JP2004213673A - Toughened reality system and method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To realize display as if a remote operator actually experiences an actual field by improving a toughened reality system in the case of performing a remote task or processing. <P>SOLUTION: This toughened reality system is provided with: a camera (19) for image acquisition movably arranged at a local site; a registration device (9) for providing a synthetic toughened reality image by generating a graphic and registering the generated graphic in an image from the camera; a display device (5) which is arranged at a remote site physically separated from the local site and displays a view including the synthetic toughened reality image; a communication link (1) for information communication between the local site and the remote site; and furthermore a specification device (7) for specifying a position and a direction at the remote site. The registration device is suitable for registering graphic display which is generated on the basis of the specified position and direction in the image, and the camera is constituted so that its position and direction are determined on the basis of the specified position and direction. <P>COPYRIGHT: (C)2004,JPO&NCIPI

Description

  The present invention provides a camera movably arranged at a local site for acquiring an image, a registration device that generates a figure, and provides a synthetic enhanced reality image by registering the generated figure in an image from the camera. A display device for displaying a view including a synthetic augmented reality image, located at a remote site physically separated from the local site, and a communication link for information communication between the local site and the remote site Reality system.

  The invention further relates to a method for remotely displaying an augmented reality view that includes graphical information superimposed on a locally acquired image.

  The systems and methods according to the present invention are suitable for remote tasks and processes performed in situations where the operator is physically remote from the task or process location. For example, the invention is useful for remote robot programming, remote maintenance and inspection, and remote collaboration.

  The invention further relates to the use of the system according to the invention for remotely programming an industrial robot.

  Augmented reality (AR) is a well-known method of overlaying computer generated information on a real world display. In the vision based on the ideal augmented reality, the user cannot recognize the difference between the real image and the computer-generated graphic, and thus can improve the recognition of the real world environment. Today, augmented reality technology is used for many applications. Augmented reality is used, for example, in the media (weather forecast) and pharmaceutical (internal organ visualization) industries, in collaborative working environments (virtual meeting rooms), and in industries that perform tasks such as maintenance and service.

  In augmented reality systems based on traditional vision, computer-generated information is added to real local scenes. Visualizing a remote environment by superimposing computer-generated graphics is hereinafter referred to as remote augmented reality. Remote augmented reality is beneficial when professionals are away from the point where they physically perform tasks, such as research in hazardous environments, and service and support tasks.

  Collaboration among users in augmented reality systems has been featured in the literature so far. This can be done by incorporating images of a plurality of remote users as graphics superimposed on the real environment. Such a system allows conference participants to visualize each other by viewing images of conference members provided in augmented reality. Further, remote users can communicate with virtual information presented in the same augmented reality environment. However, the remote user cannot see a different view of the scene at the scene from a different perspective.

  From U.S. Pat. No. 6,037,097 it is known to link augmented reality images to a computer network. The document includes one or more local stations, at least one of which must be AR, but otherwise may be AR or non-AR, and one or more remote stations, which may be AR or non-AR A real system that operates on the Internet is disclosed. Local and remote stations are networked together. The remote station can provide resources that are not available at the local AR station, such as databases, high-performance computing, and how the remote station user can correlate with the local AR station user.

  In one disclosed embodiment, a trainee is located at a local AR station and an instructor located at a remote station monitors and manages training. In a maintenance embodiment, an operator performs tasks at a local AR station, and information and assistance are located at a remote AR station. The local AR station includes a head-mounted display used by a user of the local station, a camera disposed at a fixed position with respect to the head-mounted display, and a computer having software for generating a graphic display. A video mixer that mixes the graphical display with the image to create a composite augmented reality image.

  The head-mounted display is designed to be suitable for displaying a view including a synthetic augmented reality image. The view shown on the head mounted display is based on the position and orientation of the head of the user of the local AR station. Information about the combined augmented reality is transferred to the remote station via a network connecting the local station and the remote station. The remote station has a display device suitable for displaying the transferred synthetic augmented reality. A user located at a remote station can see exactly the same view as a user wearing a head mounted display at a local station.

  Patent Literature 2 discloses an augmented reality maintenance system for processing in a hazardous environment, including an environment modeler that creates a computer model of the environment. The environment renderer creates a plurality of images each corresponding to one observation position and direction. A utility package such as a video camera is attached to the distal end of the remote control manipulator arm fixed to the fixed structure at the fixed end. The actuator moves the manipulator arm to a desired position in the environment. The position and orientation of the manipulator arm is determined by a device suitable for sensing position and orientation. This information is passed to the manipulator arm renderer, which creates images from different viewpoints from a previously stored model of the manipulator arm. The viewpoint for providing a calculated view of the environment is determined by the operator by inputting coordinates or selecting from a list of previously created viewpoint coordinates. When it is supplied to the environment renderer, the environment renderer generates an image corresponding to the shape defined by the environment geometry when viewed from the selected viewpoint.

  The resulting render, formed by combining the position and orientation of the tip of the manipulator arm, and the offset movement of the utility package allow the environment renderer to render an image corresponding to the view of the environment from the utility package perspective. Can be created. Similarly, the position and direction of the tip of the manipulator arm are supplied to the manipulator arm renderer. The manipulator arm renderer generates an image of the manipulator arm viewed from the same viewpoint as that used by the environment renderer. The video mixer superimposes the image of the manipulator arm and the image of the environment, and displays the superimposed image on a monitor. This allows the operator to visualize the position of the manipulator arm with respect to the environment.

  However, in the above system, the remote operator cannot see the local scene from different viewpoints.

The invention is particularly suitable for remote control of industrial robots. Generating high quality programs for painting, grinding and polishing, etc. requires skilled programmers with extensive processing experience. This is due to the nature of today's programming systems where processing expertise is important to meet quality requirements. Building and maintaining this capability within the end customer's organization can be both time-consuming and expensive. Costs can be saved if a remote expert supports or performs robot programming for multiple customers.
U.S. Patent Application Publication No. 2002-0010734 U.S. Pat. No. 5,745,387

  SUMMARY OF THE INVENTION It is an object of the present invention to improve an augmented reality system so that when a remote operator performs a remote task or process, it can be displayed as if the remote operator were actually experiencing the scene.

  In one aspect of the present invention, the object of the present invention is solved by the system described at the outset. The system further includes a specifying device for specifying a position and a direction at the remote site, wherein the camera is disposed at a position and a direction determined based on the specified position and the direction, and the registration device includes the specifying device for the specifying device. It is characterized in that it is configured to register a figure generated based on the specified position and direction in an image. As used herein, a local site refers to a location where a process or task is performed, and a remote site refers to a location where an operator performs the process or task.

  The system according to the invention allows the remote operator to determine the viewpoint to be displayed. The operator specifies the position and direction corresponding to the desired viewpoint, adjusts the position and direction of the camera based on the specified position and direction, and registers the figure in the image based on the specified position and direction. Is done. The system according to the invention allows the remote operator to visualize the environment of the local site and superimpose the computer-generated figures from multiple viewpoints, thus allowing the remote operator to virtually exist at the local site. can do. An advantage of the present invention is that the remote operator can view the local scene from multiple angles and can improve the perception of the environment by superimposing virtual figures. Professionals do not have to travel to local sites for service and support.

  In one embodiment of the invention, the identification device comprises a tracking device configured to determine the position and orientation of a mobile device located at a remote site with respect to a previously specified fixed remote coordinate system. Preferably, the movable device is worn by the operator or carried by the operator, thus determining the view in which the position and direction of the operator are displayed. The tracking device is a stand-alone device or is part of a mobile device. The tracking device may be any type of conventional sensor or technology used to determine position and orientation. Preferably, the movable device is a display device. The display device is preferably a hand-held display device or a head-mounted display device. Therefore, the displayed viewpoint is determined by the position and the direction of the display device. When the operator carries the display device, the actual movement of the operator controls the position and direction of the camera, and thus the displayed viewpoint.

  In a preferred embodiment of the present invention, the system further comprises a robot located at a local site, wherein the robot is equipped with a camera, and the robot has its motion relative to a pre-specified fixed local coordinate system. , Based on the specified position and direction. Preferably, the movement of the robot is determined by the movement of the mobile device. When the user wears the movable device, the robot moves according to the movement of the operator. As used herein, robot means an industrial robot. The robot includes a manipulator and a control system that controls movement of the manipulator.

  In a further embodiment of the present invention, the system comprises a graphic generator for generating a graphic display, and a registration device configured to generate the graphic based on the graphic display. It is advantageous to generate the graphic display first, and then generate the graphic based on the graphic display because the graphic display requires less bandwidth and is easy to transfer between the remote site and the local site.

  In a further embodiment of the invention, the system comprises operator input means located at a remote site for providing data relating to the graphics to be displayed to the system. By providing operator input means for supplying data relating to a graphic to be displayed, the operator can add virtual information (annotation or illustration) to the virtual environment. In hazardous or inaccessible environments, it may be necessary to introduce a robot to perform certain properties of the process or equipment. This is true in environments where nuclear radiation is present, deep seas, extreme temperature conditions, and the like. When a specialist performing such an inspection is located at a remote place, the remote enhancement system can supply a required image of the inspection object to the remote operator. This embodiment of the invention provides the remote operator with the possibility to add virtual information to the real environment and improves the operator's perception of the local scene. For example, when performing remote robot programming, the virtual information includes a programmed robot path and processing related information indicating a result of the robot processing. For remote inspection, the virtual information may include a temperature or radiological map of the actual environment or status information of the object or structure.

  According to a preferred embodiment of the present invention, the system further comprises a pointing device, and a tracking device for determining the position of the pointing device, wherein the system indicates the pointing member currently pointing based on the position of the pointing device. It is configured to generate a graphical display of a point. Preferably, the indicating device is hand-held. Such a pointing device is used, for example, by an operator at a remote site to teach a robot path. In a further embodiment, the system comprises a second pointing device and a second tracking device, wherein the second pointing device and the second tracking device are used by an operator located at the local site.

  The instruction device is particularly useful in teaching a robot program when there is no staff with the skills of robot programming at the local site where the processing is performed, or when there is no model of the object to be processed. The pointing device allows a processing professional located remotely from the robot's location to perform certain tasks on the actual target object image. By using the pointing device, a remote operator can perform a robot programming task by teaching points in the virtual real world, where the taught points are indicated by a graphical display. The remote operator can wear a head-mounted display, which can superimpose computer generated graphics on real-world graphics viewed from the camera. The computer-generated graphic represents, for example, a pointing device, processing-related information, and a programmed robot path. The pointing device is used to specify a waypoint of the robot path, and the operator input means is used to specify processing-related information to the system.

  In one embodiment of the present invention, the system includes a second specifying device for specifying a position and a direction at the local site, and generating a graphic, and displaying the generated graphic on an image of the actual environment or the environment of the local site. A second registration device for registering based on the position and the direction specified by the second specifying device, and a local display device configured to display a view including an environment of the local site and a graphic projected on the environment. And The second display device at the local site allows the on-site operator to supervise the operation of the remote operator. This embodiment of the present invention allows a remote operator and a local operator to simultaneously view the same environment overlaid with the same computer-generated graphics from their respective viewpoints. By sharing the same enhanced environment, field operators can support remote operators. The viewpoint shown to the remote operator is determined by the position and direction (for example, the position of the operator's hand or head) determined by the remote operator, and the viewpoint of the operator at the site is determined by the position determined by the local operator. Determined by the direction.

  In one embodiment of the present invention, the system includes a second mobile device located at a local site, and the second identifying device includes a second tracking device that determines a position and a direction of the second mobile device. In a preferred embodiment, the second mobile device is a local display. The local display device is, for example, a hand-held display device or a head-mounted display. This allows collaborative work between a remote operator and a local operator by describing the actual movement of the operator in a real environment with computer generated graphics. Users of the system can interact through a shared virtual environment, and both remote and local users can change perspectives.

  In one embodiment of the invention, the system further comprises a second graphic generator located at the local site for generating the second graphic display, such that the local display device displays a view including the second graphic display. It is configured. This allows the local operator to generate a graphic display and add the graphic to the displayed view. The communication link is a configuration suitable for communicating the generated graphical representation between the local site and the remote site. Thus, both the local operator and the remote operator can modify the graphic, while both operators can view the same graphic from different viewpoints. The communication bandwidth can be reduced because only the video signal from the camera and the graphic display are transferred between the local site and the remote site, rather than the graphic or the entire composite augmented reality image.

  In one embodiment of the present invention, the system comprises a second camera, located at a fixed position relative to the second mobile device, for image acquisition, and a graphic display generated on the image from the second camera. And a second registration device for registering the combined augmented reality image, wherein the local display device is configured to display a view including the combined augmented reality image. In the present embodiment, a computer-generated figure is fused to a real-world image. Alternatively, the computer-generated figure is displayed on the transmission glass worn by the operator. The generated graphic is projected on the glass, so that the operator can see the graphic for the real world and the object to be processed.

  In one embodiment of the present invention, the local display device is configured to display a view from a first viewing angle based on the position and orientation identified at the remote site. Each is configured to display the same view as the remote display from a second viewing angle based on direction.

  In one embodiment of the invention, the system comprises means for transferring audio between a local site and a remote site via a communication link. Thereby, the remote operator and the local operator can communicate by conversation. Preferably, the communication means comprises microphones or headsets at remote and local sites. Preferably, the communication link is a network such as the Internet.

  In another embodiment of the invention, the object of the invention is achieved by the method described at the outset. The method includes the steps of identifying a position and an orientation at a remote site physically distant from the local site; and determining and orienting a camera located at the local site according to the identified location and orientation; Obtaining the image from a camera, generating a graphic, generating a synthetic augmented reality image based on the image, the graphic, and the specified position and direction, and including a synthetic augmented reality image. Displaying the view.

  Yet another embodiment of the present invention is a computer program product that can be loaded directly into a computer's internal memory, the software for performing the steps of the method according to the present invention when the product is run on a computer. The object of the invention is achieved by a computer program product comprising a code part. The computer program product is provided through a computer-readable medium or a network such as the Internet.

  Yet another embodiment of the present invention is a computer readable medium having recorded thereon a program, the program being executed on a computer, the computer causing the computer to perform the steps of the method according to the present invention. The object of the invention is achieved by possible media.

  In one embodiment of the present invention, the system includes a hand-held display device including a display member and a camera. Preferably, the handheld display device is configured to give the impression that the user is looking directly through the display. The hand-held display device is effective when the user must occasionally view computer-generated information.

  According to the present invention, a task or a process (for example, generation of a robot program) can be executed from a remote place. The advantages of the present invention when used for remote robot programming are that end consumers do not need to hire and train personnel with robot programming capabilities, and that they have a group of programming professionals in a collaborative work environment. Is to improve the efficiency.

  BRIEF DESCRIPTION OF THE DRAWINGS The invention is described in more detail below with reference to several embodiments of the invention and the accompanying drawings.

  FIG. 1 illustrates an augmented reality system according to an embodiment of the present invention. The system comprises a part located at a remote site and a part located at a local site. The local part and the remote part are interconnected by a communication link 1. One or more remote operators 3 are located at remote sites. The remote unit includes a graphic display device 5 (for example, a head mounted display device), a tracking device 7 for determining the position and direction of the display device 5 with respect to a fixed remote coordinate system (xR, yR, zR), a graphic display and enhancement. The computer 9 includes software necessary for generating a real image, and a communication device 11 for causing the computer 9 to communicate with a local site. Preferably, the position and direction of the tracking device 7 are fixed with respect to the display device 5. The computer 9 is, for example, a wearable computer. Different types of hand-held displays, such as personal digital assistant (PDA) screens, can also be used.

  The remote operator 3 also has a pointing device 13 that allows the operator to indicate the location and direction of the remote site. The pointing device 13 includes a tracking device for determining the position and the direction of the pointing device 13. The position and orientation obtained from the pointing device tracking device and the display tracking device are associated with a remote coordinate system (xR, yR, zR), which is a local site local coordinate system (xL, yL, zL). It corresponds to. In a variant, the pointing device comprises a plurality of interaction means, such as push buttons, suitable for interacting with the operator and the system. The interaction means is suitable, for example, for inputting information regarding processing to the system. Alternatively, the operator supplies information to the system via a keyboard or any other known input means.

  The display tracking device 7 and the pointing device tracking device determine the position and orientation of the image recognition, accelerometer, inertial sensor, gyro, magnetic tracking device, ultrasound, laser technology, Global Positioning System (GPS), etc. Based on any prior art for.

  An object 15 to be processed or inspected is located at a local site. The local part of the system comprises one or more robot manipulators 17 and a camera device 19 mounted at one end of the robot arm for acquiring a stereoscopic or monoscopic video stream of the object 15. A tracking device for determining the position and direction of a camera mounted on the robot may be provided. Alternatively, a system for calculating the position and direction of the camera based on information from the robot control system may be installed. A robot controller 20 is connected to the robot manipulator so that the robot arm can be moved in real time according to a specified position and direction with respect to a fixed local coordinate system (xL, yL, zL). Hereinafter, the position and the direction are referred to as a pose. The local unit further includes a communication device 22 that communicates with the remote communication device 11.

  In order to use the system for a desired purpose, a fixed remote coordinate system (xR, yR, zR) and a local coordinate system (xL, yL, zL) need to be specified in advance. The pose at the remote site (7, 13) is associated with the remote coordinate system, and the pose specified at the local site is associated with the local coordinate system. Therefore, when the remote operator 3 designates a pose for the remote coordinate system, the pose of this point is defined in the local environment according to the local coordinate system.

  Any type of camera can be used, but the accuracy of the position and orientation will limit the type of camera. For example, a web camera, video camera, or CCD camera can be used. In another embodiment, one or more (eg, two) cameras arranged to provide a stereoscopic image can be used. Any known industrial robot having sufficient degrees of freedom (preferably 5 degrees of freedom) can be used. The industrial robot includes a manipulator having a plurality of arms movable with respect to each other, and a control system for controlling movement of the arms.

  In one embodiment of the present invention, the system can be operated without an operator in the local unit. In another embodiment of the invention, what the local operator does is support the remote operator. When the operator 24 is present in the local unit, the local unit includes a graphic display device 26 (for example, a head mounted display) and a camera 28 mounted on the display device for acquiring an image stream of the environment and the object 15. And a tracking device 27 for determining the position and direction of the display device 26 and the camera 28 in the local coordinate system; and generating a graphic display and an augmented reality image based on the video stream from the camera 28 and the generated graphic display. Computer 32 including software necessary for the operation.

  The camera 28 is mounted at a fixed position with respect to the display device 26. Preferably, the display device is located in the image plane of the camera along the axis of the camera view. In another embodiment, the display 26 is an optical transmissive display and the local operator 24 can view the real world environment and objects directly without a camera stream on the display, thus eliminating the need for a camera. The local unit further includes a hand-held pointing device 34, a tracking device for determining the position and direction of the pointing device with respect to the local coordinate system, and means for interacting with the system. Preferably, computer 32 is a wearable computer.

  The remote operator 3 is wearing the display device 5, and the movement of the operator is sensed by the tracking device 7. Communication between the display device 5 and the wearable computer 9 is performed via a wired or wireless link. The computer 9 includes software necessary for generating a graphic display and an augmented reality image based on the video stream from the camera 19 and the generated graphic display. The computer 9 also includes software for performing a desired task or process (for example, generating a robot program). Finally, the computer 9 includes a storage medium for saving and restoring previously saved information.

  Information communication from the pointing device 13 to the computer 9 is performed via a wired or wireless link. Depending on the type of the tracking system, the pose of the pointing device 13 can be obtained from the pointing device itself or from an external tracking device. The communication device 11 includes software and hardware necessary to enable communication from the computer 9 to the local communication device 22. Communication from the computer 9 to the communication device 11 is provided by a wired or wireless communication link. Communication between the remote communication device 11 and the local communication device 22 can be via the Internet or any other suitable communication link.

  A camera 19 is mounted on the tip of the robot 17 and transfers images of the environment and the object 15 to the remote operator 3. The robot controller 20 receives the continuously updated pose from the display tracking device 7 in the remote part, whereby the camera moves to a position according to the pose of the remote operator 3. The positions of the pointing device 34 and the display device 26 are associated with the local coordinate system. The local operator 24 wears a display 26, which provides the operator with an augmented reality view of the local environment and the object 15. When the display device 26 is of a video transmission type, the image of the local environment can be acquired using the camera 28. If the display device 26 is of the optical transmission type, no camera is required.

  The local operator 24 uses the pointing device 34 to identify poses and interact with the shared virtual environment. The position of the pointing device 34 is tracked on a local coordinate system. Computer 32 includes the software necessary to generate a local augmented reality view. Computer 32 generates graphics associated with the process or task that the remote operator has seen. Computer 32 also includes a graphics device that includes a virtual real-world graphics transfer transferred over a communication link 1 from a remote or other local unit. Therefore, there is no need to transfer graphics in video format between the local site and the remote site, which can reduce communication bandwidth requirements. Computer 32 further comprises storage used to store process related information, such as virtual graphic displays or process or task related information provided by remote or local operators. A local site may also include more local operators that interact with the system as well.

  FIG. 2 is a block diagram of an augmented reality system according to one embodiment of the present invention. The remote part of the system located at the remote site receives a video stream of images from a local camera 19 mounted on the robot. The remote operator 3 can interact with the system via the instruction and interaction device 13. The tracking device allows the pose of the pointing device to be tracked in 3D and can also be used to specify a remote coordinate system. When programming the robot, it is possible to use the pointing device 13 to specify waypoints on the robot path and to identify processing-related information, while in remote inspection, the pointing device should be used for inspection. Virtual information can be added to the object or structure.

  The output from the pointing device 13 and the information supplied to the system by the operator are transferred to the application device 36. The application device 36 includes software necessary to execute a desired process or task, such as generation of a robot program and assistance in remote inspection. The graphic device 37 includes a graphic generator that generates a 3D graphic display of the visual information shown in the application device 36. The graphic device 37 has specifications for a plurality of 3D graphic elements to be visualized. In addition to maintaining the display of the graphic from the application device 36, the graphic device 37 also includes the entire virtual graphic display obtained from another remote or local part. The positions of all graphic elements are specified with respect to a remote coordinate system. In the field of robot programming, the relevant visual information is operator-specified waypoints, actual robot paths, and task-specific information. In addition, a pose for the pointing device is provided so that the operator can obtain a visual indication of what another operator is pointing at. The generated graphic display is transferred to the registration device 38. The storage device 39 can store and load information related to the application work.

  The display following device 7 continuously transmits update data relating to the pose of the display device 5. The pause data is transferred to the communication device 11 and further transferred to the robot controller device 20. The pause data from the display following device 7 is also transferred to the registration device 38. The registration device 38 generates a graphic and registers the graphic in the image obtained from the camera device 19 based on the graphic display of the graphic device 37, thereby providing a synthetic enhanced reality image. The registration device 38 uses the pose data obtained from the display following device 7 to superimpose the computer-generated figure on the real world image. If the registration is done correctly, the computer generated graphics will be visually attached to the real world scene. The combined image is displayed on the display device 5.

  The local communication device 22 continuously receives information on the pose of the remote display device 5 on the remote coordinate system, and supplies this data to the robot controller device 20. The robot controller device 20 directs the camera 19 mounted on the robot to the same pose on the local coordinate system. The camera 19 continuously acquires images of the scene, which are returned to the communication device 22 and further to the registration device 38.

  FIG. 3 shows another embodiment of the local unit located at the local site. This embodiment shows a joint operation between a remote operator and a local operator. The communication device 22 receives specific information regarding the process specified by the remote operator, the current pose of the remote display device 5, the current pose of the remote pointing device 13, and the graphic display generated on the remote graphic device 37. . The received graphic display is transferred to the local server device 40. A camera device 28 attached to the display device 26 generates a local real-world image. The poses of the camera 28 and the display device 26 are determined according to the local coordinate system and transferred to the registration device 42 which superimposes a computer generated graphic generated from the graphic module 44 on the current real world scene. The graphics module 44 receives and displays from a remote site virtual graphics information, which is a representation of the graphics elements provided to the shared virtual environment.

  The local operator interacts with the system using the pointing and interaction device 34. The pose of the pointing device is provided with the graphical information so that it is visible in the shared virtual environment. The graphic display relating to the local operator is transferred by the communication device 22 to another operator. Information from the pointing and interaction device 34 is transferred to a local graphics module 44 that generates a graphical representation of the pointing device. This graphic display is transferred to the local registration device 42, to the communication device 22 via the server device 40, and to the remote graphic device 37 via the remote communication device 11. The remote graphic module adds the received graphic display to the graphic display generated based on the information of the remote part. In this way, the remote operator and the local operator can see the same view from different viewpoints.

  The server device 40 receives data from a remote site via the communication device 22. Further, the server device 40 holds information on the local coordinate system. The storage device 46 that communicates with the server device 40 is used to store system information (for example, application-related information, graphic display, and system configuration parameters such as a local coordinate system). One or more local operators can be subordinate to the server device 40.

  The system is configured to allow two-way audio transfer between a remote operator and a local operator via a communication link. By equipping both the local operator and the remote operator with a microphone and a headset, the operators can communicate with each other by voice.

  In the following, the use of the augmented reality system according to the present invention will be described. Both local and remote operators can initiate sessions. Alternatively, the start of the session is set in advance. First, the local coordinate system must be specified. One way to do this is the six-point calibration method, in which the operator specifies six points that define the position and direction of the coordinate system. All pose information provided by the local tracking device is associated with this local coordinate system. The position and orientation of the robot associated with this local coordinate system also need to be recognized. In this way, a point specified from a remote location can also be mapped to the robot coordinate system.

  The remote operator starts operating the system by mounting the display device. The remote operator must then specify a remote coordinate system that corresponds to the local coordinate system. The remote coordinate system can be specified in the same manner as the local coordinate system. All pose information provided by the remote tracking device is associated with a remote coordinate system. At the local site, the target object is located at a desired position in the local coordinate system.

  When the remote operator starts using the system, the movement of the remote display device is captured by the display following device and transferred to the local application server via a communication link. The local communication server then sends a command to the robot controller to move the camera to the pose specified by the remote display tracking device. Here, the position of the robot-mounted camera moves to the pose on the local coordinate system, which is the same as the position of the tracking device in the remote coordinate system.

  The pose of the remote pointing device is followed on the remote coordinate system. Pause information for the remote pointing device is transferred to the local communication device. If a local operator is present, the location of the remote operator's pointing device is used to provide computer-generated graphics on the local operator's local display. Therefore, the local operator can visualize the movement of the pointing device of the remote operator in real time using the virtual 3D graphic superimposed on the actual scene.

  At this stage, the remote user can execute a desired process (for example, generation of a robot program). While performing a particular task, process related information is visualized on a remote display along with computer generated 3D graphics. The real-world display from the local camera is superimposed on the figure and fixed with respect to the remote coordinate system. As a result, the computer-generated graphic is added to the image of the target object, and the remote user can virtually move around the target object. The movement of the remote operator is captured and transferred to the camera equipped with the local robot so that the camera equipped with the local robot moves according to the movement of the remote operator.

  If the remote user performs some off-line tasks (e.g., generating a robot path program), the local operator can also view the processing results as a remote operator by displaying the 3D computer generated graphics on the local display. Can be Processing figures are registered and superimposed on the real-world display as seen by the local operator, so when the local operator moves to a new position, the computer-generated processing figures are fixed in the local coordinate system and visualized. Is done.

  If the local operator is also wearing a pointing and interaction device, the location of the local pointing device is visualized to the local and remote operators. The position of the local pointing device is tracked on a local coordinate system. Pause information for the local pointing device is transferred to the remote site via the local and remote communication devices. The pose information of the local operator's pointing device is used to display computer generated graphics on the remote display and the local display. In this way, the remote operator can visualize the movement of the pointing device of the local operator. Even when there are more remote operators, each remote operator can visualize the movement of the pointing device of another remote operator.

  The software used to perform the method according to the invention is based in part on software known to those skilled in the art. For example, the position and the direction of the pointing member can be generated by the ARTToolKit. ARTToolKit was developed by the University of Washington and Hiroshima University and is a public software library that enables the construction of augmented reality applications using tracking technology based on accurate computer vision. For the application interface, Open GL software can be used. Open GL provides a library of 2D and 3D functions, including model correction, color, light and shading functions. Microsoft Vision SDK is a library of writing programs that perform image manipulation and analysis on a computer. Augmented reality software includes algorithms for drawing figures such as points and lines, transfer of positions and directions between different coordinate systems, extraction and generation of position and direction sequence lists, acquisition of processing-related information, and The figure includes a tip figure in the drawing, such as a painting point and a line representing painting in the direction.

  The invention is not limited to the disclosed embodiments, but can be varied and modified within the scope defined by the claims. For example, in one modified example, a plurality of cameras, a plurality of display devices, a plurality of specific devices, a plurality of graphic generators, and a plurality of registration devices are installed at a remote site, and two or more remote operators are installed. At the same time, it allows you to see the environment of the local site and the superimposed figures from your own view. In another modified embodiment, a plurality of cameras, a plurality of display devices, a plurality of specific devices, a plurality of graphic generators, and a plurality of registration devices are installed at a local site, and two or more local operators are installed. At the same time, it is possible to see the environment of the local site and the figure superimposed on it from multiple viewpoints.

  In one variation, the operator supplies the position and orientation to the system. For example, an operator specifies a position and a direction with one or more joysticks. In this embodiment, the display is a stationary computer screen.

  In one embodiment, the system comprises a handheld augmented reality display. The operator carries a hand-held display device that displays the current real world with computer-generated graphics superimposed. The computer generated graphic displays processing related information such as programmed robot path, processing specific information, target points and tasks. Either using the movement of the operator's hand with a motion-based recognition system or using an instruction and interaction device, the processing-related information is specified to the system.

  Computer-generated information indicates input by an operator for robot programming. The operator observes the result of his / her work while creating a new robot program using the hand-held AR display. Handheld displays have cameras with the same field of view as the display. The camera acquires live video of the real world. The system combines and synchronizes the live video with a computer-generated graphic that displays operator input and displays it on a handheld AR display. Since the operator can freely move the hand-held AR display in the environment, the computer-generated figure can be superimposed on the local environment including the object viewed from different viewpoints. The operator can see the robot program generated for the actual target object by "looking into" the opposite side via the hand-held AR display.

  The system according to the invention is shown in FIG. The system comprises a hand-held interaction / indication device 1, which is provided with a tracking system for determining the position and orientation of the interaction / indication device on an absolute coordinate system 60. I have. Alternatively, the system comprises a motion-based recognition system having a recognition system for recognizing and determining the position and orientation of a hand or finger in an absolute coordinate system. The system further includes a handheld display 62 (eg, a miniature PC or PDA). The display device includes a display member 64 and a camera 8 mounted or built in the display device 64 for acquiring an image stream of the environment. The camera 8 is installed at a fixed position with respect to the display device 64. The display device is located in the camera image plane along the camera view axis.

  The system further comprises a tracking system 66 for determining the position and orientation of the display device. Preferably, the tracking system is mounted on a display device. The system also includes a system for generating an augmented reality display of the computer generated graphic information superimposed on the real world display. The movement of the operator 2 is sensed by the display following system 66.

  The wearable computer 68 includes the software necessary to create an augmented reality environment based on the video stream from the camera 8 and the computer generated graphics. The wearable computer 68 also includes the necessary software to perform the desired task or process (eg, create a robot program and verify reachability). Further, the wearable computer generates a graphic that provides an augmented reality view. Finally, the wearable computer includes a storage medium for saving and restoring previously saved information. Communication of information from the interaction / instruction device 1 to the wearable computer 68 is via a wired or wireless link. The operator carries the display device 62, thereby viewing an augmented reality view of the environment. The display is of the "video transparent" type.

  Video transparency is used to create and display augmented real world on handheld AR displays. A camera built into the display device is used to obtain live video of the real world. The camera is positioned with respect to the display so as to provide the same view as the user looks through the display at the other side. The live video stream combining the computer generated graphics is displayed on a display device in real time. Additional features include camera zooming with the output of the actual camera focal length. This allows the system to accurately display computer generated graphics while changing magnification. If vision-based tracking is used for the tracking system, a camera can also be used for vision-based tracking.

1 shows an augmented reality system according to an embodiment of the present invention. FIG. 2 is a block diagram of an augmented reality system including a local part and a remote part according to an embodiment of the present invention; FIG. 7 is a block diagram of a local part of an augmented reality system according to another embodiment of the present invention. 1 shows an embodiment of the invention with a hand-held display device.

Explanation of reference numerals

DESCRIPTION OF SYMBOLS 1 Communication link 3 Remote operator 5 Graphic display device 7 Tracking device 9 Computer 11 Remote communication device 13 Pointing device 15 Object 17 Manipulator 19 Camera 20 Robot controller 22 Local communication device 24 Local operator 26 Display device 27 Tracking device 28 Camera 32 Computer 34 Handheld pointing device

Claims (33)

A camera (19) movably arranged at a local site for image acquisition;
A registration device (38) for generating a graphic and registering the generated graphic with an image from a camera to provide a synthetic augmented reality image;
A display device (5), located at a remote site physically separated from the local site, for displaying a view including the synthetic augmented reality image;
A communication link (1) for information communication between a local site and a remote site,
The apparatus further includes a specifying device (7) for specifying a position and a direction at the remote site, wherein a position and a direction of the camera are determined based on the specified position and direction, and the registration device determines the position and the direction of the camera. An augmented reality system configured to register a figure generated based on a direction and an image in an image.   The specific device comprises a tracking device (7) configured to determine the location and orientation of a mobile device (5) located at a remote site, and the registration device (38) is based on the location and orientation of the mobile device. 2. The method according to claim 1, wherein the generated graphic is configured to be registered in the image, and a position and an orientation of the camera are determined based on a position and an orientation of the movable device. 3. system.   The system according to claim 2, wherein the mobile device is a display device (5).   The robot (17, 20) located at a local site, further comprising a camera, wherein the movement of the robot is determined based on the specified position and direction. The system according to any one of claims 3 to 3.   5. A graphic generator according to claim 1, further comprising a graphic generator for generating a graphic display, wherein said registration device is configured to generate a graphic based on said graphic display. A system according to any one of the preceding claims.   There is further provided an operator input means (13) located at a remote site for providing data relating to the graphic to be displayed to the system, the system being configured to generate the graphic based on the data. The system according to any one of claims 1 to 5, characterized in that:   The operator input means includes a pointing device (13) and a follow-up device for determining the position of the pointing device, and the system determines the position of the point currently pointed to by the pointing member based on the position of the pointing device. 7. The system of claim 6, wherein the system is configured to generate a graphical display.   A second specifying device (27) for specifying a position and a direction at the local site, and generating a graphic; converting the generated graphic into an image of an actual environment or an environment of the local site; ), A second registration device (42) for registering based on the position and direction specified by the method, and a view including the environment of the local site and the generated graphic projected on the environment. System according to one of the preceding claims, characterized in that it comprises a local display (26).   A second mobile device (26) located at the local site, wherein the second identifying device comprises a second tracking device (27) for determining the position and direction of the second mobile device, The system according to claim 8.   The system according to claim 9, wherein the second mobile device is a local display (26).   A second camera (28) for image acquisition, which is fixed to the second movable device (26), is provided. The second registration device (42) transmits the generated figure to the second camera (28). 28. The method of claim 28, wherein the local display device is configured to display a view including the synthetic augmented reality image by registering with the image from 28). The system according to 9 or 10.   The local display device (26) is configured to display the view viewed from a first viewing angle based on the position and direction received from the first specifying device (7). 12. The display device according to claim 8, characterized in that it is configured to display the same view as the remote display device (5) from a second viewing angle based on the position and direction received from (27). A system according to any one of the preceding claims.   The system according to any of the preceding claims, comprising means for transferring audio between a remote site and a local site via a communication link.   14. The system according to claim 1, wherein the communication link is a network. -Determining the location and orientation at a remote site physically separated from the local site;
-Locating and orienting the camera (19) located at the local site according to the specified location and orientation;
-Obtaining an image from the camera;
-Generating the figure;
-Generating a composite augmented reality image based on the image, the graphic, and the specified position and orientation;
Displaying a view including the synthetic augmented reality image at a remote location, the method including a step of displaying the view including the synthetic augmented reality image.   The step of determining the position and direction includes determining the position and direction of a mobile device (5) located at a remote site, and determining the position and direction of the camera based on the position and direction of the mobile device (5). The method of claim 15, wherein   17. The method according to claim 16, wherein the mobile device is a remote display (5) and displays a view including the synthetic augmented reality image on the remote display.   18. The method according to any one of claims 15 to 17, comprising controlling the movement of the robot (17) and mounting a camera (19) on the robot according to the position and orientation of the mobile device (5). Method.   The method according to any one of claims 15 to 18, comprising a step of obtaining data relating to a graphic to be displayed, and a step of generating a graphic based on the data.   20. Receiving information on the position of the pointing device (13) and generating a graphic representing the point currently pointed to by the pointing member based on the position of the pointing device (13). The method according to any one of claims 1 to 4.   Specifying the position and direction at the local site, and displaying the generated graphic in a state of being projected on the environment based on the position and direction specified at the local site, together with the environment of the local site. 21. The method according to any one of claims 15 to 20, comprising:   22. The method of claim 21, wherein determining the location and orientation at the local site comprises determining the location and orientation of the second mobile device (26) located at the local site.   23. The method of claim 22, wherein the second mobile device is a local display (26), and the second view including the local site environment and graphics is displayed on the local display.   Step of acquiring an image from a second camera (28) fixedly arranged with respect to the second movable device (26), and synthesizing by registering a figure generated in the image from the second camera (28) 24. The method according to claim 22 or 23, comprising providing a real image and displaying a view including the composite augmented reality image on a local display (26).   22. The method of claim 21, further comprising: generating a second graphic; and displaying a second view including an environment of the local site and a projection of the second graphic on the environment based on the specified position and orientation. 25. The method according to any one of claims 24.   Generating a local graphic display, generating a remote graphic display, transferring the local and remote graphic displays between a local site and a remote site, and generating the first graphic based on the local and remote graphic displays 26. The method of claim 25, comprising: generating the second graphic based on local and remote graphic displays.   The view displayed on the remote site includes the environment of the local site overlaid with the figure viewed from the viewing angle based on the position and direction specified at the remote site.The view displayed on the local site includes the environment of the local site. 27. The method of claims 21 to 26, wherein the method includes superimposing a figure viewed from a viewing angle based on the position and direction specified at the local site.   28. A computer program product that can be loaded directly into a computer's internal memory, the software code portion for performing the process of any one of claims 15 to 27 when the product is executed on a computer. Computer program products, including:   28. A computer-readable medium having a program recorded thereon, wherein the computer-readable medium, when executed on a computer, causes the computer to perform the process according to any one of claims 15 to 27.   A method of using the system according to any one of claims 1 to 14 for remote programming of an industrial robot.   The system according to any one of claims 11 to 14, characterized in that it comprises a hand-held display (62) comprising a display member (64) and a camera (8).   The system of claim 31, wherein the hand-held display device is configured to give an impression as if the user was looking directly through the display.   A method of using the system according to any one of claims 1 to 13 for painting.

Images