WO2020235539A1 - Procédé et dispositif pour spécifier la position et la posture d'un objet - Google Patents

Procédé et dispositif pour spécifier la position et la posture d'un objet Download PDF

Info

Publication number
WO2020235539A1
WO2020235539A1 PCT/JP2020/019689 JP2020019689W WO2020235539A1 WO 2020235539 A1 WO2020235539 A1 WO 2020235539A1 JP 2020019689 W JP2020019689 W JP 2020019689W WO 2020235539 A1 WO2020235539 A1 WO 2020235539A1
Authority
WO
WIPO (PCT)
Prior art keywords
model
robot
virtual world
orientation
virtual
Prior art date
Application number
PCT/JP2020/019689
Other languages
English (en)
Japanese (ja)
Inventor
ロクラン ウィルソン
パーベル サフキン
Original Assignee
株式会社エスイーフォー
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 株式会社エスイーフォー filed Critical 株式会社エスイーフォー
Publication of WO2020235539A1 publication Critical patent/WO2020235539A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras

Definitions

  • the present invention relates to a method and a device for specifying the position and orientation of an object.
  • Patent Document 1 describes an image feature extraction device that estimates the position and orientation of an object by probabilistically obtaining the range in which the object exists in the captured image from the image features and then performing detailed collation with the model. It is disclosed.
  • Patent Document 2 describes a distance sensor that measures the distance to an object existing in the work environment and generates range data, and the shape and position / orientation of the object by collating the generated range data with the shape data.
  • the initial collation unit that estimates, the camera unit that generates image data that captures the image of the object, the feature extraction unit that extracts the features of the image data, and the shape and position / orientation of the object estimated by the initial collation unit as initial values.
  • a robot device equipped with a posture estimation unit that estimates the position and orientation of an object by solving a minimization problem using the feature of the image data extracted by the feature extraction unit and the error between the shape and position and orientation of the object as an evaluation function. It is disclosed.
  • Non-Patent Document 1 the PnP problem (the problem of estimating the position and orientation of the calibrated camera from the three-dimensional coordinates of n points in the world coordinate system and the image coordinates in which those points are observed) is less calculated.
  • Patent Document 1 the range in which an object exists in the captured image is stochastically obtained from the image features. In the method of Patent Document 1, it is necessary to calculate the existence probability for the entire image, so that a high calculation cost is required.
  • Patent Document 2 the position and orientation of an object are used as initial values, using the shape and position and orientation of the object estimated by collating the range data generated by measuring the distance to the object existing in the work environment with the shape data. Is estimated. Since the method of Patent Document 2 is premised on measuring the distance to an object and generating range data, it cannot be used under the condition that there is no distance measuring means or distance information cannot be obtained.
  • the method of recognizing an object in the captured image may not be able to correctly extract the feature points of the object if the quality of the captured image is low. In that case, the accuracy of collation between the object and the model is lowered, and the position / orientation of the object cannot be recognized correctly.
  • it is a method of specifying the position and orientation of an object to generate a virtual world including a display of an object in the real world and to display a model corresponding to the object in the virtual world. And in a virtual world, methods are provided that include overlaying a model on an object and comparing the object to the model to identify the position and orientation of the object.
  • it is a device for specifying the position and orientation of an object to generate a virtual world including a display of an object in the real world, and to display a model corresponding to the object in the virtual world.
  • a device with a processor configured to perform, superimpose a model on an object, and compare the object to the model to determine the position and orientation of the object in a virtual world.
  • Fig. (A) shows how to give an operation instruction to a virtual mug which is a virtual object in the virtual world
  • Fig. (B) shows how to grasp the mug in the real world with the robot hand of a robot in the real world according to the instruction. It shows how to move it.
  • FIG. 1 is a block diagram showing an embodiment of a robot control system.
  • FIG. 2 is a diagram showing a schematic configuration of an embodiment of a robot.
  • the robot control system 1 includes a robot 100, a control unit 200 that controls the robot 100, and a control device 300 that controls the control unit 200.
  • the robot 100 disclosed in the present embodiment includes at least two robot arms 120, a robot housing 140 that supports the robot arms 120, and the surrounding environment of the robot 100. It is provided with an environment sensor 160 that senses the above and a transmission / reception unit 180.
  • Each robot arm 120 in the present embodiment is, for example, a 6-axis articulated arm (hereinafter, also referred to as “arm”), and a robot hand (hereinafter, also referred to as “hand”) which is an end effector at the tip thereof. It has 122.
  • the robot arm 120 includes an actuator (not shown) having a servomotor on each rotation axis. Each servomotor is connected to the control unit 200 and is configured to control its operation based on a control signal sent from the control unit 200.
  • a 6-axis articulated arm is used as the arm 120, but the number of axes (the number of joints) of the arm can be appropriately determined according to the application of the robot 100, the functions required thereof, and the like.
  • the two-fingered hand 122 is used as the end effector, but the present invention is not limited to this, and for example, a robot hand having three or more fingers and a means of attracting by magnetic force or negative pressure are provided.
  • a robot hand equipped with a gripping means that applies the jamming (clogging) phenomenon of powder or granular material filled in a rubber film, or a robot hand that can repeatedly grip and release the gripping object by any other means. can be used. It is preferable that the hands 122a and 122b are configured to be rotatable around the wrist portion thereof.
  • the hand 122 is equipped with a kinetic sensor that detects the amount of displacement of the hand 122, the force, acceleration, vibration, etc. acting on the hand 122. Further, it is preferable that the hand 122 is provided with a tactile sensor that detects the gripping force and the tactile sensation of the hand 122.
  • the robot housing 140 may be installed in a state of being fixed on a mounting table (not shown), or may be installed on the mounting table so as to be rotatable via a rotation drive device (not shown). May be good.
  • the robot housing 140 is rotatably installed on the mounting table, the working range of the robot 100 can be expanded not only to the area in front of the robot 100 but also to the area around the robot 100.
  • the robot housing 140 can be used for vehicles, ships, submersibles, helicopters, drones, and other moving objects equipped with a plurality of wheels and endless tracks, depending on the application and usage environment of the robot 100. It may be mounted, or the robot housing 140 may be configured as part of such a moving body.
  • the robot housing 140 may have two or more legs as walking means. When the robot housing 140 has such a moving means, the working range of the robot 100 can be made wider. Depending on the application of the robot 100, the robot arm 120 may be directly fixed to a mounting table or the like without the intervention of the robot housing 140.
  • the environment sensor 160 senses the surrounding environment of the robot 100.
  • the surrounding environment includes, for example, electromagnetic waves (including visible light, invisible light, X-ray, gamma ray, etc.), sound, temperature, humidity, wind velocity, atmospheric composition, etc. Therefore, the environmental sensor 160 is a visual sensor, X-ray.
  • -It may include, but is not limited to, gamma ray sensors, auditory sensors, temperature sensors, humidity sensors, wind velocity sensors, atmospheric analyzers, and the like.
  • the environment sensor 160 is shown to be integrated with the robot 100 in the figure, the environment sensor 160 does not have to be integrated with the robot 100.
  • the environment sensor 160 may be installed at a position away from the robot 100, or may be installed on a moving body such as a vehicle or a drone. Further, the environment sensor 160 preferably includes a GPS (Global Positioning System) sensor, an altitude sensor, a gyro sensor, and the like. Further, the environment sensor 160 is used as a position detection means for detecting the position of the robot 100 outdoors or indoors. In addition to the GPS sensor, WiFi positioning, beacon positioning, self-contained navigation positioning, geomagnetic positioning, sonic positioning, UWB (Ultra Wideband) Band: Ultra-wideband) It is preferable to have a configuration for positioning, visible light / invisible light positioning, and the like.
  • GPS Global Positioning System
  • the visual sensor for example, a 2D camera, a depth sensor, a 3D camera, an RGB-D sensor, a 3D-LiDAR sensor, a Kinect TM sensor, and the like can be used.
  • the visual information obtained by the environment sensor 160 is sent to the control unit 200 and processed by the control unit 200.
  • Other environmental information obtained by the environment sensor 160 can also be transmitted to the control unit 200 and used for analysis of the surrounding environment of the robot 100.
  • the transmission / reception unit 180 transmits / receives signals / information to / from the control unit 200.
  • the transmission / reception unit 180 can be connected to the control unit 200 by a wired connection or a wireless connection, and therefore, transmission / reception of those signals / information can be performed by a wired or wireless connection.
  • the communication protocol, frequency, and the like used for transmitting and receiving those signals and information can be appropriately selected according to the application, environment, and the like in which the robot 100 is used.
  • the transmission / reception unit 180 may be connected to a network such as the Internet.
  • control unit 200 in the robot control system 1 of the present embodiment will be described.
  • control unit 200 of the system 1 includes a processor 220, a storage unit 240, and a transmission / reception unit 260.
  • the processor 220 mainly controls the drive unit and the sensor (both not shown) of the robot arm 120 and the body 140 of the robot 100, controls the environment sensor 160, processes information transmitted from the environment sensor 160, and interacts with the control device 300. It controls the operation and transmission / reception unit 260.
  • the processor 220 is composed of, for example, a central processing unit (CPU), an application specific integrated circuit (ASIC), an embedded processor, a microprocessor, a digital signal processor (DSP), or a combination thereof.
  • the processor 220 may be composed of one or more processors.
  • the processor 220 receives, for example, a control signal of the robot 100 sent from the control device 300, an operation command generated in response to the control signal, an operation of the robot 100 actually executed, and an environment sensor 160 after the operation is executed.
  • the collected ambient environment data is stored in the storage unit 240 as data, and machine learning is executed using the data to generate learning data and store it in the storage unit 240.
  • the processor 220 can generate an operation command by deciding an operation to be executed by the robot 100 based on the control signal of the robot 100 transmitted from the control device 300 from the next time onward with reference to the learning data. ..
  • the control unit 200 of the robot 100 in the real world has a machine learning function locally.
  • the storage unit 240 is a computer program for controlling the robot 100, a computer program for processing information transmitted from the environment sensor 160, and a computer for interacting with the control device 300 as described in the present embodiment.
  • -A program, a computer program for controlling the transmission / reception unit 260, a program for executing machine learning, and the like are stored.
  • the storage unit 240 stores software or a program that causes a computer to perform a process as described in this embodiment to cause a function as a control unit 200.
  • the storage unit 240 stores a computer program that can be executed by the processor 220, including instructions that implement the methods described below with reference to FIG. 3 and the like.
  • the storage unit 240 stores the data of the model of the known object as described above. Further, the storage unit 240 includes the state of each part (servo (not shown), hand 122, etc.) of the robot arm 120 of the robot 100, information transmitted from the environment sensor 160, information sent from the control device 300, control signals, and the like. It also has the role of storing at least temporarily. Further, as described above, the storage unit 240 also has a role of storing the operation instruction of the robot 100 and the operation and learning data of the robot 100 executed in response to the operation instruction. The storage unit 240 preferably includes a non-volatile storage medium that retains the storage state even when the power of the control unit 200 is turned off.
  • a hard disk drive HDD
  • SSD solid-state storage device
  • compact storage unit 240 Optical disk storage such as disk (CD), digital versatile disk (DVD), Blu-ray disk (BD), non-volatile random access memory (NVRAM), EPROM (Rrasable Programmable Read Only Memory), non-volatile storage such as flash memory Is equipped with.
  • the storage unit 240 may further include volatile storage such as a static random access memory (SRAM), but each computer program described above is a non-volatile (non-temporary) storage medium of the storage unit 340. Is remembered in.
  • the transmission / reception unit 260 transmits / receives signals / information to / from the robot 100 and transmits / receives signals / information to / from the control device 300.
  • the control unit 200 can be connected to the robot 100 by a wired connection or a wireless connection, and therefore, transmission and reception of those signals / information can be performed by a wired or wireless connection.
  • the communication protocol, frequency, and the like used for transmitting and receiving those signals and information can be appropriately selected according to the application, environment, and the like in which the robot 100 is used.
  • the transmission / reception unit 260 may be connected to a network such as the Internet.
  • the transmission / reception unit 260 transmits / receives signals / information to / from the control device 300.
  • the control unit 200 can be connected to the control device 300 by a wired connection or a wireless connection, and therefore, transmission and reception of those signals / information can be performed by a wired or wireless connection.
  • the communication protocol, frequency, and the like used for transmitting and receiving those signals and information can be appropriately selected according to the application, environment, and the like in which the robot 100 is used.
  • control unit 200 is shown as being independent of the robot 100 in FIG. 1, it is not limited to that form.
  • the control unit 200 may be provided in the housing 140 of the robot 100.
  • the number of robots 100 used in this system 1 is not limited to one, and a plurality of robots 100 may be operated independently or in cooperation with each other. In this case, a single control unit 200 may control a plurality of robots 100, or a plurality of control units 200 may cooperate to control a plurality of robots 100.
  • control device 300 in the robot control system 1 of the present embodiment will be described.
  • the control device 300 of the system 1 includes a processor 320, a storage unit 340, an input device 350, a transmission / reception unit 360, and a display 370.
  • the processor 320 mainly controls the interaction with the control unit 200, the processing based on the input performed by the user via the input device 350, the control of the transmission / reception unit 360, and the display of the display 370.
  • the processor 320 generates a control signal based on the user input input by the input device 350 and transmits it to the control unit 200.
  • the processor 220 of the control unit 200 Based on the control signal, the processor 220 of the control unit 200 generates one or a plurality of operation commands for operating each drive unit (not shown) of the robot arm 120 and the body 140 of the robot 100 and the environment sensor 160. ..
  • the processor 320 is composed of, for example, a central processing unit (CPU), an application specific integrated circuit (ASIC), an embedded processor, a microprocessor, a digital signal processor (DSP), or a combination thereof.
  • the processor 320 may be composed of one or more processors.
  • the processor 320 may be composed of one or more processors.
  • the processor 320 of the control device 300 is configured to generate a UI (user interface) screen to be presented to the user and display it on the display 370.
  • the UI screen (not shown) includes, for example, a selection button that hierarchically provides the user with a plurality of options.
  • the processor 320 generates an image or a moving image of a virtual world (simulation space) based on an image or a moving image of the real world of the surrounding environment of the robot 100 taken by the environment sensor 160 of the robot 100, and displays it on the display 370.
  • the processor 320 When the processor 320 generates an image or a moving image of a virtual world based on an image or a moving image of the real world, for example, by associating a coordinate system of the real world with a coordinate system of the virtual world, the processor 320 connects the real world and the virtual world. Build a correlation. Further, the image or moving image of the real world and the image or moving image of the virtual world (simulation space) may be displayed on the display 370 at the same time. Further, the UI screen may be superposed on the image or moving image of the surrounding environment of the robot 100 or the image or moving image of the virtual world.
  • the virtual world (simulation space) image or moving image generated based on the real world image or moving image of the surrounding environment of the robot 100 also includes an object existing in the surrounding environment of the robot 100.
  • the processor 320 By building a correlation between the real world and the virtual world as the processor 320 generates a virtual world image or video based on the real world image or video, the user in the virtual world, as described in detail below. It is possible to make a change in the real world based on the operation of, and to reflect the change in the real world in the virtual world.
  • the processor 320 of the control device 300 obtains or generates a model corresponding to an object included in an image or a moving image in a virtual world (simulation space).
  • a model corresponding to an object included in an image or a moving image in a virtual world (simulation space).
  • the mode of "obtaining the ready-made information of the model” of 1) deals with the case where the model corresponding to the scanned object is available.
  • the processor 320 of the control device 300 refers to the model of the object stored in the storage unit 340 or the model of the object existing on the network to which the control device 300 is connected to provide visual information. Get the model corresponding to the object contained in it.
  • the aspect of 2) "scanning an object to generate a model” can be dealt with when a model corresponding to the scanned object is not available.
  • the user uses the input device 350 to combine various primitive shape elements in the UI screen to generate a model corresponding to the scanned object.
  • Primitive shape elements include, for example, elements such as prisms of arbitrary angle, pyramids of arbitrary angle, cylinders, cones, and spheres.
  • the processor 320 may allow the user to draw an arbitrary shape in the UI screen and add it as a primitive shape element.
  • the user selects these various shape elements in the UI screen, changes the dimensions of each part of the selected shape elements as appropriate, and combines those elements according to the image of the scanned object to correspond to the scanned object. You can generate a model to do. When generating a model using these elements, it is also possible to represent dents and holes of objects.
  • the user combines various primitive shape elements in the UI screen to generate a model of an arbitrary object without scanning the object. It is possible to deal with the case.
  • the user's operation for generating the model in this aspect is the same as the operation described in the above aspect 2).
  • the model can be used as the corresponding object in the virtual world. It becomes possible to associate.
  • the processor 320 reproduces the model corresponding to the object by, for example, computer graphics, and displays it in the virtual world on the display 370.
  • the data of the model obtained from the outside or the model generated as described above is stored in the storage unit 340 of the control device 300.
  • the storage unit 340 is interactively performed by the user on the UI screen via a program for causing the processor 320 to execute the operation described in the present embodiment, a computer program that interacts with the control unit 200, and an input device 350. It stores a computer program that performs processing based on input, a computer program that controls the transmission / reception unit 260, a computer program that displays the display 370, and the like.
  • the storage unit 340 stores software or a program that causes the computer to perform an operation described later to generate a function as the control device 300.
  • the storage unit 340 stores a computer program that can be executed by the processor 320, including instructions that implement the methods described below with reference to FIG. 4 and the like.
  • the storage unit 340 includes an image or moving image of the surrounding environment of the robot 100 taken by the environment sensor 160 of the robot 100 and sent to the control device 300 via the control unit 200, and an image or moving image of the surrounding environment of the robot 100. It is possible to at least temporarily store an image or moving image of a virtual world (simulation space) generated by the processor 320 based on the moving image.
  • the storage unit 340 of the control device 300 is also preferably composed of a non-volatile storage medium that retains the storage state even when the power of the control device 300 is turned off. For example, a hard disk drive (HDD) or a solid-state storage device.
  • the storage unit 340 may further include volatile storage such as a static random access memory (SRAM), but each computer program described above is a non-volatile (non-temporary) storage medium of the storage unit 340. Is remembered in.
  • SRAM static random access memory
  • the storage unit 340 also functions as a database of the system 1, and as described in relation to the concept of the present invention, the operation data of the robot 100 in the real world (the operation generated by the control unit 200) operated based on the control signal. (Including commands) and the ambient environment data indicating the operation result detected by the environment sensor 160 is stored.
  • the input device 350 for example, a keyboard, a mouse, a joystick, or the like can be used, and further, a device called a tracker that can track the position and posture using infrared rays or the like and has a trigger button or the like is used. You can also.
  • the display 370 includes a touch panel type display device, the touch panel can be used as an input device.
  • the display 370 is a head-mounted display used as a display device for VR (virtual reality), AR (augmented reality), MR (mixed reality), etc., and has a user's line-of-sight tracking function
  • the line-of-sight tracking function can be used as an input device.
  • a device having a line-of-sight tracking function but not a display can use the line-of-sight tracking function as an input device.
  • a voice input device can also be used as an input device.
  • the input device 350 exemplified as an example of the input device 350, and the means that can be used for the input device 350 is not limited to these. Further, the above-mentioned means may be arbitrarily combined and used as the input device 350.
  • the user can select, for example, select a selection button, input characters, or take a picture of the robot by the environment sensor 160 of the robot 100 on the UI screen displayed on the display 370.
  • the transmission / reception unit 360 transmits / receives signals / information to / from the control unit 200.
  • the control device 300 can be connected to the control unit 200 by a wired connection or a wireless connection, and therefore, transmission and reception of these signals / information can be performed by a wired or wireless connection.
  • the communication protocol, frequency, and the like used for transmitting and receiving the signal and information can be appropriately selected according to the application and environment in which the system 1 is used.
  • the transmission / reception unit 360 may be connected to a network such as the Internet.
  • the display 370 is used as a display device such as a display monitor, a computer / tablet device (including a device equipped with a touch panel type display), VR (virtual reality) / AR (augmented reality), or MR (composite reality). Any form of display device such as a head-mounted display or a projector can be used.
  • the head-mounted display when a head-mounted display is used as the display 370, the head-mounted display provides an image or a moving image in which the left and right eyes of the user have parallax, thereby causing the user to perceive a three-dimensional image or moving image. Can be done. Further, when the head-mounted display has a motion tracking function, it is possible to display an image or a moving image according to the position and direction of the head of the user wearing the head-mounted display. Furthermore, when the head-mounted display has a user's line-of-sight tracking function as described above, the line-of-sight tracking function can be used as an input device.
  • the processor 320 of the control device 300 is an image of a virtual world (simulation space) based on a real space image or a moving image of the surrounding environment of the robot 100 taken by the environment sensor 160 of the robot 100.
  • a head-mounted display that generates a moving image and is used as a display device for VR (virtual reality), AR (augmented reality), MR (mixed reality), etc. is used as the display 370, and infrared rays or the like is used as the input device 350.
  • VR virtual reality
  • AR augmented reality
  • MR mixed reality
  • FIG. 3 is a flowchart illustrating a method of specifying the position / posture of the object in the present embodiment.
  • 4 and 5 are diagrams showing objects in the virtual world and corresponding models.
  • the ambient environment information of the robot 100 in the real world obtained by the environment sensor 160 of the robot 100 is transmitted to the control device 300 via the control unit 200.
  • the visual information may be a single still image, a plurality of images, or a moving image, and preferably includes depth information.
  • the control device 300 may store the transmitted visual information in the storage unit 340.
  • the processor 320 of the control device 300 generates a virtual world (simulation space) that reproduces the surrounding environment of the robot 100 based on the visual information of the control device 300. It is displayed on the display 370.
  • a virtual world in addition to the scenery around the robot 100 in the real world, objects in the real space existing at least in an area accessible to the robot 100 are displayed.
  • the object may be a two-dimensional or three-dimensional image of a real-world object obtained by a visual sensor, a depth map, a point cloud, or the like. Alternatively, it may be represented by computer graphics that represent the object.
  • bolts FIG. 4 (a)
  • nuts FIG. 5 (a)
  • the user uses the input device 350 of the control device 300 to select a model corresponding to the target object for which the position / posture is specified from the menu on the screen displayed on the display 370.
  • the processor 320 of the control device 300 reads the data of the model from the storage unit 340 and displays the model in the virtual world displayed on the display 370 (step S315 in FIG. 3).
  • the bolt model 40 (FIG. 4 (b))
  • the nut model 50 (FIG. 5 (b)) are displayed in the virtual world.
  • the model can be obtained or obtained by three modes: 1) obtaining information on a ready-made model, 2) scanning an object to generate a model, and 3) creating a model of the object independently. Will be generated.
  • step S320 of FIG. 3 the user superimposes the model generated as described above on the corresponding object in the virtual world.
  • the user uses the input device 350 to select the model 40 shown in FIG. 4B on the screen displayed on the display 370 and move the model 40 within the screen.
  • the objects (FIG. 4 (a)) shown in the same screen are overlapped so that their positions and postures substantially match.
  • the processor 320 compares the three-dimensional shape data (for example, the edge and / or the feature point of the three-dimensional shape) of the object (FIG. 4A) with the three-dimensional shape data of the model 40, and the object (FIG. 4).
  • the position and orientation of the model 40 are corrected so that the contour of the three-dimensional shape of (a)) and the contour of the three-dimensional shape of the model 40 match.
  • the object displayed in the virtual world (FIG. 4A) is not a mere object occupying a certain volume space, but an object (bolt) corresponding to the model 40 selected by the user.
  • the method for comparing the object and the model is not limited to the above, and any other method can be used.
  • the processor 320 of the control device 300 recognizes that the object (FIG. 4A) is the object corresponding to the model 40, and sets the virtual object corresponding to the object specified as described above, for example. It is reproduced by computer graphics and displayed so as to be superimposed on the object on the display 370.
  • the virtual object may be reproduced using, for example, an image of a model (computer graphics).
  • the user can use the input device 350 to perform operations such as moving the virtual object in the virtual world displayed on the display 370.
  • the virtual object can be moved by the user using the input device 350 in the virtual world displayed on the display 370.
  • the virtual object when using a tracker as an input device 350, by pointing a virtual object on the tracker and then pressing the trigger button, the virtual object will be adjusted to the movement of the tracker in the virtual world while the trigger button is pressed. Can be moved freely. Then, by releasing the trigger button after moving the virtual object to a desired position / posture in the virtual world, the movement operation of the virtual object can be completed.
  • the user can also operate two virtual objects at the same time in the virtual world by operating the two trackers with both hands at the same time.
  • the hand 122 of the robot 100 into an object and display it as a virtual hand, and operate the virtual hand with the tracker to move the virtual object.
  • the virtual hand is moved.
  • the virtual object to be moved is moved with the virtual hand. Allows you to move while grasping.
  • the virtual object can be moved while the virtual object is grasped by the virtual hand.
  • the trigger button after moving the virtual object to a desired position / posture in the virtual world, the movement operation of the virtual hand and the virtual object can be completed.
  • the operation of virtual objects and virtual hands in the virtual world using the input device 350 as described above, and the display thereof on the display 370 are controlled by the processor 320.
  • the processor 320 of the control device 300 generates a control signal for moving a real object corresponding to the virtual object to the hand 122 of the robot 100 in the real world in response to the operation of the virtual hand as described above.
  • the control signal is transmitted from the control device 300 to the control unit 200.
  • the control unit 200 that has received the control signal performs motion planning of the robot 100 based on the received control signal and the surrounding environment information detected by the environment sensor 160 of the robot 100, and generates an operation command to be executed by the robot 100. Then, the robot 100 is operated based on the operation command.
  • the position of the object in the virtual world by superimposing the model corresponding to the object on the object in the virtual world corresponding to the object in the real world. And the posture is specified.
  • the user specifies the target object in the virtual world. Therefore, the calculation cost can be suppressed as compared with the case where the target object and its position / orientation are specified by image processing.
  • the present embodiment even if the object displayed in the virtual world is somewhat unclear, the user recognizes the object based on his / her own perception, selects the corresponding model, and selects the corresponding model.
  • the model can be superimposed on the object to identify the position and orientation of the object. If the model corresponding to the object is selected, the processor 320 of the control device 300 uniquely identifies the position and orientation of the object by comparing the information such as the contour, feature point, and edge of the object and the model.
  • the control device 300 can specify the position and orientation of the object in the real world.
  • FIGS. 6 and 7 as another example of specifying the position / orientation of the object, an example of specifying the position / orientation of the assembly block will be shown.
  • FIG. 6A shows a state in which a plurality of assembly blocks are placed on a table in the real world.
  • the surrounding environment including these multiple assembly blocks is imaged using a depth camera DC (eg, Intel RealSense TM depth camera) whose back is visible at the bottom of FIG. 6 (a). Will be done.
  • the depth camera DC corresponds to the environment sensor 160 of the robot 100 of the present embodiment described with reference to FIGS. 1 and 2.
  • FIG. 6B shows a virtual world generated based on the image data captured by the depth camera.
  • a plurality of blocks corresponding to each assembly block in the real world of FIG. 6 (a) are displayed in the virtual world of FIG. 6 (b).
  • This virtual world is generated in the control device 300 as described above. At this stage, those blocks are recognized by the control device 300 as merely objects occupying a certain volume space, and the positions and orientations of the blocks have not yet been specified.
  • some blocks are displayed as if some of the blocks were missing. This is because when the block is imaged by the depth camera DC arranged at the fixed position, sufficient image information on the depth side cannot be obtained. If a part of the object is missing in this way, it is difficult to correctly recognize the position and orientation of the object even with image processing technology using machine learning or AI.
  • the object and its position / posture can be recognized by incorporating the human cognitive ability.
  • FIG. 6B also displays a screen showing the types and numbers of blocks displayed in the virtual world and how to combine them.
  • FIG. 7 shows an operation of superimposing a model on a block in a virtual world to specify the position and orientation of the block.
  • FIG. 7A shows a state in which the model 70 corresponding to the block blc is moved toward the block blc.
  • the user uses the input device 350 to select the model 70 in the virtual world displayed on the display 370.
  • the model 70 can be freely moved in the virtual world according to the movement of the tracker while the trigger button is pressed. be able to.
  • the movement operation of the model 70 can be completed.
  • FIG. 7B shows a state in which the model 70 is superposed on the block blc so that the positions and postures substantially match. In this state, the user releases the trigger button of the tracker to end the movement operation of the model 70.
  • the processor 320 of the control device compares the three-dimensional shape data of the block blc (for example, an edge and / or the feature point) with the three-dimensional shape data of the model 70, and outlines the three-dimensional shape of the block blc.
  • the position / orientation of the model 70 is corrected so that the contour of the three-dimensional shape of the model 70 matches the contour of the model 70.
  • the contour of the three-dimensional shape of the block blc and the contour of the three-dimensional shape of the model 70 match.
  • the controller 320 of the control device highlights this matched three-dimensional shape contour (FIG. 7 (c)).
  • the position and orientation of the block blc in the virtual world are specified.
  • the processor 320 of the control device identifies the position and orientation of the real block corresponding to the block blc in the virtual world in the real world based on the correlation between the coordinate system of the virtual world and the coordinate system of the real world. be able to.
  • the system 1 recognizes the position and orientation of the block in the real world, and the robot 100 can perform a desired operation on the block.
  • FIG. 8 is a diagram showing a robot (unmanned submersible) used in this example and a pipe to be worked underwater.
  • the robot 100 used in this example has the form of an unmanned submersible, and includes a robot arm 120 having a robot hand 122 at its tip and a housing 140 in which the robot arm 120 is installed. have.
  • the robot 100 moves in the horizontal direction (X-axis direction), the front-back direction (Y-axis direction), and the vertical direction (Z-axis direction) in water, and rotates around each XYZ axis.
  • the housing 140 is provided with an environment sensor and a transmission / reception unit described with reference to FIG. 2 and the like.
  • the housing 140 is provided with at least a visual sensor as an environment sensor, whereby visual information on the surrounding environment of the robot 100 (particularly, the environment including the robot arm 120 and the hand 122 in front of the robot 100) can be acquired. It is possible. Since other configurations of the robot (unmanned submersible) 100 used in this example are the same as those described above with reference to FIG. 2, detailed description thereof will be omitted here.
  • the system configuration and control method used in this example are the same as those described above. In this example, the characteristic points regarding the task of grasping the underwater pipe with the robot hand at the tip of the robot arm of the robot in the form of an unmanned submersible will be described.
  • FIG. 8 shows a virtual world generated based on the environmental information acquired by the environmental sensor of the robot (unmanned submersible) 100.
  • the shape and function of each part of the robot (unmanned submersible) 100 is modeled and pre-stored in at least the storage unit 340 of the control device 300, and is therefore known in the system 1. Therefore, the modeled robot (unmanned submersible) 100 is displayed in the virtual world.
  • the pipe 60 in the virtual world is displayed in a state of being reproduced based on the environmental information acquired by the environmental sensor of the robot (unmanned submersible) 100.
  • the pipe 60 Since the pipe 60 is photographed only from a specific direction by the environment sensor of the unmanned submersible 100, it is reproduced in a shape that can be recognized from the photographed direction, and the portion on the opposite side is not reproduced. In FIG. 8, the shape of the left side portion of the pipe 60 in the drawing is reproduced, but the right side portion of the pipe 60 in the drawing is displayed in a missing state.
  • FIG. 9 is a diagram showing how the position / orientation of the pipe shown in FIG. 8 is specified in the virtual world.
  • FIG. 9 shows a virtual world generated by the controller 300 according to steps S305 and S310 shown in FIG.
  • the user creates a model corresponding to the object according to the above-mentioned "2) Scanning the object to generate a model" for the pipe 60 displayed in the virtual world on the display 370.
  • the scanned object, the pipe 60 is displayed on the display 370 as shown in FIG. 9A.
  • the user generates a cylindrical model using the tracker 350 (the corresponding virtual tracker 350_vr is displayed in FIG. 9) in the UI screen displayed on the display 370 (in FIG. 9).
  • FIG. 9 (b) move this so that it overlaps the pipe 60 in the virtual world displayed on the display 370 (FIG. 9 (c)), and adjust the diameter and length (FIG. 9 (b)).
  • the display of the model corresponding to the pipe 60 which is an object step S315 in FIG. 3) and the superposition of the model on the object (step S320 in FIG. 3) are performed.
  • the processor 320 compares the three-dimensional shape data of the pipe 60, which is an object (for example, the edge and / or the feature point of the three-dimensional shape) with the three-dimensional shape data of the model, and determines the three-dimensional shape of the pipe 60. Correct the position and orientation of the model so that the contour matches the contour of the three-dimensional shape of the model.
  • the pipe 60 displayed in the virtual world is not a mere object occupying a certain volume space, but an object (pipe 60) corresponding to the model selected by the user, and the pipe 60 is virtual. It is specified in the control device 300 that the model exists in the corrected position and orientation in the world (step S325 in FIG. 3).
  • the pipe 60 whose position and orientation are specified in this way is displayed in the virtual world as a virtual pipe 60_vr by, for example, a computer graphics representation representing a model. Thereby, for example, it is possible to instruct the operation of grasping the virtual object (virtual pipe 60_vr) of the pipe 60 with the robot hand 122 of the robot (unmanned submersible) 100.
  • FIG. 10 is a diagram showing a mug 80 placed on a table.
  • the mug 80 has a handle 82 and a main body 84.
  • the handle 82 of the mug 80 can be gripped by the hand 122 provided on the arm 120 of the robot 100 described with reference to FIGS. 1 and 2.
  • the hand 122 grips the handle 82
  • the robot arm 120 For example, the mug 80 can be moved on the table.
  • FIG. 11 is a diagram showing how the position / orientation of the mug 80 on the table shown in FIG. 10 is specified in the virtual world.
  • FIG. 11 shows a virtual world created by controller 300 according to steps S305 and S310 shown in FIG.
  • the user creates a model corresponding to the object according to the above-mentioned "2) Scanning the object to generate a model" for the mug 80 displayed in the virtual world on the display 370. , Characterize the position / orientation of the object using the model.
  • the scanned object is displayed on the display 370 as shown in FIG. 11A.
  • the user generates a cylindrical model using the tracker 350 (the corresponding virtual tracker 350_vr is displayed in FIG. 11) in the UI screen displayed on the display 370.
  • This is moved so as to be superimposed on the main body 84 of the mug 80 in the virtual world displayed on the display 370 (FIG. 11 (a)), and the diameter and length are adjusted (FIG. 11 (b)).
  • the display of the model corresponding to the main body 84 of the mug 80 which is an object (step S315 in FIG. 3), and the superposition of the model on the object (step S320 in FIG. 3) are performed.
  • the processor 320 compares the three-dimensional shape data of the main body 84, which is an object (for example, the edge and / or the feature point of the three-dimensional shape) with the three-dimensional shape data of the model, and determines the three-dimensional shape of the main body 84. Correct the position and orientation of the model so that the contour matches the contour of the three-dimensional shape of the model.
  • the main body 84 displayed in the virtual world is not a mere object occupying a certain volume space, but an object (main body 84) corresponding to the model selected by the user, and further, the main body 84 is virtual. It is specified in the control device 300 that the model exists in the corrected position and orientation in the world (step S325 in FIG. 3).
  • the tracker 350 (virtual tracker 350_vr shown in FIG. 11) is used to generate a rectangular parallelepiped model corresponding to the handle 82 of the mug 80, which is displayed on the display 370. It is moved so as to be overlapped with the handle 82 of the mug 80 in the virtual world, and the height, width, and depth dimensions are adjusted (FIG. 11 (d)). In this way, the display of the model corresponding to the handle 82 of the mug 80 which is the object (step S315 in FIG. 3) and the superposition of the model on the object (step S320 in FIG. 3) are performed.
  • the processor 320 compares the three-dimensional shape data of the handle 82, which is an object (for example, the edge and / or the feature point of the three-dimensional shape) with the three-dimensional shape data of the model, and determines the three-dimensional shape of the handle 82. Correct the position and orientation of the model so that the contour matches the contour of the three-dimensional shape of the model.
  • the handle 82 displayed in the virtual world is not a mere object that occupies a certain volume space, but an object (handle 82) corresponding to the model selected by the user, and the handle 82 is virtual. It is specified in the control device 300 that the model exists in the corrected position and orientation in the world (step S325 in FIG. 3).
  • the user performs an input operation on the UI screen displayed on the display 370 to the effect that the handle 82 of the mug 80 and the main body 84 are integrated by using the tracker 350.
  • the processor 320 of the control device 300 recognizes that the handle 82 and the main body 84 are integrated at the specified positions and postures.
  • the mug 80 whose position and posture are specified in this way is displayed in the virtual world as a virtual mug by, for example, a computer graphics representation representing a model of the handle 82 and the main body 84.
  • a virtual mug by, for example, a computer graphics representation representing a model of the handle 82 and the main body 84.
  • FIG. 12A an operation of grasping and moving the virtual object of the mug 80 with the robot hand 122 of the robot 100 in the virtual world is instructed in the virtual world (FIG. 12A), and the mug 80 in the real world is displayed. It can be gripped and moved by the robot hand 122 of the robot 100 in the real world (FIG. 12 (b)).
  • the robot 100 having an arm having a hand is illustrated as the form of the robot, but the form of the robot controlled by the present invention is not limited to this, and for example, the form of the robot is a vehicle, a ship, or the like. It may be a submersible, a drone, a construction machine (excavator, bulldozer, excavator, crane, etc.) or the like.
  • the environment and applications in which robots that can be operated using the system of this embodiment are used include space development, mining, mining, resource extraction, agriculture, forestry, and water.
  • the objects operated by the robot of the present embodiment vary depending on the environment and application in which the robot is used. As an example, when a shovel car is used as a robot, excavated soil, sand, etc. are also objects.
  • Robot control system 100 Robot 200 Control unit 300 Control device

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Graphics (AREA)
  • Computer Hardware Design (AREA)
  • General Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Manipulator (AREA)

Abstract

Conformément à un mode de réalisation, la présente invention concerne un procédé pour spécifier la position et la posture d'un objet. Le procédé de la présente invention consiste à : créer un monde virtuel comprenant un affichage d'un objet dans le monde réel (S310) ; afficher, dans le monde virtuel, un modèle correspondant à l'objet (S315) ; superposer, dans le monde virtuel, le modèle sur l'objet (S320) ; et spécifier la position et la posture de l'objet en comparant l'objet et le modèle (S325).
PCT/JP2020/019689 2019-05-17 2020-05-18 Procédé et dispositif pour spécifier la position et la posture d'un objet WO2020235539A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2019094125 2019-05-17
JP2019-094125 2019-05-17

Publications (1)

Publication Number Publication Date
WO2020235539A1 true WO2020235539A1 (fr) 2020-11-26

Family

ID=73458503

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2020/019689 WO2020235539A1 (fr) 2019-05-17 2020-05-18 Procédé et dispositif pour spécifier la position et la posture d'un objet

Country Status (1)

Country Link
WO (1) WO2020235539A1 (fr)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2008046750A (ja) * 2006-08-11 2008-02-28 Canon Inc 画像処理装置および方法
JP2011174880A (ja) * 2010-02-25 2011-09-08 Canon Inc 位置姿勢推定方法及びその装置
JP2014127092A (ja) * 2012-12-27 2014-07-07 Seiko Epson Corp 画像処理装置、画像処理方法、ロボット及びプログラム
JP2015532077A (ja) * 2012-09-27 2015-11-05 メタイオ ゲゼルシャフト ミット ベシュレンクテル ハフツングｍｅｔａｉｏ ＧｍｂＨ 少なくとも１つの画像を撮影する撮影装置に関連する装置の位置及び方向の決定方法

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2008046750A (ja) * 2006-08-11 2008-02-28 Canon Inc 画像処理装置および方法
JP2011174880A (ja) * 2010-02-25 2011-09-08 Canon Inc 位置姿勢推定方法及びその装置
JP2015532077A (ja) * 2012-09-27 2015-11-05 メタイオ ゲゼルシャフト ミット ベシュレンクテル ハフツングｍｅｔａｉｏ ＧｍｂＨ 少なくとも１つの画像を撮影する撮影装置に関連する装置の位置及び方向の決定方法
JP2014127092A (ja) * 2012-12-27 2014-07-07 Seiko Epson Corp 画像処理装置、画像処理方法、ロボット及びプログラム

Similar Documents

Publication Publication Date Title
US10671874B2 (en) Generating a model for an object encountered by a robot
CN105666505B (zh) 具备扩展现实对应显示器的机器人系统
Chen et al. Human performance issues and user interface design for teleoperated robots
JP4850984B2 (ja) 動作空間提示装置、動作空間提示方法およびプログラム
US20180190014A1 (en) Collaborative multi sensor system for site exploitation
Teixeira et al. Teleoperation using google glass and ar, drone for structural inspection
CN112634318B (zh) 一种水下维修机器人的遥操作系统和方法
US11422625B2 (en) Proxy controller suit with optional dual range kinematics
CN113103230A (zh) 一种基于处置机器人遥操作的人机交互系统及方法
CN111598273B (zh) 一种基于vr技术的环控生保系统维修性检测方法及装置
WO2020203793A1 (fr) Dispositif de commande de robot, unité de commande, et système de commande de robot les comprenant
GB2598345A (en) Remote operation of robotic systems
KR20140009900A (ko) 로봇 제어 시스템 및 그 동작 방법
JP2009258884A (ja) ユーザインタフェイス
JP2021193538A (ja) 情報処理装置、移動装置、情報処理システム、および方法、並びにプログラム
Angelopoulos et al. Drone brush: Mixed reality drone path planning
Wang et al. The design of an augmented reality system for urban search and rescue
WO2020235539A1 (fr) Procédé et dispositif pour spécifier la position et la posture d'un objet
WO2020218533A1 (fr) Procédé et dispositif d'attribution d'informations d'attribut à un objet
JP5174636B2 (ja) 遠隔操作システムおよび遠隔操作装置
US20240033930A1 (en) Remote control manipulator system and remote control assistance system
US20230214004A1 (en) Information processing apparatus, information processing method, and information processing program
JP7398227B2 (ja) 作業支援システム及びプログラム
CN110807971B (zh) 基于光惯融合定位的潜水器机械臂操控虚拟现实训练系统
JP2003278159A (ja) 遠隔操作による施工方法及びシステム

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20810229

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20810229

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: JP