WO2020218533A1

WO2020218533A1 - Method and device for assigning attribute information to object

Info

Publication number: WO2020218533A1
Application number: PCT/JP2020/017745
Authority: WO
Inventors: ロクランウィルソン; パーベルサフキン
Original assignee: 株式会社エスイーフォー
Priority date: 2019-04-26
Filing date: 2020-04-24
Publication date: 2020-10-29

Abstract

A method according to one embodiment of the present invention includes: generating a virtual world reproducing an environment in the real world; acquiring a model corresponding to an object in the virtual world to which attribute information is to be assigned (S410); and assigning attribute information including information relating to at least one part of the model to the model (S415, S420).

Description

Method and device for giving attribute information to an object

The present invention relates to a method and a device for giving attribute information to an object.

In order for a robot to execute a task using an object, it is necessary to understand how the robot uses the object by knowing not only the shape of the object but also the functions of the object. Various methods for recognizing the shape and function of an object have been proposed.

In Non-Patent Document 1, local features such as depth, color, and curvature obtained from a distance image of daily necessities are learned, and as seven functional attributes provided in daily necessities, "grasp" and "cut (Grasp)" "Cut", "Scoop", "Contain", "Pond", "Support", "Grasp-Grasp" Proposed. Non-Patent Document 1 states that in order for a robot to live together with a human, it is necessary not only to know the name of the tool but also to identify each part of the tool and identify the function of each part. Has been done.

In Non-Patent Document 2, which cites Non-Patent Document 1, the object shape of daily necessities such as cups is acquired by a 3D sensor, and the "functional attributes" provided in the daily necessities are not only the local shape information of the object but also the object shape. A method has been proposed in which the shape of other parts having continuity is also used to make an estimation based on the likelihood integration of the identification result based on the local feature amount of the object shape. Non-Patent Document 2 further discloses that an object manipulation task (pour water into a mug and lift it) by a robot arm is performed by using the proposed functional attribute estimation method.

Further, in Patent Document 1, the shape of the object recognized by using the shape feature amount and the function of the object recognized by using the functional feature amount are applied to the object concept model, and the object is statistically processed. Technology that recognizes the object to be recognized by estimating the unobserved information of the object to be recognized by applying the observed information of the shape or function of the object to be recognized to the object concept model of the object and processing it statistically. Is disclosed.

Japanese Unexamined Patent Publication No. 2008-123365

All of the above prior documents provide a method of acquiring an image of an object and estimating the attributes (concept, shape and function) of the object from the image. However, the attributes of the object estimated based on the image are not always correct. For example, depending on the posture of the object in the acquired image, the attributes of the object may not be estimated accurately. Moreover, new and unique objects have no data to reference, making it difficult to estimate the attributes of such objects. In addition, since these methods require a large amount of calculation cost to recognize each object, a large amount of computer resources and time are required to recognize each object in an environment where many objects exist.

Furthermore, in the method disclosed in the above-mentioned prior literature, for example, when an electric device having various buttons is recognized as an object, the function assigned to each button cannot be recognized. For example, when recognizing a fan with "weak", "medium", and "strong" buttons for blowing air as an object, it is possible to recognize that the fan's stand base has buttons from the shape information. Even if it is, it is not possible to recognize the function assigned to each button (in this example, the air volume assigned to each button). Alternatively, in the case of a fan equipped with an air volume dial that can change the air volume according to the rotation angle, even if the shape of the air volume dial can be recognized, the air volume dial can be rotated, and further, the rotation angle and the like. It cannot be recognized that there is a correlation with the air volume.

Further, neither of the above-mentioned prior documents mentions how to use the information on the shape and function of the article to specify the action to be executed by the robot.

According to one aspect of the present invention, it is a method of giving attribute information to an object, that is, generating a virtual world that reproduces the environment of the real world, and a model corresponding to the object to which the attribute information is given in the virtual world. Is provided, and the model is given attribute information that includes information about at least one part of the model.

According to another aspect of the present invention, it is a device that imparts attribute information to an object, and corresponds to generating a virtual world that reproduces an environment in the real world and an object that imparts attribute information in the virtual world. A device is provided with a processor configured to acquire the model and to give the model attribute information, including information about at least one part of the model.

According to another aspect of the present invention, a data structure for storing attribute information of an object is provided, and the attribute information includes a data structure including information about at least one part of the object.

Other features and advantages of the present invention can be understood from the following description and accompanying drawings given exemplary and non-exhaustive.

It is a block diagram which shows one Embodiment of a robot control system. It is a figure which shows the schematic structure of one Embodiment of a robot. It is a flowchart explaining one Embodiment of the control operation of a robot. It is a flowchart which shows the method of annotating a scanned object. It is a figure which shows the scanned object and the corresponding model. It is a figure which shows the scanned object and the corresponding model. It is a figure which shows the transition of a user input operation schematicly. It is a flowchart which shows the method of specifying the operation performed by a robot with respect to the object in the real world by using the attribute information given to the object. It is a figure which shows other example of the object which gives attribute information (annotation). It is a figure which shows the robot (unmanned submersible) and the pipe which is the object of underwater work. It is a figure which shows the state of giving an annotation (attribute information) to a pipe. It is a figure which shows the mug placed on the table. It is a figure which shows the state of giving annotation (attribute information) to the mug on a table. Fig. (A) shows how to give an operation instruction to a virtual mug which is a virtual object in the virtual world, and Fig. (B) shows how to grasp the mug in the real world with the robot hand of a robot in the real world according to the instruction. It shows how to move it.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

FIG. 1 is a block diagram showing an embodiment of a robot control system. FIG. 2 is a diagram showing a schematic configuration of an embodiment of a robot.

As shown in FIG. 1, the robot control system 1 according to the present embodiment includes a robot 100, a control unit 200 that controls the robot 100, and a control device 300 that controls the control unit 200.

First, the robot 100 in the robot control system 1 of the present embodiment will be described.

As shown in FIGS. 1 and 2, as an example, the robot 100 disclosed in the present embodiment includes at least two robot arms 120, a robot housing 140 that supports the robot arms 120, and the surrounding environment of the robot 100. It is provided with an environment sensor 160 that senses the above and a transmission / reception unit 180.

Each robot arm 120 in the present embodiment is, for example, a 6-axis articulated arm (hereinafter, also referred to as “arm”), and a robot hand (hereinafter, also referred to as “hand”) which is an end effector at the tip thereof. It has 122. The robot arm 120 includes an actuator (not shown) having a servomotor on each rotation axis. Each servomotor is connected to the control unit 200 and is configured to control its operation based on a control signal sent from the control unit 200. In the present embodiment, a 6-axis articulated arm is used as the arm 120, but the number of axes (the number of joints) of the arm can be appropriately determined according to the application of the robot 100, the functions required thereof, and the like. Further, in the present embodiment, the two-fingered hand 122 is used as the end effector, but the present invention is not limited to this, and for example, a robot hand having three or more fingers and a means of attracting by magnetic force or negative pressure are provided. A robot hand equipped with a gripping means that applies the jamming phenomenon of powder or granular material filled in a rubber film, or a robot hand that can repeatedly grip and release the gripping object by any other means. Can be used. It is preferable that the

hands

122a and 122b are configured to be rotatable around the wrist portion thereof.

The hand 122 is equipped with a kinetic sensor that detects the amount of displacement of the hand 122, the force, acceleration, vibration, etc. acting on the hand 122. Further, the hand 122 preferably includes a tactile sensor that detects the gripping force and the tactile sensation of the hand 122.

For example, the robot housing 140 may be installed in a state of being fixed on a mounting table (not shown), or may be installed on the mounting table so as to be rotatable via a rotation drive device (not shown). May be good. When the robot housing 140 is rotatably installed on the mounting table, the working range of the robot 100 can be expanded not only to the area in front of the robot 100 but also to the area around the robot 100. Further, the robot housing 140 can be used for vehicles, ships, submersibles, helicopters, drones, and other moving objects equipped with a plurality of wheels and endless tracks, depending on the application and usage environment of the robot 100. It may be mounted, or the robot housing 140 may be configured as part of such a moving body. Furthermore, the robot housing 140 may have two or more legs as walking means. When the robot housing 140 has such a moving means, the working range of the robot 100 can be made wider. Depending on the application of the robot 100, the robot arm 120 may be directly fixed to a mounting table or the like without the intervention of the robot housing 140.

The environment sensor 160 senses the surrounding environment of the robot 100. The surrounding environment includes, for example, electromagnetic waves (including visible light, invisible light, X-ray, gamma ray, etc.), sound, temperature, humidity, wind velocity, atmospheric composition, etc. Therefore, the environment sensor 160 is a visual sensor, X-ray. -It may include, but is not limited to, gamma ray sensors, auditory sensors, temperature sensors, humidity sensors, wind velocity sensors, atmospheric analyzers, and the like. Although the environment sensor 160 is shown to be integrated with the robot 100 in the figure, the environment sensor 160 does not have to be integrated with the robot 100. For example, the environment sensor 160 may be installed at a position away from the robot 100, or may be installed on a moving body such as a vehicle or a drone. Further, the environment sensor 160 preferably includes a GPS (Global Positioning System) sensor, an altitude sensor, a gyro sensor, and the like. Further, the environment sensor 160 is used as a position detection means for detecting the position of the robot 100 outdoors or indoors, in addition to the above GPS sensor, WiFi positioning, beacon positioning, self-contained navigation positioning, geomagnetic positioning, sonic positioning, UWB (Ultra Wideband). Band: Ultra-wideband) It is preferable to have a configuration for positioning, visible light / invisible light positioning, and the like.

In particular, as the visual sensor, for example, a 2D camera, a depth sensor, a 3D camera, an RGB-D sensor, a 3D-LiDAR sensor, a Kinect ™ sensor, and the like can be used. The visual information obtained by the environment sensor 160 is sent to the control unit 200 and processed by the control unit 200. Other environmental information obtained by the environment sensor 160 can also be transmitted to the control unit 200 and used for analysis of the surrounding environment of the robot 100.

The transmission / reception unit 180 transmits / receives signals / information to / from the control unit 200. The transmission / reception unit 180 can be connected to the control unit 200 by a wired connection or a wireless connection, and therefore, transmission / reception of those signals / information can be performed by a wired or wireless connection. The communication protocol, frequency, and the like used for transmitting and receiving those signals and information can be appropriately selected according to the application, environment, and the like in which the robot 100 is used. Further, the transmission / reception unit 180 may be connected to a network such as the Internet.

Next, the control unit 200 in the robot control system 1 of the present embodiment will be described.

Referring to FIG. 1 again, the control unit 200 of the system 1 according to the present embodiment includes a processor 220, a storage unit 240, and a transmission / reception unit 260.

The processor 220 mainly controls the drive unit and the sensor (both not shown) of the robot arm 120 and the body 140 of the robot 100, controls the environment sensor 160, processes the information transmitted from the environment sensor 160, and interacts with the control device 300. It controls the operation and transmission / reception unit 260. The processor 220 is composed of, for example, a central processing unit (CPU), an application specific integrated circuit (ASIC), an embedded processor, a microprocessor, a digital signal processor (DSP), or a combination thereof. The processor 220 may be composed of one or more processors.

The processor 220 recognizes an object existing in the surrounding environment of the robot 100 based on the visual information obtained by the environment sensor 160 as processing of the information sent from the environment sensor 160. As an example, the processor 220 of the control device 200 detects the shape of an object included in the visual information based on the visual information (image information and the depth information in the image) obtained by the environment sensor 160, and the control unit. Attribute information of the object stored in the storage unit 240 of the 200 (data regarding the name of the object, the shape / configuration of each part, their functions, etc.), or the object existing on the network to which the control unit 200 is connected. By referring to the attribute information, the object included in the visual information is specified. To identify an object, for example, refer to the feature points of the object shape, refer to the look-up table associated with the shape data of the known object, or use any machine learning algorithm or AI technique to identify the known object. It can be executed by comparing with the shape data of.

In the present embodiment, each object specified as described above is further given an annotation indicating the function and configuration of the object itself, and further, the function and configuration of each part of the object.
For example, if the objects are bolts and nuts, the following annotations are added.

-Bolt: Consists of a head and a shaft. The head has a hexagonal column shape, the circumference is gripped when attaching and detaching the nut, the shaft has a male screw, and the nut can be fastened to the shaft (attached by rotating clockwise and removed by rotating counterclockwise).

-Nut: Hexagonal column shape, grips the outer circumference when attaching and detaching to the bolt, and a female screw is formed on the inner circumference. The female screw can be fastened to the shaft of the bolt (attached by rotating clockwise and removed by rotating counterclockwise).

Annotation of the identified object is performed, for example, by the processor 220, data regarding the function of each part of the object stored in the storage unit 240, or each part of the object existing on the network to which the control unit 200 is connected. This can be done by referring to the data related to the function. Alternatively, the processor 220 can recognize the function of each part from the shape of the specified object by using an arbitrary machine learning algorithm or AI technique, and add annotations to the object. When the information about the detected object is known or available in the system 1, the control unit 200 automatically recognizes and annotates the object by using the method like the prior art described above on the control unit 200 side as described above. It is possible to do. On the other hand, when the information about the detected object is unknown or unavailable in the system 1, the object can be recognized and annotated by the user on the control device 300 side as described later.

Information about the object identified in this way and the annotation of its function is stored in the storage unit 240. This information may also be sent to the control device 300.

Further, the processor 220 receives, for example, a control signal of the robot 100 transmitted from the control device 300, an operation command generated in response to the control signal, an operation of the robot 100 actually executed, and an environment sensor 160 after the operation is executed. The collected ambient environment data is stored in the storage unit 240 as data, and machine learning is executed using the data to generate learning data and store it in the storage unit 240. The processor 220 can generate an operation command by deciding an operation to be executed by the robot 100 based on the control signal of the robot 100 transmitted from the control device 300 from the next time onward with reference to the learning data. .. As described above, in the present embodiment, the control unit 200 of the robot 100 in the real world has a machine learning function locally.

The storage unit 240 is a computer program for controlling the robot 100, a computer program for processing information transmitted from the environment sensor 160, and a computer for interacting with the control device 300 as described in the present embodiment. -A program, a computer program for controlling the transmission / reception unit 260, a program for executing machine learning, and the like are stored. Preferably, the storage unit 240 stores software or a program that causes a computer to perform a process as described in this embodiment to cause a function as a control unit 200. In particular, the storage unit 240 stores a computer program that can be executed by the processor 220, including instructions that implement the methods described below with reference to FIG. 4 and the like.

Further, it is preferable that the storage unit 240 stores the shape data of known objects as described above, the function data of those objects, and the look-up table associated with them. Further, the storage unit 240 includes the state of each part (servo (not shown), hand 122, etc.) of the robot arm 120 of the robot 100, information transmitted from the environment sensor 160, information sent from the control device 300, control signals, and the like. It also has the role of storing at least temporarily. Further, as described above, the storage unit 240 also has a role of storing the operation instruction of the robot 100 and the operation and learning data of the robot 100 executed in response to the operation instruction. The storage unit 240 preferably includes a non-volatile storage medium that retains the storage state even when the power of the control unit 200 is turned off. For example, a hard disk drive (HDD), a solid-state storage device (SSD), or a compact storage unit 240. Optical disk storage such as disc (CD), digital versatile disc (DVD), Blu-ray disc (BD), non-volatile random access memory (NVRAM), EPROM (Rrasable Programmable Read Only Memory), non-volatile storage such as flash memory Is equipped with. The storage unit 240 may further include volatile storage such as a static random access memory (SRAM), but each computer program described above is a non-volatile (non-temporary) storage medium of the storage unit 340. Is remembered in.

The transmission / reception unit 260 transmits / receives signals / information to / from the robot 100 and transmits / receives signals / information to / from the control device 300. The control unit 200 can be connected to the robot 100 by a wired connection or a wireless connection, and therefore the signals and information thereof can be transmitted and received by a wired or wireless connection. The communication protocol, frequency, and the like used for transmitting and receiving those signals and information can be appropriately selected according to the application, environment, and the like in which the robot 100 is used. The transmission / reception unit 260 may be connected to a network such as the Internet.

Further, the transmission / reception unit 260 transmits / receives signals / information to / from the control device 300. The control unit 200 can be connected to the control device 300 by a wired connection or a wireless connection, and therefore the signals and information thereof can be transmitted and received by a wired or wireless connection. The communication protocol, frequency, and the like used for transmitting and receiving those signals and information can be appropriately selected according to the application, environment, and the like in which the robot 100 is used.

Although the control unit 200 is shown as being independent of the robot 100 in FIG. 1, it is not limited to that form. For example, the control unit 200 may be provided in the housing 140 of the robot 100. Further, the number of robots 100 used in this system 1 is not limited to one, and a plurality of robots 100 may be operated independently or in cooperation with each other. In this case, a single control unit 200 may control a plurality of robots 100, or a plurality of control units 200 may cooperate to control a plurality of robots 100.

Subsequently, the control device 300 in the robot control system 1 of the present embodiment will be described.

As shown in FIG. 1, the control device 300 of the system 1 according to the present embodiment includes a processor 320, a storage unit 340, an input device 350, a transmission / reception unit 360, and a display 370.

The processor 320 mainly controls the interaction with the control unit 200, the processing based on the input performed by the user via the input device 350, the control of the transmission / reception unit 360, and the display of the display 370. In particular, the processor 320 generates a control signal based on the user input input by the input device 350 and transmits it to the control unit 200. The processor 220 of the control unit 200 generates one or more operation commands for operating each drive unit (not shown) of the robot arm 120 and the body 140 of the robot 100 and the environment sensor 160 based on the control signal. .. The processor 320 is composed of, for example, a central processing unit (CPU), an application specific integrated circuit (ASIC), an embedded processor, a microprocessor, a digital signal processor (DSP), or a combination thereof. The processor 320 may be composed of one or more processors. The processor 320 may be composed of one or more processors.

Further, the processor 320 of the control device 300 is configured to generate a UI (user interface) screen to be presented to the user and display it on the display 370. The UI screen (not shown) includes, for example, a selection button that hierarchically provides the user with a plurality of options. Further, the processor 320 generates an image or a moving image of a virtual world (simulation space) based on an image or a moving image of the real world of the surrounding environment of the robot 100 taken by the environment sensor 160 of the robot 100, and displays it on the display 370. When the processor 320 generates an image or a moving image of a virtual world based on an image or a moving image of the real world, for example, by associating a coordinate system of the real world with a coordinate system of the virtual world, the processor 320 connects the real world and the virtual world. Build a correlation. Further, the image or moving image of the real world and the image or moving image of the virtual world (simulation space) may be displayed on the display 370 at the same time. Further, the UI screen may be superimposed on the image or moving image of the surrounding environment of the robot 100 or the image or moving image of the virtual world. The virtual world (simulation space) image or moving image generated based on the real world image or moving image of the surrounding environment of the robot 100 also includes an object existing in the surrounding environment of the robot 100. By building a correlation between the real world and the virtual world as the processor 320 generates a virtual world image or video based on a real world image or video, a user in the virtual world, as described in detail below. It is possible to make a change in the real world based on the operation of, and to reflect the change in the real world in the virtual world.

In the above, the processor 220 of the control unit 200 identifies an object existing in the surrounding environment of the robot 100 based on the visual information obtained by the environment sensor 160, and adds an annotation relating to the function of each part of the object. As described above, these processes may be performed by the processor 320 of the control device 300 in addition to the processes by the processor 220 of the control unit 200 or in place of the processes by the processor 220.

The methods for identifying and annotating an object are as follows: 1) Obtaining ready-made information about the model and its associated annotations, 2) Scanning the object and annotating it, and 3) Creating a model of the object and annotating it. There are three possible modes of doing this.

The mode of 1) "obtaining the ready-made information of the model and the annotation associated with it" deals with the case where the model corresponding to the scanned object is available and the annotation is associated with the model. is there. In this embodiment, as described above in relation to the control unit 200, the processor 320 of the control device 300 uses the attribute information of the objects (the name of the object, the shape / configuration of each part, and the like) stored in the storage unit 340. By referring to the attribute information of the object existing on the network to which the control device 300 is connected, the identification of the object included in the visual information and the annotation related to the function of each part can be performed. Grant. As described above, any machine learning algorithm or AI technique may be used for the processing of identifying the object and adding annotations related to the functions of the respective parts.

In addition, the mode of 2) "scanning an object and annotating it" can be dealt with when a model corresponding to the scanned object is not available. In this aspect, for example, the user uses the input device 350 to combine various primitive shape elements in the UI screen to generate a model corresponding to the scanned object. Primitive shape elements include, for example, elements such as prisms of arbitrary angle, pyramids of arbitrary angle, cylinders, cones, and spheres. Further, the processor 320 may allow the user to draw an arbitrary shape in the UI screen and add it as a primitive shape element. The user selects these various shape elements in the UI screen, changes the dimensions of each part of the selected shape elements as appropriate, and combines those elements according to the image of the scanned object to correspond to the scanned object. You can generate a model to do. When generating a model using these elements, it is also possible to represent dents and holes of objects.

Then, the user adds annotations to the model generated in that way in the UI screen. As described above, when the object is a bolt, annotate the part corresponding to the head and axis of the generated model to the effect that it is the head and axis, respectively, and further, for the head, , Annotate that the surrounding surface of the hexagon can be gripped, and for the shaft, a male screw is formed, a nut can be fastened (attached by rotating clockwise, removed by rotating counterclockwise), Annotate.

Alternatively, if the object is a cupboard with an open / close door and a drawer, the annotation is given that the door can rotate around the hinge and that the drawer can be pulled out by grasping the handle. Furthermore, it is also possible to add the force required to move the opening / closing door and the drawer and the direction in which the force is applied as an annotation.

Furthermore, if the object is a faucet, it is annotated that water comes out from the tip (discharge port) of the faucet by rotating the handle. In this case, the correlation between the rotation angle of the handle and the amount of water discharged from the faucet discharge port may be added as an annotation. In this way, the annotations given include not only the function of the object, but also what information comes from which part of the object (in the above example, the information "water" is generated from the outlet of the faucet. What happens is included as an annotation).

In addition, the aspect of 3) "creating a model of an object and annotating it" can be dealt with when a user combines various primitive shape elements in a UI screen to generate a model of an arbitrary object. is there. The user's operation for generating the model and annotating in this aspect is the same as the operation described in the above aspect 2). By creating a model in advance and accumulating it in the system 1, as described in the above aspect 1), when it becomes necessary to operate a real object corresponding to the model in the real world, It is possible to associate the model with the corresponding object in the virtual world.

The processor 320 reproduces the virtual object corresponding to the object specified as described above by, for example, computer graphics, and displays it in the virtual world on the display 370.

The storage unit 340 is interactively performed by the user on the UI screen via a program for causing the processor 320 to execute the operation described in the present embodiment, a computer program that interacts with the control unit 200, and an input device 350. It stores a computer program that performs processing based on input, a computer program that controls the transmission / reception unit 260, a computer program that displays the display 370, and the like. Preferably, the storage unit 340 stores software or a program that causes the computer to perform an operation described later to generate a function as the control device 300. In particular, the storage unit 340 stores a computer program that can be executed by the processor 320, including instructions that implement the methods described below with reference to FIG. 4 and the like.

Further, the storage unit 340 includes an image or moving image of the surrounding environment of the robot 100 taken by the environment sensor 160 of the robot 100 and sent to the control device 300 via the control unit 200, and an image or moving image of the surrounding environment of the robot 100. It is possible to at least temporarily store an image or moving image of a virtual world (simulation space) generated by the processor 320 based on the moving image. The storage unit 340 of the control device 300 also preferably includes a non-volatile storage medium that retains the storage state even when the power of the control device 300 is turned off. For example, a hard disk drive (HDD) or a solid-state storage device ( SSD), optical disk storage such as compact disc (CD), digital versatile disc (DVD), Blu-ray disc (BD), non-volatile random access memory (NVRAM), EPROM (Rrasable Programmable Read Only Memory), flash memory, etc. Equipped with non-volatile storage. The storage unit 340 may further include volatile storage such as a static random access memory (SRAM), but each computer program described above is a non-volatile (non-temporary) storage medium of the storage unit 340. Is remembered in.

Further, the storage unit 340 also functions as a database of the system 1, and as described in relation to the concept of the present invention, the operation data of the robot 100 in the real world (the operation generated by the control unit 200) operated based on the control signal. (Including commands) and the ambient environment data indicating the operation result detected by the environment sensor 160 is stored.

As the input device 350, for example, a keyboard, a mouse, a joystick, etc. can be used, and further, a device called a tracker, which can track the position and posture using infrared rays or the like and has a trigger button or the like, is used. You can also. When the display 370 includes a touch panel type display device, the touch panel can be used as an input device. Further, when the display 370 is a head-mounted display used as a display device for VR (virtual reality), AR (augmented reality), MR (mixed reality), etc., and has a user's line-of-sight tracking function, The line-of-sight tracking function can be used as an input device. Alternatively, even a device having a line-of-sight tracking function but not a display can use the line-of-sight tracking function as an input device. Furthermore, a voice input device can also be used as an input device. These are exemplified as an example of the input device 350, and the means that can be used for the input device 350 is not limited to these. Further, the above-mentioned means may be arbitrarily combined and used as the input device 350. By using the input device 350 as described above, the user can select, for example, select a selection button, input characters, or take a picture of the robot by the environment sensor 160 of the robot 100 on the UI screen displayed on the display 370. An object contained in the image or moving image of the surrounding environment of 100, or an image or image of a virtual world (simulation space) generated based on the image or moving image of the surrounding environment of the robot 100 taken by the environment sensor 160 of the robot 100. You can select virtual objects included in the video.

The transmission / reception unit 360 transmits / receives signals / information to / from the control unit 200. As described above, the control device 300 can be connected to the control unit 200 by a wired connection or a wireless connection, and therefore, transmission and reception of these signals / information can be performed by a wired or wireless connection. The communication protocol, frequency, and the like used for transmitting and receiving the signal and information can be appropriately selected according to the application and environment in which the system 1 is used. Further, the transmission / reception unit 360 may be connected to a network such as the Internet.

The display 370 is used as a display device such as a display monitor, a computer / tablet device (including a device equipped with a touch panel type display), VR (virtual reality) / AR (augmented reality), or MR (mixed reality). Any form of display device such as a head-mounted display or a projector can be used.

In particular, when a head-mounted display is used as the display 370, the head-mounted display provides an image or a moving image in which the left and right eyes of the user have parallax, thereby causing the user to perceive a three-dimensional image or moving image. Can be done. Further, when the head-mounted display has a motion tracking function, it is possible to display an image or a moving image according to the position and direction of the head of the user wearing the head-mounted display. Furthermore, when the head-mounted display has a user's line-of-sight tracking function as described above, the line-of-sight tracking function can be used as an input device.

In the following description of the present embodiment, the processor 320 of the control device 300 is an image of a virtual world (simulation space) based on a real-world image or moving image of the surrounding environment of the robot 100 taken by the environment sensor 160 of the robot 100. Alternatively, a head-mounted display that generates a moving image and is used as a display device for VR (virtual reality), AR (augmented reality), MR (mixed reality), etc. is used as the display 370, and infrared rays or the like is used as the input device 350. An example will be described in which a tracker capable of tracking the position and posture and having a trigger button or the like is used.

Next, with reference to FIGS. 3 to 8, as an operation example of the robot control method, a scenario in which a task of fastening bolts and nuts is performed as an assembly of components will be described as an example.

FIG. 3 is a flowchart illustrating the control operation of the robot according to the present embodiment.

In this example, first, as shown in step S305 of FIG. 3, the ambient environment information of the robot 100 in the real world obtained by the environment sensor 160 of the robot 100 is transmitted to the control device 300 via the control unit 200. Send. The visual information may be a single still image, a plurality of images, or a moving image, and preferably includes depth information. The control device 300 may store the transmitted visual information in the storage unit 340.

Next, as shown in step S310 of FIG. 3, the processor 320 of the control device 300 generates a virtual world (simulation space) that reproduces the surrounding environment of the robot 100 based on the visual information of the control device 300. It is displayed on the display 370. In the virtual world, in addition to the landscape around the robot 100 in the real world, at least real-world objects existing in the area accessible to the robot 100 are displayed. The object may be a two-dimensional or three-dimensional image of a real-world object obtained by a visual sensor, a depth map, a point cloud, or the like. Alternatively, it may be represented by computer graphics that represent the object. In this example, bolts and nuts are displayed as objects.

Then, as shown in step S315 of FIG. 3, the objects are specified and annotations relating to the functions of each part of the objects are added. As described above, these processes include 1) obtaining the ready-made information of the model and the annotation associated with it, 2) scanning the object and annotating it, and 3) creating the model of the object and annotating it. It can be done by three aspects of doing.

According to the aspect of 1) "obtaining the ready-made information of the model and the annotation associated therewith", the processor 320 of the control device 300 uses the visual information based on the visual information sent from the control unit 200 as described above. The shape of the object contained therein is detected, and the shape data of the object stored in the storage unit 340 of the control device 300 or the shape data of the object existing on the network to which the control device 300 is connected is referred to. By doing so, the object included in the visual information is specified. To identify an object, for example, refer to the feature points of the object shape, refer to the look-up table associated with the shape data of the known object, or use any machine learning algorithm or AI technique to identify the known object. It can be executed by comparing with the shape data of. Alternatively, the object can be specified by the user performing an operation of specifying the object by using the input device 350 of the control device 300. In this example of operation, at least a segment, a fixing object for fixing the segment, a bolt and a nut are specified as objects.

Further, the processor 320 is specified by referring to, for example, data on the function of the object stored in the storage unit 340 or data on the function of the object existing on the network to which the control device 300 is connected. Annotate the object. Alternatively, the processor 320 can recognize the function of each part from the shape of the specified object by using an arbitrary machine learning algorithm or AI technique, and add annotations to the object.

In this example, the following annotations are added to each specified object (bolt and nut).

The robot 100 is an element of the system 1, and the functions of each part of the robot 100 are known in the system 1. Therefore, at least the hand 122 of the robot 100 is recognized as an object in advance, and the following annotations are stored in the storage unit 240 or the storage unit 340.
・ Hand: Equipped with wrists and claws. Objects can be gripped with the claws. The wrist can be rotated.
The annotations given to each object are stored in the data structure shown in Table 1.

If the processor 320 does not recognize the object or recognizes the wrong object, the object can be identified and annotated by the mode of 2) "scanning the object and annotating it". When the object is not recognized or the wrong object is recognized, it is considered that the data corresponding to the object to be recognized is not stored, the image quality of the object to be recognized is low, and the like.

Here, a method of annotating an object according to the aspect of 2) above will be described with reference to FIGS. 4 to 6. FIG. 4 is a flowchart showing a method of annotating the scanned object. 5 and 6 are diagrams showing the scanned object and the corresponding model.

First, the processor 320 of the control device 300 displays a virtual world (simulation space) that reproduces the surrounding environment of the robot 100 generated in step S310 of FIG. 3 on the display 370 of the control device 300. At this time, in the virtual world, the object scanned by the environment sensor 160 of the robot 100 in the real world is displayed (step S405 in FIG. 4). In this example, bolts (FIG. 5 (a)) and nuts (FIG. 6 (a)) are displayed as objects. However, in the scenario of the aspect 2), it is not yet recognized in the system 1 that these objects are bolts and nuts at this stage.

Next, acquire the model corresponding to the object (step S410 in FIG. 4). As an example, the model corresponding to the scanned object can be displayed on the display 370 by the user using the input device 350 of the control device 300 to select a primitive shape element from the in-screen menu displayed on the display 370. It can be generated by combining the displayed primitive shape elements. Primitive shape elements include, for example, elements such as prisms of arbitrary angle, pyramids of arbitrary angle, cylinders, cones, and spheres, which are stored in the storage unit 340 of the control device 300.

When the object is a volt (recognized by the user) as shown in FIG. 5A, the user uses the input device 350 of the control device 300 to select from the in-screen menu displayed on the display 370. Hexagonal columns and cylinders are selected as primitive shape elements to generate the model. Then, the processor 320 of the control device 300 reads out those shape element data from the storage unit 340 and displays them on the display 370. Then, the user operates the hexagonal column and the cylinder on the display 370 using the input device 350 to appropriately change the size, the dimensional ratio, etc., and combine them to form a bolt model as shown in FIG. 5 (b). Create 50.

Further, when the object is a nut (recognized by the user) as shown in FIG. 6A, a hexagonal column and a cylinder are selected as primitive shape elements and they are combined in the same manner as described above. Then, as shown in FIG. 6B, a nut model 60 is created. In the model 60 shown in FIG. 6B, the cylinder represents a hole formed in a hexagonal column.

In the above, as one of the methods for generating a model corresponding to an object, an example of reading primitive shape elements and combining them is shown, but the method of creating a model is not limited to this.

For example, for the object (bolt) shown in FIG. 5A, a hexagon along the contour of one end face of the hexagonal column portion (head) is defined by using the input device 350, and the hexagon is defined as the object. By moving the hexagonal column portion along the longitudinal direction to the other end face, a hexagonal columnar volume space defined by the hexagonal moving region can be formed. Further, a circle along the contour of one end face of the cylindrical portion (screw axis) of the object shown in FIG. 5A is defined by using the input device 350, and the circle is defined by the other of the cylindrical portions along the longitudinal direction of the object. By moving to the end face of, a cylindrical volumetric space defined by the circular moving region can be formed. In this way, the model 50 shown in FIG. 5B is generated in the virtual world so as to be superimposed on the object shown in FIG. 5A. The model 60 shown in FIG. 6B can also be generated by the same procedure.

Alternatively, as another method, when the three-dimensional shape of the object can be recognized by the control device 300, the three-dimensional shape may be used as the model instead of generating the model using the primitive shape elements.

Next, the user gives a name to the model displayed on the display 370 as the identification information of the object by using the input device 350 of the control device 300 (step S415 in FIG. 4). Specifically, the model 50 shown in FIG. 5 (b) is given a "bolt" as the name of the corresponding object, and the model 60 shown in FIG. 6 (b) is given a "nut" as the name of the corresponding object. Give. At this time, for example, if detailed information such as a standard is known for the object, that information can also be added (for example, "M8 × 20" indicating the diameter and length of the screw for the bolt, etc. ).

The user uses the input device 350 to display the screen displayed on the display 370, and the attribute information of various objects stored in the storage unit 340 of the control device 300 (object name, shape / configuration of each part, and their). By referring to the attribute information of various objects existing on the network to which the control device 300 is connected, and selecting the name to be given to the model, the name of the object can be given to the model. Can be granted. When the attribute information of the object cannot be referred to in the case of a new object that does not exist in the storage unit 340 or an external database, for example, the user inputs the attribute information including the name of the object using the input device 350. It is possible to store the information in the storage unit 340 and to select the name and give it to the model.
Next, attribute information (annotation) is added to each part of the model created as described above (step S420 in FIG. 4).

In annotating the model 50 shown in FIG. 5B, for example, the user selects the hexagonal column portion 50a of the model 50 displayed on the display 370 using the input device 350 of the control device 300, and bolts. Annotation of "head" which means that it is the head of the head is added as identification information of the part. Further, six surfaces around the hexagonal column portion 50a of the model 50 are designated or selected by using the input device 350, and the annotation "grasping the outer circumference when attaching / detaching the nut" is added as functional information of the portion. The annotation "Grip the outer circumference when attaching and detaching the nut" uniquely specifies that the outer circumference of the bolt is gripped as the gripping area of the bolt for exerting the function of attaching and detaching the nut, but the nut is attached and detached. It means that any part of the bolt can be gripped except when (for example, when simply moving the bolt).

Further, the input device 350 is used to select the cylindrical portion 50b of the model 50 displayed on the display 370, and annotate the "axis" indicating that it is the axis of the bolt as identification information of that portion. Furthermore, the outer peripheral surface of the cylindrical portion 50b of the model 50 is instructed or selected using the input device 350, and the annotation "male screw. The nut can be fastened (attached by rotating clockwise and removed by rotating counterclockwise)" is annotated with the functional information of that portion. Granted as. Here, for example, when it is desired to prohibit gripping the shaft on which the male screw is formed, an annotation that prohibits gripping the shaft may be added as a function of the shaft. This makes it possible to prevent gripping of parts that are not desirable to be gripped by the robot's hand, such as fragile parts.

Similarly, for the model 60 shown in FIG. 6B, the outer peripheral portion 60a of the model 60 is selected, and an annotation of “outer circumference” meaning that it is the outer circumference of the nut is added as identification information of that portion. In addition, the outer peripheral portion 60a of the model 60 is annotated with "hold the outer circumference when attaching / detaching to / from the bolt" as functional information of that portion. The annotation "Grip the outer circumference when attaching / detaching to / from the bolt" also means that any part of the nut can be grasped except when attaching / detaching to / from the bolt (for example, when simply moving the nut). To do. Further, the inner peripheral portion 60b of the model 60 is selected, and an annotation of "inner circumference" meaning that it is the inner circumference of the nut is added as identification information of the portion. In addition, the inner peripheral portion 60b of the model 60 is annotated as "female screw. Can be fastened to the shaft of the bolt (attached by rotating clockwise and removed by rotating counterclockwise)" as functional information of that portion.

Here, as described in step S415, the user uses the input device 350 to display the attribute information of various objects stored in the storage unit 340 of the control device 300 on the screen displayed on the display 370. Information to be given to the model by referring to (data on the name of the object, the shape / configuration of each part, and their functions) or the attribute information of various objects existing on the network to which the control device 300 is connected. Function information can be added to the model by selecting. For example, when the functional attribute of the bolt is stored in the storage unit 340, information on the attribute information (name of the part and its function) of the head and the shaft of the bolt is selected, and the information is selected in each of the

parts

50a and 50b, respectively. Can be granted. On the other hand, when the attribute information of the object cannot be referred to because it is a new object that does not exist in the storage unit 340 or an external database, for example, the user uses the input device 350 to input the attribute information including the function of the object. It is possible to input and store it in the storage unit 340, and to select the name and give it to the model.

In this way, the control device 300 also functions as a device for imparting attribute information to the object, and the control device 300 can be used to impart attribute information to the object. The attribute information data assigned to each object is stored in the data structure shown in Table 1.

Then, in the virtual world, the model generated as described above is superimposed on the corresponding object, the model is associated with the object, and attribute information (annotation) is added to the object (annotation). Step S425 in FIG. 4). In the example shown in FIG. 5, the user uses the input device 350 to select the model 50 shown in FIG. 5B on the screen displayed on the display 370 and move the model 50 within the screen. , Overlay the objects shown in the same screen (FIG. 5 (a)) so that their positions and postures match. If the model 50 is superposed on the object and generated in advance in the virtual world, this superimposing operation is omitted. Then, the processor 320 calculates the positional relationship between the contour of the three-dimensional shape of the object and the contour of the three-dimensional shape of the model 50, and determines whether or not the model 50 is superimposed on the object based on a predetermined criterion. When it is determined that they are overlapped, the object and the model are associated with each other.

As a result, the processor 320 of the control device 300 recognizes that the object (FIG. 5A) is the object corresponding to the annotated model 50, and the processor 320 becomes the object specified as described above. The corresponding virtual object is reproduced, for example, in computer graphics and displayed so as to be superimposed on the object on the display 370. The virtual object may be reproduced using, for example, an image of a model (computer graphics). The user can use the input device 350 to perform operations such as moving the virtual object in the virtual world displayed on the display 370.

The processor 220 of the control unit 200 detects the shape of the object included in the visual information from the environment sensor 160, identifies the object included in the visual information, and adds an annotation related to each part function of the object. (That is, by referring to the attribute information of the object stored in the storage unit 240 of the control unit 200, or the attribute information of the object existing on the network to which the control unit 200 is connected. When the object can be specified), the processor 320 of the control device 300 does not perform the above-described processing, obtains information about the specified object from the control unit 200, and virtualizes the object in the virtual world. Display the object.

By adding the attribute information (annotation) to the object as described above, the object in the virtual space displayed on the display 370 of the control device 300 is not a mere object occupying a certain volume space, but is based on the given attribute information. System 1 recognizes that it represents the specified object.

The virtual object can be moved by the user using the input device 350 in the virtual world displayed on the display 370. For example, when using a tracker as an input device 350, by pointing a virtual object on the tracker and then pressing the trigger button, the virtual object is adjusted to the movement of the tracker in the virtual world while the trigger button is pressed. Can be moved freely. Then, by releasing the trigger button after moving the virtual object to a desired position / posture in the virtual world, the movement operation of the virtual object can be completed. The user can also operate two virtual objects at the same time in the virtual world by operating the two trackers with both hands at the same time.

In the virtual world on the display 370, it is possible to convert the hand 122 of the robot 100 into an object and display it as a virtual hand, and operate the virtual hand with a tracker to move the virtual object. For example, by pointing a virtual hand with a tracker and pressing the trigger button, the virtual hand is moved. Then, by aligning the claw or finger part of the virtual hand with the virtual object to be moved and releasing the trigger button being pressed, or by pressing another trigger button, the virtual object to be moved is moved with the virtual hand. Allows you to move while grasping. After that, for example, by moving the virtual hand with the tracker while pressing the trigger button, the virtual object can be moved while the virtual object is grasped by the virtual hand. Then, by releasing the trigger button after moving the virtual object to a desired position / posture in the virtual world, the movement operation of the virtual hand and the virtual object can be completed.

The operation of virtual objects and virtual hands in the virtual world using the input device 350 as described above, and the display thereof on the display 370 are controlled by the processor 320.

Next, as shown in step S320 of FIG. 3, the robot operates the virtual object displayed in the virtual world on the display 370 by the user using the input device 350 in the virtual world. Input the operation for causing 100 to execute the task. In this example, in order to cause the robot 100 to execute the task of fastening the bolts and nuts to each other, the following operations are input.

FIG. 7 is a diagram schematically showing the transition of the user input operation. As shown in FIG. 7, in the virtual world on the display 370, virtual bolts and virtual nuts corresponding to bolts and nuts, which are objects placed in the real world where the robot 100 exists, are bolts in the real world. And are displayed in the same arrangement as the nut.

As shown in FIG. 7A, in the virtual world displayed on the display 370, there are a virtual bolt 70 and a virtual nut 71 placed on a table, and two virtual hands 122a_vr and 122a_vr. There is.

First, as shown in FIG. 7B, the virtual hand 122b_vr of one hand 122b of the robot 100 displayed in the virtual world on the display 370 is moved by using the tracker of the input device 350, and the virtual hand 122b_vr is used. As an operation of grasping the virtual bolt 70, the claw portion of the virtual hand 122b_vr is overlapped with the virtual segment 70 in the virtual world. The head of the bolt is annotated to indicate that "the outer circumference can be gripped when the nut is attached or detached (in other cases, any part can be gripped)", and the claw of the hand is "grasping the object". An annotation indicating that it is possible is added. The processor 320 of the control device 300 derives the meaning of "holding the bolt with the claws of the hand" based on the relationship of the above annotations of the interacting parts of these objects. Therefore, the processor 320 of the control device 300 specifies the meaning of "holding the bolt 70 with the claw portion of the hand 122b" from the input operation of superimposing the claw portion of the virtual hand 122b_vr on the head portion of the virtual bolt 70.

Next, as shown in FIG. 7C, the virtual hand 122a_vr representing the other hand 122a of the robot 100 is moved using a tracker, and the virtual hand 122a_vr grips the virtual nut 71 as a virtual operation in the virtual world. The claw portion of the hand 122a_vr is overlapped with the virtual nut 71. Annotation indicating "Grip the outer circumference when attaching / detaching to / from the bolt (in other cases, any part can be gripped)" is attached to the outer circumference of the nut, and "Object can be gripped" on the claw part of the hand. Annotation indicating that there is is attached. The processor 320 of the control device 300 derives the meaning of "holding the nut with the claws of the hand" based on the relationship of the above annotations of the interacting parts of these objects. Therefore, the processor 320 of the control device 300 specifies the meaning of "holding the nut with the claw portion of the hand 122a" from the input operation of superimposing the claw portion of the virtual hand 122a_vr on the virtual nut 71.

Then, as shown in FIG. 7D, the virtual hand 122b_vr is moved while holding the virtual bolt 70, and the virtual hand 122a_vr is moved while holding the virtual nut 71 to fasten the bolt and the nut. As an operation, an operation of fitting the virtual nut 71 to the shaft of the virtual bolt 70 is performed. The bolt shaft is annotated to indicate that "the nut can be fastened (clockwise rotation, remove it counterclockwise)", and the inner circumference of the nut can be "fastened to the bolt shaft (clockwise rotation,). Annotation indicating "remove by turning counterclockwise)" is added. The processor 320 of the controller 300 derives that "the nut can be attached to the shaft of the bolt" based on the relationship of the above annotations of the interacting parts of these objects. Therefore, the processor 320 of the control device 300 specifies the meaning of "rotating and fastening the bolt and the nut to each other" from the input operation of fitting the virtual nut 71 to the shaft of the virtual bolt 70.

Furthermore, with the identification of the meaning of "rotating and fastening bolts and nuts to each other", for bolts, follow the annotation of "Grip the outer circumference when attaching and detaching nuts" on the head, and "Hand 122b claws". It was specified that "the outer circumference" of the bolt 70 is gripped by the part, and for the nut, according to the annotation of "grasping the outer circumference when attaching and detaching to the bolt", "the" outer circumference "of the nut is gripped by the claw part of the hand 122a. "Grip" is specified.

In this way, when the user inputs an action for executing a task to a virtual object in the virtual world, the processor 320 of the control device 300 identifies a part of the object that causes an action in each action. , The meaning of each action input (the action to be executed by the robot 100) is specified based on the attribute information about the function given to the part. As described above, the attribute information given to each object in the data structure as shown in Table 1 is associated with the operation of the virtual object in the virtual world, and the robot 100 is associated with the object in the real world. Is used to identify the actions performed by.

FIG. 8 is a flowchart showing a method of specifying an action performed by a robot on an object in the real world by using attribute information given to the object.

When a user operates a virtual object in the virtual world displayed on the display 370 of the control device 300 by using the input device 350 of the control device 300, the operation to be executed by the robot 100 is input by the control device 300 ( Step S805 in FIG. 8).

The processor 320 of the control device 300 identifies the part of the object that causes the action based on the input operation (step S810 in FIG. 8). For example, when an operation of superimposing virtual objects is performed in a virtual world, the superposed portion of these virtual objects is specified as a portion that causes an action.

Next, the processor 320 of the control device 300 refers to the functional attributes given to each part of those objects in the data structure, and identifies the meaning of the input operation based on the relationship of those functional attributes. (Step S815 in FIG. 8).

With the above, the input of the operation is completed for the task of fastening the bolt and the nut shown in step S320 of FIG. The input operation of this series of operations is stored in the storage unit 340 of the control device 300.

Next, as shown in step S325 of FIG. 3, the processor 320 generates a control signal to be transmitted to the control unit 200 based on the meaning of each operation extracted as described above, and the control device 300 generates the control unit 200. Send to. In this way, the control device 300 generates a control signal for controlling the robot 100 by using the data structure in which the attribute information of the object is stored.

The control unit 200 that has received the control signal performs motion planning of the robot 100 based on the received control signal and the surrounding environment information detected by the environment sensor 160 of the robot 100, and generates an operation command to be executed by the robot 100. Then, the robot 100 is made to execute the task based on the operation command (step S330 in FIG. 3). As a result, in the real world, the robot 100 executes the task of grasping the bolts and nuts with the

hands

122a and 122b and fastening them.

Next, as shown in step S335 of FIG. 3, the environment sensor 160 of the robot 100 detects the surrounding environment while the robot 100 is executing the task or after the task is executed, in order to detect the execution result. Then, the surrounding environment information is transmitted to the control unit 200. The surrounding environment information is stored in the storage unit 240. Further, the ambient environment information may be transmitted to the control device 300 and stored in the storage unit 340. Further, the received control signal and the generated operation command can also be stored and stored in the storage unit 240, and they may be transmitted to the control device 300 and stored and stored in the storage unit 340.

The processor 220 uses an arbitrary machine learning algorithm or AI technology to compare and learn the control signal received from the control device 300 and the operation instruction generated by the processor 220 with the surrounding environment information after the task is executed. , It is possible to improve the quality of generated operation instructions. The processor 220 can select, for example, an operation with less possibility of failure and an operation with less momentum by accumulating the generation / execution of the operation instruction and the result and learning them.

On the other hand, when such information is transmitted to the control device 300, it can also be learned by the processor 320 of the control device. In this case, the learning result can be taken into consideration when the processor 320 generates the control signal. Furthermore, it can be used for improvement when the user who sees the task execution result performs an input operation next time in the virtual world. The control unit 200 and the control device 300 may exchange and share the learning result data of the control unit 200 and the learning result data of the control device 300 with each other.
With the above, the control operation of the robot in this example is completed.

As described above, according to the present embodiment, attribute information (annotation) including functions and the like included in each part of the object in the real world is given, and the attribute information is shown in, for example, Table 1. It is stored in the data structure shown. The attribute information given to each object is used to identify the action performed by the robot 100 on the object in the real world by associating it with the action performed on the virtual object in the virtual world. Since the user can intuitively perform the instruction operation in the virtual world, it is possible to easily add the attribute information (annotation) to the object.

The attribute information includes identification information of the object represented by the model (object name, etc.), identification information indicating at least one part of the model (name of each part of the object, etc.), and information about at least one part of the model (of each part of the object). It may include (shape, function, etc.), but the attribute information given to the object may not be all of these, but only a part of them. For example, it is possible to give an object only shape information about at least one part of the model as attribute information.

In the description of the present embodiment, the robot 100 having an arm having a hand is illustrated as the form of the robot, but the form of the robot controlled by the present invention is not limited to this, and for example, the form of the robot is a vehicle, a ship, or the like. It may be a submersible, a drone, a construction machine (excavator, bulldozer, excavator, crane, etc.) or the like. In addition to the environments and applications described in this embodiment, the environment and applications in which robots that can be operated using the system of this embodiment are used include space development, mining, mining, resource extraction, agriculture, forestry, and water. There are a wide variety of environments and applications such as industry, livestock industry, search and rescue, disaster support, disaster recovery, humanitarian support, explosives disposal, removal of obstacles on the route, disaster monitoring, and crime prevention monitoring. The objects operated by the robot of the present embodiment vary depending on the environment and application in which the robot is used. As an example, when a shovel car is used as a robot, excavated soil, sand, etc. are also objects.

(Other objects)
Here, an example of another object to which attribute information (annotation) is given by the method of this embodiment is shown.

[First example]
FIG. 9A shows a model corresponding to a faucet object. This model can be generated by steps S405-S410 of the method shown in FIG. A "faucet" is added to this model as identification information of an object, and a handle 90 and a discharge port 91 are added as identification information of each part thereof (step S415 in FIG. 4). Further, the functional information thereof is added to each part of the model (step S420 in FIG. 4). For example, the following functions are given as functional information of each part of the model.
-Handle 90: Rotatable. Rotation angle is 0 ° to 180 °
-Water spout 91: Discharges water. Discharge amount is proportional to the rotation angle of the handle

By superimposing and associating the model to which the attribute information is given with the object displayed in the virtual world in this way, the attribute information possessed by the model can be given to the object. As a result, for example, when the user inputs an action in the virtual world to act on the handle of the virtual object of the faucet together with the information on the required amount of water, the robot 100 is given the handle of the faucet in the real world. It is possible to instruct the operation of rotating to a rotation angle according to the amount of water.

Note that a function indicating the relationship between the rotation angle and the amount of torque required to rotate the handle may be added as an annotation when the handle is rotated from the closed state to the open state and vice versa. The function can be modified accordingly for later optimization. Further, regarding the annotation of "the discharge amount is proportional to the rotation angle of the handle" for the spout, for example, the proportional function may be added as an annotation, and the proportional function once specified in order to optimize the proportional function. Can be changed later as appropriate.

FIG. 9B shows a model corresponding to the objects in the cupboard. This model can also be generated by steps S405 to S410 of the method shown in FIG. A "cupboard" is added to this model as object identification information, and a top plate 95, a drawer 96, a right door 97, and a left door 98 are added as identification information for each part (step S415 in FIG. 4). ). Further, each part of the model is given those functions as functional information (step S420 in FIG. 4). For example, the following functions are given as functional information of each part of the model.
-Top plate 95: Objects can be placed on top.
-Drawer 96: Has a handle. You can grab the handle and pull it out toward you. Objects can be stored inside when pulled out.
-Right door 97: Has a handle. Can be opened and closed around the hinge on the right side by grasping the handle. Objects can be stored inside in the open state.
-Left door 98: Has a handle. Can be opened and closed around the hinge on the left side by grasping the handle. Objects can be stored inside in the open state.

By superimposing and associating a model to which attribute information is given on an object in the virtual world in this way, it is possible to give the attribute information possessed by the model to the object. As a result, for example, the user inputs an action in the virtual world that causes an action on the closed right door handle of the virtual object of the cupboard, so that the robot 100 can receive the right door handle of the cupboard in the real world. You can instruct the operation to open the right door centering on the hinge.

Furthermore, by adding attribute information to each part of the object, the system 1 continues to recognize the virtual object even if each part moves in the virtual world and the entire shape of the virtual object is deformed. Becomes possible. For example, in a cupboard as shown in FIG. 10, the overall shape of the cupboard changes depending on the drawer and the opening degree of the door, but the system 1 can recognize the virtual object as the cupboard regardless of the change in the overall shape. ..
Table 2 shows a data structure for storing the attribute information given to each of the virtual objects of the faucet and the cupboard.

[Second example]
FIG. 10 is a diagram showing a robot (unmanned submersible) used in this example and a pipe to be worked underwater. As shown in FIG. 10, the robot 100 used in this example has the form of an unmanned submersible, and includes a robot arm 120 having a robot hand 122 at its tip and a housing 140 in which the robot arm 120 is installed. have. In the housing 140, the robot 100 moves in the horizontal direction (X-axis direction), the front-rear direction (Y-axis direction), and the vertical direction (Z-axis direction) in water, and rotates around each XYZ axis. There are multiple thrusters (not shown) that allow it. These thrusters consist of, for example, propellers that are rotated by an electric motor. Although not clearly shown in the drawing, the housing 140 is provided with an environment sensor and a transmission / reception unit described with reference to FIG. 2 and the like. In particular, the housing 140 is provided with at least a visual sensor as an environment sensor, whereby visual information on the surrounding environment of the robot 100 (particularly, the environment including the robot arm 120 and the hand 122 in front of the robot 100) can be acquired. It is possible. Since the other configurations of the robot (unmanned submersible) 100 used in this example are the same as those described above with reference to FIG. 2, detailed description thereof will be omitted here. The system configuration and control method used in this example are the same as those described above. In this example, the characteristic points regarding the task of grasping the underwater pipe with the robot hand at the tip of the robot arm of the robot in the form of an unmanned submersible will be described.

Note that FIG. 10 shows a virtual world generated based on the environmental information acquired by the environment sensor of the robot (unmanned submersible) 100. The shape and function of each part of the robot (unmanned submersible) 100 is at least modeled and stored in advance in the storage unit 340 of the control device 300, and is therefore known in the system 1. Therefore, the modeled robot (unmanned submersible) 100 is displayed in the virtual world. On the other hand, the pipe 40 in the virtual world is displayed in a state of being reproduced based on the environmental information acquired by the environment sensor of the robot (unmanned submersible) 100. Since the pipe 40 is photographed only from a specific direction by the environment sensor of the unmanned submersible 100, it is reproduced in a shape that can be recognized from the photographed direction, and the portion on the opposite side is not reproduced. In FIG. 10, the shape of the left side portion of the pipe 40 in the drawing is reproduced, but the right side portion of the pipe 40 in the drawing is displayed in a missing state.

FIG. 11 is a diagram showing how annotations (attribute information) are added to the pipe shown in FIG.

FIG. 11 shows a virtual world generated by the controller 300 according to steps S305 and S310 shown in FIG. In this example, the user annotates the pipe 40 displayed in the virtual world on the display 370 according to the above-mentioned "2) Scanning and annotating an object" mode, and the control device 300 It is stored in the storage unit 340 of.

More specifically, first, the scanned object, the pipe 40, is displayed on the display 370 as shown in FIG. 11A (step 405 in FIG. 4). Next, the user generates a cylindrical model using the tracker 350 (the corresponding virtual tracker 350_vr is displayed in FIG. 11) in the UI screen displayed on the display 370 (the corresponding virtual tracker 350_vr is displayed). FIG. 11 (b), by moving it so as to overlap the pipe 40 in the virtual world displayed on the display 370 (FIG. 11 (c)), and by adjusting the diameter and length (FIG. 11 (b)). 11 (d)), the model corresponding to the pipe 40 is acquired (step 410 in FIG. 4). A "pipe" is added as the identification information of the object to this model, and an "outer circumference" is added as the identification information of the portion (step S415 in FIG. 4). Further, the functional information (shape / function) is added to the portion (step S420 in FIG. 4). For example, the following functions are given as functional information of the model part.
-Outer circumference: Cylindrical shape. Can be gripped with a robot hand.
The annotation added to the pipe 40 is stored in the data structure shown in Table 3.

By superimposing and associating the model to which the attribute information is given in this way on the object (pipe 40) displayed in the virtual world, the attribute information possessed by the model can be given to the object. The pipe 40 to which the attribute information is added in this way is displayed in the virtual world as a virtual pipe by the computer graphics representation representing the model. Thereby, for example, it is possible to instruct the operation of grasping the virtual object (virtual pipe 40_vr) of the pipe 40 with the robot hand 122 of the robot (unmanned submersible) 100.

[Third example]
FIG. 12 is a diagram showing a mug 80 placed on a table.

The mug 80 has a handle 82 and a main body 84. The handle 82 of the mug 80 can be gripped by the hand 122 provided on the arm 120 of the robot 100 described with reference to FIGS. 1 and 2. When the hand 122 grips the handle 82, the robot arm 120 For example, the mug 80 can be moved on the table.

FIG. 13 is a diagram showing how annotations (attribute information) are added to the mug 80 on the table.

FIG. 13 shows a virtual world generated by the controller 300 according to steps S305 and S310 shown in FIG. In this example, the user annotates the mug 80 displayed in the virtual world on the display 370 according to the above-mentioned "2) Scanning and annotating an object" mode, and the control device 300 It is stored in the storage unit 340 of.

More specifically, first, the scanned object, the mug 80, is displayed on the display 370 as shown in FIG. 13 (a) (step 405 in FIG. 4). Next, the user generates a cylindrical model using the tracker 350 (the corresponding virtual tracker 350_vr is displayed in FIG. 13) in the UI screen displayed on the display 370. By moving this so as to be superimposed on the main body 84 of the mug 80 in the virtual world displayed on the display 370 (FIG. 13 (a)) and adjusting the diameter and length (FIG. 13 (b)). ), Acquire a model corresponding to the main body 84 (step 410 in FIG. 4).

Similarly, a tracker 350 (virtual tracker 350_vr shown in FIG. 13) is used to generate a rectangular parallelepiped model corresponding to the handle 82 of the mug 80, which is displayed on the display 370 in the virtual world of the mug 80. By moving so as to be overlapped with the handle 82 (FIG. 13 (d)), a model corresponding to the main body 84 is acquired (step 410 in FIG. 4).

A "mug" is added to this model as identification information of an object, and a handle 82 and a main body 84 are added as identification information of each part thereof (step S415 in FIG. 4). Further, the functional information thereof is added to each part of the model (step S420 in FIG. 4). For example, the following functions are given as functional information of each part of the model.
-Handle: Rectangular parallelepiped shape. Can be gripped with a robot hand.
-Main body: Cylindrical shape.
The annotation added to the mug 80 is stored in the data structure shown in Table 4.

By superimposing and associating the model to which the attribute information is given in this way on the object (mug 80) displayed in the virtual world, the attribute information possessed by the model can be given to the object. The mug 80 to which the attribute information is added in this way is displayed in the virtual world as a virtual pipe by a computer graphics representation representing the model. As a result, for example, an operation of grasping and moving the virtual object of the mug 80 with the robot hand 122 of the robot 100 in the virtual world is instructed in the virtual world (FIG. 14A), and the mug in the real world is instructed according to the instruction. The 80 can be gripped and moved by the robot hand 122 of the robot 100 in the real world (FIG. 14 (b)).

In this example, an example of assigning tax information to the handle 82 and the main body 84 of the mug 80 has been described. However, for example, if the purpose is to hold the handle 82 with the robot hand 122 of the robot 100 and move the mug 80. , Attribute information may be provided only for the handle 82, which is a part of the mug 80.

Various examples have been described for objects to which attribute information (annotation) is given by the method of the present embodiment, but the objects to which attribute information can be given by the method of the present embodiment are not limited to the above objects and are arbitrary objects. It is possible to add attribute information to the object.

For example, for a fan equipped with "weak", "medium", and "strong" buttons for blowing air, "press to switch on / off. When on, the air volume is" weak "". , "Press to switch on / off. On to switch on / off air volume" Medium "," Press to switch on / off. On to air volume "strong" ". is there. Alternatively, for a fan equipped with an air volume dial that can change the air volume according to the rotation angle, for the air volume dial, "rotatable. Rotation angle is 0 ° to 120 °. Air volume is proportional to the rotation angle." Function information can be added.

Furthermore, for vehicles equipped with multiple doors, it is possible to add functional information such as "has a doorknob. It can be opened and closed around the hinge by grasping the doorknob."

In addition, by adding information such as weight, load, friction coefficient, and center of gravity to each part of the object as annotations, it is possible to perform appropriate operation when the object itself or each part is dynamically operated. become.

The attribute information (annotation) given to the object can be stored in the data structure shown in Tables 1 to 4.

Although the present invention has been described above through the embodiments of the invention, the above-described embodiments do not limit the invention according to the claims. In addition, a form combining the features described in the embodiments of the present invention may also be included in the technical scope of the present invention. Furthermore, it will be apparent to those skilled in the art that various changes or improvements can be made to the above embodiments.

Claims

It is a method of giving attribute information to an object.
Creating a virtual world that recreates the real world environment,
In the virtual world, acquiring the model corresponding to the object to which the attribute information is given,
To give the model attribute information including information about at least one part of the model.
Including methods.
The method according to claim 1, wherein the attribute information further includes identification information of an object represented by the model and identification information indicating at least one part of the model.
In the virtual world, associating the object with the model and imparting the attribute information to the object,
The method according to claim 1 or 2, further comprising.
The method according to claim 3, wherein the association between the object and the model is performed by superimposing the model on the object in the virtual world.
A device that gives attribute information to an object
Creating a virtual world that recreates the real world environment,
In the virtual world, acquiring the model corresponding to the object to which the attribute information is given,
To give the model attribute information including information about at least one part of the model.
A device with a processor configured to run.
The device according to claim 5, wherein the attribute information further includes identification information of an object represented by the model and identification information indicating at least one part of the model.
The processor further
In the virtual world, associating the object with the model and imparting the attribute information to the object,
5. The device of claim 5 or 6, which is configured to perform.
The device according to claim 7, wherein the association between the object and the model is performed by superimposing the model on the object in the virtual world.
A data structure that stores the attribute information of an object
The attribute information is a data structure that includes information about at least one part of the object.
The data structure according to claim 9, wherein the attribute information further includes identification information of an object represented by the model and identification information indicating at least one part of the model.
The data structure according to claim 9 or 10, wherein the data structure is used for generating the control signal in a control device that generates a control signal for controlling the robot.
The control device creates a virtual world including a virtual object corresponding to the object, receives an action input for the virtual object in the virtual world by a user, and identifies a part of the object that causes an action by the action input. Is configured to
11. The data structure is used in the control device to identify the meaning of the action input based on the information about the portion and to generate the control signal based on the meaning of the action input. The data structure described.
A computer program that can be executed by a processor and includes an instruction that implements the method according to any one of claims 1 to 4.
A non-transitory computer-readable medium comprising a computer program stored on the medium and capable of being executed by a processor, including instructions for performing the method according to any one of claims 1-4. , Non-temporary computer-readable medium.