JP2021122868A - Robot, control method, information processor, and program - Google Patents

Abstract

To accurately control a first part.SOLUTION: A robot includes a processor for controlling a first part. The processor controls the first part so as to move a predetermined part of the first part to a predicted target position that is a moving target of a predetermined part when a current target position that is the moving target of the predetermined part cannot be calculated based on current data indicating at least a part of a range where the predetermined part is movable. The predicated target position is calculated based on a past target position which is a moving target in the predetermined part calculated based on the past data acquired in the past, a past reference position that is a position of a reference object calculated based on the past data, and current data.SELECTED DRAWING: Figure 3

Description

Embodiments of the present invention relate to robots, control methods, information processing devices and programs.

Conventionally, a robot having an arm provided with an end effector at the tip is known. Such a robot brings the end effector closer to the target object by deforming the arm. Then, such a robot operates the end effector to mechanically operate the target object in a state where the end effector is close to the target object. Such a robot brings the end effector closer to the target object, for example, by controlling the arm by a visual servo. That is, such a robot brings the end effector closer to the target object while detecting the positional relationship between the end effector and the target object based on the image captured by the camera.

For small robots for home use, the camera and arm are provided in a small housing. Therefore, depending on the positional relationship between the target object, the camera, and the arm, the target object may be hidden by the arm, and the camera may not be able to image the target object. In such a case, for example, the robot could not bring the end effector closer to the target object with high accuracy because the visual servo was cut off.

Japanese Unexamined Patent Publication No. 6-190756

An object to be solved by the embodiment of the present invention is to control the first portion with high accuracy.

The robot according to the embodiment includes a movable first portion and a processor that controls the first portion. When the processor cannot calculate the current target position as the movement target of the predetermined portion based on the current data indicating at least a part of the movable range of the predetermined portion of the first portion, the predetermined portion of the predetermined portion. The first portion is controlled so as to move the predetermined portion to the predicted target position which is the movement target. The predicted target position includes a past target position that is a movement target in the predetermined portion calculated based on past data acquired in the past, a past reference position that is a position of a reference object calculated based on the past data, and the current data. And, it is calculated based on.

FIG. 1 is a diagram showing the appearance of the robot according to the embodiment. FIG. 2 is a diagram showing the configuration of the robot. FIG. 3 is a diagram showing a functional configuration of an information processing device for controlling an arm by a visual servo. FIG. 4 is a diagram showing an example of an image in which the current target position can be calculated. FIG. 5 is a diagram showing an example of an image in which the target position cannot be calculated at present. FIG. 6 is a flowchart showing a processing flow of the information processing apparatus. FIG. 7 is a flowchart showing the flow of arm control processing. FIG. 8 is a flowchart showing a processing flow of the information processing apparatus according to the first modification. FIG. 9 is a flowchart showing a flow of arm control processing according to the first modification. FIG. 10 is a diagram showing an example of an image captured by a camera in the second modification. FIG. 11 is a diagram showing an example of the hardware configuration of the information processing device.

Hereinafter, embodiments will be described in detail with reference to the drawings.

FIG. 1 is a diagram showing the appearance of the robot 10 according to the embodiment. The robot 10 of the present embodiment includes at least a main body portion 22, an arm 24 (an example of a first portion), and a camera 26.

The main body 22 is a housing that is the main body of the robot 10. The main body 22 is movable, for example. The main body 22 is provided with, for example, a moving mechanism such as wheels for freely moving on the floor at the bottom.

The arm 24 is the first portion of the present embodiment and is provided on the main body portion 22. The arm 24 has one or more arms (links) and one or more joints (movable parts). Each of the one or more joints connects the arm to the body 22 or the arm to the arm, eg, rotatably or slidably. Each of the one or more joints includes a motor and changes the positional relationship between the arm and the main body 22 or the arm and the arm. The arm 24 changes into an arbitrary shape by the movement of one or more joints. The first part refers to a part of the robot 10 in addition to the arm. Further, in the present disclosure, the arm can be appropriately read as the first part.

The arm 24 includes an end effector 28 in a predetermined portion. For example, the arm 24 includes an end effector 28 at a tip end that is opposite to the main body 22.

The end effector 28 performs a predetermined mechanical operation on the target object. The target object is, for example, one independent object. The target object may be a part of the surface of the object, an edge portion or a protrusion portion of the object, as long as it can be the target of mechanical operation by the end effector 28.

For example, the end effector 28 is a gripping mechanism. The gripping mechanism can grab the target object. The arm 24, which includes a gripping mechanism as the end effector 28, can grab and move the target object. Further, the end effector 28 may have any mechanism as long as it can perform a predetermined mechanical operation on the target object. For example, the end effector 28 may be a drill for making a hole in the target object or a brush for polishing the target object.

The camera 26 is a data acquisition unit in the present embodiment and is provided in the main body unit 22. The camera 26 images (acquires) the surroundings of the robot 10. The camera 26 is provided at a position where the end effector 28 can be imaged even when the arm 24 is deformed and the end effector 28 is moved. For example, the camera 26 is provided at a position different from the mounting position of the arm 24 on the main body 22. For example, the camera 26 is provided above the position where the arm 24 is provided in the main body 22.

The camera 26 may be a device for capturing a two-dimensional image or a device for capturing a stereo image. Further, the camera 26 may be a depth camera or an infrared camera. Further, the data acquisition unit is not limited to a camera that captures such an image and acquires data, as long as it can acquire data on the position of an object or the like around the device. That is, the current data or the past data may be any data indicating the position data of an object or the like around the device. For example, the data acquisition unit may be LiDAR or the like capable of acquiring measurement data of the distance to the object at present. In the present disclosure, the camera can be appropriately read as a data acquisition unit, and the image can be appropriately read as data.

The robot 10 having such a configuration can freely move on the floor or the like. Further, the robot 10 can deform the arm 24 to move the end effector 28 in the vicinity of the target object. Then, the robot 10 can perform a predetermined operation on the target object by operating the end effector 28 in a state where the end effector 28 is moved in the vicinity of the target object.

FIG. 2 is a diagram showing a configuration of the robot 10 of the present embodiment. The main body 22 includes an information processing device 30 inside. The information processing device 30 executes a computer program to control the operations of the arm 24, the camera 26, and the end effector 28. A part of the information processing device 30 may be provided outside the main body 22. In this case, in the information processing device 30, a part inside the main body 22 and a part outside the main body 22 are connected by wireless communication, and information processing is executed in cooperation with each other.

When the information processing apparatus 30 of the present embodiment performs a predetermined operation on the target object by the end effector 28, the arm 24 is controlled by the visual servo to bring the end effector 28 closer to the target object. That is, the information processing device 30 analyzes the image captured by the camera 26 and detects the relative positional relationship between the end effector 28 and the target object. Subsequently, the information processing device 30 deforms the arm 24 so that the end effector 28 moves to a position (target position) where a predetermined operation can be performed on the target object. Then, the information processing device 30 repeatedly executes such processing in real time until the end effector 28 reaches the target position.

FIG. 3 is a diagram showing a functional configuration of an information processing device 30 for controlling an arm 24 by a visual servo. The information processing device 30 includes an acquisition unit 42, an object detection unit 44, an arm identification unit 48, a target identification unit 50, a reference object identification unit 52, a storage unit 56, and an arm control unit 58.

The acquisition unit 42 acquires an image (an example of current data) captured by the camera 26 at predetermined time intervals.

The object detection unit 44 of the present embodiment detects one or more objects included in the image acquired by the acquisition unit 42 at predetermined time intervals. Further, the object detection unit 44 may also detect each class of the detected one or a plurality of objects. The class is, for example, the type of object.

The arm identification unit 48 of the present embodiment identifies a predetermined portion of the arm 24 of one or more objects detected by the object detection unit 44. For example, the arm specifying portion 48 identifies an end effector 28 provided at the tip of the arm 24. Then, the arm specifying unit 48 calculates (identifies) the spatial position of the predetermined portion (end effector 28) on the specified arm 24. For example, the arm identification unit 48 calculates the position of the end effector 28 represented by three-dimensional Cartesian coordinates.

The target specifying unit 50 of the present embodiment calculates the current target position, which is a spatial position as a movement target of a predetermined portion of the arm 24, based on the image acquired by the acquisition unit 42. For example, the target identification unit 50 calculates the current target position represented by three-dimensional Cartesian coordinates.

In the present embodiment, the target identification unit 50 identifies the target object to be operated by the end effector 28 among one or more objects detected by the object detection unit 44. Subsequently, the target identification unit 50 calculates the spatial position of the target object. For example, the target identification unit 50 calculates the position of the target object represented by three-dimensional Cartesian coordinates.

Then, the target specifying unit 50 calculates, based on the position of the target object, a position where the end effector 28 can operate a predetermined position with respect to the target object as the current target position. For example, the target specifying unit 50 calculates a position separated from the target object by a predetermined distance in a predetermined direction as the current target position. For example, when the end effector 28 is a gripping mechanism, the target specifying unit 50 calculates a position where the end effector 28 can grip the target object as the current target position.

The reference object identification unit 52 identifies any one or a plurality of objects detected by the object detection unit 44 as a reference object. Then, the reference object specifying unit 52 calculates the spatial position of the specified reference object as the current reference position. For example, the reference object identification unit 52 calculates the current reference position represented by three-dimensional Cartesian coordinates.

The reference object identification unit 52 identifies, for example, any object other than the target object among one or a plurality of objects detected by the object detection unit 44 as a reference object. The reference object may be one independent object or a part of one object. For example, the reference object may be a pattern drawn on the surface of the object, or may be an edge portion or a protrusion portion of the object.

For example, the reference object identification unit 52 identifies an object that is fixed relative to the target object among one or a plurality of objects detected by the object detection unit 44 as a reference object. Further, for example, the reference object identification unit 52 specifies, among one or a plurality of objects detected by the object detection unit 44, an object whose distance from the target object is within a predetermined range as a reference object. More specifically, for example, the reference object identification unit 52 specifies an object farther from the target object than the preset first threshold value and closer to the preset second threshold value as the reference object. In this case, the first threshold is shorter than the second threshold.

The storage unit 56 of the present embodiment calculates the current target position calculated by the target identification unit 50 based on a past image (an example of past data or a first image), and a predetermined portion of the arm 24 (end in the present embodiment). It is stored as a past target position that is a movement target of the effector 28). Further, the storage unit 56 stores the current reference position calculated by the reference object identification unit 52 as a past reference position which is a position in the reference object calculated based on the past image. The storage unit 56 stores the past target position calculated based on the same past image and the past reference position as a pair.

The target specifying unit 50 may not be able to calculate the current target position based on the image captured by the camera 26. For example, when the target object is hidden by the arm 24 and the target object is not included in the image, the target specifying unit 50 cannot currently calculate the target position. Further, even if the target object is included in the image, the target specifying unit 50 sets the current target position even when the accuracy of the current target position is lower than the predetermined accuracy or when it is difficult to calculate the current target position due to other conditions. It may be determined that it cannot be calculated. In this case, the storage unit 56 may not store the past target position and the past reference position. Further, when the storage unit 56 stores the past target position and the past reference position calculated based on the new image, the storage unit 56 may erase the past target position and the past reference position calculated based on the previous image.

The arm control unit 58 of the present embodiment acquires the spatial position of a predetermined portion (end effector 28) on the arm 24 from the arm identification unit 48. Further, the arm control unit 58 acquires information indicating whether or not the current target position can be calculated based on the image from the target identification unit 50.

When the target specifying unit 50 can calculate the current target position based on the image, the arm control unit 58 acquires the current target position from the target specifying unit 50. The arm control unit 58 controls the arm 24 so that when the target identification unit 50 can calculate the current target position based on the image, the predetermined portion (end effector 28) of the arm 24 is brought closer to the current target position. do. For example, the arm control unit 58 deforms the arm 24 so as to reduce the difference between the position of the end effector 28 and the current target position.

When the target identification unit 50 cannot calculate the current target position based on the image, the arm control unit 58 reads the past target position and the past reference position from the storage unit 56. Further, the arm control unit 58 acquires the current reference position from the reference object identification unit 52 when the target identification unit 50 cannot calculate the current target position based on the image.

Subsequently, the arm control unit 58 calculates the position difference between the past target position and the past reference position. For example, the arm control unit 58 calculates the position difference by subtracting the past reference position from the past target position. Subsequently, the arm control unit 58 calculates a predicted target position that is a movement target of a predetermined portion based on the past target position, the past reference position, and the current reference position. For example, the arm control unit 58 calculates a predicted target position that is a movement target of a predetermined portion based on the position difference between the past target position and the past reference position and the current reference position. More specifically, for example, the arm control unit 58 calculates the predicted target position by adding the position difference to the current reference position calculated by the reference object identification unit 52. Then, the arm control unit 58 controls the arm 24 so that a predetermined portion (end effector 28) of the arm 24 approaches or reaches the predicted target position. For example, the arm control unit 58 deforms the arm 24 so as to reduce the difference between the position of the end effector 28 and the predicted target position.

The object detection unit 44, the arm identification unit 48, the target identification unit 50, the reference object identification unit 52, and the arm control unit 58 each time the acquisition unit 42 acquires an image until the end effector 28 reaches the target position. Perform the above process. As a result, the information processing device 30 can control the arm 24 by the visual servo and bring the predetermined portion of the arm 24 closer to the target position with high accuracy. For example, when the information processing device 30 receives an instruction to perform a predetermined operation on the target object by the end effector 28, the information processing device 30 controls the arm 24 by a visual servo to move to a position where the target object can be operated. The end effector 28 can be moved with high accuracy.

FIG. 4 is a diagram showing an example of an image in which the current target position can be calculated. In the example of FIG. 4, the spoon 72 is a target object. In the example of FIG. 4, the robot 10 is trying to grip the spoon 72 by the end effector 28. In this case, the camera 26 captures an image so that the end effector 28 and the spoon 72 are included in the angle of view.

When the spoon 72 is included in the image, the information processing apparatus 30 can calculate the current target position (the position where the end effector 28 can grip the spoon 72) based on the image. Therefore, in the example of FIG. 4, the information processing apparatus 30 can deform the arm 24 so that the difference between the position of the end effector 28 and the current target position calculated based on the image becomes small.

Further, the information processing device 30 identifies an object other than the spoon 72 included in the image as a reference object. In the example of FIG. 4, the information processing device 30 specifies the fork 74 as a reference object. Then, the information processing device 30 stores the current target position and the position of the fork 74 as the past target position and the past reference position. Then, the information processing device 30 stores the current target position calculated based on the image shown in FIG. 4 as the past target position, and stores the position of the fork 74 calculated based on the image shown in FIG. 4 as the past reference position.

FIG. 5 is a diagram showing an example of an image in which the target position cannot be calculated at present. While moving the end effector 28, for example, the arm 24 may enter between the camera 26 and the target object (spoon 72 in the example of FIG. 5), and all or part of the target object may be hidden from the camera 26. be. In this case, the information processing device 30 cannot identify the target object based on the image. As a result, the information processing apparatus 30 cannot calculate the current target position (the position where the spoon 72 can be gripped in the example of FIG. 5) based on the image.

When the current target position cannot be calculated based on the image, the information processing device 30 reads out the stored past target position and past reference position. For example, the information processing apparatus 30 reads out the past target position and the past reference position based on the last past image from which the current target position and the current reference position could be calculated at the end. Further, the information processing apparatus 30 may read out the past target position and the past reference position based on the past image from which the current target position and the current reference position could be calculated before a predetermined time before the last past image.

Subsequently, the information processing device 30 calculates the positional difference between the read past target position and the past reference position. Subsequently, the arm control unit 58 calculates the predicted target position based on the past target position, the past reference position, and the current reference position which is the position of the reference object (fork 74 in the example of FIG. 5) included in the image. do. For example, the arm control unit 58 calculates the predicted target position based on the calculated position difference and the current reference position which is the position of the reference object (fork 74 in the example of FIG. 5) included in the image. Then, the information processing device 30 deforms the arm 24 so that the position of the end effector 28 approaches the predicted target position.

The information processing device 30 specifies an object whose position is fixed relative to the target object as a reference object. As a result, the information processing device 30 can accurately move the end effector 28 to the target position even when the target object is moving. For example, the information processing device 30 targets the end effector 28 by specifying an object moving together with the target object as a reference object even when the target object is mounted on a conveyor belt, a trolley, or the like. It can be moved to the position with high accuracy.

Further, the information processing apparatus 30 may specify an object having a positional difference from the target object among the plurality of detected objects within a predetermined range as a reference object. As a result, when the end effector 28 approaches the target object, the information processing device 30 can fit the reference object within the angle of view of the camera 26 and include the reference object in the image. Further, the information processing device 30 can include the reference object in the image by preventing the reference object from being hidden by the arm 24 when the end effector 28 approaches the target object.

FIG. 6 is a flowchart showing a processing flow of the information processing apparatus 30. When the end effector 28 gives an instruction to perform a predetermined operation on the target object, the information processing apparatus 30 controls the arm 24 according to the flow shown in FIG.

The information processing device 30 repeats the processes from S12 to S21 at predetermined time intervals (loop processing between S11 and S22). In the loop, first, in S12, the information processing device 30 acquires an image captured by the camera 26.

Subsequently, in S13, the information processing device 30 detects one or more objects included in the image acquired in S12.

Subsequently, in S14, the information processing apparatus 30 identifies a predetermined portion of the arm 24 of the one or more objects detected in S13. In the present embodiment, the information processing device 30 specifies the end effector 28 as a predetermined portion. Subsequently, in S15, the information processing apparatus 30 calculates the spatial position of the predetermined portion (end effector 28) on the arm 24 specified in S14.

Subsequently, in S16, the information processing apparatus 30 identifies the target object among the one or more objects detected in S13.

Subsequently, in S17, the information processing apparatus 30 calculates the current target position, which is the movement target of the predetermined portion (end effector 28) in the arm 24, based on the image acquired in S12. In the present embodiment, the information processing apparatus 30 calculates a position where the end effector 28 can perform a predetermined operation on the target object specified in S16 as the current target position. More specifically, the information processing apparatus 30 calculates the spatial position of the target object specified in S16, and calculates the position separated from the calculated position of the target object by a predetermined distance in a predetermined direction as the current target position. For example, when the end effector 28 is a gripping mechanism, the information processing device 30 calculates a position where the end effector 28 can grip the target object specified in S16 as the current target position.

Note that the information processing device 30 cannot identify the target object in S16 when the target object is hidden by the arm 24 when viewed from the camera 26. In this case, the information processing device 30 cannot currently calculate the target position. Therefore, in this case, the information processing device 30 skips the process of S17.

Subsequently, in S18, the information processing apparatus 30 identifies any one of the one or a plurality of objects detected in S13 as a reference object. Subsequently, in S19, the information processing apparatus 30 calculates the spatial position of the reference object specified in S18 as the current reference position.

Subsequently, in S20, the information processing apparatus 30 uses the current target position calculated in S17 as the past target position as the movement target of the predetermined portion (end effector 28) of the arm 24 calculated based on the past image (first image). Is stored in the storage unit 56. Further, the information processing apparatus 30 stores the current reference position calculated in S19 in the storage unit 56 as a past reference position. The information processing device 30 stores the past target position and the past reference position calculated based on the same past image as a pair in the storage unit 56.

The information processing device 30 may not be able to calculate the current target position in S16. In this case, the information processing device 30 skips the process of S20.

Subsequently, in S21, the information processing device 30 controls the arm 24 so that a predetermined portion of the arm 24 approaches the target position. In the present embodiment, the information processing device 30 deforms the arm 24 so that the end effector 28 approaches a target position where the target object can be operated. For example, when the end effector 28 is a gripping mechanism, the information processing device 30 deforms the arm 24 so that the end effector 28 approaches a position where the target object can be gripped. The process of S21 will be further described with reference to FIG.

The information processing device 30 repeats the processes from S12 to S21 at predetermined time intervals until the end effector 28 reaches the target position (loop process between S11 and S22). When the end effector 28 reaches the target position, the information processing device 30 exits the loop process and ends this flow.

FIG. 7 is a flowchart showing the flow of control processing of the arm 24. The information processing device 30 executes the process shown in FIG. 7 in the control process of the arm 24 of S21 shown in FIG.

First, in S31, the information processing device 30 determines whether or not the current target position can be calculated. When the information processing apparatus 30 can currently calculate the target position (Yes in S31), the information processing apparatus 30 advances the process to S32. When the information processing apparatus 30 cannot currently calculate the target position (No in S31), the information processing apparatus 30 advances the process to S33.

In S32, the information processing apparatus 30 controls the arm 24 so that the predetermined portion (end effector 28) of the arm 24 approaches the current target position calculated in S17. More specifically, the information processing apparatus 30 deforms the arm 24 so as to reduce the difference between the position of the end effector 28 calculated in S15 and the current target position calculated in S17.

Then, when the information processing device 30 finishes the process of S32, it exits this flow and returns to the process of FIG.

In S33, the information processing device 30 reads out a pair of the past target position and the past reference position from the storage unit 56. Subsequently, in S34, the information processing apparatus 30 calculates the positional difference between the read past target position and the read past reference position. For example, the information processing device 30 calculates a position difference obtained by subtracting the past reference position from the past target position.

Subsequently, in S35, the information processing apparatus 30 calculates a predicted target position that is a movement target of a predetermined portion based on the past target position, the past reference position, and the current reference position calculated in S19. For example, the information processing apparatus 30 calculates a predicted target position as a movement target of a predetermined portion based on the position difference calculated in S34 and the current reference position calculated in S19. For example, the information processing apparatus 30 calculates the predicted target position by adding the position difference calculated in S34 to the current reference position calculated in S19.

Subsequently, in S36, the information processing apparatus 30 controls the arm 24 so that the predetermined portion (end effector 28) of the arm 24 approaches the predicted target position calculated in S35. More specifically, the information processing apparatus 30 deforms the arm 24 so as to reduce the difference between the position of the end effector 28 calculated in S15 and the predicted target position calculated in S35.

Then, when the information processing device 30 finishes the process of S36, it exits this flow and returns to the process of FIG.

When the robot 10 according to the present embodiment cannot calculate the current target position that is the movement target of the predetermined portion of the arm 24 based on the image captured by the camera 26, the past target position calculated based on the past image and the past target position A predetermined portion of the arm 24 can be moved based on the past reference position. As a result, according to the robot 10 according to the present embodiment, even when the target position cannot be calculated at present, the predetermined portion of the arm 24 can be moved with high accuracy.

For example, when the robot 10 cannot specify the target object to be operated by the end effector 28 based on the image captured by the camera 26 and therefore cannot calculate the current target position, the past specified based on the past image. The end effector 28 can be moved based on the target position and the past reference position. As a result, the robot 10 can accurately move the end effector 28 to a position that can be operated with respect to the target object even when the target object cannot be identified based on the image captured by the camera 26.

FIG. 8 is a flowchart showing a processing flow of the information processing apparatus 30 according to the first modification. When the end effector 28 gives an instruction to perform a predetermined operation on the target object, the information processing apparatus 30 may further move the end effector 28 to a predetermined operable posture.

In this case, the information processing apparatus 30 executes the process according to the flow shown in FIG. In the first modification, the information processing apparatus 30 assigns the same step numbers to the processes that are substantially the same as the processes described with reference to FIG. 6, omitting the description, and describes the differences.

In S15, the information processing apparatus 30 calculates the spatial position of the predetermined portion (end effector 28) on the arm 24, and also calculates the posture of the predetermined portion (end effector 28) on the arm 24 specified in S14.

In S17, the information processing apparatus 30 calculates the current target position, and also calculates the current target posture which is the posture target of the predetermined portion (end effector 28) in the arm 24 based on the image acquired in S12. In this modification, the information processing apparatus 30 calculates a posture in which the end effector 28 can perform a predetermined operation on the target object specified in S16 as the current target posture. More specifically, the information processing apparatus 30 calculates the spatial posture of the target object specified in S16, and calculates the posture at a predetermined angle with respect to the calculated posture of the target object as the current target posture. do. For example, when the end effector 28 is a gripping mechanism, the information processing device 30 calculates the posture in which the end effector 28 can grip the target object specified in S16 as the current target posture position.

In S18, the information processing apparatus 30 identifies any one of the one or a plurality of objects detected in S13 as a reference object. In this modification, the information processing device 30 specifies an object whose posture can be calculated as a reference object. For example, since it is difficult to detect a posture of an object such as a sphere without a pattern, in this modification, the information processing apparatus 30 is a reference object except for an object whose posture is difficult to detect. To identify.

In S19, the information processing apparatus 30 calculates the current reference position and calculates the spatial posture of the reference object specified in S18 as the current reference posture.

In S20, the information processing device 30 stores the current target position calculated in S17 as a past target position in the storage unit 56. Further, the information processing apparatus 30 stores the current reference position calculated in S19 as the past reference position in the storage unit 56. At the same time, the information processing device 30 stores the current target posture calculated in S17 in the storage unit 56 as a past target posture which is a posture target of a predetermined portion (end effector 28) of the arm 24 calculated based on the past image. Further, the information processing apparatus 30 stores the current reference posture calculated in S19 in the storage unit 56 as the past reference posture which is the posture of the reference object calculated based on the past image. The information processing device 30 stores the past target position, the past reference position, the past target posture, and the past reference posture calculated based on the same past image as a set in the storage unit 56.

In S21, the information processing device 30 controls the arm 24 so that the predetermined portion (end effector 28) of the arm 24 approaches the target position, and the predetermined portion (end effector 28) of the arm 24 approaches the target posture. Controls the arm 24. In this modification, the information processing device 30 deforms the arm 24 so that the end effector 28 approaches the target position and the target posture in which the target object can be operated. For example, when the end effector 28 is a gripping mechanism, the information processing device 30 deforms the arm 24 so that the end effector 28 approaches a position and a posture in which the target object can be gripped. The processing of S21 in this modification will be further described with reference to FIG.

The information processing device 30 repeats the processes from S12 to S21 at predetermined time intervals until the end effector 28 reaches the target position and the target posture (loop process between S11 and S22). When the end effector 28 reaches the target position and the target posture, the information processing device 30 exits the loop process and ends this flow.

FIG. 9 is a flowchart showing a flow of control processing of the arm 24 according to the first modification. The information processing device 30 executes the process shown in FIG. 9 in the control process of the arm 24 of S21 shown in FIG.

First, in S41, the information processing device 30 determines whether or not the current target position and the current target posture can be calculated. When the information processing apparatus 30 can calculate the current target position and the current target posture (Yes in S41), the information processing device 30 proceeds to S42. When the information processing apparatus 30 cannot calculate the current target position or the current target posture (No in S41), the information processing device 30 proceeds to S43.

In S42, the information processing apparatus 30 controls the arm 24 so that the predetermined portion (end effector 28) of the arm 24 approaches the current target position and the current target posture calculated in S17. More specifically, the information processing apparatus 30 reduces the difference between the position of the end effector 28 calculated in S15 and the current target position calculated in S17, and calculates the posture of the end effector 28 calculated in S15 and S17. The arm 24 is deformed so as to reduce the difference from the current target posture.

Then, when the information processing device 30 finishes the process of S42, it exits this flow and returns to the process of FIG.

In S43, the information processing device 30 reads out a set of the past target position, the past reference position, the past target posture, and the past reference posture from the storage unit 56.

Subsequently, in S44, the information processing apparatus 30 calculates the positional difference between the read past target position and the read past reference position. For example, the information processing device 30 calculates a position difference obtained by subtracting the past reference position from the past target position.

Subsequently, in S45, the information processing apparatus 30 calculates the posture difference between the read past target posture and the read past reference posture. For example, the information processing device 30 calculates a posture difference obtained by subtracting the past reference posture from the past target posture.

Subsequently, in S46, the information processing apparatus 30 calculates a predicted target position as a movement target of a predetermined portion (end effector 28) based on the position difference calculated in S44 and the current reference position calculated in S19. For example, the information processing apparatus 30 calculates the predicted target position by adding the position difference calculated in S44 to the current reference position calculated in S19.

Subsequently, in S47, the information processing apparatus 30 calculates a predicted target posture which is a posture target of a predetermined portion (end effector 28) based on the posture difference calculated in S45 and the current reference posture calculated in S19. For example, the information processing apparatus 30 calculates the predicted target posture by adding the posture difference calculated in S45 to the current reference posture calculated in S19.

In S48, the information processing apparatus 30 controls the arm 24 so that the predetermined portion (end effector 28) of the arm 24 approaches the predicted target position calculated in S46 and the predicted target posture calculated in S47. More specifically, the information processing apparatus 30 reduces the difference between the position of the end effector 28 calculated in S15 and the predicted target position calculated in S46, and calculates the posture of the end effector 28 calculated in S15 and S47. The arm 24 is deformed so as to reduce the difference from the predicted target posture.

Then, when the information processing device 30 finishes the process of S48, the information processing device 30 exits this flow and returns to the process of FIG.

When the robot 10 according to the first modification cannot calculate the current target position and the current target posture based on the image captured by the camera 26, the past target position, the past reference position, and the past calculated based on the past image. The end effector 28 can be moved based on the target posture and the past reference posture. As a result, the robot 10 according to the first modification accurately adjusts the end effector 28 to a position and posture that can be operated with respect to the target object even when the target object cannot be identified based on the image captured by the camera 26. It can be moved well.

FIG. 10 is a diagram showing an example of an image captured by the camera 26 in the second modification. The information processing device 30 according to the second modification identifies a plurality of reference objects.

In this modification, the reference object identification unit 52 identifies any plurality of reference objects among the plurality of objects detected by the object detection unit 44. For example, the reference object identification unit 52 identifies two or more objects that are relatively fixed to the target object among the plurality of objects detected by the object detection unit 44 as the plurality of reference objects. Further, for example, the reference object identification unit 52 identifies two or more objects having a distance from the target object within a predetermined range among the plurality of objects detected by the object detection unit 44 as a plurality of reference objects.

For example, in the example of FIG. 10, the object detection unit 44 identifies the fork 74 and the knife 76 as two reference objects. Then, the reference object specifying unit 52 calculates the current reference position for each of the specified plurality of reference objects. Further, the reference object specifying unit 52 may calculate the current reference posture for each of the plurality of specified reference objects.

In this modification, the storage unit 56 stores a plurality of current reference positions calculated by the reference object identification unit 52 as a plurality of past reference positions. Further, the storage unit 56 may store a plurality of current reference postures calculated by the reference object identification unit 52 as a plurality of past reference postures.

In this modified example, when the target identification unit 50 cannot calculate the current target position based on the image, the arm control unit 58 receives the past target position from the storage unit 56 and each of the plurality of reference objects. Read the past reference position. Further, the arm control unit 58 may also read past reference postures for each of the plurality of reference objects from the storage unit 56.

Subsequently, the arm control unit 58 calculates the positional difference between the past target position and the past reference position for each of the plurality of reference objects. Further, the arm control unit 58 may calculate the posture difference between the past target posture and the past reference posture for each of the plurality of reference objects.

Subsequently, the arm control unit 58 calculates the predicted target position based on the position difference for each of the plurality of reference objects and the current reference position calculated by the reference object identification unit 52. For example, the arm control unit 58 calculates the position difference between the past target position and the past reference position for each of the plurality of reference objects, and adds the current reference position to the calculated position difference to obtain an individual predicted target position. calculate. Subsequently, the arm control unit 58 calculates the predicted target position by averaging the individual predicted target positions calculated for each of the plurality of reference objects. Then, the arm control unit 58 controls the arm 24 so that the predetermined portion of the arm 24 approaches the predicted target position.

Further, the arm control unit 58 may calculate the predicted target posture based on the posture difference for each of the plurality of reference objects and the current reference posture calculated by the reference object identification unit 52. For example, the arm control unit 58 calculates the posture difference between the past target posture and the past reference posture for each of the plurality of reference objects, and adds the current reference posture to the calculated posture difference to obtain the individual predicted target posture. calculate. Subsequently, the arm control unit 58 calculates the predicted target posture by averaging the individual predicted target postures calculated for each of the plurality of reference objects. Then, the arm control unit 58 controls the arm 24 so that the predetermined portion of the arm 24 approaches the predicted target posture.

In this way, the robot 10 according to the second modification identifies a plurality of reference objects. As a result, the robot 10 according to the second modification can accurately perform the end effector 28 based on the positions and orientations of the plurality of reference objects even when the target object cannot be identified based on the image captured by the camera 26. Can be moved.

The robot 10 in the above-described embodiment may include a plurality of arms 24. Further, in that case, the arm 24 that hides the current target position from the camera 26 and the arm 24 that operates may be different. Further, the object that currently hides the target position from the camera 26 is not limited to the arm 24. Further, the current image and the past image are not limited to one image, and may be a plurality of images. Further, the present invention is not limited to an image, and may be point cloud information or the like.

FIG. 11 is a block diagram showing an example of the hardware configuration of the information processing apparatus 30 in the above-described embodiment.

A part or all of the functions provided in the information processing apparatus 30 in the above-described embodiment may be composed of hardware, or software executed by a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or the like. It may be composed of information processing of (program). When it is composed of information processing by software, the software that realizes at least a part of the functions provided in the information processing apparatus 30 in the above-described embodiment is a flexible disk or a CD-ROM (Compact Disc-Read Only Memory). ) Or a non-temporary storage medium (non-temporary computer-readable medium) such as a USB (Universal Serial Bus) memory, and the computer may read the information processing of the software. In addition, the software may be downloaded via a communication network. Further, information processing may be executed by hardware by mounting the software on a circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

The type of storage medium that stores the software is not limited. The storage medium is not limited to a removable one such as a magnetic disk or an optical disk, and may be a fixed storage medium such as a hard disk or a memory. Further, the storage medium may be provided inside the computer or may be provided outside the computer.

As shown in FIG. 11, the information processing device 30 includes, as an example, a processor 91, a main storage device 92 (memory), an auxiliary storage device 93 (memory), a network interface 94, and a device interface 95. , These may be realized as a computer 90 connected via a bus 96.

The computer 90 of FIG. 11 includes one component for each component, but may include a plurality of the same components. Further, although one computer 90 is shown in FIG. 11, software is installed on a plurality of computers, and each of the plurality of computers executes the same or different part of the software. May be good. In this case, it may be a form of distributed computing in which each computer communicates via a network interface 94 or the like to execute processing. That is, the information processing device 30 in the above-described embodiment may be configured as a system that realizes a function by executing instructions stored in one or a plurality of storage devices by one or a plurality of computers. Further, the information transmitted from the terminal may be processed by one or a plurality of computers provided on the cloud, and the processing result may be transmitted to the information processing device 30 in the robot 10.

The various operations executed by the information processing apparatus 30 in the above-described embodiment may be executed in parallel processing by using one or a plurality of processors or by using a plurality of computers via a network. Further, various operations may be distributed to a plurality of arithmetic cores in the processor and executed in parallel processing. In addition, some or all of the processes, means, etc. of the present disclosure may be executed by at least one of a processor and a storage device provided on the cloud that can communicate with the computer 90 via the network. As described above, the information processing apparatus 30 in the above-described embodiment may be in the form of parallel computing by one or a plurality of computers.

The processor 91 may be an electronic circuit (processing circuit, processing circuit, processing circuitry, CPU, GPU, FPGA, ASIC, etc.) including a control device and an arithmetic unit of a computer. Further, the processor 91 may be a semiconductor device or the like including a dedicated processing circuit. The processor 91 is not limited to an electronic circuit using an electronic logic element, and may be realized by an optical circuit using an optical logic element. Further, the processor 91 may include an arithmetic function based on quantum computing.

The processor 91 can perform arithmetic processing based on data and software (programs) input from the information processing apparatus 30 and the like of the internal configuration of the computer 90, and output the arithmetic results and control signals to each apparatus and the like. The processor 91 may control each component constituting the computer 90 by executing an OS (Operating System) of the computer 90, an application, or the like.

The information processing device 30 in the above-described embodiment may be realized by one or more processors 91. Here, the processor 91 may refer to one or more electronic circuits arranged on one chip, or may refer to one or more electronic circuits arranged on two or more chips or two or more devices. You may point. When a plurality of electronic circuits are used, each electronic circuit may communicate by wire or wirelessly.

The main storage device 92 is a storage device that stores instructions executed by the processor 91, various data, and the like, and the information stored in the main storage device 92 is read out by the processor 91. The auxiliary storage device 93 is a storage device other than the main storage device 92. Note that these storage devices mean arbitrary electronic components capable of storing electronic information, and may be semiconductor memories. The semiconductor memory may be either a volatile memory or a non-volatile memory. The storage device for storing various data in the information processing device 30 in the above-described embodiment may be realized by the main storage device 92 or the auxiliary storage device 93, or may be realized by the built-in memory built in the processor 91. good. For example, the storage unit 56 in the above-described embodiment may be realized by the main storage device 92 or the auxiliary storage device 93.

A plurality of processors may be connected (combined) or a single processor may be connected to one storage device (memory). A plurality of storage devices (memory) may be connected (combined) to one processor. When the information processing device 30 in the above-described embodiment is composed of at least one storage device (memory) and a plurality of processors connected (combined) to the at least one storage device (memory), among the plurality of processors At least one processor may include a configuration in which it is connected (combined) to at least one storage device (memory). Further, this configuration may be realized by a storage device (memory) and a processor included in a plurality of computers. Further, a configuration in which the storage device (memory) is integrated with the processor (for example, a cache memory including an L1 cache and an L2 cache) may be included.

The network interface 94 is an interface for connecting to a communication network wirelessly or by wire. As the network interface 94, an appropriate interface such as one conforming to an existing communication standard may be used. Information may be exchanged with the first external device connected via the communication network by the network interface 94. The communication network may be any one of WAN (Wide Area Network), LAN (Local Area Network), PAN (Personal Area Network), or a combination thereof, and may be a combination of the computer 90 and the first external device. Any information can be exchanged between them. An example of WAN is the Internet, an example of LAN is IEEE802.11, Ethernet (registered trademark), etc., and an example of PAN is Bluetooth (registered trademark), NFC (Near Field Communication), etc.

The device interface 95 is an interface such as USB that directly connects to the second external device. The second external device is, for example, an arm 24, a camera 26, and the like.

In the present specification (including claims), the expression (including similar expressions) of "at least one (one) of a, b and c" or "at least one (one) of a, b or c" is used. When used, it includes any of a, b, c, ab, ac, bc, or abc. It may also include multiple instances of any element, such as a-a, a-b-b, a-a-b-b-c-c, and the like. It also includes adding elements other than the listed elements (a, b and c), such as having d, such as a-b-c-d.

In the present specification (including claims), when expressions such as "with data as input / based on / according to / according to" (including similar expressions) are used, unless otherwise specified. This includes the case where various data itself is used as an input, and the case where various data is processed in some way (for example, noise-added data, normalized data, intermediate representation of various data, etc.) is used as input. In addition, when it is stated that some result can be obtained "based on / according to / according to the data", it includes the case where the result can be obtained based only on the data, and other data other than the data. It may also include cases where the result is obtained under the influence of factors, conditions, and / or conditions. In addition, when it is stated that "data is output", unless otherwise specified, various data itself is used as output, or various data is processed in some way (for example, noise is added, normal). It also includes the case where the output is output (intermediate representation of various data, etc.).

In the present specification (including claims), when the terms "connected" and "coupled" are used, direct connection / coupling and indirect connection / coupling are used. , Electrically connected / combined, communicatively connected / combined, operatively connected / combined, physically connected / combined, etc. Intended as a term. The term should be interpreted as appropriate according to the context in which the term is used, but any connection / combination form that is not intentionally or naturally excluded is not included in the term. It should be interpreted in a limited way.

When the expression "A configured to B" is used in the present specification (including claims), the physical structure of the element A can perform the operation B. Including that the element A has a configuration and the permanent or temporary setting (setting / configuration) of the element A is set (configured / set) to actually execute the operation B. good. For example, when the element A is a general-purpose processor, the processor has a hardware configuration capable of executing the operation B, and the operation B is set by setting a permanent or temporary program (instruction). It suffices if it is configured to actually execute. Further, when the element A is a dedicated processor, a dedicated arithmetic circuit, or the like, the circuit structure of the processor actually executes the operation B regardless of whether or not the control instruction and data are actually attached. It only needs to be implemented.

In the present specification (including claims), when a term meaning inclusion or possession (for example, "comprising / including" and "having", etc.) is used, the object of the term is used. It is intended as an open-ended term, including the case of containing or owning an object other than the indicated object. If the object of these terms that mean inclusion or possession is an expression that does not specify a quantity or suggests a singular (an expression with a or an as an article), the expression is interpreted as not being limited to a specific number. It should be.

In the present specification (including claims), expressions such as "one or more" or "at least one" are used in some places, and the quantity is specified in other places. Even if expressions that do not or suggest the singular (expressions with a or an as an article) are used, the latter expression is not intended to mean "one". In general, expressions that do not specify a quantity or suggest a singular (expressions with a or an as an article) should be interpreted as not necessarily limited to a particular number.

In the present specification, when it is stated that a specific effect (advantage / result) can be obtained for a specific configuration having an embodiment, unless there is a specific reason, another one or more having the configuration. It should be understood that the effect can also be obtained in the examples of. However, it should be understood that the presence or absence of the effect generally depends on various factors, conditions, and / or states, etc., and that the effect cannot always be obtained by the configuration. The effect is merely obtained by the configuration described in the examples when various factors, conditions, and / or conditions are satisfied, and in the invention relating to the claim that defines the configuration or a similar configuration. , The effect is not always obtained.

In the present specification (including claims), when terms such as "maximize" are used, the global maximum value is obtained, the approximate value of the global maximum value is obtained, and the local maximum value is obtained. Should be interpreted as appropriate according to the context in which the term is used, including finding an approximation of the local maximum. It also includes probabilistically or heuristically finding approximate values of these maximum values. Similarly, when terms such as "minimize" are used, find the global minimum, find the approximation of the global minimum, find the local minimum, and find the local minimum. It should be interpreted as appropriate according to the context in which the term was used, including finding an approximation of the value. It also includes probabilistically or heuristically finding approximate values of these minimum values. Similarly, when terms such as "optimize" are used, finding a global optimal value, finding an approximation of a global optimal value, finding a local optimal value, and local optimization It should be interpreted as appropriate according to the context in which the term was used, including finding an approximation of the value. It also includes probabilistically or heuristically finding approximate values of these optimal values.

In the present specification (including claims), when a plurality of hardware performs a predetermined process, the respective hardware may cooperate to perform the predetermined process, or some hardware may perform the predetermined process. You may do all of the above. Further, some hardware may perform a part of a predetermined process, and another hardware may perform the rest of the predetermined process. In the present specification (including claims), when expressions such as "one or more hardware performs the first process and the one or more hardware performs the second process" are used. , The hardware that performs the first process and the hardware that performs the second process may be the same or different. That is, the hardware that performs the first process and the hardware that performs the second process may be included in the one or more hardware. The hardware may include an electronic circuit, a device including the electronic circuit, or the like.

In the present specification (including claims), when a plurality of storage devices (memory) store data, each storage device (memory) among the plurality of storage devices (memory) stores only a part of the data. It may be stored or the entire data may be stored.

Although the embodiments of the present disclosure have been described in detail above, the present disclosure is not limited to the individual embodiments described above. Various additions, changes, replacements, partial deletions, etc. are possible without departing from the conceptual idea and purpose of the present invention derived from the contents defined in the claims and their equivalents. For example, in all the above-described embodiments, when numerical values or mathematical formulas are used for explanation, they are shown as examples, and the present invention is not limited thereto. Further, the order of each operation in the embodiment is shown as an example, and is not limited to these.

10 Robot 22 Main body 24 Arm 26 Camera 28 End effector 30 Information processing device 42 Acquisition unit 44 Object detection unit 48 Arm identification unit 50 Target identification unit 52 Reference object identification unit 56 Storage unit 58 Arm control unit 72 Spoon 74 Fork 76 Knife

Claims (18)

The first movable part and
With at least one processor controlling the first part,
With
When the processor cannot calculate the current target position as the movement target of the predetermined portion based on the current data indicating at least a part of the movable range of the predetermined portion of the first portion, the predetermined portion of the predetermined portion. The first part is controlled so as to move the predetermined portion to the predicted target position which is the movement target.
The predicted target position includes a past target position that is a movement target in the predetermined portion calculated based on past data acquired in the past, a past reference position that is a position of a reference object calculated based on the past data, and the current data. And the robot calculated based on. The processor
Acquire the current data and
Based on the current data, the current target position is calculated.
One of the plurality of objects included in the current data is specified as the reference object, and the object is specified.
Calculate the current reference position, which is the position of the reference object,
The robot according to claim 1, wherein the predicted target position is calculated based on the current reference position. The robot according to claim 2, wherein when the current target position can be calculated based on the current data, the first part is controlled so as to move the predetermined portion to the current target position. A data acquisition unit capable of acquiring the predetermined portion in the first portion is further provided.
The robot according to claim 2 or 3, wherein the processor acquires the current data acquired by the data acquisition unit. The robot according to claim 4, wherein the first portion is an arm. Equipped with a main body
The arm is provided on the main body portion.
The robot according to claim 5, wherein the data acquisition unit is provided at a position different from the arm in the main body unit. The predetermined portion is an end effector provided at the tip of the arm for performing a predetermined operation on an object.
The robot according to claim 5 or 6, wherein the processor calculates a predetermined operable position with respect to a target object to be operated by the end effector as the current target position. The end effector is a gripping mechanism and
The robot according to claim 7, wherein the processor calculates a position where the end effector can grip the target object as the current target position. According to any one of claims 2 to 8, the processor executes the acquisition of the current data, the calculation of the current target position, the calculation of the current reference position, and the control of the predetermined portion at predetermined time intervals. The robot described. The robot according to any one of claims 2 to 9, wherein the processor specifies, among the plurality of objects, an object that is fixed relative to the target object to be operated as the reference object. .. The processor is any one of claims 2 to 10 that specifies, among the plurality of objects, an object in which the positional difference between the past target position and the past reference position is within a predetermined range as the reference object. The robot described in the section. The processor identifies a plurality of reference objects among the plurality of objects, and the processor identifies a plurality of reference objects.
When the current target position cannot be calculated based on the current data, the processor uses the current reference position and the positions of the past target position and the past reference position for each of the plurality of reference objects. The robot according to any one of claims 2 to 11, which calculates the predicted target position based on the difference. The processor
For each of the plurality of reference objects, an individual predicted target position to be a movement target of the predetermined portion is calculated based on the current reference position and the position difference.
The robot according to claim 12, wherein the predicted target position is calculated by averaging the individual predicted target positions calculated for each of the plurality of reference objects. The processor
Based on the current data, the current target position and the current target posture, which is the posture target of the predetermined portion, are calculated.
The current reference position and the current reference posture, which is the posture of the reference object, are calculated.
When the current target position or the current target posture cannot be calculated based on the current data.
Calculate the predicted target position and
The posture target of the predetermined portion is based on the posture difference between the past target posture which is the posture target in the predetermined portion calculated based on the past data and the past reference posture which is the posture of the reference object calculated based on the past data. Calculate the predicted target posture to be
The robot according to any one of claims 2 to 13, which controls the predetermined portion to move to the predicted target position and controls the predetermined portion so as to have the current target posture. In a robot that identifies an object from the captured image and controls the first part of the object.
When the object is not specified in the captured image, the first part is controlled by using at least an image captured in the past that can identify the object in addition to the image. A robot equipped with a control device. It is a control method of a robot having a movable first part.
When the current target position that is the movement target of the predetermined part cannot be calculated based on the current data indicating at least a part of the movable range of the predetermined part of the first part, the movement target of the predetermined part is obtained. The first part is controlled so as to move the predetermined part to the predicted target position.
The predicted target position includes a past target position that is a movement target in the predetermined portion calculated based on past data acquired in the past, a past reference position that is a position of a reference object calculated based on the past data, and the current data. And the control method calculated based on. An information processing device that controls a robot equipped with a movable first part.
When the current target position that is the movement target of the predetermined part cannot be calculated based on the current data indicating at least a part of the movable range of the predetermined part of the first part, the movement target of the predetermined part is obtained. The first part is controlled so as to move the predetermined part to the predicted target position.
The predicted target position includes a past target position that is a movement target in the predetermined portion calculated based on past data acquired in the past, a past reference position that is a position of a reference object calculated based on the past data, and the current data. An information processing device calculated based on. A program executed by a processor that controls a movable first part.
To the processor
When the current target position that is the movement target of the predetermined part cannot be calculated based on the current data indicating at least a part of the movable range of the predetermined part of the first part, the movement target of the predetermined part is obtained. The first part is controlled so as to move the predetermined part to the predicted target position.
The predicted target position includes a past target position that is a movement target in the predetermined portion calculated based on past data acquired in the past, a past reference position that is a position of a reference object calculated based on the past data, and the current data. And a program calculated based on.

Images