JP2024005599A - Data collection device and control device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a data collection device and control device, which enable efficient collection of datasets for machine learning.

SOLUTION: A data collection device is provided, comprising an acquisition unit for acquiring captured image data of an object deformed by a robot arm, a determination unit configured to determine whether the object is flexible or not based on the amount of deformation of the object, and a storage unit configured to store the image data of the object in association with a determination result of the determination unit in order to use the data for controlling the robot arm.

SELECTED DRAWING: Figure 1

COPYRIGHT: (C)2024,JPO&INPIT

Description

The present invention relates to a data collection device and a control device that collect data sets for AI (artificial intelligence) learning.

Due to recent advances in robot technology, automation using robot systems in factories and distribution warehouses is progressing. In order to apply such a robot system to a store or other workplace where many general consumers are present, the robot system needs to be robust against the environment. On the other hand, especially regarding image recognition, it is difficult to ensure robustness against changing environments. In order to realize image recognition that is robust to the environment, image recognition technology using AI is required. By recognizing the type, position, and posture of an object using AI image recognition, a robot can grasp and manipulate the object.

However, image recognition using AI requires a huge amount of training data sets. Collecting huge amounts of training datasets is expensive, so there is great value in reducing collection costs. Furthermore, in order to recognize the position and orientation of an object, it is important not only to photograph the object, but also to label the photographed image with position information and orientation information of the object. Therefore, a method has been disclosed that recognizes an image of the target object and markers placed at known positions around the target object, and estimates the position and orientation of the target based on relative position information and orientation information from the markers. (Patent Document 1).

International Publication No. 2019/189661

However, with the conventional method described above, it is possible to label the position and orientation of an object in an image for a rigid object, but it is always possible to label the same position and orientation for an object that is a flexible object. Photographs will be taken in the deformed state. For this reason, there is a possibility that various deformation states of the flexible object cannot be added to the data set. Further, when a flexible object becomes deformed in a state different from that at the time of data set collection, there is a possibility that the correct position and orientation of the object cannot be estimated.

The present disclosure aims to provide a data collection device and a control device that efficiently collect data sets for machine learning.

The data collection device of the present disclosure includes an acquisition unit that acquires image data of an object deformed by a robot arm, and determines whether the object is a flexible object based on the amount of deformation of the object. It includes a determination section and a storage section that stores image data of the object in association with the determination result of the determination section for use in controlling the robot arm.

The control device of the present disclosure associates image data obtained by photographing an object deformed by a robot arm with a determination result of determining whether the object is a flexible object based on the amount of deformation of the object. The robot arm includes a storage unit that stores a control model trained using the data obtained as teacher data, and a control unit that controls the robot arm based on the control model stored in the storage unit.

According to the present disclosure, data sets for machine learning can be efficiently collected.

1 is a diagram showing the configuration of a robot arm device including a data collection device according to Embodiment 1. FIG. FIG. 6 is a diagram illustrating how image data is labeled with determination results and the position and orientation of a target object with respect to a photographing position. FIG. 3 is a diagram showing how the gripping position of the robot arm is calculated. 1 is a block diagram showing the configuration of a data collection device according to Embodiment 1. FIG. 7 is a flowchart showing the operation of the first embodiment. FIG. 3 is a diagram showing how the gripping position of the robot arm is calculated. FIG. 2 is a block diagram showing the configuration of a learning device according to Embodiment 2. FIG. FIG. 7 is a diagram showing the configuration of a robot arm device including a control device according to a third embodiment.

Embodiments of the present invention will be described below with reference to the drawings.

(Embodiment 1)
The data collection device of the present disclosure operates a robot arm and a rotary table to accurately estimate the position and orientation of a flexible object that takes various deformed states. Change the pose of the object. Then, the data collection device photographs the object with a camera while deforming the object using the robot hand, and calculates the position of the object with respect to the camera from the geometric relationship (geometric information) of the robot arm. and the posture of the object in the image data.

First, the configuration of the robot arm device 100 including the data collection device 21 according to the present disclosure will be described. FIG. 1 shows the overall configuration of a robot arm device 100 according to the first embodiment.

Robot arm device 100 includes a robot arm 7 , a camera 4 , a rotary table 5 , a control device 8 , and a data collection device 21 . The control device 8 and the data collection device 21 are electrically connected to each other. Further, the control device 8 and the data collection device 21 are electrically connected to the robot arm 7, camera 4, and rotary table 5, respectively.

The robot arm 7 includes a vertical multi-joint robot 2 in which adjacent links are connected to each other through a plurality of joints, and a robot hand 3 that functions as a hand. The vertical articulated robot 2 has an elongated shape, and its base end is fixed to the fixing base 1. This fixing base 1 is for fixing the robot arm 7 and the rotary table 5, and is fixed to the floor, for example. The robot hand 3 grips an object T placed on a rotary table 5 and is attached to the tip of the vertically articulated robot 2 .

The camera 4 is used to photograph the object T placed on the rotary table 5, and is attached to the tip of the vertically articulated robot 2.

The rotary table 5 changes the attitude (photographing angle) of the object T with respect to the camera 4, and is fixed to the fixing base 1. The rotary table 5 is rotated by a servo motor, and may be rotated, for example, in the circumferential direction 6 of a rotating shaft extending in the vertical direction. Note that in this embodiment, the rotary table 5 is used, but the present invention is not limited to this as long as the photographing angle of the object T can be changed. For example, instead of using the rotary table 5, a plurality of cameras may be arranged. This allows efficient photographing.

The data collection device 21 includes an acquisition section 22, a determination section 23, and a storage section 24.

When the object T deformed by the robot hand 3 of the robot arm 7 is photographed by the camera 4, the acquisition section 22 photographs the object T in order to determine the flexibility of the object T at the determination section 23. Image data is acquired from the camera 4. Further, the acquisition unit 22 includes a position/orientation calculation unit 22a and an area calculation unit 22b.

The area calculation unit 22b acquires image data of the object T before transformation by the robot hand 3 and image data of the object T after transformation, and calculates the area of the area of the object T in each image data. For example, the area calculation unit 22b obtains in advance background image data of the rotary table 5 in a state where the target object T is not present from the camera 4, and based on the background image data, the area calculation unit 22b generates an image of the target object T. By subtracting the background portion from the data, the object region, which is the region where the object T exists, may be extracted. Then, the area calculation unit 22b calculates the area of the target object T based on the extracted target object area. This region area changes depending on the amount of deformation of the object T, and therefore can serve as an index indicating the flexibility of the object T.

The position/orientation calculation unit 22a calculates the position and orientation of the object T with respect to the photographing position of the image data, for example, based on the geometric information of the robot arm 7. For example, the position/orientation calculation unit 22a changes the object T, whose determination is made as to whether it is a flexible object by the determination unit 23, to various positions and orientations, and changes the position and orientation of the object T relative to the photographing position of the image data. may be calculated sequentially.

At this time, the robot arm 7, camera 4, and rotary table 5 are each geometrically restrained via the fixing base 1. The position/orientation calculation unit 22a also includes a camera coordinate system 12 indicating the position and attitude of the camera 4, a base coordinate system 13 indicating the position and attitude of the base end of the robot arm 7, and the position and rotation direction of the rotary table 5. An object coordinate system 14 is set in advance. _Here , as shown in FIG. ₁ , for example, the camera coordinate system 12 is composed of unit vectors ( _e You may also set it to change. Similarly, the base coordinate system 13 and the object coordinate system 14 are each composed of unit vectors (e _x , e _y , e _z ) in mutually orthogonal x, y, and z directions. The position and orientation may be changed together with the rotary table 5. Note that since the position and orientation of the object T change depending on the position and rotation direction of the rotary table 5, the object coordinate system 14 indicates the position and orientation of the object T. Therefore, the position/orientation calculation unit 22a calculates changes in these coordinate systems based on each joint angle of the robot arm 7 and the rotation angle of the rotary table 5, and changes the camera 4 based on the calculated changes in the coordinate system. The position and orientation of the target object T relative to the target object T may be calculated.

Specifically, the position/orientation calculation unit 22 a calculates the position and orientation of the base coordinate system 13 of the robot arm 7 with respect to the camera coordinate system 12 based on the joint angles of the robot arm 7 . Further, the position/orientation calculation unit 22a calculates the position and orientation of the object T in the object coordinate system 14 with respect to the base coordinate system 13 of the robot arm 7 based on the rotation angle of the rotary table 5. As a result, the camera coordinate system 12 and the object coordinate system 14 are associated with each other via the base coordinate system 13 of the robot arm 7 . Therefore, the position/orientation calculation unit 22a calculates the object relative to the camera coordinate system 12 based on the position and orientation of the base coordinate system 13 relative to the camera coordinate system 12 and the position and orientation of the object coordinate system 14 relative to the base coordinate system 13. The position and orientation of the object coordinate system 14 may be calculated. For example, the position (x, y, z) of the camera coordinate system 12 with respect to the base coordinate system 13 is (-2, -2, -1), and the position of the object coordinate system 14 with respect to the base coordinate system 13 is (-2, -2,0), the position of the object coordinate system 14 with respect to the camera coordinate system 12 will be calculated as (0,0,1). Similarly, changes in the orientation of the object coordinate system 14 with respect to the camera coordinate system 12 are determined by, for example, the direction of the unit vector of the camera coordinate system 12 with respect to the unit vector of the base coordinate system 13, and the direction of the object coordinate system 14 with respect to the unit vector of the base coordinate system 13. It may be calculated based on the direction of the unit vector of the coordinate system 14.

The determining unit 23 determines whether the object T is a flexible object based on the amount of deformation of the object T. For example, the determining unit 23 may determine whether the target object T is a flexible object based on the area area of the target object T calculated by the area calculating unit 22b.

The storage unit 24 associates the image data of the object T with the determination result of the determination unit 23, that is, “it is a flexible object” or “it is not a flexible object” in order to use it for controlling the robot arm 7. Save the dataset. Furthermore, the storage unit 24 may store the position and orientation of the object T relative to the photographing position calculated by the position/orientation calculation unit 22a in association with the image data.

For example, as shown in FIG. 2, the storage unit 24 may label and save the image data D obtained by photographing the object T with the determination result and the position and orientation of the object T with respect to the photographing position. good. In this way, the data that is made into a set by labeling the determination result and the position and orientation of the target object T for the image data D can be used for training a machine learning model as described later.

Note that the determination unit 23 may determine whether the target object T is a rigid object. For example, the determination unit 23 may determine that the object T other than the flexible object is a rigid object. In this case, the storage unit 24 may store image data of not only the object T made of a flexible object but also the object T made of a rigid object. At this time, the method of photographing the image data D stored in the storage unit 24 may be different depending on whether the object T is a flexible object or a rigid object. For example, when the object T is a flexible object, the object T may be photographed after applying an external force with the robot hand 3, and the image data D may be stored in the storage unit 24. Thereby, image data D of the object T transformed into various shapes can be obtained. On the other hand, when the object T is a rigid object, the object T may be photographed without applying any external force, and the image data D may be stored in the storage unit 24.

The control device 8 controls each part of the robot arm device 100 in order to collect a data set for machine learning including image data D obtained by photographing the object T into the data collection device 21 .

Next, an example in which the determination unit 23 determines the target object T will be described.

First, the control device 8 operates each joint of the robot arm 7 based on the camera coordinate system 12, the base coordinate system 13, and the object coordinate system 14, and presets the tip of the robot arm 7 where the camera 4 is placed. the initial position and attitude. The initial position and initial posture may be set in advance so that the entire object T is included near the center of the photographing area of the camera 4, for example.

The control device 8 controls the camera 4 to photograph the object T when the robot arm 7 is in the initial position and initial posture. The camera 4 outputs the captured image data D to the area calculation section 22b of the acquisition section 22. Then, the area calculation unit 22b extracts the object area corresponding to the object T from the image data D captured by the camera 4.

Subsequently, the control device 8 controls the robot hand 3 to grip the object T with a preset standard gripping force, and releases the object T after several seconds have elapsed from the gripping state. Thereafter, the robot arm 7 is returned to the initial position and initial posture, and the object T is photographed again by the camera 4. Then, the area calculation unit 22b extracts the object area from the image data D captured by the camera 4. The area calculation unit 22b calculates the area of the object region in the image data D based on the extracted object region, and calculates the reduction rate of the area of the object region after gripping with respect to the area of the object region before gripping using the following formula. Calculated by 1.

The determination unit 23 determines that the object T is a flexible object when the rate of decrease in the area of the object region is equal to or greater than the reference value. On the other hand, the determination unit 23 determines that the target object T is a rigid object when the reduction rate of the area of the target object region is less than the reference value.

Subsequently, the acquisition unit 22 acquires image data D obtained by photographing various deformed states of the object T determined by the determination unit 23 to be a flexible object. At this time, the control device 8 may control the robot hand 3 to grip the object T and apply an external force in order to deform the object T.

Next, an example in which the object T is deformed by the robot hand 3 in order to collect the image data D will be described.

The robot hand 3 deforms the flexible object T by pinching the object T. At this time, in order to efficiently collect deformation pattern images of the object T, it is required to change the object T into various deformed states in a short time. It is necessary to quickly determine the force and gripping position.

Therefore, as shown in FIG. 3, the area calculation unit 22b extracts the object region R from the image data D obtained by photographing the object T in an undeformed state (no force is applied by the robot hand 3). Next, the area calculation unit 22b calculates the contour width W in the width direction D1 parallel to the rotary table 5 at a plurality of height positions (height perpendicular to the rotary table 5) based on the contour of the extracted object region R. position in direction D2). Then, the control device 8 selects the positions where the contour width W is the largest as the gripping positions P1 and P2, and controls the robot hand 3 to grip the gripping positions P1 and P2.

As a result, when the determination unit 23 determines that the object T is a flexible object, the storage unit 24 stores image data D in which the object T is deformed by grasping the positions P1 and P2. become. That is, the storage unit 24 stores image data D of the object T that has been deformed by the robot arm 7 at positions P1 and P2 where the contour width W in the predetermined width direction D1 is the longest in the undeformed object T. It is stored in association with the determination result of the determination unit 23. In this way, the robot hand 3 can deform the object T to a larger extent by grasping the positions P1 and P2 where the contour width W is the longest in the object T, thereby creating an efficient deformation pattern image. can be collected.

At this time, the gripping force of the robot hand 3 may be set according to the size of the contour width W (gripping contour width) of the gripping positions P1 and P2. For example, the control device 8 may determine the gripping force of the robot hand 3 according to the size of the gripping contour width W with respect to a preset reference contour width. Thereby, it is possible to grip the position of the object T with the largest width (largest bulge) with a gripping force corresponding to the width. A calculation formula for determining the gripping force of the robot hand 3 is shown in Formula 2 below.

When the grip contour width W is the same as the reference contour width, the control device 8 controls the robot hand 3 to grip with a preset reference grip force. Furthermore, when the gripping contour width W is larger than the reference contour width, the control device 8 controls the robot hand to grip the robot hand with a gripping force greater than the standard gripping force in accordance with the amount of increase in the gripping contour width W with respect to the standard contour width. Control 3. At this time, if the preset maximum gripping force is exceeded, the control device 8 controls the robot hand 3 to grip with the maximum gripping force. As a result, image data D obtained by applying substantially the same amount of deformation to various objects T can be collected.

In the first embodiment described above, in order to deform the object T, an external force is applied to the object T by holding the object T between the robot hands 3. This method of applying external force by pinching applies the same external force inward from both sides (gripping positions P1 and P2) of the object T, so that the object T does not move significantly on the rotary table 5. It is possible to apply an external force to T.

FIG. 4 shows the hardware configuration of the data collection device 21.

The data collection device 21 includes a storage unit 24, a processor 25, and a user interface (UI) 26 that are interconnected via a bus B.

Note that programs or instructions for realizing various functions and processes to be described later in the data collection device 21 may be downloaded from any external device via a network or the like, or may be stored in a CD-ROM (Compact Disk-Read Only Memory). , may be provided from a removable storage medium such as a flash memory.

The storage unit 24 is implemented by one or more non-transitory storage media, such as random access memory, flash memory, hard disk drive, etc., and stores the programs or instructions as well as the installed programs or instructions. Stores files, data, etc. used for execution.

The processor 25 may be realized by one or more CPUs (Central Processing Units), GPUs (Graphics Processing Units), processing circuits, etc. that may be configured from one or more processor cores. The processor 25 executes various functions and processes of the control device 8, which will be described later, in accordance with programs, instructions, and data such as parameters necessary to execute the programs or instructions stored in the storage unit 24.

The UI 26 may be composed of input devices such as a keyboard, mouse, camera, microphone, etc., output devices such as a display, speaker, headset, printer, etc., and input/output devices such as a touch panel. Realize the interface. For example, the user may operate the control device 8 by operating a GUI (Graphical User Interface) displayed on a display or touch panel using a keyboard, a mouse, or the like.

Note that the hardware configuration described above is merely an example, and the control device 8 according to the present disclosure may be realized by any other suitable hardware configuration.

Next, the data collection operation of this embodiment will be explained with reference to the flowchart shown in FIG.

First, in step S1, the control device 8 moves the robot arm 7 to a preset initial position and initial posture. The initial position and initial posture may be set, for example, so that the entire object T placed on the rotary table 5 is included near the center of the imaging area of the camera 4. The control device 8 controls the robot arm 7 based on geometric information of the robot arm 7, such as a camera coordinate system 12, a base coordinate system 13, and an object coordinate system 14, so that the tip of the robot arm 7 is at an initial position and an initial posture. 7 joints are driven.

Subsequently, the control device 8 controls the camera 4 to photograph the object T. The camera 4 outputs the captured image data D to the area calculation section 22b of the acquisition section 22. Then, as shown in FIG. 6, the area calculation unit 22b extracts the object region R corresponding to the object T from the image data D and calculates the area of the object region R in step S2.

When the object T is photographed in this manner, the control device 8 controls the robot arm 7 to grip the predetermined positions Pa1 and Pa2 of the object T with a standard gripping force in step S3. At this time, the predetermined positions Pa1 and Pa2 may control the robot arm 7 to, for example, grip a side portion corresponding to the center portion C of the object region R in the height direction D2 from the photographing direction. That is, the control device 8 causes the robot to grip two points Pa1 and Pa2 where a straight line passing through the center C of the object region R in the height direction D2 and extending in the width direction D1 intersects the outline of the object region R. Controls arm 7.

After that, the control device 8 moves the robot arm to the initial position and initial posture again in step S4. Further, the control device 8 controls the camera 4 to photograph the object T, and the photographed image data D is output from the camera 4 to the acquisition unit 22. Then, the area calculation unit 22b of the acquisition unit 22 extracts the object region R from the image data D and calculates the area of the object region R in step S5.

Subsequently, in step S6, the determination unit 23 determines whether or not the object T is a flexible object based on the rate of decrease in the area of the object region R after gripping relative to the object region R before gripping. .

When it is determined that the object T is not a flexible object, that is, it is determined that it is a rigid object, the control device 8 rotates the rotary table 5 by a certain angle and then controls the camera to photograph the object T in step S7. Control 4. Subsequently, the control device 8 controls the camera 4 to photograph the object T after further rotating the rotary table 5 by the same fixed angle as before. In this way, the control device 8 repeatedly photographs the object T from different angles until the rotary table 5 rotates once.

When the rotary table 5 rotates once, the control device 8 changes each joint angle of the robot arm 7, that is, changes the position (for example, height position) of the object T with respect to the imaging position, and again rotates the rotary table 5 at a constant angle. The camera 4 is controlled to photograph the object T while rotating it. The control device 8 repeats photographing the object T from different angles until the rotary table 5 rotates once. In this way, the object T is photographed at various positions and postures. Then, in step S8, the storage unit 24 stores the image data D of the object T photographed at various positions and postures in association with the determination result that the object T is a “rigid object” determined by the determination unit 23. do. Furthermore, the storage unit 24 may store information related to the object region in association with the image data D of the rigid object.

In this way, the data collection device 21 can efficiently collect a data set for machine learning including the image data D of a rigid object. Furthermore, by changing the angle of each joint of the robot arm 7, it is possible to photograph the object T at different angles of view, and it is possible to collect image data D of the rigid body photographed at various positions and postures. .

At this time, the position/orientation calculation unit 22a calculates the position of the object T relative to the photographing position of the image data D based on the geometric information of the robot arm 7, for example, the camera coordinate system 12, the base coordinate system 13, and the object coordinate system 14. and the posture may be calculated. Then, the storage unit 24 stores image data D of the object T photographed at various positions and angles in association with the position and orientation of the object T with respect to the photographing position.

Thereby, it is possible to efficiently collect teacher data in which the image data D of the rigid body is labeled with the position and orientation relationship between the camera 4 and the target object T.

Further, in step S9, the control device 8 determines whether each joint angle of the robot arm 7 has been changed a predetermined number of times or more. If each joint angle of the robot arm 7 has not been changed more than the specified number of times, the control device 8 changes each joint angle of the robot arm 7 in step S10, and then returns to step S7 to photograph the object T. The camera 4 is controlled as follows. The control device 8 repeats steps S7, S8, S9, and S10 until it is determined that each joint angle of the robot arm 7 has been changed a specified number of times or more. Then, in step S9, when the control device 8 changes each joint angle of the robot arm 7 a specified number of times or more, it ends the collection of the image data D.

On the other hand, if it is determined in step S6 that the object T is a flexible object, the process proceeds to step S11, where the area calculation unit 22b extracts the object region R from the image data D. As shown in FIG. 3, the control device 8 determines the gripping positions P1 and P2 of the robot hand 3 and the gripping force based on the size of the contour width W of the extracted object region R. Next, in step S12, the control device 8 controls the robot hand 3 to apply an external force to the object T using the determined gripping positions P1 and P2 and the gripping force.

The control device 8 controls the camera 4 to photograph the object T after rotating the rotary table 5 by a certain angle in step S13. Subsequently, the control device 8 controls the camera 4 to photograph the object T after further rotating the rotary table 5 by the same fixed angle as before. In this way, the control device 8 repeatedly photographs the object T from different angles until the rotary table 5 rotates once.

When the rotary table 5 rotates once, the control device 8 changes each joint angle of the robot arm 7, that is, changes the position (for example, height position) of the object T with respect to the imaging position, and again rotates the rotary table 5 at a constant angle. The camera 4 is controlled to photograph the object T while rotating it. The control device 8 repeats photographing the object T from different angles until the rotary table 5 rotates once. In this way, the object T is photographed at various positions and postures. Then, in step S14, the storage unit 24 stores the image data D of the object T photographed at various positions and postures in association with the determination result that the object T is a “flexible object” determined by the determination unit 23. do. As a result, the image data D of the object T is stored in the storage unit 24 in association with the determination result that it is a "flexible object" or a "rigid object." At this time, the storage unit 24 determines whether or not the object T is a flexible object with respect to two image data D obtained by photographing the object T before and after deformation by the robot arm 7 (before and after grasping). The determined results may be stored in association with each other. Furthermore, the storage unit 24 may store information related to the object region R in association with the image data D of the flexible object.

In this way, the data collection device 21 can efficiently collect a data set for machine learning including the image data D of the flexible object. Furthermore, by changing the angle of each joint of the robot arm 7, the object T can be photographed at different angles of view, and image data D of the flexible object photographed at various positions and postures can be collected. .

At this time, the position/orientation calculation unit 22a may calculate the position and orientation of the object T with respect to the photographing position of the image data D based on the geometric information of the robot arm 7. Then, the storage unit 24 stores the image data D of the object T photographed at various positions and postures in association with the position and posture of the object T.

Thereby, it is possible to efficiently collect teacher data in which the image data D of the flexible object is labeled with the position and posture relationship between the camera 4 and the target object T. In this way, the storage unit 24 stores the determination result of whether the image data D is a flexible object or a rigid object, the position of the object T with respect to the photographing position, and the posture of the object T with respect to the photographing posture. Labeled teacher data will be collected.

Further, in step S15, the control device 8 determines whether each joint angle of the robot arm 7 has been changed a predetermined number of times or more. If each joint angle of the robot arm 7 has not been changed more than the specified number of times, the control device 8 changes each joint angle of the robot arm 7 in step S16, and then returns to step S11 to change the newly extracted target. Based on the object area R, the gripping positions P1 and P2 of the robot hand 3 and the gripping force are determined. The control device 8 repeats step S11, step S12, step S13, step S14, step S15, and step S16 until it is determined that each joint angle of the robot arm 7 has been changed a specified number of times or more. Then, in step S15, when the control device 8 changes each joint angle of the robot arm 7 a specified number of times or more, it ends the collection of the image data D.

According to the present embodiment, the image data D obtained by photographing the object T deformed by the robot arm 7 is obtained, and based on the amount of deformation of the object T, it is determined whether the object T is a flexible object or not. The determination unit 23 makes a determination. Then, the image data D of the object T is stored in the storage unit 24 in association with the determination result of the determination unit 23 for use in controlling the robot arm 7 . This makes it possible to efficiently collect data sets for machine learning.

(Embodiment 2)
Embodiment 2 of the present disclosure will be described below. Here, differences from the first embodiment described above will be mainly explained, and common points with the first embodiment described above will be denoted by common reference numerals, and detailed explanation thereof will be omitted.

In the second embodiment, a control model for the robot arm 7 is constructed using the data set collected in the first embodiment as training data.

For example, as shown in FIG. 7, a learning device 31 may be connected to the data collection device 21 of the first embodiment. The learning device 31 includes a learning section 32 and a control model storage section 33.

The learning unit 32 performs machine learning using the data set collected by the data collection device 21 as teacher data, and constructs a control model used to control the robot arm 7. That is, the control model associates image data D obtained by photographing the object T deformed by the robot arm 7 with the determination result of whether or not the object T is flexible. Trained as teacher data. At this time, the control model generates data in which the determination results of the object T are associated with two image data D obtained by photographing the object T before and after deformation by the robot arm 7 (before and after grasping). It may also be trained as training data. Thereby, the control model is constructed so as to input the image data D of the object T and output whether or not the object T is flexible. Further, the control model may be trained using a data set of the image data D, the determination result, and the position and orientation of the target object T as teacher data. Thereby, the control model is constructed so as to further output the position and orientation of the object T in response to the input of the image data D of the object T.

The control model storage unit 33 stores the control model constructed by the learning unit 32.

Note that the hardware configuration of the learning device 31 is the same as that of the data collection device 21 of Embodiment 1, so a description thereof will be omitted.

According to the present embodiment, the learning unit 32 determines that the object T is a flexible object based on the amount of deformation of the object T with respect to the image data D obtained by photographing the object T deformed by the robot arm 7. A control model is constructed by training the data associated with the determination results as teacher data. In this way, since the efficiently collected data set for machine learning is used, a control model can be easily constructed.

(Embodiment 3)
Embodiment 3 of the present disclosure will be described below. Here, we will mainly explain the differences from the above embodiments 1 and 2, and common points with the above embodiments 1 and 2 will be described in detail using common reference numerals. Omitted.

In the third embodiment, the robot arm 7 is controlled using the control model constructed in the second embodiment.

FIG. 8 shows a configuration of a robot arm device 100 including a control device 41 according to the present disclosure. The robot arm device 100 includes a robot arm 7, a camera 4, a rotary table 5, and a control device 41. Control device 41 is electrically connected to robot arm 7, camera 4, and rotary table 5.

The control device 41 includes a control section 42 and a storage section 43.

The storage unit 43 stores the control model constructed by the learning unit 32 of the second embodiment. That is, the storage unit 43 determines whether or not the object T is a flexible object based on the amount of deformation of the object T with respect to the image data D obtained by photographing the object T deformed by the robot arm 7. A control model trained using data associated with determination results as teacher data is stored. For example, the storage unit 43 may store a control model trained on the image data D using data in which the determination result is associated with the position and orientation of the target object T as teacher data.

The control unit 42 controls each part of the robot arm device 100 based on the control model stored in the storage unit 43. For example, when information indicating that the object T is a flexible object and the position and orientation of the object T are output from the control model in response to input of the image data D, the control unit 42 uses the information based on these information. The robot arm 7 may be controlled to grip a predetermined position with an appropriate gripping force.

Note that the hardware configuration of the control device 41 is the same as that of the data collection device 21 of Embodiment 1, so a description thereof will be omitted.

Next, the operation for controlling the robot arm of this embodiment will be explained.

First, the control unit 42 controls the robot arm 7 to grip the object T and place it on the rotary table 5. Then, the control unit 42 controls the camera 4 to photograph the object T that is not gripped by the robot arm 7, that is, the object T before deformation. Subsequently, the control unit 42 controls the robot arm 7 to grasp and deform the object T placed on the rotary table 5. For example, the control unit 42 may control the robot arm 7 to grip the object T with a preset standard gripping force. Then, the control unit 42 controls the camera 4 again to photograph the object T placed on the rotary table 5.

When the object T before and after deformation is photographed in this way, the control unit 42 inputs the two image data D before and after the deformation into the control model stored in the storage unit 43. Thereby, the control model outputs whether or not the target object T is a flexible object. When the object T is a flexible object, the control unit 42 grips the object T with a gripping force for a flexible object that is preset according to the flexible object, and performs a predetermined work such as store work. Controls the robot arm 7. On the other hand, if the object T is not a flexible object, the robot arm 7 is controlled to grip the object T with a gripping force greater than the gripping force for flexible objects, for example.

Generally, stores carry products with various flexibility. For this reason, the robot arm 7 that performs store work is required to grip the product with a gripping force that corresponds to the flexibility of the product. Therefore, in the present disclosure, the control unit 42 controls the robot arm 7 to grip the object T with a gripping force that corresponds to the flexibility of the object T, based on the above control model. Thereby, work on the target object T can be carried out appropriately.

Further, when the position and orientation of the target object T are further output from the control model, the control unit 42 controls the robot to grasp the target object T based on the output position and orientation of the target object T. Controls arm 7. Thereby, the robot arm 7 can grasp the object T at an appropriate position and perform the work.

According to the present embodiment, the control unit 42 determines whether the object T is a flexible object based on the amount of deformation of the object T, in response to the image data D obtained by photographing the object T deformed by the robot arm 7. The robot arm 7 is controlled based on a control model trained using data associated with the determination result of determining whether the robot exists or not as teacher data. Thereby, the control unit 42 can control the robot arm 7 to grip the object T with an appropriate gripping force based on whether or not the object T is a flexible object.

Although specific examples of the present disclosure have been described in detail above, these are merely examples and do not limit the scope of the claims. The technology described in the claims includes various modifications and changes to the specific examples illustrated above.

The data collection device and control device of the present invention can efficiently perform AI learning in automated robots that operate various objects.

1 Fixation base 2 Vertical articulated robot 3 Robot hand 4 Camera 5 Rotary table 6 Circumferential direction 7 Robot arm 8 Control device 12 Camera coordinate system 13 Base coordinate system 14 Object coordinate system 21 Data acquisition device 22 Acquisition unit 22a Position/ Attitude calculation unit 22b Area calculation unit 23 Judgment unit 24 Storage unit 25 Processor 26 User interface 31 Learning device 32 Learning unit 33 Control model storage unit 41 Control device 42 Control unit 43 Storage unit 100 Robot arm device C Central portion D1 Width direction D2 Height Horizontal direction P1, P2, Pa1, Pa2 Gripping position R Object area T Object W Contour width

Claims (5)

an acquisition unit that acquires image data of an object transformed by the robot arm;
a determination unit that determines whether the object is a flexible object based on the amount of deformation of the object;
a storage unit that stores image data of the object in association with the determination result of the determination unit for use in controlling the robot arm;
Data collection equipment. The acquisition unit calculates a position and orientation of the object with respect to a shooting position of the image data based on geometric information of the robot arm,
the storage unit stores the position and orientation of the object in association with the image data;
The data collection device according to claim 1. The acquisition unit acquires image data of the object before deformation and image data of the object after deformation, and calculates a region area of the object in each image data,
The determination unit determines whether the target object is a flexible object based on a region area of the target object.
The data collection device according to claim 1. When it is determined that the object is a flexible object, the storage section stores the object that has been deformed by the robot arm at a position where the outline width in a predetermined width direction is the longest in the undeformed object. storing image data of the object in association with the determination result of the determination unit;
The data collection device according to claim 3. A teacher uses data in which a judgment result of determining whether or not the object is a flexible object based on the amount of deformation of the object is associated with image data taken of an object deformed by a robot arm. a storage unit that stores the trained control model as data;
a control unit that controls the robot arm based on the control model stored in the storage unit;
Control device.

Images