WO2022107943A1

WO2022107943A1 - Intelligent smart logistics automation information processing device

Info

Publication number: WO2022107943A1
Application number: PCT/KR2020/016547
Authority: WO
Inventors: 이상설; 장성준; 박종희
Original assignee: 한국전자기술연구원
Priority date: 2020-11-23
Filing date: 2020-11-23
Publication date: 2022-05-27
Also published as: KR20220070592A

Abstract

An intelligent smart logistics automation information processing device is provided. In a robot arm control method according to an embodiment of the present invention, an optical flow is measured while a camera is moved in the distance direction of an object, whether the center of a camera image coincides with the center of weight of the object is confirmed on the basis of the measured optical flow, and when it is determined that the centers do not coincide, the camera is moved on a plane perpendicular to the distance direction of the object. Accordingly, through image generation and object recognition through a lightweight camera and a lightweight deep learning network and object-oriented tracking through optical flow measurement, a logistics object can be stably picked up with low power in a small quantity/multi-product logistics system.

Description

Intelligent smart logistics automation information processing device

The present invention relates to smart logistics automation technology, and more particularly, to a method and apparatus for controlling an operation of a robot arm gripping an object using a camera image.

In the conventional robot automation of industrial production/logistics sites, the smart camera system has a problem in that the camera and information processing (eg, object recognition) are dualized, so it is difficult to install and the price is high. Such a structure is difficult to use in a field requiring a lightweight system.

In addition, additional costs are continuously required for the construction and operation of the server system, and there is a problem in that all systems must be periodically replaced in order to improve performance.

In addition, in order to be applied to a logistics robot, a camera device must be designed to be lightweight, and there is a lack of device design technology that satisfies this.

The present invention has been devised to solve the above problems, and an object of the present invention is to apply a high-complexity deep learning-based object recognition and location estimation technology to a lightweight embedded system, a monocular camera or a depth camera To provide a method for controlling a robot arm that grips an object using

According to an embodiment of the present invention for achieving the above object, a method for controlling a robot arm includes: measuring an optical flow while moving a camera in a distance direction of an object; checking whether the center of the camera image coincides with the center of gravity of the object based on the measured light flow; and if it is determined that they do not match, moving the camera on a plane perpendicular to the distance direction of the object.

And, in the confirmation step, movements at the edges of the object are identified through the optical flow, and if the movements are all the same, the center of the camera image may coincide with the center of gravity of the object.

The robot arm control method according to an embodiment of the present invention, if it is confirmed that the match, the step of controlling the robot arm to move to the center of gravity of the object to grip the object; may further include.

A method for controlling a robot arm according to an embodiment of the present invention, comprising: acquiring an image; detecting an object in the image; It may further include; estimating the distance to the detected object.

In the movement step, the distance per pixel of the object is calculated based on the distance to the object, the number of pixels from the center of the camera image to the center of gravity of the object is calculated, and the distance per pixel is multiplied by the number of pixels to determine the amount of movement of the camera. can

The image may be an RGB image or a depth image. And, in the moving step, the direction of the camera may be rotated.

On the other hand, according to another embodiment of the present invention, a robot arm control device, a camera; and measuring the optical flow while moving the camera in the distance direction of the object, and based on the measured optical flow, check whether the center of the camera image coincides with the center of gravity of the object. It includes; a processor that moves on a vertical plane.

As described above, according to the embodiments of the present invention, through image generation using a lightweight camera and a lightweight deep learning network, and object-oriented tracking through object recognition and optical flow measurement, it is stable with low power in a small quantity/multiple product distribution system Logistics picking becomes possible.

1 is an example of industrial sites and object recognition conditions;

2 is an example of an inverse proportion of network complexity and performance;

3 is a view showing a smart logistics system to which the present invention is applicable;

4 is an external perspective view of a robot arm control system according to an embodiment of the present invention;

5 is a view illustrating cases in which the center of gravity of the object is located at the center of the camera image;

6 is a flowchart provided for the description of a method for controlling a robot arm according to an embodiment of the present invention;

7 is a view showing the movements of the edges of the object identified through the optical flow;

8 is a diagram showing the relationship between the FOV of the camera, the distance to the object and the size of the object;

9 is a view showing the process of matching the camera center to the center of gravity of the object while grasping the movements at the edges of the object through the optical flow;

10 and 11 are views showing a case in which the plane on which the center of gravity of the object is located does not coincide with the camera image plane;

12 is an internal block diagram of the robot arm control system shown in FIG.

Hereinafter, the present invention will be described in more detail with reference to the drawings.

In an embodiment of the present invention, an intelligent smart logistics automation information processing device is provided. Specifically, we present a method to directly control a robot arm without going through a server in a low-spec embedded system using a monocular camera image or a depth camera and a lightweight deep learning network.

In logistics automation in smart factories, marts, convenience stores, etc., 'machine learning and deep learning'-based technologies are required to accurately recognize objects (objects) in various environments and object states to be controlled.

In particular, as shown in FIG. 1, in the case of an industrial site, complexity (background, object diversity, etc.), lighting (illuminance, ceramics, etc.) and product characteristics (size, shape, material, arrangement, etc.) are different depending on the work environment. Therefore, even for the same object, it is very difficult to precisely perform object recognition with a generalized network model because changes in the surrounding environment, object appearance and texture, etc. vary.

In the case of industrial sites, high-precision recognition performance is required for factors such as work efficiency, but in general, network performance and execution speed have a trade-off relationship as shown in FIG. 2 .

This is because, due to the nature of deep learning, a network with high complexity improves the ability to distinguish nonlinear data that is difficult to classify through a large number of activation functions and features. Therefore, the amount of calculation should be considered according to the precision required in the industrial site to be applied.

3 is a diagram illustrating a smart logistics system to which the present invention is applicable. The smart logistics system to which the present invention is applicable is configured to include a robot arm 10 and a robot arm control system 100, as shown.

The robot arm 10 moves the target object to a predetermined position after picking it using a suction device. Picking the object's center of gravity is stable and consumes less power. Therefore, it is necessary to find the center of gravity of the object and move the robot arm 10 to the corresponding position.

4 is an external perspective view of the robot arm control system 100 according to an embodiment of the present invention. The robot arm control system 100 detects an object in an image generated by a camera (monocular camera or depth camera), and controls the robot arm 10 to move to the center of gravity of the detected object.

Since the robot arm 10 is set to move toward the center of the camera image, when the center of gravity of the object is located at the center of the camera image, the robot arm 10 may move to the center of gravity of the object.

5 exemplifies cases in which the center of gravity of the object is located at the center of the camera image. In FIG. 5 , circles and squares are the top surfaces of objects displayed in the camera image, and the + sign indicates the center of the camera image.

Hereinafter, a process of the robot arm control system 100 controlling the robot arm 10 will be described in detail with reference to FIG. 6 . 6 is a flowchart provided for explaining a method for controlling a robot arm according to an embodiment of the present invention.

In order to control the robot arm 10 to move to the center of gravity of the object, first, the robot arm control system 100 acquires an image using a camera (S210), and detects/classifies an object from the acquired image (S220).

The image acquired in step S210 may be an RGB image or a depth image. And, object detection in step S220 may be performed using a network learned to detect/classify objects by receiving an image through a lightweight deep learning network.

Next, the robot arm control system 100 estimates the distance to the object detected in step S220 (S230).

If the image acquired in step S210 is a depth image, distance estimation in step S230 is not a problem. When the image obtained in step S210 is an RGB image, the distance estimation in step S230 may be found by comparing the actual size (pre-stored) of the detected object with the image size.

Thereafter, the optical flow is measured while moving the camera downward along the distance direction of the object detected in step S220 ( S240 ), and based on the measured optical flow, it is confirmed whether the center of the camera image coincides with the center of the object do (S250).

If it is confirmed that they do not match in step S250 (S260-N), the camera is moved on a plane perpendicular to the distance direction of the object (S270).

In step S250, it is possible to check whether the center coincides with the movements at the edges of the object through the optical flow. If all the detected movements are the same, it is determined that the center of the camera image matches the center of gravity of the object. The center of gravity of the object is pre-stored for each object in the robot arm control system 100 .

7 shows a situation in which the edges of the object detected through the optical flow move greatly from the left. This means that the camera center and the object's center of gravity do not match.

In a situation as shown in FIG. 7 , it is required to move the camera in a left direction in which the movement of object edges is large.

When the camera moves in step S270, the distance to the object estimated in step S230 is used. Specifically, the FOV of the camera is as shown in FIG. 8 , and the size of the object is known to the robot arm control system 100 , so the distance per pixel of the object can be calculated based on the distance to the object.

Therefore, after calculating the number of pixels from the center of the camera to the center of the object, the distance per pixel is multiplied by the number of pixels to determine the amount of movement of the camera.

After that, it repeats from step S210.

Accordingly, when it is confirmed that the center of the camera image matches the center of gravity of the object in step S250 (S260-Y), the robot arm 10 is moved to the center of the object and controlled to grip the object (S260).

In step S260, it is possible by a method of lowering the robot arm 10 until the distance to the object estimated in step S230 becomes 0.

9 shows the actual process of matching the camera center to the object's center of gravity while grasping movements at the edges of the object through the optical flow.

Meanwhile, as shown in FIG. 10 , there may be a case where the plane on which the center of gravity of the object is located does not coincide with the camera image plane. To this end, the robot arm control system 100 may hold information about the center of gravity of the object as 3D information, as shown in FIG. 11 .

In this case, in step S270, in addition to moving the camera, a direction change is also performed. At this time, it is necessary to switch the direction of the robot arm 10 in conjunction with the direction of the camera.

12 is an internal block diagram of the robot arm control system 100 shown in FIG. The robot arm control system 100 is configured to include a camera 110 , a processor 120 , a control unit 130 , and a storage unit 140 , as shown in FIG. 12 .

The camera 110 is a means for acquiring an image of an object, and is implemented as a monocular camera or a depth camera. Furthermore, the camera 110 may be implemented with a specification that includes both.

and the processor 120 drives a deep learning network for detecting/classifying an object in the image, estimating the distance to the detected object, and measuring the optical flow so that the camera center matches the center of gravity of the object. ) to control

The controller 130 controls the robot arm 10 to move to the center of the object to grip the object.

The storage unit 140 stores information on the center of gravity of the object, and provides a storage space necessary for the processor 120 to function.

So far, for the intelligent smart logistics automation information processing device, the robot arm control system has been described in detail as an embodiment.

In the above embodiment, through accurate object recognition using depth information, optical flow information, and deep learning, it was possible to control the gripping/picking operation of the robot arm with high precision/high speed only with a small embedded system.

In particular, it made it possible to implement high-complexity deep learning-based object recognition and location estimation in a lightweight embedded system, and presented a structure that can be applied simultaneously when combining a single sensor and multiple sensors.

In addition, it is a lightweight object recognition and location estimation that can be applied to small-volume and multi-variety logistics systems, and it is a model that can be maintained for flexible deep learning devices and new applications. It is also a smart logistics automation device.

On the other hand, it goes without saying that the technical idea of the present invention can also be applied to a computer-readable recording medium containing a computer program for performing the functions of the apparatus and method according to the present embodiment. In addition, the technical ideas according to various embodiments of the present invention may be implemented in the form of computer-readable codes recorded on a computer-readable recording medium. The computer-readable recording medium may be any data storage device readable by the computer and capable of storing data. For example, the computer-readable recording medium may be a ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical disk, hard disk drive, or the like. In addition, the computer-readable code or program stored in the computer-readable recording medium may be transmitted through a network connected between computers.

In addition, although preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above, and the technical field to which the present invention belongs without departing from the gist of the present invention as claimed in the claims In addition, various modifications are possible by those of ordinary skill in the art, and these modifications should not be individually understood from the technical spirit or perspective of the present invention.

Claims

measuring the light flow while moving the camera in the distance direction of the object;

checking whether the center of the camera image coincides with the center of gravity of the object based on the measured light flow;

If it is confirmed that they do not match, moving the camera on a plane perpendicular to the distance direction of the object; Robot arm control method comprising a.
The method according to claim 1,

The verification step is

A method of controlling a robot arm, characterized in that it is determined that the movements at the edges of the object are identified through the optical flow, and that the center of the camera image coincides with the center of gravity of the object if the movements are all the same.
The method according to claim 1,

When it is confirmed that they match, the robot arm moves to the center of gravity of the object and controls to grip the object; the robot arm control method further comprising a.
The method according to claim 1,

acquiring an image;

detecting an object in the image;

Estimating the distance to the detected object; Robot arm control method characterized in that it further comprises.
5. The method according to claim 4,

moving steps,

Calculating the distance per pixel of the object based on the distance to the object, calculating the number of pixels from the center of the camera image to the center of gravity of the object, and determining the movement amount of the camera by multiplying the distance per pixel by the number of pixels How to control a robot arm.
5. The method according to claim 4,

the video,

A method of controlling a robot arm, characterized in that it is an RGB image or a depth image.
The method according to claim 1,

moving steps,

Robot arm control method, characterized in that rotating the direction of the camera.
camera; and

Measure the light flow while moving the camera in the distance direction of the object, and based on the measured light flow, check whether the center of the camera image coincides with the center of gravity of the object. A robot arm control device comprising a; processor for moving on one plane.