KR20230174126A - Method for depalletizing - Google Patents

Abstract

The depalletizing method according to an embodiment of the present invention includes the steps of estimating posture information of boxes, extracting roll container side grid information, and based on the posture information of the boxes and the roll container side grid information. It may include controlling to perform a depalletizing operation on the boxes.

Description

Depalletizing method {METHOD FOR DEPALLETIZING}

The present invention relates to robot depalletizing technology.

Box depalletizing, which uses a robot to pick up boxes stacked in a given space one by one and place them on a conveyor belt, is an essential technology in logistics automation.

In the case of large-sized parcel boxes, most of them are stacked neatly on flat pallets, so it is easy to select the gripper gripping position using normal two-dimensional image analysis methods. On the other hand, in the case of small-sized parcel boxes, parcels are transported irregularly in roll containers or roll pallets made of wire mesh on three sides to prevent the cargo from falling.

A conventional depalletizing device selects a target box based on the location of the parcel box, lifts the selected parcel box, and performs depalletizing.

However, since the conventional depalletizing device does not take into account the position and structure of the roll container, the robot gripper gets caught in the side grid of the roll container while lifting the parcel box, resulting in work interruption or delay. .

The purpose of the present invention is to provide a depalletizing method for stably depalletizing small-sized boxes in a path that does not interfere with the roll container.

Additionally, an object of the present invention is to provide a depalletizing method for depalletizing small-sized boxes in the fastest possible order without interfering with the roll container.

The depalletizing method according to the present invention for achieving the above object includes the steps of estimating posture information of boxes, extracting roll container side grid information, the posture information of the boxes, and the roll container side grid information. It may include controlling to perform a depalletizing operation on the boxes based on .

According to the present invention, a path in which the box or gripper does not collide with the side grid of the roll container can be created even without fixing the roll container in a specific position, thereby significantly reducing the need for operator intervention due to an emergency stop.

In addition, the present invention can automate the depalletizing task because boxes that cannot be picked are expanded in the direction of the side clearance where picking is possible when processing the surrounding boxes first.

In addition, the present invention can utilize not only the upper surface of the roll container but also the front opening as a picking and transfer path, so the processing speed per roll container can be increased by optimally selecting the pick and place path.

Figure 1 is a block diagram showing a depalletizing device according to an embodiment of the present invention.
Figures 2 and 3 are diagrams showing a depalletizing work environment according to an embodiment of the present invention.
Figure 4 is a flowchart for explaining the depalletizing process according to an embodiment of the present invention.
Figure 5 is a diagram showing a point group subject to roll container recognition.
Figure 6 is a diagram showing a point group to be recognized as a roll container after removing the plane.
Figure 7 is a diagram showing the results of iterative RANSAC line fitting and endpoint calculation.
Figure 8 is a diagram showing orthogonal line segments connected to each target line segment.
Figure 9 is a diagram showing how the roll container side grid is recognized using the largest connected element of the undirected graph.
Figure 10 is a diagram showing top bar recognition and side settings for the roll container coordinate system.
Figure 11 is a diagram showing the upper free space defined.
Figure 12 shows the results of estimating the posture of the box.
Figure 13 shows the results of visualizing the free space at the top of box picking.
Figure 14 is a diagram showing the judgment reference point of P _ij .
Figure 15 is a diagram for explaining picking priority.
Figure 16 is a block diagram showing the configuration of a computer system according to an embodiment.

The advantages and features of the present invention and methods for achieving them will become clear by referring to the embodiments described in detail below along with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and will be implemented in various different forms. The present embodiments only serve to ensure that the disclosure of the present invention is complete and that common knowledge in the technical field to which the present invention pertains is not limited. It is provided to fully inform those who have the scope of the invention, and the present invention is only defined by the scope of the claims. Like reference numerals refer to like elements throughout the specification.

Although terms such as “first” or “second” are used to describe various components, these components are not limited by the above terms. The above terms may be used only to distinguish one component from another component. Accordingly, the first component mentioned below may also be the second component within the technical spirit of the present invention.

The terms used in this specification are for describing embodiments and are not intended to limit the invention. As used herein, singular forms also include plural forms, unless specifically stated otherwise in the context. As used in the specification, “comprises” or “comprising” implies that the mentioned component or step does not exclude the presence or addition of one or more other components or steps.

Unless otherwise defined, all terms used in this specification can be interpreted as meanings commonly understood by those skilled in the art to which the present invention pertains. Additionally, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless clearly specifically defined.

In this document, “A or B,” “at least one of A and B,” “at least one of A or B,” “at least one of A, B, or C,” and “at least one of A, B, or C.” Each of the phrases may include any one of the items listed with the corresponding phrase, or any possible combination thereof.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. When describing with reference to the drawings, identical or corresponding components will be assigned the same reference numerals and redundant description thereof will be omitted. .

Figure 1 is a block diagram showing a depalletizing device according to an embodiment of the present invention.

Referring to FIG. 1, the depalletizing device 100 according to an embodiment of the present invention may include a camera unit 110, a control unit 130, and a box transfer unit 150.

The camera unit 110 can photograph the roll container and the boxes loaded on the roll container. The camera unit 110 may be a 3D camera.

The control unit 130 may obtain point cloud data from the image obtained from the camera unit 110 and detect the side grid of the roll container using the point cloud data.

The control unit 130 can measure the upper free space up to the roll container side grid boundary with respect to the rectangular planes of the box, and determine the picking order of the boxes based on the free space. A vision system may be used as this control unit 130, but its type is not limited.

The box transfer unit 150 may perform the task of moving boxes according to the picking order determined by the control unit. The box transfer unit 150 may include a box transfer robot.

Figures 2 and 3 are diagrams showing a depalletizing work environment according to an embodiment of the present invention.

As shown in FIG. 2, the camera unit 110 is placed on the upper part of the box 10 to photograph and recognize the bottom of the roll container 30, the roll container 30, and the box 10 loaded on the roll container. You can.

The box transfer unit 150 may include a robot. The robot may be composed of a gripper and joints, but its configuration is not limited. A rectangular plane is attached to the gripper, and the size, midpoint, and direction vector of the gripper rectangular plane can be obtained. Here, the direction vector of the rectangular plane may include a normal vector and two direction vectors parallel to the long side and the short side.

As shown in FIG. 3, when an operator requests work after opening the front door of the roll container 30, the box transport unit 150 sequentially conveyors the boxes 10 based on the control signal from the control unit 130. Depalletizing can be done on belt 50.

Below, we will look at the depalletizing process in more detail.

Figure 4 is a flowchart for explaining the depalletizing process according to an embodiment of the present invention. Here, the depalletizing process may be performed in a depalletizing device.

Referring to FIG. 4, when a depalletizing request is received from an operator, the box transfer unit 150 may prepare a gripper plane (S200). The box transfer unit 150 may provide a request message for requesting plane posture information to the control unit (S210).

The control unit 130 obtains an image of the floor without a roll container from the camera unit (S100) and performs plane fitting to detect the floor plane and a normal vector perpendicular to the floor plane (S100). S110).

The control unit 130 may obtain the size, midpoint, and direction vector of the gripper rectangular plane from the rectangular plane attached to the robot gripper (S120). The direction vector of a rectangular plane can be composed of a normal vector and two direction vectors parallel to the long side and the short side. Additionally, if necessary, the positive direction of each direction vector can be additionally identified by changing the angle of the robot joint.

The box transfer unit 150 can wait for the roll pallet to be introduced (S220) and prepare for the waiting posture (S230). The box transfer unit 150 may transmit a message for requesting selection of a work target box. (S240)

The control unit 130 can process the point cloud data obtained from the camera unit to calculate and detect the midpoint, size, and direction vector of the rectangular planes of the box in the roll container (S130).

The control unit 130 selects the point group remaining from the point group from the detected box rectangular planes to the camera, excluding the detected box rectangular plane point group, as the roll container side grid detection target point group.

By performing a straight line fit on this point group, the rolltainer side grid can be detected and the region of interest (ROI) boundary can be set (S140).

The control unit 130 may determine the upper free space up to the roll container side grid boundary with respect to the box rectangular planes. The control unit 130 may group the upper free space of the box rectangular planes into each box and use it as the upper and side free space of each box.

The control unit 130 determines the box parking order using the free space on the top and side of the box and provides the pick work free space information of the highest priority rectangular plane information and the remaining box quantity to the box transfer unit (S150) ).

The box transfer unit 150 can convert the work target rectangular plane information and pick work free space information into the robot coordinate system by referring to the gripper plane coordinates.

The box transfer unit 150 can plan a path for the work object (S250) and perform pick & place operations (S260, S270).

Below, we look at the rolltainer side grid recognition method for ROI selection.

FIG. 5 is a diagram showing a point cloud to be recognized as a roll container, and FIG. 6 is a diagram showing a point cloud to be recognized as a roll container after removing the plane.

Referring to FIG. 5, in the case of a roll container containing a box, point groups that fit the RANSAC plane with a normal box plane size can also be repeatedly removed.

As shown in FIG. 6, in the case of an empty roll container, the point cloud on the roll container floor and the workshop floor can be cropped based on the distance from the camera so that only the point cloud on the side grid remains.

For the remaining point clouds, RANSAC line fitting, Euclidean clustering, and Moment of Inertia estimation of PCL (PointCloud Library) are sequentially and repeatedly applied to obtain the center point, direction vector, and length information of line segments of a certain length (e.g., 0.5 m) or more. You can get it.

Figure 7 is a diagram showing the results of iterative RANSAC line fitting and endpoint calculation.

As shown in FIG. 7, the line segments detected as a result of RANSAC line fitting may belong to various objects other than the roll container, but the line segments L that make up the roll container side grid have the characteristic of being parallel or perpendicular to each other.

If two mutually perpendicular lines belong to one side, they meet at one point, and if two mutually perpendicular lines belong to different sides, there is a distance equal to the width of the hinge. Line segments belonging to the same side but parallel to each other may be associated with a third line segment perpendicular to them.

Therefore, by calculating the angle between all detected line segments (L) and the distance between each line segment, it is possible to find line segments that are perpendicular to each other and within a predetermined distance, for example, 15 cm, and connect them logically.

This process is similar to creating a level 2 tree with each line segment as the root node and the connected orthogonal line segments as descendant nodes.

Figure 8 is a diagram showing orthogonal line segments connected to each target line segment, and Figure 9 is a diagram showing the roll container side grid recognized using the largest connected element of the undirected graph.

As shown in FIG. 8, in order to perform clustering that logically connects all interconnected line segments, an undirected graph such as FIG. 9a is created with the index of the line segment (L) detected as a tree as a vertex. You can make it.

As shown in Figure 9, a graph adjacency list is extracted from the graph adjacency matrix, the connected component with the most orthogonal line segments connected to each other is determined to be the side grid of the rolltainer, and the connected component with relatively few connected line segments is determined as the side grid of the rolltainer. The elements can be considered noise components belonging to objects other than the roll container.

Figure 10 is a diagram showing top bar recognition and side settings for the roll container coordinate system.

As shown in Figure 10a, the midpoint position vector of the line segment (C _i ) in the order of proximity to the origin (O) of the camera unit. and the direction vector of the line segment (C _i ) Evaluate whether the angle between is within a predetermined standard and use it as the top bar of the roll container. As shown in Figure 10b, find a pair of top bars that are perpendicular to each other among the selected top bars and determine the height of the roll container in the vector cross product direction between the two direction vectors. You can set as many ROI boundary planes as you want.

Here, the horizontal bar can be measured by Equation 1.

[Equation 1]

Additionally, the vertical bar can be measured by Equation 2.

[Equation 2]

Meanwhile, the method for determining the picking order is as follows.

Figure 11 is a diagram showing the upper free space defined, and Figure 12 shows the result of estimating the posture of the box.

Figure 11a shows the box rectangular plane, Figure 11b shows the grid interference area on the side of the roll container, and Figure 11c shows the free space at the top.

In order to determine the upper free space with respect to the rectangular plane of the box, the rectangular planes of the box can be divided to determine each posture, such as the normal vector of the rectangular plane, the midpoint, and the length of each side.

For example, point cloud data of the work object is obtained from an image captured using a 3D camera, and a rectangular plane area is divided from the point cloud data to obtain posture information of the box 10, as shown in FIG. 12. can be estimated. In addition, the obtained point cloud data can be processed to calculate the midpoint, size, and direction vectors of the box rectangular planes in the roll container.

In the case of boxes randomly stacked in a roll container, a square plane must be fixed with a gripper to lift the box and be lifted in the normal direction of the square plane to remove it from between adjacent boxes without applying force to adjacent boxes. If the square plane is inclined with respect to the ground, the parcel box or gripper may collide with the side grid of the roll container while lifting in the normal direction of the square plane. The transportable distance before collision is defined as the upper clearance space, The top clearance must be determined for each box rectangular plane.

For this purpose, there is a space between the grids on the side of the roll container, but it may not be captured by the top bar, so a point cloud that fills all of the previously recognized ROI boundary planes can be added to the original point cloud.

In the coordinate system set at the center of the rectangle recognized as the box surface, the minimum value of the z-coordinate of the point cloud remaining after cutting out everything outside the rectangular area means the free picking space in that direction. If this value is smaller than the sum of the normal box height and gripper height, it can be determined that picking in that direction is not possible.

Figure 13 shows the results of visualizing the free space at the top of box picking.

As shown in Figure 13, the free space above box picking can be made up of two cases. The first box picking upper free space (Z1) may include instances where it does not meet the roll container side grid. The second box picking upper clearance (Z2) may contain instances of collision with the roll container side grid.

Next, you can determine the side clearance. A maximum of three sides of one rectangular box can be photographed at the same time. If the top surface of the box to be picked by the gripper has little free space to the side grid of the roll container, the box can be adsorbed and fixed and then transferred in the direction of the free space on the remaining two sides. .

For this purpose, by using the midpoint P _si and the rotation matrix R _si representing the coordinate system of each of the rectangles detected from the point cloud, the jth midpoint P _ij and the relative coordinate system rotation matrix R _ij for the ith coordinate system can be obtained as shown in Equation 3. .

[Equation 3]

If one of the Euler angles of the rotation matrix R _ij has a component close to 90 degrees, the two rectangles are perpendicular and may belong to one box.

Figure 14 is a diagram showing the judgment reference point of P _ij .

As shown in FIG. 14, the distance between point P _ij and the next point can be calculated, and if it is less than a predetermined distance, it can be determined that it belongs to one parcel box.

If these two evaluation criteria are satisfied, the top free space of the jth rectangle is judged as the side free space of the ith rectangle. If there are no more boxes with sufficient top free space, boxes with long side free space are picked first. do.

If the picking operation is performed using the side free space as described above, the side of the roll container can be used as collision information in the path planning algorithm. Even if a box only has insufficient upper space, if adjacent boxes are processed and side space is secured, the picking work can be completed by moving to the side immediately after gripping by the gripper.

In addition, when the side of the roll container is opened, not only the upward direction but also the front opening direction of the roll container can be automatically used as a picking and transport path.

Figure 15 is a diagram for explaining picking priority.

As shown in Figure 15, the possible picking directions include an upper free space and a side free space, and the picking order can be determined in the order of a rectangle with a top free space as shown in Figure 15a and a rectangle with a side free space as shown in Figure 15b. there is.

The depalletizing device according to the embodiment may be implemented in a computer system such as a computer-readable recording medium.

Figure 16 is a block diagram showing the configuration of a computer system according to an embodiment.

Referring to FIG. 16, the computer system 1000 according to the embodiment includes one or more processors 1010, a memory 1030, a user interface input device 1040, and a user interface output device ( 1050) and storage 1060. Additionally, the computer system 1000 may further include a network interface 1070 connected to a network.

The processor 1010 may be a central processing unit or a semiconductor device that executes programs or processing instructions stored in memory or storage. The processor 1010 is a type of central processing unit and can control the overall operation of the depalletizing device.

The processor 1010 may include any type of device capable of processing data. Here, 'processor' may mean, for example, a data processing device built into hardware that has a physically structured circuit to perform a function expressed by code or instructions included in a program. Examples of data processing devices built into hardware include a microprocessor, central processing unit (CPU), processor core, multiprocessor, and application-specific integrated (ASIC). circuit) and FPGA (field programmable gate array), but are not limited thereto.

The memory 1030 may store various data for overall operation, such as a control program for performing a depalletizing method according to an embodiment. Specifically, a number of application programs running on the depalletizing device, data and commands for operating the depalletizing device may be stored in the memory.

The memory 1030 and storage 1060 may be storage media including at least one of volatile media, non-volatile media, removable media, non-removable media, communication media, and information transfer media. For example, memory 1030 may include ROM 1031 or RAM 1032.

According to one embodiment, a computer-readable recording medium storing a computer program, comprising: estimating posture information of boxes, extracting grid information on the side of the roll container, posture information on the boxes, and the side of the roll container. It may include instructions for causing a processor to perform a method including controlling to perform a depalletizing operation on the boxes based on grid information.

According to one embodiment, it is a computer program stored in a computer-readable recording medium, which includes an operation of estimating posture information of boxes, an operation of extracting roll container side grid information, and the posture information of the boxes and the roll container side grid. It may include instructions for causing the processor to perform a control operation to perform a depalletizing operation on the boxes based on the information.

The specific implementations described in the present invention are examples and are not intended to limit the scope of the present invention in any way. For the sake of brevity of the specification, descriptions of conventional electronic components, control systems, software, and other functional aspects of the systems may be omitted. In addition, the connections or connection members of lines between components shown in the drawings exemplify functional connections and/or physical or circuit connections, and in actual devices, various functional connections or physical connections may be replaced or added. Can be represented as connections, or circuit connections. Additionally, if there is no specific mention such as “essential,” “important,” etc., it may not be a necessary component for the application of the present invention.

Therefore, the spirit of the present invention should not be limited to the above-described embodiments, and the scope of the patent claims described below as well as all scopes equivalent to or equivalently changed from the scope of the claims are within the scope of the spirit of the present invention. It will be said to belong to

10: box
30: Rolltainer
110: Camera unit
130: control unit
150: Box transfer unit

Claims (1)

estimating posture information of boxes;
Extracting roll container side grid information; and
Controlling to perform a depalletizing operation on the boxes based on the attitude information of the boxes and the roll container side grid information;
A depalletizing method comprising:

Images