KR20210138359A - Apparatus and method for controlling object loading - Google Patents

Abstract

The present invention relates to an apparatus and a method for controlling object loading. More specifically, an apparatus for controlling object loading, in accordance with an embodiment of the present invention, comprises: an object information acquiring unit for acquiring a projected image and a first height value of an object to be loaded based on a first image and first data output from a first image sensor; a weight map setting unit for acquiring second height values for a space sensed by a second image sensor based on second data output from the second image sensor, and setting a weight map for a second image output from the second image sensor based on the second height values; a loading position determining unit for determining a loading position of the object to be loaded based on contact point information about a point where a robot arm and the object to be loaded are in contact with each other, the first height value, the projected image, and the weight map; and a control unit for controlling the robot arm for the robot arm to load the object to be loaded into the loading position, when the loading position is determined. The present invention can prevent malfunction by accurately determining the loading position.

Description

LOAD CONTROL DEVICES AND METHOD FOR CONTROLLING OBJECT LOADING

The present invention relates to a load control device and a load control method.

As Internet commerce is rapidly becoming common, the amount of transport carrying goods is rapidly increasing. When there is information on the order and size of cargo in a cargo transport system such as a logistics transport system, a courier system, and a manufacturing process system, the system loads the cargo in advance using a robot arm, a control device, etc. You can decide whether to do it or not. The system may also perform an automated process of loading cargo into loading boxes in the order they arrive.

However, if the size of each cargo is different and there is no information on the size, it may be difficult to determine the loading position for loading the cargo in advance if there is no information on the order in which the cargo is received.

In addition, when the sizes of the cargos are different or the shapes in which the cargos are placed are different, the picking positions at which the robot arm picks the cargos may not be constant due to errors of sensors and grippers. In this case, even if the loading location is determined in advance, the cargo may not be accurately loaded at the location.

Therefore, even when information on the size of an object is not provided, information on the space that can be loaded in real time, information on where the gripper picked the cargo, height of the picked product, projected area, etc. It is necessary to determine the loading location based on the

The problem to be solved by the present invention is to maximally load the cargo in the loading box by securing the maximum possible loading space, even if the cargos have different sizes or information on the sizes is not provided in advance.

In addition, another problem to be solved by the present invention is to prevent malfunction by accurately determining a loading position even if cargos have different sizes or information about the sizes is not provided in advance.

In addition, another problem to be solved by the present invention is to minimize loading costs by maximizing loading efficiency, even if cargos have different sizes or information about the sizes is not provided in advance.

The problems of the present invention are not limited to the problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following description.

In order to solve the above problems, in one aspect, the loading control device according to an embodiment of the present invention, a projection image of the loading target object and the second based on the first image and the first data output from the first image sensor The object information acquisition unit for acquiring the first height value, acquires second height values for a space sensed by the second image sensor based on second data output from the second image sensor, and based on the second height values A weight map setting unit for setting a weight map for the second image output from the second image sensor, contact point information about a point where the robot arm and the load target contact each other, a first height value, a projection image, and a weight map and a loading position determining unit that determines a loading position of the loading target based on the loading position determining unit, and a control unit that controls the robot arm so that the robot arm loads the loading target to the loading position when the loading position is determined.

In another aspect, the loading control method according to an embodiment of the present invention includes the steps of: acquiring a projection image and a first height value of a loading target object based on a first image and first data output from a first image sensor; Based on the second data output from the second image sensor, second height values for the space sensed by the second image sensor are obtained, and based on the second height values, the second height values are applied to the second image output from the second image sensor. setting a weight map for the loading target, determining a loading position of the loading target based on contact point information about a point where the robot arm and the loading target contact each other, a first height value, a projection image, and a weight map, and and controlling the robot arm to load the object to be loaded into the loading position when the loading position is determined.

The details of other embodiments are included in the detailed description and drawings.

As described above, in the embodiments of the present invention, even if the cargos have different sizes or information on the sizes is not provided in advance, the cargo can be maximally loaded into the loading box by maximizing the loadable space.

In addition, embodiments of the present invention can prevent malfunctions by accurately determining the loading position even if cargos have different sizes or information on the sizes is not provided in advance.

In addition, embodiments of the present invention can minimize loading costs by maximizing loading efficiency, even if cargos have different sizes or information about the sizes is not provided in advance.

Effects according to the embodiments are not limited by the contents exemplified above, and more various effects are included in the present specification.

1 to 3 are views for explaining an object loading system according to an embodiment of the present invention.
4 is a block diagram illustrating a loading control apparatus according to an embodiment of the present invention.
5 is a view for explaining an embodiment of measuring the first height value of the object to be loaded.
FIG. 6 is a diagram exemplarily illustrating a first image generated by the first image sensor illustrated in FIG. 5 .
7 to 10 are views for explaining an embodiment of measuring the second height value of the loading box.
11 is a graph illustrating second height values of regions corresponding to the first portion shown in FIG. 8 .
FIG. 12 is a graph illustrating second height values of regions corresponding to the second portion shown in FIG. 8 .
13 is a diagram exemplarily illustrating a plurality of sub-images included in an image-processed second image.
14 is a diagram illustrating a weight map according to an embodiment of the present invention.
15 is a diagram for explaining an embodiment in which a weight is set for each of a plurality of groups.
16 is a view for explaining an embodiment of determining a first loading position for a loading target object.
17 is a view exemplarily showing a plurality of sub-images included in the image-processed second image when the loading target object is loaded.
18 is a view for explaining an embodiment of determining a second loading position with respect to another loading target when the loading target is loaded.
19 is a view exemplarily showing a plurality of sub-images included in the image-processed second image when loading target objects are loaded.
20 to 22 are views for explaining an embodiment of determining the third loading position on the loading target object loaded in the loading box.
23 is a flowchart illustrating a loading control method according to an embodiment of the present invention.

The objects, specific advantages and novel features of the present invention will become more apparent from the following detailed description and preferred embodiments taken in conjunction with the accompanying drawings. In the present specification, in adding reference numbers to the components of each drawing, it should be noted that only the same components are given the same number as possible even though they are indicated on different drawings. In addition, in describing the present invention, if it is determined that a detailed description of a related known technology may unnecessarily obscure the subject matter of the present invention, the detailed description thereof will be omitted.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 to 3 are views for explaining an object loading system according to an embodiment of the present invention.

1 to 3 , the object loading system 10 according to an embodiment of the present invention selects a loading target object OB from among the objects included in a first loading box LB1 and , it may be a system that performs an operation of loading the selected loading target object OB into the second loading box (LB2).

The object loading system 10 may include a loading control device 100 , a robot arm 200 , a first image sensor 300 , and a second image sensor 400 .

The loading control device 100 may control the robot arm 200 , the first image sensor 300 , and the second image sensor 400 .

In an embodiment, the loading control apparatus 100 may control the operation of the robot arm 200 . Here, the operation of the robot arm 200 is a picking operation in which the robot arm 200 picks the load target object OB using a gripper GR, and the robot arm 200 uses the gripper GR. A transport operation of moving the load object (OB) while holding the load object (OB) using the It may include a loading operation to

1 and 2, for example, the loading control device 100, the loading target object OB located inside the first loading box LB1 is located in the object recognition area, the robot arm 200 It is possible to output a first control signal for controlling the picking operation and the transport operation of the . In this case, the robot arm 200 receives the first control signal, picks the loading target object OB located inside the first loading box LB1, and transfers the loading target object OB to the first loading box ( LB1) can be moved to the object recognition area.

Here, the object recognition area may mean an area for measuring the size of the height and volume of the loading target object OB, and a projection shape corresponding to the cross-sectional area. The object recognition area may correspond to a sensing range (or a photographing range) of the first image sensor 300 . For example, as shown in FIGS. 1 to 3 , when the first image sensor 300 is positioned in the xy plane formed by the x-axis and the y-axis and photographed in the z-axis direction, the object recognition area is the second One of the position coordinates P2 of the image sensor 300 may be the same as the x-coordinate and the y-coordinate, and only the z-coordinate among the position coordinates P2 of the first image sensor 300 may correspond to different position coordinates. . However, the present invention is not limited thereto.

On the other hand, with reference to FIGS. 2 and 3 , for example, the loading control device 100 may include the robot arm 200 so that the loading target object OB located in the object recognition area is loaded in the second loading box LB2. ) may output a second control signal for controlling the transport operation and the loading operation. In this case, the robot arm 200 receives the second control signal, moves the loading target OB from the object recognition area to the inside of the second loading box LB2, and moves the loading target OB to the second It can be loaded inside the loading box LB2.

Meanwhile, according to an embodiment, the loading control apparatus 100 may control the sensing operation of the first image sensor 300 . Here, the sensing operation of the first image sensor 300 includes an operation in which the first image sensor 300 generates a first image (or image data) by photographing within a predetermined photographing range, and the first image sensor 300 . The operation of generating first data by measuring the height, cross-sectional area, etc. of the provisional loading target object OB, and the first image sensor 300 transmits the first image and an electrical signal corresponding to the first data to the loading control device 100 ) can be included in the output operation.

Referring to FIGS. 1 and 2 , for example, the loading control device 100 includes a third control for controlling the sensing operation of the first image sensor 300 when the loading target object OB is located in the object recognition area. signal can be output. In this case, the first image sensor 300 receives the third control signal, captures the image in one direction (eg, the z-axis direction) to generate a first image and first data, and generates the first image and the first data. An electrical signal corresponding to the data may be output to the loading control device 100 .

Here, the first image may include a projection image of the load target object OB, an image of another object existing around the load target object OB, and the like, and the first data includes the load target object OB and the second image. 1 may include a distance value between the image sensors 300 , a cross-sectional area value of the load target object OB, and the like. A detailed description thereof will be described later with reference to FIGS. 5 and 6 .

Meanwhile, according to an embodiment, the loading control apparatus 100 may control the sensing operation of the second image sensor 400 . Here, the sensing operation of the second image sensor 400 includes an operation in which the second image sensor 400 generates a second image (or image data) by photographing within a predetermined photographing range, and the second image sensor 400 . An operation of generating second data by measuring a distance between the second loading box LB2 or the loaded loading target object OB and the second image sensor 400, and the second image sensor 400 using the second image and outputting an electrical signal corresponding to the second data to the loading control device 100 .

2 and 3 , the loading control apparatus 100 may output a fourth control signal. The second image sensor 400 receives the fourth control signal and generates a second image and second data by photographing it in one direction (eg, the same direction as the zx plane formed by the x-axis and the z-axis). can do. The second image sensor 400 may output an electrical signal corresponding to the second image and the second data to the loading control device 100 .

Here, the second image may include an image for the empty second loading box LB2 (see FIG. 2 ), and an image for the second loading box LB2 in which the loading target object OB is loaded. may be included (see FIG. 3 ), and may include an image for each of the second loading box LB2 and other objects positioned around the second loading box LB2 . And, the second data is a distance value between the second loading box LB2 and the second image sensor 400 (see FIG. 10 to be described later) and/or the loading target object OB loaded in the second loading box LB2. and a distance value between the second image sensor 400 and the second image sensor 400 (see FIG. 21 to be described later).

The robot arm 200 may be driven by receiving a control signal output from the loading control device 100 . Specifically, the robot arm 200 may pick the load target object OB, move the load target object OB to a required position, and load the load target object OB to a required loading position. The robot arm 200 may include a gripper GR. Here, the gripper GR may hold the load target object OB or release the holding target object OB. The shape of the gripper GR may be a vacuum suction gripper capable of adsorption as shown in FIGS. 1 to 3 . Alternatively, in various embodiments, the gripper GR may be in the form of tongs for holding and lifting the load object OB. However, the shape of the gripper GR is not limited to the above-described example, and any shape is possible as long as it can pick, hold, and load the load target object OB.

In an embodiment, the robot arm 200 may transmit information about the position coordinates of the gripper GR and the first yaw rotation angle to the loading control apparatus 100 . Here, the position coordinates of the gripper GR may mean the position coordinates of the gripper GR when the loading target object OB is located in the object recognition area. In addition, the first yaw rotation angle may mean a yaw rotation angle of the gripper GR when the loading target object OB is positioned in the object recognition area.

The first image sensor 300 may be, for example, a 3D camera. However, the present invention is not limited thereto. When the first image sensor 300 is a 3D camera, the first image sensor 300 generates a first image by photographing a predetermined area, and creates a second image between the first image sensor 300 and the loading target object OB. The first distance value may be calculated and first data including the first distance value may be generated. Here, the first distance value may be calculated by a method using structured light, a method using time of flight (ToF), and a method using depth from defocus (DFD) using defocusing. . However, the present invention is not limited thereto. Hereinafter, for convenience of description, it is assumed that the first image sensor 300 is a 3D camera.

The first image sensor 300 may be positioned below the gripper GR of the robot arm 200 as shown in FIG. 1 , but is not limited thereto. Here, the position coordinates at which the first image sensor 300 is disposed may be stored in advance in a memory (not shown) of the first image sensor 300 or stored in advance in a memory unit (not shown) of the loading control device 100 . have. When the position coordinates P2 of the first image sensor 300 are previously stored in the memory of the first image sensor 300 , the first data may further include the position coordinates P2 of the first image sensor 300 . have. A detailed description thereof will be described later with reference to FIGS. 5 and 6 .

The second image sensor 400 may be the same image sensor as the first image sensor 300 . However, the present invention is not limited thereto. Hereinafter, for convenience of description, it is assumed that the second image sensor 400 is a 3D camera in the same manner as the first image sensor 300 . On the other hand, when the second image sensor 400 is a 3D camera, the second image sensor 400 calculates a second distance value between the second image sensor 400 and the second loading box LB2 to receive the second data. can create

The first loading box (LB1) and the second loading box (LB2) may have a box shape with an open top surface as shown in FIGS. 1 to 3 and defined by a side surface and a bottom surface, but is not limited thereto. It may be in the form of a palette having a bottom of the thickness of .

The first support member SM1 may support or support the first loading box LB1.

The second support member SM2 may support or support the second loading box LB2 similarly to the first support member SM1 . The second support member SM2 may include a support pole SP. The support pole SP may extend in one direction (eg, the x-axis direction) and may extend in a direction different from the one direction (eg, the z-axis direction), but is not limited thereto. In addition, the second image sensor 400 may be disposed at one end of the support pole SP at a predetermined angle.

The first support member SM1 and the second support member SM2 may be fixed members as shown in FIGS. 1 to 3 , but are not limited thereto, and may be a conveyor belt. That is, the first loading box LB1 and the second loading box LB2 may be located on the conveyor belt.

Although not shown, the loading control device 100, the robot arm 200, the first image sensor 300, and the second image sensor 400 included in the object loading system 10 communicate by wire or wirelessly. It can transmit or receive electrical signals.

Although not shown, the first loading box (LB1) may also be provided with a support pole (SP) similar to the support pole (SP) provided in the second loading box (LB2), provided in the first loading box (LB1) The support pole SP may support a third image sensor capable of sensing the first loading box LB1.

Hereinafter, the loading control apparatus 100 according to an embodiment of the present invention will be described in detail.

4 is a block diagram illustrating a loading control apparatus according to an embodiment of the present invention.

Referring to FIG. 4 , the loading control apparatus according to an embodiment of the present invention includes an object information acquisition unit 110 , a weight map setting unit 120 , a loading position determining unit 130 , and a control unit 140 , etc. can do.

The object information acquisition unit 110 receives an electrical signal corresponding to the first image and the first data from the first image sensor 300 , and determines the load target object OB based on the first image and the first data. A first height value and a projection image may be acquired.

Specifically, when the robot arm 200 positions the loading target object OB in the object recognition area, the first image sensor 300 transmits an electrical signal corresponding to the first image and the first data to the object information acquisition unit 110 . ) can be sent to The object information obtaining unit 110 may calculate a first height value of the loading target object OB based on the first data and obtain a projection image based on the first image. A method of acquiring the first height value and the projection image of the load target object OB will be described later with reference to FIGS. 5 and 6 .

The object information acquisition unit 110 receives an electrical signal corresponding to the position coordinates of the robot arm 200 and the first yaw rotation angle, and applies the coordinate image for the position coordinates of the robot arm 200 to the above-described projection image. can be displayed A detailed description thereof will be described later with reference to FIG. 6 .

The object information acquisition unit 110 may also include a frame grabber, software, and the like that converts the image output from the first image sensor 300 into image data.

The weight map setting unit 120 obtains second height values for a space sensed by the second image sensor 400 based on the second data output from the second image sensor 400 , and sets the second height values. Based on , a weight map for the second image output from the second image sensor 400 may be set.

The weight map may mean data weighted for each of a plurality of regions included in the image. The weight may be an integer. However, the present invention is not limited thereto. A method of setting the weight map will be described later with reference to FIGS. 14 to 16 .

In an embodiment, the weight map setting unit 120 obtains second height values for each of a plurality of regions (not shown) in the second image based on the second data, and among the second height values A weight map may be set by assigning a weight to an area having a second height value equal to or less than the reference value.

In an embodiment, the weight map setting unit 120 receives an electrical signal corresponding to the second image and the second data from the second image sensor 400 , and loads the second based on the second image and the second data. A loadable space image corresponding to the loading space of the box LB2 may be set. Here, setting as the loadable space image may mean image processing for extracting an image corresponding to the second loading box LB2 from the second image. According to this, there is an advantage of preventing a malfunction.

Here, the loading space is, for example, when the 2nd loading box LB2 is empty, the space inside the 2nd loading box LB2, the loading target object OB is loaded in the inside of the 2nd loading box LB2. In this case, it may mean the remaining space except for the size of the loading target object OB among the internal spaces of the second loading box LB2. A detailed description thereof will be described later with reference to FIGS. 10 to 13 , 17 , 19 , and 20 .

In an embodiment, the weight map setting unit 120 may be configured to correspond to an object other than the space and/or the second loading box LB2 that cannot be loaded in the second loading box LB2 based on the second image and the second data. Unloadable space image can be set.

Here, the non-loadable space may mean a space other than the loading space, for example, the side of the second loading box LB2, the size of the loading target object OB loaded inside the second loading box LB2. It may include a space corresponding to , an external space of the second loading box LB2, and the like. A detailed description thereof will be described later with reference to FIGS. 10 to 13 , 17 , 19 , and 20 .

In an embodiment, the weight map setting unit 120 obtains second height values for each of a plurality of regions in the second image based on the second data, and a second height that is less than or equal to a reference value among the second height values. A first weight may be given to an area having a value, and a second weight may be given to an area having a second height value greater than a reference value among the second height values.

Here, the first weight and the second weight may be different from each other. However, the present invention is not limited thereto.

The weight map setting unit 120 may also include a frame grabber that converts the image signal output from the second image sensor 400 into image data, software, and the like.

The loading position determining unit 130 may receive or store in advance the position coordinates and the first yaw rotation angle of the gripper GR of the robot arm 200 , and the first height value and the projection image from the object information obtaining unit 110 . It is possible to receive an electrical signal corresponding to , and an electrical signal for the loadable space image (and the non-loadable space image) from the weight map setting unit 120 .

The loading position determining unit 130 determines the loading target object OB based on the contact point information about the point where the robot arm 200 and the loading target object OB come into contact with each other, the first height value, the projection image, and the weight map. can determine the location of the

Here, the contact point information may include, for example, the position coordinates of the gripper GR of the robot arm 200 and/or the first yaw rotation angle.

In one embodiment, the loading position determining unit 130 is a loading target object based on the position coordinates and the first yaw rotation angle of the gripper GR of the robot arm 200 , the first height value and the projected image, and the weight map. It is possible to determine the loading position of (OB).

In one embodiment, the loading position determining unit 130 arranges a projection image in which a coordinate image corresponding to a point where the robot arm 200 and the loading target object OB come into contact with each other is displayed on an image in which a weight map is set, Weights given to each of the regions overlapping at least partially with the projected image among the regions may be extracted from the weight map, the extracted weights may be calculated to calculate the placement value of the projected image, and the loading position may be determined based on the placement value. .

In an embodiment, the loading position determining unit 130 may determine a position of the projection image such that the placement value satisfies a predetermined loading condition as the loading position. The loading condition may be, for example, a case in which the batch value belongs to a range of weights or has a maximum or minimum value.

In one embodiment, the loading position determining unit 130 arranges a projection image in which a coordinate image for a position coordinate of the robot arm 200 is displayed in a sub-image, so that each of the regions overlapping at least partially with the projected image among the regions Weights assigned to . may be extracted from the weight map, and a loading position may be determined based on a projection image that satisfies a loading condition in which the sum of the weights is the minimum.

In one embodiment, the loading position is information about the position coordinates and the second yaw rotation angle of the gripper GR of the robot arm 200 required for the loading target object OB to be loaded in the second loading box LB2. may include A method of determining the loading position will be described later with reference to FIGS. 17 to 19 .

When the loading position is determined, the controller 140 may control the robot arm 200 to load the loading target object OB to the loading position.

In this case, the robot arm 200 horizontally moves the load target object OB located in the object recognition area (eg, the load target object OB moves parallel to the xy plane shown in FIGS. 1 to 3 ) ), vertical movement (for example, the load object OB is moved in the z-axis direction shown in FIGS. 1 to 3 ) and yaw rotation (for example, the load object OB is moved in the 1 to 3 ) may be rotated based on the z-axis, and the loading target object OB may be loaded at the loading position by the robot arm 200 . In an embodiment, the robot arm 200 may move in a combined form of at least two of horizontal movement, vertical movement, or yaw rotation.

Although not shown, the load control apparatus according to an embodiment of the present invention may further include a memory unit including a volatile memory and a non-volatile memory. The memory unit may store a first height value, a second height value, a weight map, a first distance value, a second distance value, a first reference distance value, a second reference distance value 10 , a reference value, and the like, which will be described later.

Hereinafter, an embodiment of calculating the first height value of the loading target object OB will be described in detail.

FIG. 5 is a view for explaining an embodiment of measuring a first height value of an object to be loaded, and FIG. 6 is a view exemplarily showing a first image generated by the first image sensor shown in FIG. 5 .

Referring to FIG. 5 , when the robot arm 200 (specifically, the gripper GR of the robot arm 200 ) positions the loading target object OB in the object recognition area, the loading target object OB is the robot It may be in a stationary state while being rotated by the first yaw rotation angle θ1 of the gripper GR of the arm 200 . Here, the object recognition region may mean a region in which the position coordinate P1 of the gripper GR is on a vertical line (eg, a straight line in the z-axis direction) of the first image sensor 300 . That is, the fact that the loading target object OB is located in the object recognition area means that the x coordinate and y coordinate of the gripper GR are the same as the x coordinate and y coordinate of the first image sensor 300 , and the z coordinate of the gripper GR is the same as that of the first image sensor 300 . The coordinates and the z coordinates of the first image sensor 300 may mean different things.

Meanwhile, since the first image sensor 300 generates the first data including the first distance value d1 , the object information acquisition unit 110 , the load target object OB and the first image from the first data The first distance value d1 between the sensors 300 may be extracted.

When the first distance value d1 is extracted, the object information obtaining unit 110 calculates the first height value h1 based on the preset first reference distance value d0 and the first distance value d1. can

In an embodiment, the first reference distance value d0 may be calculated based on the position coordinate P2 of the first image sensor 300 and the position coordinate P1 of the gripper GR. Here, the position coordinate P2 of the first image sensor 300 may be previously stored in the first image sensor 300 and/or the loading control device, and the position coordinate P1 of the gripper GR is the robot arm 200 and/or the load control device.

For example, the x and y coordinates of the gripper GR are the same as the x and y coordinates of the first image sensor 300 , and the z coordinate of the gripper GR and the z coordinate of the first image sensor 300 are In different cases, the first reference distance value d0 may be a difference value between the z coordinate value of the gripper GR and the z coordinate value of the first image sensor 300 .

As an embodiment, the first height value h1 may be a value obtained by subtracting the first distance value d1 from the first reference distance value d0.

Meanwhile, referring to FIG. 6 , the first image IM1 may include a projection image PIM of the load target object OB. Here, the projection image PIM may be generated through perspective projection transformation or the like.

In an embodiment, the first image IM1 may include a coordinate image P1 ′ with respect to the position coordinate P1 of the gripper GR. In this case, the coordinate image P1 ′ with respect to the position coordinate P1 of the gripper GR may be displayed in the projection image PIM by overlapping the projection image PIM. Since the projection image PIM and the coordinate image P1' for the position coordinate P1 of the gripper GR are displayed together, there is an advantage in that the loading position can be accurately determined.

In an embodiment, the first height value h1 and the cross-sectional area value of the projection image PIM may be displayed on the first image IM1 .

The first height value h1, the projected image PIM, and the cross-sectional area value may be stored in a memory unit (not shown). When the number of loading target objects OB is plural, the plurality of first height values h1 , the plurality of projection images PIM, and the plurality of cross-sectional area values may be stored in the memory unit.

As described above, there is an advantage in that information on the size of each of the cargos can be easily obtained by using the first image sensor 300 even when there is no information on the size of each of the cargos in advance. In addition, there is an advantage in that the loading position can be accurately determined by obtaining information on the size of each of the cargoes.

On the other hand, in order for the loading target object OB to be located in the internal space of the second loading box LB2, it is necessary to recognize a loadable space in the internal space of the second loading box LB2.

Hereinafter, a method of measuring the second height value of the loading box required to recognize the loadable space will be described in detail.

7 to 10 are views for explaining an embodiment of measuring the second height value of the loading box.

7 to 10 , the weight map setting unit 120 calculates the second height values h21 and h22 of the second loading box LB2 from the second data to the second image sensor 400 . ) and the second distance values l1 and l2 between the second loading box LB2 may be extracted. Here, the distance value between the second image sensor 400 and the object is the plurality of regions X11, X12, X1n, X21, Xm1, Xmn (refer to FIG. 9 ) included in the second image IM2 (see FIG. 8 ). ), and the number of distance values between the second image sensor 400 and the object is the number of the plurality of regions X11, X12, X1n, X21, Xm1, and Xmn included in the second image IM2. can be the same as

As an example, the plurality of regions X11, X12, X1n, X21, Xm1, and Xmn included in the second image IM2 may correspond to regions included in the second image IM2. In an embodiment, the area included in the second image IM2 may be a unit area.

As an embodiment, the number of the plurality of regions X11, X12, X1n, X21, Xm1, and Xmn included in the second image IM2 is m*n(m) as shown in FIGS. 8 and 9 . , n may be a natural number). In this case, the number of distance values between the second image sensor 400 and the object may also be m*n.

Meanwhile, the number of the plurality of rows Row1, Row2, and Rowm may be m. And, when the plurality of areas X11, X12, X1n, X21, Xm1, and Xmn are areas corresponding to areas included in the second image IM2, a specific one of the plurality of rows Row1, Row2, and Rowm A row may correspond to any one horizontal line image in the second image IM2 . For example, the horizontal line image of the first portion A of the second image IM2 illustrated in FIG. 8 may correspond to the second row Row2 illustrated in FIG. 9 .

Meanwhile, although not clearly illustrated, the horizontal line image of the second portion B of the second image IM2 illustrated in FIG. 8 may correspond to the k-th row (not illustrated) illustrated in FIG. 9 . However, the present invention is not limited thereto. Hereinafter, for convenience of description, the present embodiments will be described with a specific row as a specific area row.

However, when the second image sensor 400 is disposed at one end of the support pole SP as shown in FIG. 7 , the image for the second loading box LB2 is somewhat distorted as shown in FIG. 8 . , an error may occur with respect to the second height values h21 and h22 calculated based on the second distance values l1 and l2. Accordingly, in order to accurately calculate the second height values h21 and h22 of the second loading box LB2, it is necessary to image-process and data-process the second image IM2 and the second data. Accordingly, the weight map setting unit 120 may process the second image IM2 as a top view image, and process the second data to data-process the second distance included in the second data. The values 111, 112, 11n, 121, lm1, and lmn may be converted into a vertical distance (eg, a distance in the z-axis direction) between the second image sensor 400 and the second loading box LB2. .

Here, the top view-processed second image (not shown) is the second image sensor 400 in the vertical direction (for example, z-axis direction) may be the same as the image taken. In this case, the z coordinate among the virtual position coordinates P3 ′ and the z coordinate among the actual position coordinates P3 of the second image sensor 400 may be the same (see FIG. 7 ). In addition, as shown in FIGS. 7 and 10 , the second distance values 111, 112, 11n, 121, lm1, and lmn included in the processed second data DA are virtual position coordinates P3'. It may be the same as a linear distance value that is a difference value between the z coordinate of the z coordinate and the z coordinate among the position coordinates of the second loading box LB2, and is included in the processed second data DA as shown in FIGS. 9 and 10 . The number of the second distance values 111, 112, l1n, l21, lm1, and lmn is also the number of the plurality of regions X11, X12, X1n, X21, Xm1, Xmn included in the second image IM2. can be the same as

In one embodiment, the weight map setting unit 120 is a second loading box (LB2) corresponding to each of the plurality of regions (X11, X12, X1n, X21, Xm1, Xmn) included in the second image (IM2) The second distance values 111, 112, 11n, 121, lm1, and lmn between the and the second image sensor 400 may be extracted.

Meanwhile, referring to FIG. 7 , when the second distance values l1 and l2 are extracted, the weight map setting unit 120 based on the second distance values l1 and l2 and a preset second reference distance value l0. Thus, the second height values h21 and h22 may be calculated.

In an embodiment, the second reference distance value 10 is based on the position coordinates P3 of the second image sensor 400 and the position coordinates of the second support member SM2 supporting the second loading box LB2. can be calculated by The position coordinates P3 of the second image sensor 400 may be previously stored in the second image sensor 400 and/or the loading control device, and the position coordinates P3 of the second support member SM2 and/or the second Coordinates of a surface (eg, xy plane) on which the support member SM2 supports the second loading box LB2 may be stored in the loading control device.

For example, the second reference distance value 10 is the second support member at the position coordinate P3 of the second image sensor 400 (or the virtual position coordinate P3 ′ of the second image sensor 400 ). (SM2) may be the same as the linear distance value (or the shortest distance value) to the surface (eg, xy plane) supporting the second loading box LB2. In this case, when the z coordinate among the position coordinates O of the second support member SM2 is 0, the second reference distance value 10 may be the same as the z coordinate value of the second image sensor 400 .

On the other hand, referring to FIGS. 7 to 9 , the loading box image LBIM for the second loading box LB2 is an image of the side surface of the second loading box LB2 and the bottom surface of the second loading box LB2 ( or internal). Accordingly, the second height value h21 for the side surface of the second loading box LB2 and the second height value h22 for the bottom surface of the second loading box LB2 may be calculated, respectively. That is, the weight map setting unit 120 extracts the loading box image LBIM from the second image IM2, divides the loading box image LBIM into a plurality of regions, Two height values h21 and h22 may be obtained based on the second data. Here, the plurality of regions may be, for example, a set of parts of a side surface of the loading box image LBIM, and may be a set of parts of a bottom surface of the loading box image LBIM.

Referring to FIG. 7 , for example, the second height value h21 for the side surface of the second loading box LB2 is the z coordinate value of the second image sensor 400 and the side surface of the second loading box LB2. It may be a value corresponding to the difference value between the z coordinate values, and the second height value h22 for the bottom of the second loading box LB2 is the z coordinate value of the second image sensor 400 and the second loading box LB2. may be a value corresponding to a difference value between the z coordinate values of the bottom surface (or the z coordinate value of the second image sensor 400 when the z coordinate is 0 among the position coordinates O of the second support member SM2) .

As described above, the second height values h21 for the side surface of the second loading box LB2 and the second height values h22 for the bottom surface of the second loading box LB2 are also shown in FIGS. 8 and 9 . As illustrated in FIG. 1 , it may be calculated for each of the plurality of regions X11 , X12 , X1n , X21 , Xm1 , and Xmn included in the second image IM2 . That is, the weight map setting unit 120 sets the second height values h21, h22) may be calculated for each of the plurality of regions included in the second image IM2.

On the other hand, the second height values h21 for the side surface of the second loading box LB2 may be generally the same, and the second height values h22 for the bottom surface of the second loading box LB2 are also common. may be the same as each other, and the second height value h21 for the side surface of the second loading box LB2 may be generally larger than the second height value h22 for the bottom surface of the second loading box LB2. have. Accordingly, the space inside the second loading box LB2 may be measured using second height values h2 of specific areas corresponding to the horizontal line image of the second image IM2 .

Hereinafter, a method of measuring the inner space of the second loading box LB2 and generating an image thereof will be described for areas corresponding to the first portion A of the second image IM2 shown in FIG. 8 . The second height values h2 and second height values h2 of regions corresponding to the second portion B of the second image IM2 will be used as an example.

11 is a graph showing second height values for regions corresponding to the first part shown in FIG. 8 , and FIG. 12 is a graph showing second height values for regions corresponding to the second part shown in FIG. 8 . It is a graph showing, and FIG. 13 is a view exemplarily showing a plurality of sub-images included in the image-processed second image.

8 and 11, the image included in the first part (A) is different from the image for the side of the second loading box (LB2) as shown in FIG. 8 (eg, on the floor) It may include an image for In this case, since the height of the side surface of the second loading box LB2 is generally constant, the graph of the second height value h2 for the regions corresponding to the first portion A has the form shown in FIG. 11 . can be

The weight map setting unit 120 compares a preset reference value (Threshold) and the second height value (h2), and divides a section in which the second height value (h2) is greater than the reference value (Threshold) and a section in which the reference value (Threshold) is less. can In this case, in the graph shown in FIG. 11 , the section in which the second height value is greater than the reference value (Threshold) is the A2 section, and the section in which the second height value (h2) is equal to or less than the reference value (Threshold) is the A1 section and the A3 section. .

At this time, the image corresponding to the areas included in the A2 section may be an image of the side of the second loading box (LB2), and the image corresponding to the areas included in each of the A1 section and the A3 section is the second loading box It can be an image for (LB2) and another object (eg floor).

On the other hand, referring to FIGS. 8 and 12 , the image included in the second part (B) is an image of another object (eg, the floor) as shown in FIG. 8 , and the second loading box (LB2) It may include an image for the side surface of, and an image for the bottom surface of the second loading box (LB2). In this case, the height of the bottom surface of the second loading box (LB2) is also generally constant, but is smaller than the height of the side surface of the second loading box (LB2), so the second for the areas corresponding to the second part (B) The graph of the height value h2 may have a form as shown in FIG. 12 .

The weight map setting unit 120 compares the above-described reference value (Threshold) with the second height value (h2), and distinguishes a section in which the second height value (h2) is greater than the reference value (Threshold) and a section in which the reference value (Threshold) is less than the reference value (Threshold). can In this case, in the graph shown in FIG. 12 , the sections in which the second height value h2 is greater than the reference value (Threshold) are sections B2 and B4, and the sections in which the second height value (h2) is less than or equal to the reference value (Threshold) are sections B1 , section B3, and section B5.

At this time, the images corresponding to the regions included in each of the section B2 and section B4 may be images of the side surface of the second loading box LB2, and the images corresponding to the regions included in the section B3 are the second loading box It may be an image of the bottom (or inside) of (LB2), and the image corresponding to the areas included in each of the sections B1 and B5 is an image of an object (eg, floor) different from the second loading box (LB2). can be

Meanwhile, referring to FIG. 13 , according to an embodiment, the weight map setting unit 120 is configured to correspond to regions included in a section (eg, B3) in which the second height value h2 is equal to or less than a reference value (Threshold). The image may be image-processed as the first sub-image SIM1, and the second height value h2 corresponds to regions included in a section (eg, A2, B2, B4) greater than the reference value (Threshold). The image may be image-processed as the second sub-image SIM2.

Here, the first sub-image SIM1 may mean an image displayed with the lowest gradation (black gradation), and the second sub-image SIM2 may mean an image displayed with the highest gradation (white gradation).

In addition, the weight map setting unit 120 may set the first sub-image SIM1 as the loadable space image and set the second sub-image SIM2 as the non-loadable space image.

On the other hand, since the second height value h2 is equal to or less than the threshold in the sections A1 and A3 shown in FIG. 11 and sections B1 and B5 shown in FIG. 12 , the weight map setting unit 120 sets the A1 section , an image corresponding to regions included in each of the sections A3, B1, and B5 may be image-processed as the third sub-image SIM3. Here, the third sub-image SIM3 may refer to an image displayed with the lowest gray scale (black gray scale) in the same manner as the first sub-image SIM1 . However, since the third sub-image SIM3 corresponds to a non-loadable space, although not shown, the weight map setting unit 120 removes the third sub-image SIM3 from the image-processed second image IM2'. Image processing can be performed.

Meanwhile, referring to FIGS. 10 and 13 , the first part A and the second part B of the second image IM2 shown in FIG. 10 are The first part (A) and the second part (B) may be indicated. According to this, there is an advantage in that the loading space can be changed to a single cycle image to perform faster calculations.

Although not shown, the above-described image processing may be omitted. In this case, the weight map setting unit 120 is configured in regions (eg, A1, A3, B1, B3, and B5) having a second height value h2 that is less than or equal to the threshold. To provide the loading position determining unit 130 with the first information about the first information and the second information about the regions having a second height value greater than the threshold (eg, section A2, section B2, and section B4). can

14 is a diagram illustrating a weight map according to an embodiment of the present invention, and FIG. 15 is a diagram for explaining an embodiment in which a weight is set for each of a plurality of groups.

Referring to FIG. 14 , the weight map setting unit 120 sets weights W11, W12, W1n, W21, Wm1, and weights W11, W12, W1n, W21, Wm1, Wmn) to set the weight map WM.

13 and 14 , the weight map setting unit 120 sets weights (W11, W12, W1n, W21, Wm1, Wmn) for each of a plurality of regions included in the first sub-image SIM1 set as the loading space. It is possible to set the weight map WM for the first sub-image SIM1 by giving .

Here, the weight map WM may mean a digitized pattern determined so that the loading target OB is loaded according to a required loading order (or loading scheduling). Weights W11, W12, W1n, W21, Wm1, and Wmn of the weight map WM may be assigned to each of a plurality of regions included in the image. The number of weights W11, W12, W1n, W21, Wm1, and Wmn may be the same as the number of regions included in the image, and the weights W11, W12, W1n, W21, Wm1 and Wmn) may have different values, and at least some of them may have the same value.

Meanwhile, although not clearly shown, according to an embodiment, the weight map setting unit 120 sets weights W11, W12, W1n, W21, Wm1, Wmn) can be assigned. However, since the second sub-image SIM2 and the third sub-image SIM3 are images for a non-loadable space, the loading position determining unit 130 determines the second sub-image SIM2 and the third sub-image SIM3. A weight applied to a plurality of regions included in , and a weight applied to a plurality of regions included in the first sub-image SIM1 may be controlled differently. By differently controlling weights given to a plurality of regions included in the second sub-image SIM2 and the third sub-image SIM3, the plurality of regions included in the second sub-image SIM2 and the third sub-image SIM3 are controlled differently. Areas can be identified as non-loadable spaces. In an embodiment, the loading position determining unit 130 may assign different weights to each of the first to third sub-images SIM1 to 3 .

As an embodiment, the second weight may be greater than a first calculated value of all the first weights applied to each of the regions in the second image IM2 . Here, the first calculated value may mean a value that is increased by an operation, such as a sum of all first weights and a product of all first weights.

As an embodiment, the second weight may be smaller than a second calculated value of all the first weights applied to each of the regions in the second image IM2 . Here, the second calculated value may mean a value reduced by an operation such as subtraction of all first weights.

Meanwhile, according to an embodiment, the weights W11, W12, W1n, W21, Wm1, and Wmn may be assigned to each group PG in which regions are grouped according to a predetermined criterion. For example, the weight map setting unit 120 assigns a first weight to a first group of regions included in the loadable space image (eg, the first sub-image SIM1), and the loadable space image A second weight different from the first weight may be given to the second group of regions included in (eg, the first sub-image SIM1 ).

The embodiment of FIG. 15 illustrates a case in which weights are assigned to a plurality of groups each including a plurality of regions.

Specifically, for example, the weight map setting unit 120 assigns a weight W1 to the first group PG1 , a weight W2 to the second group PG2 , and a weight W4 to the third group PG3 . , and a weight Wm may be assigned to the mth group PGm. However, the present invention is not limited thereto.

Hereinafter, a method for determining the loading position will be described, but for convenience of explanation, it is assumed that the number of a plurality of regions included in the loadable space image (eg, the first sub-image SIM1) is 5*5. do.

16 is a view for explaining an embodiment of determining a first loading position for a loading target object.

Referring to FIG. 16 , the loading position determining unit 130 displays a projection image (eg, P1 ') in which a coordinate image (eg, P1') corresponding to a point where the robot arm 200 and the loading target object OB are in contact with each other is displayed. For example, the PIM may be disposed on the sub-image in which the weight map WM is set. In addition, the loading position determining unit 130 is provided to each of the regions (eg, X11, X12, X1n, X21, Xm1, Xmn) overlapping the projection image (eg, PIM) at least in part. Weights (eg, W11, W12, W1n, W21, Wm1, Wmn) may be extracted from the weight map WM. In addition, the loading position determining unit 130 may calculate an arrangement value of the projected image by calculating the extracted weights. And, the loading position determining unit 130 may determine the loading position based on the arrangement value. Here, the calculation of the extracted weights means, for example, a sum of weights, a product of weights, a difference between weights, and the like, and an arrangement value may mean a value obtained by calculating weights. However, the present invention is not limited thereto.

As an embodiment, the loading position determining unit 130 may virtually arrange the projected image in which the position coordinates of the robot arm 200 are displayed on the first sub-image SIM1 in which the weight map WM is set.

For example, a first projection image PIM1 for a first load target (eg, load target OB shown in FIG. 1 ) is obtained and it is assumed that the first projection image PIM1 is rectangular in shape. Assume that the first projection image PIM1 may display a coordinate image P1 ″ for the position coordinate P1 of the gripper GR holding the first loading target object. In addition, the first projection image PIM1 may be arbitrarily disposed on the first sub-image SIM1 to which the weight map WM is set.

When the first projection image PIM1 is virtually disposed on the first sub-image SIM1 to which the weight map WM is set, the loading position determining unit 130 may generate a loadable space image (eg, the first sub-image ( The sum of weights given to each of the regions at least partially overlapping the first projection image PIM1 among the plurality of regions included in SIM1)) may be calculated. In this case, the number of cases in which the first projection image PIM1 is virtually disposed on the first sub-image SIM1 to which the weight map WM is set may be infinitely large.

For example, the first projection image PIM1 may be disposed on the upper left side based on the first sub-image SIM1 to which the weight map WM shown in FIG. 16 is set. In this case, since the weights given to each of the regions at least partially overlapping the first projection image PIM1 disposed at the upper left are W11, W12, W21, and W22, the calculated value of the weights, for example, the sum of the weights, is W11+W12+W21+W22, which may be an arrangement value of the first projection image PIM1.

For example, the first projection image PIM1 may be disposed at the upper right side with respect to the first sub-image SIM1 to which the weight map WM shown in FIG. 16 is set, and a value calculated by weights, that is, the weights. The sum of them is W14+W15+W24+W25, which may be the arrangement value of the first projection image PIM1.

For example, the first projection image PIM1 may be disposed at the lowermost right of the first sub-image SIM1 to which the weight map WM shown in FIG. 16 is set. In this case, since the weights given to each of the regions at least partially overlapping the first projection image PIM1 are W43, W44, W45, W54, and W55, the calculated value of the weights, that is, the sum of the weights, is W34+W43+W44 +W45+W54+W55, which may be an arrangement value of the first projection image PIM1.

In one embodiment, the loading position determining unit 130 may determine the loading position based on the position of the projected image that satisfies the loading condition in which the placement value is preset. Here, the loading condition may be a condition in which the batch value belongs to a certain range of weights or the batch value has a maximum value or a minimum value.

For example, when the sum of the weights is calculated, the loading position determining unit 130 may determine a projection image that satisfies the loading condition in which the batch value is the minimum value, and determine the loading position corresponding to the projection image that satisfies the loading condition. have.

In the above example, when W11+W12+W21+W22, which is one of the sums of weights, is the minimum, the first projection image PIM1 satisfying the loading condition is within the first sub-image SIM1 in which the weight map WM is set. may correspond to an image disposed at the upper left of the . However, the present invention is not limited thereto, and a weight may be appropriately set so that the target object is loaded randomly or loaded in a specific direction.

For example, the loading positioning unit 130 may be configured such that the first projected image PIM1 is disposed at the upper left in the loadable space image (or the first projected image PIM1 is at the upper right in the loadable space image). to be disposed on), the loading position may be determined by calculating the position coordinates P1 of the gripper GR and the second yaw rotation angle.

In an embodiment, the loading position determining unit 130 is loaded based on the position coordinates of the robot arm 200 (eg, the position coordinates P1 of the gripper GR) and the first height value h1 ). Calculate the target position coordinates corresponding to the coordinate image P1'' displayed on the projection image that satisfies the condition, calculate the second yaw rotation angle based on the first yaw rotation angle θ1 and the projection image, and calculate the target position It is possible to determine the loading position including the coordinates and the second yaw rotation angle.

5, 7, and 16, for example, the x-coordinate and y-coordinate values among the target position coordinates of the second supporting member SM2 are obtained from the previously stored position coordinate values of the second supporting member SM2. can be decided. In addition, assuming that the z coordinate among the position coordinates O of the second support member SM2 is 0, the z coordinate value among the position coordinates is the z coordinate value among the position coordinates P1 of the gripper GR shown in FIG. 5 . The value may be the same as a value obtained by subtracting the first height value h1 from the value. And, the second yaw rotation angle is the first projection image PIM1 included in the first image IM1 obtained when the first loading target object is positioned in the object recognition area is yaw rotated to satisfy the loading condition It may be calculated by adding or subtracting a predetermined rotation angle from the first yaw rotation angle θ1 to match the first projection image PIM1 .

Although the above-described embodiment has described a method of determining the loading position based on that the value for which the weights are calculated is the sum of the weights and the batch value is the minimum value, a calculation method in which the batch value is increased by calculating the weights (eg, multiplication) , power, etc.) may be applied in the same manner as in the above-described embodiment.

In addition, although not shown, the method of determining the loading position based on that the batch value is the maximum value is also similar to that described above. In this case, the calculated value of the weights may be subtraction. In addition, a second weight applied to the second sub-image SIM2 (and the third sub-image SIM3 ) may be smaller than a minimum value of an arrangement value calculated using only the first weights applied to each of the regions.

In FIG. 16 , the projection image, for example, the first projection image PIM1 and the weight map WM are shown as at least partially overlapped, but the projection image, for example, the first projection image PIM1 is processed (or processed). (processing) to the secondary image, for example, a rectangular image, a circular image, an elliptical image, etc. corresponding to the boundary portion of the projection image, the weight map WM is at least partially overlapped, so that the loading position can be determined as described above. .

17 is a view exemplarily showing a plurality of sub-images included in the image-processed second image when a loading target is loaded, and FIG. 18 is a second example of another loading target when the loading target is loaded 2 is a view for explaining an embodiment of determining the loading position, and FIG. 19 is a view exemplarily showing a plurality of sub-images included in the image-processed second image when loading target objects are loaded.

13, 16, and 17 , the weight map setting unit 120 may set the projection image satisfying the loading condition as the non-loadable space image when the loading target is disposed at the determined loading position. This is to exclude the space occupied by the loading target object OB already disposed in the second loading box LB2 from the loading space.

For example, when the first loading target shown in FIG. 16 is disposed at the determined loading position, the first projection image PIM1 satisfying the loading condition may be image-processed as the second sub-image SIM2 ′. . Here, the second sub-image SIM2' may be an image displayed with the highest grayscale as shown in FIG. 13 . In this case, the first sub-image SIM1' set as the loadable space image may be reduced by the first projected image PIM1, and the second sub-image SIM2' set as the non-loadable space image is the first projected image It can be extended by (PIM1). Meanwhile, the third sub-image SIM3 may be maintained.

On the other hand, the method of loading the 2nd loading target object selected after the 1st loading target object is loaded in 2nd loading box LB2 is also similar to the above-mentioned.

For example, when the second projection image PIM2 of the second loading target is acquired, the second projection image PIM2 in which the coordinate image P1''' for the position coordinate P1 of the gripper GR is displayed. The false weight map WM may be arbitrarily disposed in the set first sub-image SIM1'.

And, as a number of cases, for example, the second projection image PIM2 may be disposed at the top with respect to the first sub-image SIM1 ′ set as the loadable space image shown in FIG. 18 . The sum may be W12+W13+W22+W23.

In addition, as a number of other cases, for example, the second projection image PIM2 may be disposed at the left middle with respect to the first sub-image SIM1 ′ set as the loadable space image shown in FIG. 18 . The sum of them may be W31+W32+W41+W42.

Although not shown, the number of cases in which the second projection image PIM2 is virtually disposed on the first sub-image SIM1 ′ to which the weight map WM is set may be innumerable, and the sum of the weights is W12+W13+W22 It can be expressed as a sum of weights less than +W23 (or W31+W32+W41+W42) or as a sum of weights more than W12+W13+W22+W23 (or W31+W32+W41+W42).

For example, in the above example, when W12+W13+W22+W23, which is one of the sums of weights, is the minimum, the second projection image PIM2 satisfying the loading condition is the first sub-image to which the weight map WM is set. (SIM1') may correspond to the image disposed at the top. The loading position determining unit 130 determines the loading position by calculating the position coordinates P1 and the second yaw rotation angle of the gripper GR so that the second projected image PIM2 is disposed at the top in the loadable space image. can decide However, the present invention is not limited thereto.

On the other hand, referring to FIG. 19 , when the second loading target object is loaded, the second projection image PIM2 that satisfies the loading condition may be set as a non-loadable space image, and specifically, the second projection image PIM2 is The second sub-image IM2'' may be image-processed, and in this case, the first sub-image IM1'' may be reduced by the first projection image PIM1 and the second projection image PIM2, The second sub-image IM2 ″ may extend as much as the first projection image PIM1 and the second projection image PIM2 . Meanwhile, the third sub-image SIM3 may be maintained.

As described above, when the loadable objects OB are loaded in the second loading box LB2, the loadable space image may be reduced, and the loadable space image may be expanded.

On the other hand, when the loading target objects OB are sufficiently loaded on the bottom surface of the second loading box LB2 and the loading target objects OB can no longer be loaded on the bottom surface of the second loading box LB2, the loaded loading In some cases, loading target objects OB may be additionally loaded on target objects OB.

Hereinafter, a method of loading the loading target objects OB as a multi-layer having two or more layers on which the loading target objects OB are loaded will be described in detail.

20 to 22 are views for explaining an embodiment of determining the third loading position on the loading target object loaded in the loading box.

Referring to FIGS. 20 and 21 , when loading objects are loaded in the second loading box LB2 , there is a case where a projection image of another loading target cannot be arranged in the loadable space image.

Specifically, for example, the first projection image PIM1 to the fifth projection image PIM5 are image-processed into a second sub-image (eg, image) displayed with the highest gradation as shown in FIG. 20 , The first sub-image or the first sub-image displayed with the lowest gradation (black gradation) in the space adjacent to the 5 projection image PIM5 exists as a loadable space image, but the size of the projection image of the additional loadable object remains. If it is larger than the sub images, it may be determined that the loading position does not exist. In this case, the loading position determining unit 130 may output a flag signal for notifying that the loading is impossible to the weight map setting unit 120 . The weight map setting unit 120 may change the reference value in response to the flag signal. As an embodiment, the weight map setting unit 120 may change the reference value until a flag signal is not received.

On the other hand, Figure 21 is a side view when viewed from the inside of the second loading box along the dotted line s-s' shown in Figure 20. Referring to FIG. 21 , the height (eg, the first height value h1 ) of the loading target objects may be the same or different from each other.

In one embodiment, the loading target objects loaded in the second loading box LB2 may be regularly arranged, such as arranged side by side without falling over as shown in FIG. 21 . In this case, the height value of each of the loading target objects is a value obtained by the first image sensor 300 (eg, the first height value h1), as described above with reference to FIGS. 2 and 5 . may be the same as

Although not shown, in an embodiment, when at least one of the loading target objects loaded in the second loading box LB2 is inclined, etc., the loading target objects loaded in the second loading box LB2 are irregularly aligned. , The height value of each of the loading objects is measured in a direction perpendicular to the bottom of the second loading box LB2, for example, in the z-axis direction shown in FIGS. 1 to 3, 5, and 7 It may be equal to the value of the largest length among the lengths. In this case, a direction perpendicular to the bottom of the second loading box LB2 (eg, a length measured in the z-axis direction may be a value obtained by the second image sensor 300 ).

Referring to FIG. 21 , when the height of the fifth load target object OB5 is lower than the height of each of the first load target object OB1 to the fourth load target object OB4, based on the images shown in FIG. 20 . Even if it is determined that it cannot be loaded, if another load target OB can be additionally loaded in the area L on the fifth load target object OB5, more load target objects OB are placed in the second loading box LB2 ), the loading efficiency can be increased.

Accordingly, according to an embodiment, the weight map setting unit 120 may change a preset threshold value in response to a flag signal indicating that loading is not possible from the loading position determining unit 130 .

For example, the weight map setting unit 120 may stepwise increase the threshold by a preset value until a flag signal is not received.

Also, in one embodiment, the weight map setting unit 120, in response to the flag signal from the loading position determining unit 130, sets a reference value (Threshold) to the height value of at least one loading target loaded in the loading box (eg, For example, it can be changed to the first height value (h1)). In an embodiment, the weight map setting unit 120 may change the threshold to a value within a predetermined range from the height value of the loading target in response to the flag signal.

For example, the reference value may be changed within a preset offset range based on the height value of the at least one loading target loaded in the loading box.

When there are a plurality of loading target objects loaded in the loading box, the changed threshold value may be within an offset range based on the minimum value among the height values (eg, the first height value h1).

Referring to FIG. 21 , for example, the first height value h15 of the fifth load target object OB5 is the first height value of each of the first load target object OB1 to the fourth load target object OB4 . (h11, h12, h13, h14), the reference value (Threshold) may be changed to the same value as the first height value (h15) of the fifth load target object (OB5).

When the reference value (Threshold) is changed, the weight map setting unit 120 similarly to those described above with reference to FIGS. 11 and 12 , regions included in the section in which the first height value h1 is equal to or less than the threshold value (Threshold). The image corresponding to can be image-processed as a first sub-image and a third sub-image SIM3, and an image corresponding to regions included in a section in which the first height value h1 is greater than the reference value (Threshold) is generated. The image may be processed as two sub-images, and a second weight may be assigned only to the second sub-image SIM2 , or a second weight may be assigned to the second sub-image SIM2 and the third sub-image SIM3 .

The weight map setting unit 120 may set the weight map by assigning weights to each of the regions included in the sub-image having a second height value equal to or less than the changed reference value.

Although not shown, in the same manner as described above, in one embodiment, when the loading target objects loaded in the second loading box LB2 are irregularly aligned, the loading target objects are loaded in the second loading box LB2. Then, since the height value of each of the loading target objects loaded in the second loading box LB2 may be acquired by the second image sensor 300 , the reference value is the height value acquired by the second image sensor 300 . may be changed based on , or may be changed to a smallest height value among a plurality of height values obtained by the second image sensor 300 .

Referring to FIGS. 20 and 22 , the fifth projection image PIM5 shown in FIG. 20 is image-processed as a first sub-image SIM1''' as shown in FIG. 22 after the threshold is changed. and can be set as a loadable space image. The first sub-image SIM1''' included in the second image IM2'' shown in FIG. 22 may be expanded compared to that shown in FIG. 20 , and the second image IM2' shown in FIG. 22 . The second sub-image SIM2''' included in ') may be reduced compared to that shown in FIG. 20 , and the third sub-image SIM3 included in the second image IM2'' shown in FIG. 22 . ) can be maintained.

As described above, there is an advantage in that loading efficiency can be maximized by loading the loading target object in a multi-layer manner in the loading box.

Hereinafter, a loading control method capable of performing all of the above-described embodiments will be described.

23 is a flowchart illustrating a loading control method according to an embodiment of the present invention.

Referring to FIG. 23 , the loading control method according to an embodiment of the present invention includes the steps of controlling the robot arm so that the loading target is located in the object recognition area ( S100 ), and the first image output from the first image sensor and A step (S200) of obtaining a first height value and a projection image of the object to be loaded on the basis of the first data, and a second image output from the second image sensor and the second data of the loading box Determining the loadable space image and the loadable space image (S300), setting a preset weight map WM in the loadable space image, and loading target based on the first height value, the projection image, and the weight map Determining the loading position of the object (S400) and when the loading position is determined, the robot arm may include controlling the robot arm to load the loading target object to the loading position (S500).

As described above, in the embodiments of the present invention, even if the cargos have different sizes or information on the sizes is not provided in advance, the cargo can be maximally loaded into the loading box by maximizing the loadable space.

In addition, embodiments of the present invention can prevent malfunctions by accurately determining the loading position even if cargos have different sizes or information on the sizes is not provided in advance.

In addition, embodiments of the present invention can minimize loading costs by maximizing loading efficiency, even if cargos have different sizes or information about the sizes is not provided in advance.

In the above, the present invention has been described focusing on the embodiments of the present invention, but this is only an example and does not limit the present invention. It will be appreciated that various combinations or modifications and applications not illustrated in the embodiments are possible within the scope. Accordingly, the descriptions related to the variations and applications that can be easily derived from the embodiments of the present invention should be interpreted as being included in the present invention.

10: object loading system 100: loading control device
110: object information acquisition unit 120: weight map setting unit
130: loading position determining unit 140: control unit
200: robot arm 300: first image sensor
400: second image sensor

Claims (16)

an object information acquisition unit configured to acquire a projection image and a first height value of the loading target object based on the first image and the first data output from the first image sensor;
Second height values for a space sensed by the second image sensor are obtained based on second data output from the second image sensor, and second height values output from the second image sensor are obtained based on the second height values. 2 a weight map setting unit for setting a weight map for the image;
a loading position determining unit configured to determine a loading position of the loading target based on contact point information on a point where the robot arm and the loading target contact each other, the first height value, the projected image, and the weight map; and
When the loading position is determined, the loading control device comprising a control unit for controlling the robot arm to load the object to be loaded in the loading position. According to claim 1,
The object information acquisition unit,
extracting a first distance value between the loading target object and the first image sensor from the first data,
Loading control device, characterized in that for calculating the first height value based on a first reference distance value and the first distance value set in advance. According to claim 1,
The weight map setting unit,
Loading control apparatus, characterized in that the weight map is set by assigning weights to each of the regions included in the sub-image having a second height value that is less than or equal to a preset reference value among the second height values. 4. The method of claim 3,
The loading position determining unit,
By arranging the projection image in which the coordinate image corresponding to the point is displayed on the sub-image, weights given to each of the regions included in the sub-image that at least partially overlap with the projected image are extracted from the weight map do,
calculating the placement value of the projected image by calculating the extracted weights;
Loading control device, characterized in that determining the loading position based on the placement value. 5. The method of claim 4,
The loading position determining unit,
and determining the loading position based on a position of the projected image in which the placement value satisfies a preset loading condition. 6. The method of claim 5,
The loading position determining unit,
Target position coordinates for loading the object to be loaded into the loading position based on the contact point information, the first height value, the projected image, and the position of the projected image at which the placement value satisfies the loading condition, and Loading control device, characterized in that calculating the yaw rotation angle. 6. The method of claim 5,
The loading position determining unit,
When the batch value does not satisfy the loading condition, a flag signal indicating that loading is impossible is provided to the weight map setting unit,
The weight map setting unit,
and changing the reference value in response to the flag signal. According to claim 1,
The weight map setting unit,
obtaining second height values for each of a plurality of regions in the second image based on the second data;
A first weight is given to an area having a second height value equal to or less than a preset reference value among the second height values;
By giving a second weight different from the first weight to an area having a second height value greater than the reference value among the second height values,
Loading control device, characterized in that setting the weight map. 9. The method of claim 8,
The loading position determining unit,
By disposing the projection image in which the coordinate image corresponding to the point is displayed on the second image, weights given to each of the regions at least partially overlapping with the projected image among the regions included in the second image are weighted. extracted from the map,
calculating the placement value of the projected image by calculating the extracted weights;
Loading control device, characterized in that determining the loading position based on the placement value. 10. The method of claim 9,
The loading position determining unit,
and determining, as the loading position, a position of a projection image in which the placement value satisfies a preset loading condition. 11. The method of claim 10,
The loading control device, characterized in that the batch value reflected in the at least one second weight does not satisfy the loading condition. 11. The method of claim 10,
The loading position determining unit,
Target position coordinates for loading the loading target object to the loading position based on the contact point information, the first height value, the projected image, and the position of the projection image at which the placement value satisfies the loading condition; Loading control device, characterized in that calculating the yaw rotation angle. 12. The method of claim 11,
The loading position determining unit,
When the batch value does not satisfy the loading condition, a flag signal indicating that loading is impossible is provided to the weight map setting unit,
The weight map setting unit,
and changing the reference value in response to the flag signal. 14. The method of claim 13,
The weight map setting unit,
Loading control device, characterized in that for changing the reference value within a preset offset range based on the height value of the at least one loading target loaded in the loading box. 15. The method of claim 14,
The weight map setting unit,
Loading control device, characterized in that for changing the reference value within the offset range based on a minimum value among height values when the number of loading objects loaded in the loading box is plural. acquiring a projection image and a first height value of the load target object based on the first image and the first data output from the first image sensor;
Second height values for a space sensed by the second image sensor are obtained based on second data output from the second image sensor, and second height values output from the second image sensor are obtained based on the second height values. 2 setting a weight map for the image;
determining a loading position of the loading target based on contact point information on a point where the robot arm and the loading target contact each other, the first height value, the projected image, and the weight map; and
When the loading position is determined, the loading control method comprising the step of controlling the robot arm to load the object to be loaded in the loading position.

Images