KR102307047B1 - Apparatus and method for determining picking order of the picking robot - Google Patents

Abstract

A picking priority selection device is provided. The picking priority selection device comprises: an acquisition unit configured to acquire a two-dimensional image and depth information for a setting area on which a plurality of picking objects are placed; a detection unit configured to detect the object using the two-dimensional image and the depth information; an analysis unit that analyzes the two-dimensional image and the object detected in the depth information to calculate at least two pieces of environment information for each object; and a picking priority selection unit configured to select a picking priority of the plurality of picking objects by applying a predetermined weight to the at least two pieces of environment information. An object of the present invention is to provide the picking priority selection device and the picking priority selection method for selecting and giving a priority to be picked by a robot for a plurality of picking objects.

Description

Apparatus and method for determining picking order of the picking robot

The present invention relates to an apparatus and method for selecting a picking order of an object to be picked by a robot.

In order to control a robot that processes or processes an object according to the object, a sensor for detecting an object may be used.

The processing accuracy of the robot on the object may be related to the measurement precision of the sensor. In order to improve the processing accuracy of the robot, the higher the measurement precision of the sensor, the more advantageous.

However, since a sensor having high measurement accuracy is very expensive, popularization is difficult. Also, even when a sensor having high measurement accuracy is provided, a situation in which measurement accuracy is lowered due to diffuse reflection or the like may occur depending on the type of object.

When performing a picking operation on an object detected by using a sensor, a problem of selecting which object to pick from among a plurality of objects may occur.

Korean Patent Application Laid-Open No. 2019-0072285 discloses a technique of grasping the position of a gripping object selected through a picking view camera and gripping the gripping object using a gripper.

Korean Patent Publication No. 2019-0072285

It is an object of the present invention to provide a picking priority selection device and a picking priority selection method for selecting and giving a priority to be picked by a robot for a plurality of picking objects.

According to an embodiment of the present invention, a picking priority selection apparatus is provided. The selection robot control apparatus may include: an acquisition unit configured to acquire a two-dimensional image and depth information for a setting area on which a plurality of picking objects are placed; a detector configured to detect the object using the two-dimensional image and the depth information; an analysis unit that analyzes the two-dimensional image and the object detected in the depth information and calculates at least two pieces of environment information for each object; and a picking priority selector configured to select a picking priority of the plurality of picking objects by applying a predetermined weight to the at least two or more pieces of environment information.

The environment information includes at least two of an interference degree indicating a rate at which the object is interfered with by another object, a height of a position where the object is placed, an area of a masking area of the object, and a distance of interest between the center of the object and the center of the two-dimensional image. It may contain more than one.

The acquiring unit may further receive information on an effector for picking the picking target, and the picking order selecting unit may change at least one of the environment information and the weight according to the information on the effector.

The analysis unit may correct the interference degree by using information about the effector received from the acquisition unit.

The analysis unit may extract coordinate information of a segmentation pixel of the object by using a deep learning model targeting the two-dimensional image.

The analysis unit may set the interference check area of the object by adding a margin necessary for the entry of the effector of the robot picking the object to the masking area of the object corresponding to the set of coordinate information.

The analysis unit may calculate the degree of interference according to an intrusion ratio of another object to the interference check area of a specific object.

The analyzer may classify the area of the object by using the coordinate information.

The analyzer may extract a depth value of a topmost point corresponding to the height in the area of the object.

The analyzer may extract coordinates of a central pixel disposed at a center among the plurality of divided pixels forming the object.

The analyzer may calculate the distance of interest corresponding to a distance between the central coordinates of a region of interest (ROI) set in the camera for capturing the set region and the coordinates of the central pixel.

In addition, according to another embodiment of the present invention, there is provided a robot having an arm and an effector installed on the arm. The robot may include: a picking priority selection device; and a control device for controlling the arm and the effector to pick an object according to the picking priority received from the picking priority selection device.

In addition, according to another embodiment of the present invention, a picking priority selection method for selecting a picking priority of a plurality of picking objects is provided. The picking priority selection method may include: acquiring a two-dimensional image and depth information for a setting area in which the plurality of picking objects are placed; detecting the object using the two-dimensional image and the depth information; calculating at least two pieces of environmental information for each object by analyzing the two-dimensional image and the object detected in the depth information; and selecting a picking priority of the plurality of picking objects by applying a predetermined weight to the at least two pieces of environment information.

The environment information includes at least two of an interference degree indicating a rate at which the object is interfered with by another object, a height of a position where the object is placed, an area of a masking area of the object, and a distance of interest between the center of the object and the center of the two-dimensional image. It may contain more than one.

A picking priority selection method may be provided, wherein information on an effector for picking the picking target is further input, and at least one of the environment information and the weight is changed according to the information on the effector.

A picking priority selection method may be provided, characterized in that the interference degree is corrected using the information on the effector.

A robot that extracts coordinate information of segmentation pixels of the object by using a deep learning model targeting the two-dimensional image, and picks the object in the masking area of the object corresponding to the set of coordinate information Picking priority selection, characterized in that the interference check area of the object is set by adding a margin necessary for the entry of the effector of A method may be provided.

A picking priority selection method may be provided, wherein the area of the object is divided by using the coordinate information, and a depth value of the highest uppermost point corresponding to the height is extracted from the area of the object.

The coordinates of the central pixel arranged in the center of the plurality of divided pixels forming the object are extracted, and the distance between the central coordinates of the ROI (Region Of Interest) set in the camera for photographing the set area and the coordinates of the central pixel. A picking priority selection method for calculating the corresponding distance of interest may be provided.

According to the present invention, a picking priority may be selected according to an order in which picking is easy among a plurality of objects arranged in a setting area. As a result, the lead time for later searching for the picking point, the contact point, and the grab point can be shortened and the picking success rate can be improved.

According to the present invention, candidates to be picked may be selected in the order of the highest picking success rate. Further, according to the present invention, a weight concept may be introduced to improve the analysis accuracy of the picking success rate. In particular, the weight applied to each different environmental information may be changed according to the type of the effector for picking the object.

According to the present invention, in a state in which a plurality of objects are congested, the objects having a high picking success rate can be processed one by one. Whenever an object is picked up one by one, an obstacle to picking up another object is reduced, so that the pickup success rate or the picking success rate of the object can be maintained above the set probability regardless of the number of pickups.

1 is a block diagram illustrating an apparatus for selecting a picking priority according to an embodiment of the present invention.
2 is a schematic diagram showing a picking apparatus according to an embodiment of the present invention.
3 is a schematic diagram illustrating a process of calculating an interference degree.
4 is a schematic diagram illustrating a distance of interest.
5 is a schematic diagram illustrating adjustment of weights according to types of effectors.
6 is a schematic diagram illustrating an interference degree considering an interference check area.
7 is a flowchart illustrating a picking priority selection method according to an embodiment of the present invention.
8 is a diagram illustrating a computing device according to an embodiment of the present invention.

Hereinafter, with reference to the accompanying drawings, embodiments of the present invention will be described in detail so that those of ordinary skill in the art to which the present invention pertains can easily implement them. However, the present invention may be embodied in many different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

In the present specification, duplicate descriptions of the same components will be omitted.

Also, in this specification, when it is mentioned that a certain element is 'connected' or 'connected' to another element, it may be directly connected or connected to the other element, but another element in the middle. It should be understood that there may be On the other hand, in this specification, when it is mentioned that a certain element is 'directly connected' or 'directly connected' to another element, it should be understood that the other element does not exist in the middle.

In addition, the terms used herein are used only to describe specific embodiments, and are not intended to limit the present invention.

Also, in this specification, the singular expression may include the plural expression unless the context clearly dictates otherwise.

Also in this specification, terms such as 'include' or 'have' are only intended to designate that the features, numbers, steps, operations, components, parts, or combinations thereof described in the specification exist, and one or more It should be understood that the existence or addition of other features, numbers, steps, operations, components, parts or combinations thereof is not precluded in advance.

Also in this specification, the term 'and/or' includes a combination of a plurality of listed items or any of a plurality of listed items.

Also, in this specification, detailed descriptions of well-known functions and configurations that may obscure the gist of the present invention will be omitted.

1 is a block diagram illustrating an apparatus for selecting a picking priority according to an embodiment of the present invention.

As shown in the figure, the picking priority selection apparatus according to an embodiment of the present invention may include an acquisition unit 210 , a detection unit 230 , an analysis unit 250 , and a picking priority selection unit 270 . have.

The acquisition unit 210 may acquire a scan result of the setting area. The scan result may include at least one of a two-dimensional image, depth information, and a depth image of the setting area. Here, the depth image is one of methods for representing depth information by mapping the depth information of each pixel to a 2D image or a 3D image.

The detector 230 may analyze the scan result and detect a plurality of picking objects 90 arranged in the setting area.

The analyzer 250 may analyze environmental information of each of the plurality of objects 90 when a plurality of objects 90 that the effector 20 can pick are arranged in the setting area.

The picking ranking selecting unit 270 may select a picking candidate having a high picking success rate among the plurality of objects 90 . As an example, the picking ranking selection unit 270 may compare the environmental information of each object 90 analyzed by the analysis unit 250 . The picking priority selection unit 270 may select a picking candidate or a picking priority of the object 90 according to the comparison result of the environmental information of each object 90 .

The picking order selection unit 270 may independently evaluate at least one of environmental information among a height, an area, a distance of interest, and a degree of interference around each object 90 . The picking priority selection unit 270 may select the picking priority of each object 90 according to the evaluation result of the environment information for each object 90 .

The picking candidates or picking priorities selected by the picking priority selection unit 270 may be provided to the picking apparatus 100 that operates to pick the object 90 disposed in the setting area. The picking prioritization apparatus of the present invention may be formed integrally with the picking apparatus 100 or may be formed separately.

2 is a schematic diagram illustrating a picking apparatus 100 according to an embodiment of the present invention.

As shown in the drawing, the picking apparatus 100 according to an embodiment of the present invention may include photographing units 110 and 120 and a robot 10 .

The photographing units 110 and 120 may photograph the setting area. For example, when a bin 80 containing the object 90 is disposed in the setting area, the photographing units 110 and 120 may photograph or scan the bin 80 . In the bin 80 of FIG. 2, three objects ①, ②, and ③ are arranged. In this case, the setting area may include a bottom surface of the bin 80 in which the object 90 is accommodated or supported.

The photographing unit may have a single photographing unit or a plurality of photographing units in consideration of scanning (photographing) capability, initial installation cost, and the like.

As an example, the first photographing unit 110 installed on the support 109 disposed to face the setting area and photographing the setting area, and the object 90 between the first photographing unit 110 and the setting area. A second photographing unit 120 may be provided.

Each of the first photographing unit 110 and the second photographing unit 120 may be provided as an RGB-D sensor, a stereo camera, a time of flight (TOF) sensor, an ultrasonic sensor, or the like. Here, the RGB-D sensor may sense an RGB (color) two-dimensional image and depth information (Depth). A stereo camera can sense image information and depth information by arranging two image sensors in parallel. Here, the RGB-D sensor, the stereo camera, the TOF sensor, and the ultrasonic sensor are only examples, and the first photographing unit 110 and the second photographing unit 120 may be composed of various types of sensors or combinations thereof. am. In some cases, the first photographing unit 110 and the second photographing unit 120 may each be provided as a single image sensor to photograph only a two-dimensional image of the object 90 .

The first photographing unit 110 may be installed at a first position p1 on one surface of the support 109 facing the setting area.

The scan result of the first photographing unit 110 may be used to detect the object 90 disposed in the setting area or to select a candidate to be picked from among the plurality of objects 90 .

The second photographing unit 120 may be installed at a second position p2 between the first position p1 and the setting area in the direction of gravity. For example, the second photographing unit 120 may be provided on the arm 11 of the robot 10 moving between the first position p1 and the setting area.

When the second photographing unit 120 is formed separately from the robot 10 , a detachable unit 180 for mounting the second photographing unit 120 to the robot 10 may be additionally provided. For example, the detachable part 180 may be formed in a band shape surrounding the arm 11 of the robot 10 . In a state in which the arm 11 of the robot 10 is wrapped, both ends of the detachable part 180 are disposed to face each other and may be fastened to each other by a fastening means 189 such as a bolt.

The scan result of the second photographing unit 120 may be used to estimate the posture of the object 90 selected as a picking candidate, and to designate a point at which picking is to be performed on the object 90 .

An effector 20 (end effector) for picking an object 90 may be provided at an end of the arm 11 of the robot 10 .

The effector 20 may include a suction unit for adsorbing and lifting the object 90 using air suction. As another example, the effector 20 may include a plurality of fingers each gripping one side and the other side of the object 90 . When the plurality of fingers are tightened in a direction approaching each other, the object 90 disposed between the plurality of fingers may be grabbed by the fingers.

When a picking candidate is selected by analyzing the scan result of the first photographing unit 110, the robot 10 in which the effector 20 is disposed at the initial position, the effector 20 is positioned at the passing point pr facing the picking candidate. can move When the effector 20 is located at the passing point pr, the posture or the picking point of the picking candidate may be analyzed by analyzing the scan result of the second camera. The robot 10 may move the effector 20 to take a posture with the highest probability of picking a picking candidate corresponding to the posture of the picking candidate. Alternatively, the robot 10 may move the effector 20 toward a picking point having the highest probability of picking a picking candidate.

As another example, when only one of the first photographing unit 110 and the second photographing unit 120 is provided, a picking candidate may be selected using a single photographing unit, and a posture and a picking point of the picking candidate may be determined.

The picking priority selection apparatus shown in FIG. 1 selects a picking candidate to be picked by the effector 20 from among a plurality of objects 90 arranged in the setting area, or selects the picking priority of each object 90. have.

The acquisition unit 210 may acquire the scan result of the setting area from the imaging units 110 and 120 .

The acquisition unit 210 may additionally acquire depth information of the object 90 from the photographing units 110 and 120 when the photographing units 110 and 120 can additionally measure the depth information of the object 90 . have.

The picking order selection unit 270 may analyze a plurality of pieces of environment information of each object 90 through the scan result. As an example, the picking ranking selecting unit 270 may include an interference degree indicating a rate at which each object 90 is interfered with by another object, the height of each object 90, the masking area width of each object 90, each object ( 90) and the center of a region of interest (ROI) may be used to select the picking priority of each object 90 using at least one of the distances. In this case, the corresponding interference degree, the height of the object 90 , the masking area area, the distance between the center of the object 90 and the center of the ROI, etc. may correspond to the environment information analyzed from the scan result. Here, the degree of interference, the height, and the distance between the center of the object 90 and the center of the ROI as examples of the above-described environment information are merely examples, and various kinds of factors may be added or changed. Here, the height of the object 90 is measured from the bottom surface of the bin 80 in a direction opposite to the direction of gravity, and may be calculated from depth information obtained from the above-described photographing units 110 and 120 .

The picking order selection unit 270 may normalize a plurality of pieces of environment information and calculate cumulative normal distribution function scores for the plurality of pieces of environment information. The picking ranking selecting unit 270 may assign a predetermined weight to each environment information and calculate a total score of each environment information in consideration of the weight. That is, the selector 270 may tune the corresponding environmental information by assigning a weight in consideration of the importance of the corresponding environmental information to each environmental information.

The picking priority selection unit 270 may select a picking priority to be picked preferentially among the plurality of objects 90 by comparing the calculated overall scores for the plurality of objects 90 . As an example, the picking order selection unit 270 may give priority in order of the highest overall score.

The analysis unit 250 may include a first analysis unit 251 , a second analysis unit 252 , a third analysis unit 253 , and a fourth analysis unit 254 . Here, it has been described that the analysis unit 250 includes all of the first to fourth analysis means 251 , 252 , 253 , and 254 , but any one of them may be omitted. The analysis unit 250 may include at least one of the first to fourth analysis means 251 , 252 , 253 , and 254 .

The first analysis means 251 may analyze the area of the masking area of the object 90 .

The first analysis means 251 may analyze the scan result using the deep learning model, estimate the area of the object 90, and mask the area.

The picking order selection unit 270 may select a picking candidate using the area of the masking area of the object 90 calculated by the first analysis means 251 . In the case of the effector 20 provided with a suction unit for sucking air and adsorbing the object 90, in order for the picking to be performed successfully, the exposed portion of the object 90 must be sufficiently exposed to be adsorbed by the suction unit. do. In this case, the area of the object 90 exposed to the suction unit may correspond to the area of the masking area. The area of the masking area of the specific object to be picked may be reduced when another object is stacked on the corresponding object 90 . That is, the wide masking area of the object 90 means that the degree of being covered by other objects is small.

The width of the masking area may be variously recognized.

For example, the acquisition unit 210 may acquire an RGB (red, green, blue) image from the photographing units 110 and 120 , and the like. The first analysis means 251 may extract coordinate information of a segmentation pixel of the object 90 by using a deep learning model targeting an RGB image. The first analysis means 251 may calculate the sum of the number of divided pixels of the object 90 as the masking area area of the object 90 . The picking order selection unit 270 may select a picking candidate by using the masking area width.

3 is a schematic diagram illustrating a process of calculating an interference degree.

An interference degree indicating a rate at which a specific object existing in a bin 80 corresponding to the setting area g3 is interfered with by other objects existing in the setting area g3 may be defined.

The second analysis means 252 may analyze the degree of interference with respect to the object 90 . The second analysis means 252 may analyze the degree of interference by using the first depth information of g1 inside the masking area and the second depth information of g2 outside the masking area. Since the first depth information and the second depth information are based on the masking area, the calculation result of the first analysis means 251 for analyzing the masking area or its area in the interference analysis process may be additionally used. The picking order selection unit 270 may select a picking candidate using the degree of interference.

In order to obtain the first depth information or the second depth information, the acquisition unit 210 obtains, as described above, depth information on the object 90 from a Time-Of-Flight (ToF) camera, a distance measurer, a vision, and the like. can be obtained In summary, for the analysis of the degree of interference, the acquisition unit 210 may acquire both RGB image information and depth information as a scan result. Depth information may be provided in the form of a two-dimensional image, and depth information in the form of a two-dimensional image will be referred to as a depth image.

The detector 230 may detect a specific object existing in the setting area using at least one of RGB image information and a depth image.

When a specific object is detected, the picking order selector 270 may extract a second area from among the set areas except for the first area in which the specific object is disposed by using the RGB image.

The picking priority selection unit 270 generates a peripheral depth information map based on a specific object through analysis of only the corresponding area of the second area in the depth image, except for the corresponding area of the first area in the depth image. can

The picking order selection unit 270 may calculate a degree of interference around a specific target by using a depth information map based on the specific target, and select a picking candidate using the degree of interference. For example, the picking priority selection unit 270 may select the picking priority according to the order of the low degree of interference. When the degree of interference for a specific object is low, the specific object is removed from the setting area using the robot 10 It's easy to bet on, and it can mean that it's more likely that other objects won't fluctuate if you lift it. If another object collapses and is changed to a different state during the picking process of a specific object, the picking priority selected for the plurality of objects 90 at the current time point may become useless. Therefore, it is advantageous that the picking operation is performed from the object 90 having a low degree of interference.

The first analysis means 251 masks the area of the object 90 using a deep learning model targeting the RGB image to extract the coordinate information of the segmentation pixel of the area in which the object 90 is segmented. can The second analyzing means 252 may calculate the degree of interference of another object with respect to a specific object by using the pixel coordinate information and the depth information of the segmentation area extracted by the first analyzing means 251 .

The second analysis means 252 adds a margin necessary for the entry of the effector 20 of the robot 10 picking the object 90 to the masking area of the object 90 corresponding to the set of coordinate information, and the object 90 ) to set the interference check area. This is because, according to the type of the effector 20 , the free space required for picking the object 90 is different. For example, if the effector 20 is in the shape of a finger (finger), since the finger has to pinch the edge of the object 90, a lot of free space without interference is required around the object 90, so that the interference check area is On the other hand, when the effector 20 pneumatically sucks the central portion of the object 90 , the free space required for picking the object 90 is relatively small, so the interference check area may be small. That is, the interference check area may vary according to the type and type of the effector 20 .

6 is a schematic diagram illustrating an interference degree considering an interference check area.

6A is a plan view illustrating the interference check area ch, and FIG. 6B is a side view illustrating the interference check area ch.

In the case of the finger-type effector 20 , the interference check region ch may include a region in which the thickness g9 of the finger and the entry distance g8 are added to the outer peripheral surface or the periphery of the masking region g1 . For example, a range or region satisfying within the thickness g9 of the finger and within the entry distance g8 may correspond to the interference check area ch.

In order to reliably pick up the object 90 using a plurality of fingers, the corresponding finger must enter a set distance from the upper end of the object 90 toward the lower end of the object 90, and at this time, the set distance corresponds to the entry distance g8 can be

If another object is invading the interference check area to which the thickness g9 of the finger and the entry distance g8 are added, it may be determined that the picking operation is interfered with the corresponding object by the other object. In this case, some objects disposed in the setting area and being picked may function as obstacles 99 that prevent picking of a specific object. It is preferable that the picking apparatus 100 picks the object 90 while avoiding the obstacle 99 .

The second analysis means 252 may calculate the degree of interference according to the intrusion ratio of the other object to the interference check area. For example, when the area of the interference check area ch is 100, when the obstacle 99 occupies the area 10 of the interference check area ch, the invasion rate may be calculated as 10%.

The third analysis means 253 may analyze the top height of each of the plurality of objects 90 included in the depth image. The upper end of the object 90 may represent the highest portion of the object 90 in the direction of gravity.

The picking ranking selecting unit 270 may select a picking candidate by using an upper height corresponding to the height of the environment information. For example, the picking priority selecting unit 270 may select the picking priority in the order of the objects 90 disposed at a high position using the top height. Even if a specific object placed in the highest position is lifted, other objects below it can maintain their current state without change. In addition, the picking operation of the highest specific object can also be smoothly performed.

The height of the upper end of the object 90 may be specifically analyzed through the following process.

The picking ranking selecting unit 270 uses a deep learning model targeting an RGB image to extract the coordinate information of the segmentation pixel of the segmented area of the object 90 by using the first analysis means 251 and extraction A third analysis means 253 for classifying the area of the object 90 on the depth image by using the coordinate information may be included.

The third analysis means 253 may extract a depth value of the highest uppermost point in the area of the object 90 . The picking order selection unit 270 may select a picking candidate by using the depth value of the highest point. That is, the third analyzing means 253 may extract the pixel of the highest uppermost point and the depth value of the corresponding pixel by using the depth values of the pixel coordinates of the segmentation region of the object. The selection unit 270 compares the depth value of the highest upper point of each of the plurality of objects 90 provided by the third analysis means 253 and prioritizes the object 90 corresponding to the highest upper point among them. It can be selected as a high ??king-floor.

4 is a schematic diagram illustrating a distance of interest.

The fourth analysis means 254 may acquire central information of a region of interest (ROI) that can be picked by the effector 20 of the robot 10 and central information of the object 90 .

The fourth analysis means 254 may measure or calculate the distance between the centers of the respective objects 90 from the midpoint o of the region of interest (ROI) as the distance of interest. The picking ranking selecting unit 270 may select a picking candidate using the distance of interest. That is, the distance of interest means a distance between the midpoint o of the ROI and the center of each object 90 . The midpoint o of the ROI may correspond to the central point of the RGB image area captured by the first imaging unit 110 or the second imaging unit 120 described above.

For example, the 2D image area photographed by the second photographing unit 120 mounted on the robot 10 may be smaller than the 2D image area photographed by the first photographing unit 110 . This may occur when the focal length, field of view, and the like of the lenses used in the first photographing unit 110 and the second photographing unit 120 are different. In this case, the region photographed by the second imaging unit 120 having a relatively small two-dimensional image region may be used as the region of interest (ROI). When the two-dimensional image area is small, high-speed calculation may be possible because the amount of calculation for calculating the distance of interest is relatively small.

Meanwhile, the distance of interest related to the region of interest (ROI) may be related to the distance between the effector 20 installed at the end of the robot 10 and the object 90 . That is, as the distance of interest is shorter, it may mean that the distance between the effector 20 and the corresponding object 90 is shorter. Accordingly, in order to shorten the total working time by minimizing the movement time for the effector 20 to move to the object 90 , picking candidates may be selected in the order of the shortest distance of interest or the picking priority may be given. Accordingly, in order to minimize the movement time, the distance of interest may be calculated based on the image captured by the second photographing unit 120 attached to the arm of the robot 10 .

The distance of interest can be specifically analyzed as follows.

Coordinate information of segmentation pixels of the object 90 may be extracted from the image photographed by the second photographing unit 120 by the first analysis means 251 using deep learning.

The fourth analysis means 254 may obtain the midpoint coordinate o of a region of interest (ROI) from the image captured by the second photographing unit 120 .

The fourth analysis means 254 may extract the coordinates w of the center pixel disposed at the center of each object 90 among a plurality of segmentation pixels forming each object 90 . The fourth analysis means 254 may calculate a distance of interest U corresponding to a distance between the coordinates o of the center point of the region of interest (ROI) and the coordinates w of the center pixel, and select a picking candidate using the distance of interest. That is, the picking priority may be determined so as to preferentially pick the object 90 corresponding to the central pixel having a small distance of interest.

When the detection unit 230 recognizes the object 90 in the image taken using the bounding box method, the fourth analysis means is the inner center of the polygon-shaped bounding box BB and the region of interest (ROI). The distance of the focal point can also be set as the distance of interest. In the case of FIG. 4 , there are four objects 90 , and if an object existing on the midpoint of the region of interest (ROI) is excluded, three bounding boxes may exist. Among the three bounding boxes, picking priorities 0, 1, and 2 may be given in the order of the nearest U of interest.

The picking order selection unit 270 may acquire information about the effector 20 related to the effector 20 . The picking priority selecting unit 270 may correct the picking priority selected by the above-described various environmental information according to the effector 20 information.

As an example, the picking order selection unit 270 may set weights for a plurality of pieces of environment information based on the effector 20 information. When the type of the effector 20 is changed, the weight may be changed by the picking priority selector 270 , and the priority may be changed due to application of the changed weight.

If the picking order selection unit 270 is a finger type in which the effector 20 picks up the object 90 by tightening a plurality of fingers disposed to face each other with the object 90 interposed therebetween, the degree of ambient interference among the plurality of environment information can be set to the highest weight. This is because, if the degree of ambient interference is not guaranteed, the effector 20 cannot pick up the object 90 itself.

The picking priority selection unit 270 may select the picking priority by using the evaluation result for each environment information to which the corresponding weight is reflected.

As an example, the picking priority selector 270 may independently evaluate the masking area area (as described above, the sum of the number of pixels of the divided pixels) and the surrounding interference degree for each object 90 .

When the picking order selection unit 270 includes a suction unit that the effector 20 absorbs and lifts the object 90 using air pressure (pneumatic) suction, the weight for the area of the object 90 among a plurality of environmental information can be set to be larger than the interference degree. This is because even in the presence of some interference, if an adsorption surface of a sufficient area is ensured, it is possible to pick up the object 90 in spite of the surrounding interference.

The picking priority selection unit 270 may select the priority by using the evaluation result for each environment information to which the weight is reflected.

5 is a schematic diagram illustrating adjustment of a weight according to the type of the effector 20 .

5 is environmental information involved in the selection of the picking priority, and the height A, the ambient interference degree B, the area (masking area area) C, and the distance of interest D can be analyzed by the analysis unit 250 for each object 90. have.

Here, as described above, each of the objects 90 is those detected by the detector 230 in the image captured by the first photographing unit 110 or the second photographing unit 120 .

As height A increases, the score of the picking priority may increase. That is, the higher the object 90 that exists in the bin 80, the more preferentially it can be picked.

The lower the degree of ambient interference B, the higher the priority score may be. That is, as the interference caused by other nearby objects is smaller, the corresponding object 90 may be preferentially picked.

As the area C increases, the priority score may increase. That is, the larger the size of the object 90 (to be precise, the area of the divided pixel area occupied in the photographed image) is, the more preferentially the picking can be.

The shorter the distance D of interest, the higher the priority score may be. That is, the object 90 located closer to the arm of the robot 10 can be picked preferentially.

Figure 5 (a) shows a general weight. When the type of the effector 20 is set to normal, the picking order selection unit 270 may adjust all the weights of each environment information to 1. It corresponds to the weight corresponding to "general type" in [Table 1] to be described later.

FIG. 5B shows weights for the finger-type effector 20 . It corresponds to a weight corresponding to "finger type" in [Table 1] to be described later. When the type of the effector 20 is set to the finger type, the picking order selection unit 270 may adjust the weight of the ambient interference degree B to 2, which is the highest. The picking order selection unit 270 may adjust the weight of the height A to 1 and the weights of the area C and the distance of interest D to 0. It can be seen that the degree of ambient interference, which is an important factor for the finger-type effector 20 to pick up the object 90, is most reflected.

5 (c) shows the weight of the effector 20 of the suction type. It corresponds to the weight corresponding to the "suction type" in [Table 1] to be described later. When the type of the effector 20 is set to the suction type, the picking order selection unit 270 may adjust the weights of the height A and the width C to 2, and adjust the weights of the ambient interference degree B and the distance of interest D to 0. . It can be seen that the width, which is the most important factor for the suction-type effector 20 to pick up the object 90, is reflected larger than the surrounding interference degree.

The weights described in (a), (b), and (c) of FIG. 5 above are only examples, and the weights can be variously changed depending on the type of the object 90 , the shape of the effector 20 , and the like.

Table 1 shows the example scores in the state in which the weight of FIG. 5 is applied.

Table 1 may target the three objects ①, ②, ③ arranged in FIG.

Object (90) identification number A: height B: Ambient Interference Degree C: width D: point of interest general type
(A+B+C+D) finger type
(2B+A) Suction type
(2A+2C) ① 60 60 30 80 230 180 180 ② 60 90 40 10 200 240 200 ③ 90 10 60 10 170 110 300

In Table 1, A, B, C, and D may represent scores evaluated by the analysis unit 250 . When the finger type is changed in the situation where the environmental information analysis scores of the objects ①, ②, and ③ of the objects ①, ②, and ③ by the analysis unit 250 are changed as shown in Table 1, the picking priority may be changed as shown in FIG. 5 by adjusting the weight. If the effector 20 corresponds to the general type, the picking priority may be given in the order of the first object ①, the second object ②, and the third object ③.

If the effector 20 corresponds to the finger type, the picking priority may be given in the order of the second object ②, the first object ①, and the third object ③.

If the effector 20 corresponds to the suction type, the picking priority may be given in the order of the third object ③, the second object ②, and the third object ③.

7 is a flowchart illustrating a method for selecting a picking priority according to an embodiment of the present invention.

The method for selecting a picking priority of FIG. 7 may be performed by the apparatus for selecting a picking priority shown in FIG. 1 . The picking priority selection apparatus may select a picking priority for a plurality of objects 90 picked by the effector 20 of the robot 10 .

The picking priority selection method of the present invention may include a scan step (S510), an analysis step (S520), a calculation step (S530), and a selection step (S540).

In the scanning step ( S510 ), a setting area in which the plurality of objects 90 are disposed may be scanned. The scanning step S 510 may be an operation performed by the imaging units 110 and 120 or the acquisition unit 210 .

In the analysis step ( S520 ), a corresponding scan result may be analyzed, and a plurality of environmental information may be analyzed for each object 90 . The analysis step ( S520 ) may be performed by the analysis unit 250 .

In the calculation step ( S530 ), a total score may be calculated by applying different weights for each environmental information according to the type of the effector 20 . Here, the calculation step may be performed by the above-described analysis unit 250 .

In the selection step ( S540 ), the picking priority of the plurality of objects 90 may be selected using the overall score. The calculation step ( S530 ) and the selection step ( S540 ) may be performed by the picking rank selection unit 270 .

The picking priority selected through the selection step may be provided to the robot 10 for picking the object 90 .

According to the picking priority selection method of the present invention, if the effector 20 is a finger type, the picking priority of the object 90 may be selected by applying the highest weight to the degree of interference among a plurality of environmental information. According to the picking priority selection method of the present invention, if the effector 20 is a suction type, the picking priority of the object 90 may be selected by applying the highest weight to the masking area among the plurality of environmental information.

8 is a diagram illustrating a computing device according to an embodiment of the present invention. The computing device TN100 of FIG. 8 may be a device (eg, a picking priority selection device, etc.) described herein. In the embodiment of FIG. 8 , the computing device TN100 may include at least one processor TN110 , a transceiver device TN120 , and a memory TN130 . Also, the computing device TN100 may further include a storage device TN140 , an input interface device TN150 , an output interface device TN160 , and the like. Components included in the computing device TN100 may be connected by a bus TN170 to communicate with each other.

The processor TN110 may execute a program command stored in at least one of the memory TN130 and the storage device TN140. The processor TN110 may mean a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor on which methods according to an embodiment of the present invention are performed. The processor TN110 may be configured to implement procedures, functions, and methods described in connection with an embodiment of the present invention. The processor TN110 may control each component of the computing device TN100 .

Each of the memory TN130 and the storage device TN140 may store various information related to the operation of the processor TN110 . Each of the memory TN130 and the storage device TN140 may be configured as at least one of a volatile storage medium and a non-volatile storage medium. For example, the memory TN130 may include at least one of a read only memory (ROM) and a random access memory (RAM).

The transceiver TN120 may transmit or receive a wired signal or a wireless signal. The transceiver TN120 may be connected to a network to perform communication.

On the other hand, the various methods according to the embodiment of the present invention described above may be implemented in the form of a program readable by various computer means and recorded in a computer readable recording medium. Here, the recording medium may include a program command, a data file, a data structure, etc. alone or in combination. The program instructions recorded on the recording medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. For example, the recording medium includes magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floppy disks ( magneto-optical media), and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions may include not only machine language wires such as those created by a compiler, but also high-level language wires that can be executed by a computer using an interpreter or the like. Such hardware devices may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

In the above, although an embodiment of the present invention has been described, those of ordinary skill in the art can add, change, delete or add components within the scope that does not depart from the spirit of the present invention described in the claims. The present invention may be variously modified and changed by, etc., and this will also be included within the scope of the present invention.

10...Robot 11...arm
20...effector 80...bin
90...object 99...obstacle
100...picking device 110, 120...shooting unit
110... 1st filming unit 109... Support
120...Second Filming Division 210...Acquisition Division
230...detection unit 250...analysis unit
251...first analysis means 252...second analysis means
253...third analysis means 254...fourth analysis means
270...Picking ranking selection unit

Claims (15)

an acquisition unit configured to acquire a two-dimensional image and depth information for a setting area on which a plurality of picking objects are placed;
a detector configured to detect the object using the two-dimensional image and the depth information;
an analysis unit that analyzes the two-dimensional image and the object detected in the depth information and calculates at least two pieces of environment information for each object; and
a picking priority selection unit for selecting a picking priority for selecting an object to be picked preferentially from among the plurality of picking objects by applying a predetermined weight to the at least two or more pieces of environment information; and
If the height, ambient interference degree, area, and distance of interest are analyzed for each of the plurality of picking objects, and the effector for picking the picking object is set as a finger type,
The picking ranking selecting unit adjusts the weight of the ambient interference degree to the highest, adjusts the weight of the area to the lowest, and adjusts the weight of the height between the weight of the ambient interference degree and the weight of the area. Picking prioritization device.
According to claim 1,
The environmental information includes at least two of an interference degree indicating a rate at which the object is interfered with by another object, a height of a position where the object is placed, an area of a masking area of the object, and a distance of interest between the center of the object and the center of the two-dimensional image. Picking priority selection device, characterized in that it includes more than one.
delete delete 3. The method of claim 2,
The analysis unit extracts coordinate information of segmentation pixels of the object using a deep learning model targeting the two-dimensional image,
The analysis unit sets the interference check area of the object by adding a margin necessary for the entry of the effector of the robot picking the object to the masking area of the object corresponding to the set of coordinate information,
Picking priority selection device, characterized in that the analysis unit calculates the degree of interference according to a ratio of a specific object to the interference check area of another object intrusion.
3. The method of claim 2,
The analysis unit extracts coordinate information of segmentation pixels of the object using a deep learning model targeting the two-dimensional image,
The analysis unit divides the area of the object by using the coordinate information,
Picking priority selection device, characterized in that the analysis unit extracts the depth value of the highest point corresponding to the height in the area of the object.
3. The method of claim 2,
The analysis unit extracts coordinate information of segmentation pixels of the object using a deep learning model targeting the two-dimensional image,
The analysis unit extracts the coordinates of the central pixel disposed in the center of the plurality of divided pixels forming the object,
and the analysis unit calculates the distance of interest corresponding to a distance between the coordinates of a central point of a region of interest (ROI) set in a camera for capturing the set region and the coordinates of the central pixel.
A robot having an arm and an effector installed on the arm, the robot comprising:
A picking priority selection device according to any one of claims 1 to 2, 5 to 7; and
A control device for controlling the arm and the effector to pick an object according to the picking priority received from the picking priority selection device
A robot comprising a.
In the picking priority selection method for selecting the picking priority of a plurality of picking objects,
acquiring a two-dimensional image and depth information for a setting area in which the plurality of picking objects are placed;
detecting the object using the two-dimensional image and the depth information;
calculating at least two pieces of environmental information for each object by analyzing the two-dimensional image and the object detected in the depth information; and
selecting a picking priority for selecting an object to be picked preferentially from among the plurality of picking objects by applying a predetermined weight to the at least two or more pieces of environmental information;
If the height, ambient interference degree, area, and distance of interest are analyzed for each of the plurality of picking objects, and the effector for picking the picking object is set as a finger type,
The step of selecting the picking priority includes adjusting the weight of the peripheral interference degree to the highest, adjusting the weight of the area to the lowest, and adjusting the weight of the height between the weight of the ambient interference degree and the weight of the area do,
When the height, ambient interference degree, area, and distance of interest are analyzed for each of the plurality of picking objects, and the effector for picking the picking object is set as a suction type,
The selecting of the picking priority comprises not using the ambient interference degree and the distance of interest, and adjusting the weight to use the height and the width.
10. The method of claim 9,
The environmental information includes at least two of an interference degree indicating a rate at which the object is interfered with by another object, a height of a position where the object is placed, an area of a masking area of the object, and a distance of interest between the center of the object and the center of the two-dimensional image. Picking priority selection method, characterized in that it includes more than one.
delete delete 11. The method of claim 10,
Extracting coordinate information of segmentation pixels of the object using a deep learning model targeting the two-dimensional image,
Setting the interference check area of the object by adding a margin necessary for the entry of the effector of the robot picking the object to the masking area of the object corresponding to the set of coordinate information,
A picking priority selection method, characterized in that the interference degree is calculated according to a ratio of a specific object's intrusion into the interference check area of another object.
11. The method of claim 10,
Extracting coordinate information of segmentation pixels of the object using a deep learning model targeting the two-dimensional image,
Classifying the area of the object using the coordinate information,
Picking priority selection method, characterized in that extracting the depth value of the highest point corresponding to the height in the area of the object.
11. The method of claim 10,
Extracting coordinate information of segmentation pixels of the object using a deep learning model targeting the two-dimensional image,
extracting the coordinates of the central pixel disposed in the center among the plurality of divided pixels forming the object,
Picking priority selection method, characterized in that calculating the distance of interest corresponding to the distance between the coordinates of the center point of the ROI (Region Of Interest) set in the camera for photographing the set area and the coordinates of the center pixel.

Images