KR20230032852A - Robot and controlling method of robot - Google Patents

Abstract

A robot and a method for controlling the same are disclosed. A robot according to the present disclosure comprises: a manipulator; a plurality of motors for controlling the operation of the manipulator; a memory storing at least one instruction; and a processor to execute the at least one instruction. In addition, the processor may obtain an image of an object through a camera, obtain, based on the obtained image, point cloud information including a plurality of points indicating the object, identify, among the plurality of points, a plurality of edge points indicating the edge of the object and a plurality of body points indicating a body region excluding the edge of the object, obtain a plurality of pose values indicating a pose when the manipulator grips the object by using at least some points among the plurality of edge points and the plurality of body points as griping points, respectively, obtain evaluation information indicating stability when the manipulator grips the object according to the plurality of pose values, identify, based on the evaluation information, one pose value among the plurality of pose values, and control, based on the identified one pose value, the plurality of motors so that the manipulator operates. To this end, a robot capable of stably and efficiently gripping an object and a method for controlling the same are provided.

Description

Robot and control method of robot {ROBOT AND CONTROLLING METHOD OF ROBOT}

The present disclosure relates to a robot and a method for controlling the robot, and more specifically, to a robot capable of gripping an object using a manipulator and a method for controlling the robot.

Recently, thanks to the development of robot technology, various types of robots such as cleaning robots, service robots, and industrial robots are being used. , research on manipulators is being actively conducted. A technology related to gripping an object using a robot manipulator can be said to be the most basic and highly utilized technology in a technology field related to a manipulator.

In gripping an object using a manipulator, the prior art acquires an image of the object by photographing the object at 360 degrees using a camera, or relies on deep learning technology to increase the gripping rate of the object. Currently, a method for analyzing an image is being used.

However, when an object is photographed at 360 degrees using a camera according to the prior art, there is a limit in that it takes a lot of time to acquire an image of the object.

On the other hand, in the case of using a method of analyzing an image of an object relying on deep learning technology, not only does it take a lot of time in the process of analyzing the image by deep learning, but also requires a large amount of training data for learning the neural network model. In addition, the limitation that high-performance hardware such as GPU (graphics processing unit) is required is pointed out.

The present disclosure is to overcome the limitations of the prior art as described above, and an object of the present disclosure is to acquire an image of an object and geometrically analyze the obtained image, thereby enabling a robot capable of stably and efficiently gripping an object. And to provide a control method of the robot.

According to an embodiment of the present disclosure for achieving the above object, the robot includes a camera, a manipulator, a plurality of motors for controlling the operation of the manipulator, and a memory for storing at least one instruction. And a processor that executes the at least one instruction, wherein the processor obtains an image of an object through the camera, and point cloud information including a plurality of points representing the object based on the obtained image ( point cloud information) is acquired, and among the plurality of points, a plurality of edge points representing an edge of the object and a plurality of body points representing a body area excluding the edge of the object are identified, and the manipulator acquires a plurality of pose values representing poses when gripping the object using at least some of the plurality of edge points and the plurality of body points as gripping points, and the manipulator according to the plurality of pose values obtains evaluation information indicating stability when holding the object, identifies one pose value among the plurality of pose values based on the evaluation information, and determines that the manipulator operates based on the identified one pose value. Control the plurality of motors to operate.

Here, the processor identifies, for each of the plurality of points, a zero moment shift (ZMS) vector directed from the plurality of points to the center point of points included in a circle having a predetermined size centered on the plurality of points, and Points corresponding to vectors having a size of the ZMS vector greater than or equal to a preset threshold among points of are identified as the edge points, and among the plurality of points, points corresponding to vectors having a size of the ZMS vector less than the threshold are identified. It can be identified as the body point.

Here, the processor determines the one point when an angle between a ZMS vector for one point identified as the edge point and a ray vector directed from the camera to the one point is equal to or greater than a preset threshold angle. It can be identified as a body point.

Here, the processor causes the z-axis of the manipulator, which is defined to face in a direction opposite to the direction toward the forearm joint of the manipulator, to correspond to the direction of the ZMS vector with respect to the identified edge point, and the one included in the manipulator By identifying the pose of the manipulator so that the y-axis of the manipulator defined from a finger to the other finger in the opposite direction corresponds to a direction perpendicular to the surface of the identified edge point, the plurality of pose values Pose values corresponding to the edge points of can be obtained.

Here, the processor identifies a pose of the manipulator so that the z-axis corresponds to a direction perpendicular to the surface of the plurality of points and the y-axis corresponds to a direction of greatest curvature at the plurality of points. , Among the plurality of pose values, pose values corresponding to the plurality of body points may be obtained.

Here, the processor may augment the plurality of pose values by changing some coordinate axes of pose values corresponding to the plurality of edge points and pose values corresponding to the plurality of body points.

Here, the evaluation information is a first score calculated according to an angle between a direction perpendicular to the surface of two fingers included in the manipulator and a direction perpendicular to the surface of a contact area where the two fingers come into contact with the object; A second score calculated according to the number of points included in the contact area, a distance between one of the two fingers and the contact area, and a distance between another one of the two fingers and the contact area It may include at least one of a third score calculated according to the difference between and a fourth score calculated according to whether the at least some points are close to the center of gravity of the object.

Here, the processor may obtain evaluation information for each of the plurality of pose values by calculating a final score based on a weighted sum of the first score, the second score, the third score, and the fourth score. there is.

Here, the processor obtains information on a plurality of estimated points corresponding to a non-observation area included in the image based on the ZMS vector and the ray vector for the identified edge point, and obtains surface area of the two fingers The contact area may be identified based on the information about, the plurality of edge points of the manipulator, the plurality of body points, and the plurality of estimated points.

According to an embodiment of the present disclosure for achieving the above object, a control method of a robot including a manipulator and a plurality of motors for controlling an operation of the manipulator includes acquiring an image of an object. Obtaining point cloud information including a plurality of points representing the object based on the obtained image, a plurality of edge points representing the edge of the object among the plurality of points, and Identifying a plurality of body points representing a body area excluding an edge of the object, wherein the manipulator uses at least some of the plurality of edge points and the plurality of body points as gripping points to hold the object obtaining a plurality of pose values representing poses when gripping the object; obtaining evaluation information representing stability when the manipulator grips the object according to the plurality of pose values; The method includes identifying one pose value among a plurality of pose values and controlling the plurality of motors to operate the manipulator based on the identified pose value.

1 is a diagram showing a robot according to an embodiment of the present disclosure;
2 is a block diagram briefly showing the configuration of a robot according to an embodiment of the present disclosure;
3 is a diagram for explaining a process of identifying an edge point and a body point according to an embodiment of the present disclosure;
4 is a diagram for explaining a process of distinguishing a true edge point and a false edge point according to an embodiment of the present disclosure;
5 is a diagram for explaining a process of estimating a non-observation area according to an embodiment of the present disclosure;
6 to 8 are diagrams for explaining a process of obtaining a plurality of pose values according to an embodiment of the present disclosure;
9 and 10 are diagrams for explaining in detail evaluation information according to an embodiment of the present disclosure;
11 is a block diagram showing in detail the configuration of a robot according to an embodiment of the present disclosure, and
12 is a flowchart illustrating a method for controlling a robot according to an embodiment of the present disclosure.

Since the present embodiments can apply various transformations and have various embodiments, specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the scope to the specific embodiments, and should be understood to include various modifications, equivalents, and/or alternatives of the embodiments of the present disclosure. In connection with the description of the drawings, like reference numerals may be used for like elements.

In describing the present disclosure, if it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the gist of the present disclosure, a detailed description thereof will be omitted.

In addition, the following embodiments may be modified in many different forms, and the scope of the technical idea of the present disclosure is not limited to the following embodiments. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the spirit of the disclosure to those skilled in the art.

Terms used in this disclosure are only used to describe specific embodiments, and are not intended to limit the scope of rights. Singular expressions include plural expressions unless the context clearly dictates otherwise.

In the present disclosure, expressions such as “has,” “can have,” “includes,” or “can include” indicate the presence of a corresponding feature (eg, numerical value, function, operation, or component such as a part). , which does not preclude the existence of additional features.

In this disclosure, expressions such as “A or B,” “at least one of A and/and B,” or “one or more of A or/and B” may include all possible combinations of the items listed together. . For example, “A or B,” “at least one of A and B,” or “at least one of A or B” (1) includes at least one A, (2) includes at least one B, Or (3) may refer to all cases including at least one A and at least one B.

Expressions such as "first," "second," "first," or "second," used in the present disclosure may modify various elements regardless of order and/or importance, and may refer to one element as It is used only to distinguish it from other components and does not limit the corresponding components.

A component (e.g., a first component) is "(operatively or communicatively) coupled with/to" another component (e.g., a second component); When referred to as "connected to", it should be understood that the certain component may be directly connected to the other component or connected through another component (eg, a third component).

On the other hand, when an element (eg, a first element) is referred to as being “directly connected” or “directly connected” to another element (eg, a second element), the element and the above It may be understood that other components (eg, a third component) do not exist between the other components.

The expression “configured to (or configured to)” as used in this disclosure means, depending on the situation, for example, “suitable for,” “having the capacity to.” ," "designed to," "adapted to," "made to," or "capable of." The term "configured (or set) to" may not necessarily mean only "specifically designed to" hardware.

Instead, in some contexts, the phrase "device configured to" may mean that the device is "capable of" in conjunction with other devices or components. For example, the phrase "a processor configured (or configured) to perform A, B, and C" may include a dedicated processor (eg, embedded processor) to perform the operation, or by executing one or more software programs stored in a memory device. , may mean a general-purpose processor (eg, CPU or application processor) capable of performing corresponding operations.

In an embodiment, a 'module' or 'unit' performs at least one function or operation, and may be implemented with hardware or software, or a combination of hardware and software. In addition, a plurality of 'modules' or a plurality of 'units' may be integrated into at least one module and implemented by at least one processor, except for 'modules' or 'units' that need to be implemented with specific hardware.

Meanwhile, various elements and regions in the drawings are schematically drawn. Therefore, the technical spirit of the present invention is not limited by the relative size or spacing drawn in the accompanying drawings.

Hereinafter, with reference to the accompanying drawings, an embodiment according to the present disclosure will be described in detail so that those skilled in the art can easily implement it.

1 is a diagram showing a robot 100 according to an embodiment of the present disclosure, and FIG. 2 is a block diagram briefly showing the configuration of the robot 100 according to an embodiment of the present disclosure.

The robot 100 according to the present disclosure refers to a device capable of gripping an object using the manipulator 120 . The robot 100 according to the present disclosure may have a structure and shape as shown in FIG. 1, but this is only an example. That is, there are no particular limitations on the structure, shape, size, etc. of the robot 100 and the manipulator 120 according to the present disclosure.

As shown in FIG. 1 , a robot 100 according to an embodiment of the present disclosure may include a camera 110 and a manipulator 120. And, although not shown in FIG. 1, the robot 100 may include a plurality of motors 130, a memory 140, and a processor 150 inside the robot 100, as shown in FIG. there is.

The camera 110 may acquire an image of at least one object. Specifically, the camera 110 includes an image sensor, and the image sensor may convert light entering through a lens into an electrical image signal.

In particular, in various embodiments according to the present disclosure, the camera 110 may acquire an image of an object, and an image of an object according to the present disclosure refers to a single view image of the object. That is, in the present disclosure, the processor 150 does not acquire a multi-view image of an object while moving the position or angle of the camera 110, but a single-view image that is one image of the object. , and based on the obtained single-view image, a pose value for holding an object may be obtained as will be described later.

The manipulator 120 can perform similar motions to human arms and hands. Specifically, the manipulator 120 may include links corresponding to the upper arm and lower arm of the user's arm, and may include a hand including a plurality of fingers. The manipulator 120 may be connected to a plurality of motors 130, and in particular, may be connected to a plurality of motors 130 corresponding to each of the upper arm, lower arm, hand, and plurality of fingers of the manipulator 120.

Also, the processor 150 may control the operation of the manipulator 120 by controlling the plurality of motors 130 based on the pose value according to the present disclosure. The manipulator 120 can provide various motions similar to those of the human arm and hand, but in the present disclosure, various embodiments according to the present disclosure will be described on the assumption that an object is gripped using the manipulator 120. do.

At least one instruction for the robot 100 may be stored in the memory 140 . And, an O/S (Operating System) for driving the robot 100 may be stored in the memory 140 . Also, various software programs or applications for operating the robot 100 may be stored in the memory 140 according to various embodiments of the present disclosure. Also, the memory 140 may include a semiconductor memory such as a flash memory or a magnetic storage medium such as a hard disk.

Specifically, various software modules for operating the robot 100 may be stored in the memory 140 according to various embodiments of the present disclosure, and the processor 150 executes various software modules stored in the memory 140 to The operation of the robot 100 can be controlled. That is, the memory 140 is accessed by the processor 150, and data can be read/written/modified/deleted/updated by the processor 150.

Meanwhile, in the present disclosure, the term memory 140 refers to the memory 140, a ROM (not shown) in the processor 150, a RAM (not shown), or a memory card (not shown) mounted in the robot 100 (eg For example, micro SD card, memory stick) can be used as a meaning including.

In particular, in various embodiments according to the present disclosure, the memory 140 includes an image of an object, point cloud information, information about a plurality of edge points, information about a plurality of body points, and information about a plurality of body points. Information on a pause value, evaluation information on each of a plurality of pause values, and the like may be stored. In addition, various types of information such as information on zero moment shift (ZMS) vectors for each of a plurality of points and information on the coordinate system of the manipulator 120 may be stored in the memory 140 .

In addition, various information required within the scope of achieving the object of the present disclosure may be stored in the memory 140, and the information stored in the memory 140 may be updated as it is received from a server or an external device or input by a user. may be

The processor 150 controls the overall operation of the robot 100. Specifically, the processor 150 is connected to the configuration of the robot 100 including the camera 110, the manipulator 120, a plurality of motors 130, and the memory 140, and the memory 140 as described above. By executing at least one instruction stored in , it is possible to control the overall operation of the robot 100 .

Processor 150 may be implemented in a variety of ways. For example, the processor 150 may include an application specific integrated circuit (ASIC), an embedded processor, a microprocessor, hardware control logic, a hardware finite state machine (FSM), a digital signal processor Processor, DSP) may be implemented as at least one. Meanwhile, in the present disclosure, the term processor 150 may be used to include a Central Processing Unit (CPU), a Graphic Processing Unit (GPU), and a Main Processing Unit (MPU).

In particular, in various embodiments according to the present disclosure, the processor 150 may obtain a pose value of the manipulator 120 for gripping the object based on geometrically analyzing an image of the object.

Specifically, the processor 150 may acquire an image of an object through the camera 110 . In particular, it has been described above that an image of an object in the present disclosure refers to a single view image of the object.

When an image of an object is obtained, the processor 150 may obtain point cloud information including a plurality of points representing the object based on the obtained image. Here, the point cloud refers to a set of a plurality of points corresponding to points on the object surface. Specifically, the processor 150 transmits a light/signal to the object, and then the light/signal is reflected from the surface of the object and is received by the image sensor included in the camera 110, the distance to the points on the surface of the object Information may be calculated, and point cloud information including a plurality of points corresponding to points on the surface of the object may be obtained based on the calculated information.

In one embodiment, when the point cloud information is obtained, the processor 150 performs a pre-processing process to remove noise included in the point cloud information, and pluralizes edge points based on the pre-processed point cloud information. and a plurality of body points may be identified. However, the preprocessing process is only for increasing the accuracy of the pose value of the manipulator 120 for gripping the object, and is not a process that must be performed in the present disclosure.

When the point cloud information is acquired, the processor 150 may identify a plurality of edge points representing the edge of the object and a plurality of body points representing the body area excluding the edge of the object from among the plurality of points. That is, the processor 150 may distinguish a plurality of points included in the point cloud information into a plurality of edge points and a plurality of body points based on the point cloud information. Here, the edge point refers to a point included in the outer area of the object, and the body point refers to a point included in the remaining area excluding the outer area of the object.

In an embodiment, the processor 150 may identify a zero moment shift (ZMS) vector pointing to a center point of points included in a circle having a preset size centered on the plurality of points, for each of the plurality of points. can When ZMS vectors are identified for each of a plurality of points, the processor 150 identifies points corresponding to vectors whose magnitudes of ZMS vectors are equal to or greater than a predetermined threshold among the plurality of points as edge points, and among the plurality of points, the magnitude of ZMS vectors Points corresponding to vectors for which is less than the threshold may be identified as body points. A process of identifying edge points and body points will be described in more detail with reference to FIGS. 3 and 4 .

If the plurality of edge points and the plurality of body points are identified, the processor 150 determines a pose when the manipulator 120 grips the object using at least some of the plurality of edge points and the plurality of body points as gripping points, respectively. It is possible to obtain a plurality of pose values representing . Here, the pose value may include information on which point of the object and in which direction the manipulator 120 is to be held. And, the pose value may include information about the gripping point and information about the direction of the coordinate axes constituting the coordinate system of the manipulator 120, and the like.

Specifically, the processor 150 may randomly sample at least some of the plurality of edge points and the plurality of body points, and obtain a plurality of pose values for each of the sampled partial points. Also, the processor 150 may acquire a plurality of pose values by applying different methods depending on whether some of the sampled points are edge points or body points.

In one embodiment, the processor 150 sets the z-axis of the manipulator 120, which is defined to face in the opposite direction to the direction toward the lower arm joint of the manipulator 120, to correspond to the direction of the ZMS vector for the identified edge point. and the y-axis of the manipulator 120 defined from one finger included in the manipulator 120 to the other finger in the opposite direction corresponds to a direction perpendicular to the surface of the identified edge point. By identifying the pose of , it is possible to obtain pose values corresponding to a plurality of edge points among a plurality of pose values.

In one embodiment, the processor 150 sets the z-axis to correspond to a direction perpendicular to the surface of the plurality of points, and the y-axis to correspond to the direction of greatest curvature at the plurality of points. By identifying the pose, it is possible to obtain pose values corresponding to a plurality of body points among a plurality of pose values.

A process of sampling at least some of the plurality of edge points and the plurality of body points and obtaining a plurality of pose values will be described in more detail with reference to FIGS. 6 to 8 .

When the plurality of pose values are obtained, the processor 150 may obtain evaluation information indicating stability when the manipulator 120 grips the object according to the plurality of pose values. Here, the evaluation information is a first score calculated according to an angle between a direction perpendicular to the surface of two fingers included in the manipulator 120 and a direction perpendicular to the surface of the contact area where the two fingers contact the object, contact The second score calculated according to the number of points included in the area, the distance between one of the two fingers and the contact area, and the distance between the other one of the two fingers and the contact area Calculated according to the difference It may include at least one of a third score and a fourth score calculated according to whether at least some points are close to the center of gravity of the object.

The processor 150 may obtain evaluation information for each of a plurality of pose values by calculating a final score based on a weighted sum of the first score, the second score, the third score, and the fourth score. The meaning of each of the first score, the second score, the third score, and the fourth score included in the evaluation information will be described in more detail with reference to FIGS. 9 and 10 .

When the evaluation information is acquired, the processor 150 may identify one pose value among a plurality of pose values based on the evaluation information. Also, if one of the plurality of pose values is identified, the processor 150 may control the plurality of motors 130 to operate the manipulator 120 based on the identified one pose value. Specifically, the processor 150 may identify one pose value having the highest final score included in the evaluation information among a plurality of pose values, and control the operation of the manipulator 120 based on the identified one pose. Can hold objects.

According to the embodiments described above and various embodiments to be described in detail below, the robot 100 acquires only a single-view image of an object, and geometrically converts the single-view image without using a neural network model and deep learning technology. By analyzing it, it is possible to stably and efficiently hold an object using only the CPU without using high-performance hardware such as GPU. In particular, according to the present disclosure, an object can be gripped with a high gripping rate, and an object can be gripped without damage to the object while minimizing a collision with the object.

3 is a diagram for explaining a process of identifying an edge point and a body point according to an embodiment of the present disclosure, and FIG. 4 is a diagram for a process of distinguishing a true edge point from a false edge point according to an embodiment of the present disclosure. It is a drawing for explanation.

The left view of FIG. 3 is a view showing point cloud information including a plurality of points representing objects. And, the drawing on the right side of FIG. 3 is a drawing showing enlarged two points among a plurality of points representing an object.

As described above, the processor 150 may identify a ZMS vector for each of a plurality of points, and may identify a plurality of edge points and a plurality of body points based on the size of the identified zero moment shift (ZMS) vector. Specifically, the processor 150 may identify, for each plurality of points, a ZMS vector pointing to a center point of points included in a circle having a predetermined size centered on the plurality of points. Then, the processor 150 identifies points corresponding to vectors having a ZMS vector size equal to or greater than a preset threshold among a plurality of points as edge points, and corresponds to vectors having a ZMS vector magnitude less than the threshold value among the plurality of points. Points that become points can be identified as body points.

For example, as shown in FIG. 3 , the processor 150 may identify a ZMS vector pointing to a center point of points included in a preset circle B having point a as a center. Since the point a is a point included in the area corresponding to the outer edge of the object, the center points (for example, points representing the center of gravity) of the points included in the preset circle (B) centered on the point a are based on the point a. may be biased in a particular direction. Accordingly, the magnitude (m) of the ZMS vector corresponding to point a, that is, the norm, may have a predetermined value greater than zero. The processor 150 may identify point a as an edge point when the size of the ZMS vector m corresponding to point a is equal to or greater than a predetermined threshold value.

Meanwhile, as shown in FIG. 3 , the processor 150 may identify a ZMS vector pointing to the center point of points included in a preset circle B centered at point b. Since the point b is a point included in the area other than the outer periphery of the object, the center points of the points included in the preset circle B having the point b as the center may coincide with the point b or may be located at a close distance. Accordingly, the magnitude (m) of the ZMS vector corresponding to the point b may have a value close to 0. The processor 150 may identify point b as the body point when the magnitude of the ZMS vector corresponding to point b is less than a predetermined threshold value.

As described above, the threshold value for identifying the ZMS vector corresponding to the edge point and the threshold value for identifying the ZMS vector corresponding to the body point may be the same or different. Further, it is needless to say that the threshold value for identifying the ZMS vector corresponding to the edge point and the threshold value for identifying the ZMS vector corresponding to the body point may be changed according to the settings of the user or developer.

On the other hand, as described above, since the image of an object in the present disclosure means one single-view image of the object, the image of the object includes an area that cannot be observed with only a single-view image, that is, an unobservable area. ) may be included. Therefore, when edge points are identified by the above-described method, a problem may arise in that points included in the outer area of the non-observation area are identified as edge points, rather than points included in the outer area of the actual object.

Therefore, as described above, the processor 150 primarily identifies the plurality of edge points and the plurality of body points based on the size of the ZMS vector, and then, among the plurality of primarily identified edge points, the outer area of the real object. It is possible to distinguish between true edge points included in and false edge points included in the outer area of the non-observation area.

Specifically, the processor 150 determines true edge points and false edge points based on an angle between a ZMS vector for one point identified as an edge point and a ray vector pointing to one point from the camera 110. , and treat the identified false edge points as body points. More specifically, if the angle between the ZMS vector for the first edge point and the ray vector directed from the camera 110 toward the first edge point is less than a preset threshold angle, the processor 150 determines the first edge point as a true edge point. can be identified by On the other hand, if the angle between the ZMS vector for the second edge point and the ray vector directed to the second edge point from the camera 110 is equal to or greater than a preset threshold angle, the processor 150 sets the second edge point as a false edge. points can be identified. Here, the first edge point identified as a true edge point may be finally treated as an edge point, and the second edge point identified as a false edge point may be finally treated as a true edge point.

)를 향하는 제1 레이 벡터와 제1 엣지 포인트(

)의 ZMS 벡터 사이의 각도는 90도 미만이다. 반면, 카메라(110)에서 비관측 영역의 외곽 영역에 포함되는 제2 엣지 포인트(

)를 향하는 제2 레이 벡터와 제2 엣지 포인트(

)의 ZMS 벡터 사이의 각도는 90도 이상이다. 이 경우, 제1 엣지 포인트는 트루 엣지 포인트로 식별되며 제2 엣지 포인트는 폴스 엣지 포인트로 식별되고, 따라서 제1 엣지 포인트는 최종적으로 엣지 포인트로 취급되는 반면, 제2 엣지 포인트는 최종적으로 바디 포인트로 취급될 수 있다.For example, referring to FIG. 4 , a first edge point included in the outer area of the real object in the camera 110 (

) and the first ray vector and the first edge point (

) is less than 90 degrees. On the other hand, the second edge point included in the outer area of the non-observation area in the camera 110 (

) and the second ray vector and the second edge point (

) is greater than 90 degrees. In this case, the first edge point is identified as a true edge point and the second edge point is identified as a false edge point, so the first edge point is finally treated as an edge point, while the second edge point is finally treated as a body point. can be treated as

5 is a diagram for explaining a process of estimating a non-observation area according to an embodiment of the present disclosure.

Meanwhile, as described above, in the present disclosure, an image of an object may include an area that cannot be observed with only a single view image, that is, a non-observation area. In addition, in order to effectively grip the object using the manipulator 120, it is necessary to estimate the non-observation area. For example, referring to the first image 510 of FIG. 5 , when an image of an object is acquired through the camera 110 and point cloud information is obtained based on the obtained image, the point cloud information is obtained from the object to the camera. Information on points corresponding to areas located on the opposite surface of (110) may not be included.

Accordingly, the processor 150 may obtain information on a plurality of estimation points corresponding to the non-observation area included in the image, based on the ZMS vector and the ray vector for the identified edge point. Here, since the identified edge point is an edge point corresponding to a boundary dividing the non-observation area and the observation area, it may be a false edge point as described above.

Specifically, the processor 150 may estimate a line segment of a predetermined length located in the opposite direction of the ZMS vector for the edge point from one edge point as part of the non-observation area. Here, estimating a line segment of a predetermined length may be considered to compensate for the loss in consideration of the possibility of data loss at the false edge point. Meanwhile, the estimated line segment length may be changed according to a setting of a user or developer, and may be determined based on the size of an object.

In addition, the processor 150 may estimate a curve that extends from the estimated line segment and does not deviate from a ray vector toward one edge point in the camera 110 as another part of the non-observation area. Specifically, the surface of the non-viewing area cannot but be located on the same line as the ray vector or inside the direction of the ray vector toward the object. Accordingly, the processor 150 may estimate a curve extending from the previously estimated line segment to the ground within a range that does not deviate from the ray vector. In an embodiment, the processor 150 may estimate a curve by considering the symmetry of the non-observed area with the opposite area, but is not limited thereto.

)에서부터 그 엣지 포인트에 대한 ZMS 벡터(m)의 반대 방향에 위치하는 기 설정된 길이의 선분을 추정할 수 있다. 그리고, 프로세서(150)는 추정된 선분에서부터 연장되며, 카메라(110)에서 하나의 엣지 포인트를 향하는 레이 벡터(r)를 벗어나지 않는 곡선을 비관측 영역의 또 다른 일부로 추정할 수 있다. 도 3의 두번째 이미지(520)에서 추정된 비관측 영역을 실선으로 표시하였으며, 도 3의 세번째 이미지(530)에서는 본 개시에 따른 비관측 영역의 추정에 의해 획득된 복수의 추정 포인트들을 나타내었다.For example, referring to the second image 520 of FIG. 5 , the processor 150 has one edge point (

), it is possible to estimate a line segment with a preset length located in the opposite direction of the ZMS vector (m) for the edge point. Further, the processor 150 may estimate a curve extending from the estimated line segment and not departing from the ray vector r toward one edge point in the camera 110 as another part of the non-observation area. In the second image 520 of FIG. 3 , the estimated non-observation area is indicated by a solid line, and in the third image 530 of FIG. 3 , a plurality of estimation points obtained by estimating the non-observation area according to the present disclosure are indicated.

As described above, if the non-observation area is estimated, the processor 150 treats a plurality of estimation points corresponding to the non-observation area as body points and uses them to obtain a plurality of pose values. Also, the processor 150 may use a plurality of estimation points corresponding to the non-observation area to identify a contact area as described below. Identification of the contact area will be described in more detail with reference to FIG. 9 .

6 to 8 are diagrams for explaining a process of obtaining a plurality of pose values according to an embodiment of the present disclosure.

When the plurality of edge points and the plurality of body points are identified through the above process, the processor 150 determines that the manipulator 120 uses at least some of the plurality of edge points and the plurality of body points as gripping points, respectively. A plurality of pose values indicating a pose when holding an object may be acquired. Here, the pose value may indicate at which point of the object and in which direction the manipulator 120 is to be gripped. And, the pose value may include information about the gripping point and information about the direction of the coordinate axes constituting the coordinate system of the manipulator 120, and the like.

Specifically, the processor 150 may randomly sample at least some of the plurality of edge points and the plurality of body points, and obtain a plurality of pose values for each of the sampled partial points. Also, the processor 150 may acquire a plurality of pose values by applying different methods depending on whether some of the sampled points are edge points or body points.

, y축에 대응되는 단위 벡터는

, 그리고 z축에 대응되는 단위 벡터는

로 나타내었다. FIG. 6 is a diagram for explaining the coordinate system of the manipulator 120 according to an embodiment of the present disclosure, and unit vectors corresponding to the coordinate system of the manipulator 120 are shown in FIG. 6 . Here, the unit vector corresponding to the x-axis is

, the unit vector corresponding to the y-axis is

, and the unit vector corresponding to the z-axis is

indicated by

Specifically, the z-axis of the coordinate system of the manipulator 120 may be defined to face in a direction opposite to a direction toward the lower arm joint of the manipulator 120 . In other words, the z-axis of the coordinate system of the manipulator 120 may be defined to face the opposite direction of the arm of the manipulator 120 . Also, the y-axis of the coordinate system of the manipulator 120 may be defined from one finger included in the manipulator 120 to another finger in the opposite direction. In other words, the y-axis of the coordinate system of the manipulator 120 may be defined to be parallel to the direction in which the fingers of the manipulator 120 are closed. If the z-axis and the y-axis are defined, the x-axis of the coordinate system of the manipulator 120 may be determined based on a cross product between a unit vector corresponding to the x-axis and a unit vector corresponding to the y-axis.

Hereinafter, a process of obtaining coordinate values for an edge point according to an embodiment of the present disclosure will be described with reference to FIG. 7 , and obtaining coordinate values for a body point according to an embodiment of the present disclosure will be described with reference to FIG. 8 . explain the process of

In one embodiment, the processor 150 causes the z-axis of the manipulator 120 to correspond to the direction of the ZMS vector for the identified edge point, and the y-axis of the manipulator 120 to be perpendicular to the surface of the identified edge point. Pose values corresponding to a plurality of edge points among a plurality of pose values may be obtained by identifying poses of the manipulator 120 to correspond to the in direction. Here, the direction perpendicular to the surface of the edge point refers to the direction of a normal vector from the edge point toward the ground.

)에 대한 ZMS 벡터의 방향에 대응되도록 하고, 매니퓰레이터(120)의 y축이 엣지 포인트(

)의 표면에 수직인 방향에 대응되도록 하기 위한 포즈 값을 획득할 수 있다. 다시 말해, 프로세서(150)는 매니퓰레이터(120) 좌표계의 z축을 해당 엣지 포인트의 ZMS 벡터(

) 방향으로 정렬시키고, 매니퓰레이터(120) 좌표계의 y축을 해당 엣지 포인트에서 지면을 향하는 법선 벡터(

)의 방향으로 정렬시킴으로써, 각각의 엣지 포인트에 대응되는 포즈 값을 획득할 수 있다.Referring to the example of FIG. 7 , the z-axis of the manipulator 120 is the edge point (

), and the y-axis of the manipulator 120 corresponds to the edge point (

) It is possible to obtain a pose value to correspond to the direction perpendicular to the surface of the surface. In other words, the processor 150 converts the z-axis of the coordinate system of the manipulator 120 to the ZMS vector of the corresponding edge point (

) direction, and the y-axis of the coordinate system of the manipulator 120 is the normal vector pointing to the ground at the corresponding edge point (

), it is possible to obtain a pose value corresponding to each edge point.

In one embodiment, the processor 150 causes the z-axis of the manipulator 120 to correspond to a direction perpendicular to the surface of a plurality of points, and the y-axis of the manipulator 120 to have curvature at the plurality of points. By identifying the pose of the manipulator 120 to correspond to the largest direction, it is possible to obtain pose values corresponding to a plurality of body points among a plurality of pose values. Here, the direction perpendicular to the surface of the body point refers to a direction of a normal vector from the body point toward the ground.

)의 표면에 수직인 방향에 대응되도록 하고, 매니퓰레이터(120)의 y축이 바디 포인트(

)에서 곡률이 가장 큰 방향에 대응되도록 하기 위한 포즈 값을 식별할 수 있다. 다시 말해, 프로세서(150)는 매니퓰레이터(120) 좌표계의 z축을 해당 바디 포인트에서 지면을 향하는 법선 벡터의 방향으로 정렬시키고, 매니퓰레이터(120) 좌표계의 y축은 해당 바디 포인트에서 곡률이 가장 큰 방향, x축은 해당 바디 포인트에서 곡률이 가장 작은 방향으로 정렬시킴으로써, 각각의 바디 포인트에 대응되는 포즈 값을 획득할 수 있다.Referring to the example of FIG. 8 , the z-axis of the manipulator 120 is the body point (

), and the y-axis of the manipulator 120 corresponds to the body point (

), it is possible to identify a pose value for corresponding to the direction in which the curvature is greatest. In other words, the processor 150 aligns the z axis of the coordinate system of the manipulator 120 with the direction of the normal vector from the body point to the ground, and the y axis of the coordinate system of the manipulator 120 is the direction of greatest curvature at the body point, x A pose value corresponding to each body point may be obtained by aligning the axis in a direction having the smallest curvature at the corresponding body point.

Meanwhile, when pose values corresponding to randomly sampled edge points and body points are obtained as described above, the processor 150 may augment the pose values based on the obtained pose values. Specifically, the processor 150 may augment a plurality of pose values by changing some coordinate axes of pose values corresponding to a plurality of edge points and pose values corresponding to a plurality of body points.

In other words, the processor 150 performs a process for obtaining pose values corresponding to randomly sampled edge points and body points (which may be referred to as a so-called coarse sampling process), and then the obtained pose values indicate A process of augmenting a plurality of pose values (which may be referred to as a so-called fine sampling process) may be performed by changing at least one coordinate axis among the x-axis, y-axis, and z-axis within a preset threshold angle. Accordingly, the processor can obtain a large number of pose values in an efficient manner.

9 and 10 are diagrams for explaining in detail evaluation information according to an embodiment of the present disclosure.

As described above, when a plurality of pose values are obtained, the processor 150 may obtain evaluation information representing stability when the manipulator 120 grips an object according to the plurality of pose values. Here, the evaluation information may include a first score, a second score, a third score, and a fourth score as described later.

Hereinafter, the contact area of the manipulator 120 is defined with reference to FIG. 9, and the first score, the second score, the third score, and the fourth score according to the present disclosure are described in detail with reference to FIG. 10. .

The contact area of the manipulator 120 may be defined as shown in FIG. 9 . Specifically, a region between the surfaces of two fingers included in the manipulator 120 may be defined as a closing region, and a region where the two fingers contact the object in the closing region has a predetermined thickness (t _c ) may be defined as a contact region. And, the contact region is a left contact region where one of the two fingers comes into contact with the object and a right contact region where the other one of the two fingers comes into contact with the object. ) may be included.

Meanwhile, as described above, the image of the object may include a non-observation area, and the processor 150 may obtain information on a plurality of estimation points corresponding to the non-observation area. Then, when information on a plurality of estimated points corresponding to the non-observation area is obtained, the processor 150 obtains information on the surface areas of the two fingers, a plurality of edge points of the manipulator 120, a plurality of body points, and a plurality of A contact area may be identified based on the estimated point.

,

)의 방향이 일치할수록 제1 스코어는 높게 산출될 수 있는바, 이는 두 개의 핑거에서 전달되는 힘이 많이 전달될수록 높은 스코어를 할당하기 위한 것이라고 할 수 있다. 제1 스코어는 소위 antipodal quality를 평가하기 위한 것이라고 할 수 있다. The first score may be calculated according to an angle between a direction perpendicular to the surface of the two fingers included in the manipulator 120 and a direction perpendicular to the surface of the contact area where the two fingers come into contact with an object. Specifically, the first score may be calculated higher as the direction perpendicular to the surface of the two fingers coincides with the direction perpendicular to the surface of the contact area where the two fingers contact the object. That is, the direction of the normal vector of the two finger surfaces included in the manipulator 120 and the normal vector of the contact area (

,

), the first score can be calculated higher as the directions of the two fingers coincide, which can be said to assign a higher score as more force transmitted from the two fingers is transmitted. The first score can be said to be for evaluating the so-called antipodal quality.

The second score may be calculated according to the number of points included in the contact area. Specifically, the second score may be calculated higher as the contact area of the two fingers included in the manipulator 120 with the object is wider. This can be said to assign a higher score as the frictional force between the two fingers and the contact area is higher. The second score can be said to be for evaluating so-called contact quality.

The third score may be calculated according to the difference between the distance between one of the two fingers and the contact area and the distance between the other one of the two fingers and the contact area. Specifically, the third score may be calculated higher as the left contact area and the right contact area are closer to left-right symmetry with respect to the center of the object. The third score may be for evaluating so-called displacement quality.

The fourth score may be calculated according to whether at least some of the plurality of edge points and the plurality of body points are close to the center of gravity of the object. Specifically, the fourth score may be calculated higher as points used to obtain the plurality of pose values are closer to the center of gravity of the object. The first score can be said to be for evaluating the so-called gravity quality.

As described above, when the first score, the second score, the third score, and the fourth score are obtained, the processor 150 calculates based on the weighted sum of the first score, the second score, the third score, and the fourth score. By calculating the final score, evaluation information for each of a plurality of pose values may be obtained.

If one of the plurality of pose values is identified, the processor 150 may control the plurality of motors 130 to operate the manipulator 120 based on the identified one pose value. Specifically, the processor 150 may identify one pose value having the highest final score included in the evaluation information among a plurality of pose values, and control the operation of the manipulator 120 based on the identified one pose. Can hold objects.

Meanwhile, in the foregoing, a process of obtaining evaluation information representing stability when the manipulator 120 grips an object based on the first score, the second score, the third score, and the fourth score has been described, but this is an exemplary embodiment. It is only an example, and various methods capable of indicating stability when the manipulator 120 grips an object can be applied to the present disclosure, of course.

In particular, in the present disclosure, the case where the manipulator 120 grips an object using the first score, the second score, the third score, and the fourth score as described above without using an artificial intelligence model or a neural network model. Evaluation information indicating stability may be obtained, and thus stability when the manipulator 120 grips an object using only a central processing unit (CPU) without using high resources using a graphics processing unit (GPU). can be evaluated.

11 is a block diagram showing the configuration of the robot 100 according to an embodiment of the present disclosure in detail.

As shown in FIG. 2 , the robot 100 according to an embodiment of the present disclosure includes a camera 110 , a manipulator 120 , and a plurality of motors 130 . In addition, as shown in FIG. 3 , the robot 100 according to an embodiment of the present disclosure may further include a communication interface 170, an input interface 180, and an output interface 190. However, the configurations shown in FIGS. 2 and 3 are merely exemplary, and new configurations may be added or some configurations may be omitted in addition to the configurations shown in FIGS. 2 and 3 in practicing the present disclosure. Of course there is.

The sensor 160 may detect various information inside and outside the robot 100 . Specifically, the sensor 160 includes a Global Positioning System (GPS) sensor 160, a gyro sensor 160 (gyro sensor, gyroscope), an acceleration sensor 160 (acceleration sensor, accelerometer), and a lidar sensor 160 ( lidar sensor), an inertial sensor 160 (Inertial Measurement Unit, IMU), and a motion sensor 160. In addition, the sensor 160 may include various types of sensors 160 such as a temperature sensor 160, a humidity sensor 160, an infrared sensor 160, and a biosensor 160.

In particular, in various embodiments according to the present disclosure, the processor 150 may acquire various information such as the distance between the robot 100 and an object, the motion of the object, etc. through the sensor 160, The information may be used together with the pose value of the manipulator 120 in gripping an object by controlling the operation of the manipulator 120 .

The communication interface 170 includes a circuit and can perform communication with a server or an external device. Specifically, the processor 150 may receive various data or information from a server or an external device connected through the communication interface 170, and may transmit various data or information to the server or external device.

The communication interface 170 may include at least one of a WiFi module, a Bluetooth module, a wireless communication module, an NFC module, and a UWB module (Ultra Wide Band). Specifically, each of the WiFi module and the Bluetooth module may perform communication using a WiFi method or a Bluetooth method. In the case of using a WiFi module or a Bluetooth module, various connection information such as an SSID is first transmitted and received, and various information can be transmitted and received after communication is connected using this.

In addition, the wireless communication module may perform communication according to various communication standards such as IEEE, Zigbee, 3rd Generation (3G), 3rd Generation Partnership Project (3GPP), Long Term Evolution (LTE), and 5th Generation (5G). In addition, the NFC module may perform communication using a Near Field Communication (NFC) method using a 13.56 MHz band among various RF-ID frequency bands such as 135 kHz, 13.56 MHz, 433 MHz, 860 ~ 960 MHz, and 2.45 GHz. In addition, the UWB module can accurately measure Time of Arrival (ToA), which is the time at which a pulse arrives at a target, and Ange of Arrival (AoA), which is the angle of arrival of a pulse in a transmitting device, through communication between UWB antennas. It is possible to recognize precise distance and location within the error range of several tens of cm.

In particular, in various embodiments according to the present disclosure, the processor 150 may include an image of an object, point cloud information, information about a plurality of edge points, information about a plurality of body points, information about a plurality of pose values, The communication interface 170 is used to transmit evaluation information for each of a plurality of pose values, information on zero moment shift (ZMS) vectors for each of a plurality of points, and information on the coordinate system of the manipulator 120 to a server or an external device. It can be controlled, and various information including the above information can be received through the communication interface 170.

The input interface 180 includes a circuit, and the processor 150 may receive a user command for controlling the operation of the robot 100 through the input interface 180 . Specifically, the input interface 180 may include components such as a microphone, a camera 110 (not shown), and a remote control signal receiver (not shown). Also, the input interface 180 may be implemented as a touch screen included in a display. In particular, the microphone may receive a voice signal and convert the received voice signal into an electrical signal.

In particular, in various embodiments according to the present disclosure, the processor 150 may receive a user command for gripping an object using the manipulator 120 through the input interface 180 .

The output interface 190 includes a circuit, and the processor 150 may output various functions that the robot 100 may perform through the output interface 190 . Also, the output interface 190 may include at least one of a display, a speaker, and an indicator.

The display may output image data under the control of the processor 150 . Specifically, the display may output an image pre-stored in the memory 140 under the control of the processor 150 . In particular, the display according to an embodiment of the present disclosure may display a user interface stored in the memory 140 .

The display may be implemented with a liquid crystal display panel (LCD), organic light emitting diodes (OLED), or the like, and the display may also be implemented with a flexible display, a transparent display, or the like, depending on circumstances. However, the display according to the present disclosure is not limited to a specific type.

The speaker may output audio data under the control of the processor 150, and the indicator may be turned on under the control of the processor 150.

In particular, in various embodiments according to the present disclosure, the processor 150 may output various information, such as information indicating that object grasping is in progress, information indicating that object grasping has been completed, and the like, through the output interface 190. there is.

12 is a flowchart illustrating a control method of the robot 100 according to an embodiment of the present disclosure.

Referring to FIG. 12 , the robot 100 may acquire an image of an object through the camera 110 (S1210). In particular, it has been described above that an image of an object in the present disclosure refers to a single view image of the object.

When an image of an object is obtained, the robot 100 may obtain point cloud information including a plurality of points representing the object based on the obtained image (S1210).

In one embodiment, when point cloud information is acquired, the robot 100 performs a pre-processing process for removing noise included in the point cloud information, and pluralizes edge points based on the pre-processed point cloud information. and a plurality of body points may be identified.

When the point cloud information is obtained, the robot 100 may identify a plurality of edge points representing the edge of the object and a plurality of body points representing the body area excluding the edge of the object from among the plurality of points (S1230).

In one embodiment, the robot 100 identifies a zero moment shift (ZMS) vector directed toward the center point of points included in a circle having a preset size centered on the plurality of points, for each of the plurality of points. can When ZMS vectors are identified for each of a plurality of points, the robot 100 identifies points corresponding to vectors whose magnitudes of ZMS vectors are equal to or greater than a predetermined threshold among the plurality of points as edge points, and among the plurality of points, the magnitudes of ZMS vectors Points corresponding to vectors for which is less than the threshold may be identified as body points.

If the plurality of edge points and the plurality of body points are identified, the robot 100 poses when the manipulator 120 grips the object by using at least some of the plurality of edge points and the plurality of body points as gripping points, respectively. It is possible to obtain a plurality of pause values representing (S1240).

Specifically, the robot 100 may randomly sample at least some of the plurality of edge points and the plurality of body points, and obtain a plurality of pose values for each of the sampled partial points. In addition, the robot 100 may acquire a plurality of pose values by applying different methods depending on whether some of the sampled points are edge points or body points.

In one embodiment, the robot 100 makes the z-axis defined to face the direction opposite to the direction toward the lower arm joint of the manipulator 120 correspond to the direction of the ZMS vector for the identified edge point, and the manipulator 120 ) By identifying the pose of the manipulator 120 so that the y-axis defined from one finger to the other finger in the opposite direction corresponds to the direction perpendicular to the surface of the identified edge point, a plurality of pose values Among them, pose values corresponding to a plurality of edge points may be obtained.

In one embodiment, the robot 100 has a z-axis corresponding to a direction perpendicular to the surface of a plurality of points, and a y-axis of the manipulator 120 to correspond to a direction with the greatest curvature at the plurality of points. By identifying the pose, it is possible to obtain pose values corresponding to a plurality of body points among a plurality of pose values.

When a plurality of pose values are obtained, the robot 100 may obtain evaluation information representing stability when the manipulator 120 grips an object according to the plurality of pose values (S1250).

Here, the evaluation information is a first score calculated according to an angle between a direction perpendicular to the surface of two fingers included in the manipulator 120 and a direction perpendicular to the surface of the contact area where the two fingers contact the object, contact The second score calculated according to the number of points included in the area, the distance between one of the two fingers and the contact area, and the distance between the other one of the two fingers and the contact area Calculated according to the difference It may include at least one of a third score and a fourth score calculated according to whether at least some points are close to the center of gravity of the object.

The robot 100 may obtain evaluation information for each of a plurality of pose values by calculating a final score based on a weighted sum of the first score, the second score, the third score, and the fourth score.

When the evaluation information is acquired, the robot 100 may identify one pose value among a plurality of pose values based on the evaluation information (S1260). If one of the plurality of pose values is identified, the robot 100 may control the plurality of motors 130 to operate the manipulator 120 based on the identified one pose value (S1270). Specifically, the robot 100 may identify one pose value having the highest final score included in the evaluation information among a plurality of pose values, and control the operation of the manipulator 120 based on the identified one pose. Can hold objects.

Meanwhile, the robot control method according to the above-described embodiment may be implemented as a program and provided to the robot. In particular, a program including a method for controlling a robot may be stored and provided in a non-transitory computer readable medium.

Specifically, in a non-transitory computer-readable recording medium containing a program that executes a method for controlling a robot, the method for controlling a robot includes acquiring an image of an object, a plurality of points indicating the object based on the acquired image. Obtaining point cloud information including a plurality of edge points representing an edge of an object among a plurality of points and a plurality of body points representing a body area excluding the edge of the object Identifying, obtaining a plurality of pose values indicating poses when the manipulator 120 grips an object using at least some of the plurality of edge points and the plurality of body points as gripping points, the plurality of poses Obtaining evaluation information representing stability when the manipulator 120 grips the object according to the values, identifying one pose value among a plurality of pose values based on the evaluation information, and determining the identified one pose value. Based on the method, a step of controlling the plurality of motors 130 to operate the manipulator 120 may be included.

In the above, the control method of the robot and the computer readable recording medium including the program for executing the control method of the robot have been briefly described, but this is only for omitting redundant description, and various embodiments of the robot Of course, it can also be applied to a computer readable recording medium including a control method and a program for executing the control method of the robot.

The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-temporary storage medium' only means that it is a tangible device and does not contain signals (e.g., electromagnetic waves), and this term refers to the case where data is stored semi-permanently in the storage medium and temporary It does not discriminate if it is saved as . For example, a 'non-temporary storage medium' may include a buffer in which data is temporarily stored.

According to one embodiment, the method according to various embodiments disclosed in this document may be included and provided in a computer program product. Computer program products may be traded between sellers and buyers as commodities. A computer program product is distributed in the form of a device-readable storage medium (eg compact disc read only memory (CD-ROM)), or through an application store (eg Play Store ^TM ) or between two user devices ( It can be distributed (eg downloaded or uploaded) online, directly between smartphones. In the case of online distribution, at least a portion of a computer program product (eg, a downloadable app) is a device-readable storage such as a manufacturer's server, an application store's server, or a relay server's memory 140. It can be at least temporarily stored in a medium or temporarily created.

Each of the components (eg, modules or programs) according to various embodiments of the present disclosure as described above may be composed of a single object or a plurality of entities, and some of the sub-components described above are omitted. or other sub-elements may be further included in various embodiments. Alternatively or additionally, some components (eg, modules or programs) may be integrated into one entity and perform the same or similar functions performed by each corresponding component prior to integration.

According to various embodiments, operations performed by modules, programs, or other components may be executed sequentially, in parallel, repetitively, or heuristically, or at least some operations may be executed in a different order, may be omitted, or other operations may be added. can

On the other hand, the term "unit" or "module" used in the present disclosure includes units composed of hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic blocks, parts, or circuits, for example. can A “unit” or “module” may be an integrated component or a minimum unit or part thereof that performs one or more functions. For example, the module may be composed of an application-specific integrated circuit (ASIC).

Various embodiments of the present disclosure may be implemented as software including commands stored in a storage medium readable by a machine (eg, a computer). The device calls the stored commands from the storage medium. And, as a device capable of operating according to the called command, it may include an electronic device (eg, the robot 100) according to the disclosed embodiments.

When the command is executed by a processor, the processor may directly or use other elements under the control of the processor to perform a function corresponding to the command. An instruction may include code generated or executed by a compiler or interpreter.

Although the preferred embodiments of the present disclosure have been shown and described above, the present disclosure is not limited to the specific embodiments described above, and is common in the art to which the disclosure belongs without departing from the gist of the present disclosure claimed in the claims. Of course, various modifications are possible by those with knowledge of, and these modifications should not be individually understood from the technical spirit or perspective of the present disclosure.

100: robot 110: camera
120: manipulator 130: multiple motors
140: memory

Claims (10)

in robots,
camera;
manipulators;
a plurality of motors for controlling the operation of the manipulator;
a memory for storing at least one instruction; and
a processor to execute the at least one instruction; including,
the processor,
Obtaining an image of an object through the camera,
Obtaining point cloud information including a plurality of points representing the object based on the obtained image;
Identifying a plurality of edge points representing an edge of the object and a plurality of body points representing a body area excluding the edge of the object among the plurality of points;
obtaining a plurality of pose values representing a pose when the manipulator grips the object by using at least some of the plurality of edge points and the plurality of body points as gripping points, respectively;
obtaining evaluation information representing stability when the manipulator grips the object according to the plurality of pose values;
Identifying one pose value among the plurality of pose values based on the evaluation information;
A robot that controls the plurality of motors to operate the manipulator based on the identified one pose value.
According to claim 1,
the processor,
For each of the plurality of points, a zero moment shift (ZMS) vector directed from the plurality of points to a center point of points included in a circle having a preset size centered on the plurality of points is identified,
Among the plurality of points, points corresponding to vectors having a magnitude of the ZMS vector greater than or equal to a predetermined threshold value are identified as the edge points;
Among the plurality of points, the robot identifies points corresponding to vectors in which the size of the ZMS vector is less than the threshold value as the body point.
According to claim 2,
the processor,
If the angle between the ZMS vector for the one point identified as the edge point and the ray vector directed from the camera to the one point is equal to or greater than a preset threshold angle, identifying the one point as the body point robot.
According to claim 3,
the processor,
The z-axis of the manipulator, which is defined to face in a direction opposite to the direction toward the lower arm joint of the manipulator, corresponds to the direction of the ZMS vector for the identified edge point, and in one finger included in the manipulator, in the opposite direction Corresponding to the plurality of edge points among the plurality of pose values by identifying poses of the manipulator so that the y-axis of the manipulator, which is defined to face the other finger, corresponds to a direction perpendicular to the surface of the identified edge points. A robot that obtains the pose values to be.
According to claim 4,
the processor,
The plurality of poses by identifying a pose of the manipulator so that the z-axis corresponds to a direction perpendicular to the surface of the plurality of points and the y-axis corresponds to a direction of greatest curvature at the plurality of points A robot that obtains pose values corresponding to the plurality of body points among values.
According to claim 5,
the processor,
A robot that augments the plurality of pose values by changing some coordinate axes of pose values corresponding to the plurality of edge points and pose values corresponding to the plurality of body points.
According to claim 6,
The evaluation information is a first score calculated according to an angle between a direction perpendicular to the surface of two fingers included in the manipulator and a direction perpendicular to the surface of the contact area where the two fingers contact the object, the contact A second score calculated according to the number of points included in the area, a difference between the distance between one of the two fingers and the contact area and the distance between the other of the two fingers and the contact area A robot including at least one of a third score calculated according to and a fourth score calculated according to whether the at least some points are close to the center of gravity of the object.
According to claim 7,
the processor,
A robot that obtains evaluation information for each of the plurality of pose values by calculating a final score based on a weighted sum of the first score, the second score, the third score, and the fourth score.
According to claim 8,
the processor,
Obtaining information on a plurality of estimation points corresponding to a non-observation area included in the image based on the ZMS vector and the ray vector for the identified edge point;
and identifying the contact area based on the information on the surface areas of the two fingers, the plurality of edge points of the manipulator, the plurality of body points, and the plurality of estimated points.
A control method of a robot including a manipulator and a plurality of motors for controlling the operation of the manipulator, comprising:
acquiring an image of the object;
obtaining point cloud information including a plurality of points representing the object based on the obtained image;
identifying a plurality of edge points representing an edge of the object and a plurality of body points representing a body area excluding the edge of the object from among the plurality of points;
obtaining a plurality of pose values representing poses when the manipulator grips the object by using at least some of the plurality of edge points and the plurality of body points as gripping points;
obtaining evaluation information representing stability when the manipulator grips the object according to the plurality of pose values;
identifying one of the plurality of pose values based on the evaluation information; and
controlling the plurality of motors to operate the manipulator based on the identified one pose value; A robot comprising a.

Images