KR20240030730A - Apparatus and method for controlling a robot manipulator using a gimbal mechanism - Google Patents

Abstract

A control device is provided. The control device corresponds to the positional relationship between the first joint and the gimbal when a gimbal is installed on a first arm having a first joint in a robot arm and a target for shooting a camera installed on the gimbal is provided. an acquisition unit that acquires a second relationship; a processing unit that converts target coordinates corresponding to the position of the target into a reference coordinate system of the robot arm using the first relationship; and a control unit that controls the gimbal using the target coordinates converted to the reference coordinate system, wherein the control unit controls the gimbal so that the camera faces the target coordinates regardless of the operation of the robot arm. .

Description

Apparatus and method for controlling a robot manipulator using a gimbal mechanism}

The present invention relates to an apparatus and method for controlling a robot manipulator equipped with a gimbal.

In most robot systems, an imaging means such as a camera for the robot's work is installed near the terminal effect device of the robot manipulator.

In this case, the range of vision, or visual information, delivered to the user or robot may be limited depending on the installation location of the robot's end effect device. As a result, the robot's work range may be limited or it may be difficult to secure the user's field of vision.

Korean Patent Publication No. 2162756 discloses a scan camera that acquires location information about the code mark attached to the floor and the center position.

Korean Patent Publication No. 2162756

The present invention is to provide a control device and method for controlling a gimbal or a robot using a gimbal installed on a robot arm to continuously photograph a target set by a user regardless of the operation of the robot arm.

The control device of the present invention provides a positional relationship between the first joint and the gimbal when a gimbal is installed on a first arm having a first joint in a robot arm and a target for shooting a camera installed on the gimbal is provided. an acquisition unit that acquires a second relationship corresponding to; a processing unit that converts target coordinates corresponding to the position of the target into a reference coordinate system of the robot arm using the second relationship; and a control unit that controls the gimbal using the target coordinates converted to the reference coordinate system, wherein the control unit controls the gimbal so that the camera faces the target coordinates regardless of the operation of the robot arm. .

The control method of the present invention includes an acquisition step; processing steps; A control step may be included.

In the acquisition step, a first relationship corresponding to a positional relationship between the robot arm and the first joint is acquired, a second relationship corresponding to a positional relationship between the first joint and the gimbal is obtained, and the gimbal and the gimbal are obtained. A third relationship corresponding to the positional relationship between cameras can be obtained, and a fourth relationship corresponding to the positional relationship between the camera and the target can be obtained.

In the processing step, the target coordinates corresponding to the position of the target may be converted into a reference coordinate system of the robot arm using the first relationship, the second relationship, the third relationship, and the fourth relationship.

In the control step, the gimbal may be controlled so that the camera faces the target coordinates regardless of the operation of the robot arm, using the target coordinates converted to the reference coordinate system.

According to the present invention, a robot manipulator system including a gimbal mechanism can be provided.

An imaging means that functions as a sensor for the operation of the robot may be installed at the end of the manipulator. The gimbal of the present invention is intended to support a camera that is provided separately from the corresponding photographing means.

The image captured by the imaging means disposed at the end of the robot arm or the end of the manipulator is for obtaining information necessary for the operation of the robot, and can capture images based on the robot. However, there are cases where the video is difficult to recognize from the perspective of the user managing the robot. This can be a major problem from the perspective of users instructing robots to perform various tasks or monitoring precise tasks.

The control device of the present invention may be provided with a separate gimbal and an imaging means that functions as a sensor at the end of the robot arm. The gimbal may be detachably formed on any one of the joints that form the robot arm together with the camera.

The control device of the present invention can provide a hardware gimbal and at the same time a means to control the gimbal.

The control device can control the gimbal to continuously photograph a specific location set by the user regardless of the movement of the robot arm, so that tasks can be performed while continuously looking at the specific location set by the user.

According to the present invention, compared to existing systems that install a camera on the end effector of a robot, it is possible to freely secure a view of the location to work regardless of the position and orientation of the end effector of the robot by using a gimbal. In particular, it is very useful in remote control.

According to the present invention, a gimbal that is detachable from the robot arm can be provided, and at the same time, a control means for the gimbal whose control conditions vary depending on the installation object or installation location can be provided. As a result, according to the present invention, a fusion system combining a robot and a gimbal can be provided in a realistically usable state.

1 is a schematic diagram showing the control device of the present invention.
Figure 2 is a schematic diagram showing the operation of the control device.
Figure 3 is a schematic diagram showing a gimbal.
Figure 4 is another schematic diagram showing a gimbal.
Figure 5 is another schematic diagram showing a gimbal.
Figure 6 is a schematic diagram showing the positional relationship between the robot's reference coordinate system and the camera coordinate system.
Figure 7 is a schematic diagram showing the process of obtaining the transformation matrix T.
Figure 8 is a flowchart showing the control method of the present invention.
Figure 9 is a flowchart of obtaining a transformation matrix T representing the position and orientation relationship between the joint and the gimbal.
Figure 10 is a schematic diagram showing a robot equipped with a gimbal.
11 is a diagram illustrating a computing device according to an embodiment of the present invention.

Below, with reference to the attached drawings, embodiments of the present invention will be described in detail so that those skilled in the art can easily implement the present invention. However, the present invention may be implemented in many different forms and is not limited to the embodiments described herein. In order to clearly explain the present invention in the drawings, parts that are not related to the description are omitted, and similar parts are given similar reference numerals throughout the specification.

In this specification, duplicate descriptions of the same components are omitted.

Also, in this specification, when a component is mentioned as being 'connected' or 'connected' to another component, it may be directly connected or connected to the other component, but may be connected to the other component in the middle. It should be understood that may exist. On the other hand, in this specification, when it is mentioned that a component is 'directly connected' or 'directly connected' to another component, it should be understood that there are no other components in between.

Additionally, the terms used in this specification are merely used to describe specific embodiments and are not intended to limit the present invention.

Also, in this specification, singular expressions may include plural expressions, unless the context clearly dictates otherwise.

In addition, in this specification, terms such as 'include' or 'have' are only intended to designate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, and one or more It should be understood that this does not exclude in advance the presence or addition of other features, numbers, steps, operations, components, parts, or combinations thereof.

Also, in this specification, the term 'and/or' includes a combination of a plurality of listed items or any of the plurality of listed items. In this specification, 'A or B' may include 'A', 'B', or 'both A and B'.

Additionally, in this specification, detailed descriptions of well-known functions and configurations that may obscure the gist of the present invention will be omitted.

Figure 1 is a schematic diagram showing the control device 100 of the present invention. Figure 2 is a schematic diagram showing the operation of the control device 100.

The control device 100 shown in FIG. 1 may include an acquisition unit 110, a processing unit 130, and a control unit 150.

In the robot arm 30 (arm), a gimbal 200 is installed on the first arm 31 having a first joint, and a target s, which is an object of filming for the camera 201 installed on the gimbal 200, is provided. You can.

The robot arm 30 may include one or more rod-shaped nodes. Each node can be formed to be movable through a joint formed to be rotatable. Among one or more nodes, the arm on which the gimbal 200 is installed may correspond to the first arm 31.

The robot arm 30 may be supported on the robot body 90. The robot body 90 may be fixedly installed on the ground or may move along the ground using a moving means 91 such as wheels.

An end effector, such as a work tool that performs various tasks, may be installed at the end of the robot arm 30 along with the imaging means 10. The imaging means 10 may collect information for controlling the end effector. Due to this, the photographing means 10 can photograph only the robot-centered work position from the position of the end effector, regardless of the target that the user must recognize during various tasks.

For example, when a robot performs welding work on an object located at a first location, the photographing means 10 may photograph the first location. At this time, the user needs to look at surrounding materials at the second location that may collide with the robot arm 30. However, since it is impossible to photograph the second position using the photographing means 10 alone, the user has no choice but to look directly at the surrounding materials at the work site. In this way, the location in which the robot's photographing means 10 is interested in the work site and the target s in which the user is interested may be different. In order to provide an image of the target s to the user, a camera 201 separate from the photographing means 10 may be additionally installed on the robot arm 30.

The camera 201 installed on the robot arm 30 needs to photograph the target s in a position different from the area of interest to the robot arm 30 and the imaging means 10 at a set angle. The image captured through the camera 201 can be displayed on the display 20 used by the user through a wired or wireless communication network.

In reality, where the robot arm 30 supporting the camera 201 operates independently of the target s, a gimbal 200 may be provided to ensure that the camera 201 always faces the target s. At this time, the gimbal 200 needs to operate so that the optical axis of the camera 201 always points toward the target s, regardless of the movement of the robot arm 30.

For this, the target's position needs to be converted to a robot-centered reference coordinate system. The acquisition unit 110 and the processing unit 130 may be used for the conversion.

The acquisition unit 110 may acquire a first relationship corresponding to the positional relationship between the robot arm 30 and the first joint.

The acquisition unit 110 may acquire a second relationship corresponding to the positional relationship between the first joint and the gimbal 200.

The acquisition unit 110 may acquire a third relationship corresponding to the positional relationship between the gimbal 200 and the camera 201.

The acquisition unit 110 may acquire a fourth relationship corresponding to the positional relationship between the camera 201 and the target.

The processing unit 130 may convert the target coordinates corresponding to the location of the target into the reference coordinate system of the robot arm 30 using the first relationship, second relationship, third relationship, and fourth relationship.

By connecting the first relationship, second relationship, third relationship, and fourth relationship in order, a final relationship corresponding to the position relationship between the robot arm 30 and target s can be obtained.

The control unit 150 can control the gimbal 200 using target coordinates converted to a reference coordinate system. For example, the control unit 150 may control the gimbal 200 so that the camera 201 faces the target coordinates regardless of the operation of the robot arm 30.

The target can be changed by the user through manipulation of a remote control, etc.

Figure 3 is a schematic diagram showing the gimbal 200. Figure 4 is another schematic diagram showing the gimbal 200. Figure 5 is another schematic diagram showing the gimbal 200.

The gimbal 200 uses a gyro sensor and an acceleration sensor internally to minimize image shaking while looking in a specific direction. According to the gimbal 200, regardless of the motion of the robot, the camera 201 gazes in a specific direction where the target s is located, and the user can be provided with a field of view (vision) that can work efficiently.

The gimbal 200 may be provided with a first unit 210, a second unit 220, a third unit 230, and a fourth unit 240.

The first unit 210 may be installed on the first arm 31 to be capable of first rotation about a first virtual axis extending along the longitudinal direction of the first arm 31. The first virtual axis may be formed on the same axis as the central axis of the first arm 31. In this case, the first unit 210 has a first degree of rotational freedom with the longitudinal direction or central axis of the first arm 31 as the center of rotation.

The first arm 31 may be formed in a pillar or bar shape. Correspondingly, the first unit 210 may be formed in a band shape or ring shape surrounding the first arm 31. A through hole 219 through which the first arm 31 is installed may be formed in the center of the first unit 210.

The second unit 220 may be installed in the first unit 210 to be capable of second rotation about a second virtual axis extending along a direction perpendicular to the first virtual axis. In other words, the second unit 220 may have a second rotational degree of freedom with the second virtual axis extending perpendicular to the first arm 31 as the rotation center.

The third unit 230 may be installed in the second unit 220 to enable a third rotation about a third virtual axis extending along a direction perpendicular to the second virtual axis. In other words, the third unit 230 may have a third rotational degree of freedom with the third virtual axis as the rotation center.

The fourth unit 240 may be installed in the third unit 230 to be capable of fourth rotation about a fourth virtual axis extending along a direction perpendicular to the third virtual axis. The fourth virtual axis may extend perpendicularly not only to the third virtual axis but also to the second virtual axis. The fourth unit 240 may have a fourth rotational degree of freedom with the fourth virtual axis as the rotation center.

The fourth unit 240 may be provided with a holder 260 on which the camera 201 is mounted and a detachment means 250 on which the camera 201 is attached and detached. As an example, the attaching and detaching means 250 may include a screw coupled to the camera 201. According to the holder 260 and the attachment/detachment means 250, the camera 201 can be mounted on the fourth unit 240 and aligned in a set posture (including position and orientation) with respect to the fourth unit 240.

A motor 290 that first rotates the first unit 210 with respect to the first arm 31 may be additionally provided in the first unit 210. Likewise, the second unit 220, the third unit 230, and the fourth unit 240 may each be provided with a motor 290 that performs the second rotation, third rotation, and fourth rotation.

The second unit ( At least one of the third unit 220 and 230 may be bent.

For example, the second unit 220 may be formed in an 'L' shape, and the third unit 230 may be formed in an 'I' shape. If the third unit 230 is formed at the end of the second unit 220, it may have an overall 'U' shape. According to this, the camera 201 can be placed at the same point on the second virtual axis regardless of the first rotation, second rotation, and third rotation. When the camera 201 is placed at a certain position on the second virtual axis, position control of the camera 201, which rotates in multiple stages through first rotation, second rotation, third rotation, and fourth rotation, can be simplified.

At this time, the acquisition unit 110, the processing unit 130, and the control unit 150 may be provided in the first unit 210.

Let's look at the processing process of the processing unit 130 in detail.

Figure 6 is a schematic diagram showing the positional relationship between the reference coordinate system of the robot and the coordinate system of the camera 201. Figure 7 is a schematic diagram showing the process of obtaining the transformation matrix T.

The position of target s read from the camera 201 of the gimbal 200 When defining , a process of converting the definition value into the robot reference coordinate system is required. Therefore, the following equation 1 is required:

[Formula 1]

In Equation 1, the relationship from the reference coordinate system to the rotation joint near the camera 201 (joint near the camera 201, first joint) can be expressed by the kinematics of the manipulator. Joints may be formed at both ends of the first arm 31, respectively. At this time, the first joint may represent a joint of the first arm 31 that is closer to the base of the robot arm 30.

And, the position to the target point based on the camera 201 Can be given by the camera 201 mounted on the gimbal 200. Therefore, in this system, the relationship between the coordinate system of the robot joint located near the camera 201 and the coordinate system of the camera 201 is needed. To obtain, Equation 2 below was expressed as shown in Figure 6. The first joint corresponding to the joint near the camera 201 is expressed as joint A.

[Formula 2]

When the coordinate system of the first joint is {joint A}, a transformation matrix in three-dimensional space is used to obtain the position and direction to the camera 201. ) can be used. The upper left subscript indicates the reference coordinate system, and the lower left subscript indicates the target coordinate system. thus, represents the position and orientation of the {i} coordinate system relative to the {i-1} coordinate system.

[Formula 3]

Here, ‘joint’ may represent the first joint. R of the transformation matrix represents a 3 by 3 rotation matrix, and P represents a 3 by 1 position vector.

In formula 3: is a value determined through the kinematics of the gimbal 200 and can be determined by the user considering the design value of the gimbal 200 and the camera 201. is determined based on the position of the link where the user attaches the gimbal 200.

Transformation matrix according to the attachment position of the gimbal 200 The process of obtaining is shown in Figure 7.

Step1: The processing unit 130 may calculate the distance dz between the first joint and the gimbal 200 (first unit 210) on the z-axis perpendicular to the longitudinal direction of the first arm 31. In the case of Figure 7, dz = 0.

Step2: The processing unit 130 may calculate a difference value θz between the angle of the first joint and the angle of the gimbal 200, based on the z-axis perpendicular to the longitudinal direction of the first arm 31. In the case of FIG. 7 where the first arm 31 is straight, θz = 0.

Step3: The processing unit 130 may calculate the distance dx between the first joint and the gimbal 200 (first unit 210) on the x-axis parallel to the longitudinal direction of the first arm 31.

Step4: The processing unit 130 may calculate a difference value θx between the angle of the first joint and the angle of the gimbal 200, based on the x-axis parallel to the longitudinal direction of the first arm 31.

By going through the process of Figure 7, Equation 4 can be obtained. Can be used to control the robot or gimbal 200.

[Formula 4]

transformation matrix Through Equation 5 below, the target s obtained from the camera 201 is based on a reference coordinate system (a coordinate system based on the base of the robot arm 30 or the body 90 supporting the robot arm 30). The location and direction can be calculated.

[Formula 5]

Figure 8 is a flowchart showing the control method of the present invention.

The control method of FIG. 8 may be performed by the control device 100 shown in FIG. 1.

The control method may include an acquisition step (S 510), a processing step (S 520), and a control step (S 530).

In the robot arm 30 (arm), a gimbal 200 may be installed on the first arm 31 having a first joint, and a target s, which is an image target of the camera 201 installed on the gimbal 200, may be provided. At this time, the acquisition step (S 510) may acquire the first relationship corresponding to the positional relationship between the robot arm 30 (using a coordinate system based on the robot arm 30 or the body 90) and the first joint. there is.

In the acquisition step (S510), a second relationship corresponding to the positional relationship between the first joint and the gimbal 200 may be acquired.

In the acquisition step (S510), a third relationship corresponding to the positional relationship between the gimbal 200 and the camera 201 may be acquired.

In the acquisition step (S510), a fourth relationship corresponding to the positional relationship between the camera 201 and the target may be acquired.

The processing step S 520 is a first relationship (e.g. ), a second relationship (e.g. ), third relationships (e.g. ), fourth relation (e.g. ), the target coordinates corresponding to the position of the target can be converted into the reference coordinate system of the robot arm 30.

In the control step (S530), the gimbal 200 can be controlled so that the camera 201 faces the target coordinates regardless of the operation of the robot arm 30 using the target coordinates converted to a reference coordinate system.

Figure 9 is a flowchart for obtaining a transformation matrix T representing the position and orientation relationship between the joint and the gimbal 200.

9 may be performed in the processing step (S520).

On the z-axis perpendicular to the longitudinal direction of the first arm 31, the distance dz between the first joint and the gimbal 200 (first unit 210) can be calculated (S 521). From another perspective, dz may mean a movement distance that causes the x-axis of the first joint and the x-axis of the gimbal 200 to intersect.

Based on the z-axis perpendicular to the longitudinal direction of the first arm 31, the difference value θz between the angle of the first joint and the angle of the gimbal 200 can be calculated (S 523). Looking at it from another perspective, θz may mean a rotation angle that matches the x-axis of the first joint and the x-axis of the gimbal 200.

On the x-axis parallel to the longitudinal direction of the first arm 31, the distance dx between the first joint and the gimbal 200 (first unit 210) can be calculated (S 525). From another perspective, dx may mean a movement distance that causes the z-axis of the first joint and the z-axis of the gimbal 200 to intersect.

Step4: The processing unit 130 may calculate a difference value θx between the angle of the first joint and the angle of the gimbal 200, based on the x-axis parallel to the longitudinal direction of the first arm 31 (S 527). From another perspective, θx may mean a rotation angle that matches the z-axis of the first joint and the z-axis of the gimbal 200.

Figure 10 is a schematic diagram showing a robot equipped with a gimbal 200.

The control unit 150 shown in FIG. 10 can control the gimbal 200. The robot arm 30 or manipulator may be controlled by the controller 160. At this time, the control unit 150 and the controller 160 may be formed integrally. If the control unit 150 is formed separately from the controller 160, the control unit 150 may be built into the gimbal 200. In this case, the robot and controller 160 may be already installed. In other words, if the control device 100 of the present invention equipped with the gimbal 200 and the control unit 150 is added to the existing robot and controller 160, an environment in which target s can be continuously monitored will be provided. You can.

11 is a diagram illustrating a computing device according to an embodiment of the present invention. The computing device TN100 of FIG. 11 may be a device described herein (eg, control device 100, etc.).

In the embodiment of FIG. 11 , the computing device TN100 may include at least one processor TN110, a transceiver device TN120, and a memory TN130. Additionally, the computing device TN100 may further include a storage device TN140, an input interface device TN150, an output interface device TN160, etc. Components included in the computing device TN100 may be connected by a bus TN170 and communicate with each other.

The processor TN110 may execute a program command stored in at least one of the memory TN130 and the storage device TN140. The processor TN110 may refer to a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor on which methods according to embodiments of the present invention are performed. Processor TN110 may be configured to implement procedures, functions, and methods described in connection with embodiments of the present invention. The processor TN110 may control each component of the computing device TN100.

Each of the memory TN130 and the storage device TN140 can store various information related to the operation of the processor TN110. Each of the memory TN130 and the storage device TN140 may be comprised of at least one of a volatile storage medium and a non-volatile storage medium. For example, the memory TN130 may be comprised of at least one of read only memory (ROM) and random access memory (RAM).

The transceiving device TN120 can transmit or receive wired signals or wireless signals. The transmitting and receiving device (TN120) can be connected to a network and perform communication.

Meanwhile, the embodiments of the present invention are not only implemented through the apparatus and/or method described so far, but may also be implemented through a program that realizes the function corresponding to the configuration of the embodiment of the present invention or a recording medium on which the program is recorded. This implementation can be easily implemented by anyone skilled in the art from the description of the above-described embodiments.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements made by those skilled in the art using the basic concept of the present invention defined in the following claims are also possible. It falls within the scope of invention rights.

10...Shooting means 20...Display
30...robot arm 31...first arm
90...Body 91...Movement
100...Control unit. 110...Acquisition Department
130...processing unit 150...control unit
160...Controller 200...Gimbal
201...camera 210...first unit
219...through hole 220...2nd unit
230...3rd unit 240...4th unit
250...removable means 260...holder
290...motor

Claims (4)

When a gimbal is installed on a first arm having a first joint in a robot arm, and a target for filming by a camera installed on the gimbal is provided,
an acquisition unit that acquires a second relationship corresponding to a positional relationship between the first joint and the gimbal;
a processing unit that converts target coordinates corresponding to the position of the target into a reference coordinate system of the robot arm using the second relationship;
It includes a control unit that controls the gimbal using the target coordinates converted to the reference coordinate system,
The control unit is a control device that controls the gimbal to direct the camera to the target coordinates regardless of the operation of the robot arm.
According to paragraph 1,
A first unit, a second unit, a third unit, and a fourth unit are provided,
The first unit is installed on the first arm to be capable of first rotation about a first virtual axis extending along the longitudinal direction of the first arm,
The second unit is installed in the first unit to be capable of second rotation about a second virtual axis extending along a direction perpendicular to the first virtual axis,
The third unit is installed in the second unit to enable a third rotation about a third virtual axis extending along a direction perpendicular to the second virtual axis,
The fourth unit is installed in the third unit to enable a fourth rotation about a fourth virtual axis extending along a direction perpendicular to the third virtual axis,
The fourth unit is provided with attachment and detachment means for attaching and detaching the camera,
The second unit and the fourth unit are configured such that the camera mounted on the fourth unit through the attachment/detachment means is disposed at the same point on the second virtual axis regardless of the first rotation, the second rotation, and the third rotation. A control device in which at least one of the three units is bent.
According to paragraph 1,
The acquisition unit acquires a first relationship corresponding to a positional relationship between the robot arm and the first joint,
The acquisition unit acquires a third relationship corresponding to the positional relationship between the gimbal and the camera,
The acquisition unit acquires a fourth relationship corresponding to the positional relationship between the camera and the target,
The processing unit is a control device that converts the target coordinates into the reference coordinate system using the first relationship, the second relationship, the third relationship, and the fourth relationship.
In a control method performed by a control device,
When a gimbal is installed on a first arm having a first joint in a robot arm, and a target for filming by a camera installed on the gimbal is provided,
acquisition phase;
processing steps;
It includes a control step;
The acquisition step is,
Obtaining a first relationship corresponding to a positional relationship between the robot arm and the first joint,
Obtaining a second relationship corresponding to the positional relationship between the first joint and the gimbal,
Obtaining a third relationship corresponding to the positional relationship between the gimbal and the camera,
Obtaining a fourth relationship corresponding to a positional relationship between the camera and the target,
The processing steps are:
Converting target coordinates corresponding to the position of the target into a reference coordinate system of the robot arm using the first relationship, the second relationship, the third relationship, and the fourth relationship,
The control step is,
A control method for controlling the gimbal so that the camera faces the target coordinates regardless of the operation of the robot arm, using the target coordinates converted to the reference coordinate system.

Images