JP7504374B2 - ROBOT CONTROL DEVICE, ROBOT CONTROL METHOD, AND ROBOT CONTROL SYSTEM - Google Patents

Description

The present invention relates to a robot control device, a robot control method, and a robot control system for controlling a robot.

Systems that use robots to perform tasks such as arranging products in work areas such as stores are being introduced. Patent Document 1 discloses a system in which a robot performs tasks such as arranging products on shelves.

JP 2018-147138 A

Due to constraints on the location where the robot works or on the structure of the robot, there are cases where the robot must work using the arm opposite to the operator's dominant arm. When an operator operates a robot using their non-dominant arm, there is a problem in that work efficiency decreases.

The present invention was made in consideration of these points, and aims to improve the work efficiency when a robot is made to perform a task using the non-dominant arm of an operator who remotely controls the robot.

The robot control device of the first aspect of the present invention has a data receiving unit that receives captured image data generated by an imaging unit provided in the robot, a display processing unit that displays an inverted image based on inverted image data obtained by left-right inverting the captured image data on a display device visible to an operator of the robot, a motion identification unit that identifies a motion of the operator, and a motion control unit that controls the robot by transmitting to the robot control data for the robot to perform a motion obtained by left-right inverting the motion of the operator identified by the motion identification unit.

The operation control unit may transmit to the robot, as the control data, inverted operation information obtained by left-right inverting at least one of coordinate information indicating the position of one or more body parts of the operator detected using an operating device used by the operator to operate the robot, or rotation information indicating the direction of rotation of the one or more body parts.

The data receiving unit may further receive at least one of robot status data, which may be haptic data indicating haptics detected by the robot, sound data indicating sounds collected by the robot, or joint status data indicating the status of the robot's joints, and the robot control device may further have a status data processing unit that transmits inverted status data, which is the robot status data that is inverted from left to right, to an operating device used by the operator.

The apparatus may further include a setting unit that accepts a setting of an inversion mode that transmits to the robot the control data for the robot to perform an action that is a left-right mirror image of the action of the operator, and when the setting unit accepts the setting of the inversion mode, the action control unit may transmit to the robot the control data for the robot to perform an action that is a left-right mirror image of the action of the operator.

The robot control device may further have a setting unit that accepts the setting of the operator's dominant arm, and the motion control unit may determine whether or not to operate in an inversion mode in which the control data for the robot to perform an action that is a left-right inversion of the operator's action is transmitted to the robot based on the relationship between the operator's dominant arm and the arm used in the task performed by the robot.

When the motion control unit determines that the robot will perform a task with the left hand, it may operate in an inverted mode in which it transmits to the robot the control data for the robot to perform a task that is a left-right inverted version of the motion of the operator, and when the motion control unit determines that the robot will perform a task with the right hand, it may operate in a non-inverted mode in which it transmits to the robot the control data for the robot to perform a task that is not a left-right inverted version of the motion of the operator.

The operation control unit may operate the robot without using either the right arm or the left arm based on the result of judging whether or not it is necessary to use each of the robot's right arm or left arm.

The robot control device may further have a work content specification unit that specifies the content of the work to be performed by the robot, and the motion control unit may determine which of the right arm and the left arm to use based on the content of the work specified by the work content specification unit.

When the motion control unit determines that the right arm of the robot will not be used, the motion control unit may transmit control data to the robot for the robot to perform a motion that is a left-right inversion of the motion of the operator.

The display processing unit may display on the display device at least one of information indicating that the robot is operating in an inversion mode in which the operator's movements are reversed left and right to operate the robot, or information indicating that the robot is operating in a movement restriction mode in which one arm of the robot is not used.

The display processing unit may determine an order of a plurality of tasks so that the tasks of operating the robot in a reverse mode in which control data for the robot to perform an action that is a left-right reverse of the action of the operator is transmitted to the robot, and information indicating the determined order may be displayed on the display device.

The display processing unit may identify a text area containing text in the captured image data, and display an image of the text area on the display device without being mirror-flipped.

The robot control system of the second aspect of the present invention includes an operating device used by an operator who operates a robot, and a robot control device that controls the robot, and has a display processing unit that displays an inverted image based on inverted image data obtained by left-right inverting captured image data generated by an imaging unit provided in the robot on a display device visible to the operator of the robot, a motion identification unit that identifies a motion of the operator, and a motion control unit that controls the robot to perform a motion that is the left-right inverted version of the motion of the operator identified by the motion identification unit.

The robot control method of the third aspect of the present invention includes the steps of receiving captured image data generated by an imaging unit provided in a robot, executed by a computer, displaying an inverted image based on inverted image data obtained by left-right inverting the captured image data on a display device visible to an operator of the robot, identifying a motion of the operator, and controlling the robot by transmitting control data to the robot for the robot to perform a motion obtained by left-right inverting the identified motion of the operator.

The present invention has the effect of improving work efficiency when a robot is made to perform a task using the non-dominant arm of an operator who remotely controls the robot.

FIG. 1 is a diagram showing an overview of a robot control system. FIG. 13 is a diagram for explaining a reversal mode. FIG. 2 is a diagram illustrating a functional configuration of a robot control device. FIG. 11 is a diagram showing an example of data indicating a relationship between a task content and a holding means used by a robot. FIG. 13 is a diagram showing an example of data indicating on which arm each holding means of the robot is provided. FIG. 4 is a sequence diagram showing a processing flow in the robot control system. 10 is a flowchart showing an example of an operation performed by an operation control unit to determine an operation mode.

[Overview of Robot Control System S]
1 is a diagram showing an overview of a robot control system S. The robot control system S is a system that provides a telexistence environment in which an operator U can operate a remote object in real time while making the user U feel as if the object is nearby.

The robot control system S has a robot control device 1, an operation device 2, and a robot 3. The robot control device 1 is a device with a cockpit function for controlling the robot 3 in response to the operation of the operator U, and is, for example, a computer. The operation device 2 is a device worn by the operator U, and includes a display device 21 and an operation device 22. The display device 21 has a display visible to the operator U, and is, for example, a goggle-type display device. The display device 21 may generate sound from a speaker provided on the right side and a speaker provided on the left side.

The operating device 22 is a device used by the operator U to operate the robot 3. The operating device 22 has, for example, a reflective member that reflects infrared light irradiated from the robot control device 1 so that the robot control device 1 can detect the movement of parts of the operator U's body (e.g., the hands and arms). The operating device 22 may have an acceleration sensor that detects the movement of the operator U. The operating device 22 also has an element for generating heat, pressure, vibration, etc. corresponding to the state of the robot 3 so that the operator U can sense the state of the robot 3.

The robot 3 operates based on control data received from the robot control device 1 via the network N. The robot 3 also has a camera capable of photographing the surroundings, and transmits captured image data generated by the camera to the robot control device 1. The robot 3 also transmits at least one of the following robot status data to the robot control device 1: tactile data indicating touch detected by the robot 3, sound data indicating sounds collected by the robot 3, or joint status data indicating the status of the robot 3's joints. As an example, the robot 3 according to this embodiment performs the task of arranging products on shelves in a store where many products are sold lined up on shelves, but the location where the robot 3 is installed and the content of the task are arbitrary.

The robot 3 has multiple types of mechanisms as holding means used to hold products. For example, the robot 3 has a clamping section at the end of at least one arm for holding the product by pinching it. The clamping section includes multiple actuators with variable angles, like human fingers. The robot 3 also has a suction section at the end of at least one arm for holding the product by suctioning it. The suction section has an opening formed therein for sucking in air.

The robot control device 1 displays an image based on the captured image data received from the robot 3 on the display device 21. The robot control device 1 also heats and vibrates the operation device 22 based on the robot status data received from the robot 3. With this configuration, the operator U can experience the environment around the robot 3 while in a location away from the robot 3, as if he or she were inside the robot 3.

The robot control device 1 also identifies the motion content of the operator U based on information indicating the motion content of the operator U transmitted from the operation device 22. The information indicating the motion content of the operator U is, for example, information indicating the movement of each part of the operator U. For example, the robot control device 1 irradiates infrared light toward the operator U, and identifies the motion content of the operator U based on the light of the irradiated infrared light reflected by the operation device 22. The robot control device 1 creates control data for operating the robot 3 based on the identified motion content, and controls the motion of the robot 3 by transmitting the created control data to the robot 3.

However, there are cases where the robot 3 must perform a task using the arm opposite to the dominant arm of the operator U. For example, even though the operator U is right-handed, the robot 3 may perform a task that is difficult to perform using the right arm. Also, if the functions of the right arm and the left arm of the robot 3 are different, depending on the task performed by the robot 3, the robot 3 may perform the task using the arm opposite to the dominant arm of the operator U.

The robot control device 1 according to this embodiment can switch between a non-inversion mode and an inversion mode so that the operator U can operate using his/her dominant arm regardless of the relationship between the dominant arm of the operator U and the arm used by the robot 3. The non-inversion mode is an operation mode when, for example, the dominant arm of the operator U and the arm used by the robot 3 match. In the non-inversion mode, the robot control device 1 transmits the captured image data transmitted from the robot 3 to the display device 21 without inverting it horizontally. The robot control device 1 also transmits the robot status data transmitted from the robot 3 to the operation device 22. The robot control device 1 also transmits control data that directly reflects the operation content of the operator U to the robot 3.

The inversion mode is an operation mode in which, for example, the dominant arm of the operator U does not match the arm used by the robot 3. In the inversion mode, the robot control device 1 transmits inverted image data obtained by inverting the left and right of the captured image data transmitted from the robot 3 to the display device 21. The robot control device 1 also transmits inverted state data obtained by inverting the left and right of the robot state data transmitted from the robot 3 to the operation device 22. The robot control device 1 also controls the robot 3 so that the arm opposite to the dominant arm (e.g. the left arm) of the robot 3 moves in response to the operator U operating the robot with his dominant arm (e.g. the right arm). Specifically, in the inversion mode, the robot control device 1 transmits control data to the robot 3 for the robot 3 to perform an action obtained by inverting the left and right of the action of the operator U.

Figure 2 is a diagram for explaining the inversion mode. Figure 2(a) is a schematic diagram of a product shelf on which the robot 3 is working. Figure 2(a) shows a state in which multiple bottles, indicated by solid lines, have been arranged, and bottles need to be arranged in the position indicated by dashed lines. In this way, when products need to be arranged at the right edge of the product shelf, it is difficult for the robot 3 to use its right arm to arrange the products. In particular, if there is a wall to the right of the robot 3, it may be physically impossible to move the right arm.

Figure 2(b) shows the inverted image data displayed on the display device 21 in this state. In Figure 2(b), the image is inverted left to right except for the text, and the position indicated by the dashed line is at the left edge. When the operator U looks at this image and uses his right arm to place a product at the position indicated by the dashed line, the robot control device 1 transmits to the robot 3 control data created by inverting the operator U's action left to right. As a result, in response to the operator U using his right arm to place the product at the position indicated by the dashed line in Figure 2(b), the robot 3 uses his left arm to place the product at the position indicated by the dashed line in Figure 2(a).

By operating the robot control device 1 in this manner, the operator U can perform work using the arm opposite to the dominant arm of the robot 3 while feeling as if the robot 3 is also working with the dominant arm of the operator U, thereby improving work efficiency.
The configuration and operation of the robot control device 1 will be described in detail below.

[Configuration of robot control device 1]
3 is a diagram showing the functional configuration of the robot control device 1. The robot control device 1 has a first communication unit 11, a second communication unit 12, a movement detection unit 13, an operation unit 14, a storage unit 15, and a control unit 16. The control unit 16 has a data receiving unit 161, a display processing unit 162, a status data processing unit 163, an action identification unit 164, a setting unit 165, a task content identification unit 166, and an action control unit 167.

The first communication unit 11 has a communication controller for transmitting and receiving various data to and from the robot 3. The first communication unit 11 receives captured image data transmitted by the robot 3, and inputs the received captured image data to the data receiving unit 161. The first communication unit 11 also transmits control data input from the operation identification unit 164 to the robot 3.

The second communication unit 12 has a communication controller for transmitting the inverted image data input from the display processing unit 162 to the display device 21 and transmitting the inverted state data input from the state data processing unit 163 to the operation device 22. The second communication unit 12 transmits inverted image data based on the captured image data received from the robot 3 to the display device 21 and transmits inverted state data based on the state data received from the robot 3 to the operation device 22, using a short-range wireless communication method such as Wi-Fi (registered trademark) or Bluetooth (registered trademark).

The movement detection unit 13 detects the movement of the operator U wearing the operation device 22, and inputs data indicating the detected movement to the action identification unit 164. The movement detection unit 13 has, for example, a light source that emits infrared light, and a light sensor that detects light reflected by the operation device 22.

The operation unit 14 includes a display, a mouse, a keyboard, or a touch panel as a device for accepting setting operations by the operator U or an assistant of the operator U. The operation unit 14 may be a device of another type, such as a head-mounted display that displays a virtual reality image or a device that can be operated with the feet. The operation unit 14 inputs data indicating the contents of the input settings to the operation control unit 167.

The storage unit 15 includes storage media such as a ROM (Read Only Memory), a RAM (Random Access Memory), and a hard disk. The storage unit 15 stores programs executed by the control unit 16. The storage unit 15 also stores various data necessary for the control unit 16 to operate the robot 3.

Figures 4 and 5 are diagrams showing an example of data stored in the memory unit 15. Figure 4 is a diagram showing an example of data showing the relationship between the work content and the holding means used by the robot 3. In the example shown in Figure 4, it is assumed that the clamping unit is used for the work of replenishing the shelves with PET bottles, the suction unit is used for the work of replenishing canned drinks or rice balls, and both the clamping unit and the suction unit are used for the work of replenishing boxed lunches.

Figure 5 shows an example of data indicating which arm each holding means of the robot 3 is provided on. A robot 3 with a robot ID of "A-01" (for example, the first robot in store A), which is information for identifying the robot 3, has a clamping unit provided on its right arm and a suction unit provided on its left arm. A robot 3 with a robot ID of "B-01" (for example, the first robot in store B) has a clamping unit provided on its left arm and a suction unit provided on its right arm.

Next, the control unit 16 will be described in detail with reference to FIG. 3. The control unit 16 executes the programs stored in the storage unit 15, thereby functioning as a data receiving unit 161, a display processing unit 162, a status data processing unit 163, an operation identification unit 164, a setting unit 165, a task content identification unit 166, and an operation control unit 167.

The data receiving unit 161 receives captured image data generated by an imaging unit provided in the robot 3 via the first communication unit 11. The data receiving unit 161 inputs the received captured image data to the display processing unit 162.

The data receiving unit 161 also receives at least any one of the following robot status data: haptic data indicating the haptics detected by the robot 3, sound data indicating the sounds collected by the robot 3, or joint status data indicating the status of the joints of the robot 3. The data receiving unit 161 inputs the received robot status data to the status data processing unit 163.

The display processing unit 162 causes the display device 21 to display various types of information via the second communication unit 12. For example, the display processing unit 162 causes the display device 21 to display an image based on the captured image data input from the data receiving unit 161. For example, the display processing unit 162 determines whether to operate in non-inversion mode or inversion mode based on information indicating the operation mode notified from the operation control unit 167 described later.

When the display processing unit 162 is operating in the non-inversion mode, it transmits the captured image data to the display device 21 as is, thereby causing the display device 21 to display an uninverted captured image. When the display processing unit 162 is operating in the inversion mode, it causes the display device 21 to display an inverted image based on inverted image data obtained by inverting the captured image data input from the data receiving unit 161 left and right. The display processing unit 162 creates inverted image data, for example, by swapping the pixel values of each pixel on the right side of the captured image data with the pixel values of the corresponding pixels on the left side, based on the center line in the left-right direction of the captured image data.

At this time, as shown in FIG. 2B, the display processing unit 162 may identify a text area containing text in the captured image data, and display the image of the text area on the display device 21 without left-right inversion. Specifically, the display processing unit 162 creates inverted image data in which the text contained in the identified text area is converted into mirror text, and transmits the inverted image data to the display device 21. By operating the display processing unit 162 in this manner, the operator U can work while visually recognizing the text in the same state as in non-inverted mode, thereby preventing a decrease in work efficiency in, for example, checking the expiration date before arranging products on a shelf.

In addition to the captured image data, the display processing unit 162 may display on the display device 21 various types of information to assist the operation of the operator U. For example, the display processing unit 162 displays on the display device 21 at least one of information indicating that the robot 3 is operating in an inverted mode or information indicating that the robot 3 is operating in a motion restriction mode in which one arm of the robot 3 is not used. By having the display processing unit 162 display such information on the display device 21, it is possible to prevent the operator U from becoming confused when, for example, the inverted mode and non-inverted mode are automatically switched while the operator U is operating the robot 3.

The display processing unit 162 may also determine the order of a plurality of tasks so that the tasks of operating the robot 3 in the inversion mode are consecutive, and display information indicating the determined order on the display device 21. As an example, a case will be described in which the robot 3 needs to perform the tasks of refilling plastic bottles, refilling canned drinks, refilling rice balls, and refilling lunch boxes shown in FIG. 4.

When the robot ID of the robot 3 used is A-01 shown in FIG. 5, the left arm is used for the tasks of refilling canned drinks and refilling rice balls, which require the use of the suction part. Therefore, if the operator U's dominant arm is the right arm, the robot control device 1 operates in the inverted mode when the robot 3 performs the tasks of refilling canned drinks and refilling rice balls. Therefore, the display processing unit 162 decides to perform the tasks of refilling canned drinks and refilling rice balls consecutively, and displays this on the display device 21 so that the operator U can recognize that the tasks will be performed in this order. By operating in this manner, the display processing unit 162 can prevent the operating mode from being frequently switched, reducing the frequency with which captured image data is switched and improving operability for the operator U.

The status data processing unit 163 transmits data based on the robot status data input from the data receiving unit 161 to the operation device 22 via the second communication unit 12. The status data processing unit 163, like the display processing unit 162, determines whether to operate in non-inverted mode or inverted mode based on, for example, information indicating the operation mode notified from the operation control unit 167 described later.

While operating in the non-inverted mode, the status data processing unit 163 transmits the robot status data input from the data receiving unit 161 to the operation device 22 as is, thereby enabling the operator U to feel the touch, sound, and joint state detected on the right side of the robot 3 (right arm or right ear, etc.) on the right side.While operating in the inverted mode, the status data processing unit 163 transmits inverted status data, which is the robot status data input from the data receiving unit 161 that is inverted left and right, to the operation device 22, thereby enabling the operator U to feel the touch, sound, and joint state detected on the left side of the robot 3 (left arm or left ear, etc.) on the right side.

The motion identification unit 164 identifies the motion of the operator U. For example, the motion identification unit 164 controls the motion detection unit 13 so that the motion detection unit 13 irradiates infrared light, and creates motion information indicating the motion of the operator U based on the reflected light input from the motion detection unit 13. The motion identification unit 164 creates motion information such as position information indicating the relative position of each part of the operator U's fingers, hands, arms, head, feet, etc. relative to a reference position, speed information indicating the speed of each part, or acceleration information indicating the acceleration of each part. The reference position is, for example, the position of each part when the operator U takes a predetermined posture (for example, a posture in which the arms are stretched out and placed on both sides of the torso) at the time when the operator U starts operating the robot 3. The motion identification unit 164 notifies the motion control unit 167 of the created motion information.

The motion identification unit 164 also identifies the center line of the operator U in the left-right direction, and notifies the motion control unit 167 of center line information indicating the position of the identified center line. For example, the motion identification unit 164 identifies as the center line a line connecting the center position of the operator U's head and the midpoints of the left and right feet when the operator U is in a predetermined posture at the time when the operator U starts operating the robot 3. The identified center line is used by the motion control unit 167 when it horizontally inverts the motion information in the inversion mode.

The setting unit 165 accepts various settings based on the contents of the setting operation performed in the operation unit 14. The setting unit 165 may accept various settings based on information transmitted from the display device 21 or the operation device 22 via the second communication unit 12. In this case, the setting unit 165 displays a screen showing candidate setting contents on the display device 21 via, for example, the display processing unit 162, and accepts the setting contents selected by the line of sight of the operator U. The setting unit 165 accepts, for example, a setting of the inversion mode. The setting unit 165 notifies the operation control unit 167 that the inversion mode has been set.

The setting unit 165 also accepts settings for the work content to be performed by the robot 3. The work content includes information for identifying the product on which the robot 3 is to work, as shown in FIG. 4, for example. The setting unit 165 notifies the work content identification unit 166 of the set work content.

The setting unit 165 also accepts the setting of the dominant arm of the operator U. Specifically, the setting unit 165 accepts the setting of whether the operator U is right-handed or left-handed. Furthermore, the setting unit 165 may accept the setting of the arm to be used by the robot 3. The setting unit 165 notifies the operation control unit 167 of information for identifying the set dominant arm of the operator U and information for identifying the arm to be used by the robot 3.

The work content identification unit 166 identifies the content of the work to be performed by the robot 3. The work content identification unit 166 identifies the content of the work to be performed by the robot 3, for example, based on the work content notified from the setting unit 165. The work content identification unit 166 may identify the content of the work by identifying the product to be the subject of the work based on the image of the product indicated in the captured image data received via the data receiving unit 161. For example, when the captured image data includes image data of canned drinks, the work content identification unit 166 identifies that the work content is refilling canned drinks. The work content identification unit 166 notifies the operation control unit 167 of the identified content of the work. The work content identification unit 166 may identify the content of the work to be performed based on a list of work contents previously stored in the memory unit 15.

The operation control unit 167 creates control data for controlling the robot 3 based on the operation information input from the operation identification unit 164, and controls the robot 3 by transmitting the created control data to the robot 3. In the non-inversion mode, the operation control unit 167 transmits, for example, via the first communication unit 11, control data that is packet data in which the robot ID of the robot 3 is added to the operation information. The operation control unit 167 may transmit control data to which time information indicating the time at which the operation control unit 167 received the operation information from the operation identification unit 164 is added, so that the robot 3 can execute an operation based on the control data at an appropriate timing.

When the setting unit 165 accepts the setting of the inversion mode, the motion control unit 167 controls the robot 3 by transmitting to the robot 3 control data for the robot 3 to perform a motion that is a left-right inversion of the motion of the operator U identified by the motion identification unit 164. Specifically, the motion control unit 167 creates inversion motion information by left-right inverting information indicating the left-right position in the motion information input from the motion identification unit 164, and transmits control data that is packet data in which the robot ID of the robot 3 is added to the inversion motion information. More specifically, the motion control unit 167 transmits at least one of coordinate information indicating the position of one or more parts of the operator U detected using the operation device 22, or rotation information indicating the rotation direction of one or more parts, to the robot 3 as control data including inversion motion information that is left-right inverted based on the center line indicated by the center line information notified from the motion identification unit 164.

The operation control unit 167 may determine whether to operate in the inverted mode or the non-inverted mode based on the content of the work notified by the work content identification unit 166, regardless of the setting content of the operation mode notified by the setting unit 165. The operation control unit 167 identifies which holding means is used for the work performed by the robot 3, for example, by referring to data showing the relationship between the work content and the holding means shown in FIG. 4. Then, the operation control unit 167 identifies whether the identified holding means is provided on the right arm or the left arm of the robot 3, by referring to data showing the position of the holding means shown in FIG. 5. For example, when the work content is refilling canned drinks, the suction unit is used as the holding means, and when the robot ID of the robot 3 is A-01, the operation control unit 167 identifies that the left arm on which the suction unit is provided is to be used.

When the operation control unit 167 determines that the robot 3 will perform a task with the left hand, it decides to operate in an inverted mode in which control data is sent to the robot 3 for the robot 3 to perform a task that is a left-right inverted version of the operator U's motion. When the operation control unit 167 determines that the robot 3 will perform a task with the right hand, it decides to operate in a non-inverted mode in which control data is sent to the robot 3 for the robot 3 to perform a task that is not a left-right inverted version of the operator U's motion. The operation control unit 167 also decides to operate in the non-inverted mode when it determines that it is necessary to use both the right and left arms, such as in the lunch box refilling task shown in FIG. 4.

The operation control unit 167 may determine whether or not to operate in an inverted mode in which control data is sent to the robot 3 for the robot 3 to perform an operation in which the operation of the operator U is inverted left and right based on the relationship between the dominant arm of the operator U and the arm used by the robot 3 in the task performed by the robot 3. Specifically, the operation control unit 167 operates in the inverted mode when the dominant arm of the operator U does not match the arm used by the robot 3, and operates in the non-inverted mode when the dominant arm of the operator U matches the arm used by the robot 3. By operating the operation control unit 167 in this manner, the robot control device 1 can improve the operability of the operator U regardless of the task, even if the operator U is right-handed or left-handed.

The means by which the operation control unit 167 identifies the dominant arm of the operator U is arbitrary, but the operation control unit 167 identifies the dominant arm based on information set by the setting unit 165 via the operation unit 14 when the operator U starts operating the robot 3, for example. The operation control unit 167 may identify the dominant arm by reading out the dominant arm information stored in the storage unit 15 in association with the operator ID of the logged-in operator U from among multiple pieces of dominant arm information stored in the storage unit 15 in association with information for identifying the operator in advance (for example, an operator ID).

The operation control unit 167 may operate the robot 3 in an operation restriction mode in which one of the right arm and the left arm is not used, based on the result of judging whether or not it is necessary to use the right arm or the left arm of the robot 3. The operation control unit 167 identifies the arm to be used, either the right arm or the left arm of the robot 3, based on, for example, the content of the work notified by the work content identification unit 166 or the setting content notified by the setting unit 165. The operation control unit 167 may limit the operation of the arm that has been set in advance not to be used by the operator U.

In the movement restriction mode, the movement control unit 167 causes the robot 3 to work with the arm that is determined not to be used extended and attached to the body of the robot 3, or with the arm folded, regardless of the movement content of the operator U. Specifically, if the operator U is right-handed and the robot 3 uses its right arm, even if the operator U moves his left arm, the movement control unit 167 transmits to the robot 3 control data that causes the robot 3 to not move its left arm (i.e., control data indicating that the operator U is not moving his left arm). Similarly, if the operator U is right-handed and the robot 3 uses its left arm, even if the operator U moves his left arm, the movement control unit 167 transmits to the robot 3 control data that causes the robot 3 to not move its right arm (i.e., control data indicating that the operator U is not moving his left arm). By operating the operation control unit 167 in this manner, in cases where moving the unused arm of the robot 3 would put it in a dangerous state, such as colliding with another object or person, the arm of the robot 3 will not move even if the operator U mistakenly moves the unused arm, thereby improving safety.

Regardless of whether the setting unit 165 has accepted the inversion mode setting, when it has determined that the right arm of the robot 3 will not be used (i.e., when it has determined that the left arm of the robot 3 will be used), the operation control unit 167 may decide to operate in the inversion mode and transmit to the robot 3 control data for causing the robot 3 to perform an operation that is a left-right inversion of the operation of the operator U. Since the majority of operators U are generally right-handed, by operating the operation control unit 167 in this manner, the inversion mode is set when the operator U works using the left arm of the robot 3 without setting a dominant arm, thereby improving the work efficiency of the operator U.

[Processing flow in the robot control system S]
Fig. 6 is a sequence diagram showing the flow of processing in the robot control system S. The sequence diagram shown in Fig. 6 starts at the point in time when the operator U starts a predetermined task.

The operation control unit 167 determines the operation mode based on the various methods described above. Here, it is assumed that the operation control unit 167 determines that the robot will operate in the inversion mode (S1). While the robot control device 1 is operating in the inversion mode, the robot 3 creates captured image data (S2) and transmits the created captured image data to the robot control device 1.

The display processing unit 162 creates inverted image data by inverting the captured image data received by the data receiving unit 161 (S3), and transmits the created inverted image data to the display device 21. During this time, the operation device 22 creates motion information indicating the content of the motion of the operator U (S4), and transmits the created motion information to the robot control device 1. The motion control unit 167 creates control data by inverting the positions and movement directions of each part of the operator U indicated by the received motion information (S5), and transmits the created control data to the robot 3. Having received the control data, the robot 3 executes a motion in which the motion of the operator U is inverted left to right (S6).

[Flowchart of Operation Mode Determination Process]
Fig. 7 is a flowchart showing an example of an operation of determining an operation mode by the operation control unit 167. The flowchart shown in Fig. 7 starts with a process in which the operation control unit 167 specifies the operation content to be performed by the robot 3 based on, for example, a notification from the operation content specifying unit 166 (S11).

The motion control unit 167 identifies the dominant arm of the operator U, for example, based on a notification from the setting unit 165 (S12). The motion control unit 167 also identifies the arm to be used by the robot 3, based on the task content notified by the setting unit 165 and the data shown in Figures 4 and 5 (S13). Next, the motion control unit 167 determines whether the dominant arm of the operator U matches the arm to be used by the robot 3 (S14).

When the operation control unit 167 determines that the dominant arm of the operator U and the arm used by the robot 3 match (YES in S14), it decides to operate in the non-inverted mode (S15). On the other hand, when the operation control unit 167 determines that the dominant arm of the operator U and the arm used by the robot 3 do not match (NO in S14), it decides to operate in the inverted mode (S16). The operation control unit 167 creates control data corresponding to the determined operation mode and transmits the created control data to the robot 3 (S17).

Then, the operation control unit 167 monitors whether the work content has changed (S18). If the operation control unit 167 determines that the work content has changed (YES in S18), the process returns to S12 and the operation mode is determined again. On the other hand, if the operation control unit 167 determines that the work content has not changed (NO in S18), the process proceeds to S19 and determines whether to end the operation (S19).

If the operation control unit 167 determines not to end the operation (NO in S19), it returns to S17 and continues the operation of transmitting control data to the robot 3 and controlling the robot 3. If the operation control unit 167 determines to end the operation (YES in S19), it ends the process of determining the operation mode.

[Modification]
In the above description, the robot controller 1 and the robot 3 directly communicate captured image data and control data via the network N, but the robot controller 1 and the robot 3 may transmit and receive captured image data and control data via another device (e.g., a server). In the above description, the robot controller 1 is a device different from the operation device 2, but the robot controller 1 may be built into the display device 21 or the operation device 22.

In the above description, the robot control device 1 is provided on the operation device 2 side relative to the network N, but the robot control device 1 may be provided on the robot 3 side relative to the network N (e.g., inside the robot 3). That is, the robot control device 1 may transmit captured image data to the display device 21 via the network N, or obtain motion information indicating the motion of the operator U from the operation device 22.

In the above description, a case has been exemplified in which a clamping portion is provided on one arm and an adsorption portion is provided on the other arm, but the type and combination of holding means provided on each arm are arbitrary. For example, an adsorption portion may be provided on one arm and an adsorption portion and a clamping portion may be provided on the other arm. Also, at least one of multiple adsorption portions or multiple clamping portions may be provided on one arm.

[Effects of the robot control device 1]
As described above, in the robot control device 1 according to this embodiment, when operating in the inversion mode, the display processing unit 162 causes the display device 21 to display an inverted image based on inverted image data obtained by left-right inverting the captured image data received from the robot 3. In addition, the operation control unit 167 controls the robot 3 by transmitting to the robot 3 control data for causing the robot 3 to perform an operation obtained by left-right inverting the operation of the operator U identified by the operation identification unit 164. By operating the display processing unit 162 and the operation control unit 167 in this manner, when the operator U who remotely operates the robot 3 has the robot 3 perform a task using his/her non-dominant arm, the operator U can operate the robot 3 using his/her dominant arm, thereby improving work efficiency.

Although the present invention has been described above using embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes are possible within the scope of the gist of the invention. For example, all or part of the device can be configured by distributing or integrating functionally or physically in any unit. In addition, new embodiments resulting from any combination of multiple embodiments are also included in the embodiments of the present invention. The effect of the new embodiment resulting from the combination also has the effect of the original embodiment.

Reference Signs List 1 Robot control device 2 Operation device 3 Robot 11 First communication unit 12 Second communication unit 13 Movement detection unit 14 Operation unit 15 Memory unit 16 Control unit 21 Display device 22 Operation device 161 Data receiving unit 162 Display processing unit 163 Status data processing unit 164 Action identification unit 165 Setting unit 166 Work content identification unit 167 Action control unit

Claims (13)

a data receiving unit that receives captured image data generated by an imaging unit provided in the robot;
a display processing unit that displays an inverted image based on inverted image data obtained by horizontally inverting the captured image data on a display device that is visible to an operator of the robot;
A motion identification unit that identifies a motion of the operator;
a motion control unit that controls the robot by transmitting to the robot control data for the robot to perform a motion that is a left-right inversion of the motion of the operator identified by the motion identification unit;
A setting unit that accepts a setting of a dominant hand of the operator;
having
the motion control unit determines whether or not to operate in an inversion mode in which the control data for the robot to perform an action that is a left-right inversion of the action of the operator is transmitted to the robot based on a relationship between the dominant arm of the operator and the arm used in the task performed by the robot;
Robot control device. the operation control unit transmits to the robot, as the control data, inverted operation information obtained by left-right inverting at least one of coordinate information indicating the position of one or more body parts of the operator detected using an operation device used by the operator to operate the robot, or rotation information indicating a rotation direction of the one or more body parts;
The robot control device according to claim 1 . the data receiving unit further receives at least any one of robot status data, which is haptic data indicating a haptic sensation detected by the robot, sound data indicating a sound collected by the robot, or joint status data indicating a state of a joint of the robot;
The robot control device further includes a state data processing unit that transmits inverted state data obtained by inverting the robot state data to an operation device used by the operator.
The robot control device according to claim 1 or 2. a setting unit that receives a setting of an inversion mode in which the control data is transmitted to the robot so that the robot performs an action that is a left-right inversion of the action of the operator;
When the setting unit accepts the setting of the inversion mode, the movement control unit transmits to the robot the control data for the robot to perform an movement obtained by horizontally inverting the movement of the operator.
The robot control device according to any one of claims 1 to 3. the motion control unit operates in an inversion mode in which the control data is transmitted to the robot so that the robot performs a motion in which the motion of the operator is reversed left and right when it is determined that the robot performs a work with the left hand, and operates in a non-inversion mode in which the control data is transmitted to the robot so that the robot performs a motion in which the motion of the operator is not reversed left and right when it is determined that the robot performs a work with the right hand.
The robot control device according to any one of claims 1 to 4 . the operation control unit operates the robot without using one of the right arm and the left arm based on a result of determining whether or not it is necessary to use each of the right arm and the left arm of the robot.
The robot control device according to any one of claims 1 to 5 . A task content specification unit that specifies the task content to be performed by the robot,
The movement control unit determines which arm to use, the right arm or the left arm, based on the content of the work identified by the work content identification unit.
The robot control device according to claim 6 . When it is determined that the right arm of the robot will not be used, the movement control unit transmits to the robot control data for causing the robot to perform a movement obtained by left-right inverting the movement of the operator.
The robot control device according to claim 6 or 7 . The display processing unit causes the display device to display at least one of information indicating that the robot is operating in an inversion mode in which the robot is operated by reversing the motion of the operator left and right, or information indicating that the robot is operating in a motion restriction mode in which one arm of the robot is not used.
The robot control device according to any one of claims 1 to 8 . the display processing unit determines an order of a plurality of tasks so that tasks of operating the robot in a reverse mode in which control data for causing the robot to perform an action obtained by laterally reversing the action of the operator is transmitted to the robot, and causes the display device to display information indicating the determined order.
The robot control device according to any one of claims 1 to 9 . the display processing unit identifies a character area including characters in the captured image data, and causes the display device to display an image of the character area without flipping the image horizontally.
The robot control device according to any one of claims 1 to 10 . The robot control device includes an operating device used by an operator to operate the robot, and a robot control device that controls the robot,
a display processing unit that displays an inverted image based on inverted image data obtained by left-right inverting the captured image data generated by the imaging unit provided in the robot on a display device that is visible to an operator of the robot;
A motion identification unit that identifies a motion of the operator;
a motion control unit that controls the robot to perform a motion that is a left-right inversion of the motion of the operator identified by the motion identification unit;
A setting unit that accepts a setting of a dominant hand of the operator;
having
the motion control unit determines whether or not to operate in an inverted mode in which the motion of the operator is inverted left and right based on a relationship between a dominant arm of the operator and an arm used in a task performed by the robot;
Robot control system. The computer executes
receiving captured image data generated by an imaging unit provided in the robot;
displaying an inverted image based on inverted image data obtained by horizontally inverting the captured image data on a display device that is visible to an operator of the robot;
identifying an action of the operator;
a step of controlling the robot by transmitting control data to the robot for causing the robot to perform a motion that is a left-right inverse of the identified motion of the operator;
A robot control method comprising:
accepting a setting of a dominant hand of the operator;
determining whether to operate in an inversion mode in which the control data for the robot to perform an action that is a left-right inversion of the action of the operator is transmitted to the robot based on a relationship between the dominant arm of the operator and the arm used in a task performed by the robot;
The robot control method further comprises :

Images