WO2023054535A1 - 情報処理装置、ロボットコントローラ、ロボット制御システム、及び情報処理方法 - Google Patents

情報処理装置、ロボットコントローラ、ロボット制御システム、及び情報処理方法 Download PDF

Info

Publication number
WO2023054535A1
WO2023054535A1 PCT/JP2022/036312 JP2022036312W WO2023054535A1 WO 2023054535 A1 WO2023054535 A1 WO 2023054535A1 JP 2022036312 W JP2022036312 W JP 2022036312W WO 2023054535 A1 WO2023054535 A1 WO 2023054535A1
Authority
WO
WIPO (PCT)
Prior art keywords
information
robot
environment
control unit
information processing
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2022/036312
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
則夫 冨家
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Kyocera Corp
Original Assignee
Kyocera Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Kyocera Corp filed Critical Kyocera Corp
Priority to US18/696,301 priority Critical patent/US20240391105A1/en
Priority to CN202280065536.4A priority patent/CN118019623A/zh
Priority to EP22876401.5A priority patent/EP4410500A4/en
Priority to JP2023551815A priority patent/JP7678121B2/ja
Publication of WO2023054535A1 publication Critical patent/WO2023054535A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Program-controlled manipulators
    • B25J9/16—Program controls
    • B25J9/1656—Program controls characterised by programming, planning systems for manipulators
    • B25J9/1661—Program controls characterised by programming, planning systems for manipulators characterised by task planning, object-oriented languages
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Program-controlled manipulators
    • B25J9/16—Program controls
    • B25J9/1656—Program controls characterised by programming, planning systems for manipulators
    • B25J9/1664—Program controls characterised by programming, planning systems for manipulators characterised by motion, path, trajectory planning
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B19/00—Program-control systems
    • G05B19/02—Program-control systems electric
    • G05B19/42—Recording and playback systems, i.e. in which the program is recorded from a cycle of operations, e.g. the cycle of operations being manually controlled, after which this record is played back on the same machine
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Program-controlled manipulators
    • B25J9/16—Program controls
    • B25J9/1656—Program controls characterised by programming, planning systems for manipulators
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Program-controlled manipulators
    • B25J9/16—Program controls
    • B25J9/1679—Program controls characterised by the tasks executed
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B2219/00—Program-control systems
    • G05B2219/30—Nc systems
    • G05B2219/40—Robotics, robotics mapping to robotics vision
    • G05B2219/40607—Fixed camera to observe workspace, object, workpiece, global

Definitions

  • the present disclosure relates to an information processing device, a robot controller, a robot control system, and an information processing method.
  • Patent Document 1 Conventionally, there is known a device that causes a robot to execute a work group even if there is a lack of instructions from the user (see Patent Document 1, for example).
  • An information processing apparatus includes a control unit that estimates suitability of a first control program candidate for controlling a first robot from a plurality of control programs for controlling robots recorded in a storage unit. .
  • the control unit acquires first environment information indicating a work environment of the first robot, acquires second environment information indicating an environment for executing the first control program, and stores the first environment information and the second environment information. Based on the degree of similarity with the environment information, the suitability of the candidate for the first control program is estimated.
  • a robot controller controls the robot by executing a control program output after the information processing apparatus determines that the state is appropriate.
  • a robot control system includes the information processing device, the robot controller, and the robot.
  • An information processing method is an information processing device for estimating suitability of a first control program candidate for controlling a first robot from a plurality of control programs for controlling robots recorded in an information processing device. performed by The information processing method includes obtaining, by the information processing device, first environment information indicating a working environment of the first robot. The information processing method includes acquiring second environment information indicating an environment in which the information processing device executes the first control program. The information processing method includes the information processing device estimating suitability of a first control program candidate based on a degree of similarity between the first environment information and the second environment information.
  • FIG. 1 is a schematic diagram showing a configuration example of a robot control system according to an embodiment
  • FIG. 1 is a block diagram showing a configuration example of a robot control system according to an embodiment
  • FIG. It is a figure which shows an example of a work environment image.
  • 4 is a table showing an example of attribute information of objects located in the working environment;
  • 4 is a table showing types of combinations of selected control programs and actual work environments;
  • 7 is a flow chart showing an example of a procedure for associating a control program with past record information;
  • 6 is a flow chart showing an example procedure of an information processing method according to an embodiment;
  • a robot control system 100 includes a robot 1, a robot controller 2, an information acquisition device 3, a terminal device 9, and an information processing device .
  • the robot control system 100 causes the robot 1 to perform work on an object placed on the workbench 5 located in the work environment 4 .
  • the robot 1 can move the work 8 accommodated in the first tray 6 to the second tray 7 .
  • a program for controlling the motion of the robot 1 may be created, for example, while the user is actually operating the robot 1, and stored in the robot controller 2 or the information processing device 10.
  • a program that controls the operation of the robot 1 is also called a control program.
  • the control program may be created using a robot simulator or the like. In this case, for example, the control program may be created while not connected to the network (offline).
  • the robot 1 performs work by operating in a predetermined processing pattern.
  • a processing pattern is represented, for example, as a combination of actions for the robot 1 to accomplish an instructed task.
  • the processing pattern can be expressed as a combination of an action of gripping the work 8, an action of transporting the gripped work 8, and an action of arranging the transported work 8 at a predetermined position.
  • a task is a concept that defines a goal to be achieved by executing a series of operations included in a processing pattern.
  • a work may be a concept defining a goal of transporting a work 8 from point A to point B, for example.
  • the work may be, for example, a concept that defines the goal of placing the work 8 on the C point.
  • a work may be, for example, a concept that defines the goal of combining a workpiece 8 with another object.
  • a processing pattern can also be said to be a combination of actions for the robot 1 to complete a task.
  • the control program can be said to be a group of commands that cause the robot 1 to execute a combination of actions represented by a processing pattern so that the robot 1 can complete the task.
  • the control program can also be said to be software in which processing patterns for accomplishing predetermined tasks are programmed.
  • the plurality of control programs may include a control program in which different work performed on the same kind of work 8 is recorded, or a control program in which the same kind of work performed on the work 8 of the different kind is recorded. It may be configured including a control program. Also, one type of work for one type of workpiece 8 may be programmed in one control program.
  • one control program corresponding to the task to be performed is prepared in advance.
  • a plurality of control programs corresponding to each task are prepared in advance.
  • the user instructs the robot controller 2 to select a control program corresponding to the work to be performed by the robot 1, thereby causing the robot 1 to perform the work.
  • the workpiece in the robot controller estimates the workpiece existing in the working environment based on the image of the working environment, and estimates the work to be executed by the robot on the workpiece existing in the working environment.
  • the work estimation method according to the comparative example for example, when there is a possibility that a plurality of different works are performed on the same work, or when an incorrect work is arranged in the working environment, the robot controller It is difficult to judge appropriate work.
  • the information processing device 10 of the robot control system 100 obtains information about at least a part of the work environment 4 based on an image of at least a part of the work environment 4 of the robot captured by the information acquisition device 3. to get The information processing device 10 of the robot control system 100 compares at least part of the information of the working environment 4 with information associated with candidate control programs.
  • the information processing device 10 of the robot control system 100 can determine whether the candidate control program is suitable as a control program for causing the robot to work.
  • the information processing device 10 of the robot control system 100 can determine whether the candidate control program is appropriate for the working environment 4 .
  • the information processing device 10 of the robot control system 100 can determine whether the work environment 4 is suitable as an environment for executing the candidate control program.
  • the information processing device 10 of the robot control system 100 selects a control program for controlling the robot 1 to be controlled from a plurality of control programs for controlling the robot 1 recorded in the storage unit 12 of the information processing device 10.
  • Estimate the suitability of candidates for The robot 1 to be controlled is also referred to as the first robot for the sake of distinction only.
  • the control program that controls the first robot is also referred to as the first control program for the sake of distinction.
  • the information processing device 10 of the robot control system 100 may notify the user or extract and present an appropriate control program. You may By doing so, the possibility of the user making a mistake in selecting the control program can be reduced.
  • a robot control system 100 includes a robot 1, a robot controller 2, an information acquisition device 3, a terminal device 9, and an information processing device . At least one component of the robot control system 100 may be communicatively connected via the network 80 or may be communicatively connected without the network 80 . At least one component of the robot control system 100 may be communicatively connected by wire or wirelessly. At least one component of the robot control system 100 may be communicatively connected via a dedicated line. At least one component of the robot control system 100 is not limited to these examples, and may be communicatively connected to each other in various other forms. Each configuration of the robot control system 100 will be specifically described below.
  • the information processing device 10 includes a control section 11 and a storage section 12 .
  • the information processing apparatus 10 is communicatively connected to other components via the network 80 or directly without the network 80 .
  • the control unit 11 may include at least one processor in order to implement various functions of the information processing device 10 .
  • the processor can execute programs that implement various functions of the information processing device 10 .
  • a processor may be implemented as a single integrated circuit.
  • An integrated circuit is also called an IC (Integrated Circuit).
  • a processor may be implemented as a plurality of communicatively coupled integrated and discrete circuits.
  • the processor may be configured including a CPU (Central Processing Unit).
  • the processor may be configured including a DSP (Digital Signal Processor) or a GPU (Graphics Processing Unit). Processors may be implemented based on various other known technologies.
  • the information processing device 10 further includes a storage unit 12 .
  • the storage unit 12 may be configured including an electromagnetic storage medium such as a magnetic disk, or may be configured including a memory such as a semiconductor memory or a magnetic memory.
  • the storage unit 12 may be configured as an HDD (Hard Disk Drive) or an SSD (Solid State Drive).
  • the storage unit 12 stores various information, programs executed by the control unit 11, and the like.
  • the storage unit 12 may function as a work memory for the control unit 11 . At least part of the storage unit 12 may be included in the control unit 11 . At least part of the storage unit 12 may be configured as a separate storage device from the information processing device 10 .
  • the information processing apparatus 10 may be configured including a communication device configured to be capable of wired or wireless communication.
  • a communication device may be configured to be able to communicate with communication schemes based on various communication standards.
  • the information processing device 10 may be configured including one or more servers.
  • the information processing apparatus 10 may be configured to allow multiple servers to execute parallel processing.
  • the information processing device 10 does not need to include a physical housing, and may be configured based on virtualization technology such as a virtual machine or a container orchestration system.
  • the information processing device 10 may be configured using a cloud service.
  • cloud services When the information processing apparatus 10 is configured using cloud services, it can be configured by combining managed services. That is, the functions of the information processing device 10 can be implemented as a cloud service.
  • the information processing device 10 may include at least one server group and at least one database group.
  • the server group functions as the control unit 11 .
  • the database group functions as the storage unit 12 .
  • the number of server groups may be one or two or more. When the number of server groups is one, functions realized by one server group include functions realized by each server group. Each server group is communicably connected to each other by wire or wirelessly.
  • the number of database groups may be one or two or more. The number of database groups may be increased or decreased as appropriate based on the capacity of data managed by the information processing device 10 and availability requirements for the information processing device 10 .
  • the database group is communicably connected to each server group by wire or wirelessly.
  • the information processing device 10 may be connected to an external database. An information processing system including the information processing device 10 and an external database may be configured.
  • the information processing apparatus 10 is described as one configuration in FIGS. 1 and 2, multiple configurations can be regarded as one system and operated as necessary. That is, the information processing apparatus 10 is configured as a platform with variable capacity. By using a plurality of configurations for the information processing apparatus 10, even if one configuration becomes inoperable due to an unforeseen event such as a natural disaster, the other configurations are used to continue the operation of the system. In this case, each of the plurality of components is connected by a line, whether wired or wireless, and configured to be able to communicate with each other. These multiple configurations may be built across the cloud environment and the on-premises environment.
  • the information processing device 10 is communicably connected to at least one component of the robot control system 100 by a line that may be wired or wireless.
  • the information processing device 10 and at least one configuration of the robot control system 100 have an interface using a mutual standard protocol, and are capable of two-way communication.
  • Terminal device 9 is communicably connected to at least one configuration of the robot controller 2 and the information processing device 10 of the robot control system 100 .
  • the terminal device 9 may be communicably connected to other components of the robot control system 100 .
  • the terminal device 9 and at least one component of the robot control system 100 may be communicatively connected via the network 80 or directly without the network 80 .
  • the terminal device 9 may be configured including at least one processor.
  • the processor of the terminal device 9 may be configured identically or similarly to the processor of the control unit 11 of the information processing device 10 .
  • the terminal device 9 may be configured including a storage device.
  • the storage device of the terminal device 9 may be configured the same as or similar to the storage unit 12 of the information processing device 10 .
  • the terminal device 9 may be configured including a communication device.
  • the communication device of the terminal device 9 may be configured identically or similarly to the communication device of the information processing device 10 .
  • the terminal device 9 may be equipped with an input device.
  • the input device may include, for example, a touch panel or touch sensor, or a pointing device such as a mouse.
  • the input device may be configured including physical keys.
  • the input device may include an audio input device such as a microphone. Input devices are not limited to these examples, and may be configured to include other various devices.
  • the terminal device 9 may be equipped with an output device.
  • An output device may be configured including a display device.
  • the display device may include, for example, a liquid crystal display (LCD), an organic EL (Electro-Luminescence) display or an inorganic EL display, or a plasma display (PDP: Plasma Display Panel).
  • the display device is not limited to these displays, and may be configured to include other various types of displays.
  • the display device may include a light-emitting device such as an LED (Light Emitting Diode).
  • the display device may be configured including other various devices.
  • the output device may include an audio output device such as a speaker that outputs auditory information such as voice. Output devices are not limited to these examples, and may be configured to include other various devices.
  • the number of terminal devices 9 provided in the robot control system 100 is not limited to one, and may be two or more.
  • each terminal device 9 may receive input from the user.
  • the terminal device 9 may be configured as a tablet terminal.
  • the terminal device 9 may be configured as a mobile phone terminal such as a feature phone or a smart phone.
  • the terminal device 9 may be configured as a PC terminal such as a desktop PC (Personal Computer) or a notebook PC.
  • the terminal device 9 is not limited to these examples, and may be configured as various devices capable of providing GUI (Graphical User Interface) and communication functions.
  • GUI Graphic User Interface
  • the terminal device 9 may be used by the user to store the control program in the information processing device 10 in advance. Terminal device 9 may be used to monitor the status of robot 1 .
  • the terminal device 9 is not limited to these examples and can provide various other functions.
  • the terminal device 9 may be provided side by side with the robot controller 2 . When the terminal device 9 is installed side by side with the robot controller 2, the robot controller 2 itself may have an input device or an output device. A robot controller 2 may be included in the terminal device 9 .
  • the robot controller 2 downloads the control program from the information processing device 10 .
  • the robot controller 2 outputs information for controlling the operation of the robot 1 to the robot 1 and causes the robot 1 to perform the work specified by the control program.
  • the robot controller 2 may execute a control program held by the robot controller 2 itself.
  • the robot controller 2 may cause the robot 1 to move the workpiece 8 from the first tray 6 to the second tray 7 as illustrated in FIG.
  • the robot controller 2 may cause the robot 1 to perform various tasks without being limited to this.
  • the robot controller 2 may or may not be connected to the cloud computing environment. When the robot controller 2 is not connected to the cloud computing environment, the operation of the robot controller 2 is completed in the on-premises environment. If the operation of the robot controller 2 is completed in the on-premises environment, the operation of the information processing device 10 is executed by the robot controller 2 .
  • the robot controller 2 may be equipped with a communication device that downloads the control program from the information processing device 10 .
  • the communication device of the robot controller 2 may be configured identically or similarly to the communication device of the information processing device 10 .
  • the robot controller 2 may comprise a processor that generates information for controlling the motion of the robot 1 by executing control programs.
  • the processor of the robot controller 2 may be configured identically or similarly to the processor of the control unit 11 of the information processing device 10 .
  • one robot controller 2 is connected to one robot 1.
  • One robot controller 2 may be connected to two or more robots 1 .
  • One robot controller 2 may control only one robot 1 or may control two or more robots 1 .
  • the number of robot controllers 2 and robots 1 is not limited to two, and may be one or three or more.
  • the robot controller 2 may be integrated with the information processing device 10 and the function of the robot controller 2 may be implemented as one function of the information processing device 10 .
  • the robot 1 may be configured as a robot arm with an arm.
  • the arm may be configured as a 6-axis or 7-axis vertical articulated robot, for example.
  • the arm may be configured as a 3-axis or 4-axis horizontal articulated robot or a SCARA robot.
  • the arm may be configured as a 2-axis or 3-axis Cartesian robot.
  • the arm may be configured as a parallel link robot or the like. The number of axes forming the arm is not limited to the illustrated one.
  • the robot 1 may be equipped with an end effector attached to the arm.
  • the end effector may include, for example, a gripping hand configured to grip a work object.
  • the grasping hand may have multiple fingers. The number of fingers of the grasping hand may be two or more.
  • the fingers of the grasping hand may have one or more joints.
  • the end effector may include a suction hand configured to be able to suction the work target.
  • the end effector may include a scooping hand configured to scoop the work object.
  • the end effector includes a tool such as a drill, and may be configured to be able to perform various machining operations such as drilling a hole in a work object.
  • the end effector is not limited to these examples, and may be configured to perform various other operations.
  • the robot 1 can control the position of the end effector by operating the arm.
  • the end effector may have an axis that serves as a reference for the direction in which it acts on the work object. If the end effector has an axis, the robot 1 can control the direction of the end effector axis by operating the arm.
  • the robot 1 controls the start and end of the action of the end effector acting on the work object.
  • the robot 1 can move or process a work object by controlling the position of the end effector or the direction of the axis of the end effector and controlling the operation of the end effector.
  • the robot 1 may be configured as an automated guided vehicle (AGV).
  • AGV automated guided vehicle
  • the robot 1 may be configured as a drone.
  • the robot 1 is not limited to a robot arm or AGV, and may be configured in various other forms such as a vehicle, an electronic device, or a control machine.
  • the robot 1 may further include sensors that detect the state of at least one component of the robot 1 .
  • the sensors may detect information about the real position or pose of at least one component of the robot 1 or the velocity or acceleration of at least one component of the robot 1 .
  • the sensors may detect forces acting on at least one component of the robot 1 .
  • the sensors may detect the current flowing through or the torque of the motors driving at least one component of the robot 1 .
  • the sensors can detect information resulting from the actual movement of the robot 1 .
  • the robot controller 2 can grasp the result of the actual motion of the robot 1 by acquiring the detection result of the sensor.
  • the robot 1 performs a predetermined work on a product, and unlike a device that processes different types of products by setting conditions according to the type of product, the robot 1 can generate a control program suitable for the work, and control Programs can be generated to operate to perform various tasks.
  • the information acquisition device 3 acquires at least part of the information on the work environment 4 of the robot 1 .
  • the information acquisition device 3 may include a camera that captures at least part of the working environment 4 .
  • the information acquisition device 3 may include a sensor that measures the position, shape, or size of objects present in at least part of the working environment 4 .
  • the information acquisition device 3 may be configured including a 3D sensor, a distance sensor, or the like.
  • the information acquisition device 3 may include a sensor that measures the temperature or humidity of at least part of the work environment 4 . If at least part of the work environment 4 is a clean room, the information acquisition device 3 may include a sensor that measures particle density within the clean room. At least some information about the work environment 4 may include clean room classification information.
  • the information acquisition device 3 may include a sensor that measures noise in at least part of the working environment 4 . Further, the information acquisition device 3 may be configured to include a sensor such as a current sensor that measures current or the like, indicating the operation of the peripheral device. The information acquisition device 3 may be fixed at a position where at least part of the information of the working environment 4 of the robot 1 can be acquired. The information acquisition device 3 may be attached to a robot arm or an end effector. The number of information acquisition devices 3 is not limited to one, and may be two or more. A plurality of information acquisition devices 3 may be connected so as to be able to communicate with each other. At least one information acquisition device 3 among the plurality of information acquisition devices 3 may be communicably connected to the robot controller 2 or the information processing device 10 . The information acquisition device 3 may include a plurality of physically separated information acquisition units. A plurality of information acquisition units may be connected so as to be able to communicate with each other.
  • the information acquired by the information acquisition device 3 as at least a part of the work environment 4 is within the work range of the robot 1, such as the range where the work table 5 and the objects placed on the work table 5 are located, as illustrated in FIG. may contain information about
  • the information that the information acquisition device 3 acquires as information about at least a part of the working environment 4 may include information different from that of the work object itself or the robot 1 itself.
  • the information acquired by the information acquisition device 3 as at least part of the information on the work environment 4 is not limited to these pieces of information, and may include information outside the work range of the robot 1 .
  • the information that the information acquisition device 3 acquires as at least part of the information on the work environment 4 may include information on the range in which the peripheral device is located.
  • the peripheral device may include a parts feeder for placing the workpiece 8 or a tray changer for placing the first tray 6 or the second tray 7 or the like.
  • the peripheral device may include a work inverting device that turns the work 8 upside down.
  • Peripherals may include a tool changer for changing end effectors of robot 1 .
  • the peripheral device may include a manufacturing device such as a grinder that processes the workpiece 8 .
  • Peripheral devices are not limited to these and may include various devices.
  • the information that the information acquisition device 3 acquires as at least part of the information on the work environment 4 may include information on the range where the work 8 is not located.
  • the information that the information acquisition device 3 acquires as at least part of the information on the work environment 4 may include the attribute information of the end effector of the robot 1 or the attribute information of the arm.
  • the end effector attribute information may include information specifying whether the end effector is a hand or a suction cup.
  • the arm attribute information may include information specifying the number of joints or the range of movement of the arm, or the length of the arm.
  • the range from which the information acquisition device 3 acquires information may be the motion range of the robot 1 . Further, the range from which the information acquisition device 3 acquires information is not limited to the motion range of the robot 1, and may include a range outside the motion range of the robot 1 or a range around the motion range. The information acquisition device 3 may acquire information on the range outside the motion range of the robot 1 or the range around the motion range as at least part of the information on the working environment 4 .
  • the range around the range of motion may be a range within a predetermined distance from the range of motion. The predetermined distance may be set as a distance between two points, for example, within the same factory building, within the same room, or within the same production line.
  • Peripheral equipment may include a belt conveyor that feeds the workpiece 8 or the like toward the robot 1 within its range of motion.
  • the robot control system 100 can select a control program based on the state of the work 8 on the belt conveyor located outside the operating range of the robot 1 .
  • the robot control system 100 may store the control program (processing pattern) in the information processing device 10 in association with the program number, program name, or task name.
  • the robot control system 100 associates at least part of the information of the work environment 4 in which the control program is executed with the program number or the like.
  • the control program is associated with at least part of information of the work environment 4 in which the control program is executed.
  • At least part of the information of the working environment 4 prepared by the user is also referred to as prepared environment information or first environment information.
  • At least part of the information on the work environment 4 associated with the control program is information on at least part of the work environment 4 when the control program was recorded in the past, and is also referred to as past recorded information or second environment information. be done.
  • the past record information or the second environment information corresponds to at least part of information of the working environment 4 when the robot 1 performed the work by the robot controller 2 executing the control program.
  • the past record information or the second environment information may include at least part of the work environment 4 registered by the user.
  • the terminal device 9 may be configured to receive input for registering information of at least part of the work environment 4 from the user.
  • the robot control system 100 may associate the information simulated by the user with the robot control program (processing pattern) number or the like.
  • a user who can register the past record information or the second environment information of the first robot may be, for example, a user who has a track record of causing the first robot to perform work in the past, or a user of another robot. good.
  • a user who can register the past record information of the first robot or the second environment information may be a manager of a factory or the like, or may be a worker or the like.
  • the past record information only needs to contain information that can be compared with the preparation environment information. That is, the past record information only needs to include information corresponding to the preparation environment information. For example, if the preparation environment information has an image, the past record information may also have an image. has numerical data such as sensor output, the past record information may also have numerical data. In other words, the preparation environment information may include information that can be compared with past record information.
  • the past record information may include, for example, at least a partial image of the working environment 4 taken when the control program is executed.
  • the past record information may include attribute information of objects present in at least part of the work environment 4 obtained by analyzing images of at least part of the work environment 4 .
  • the object attribute information may include information on the shape of the object such as its contour, the color or texture of the object, or the appearance of the object such as the size of the object.
  • the object attribute information may include two-dimensional information captured in an image of at least a portion of the work environment 4 or may include three-dimensional information based on depth information of at least a portion of the work environment 4 .
  • Attribute information is not limited to information about the appearance of an object, and may include information such as the material or density of the object.
  • the past record information may include position information such as coordinates of objects existing in at least part of the work environment 4 .
  • Position information may represent a relative position from a reference position set in the work environment 4 .
  • the position information may represent an absolute position set in the information acquisition device 3 .
  • the positional information may represent the positional relationship of multiple objects around the robot 1 .
  • the positional information may represent the positional relationship of multiple objects existing in at least part of the work environment 4 .
  • the positional information may represent the positional relationship between the work 8 and objects around the work 8 .
  • the past recording information may include the type of camera that captured at least part of the working environment 4, or the capturing conditions such as the position or orientation of the camera.
  • the robot control system 100 stores the control program associated with the program number or program name in the information processing apparatus 10 in association with the past record information of the control program. Specifically, the robot control system 100 may store the following information in the information processing device 10 .
  • Program number (2) Control program (processing pattern) (3) Image of at least part of work environment 4 (4) Attribute information and position information of object recognized from image of at least part of work environment 4
  • FIG. 1 An example of an image of at least part of the working environment 4 is shown in FIG.
  • a workbench 5 is positioned in the work environment 4 .
  • a first tray 6 and a second tray 7 are positioned on the workbench 5 .
  • Workpieces 8 are accommodated in the first tray 6 .
  • FIG. 4 shows an example of attribute information and position information of an object recognized from an image of at least part of the work environment 4.
  • the total number of objects recognized from the images is 11.
  • Workbench 5 is associated with No. 1.
  • the second tray 7 is associated with No.2.
  • the first tray 6 is associated with number 3.
  • Work 8 is associated with No. 4.
  • the number of workbench 5, first tray 6 and second tray 7 is one each.
  • the number of works 8 is eight.
  • the color information of each object is expressed as average color information that is the average of the color information of each part of the object.
  • the color information is expressed as a numerical value corresponding to the gradation of each RGB (Red Green Blue) component of the image.
  • the outline and texture of each object are represented by its image.
  • the size of each object is represented by the width (W) x depth (D) x height (H) of each object.
  • the coordinates of each object may be represented by coordinates relative to the coordinates of the reference position set in the work environment 4, or by absolute coordinates set in the information acquisition device 3 (for example, the pixel position of the captured image). ) may be represented by In this embodiment, coordinates are represented by XYZ coordinates.
  • the robot control system 100 stores control programs in the information processing device 10 in advance. If the control program has not yet been associated with past record information when the control program is executed, the robot control system 100 may acquire at least part of the information of the work environment 4 when the control program is executed. The robot control system 100 may associate at least part of the acquired information of the work environment 4 with the control program as past record information. For example, when the control program is executed for the first time, the robot control system 100 may acquire at least part of the information of the work environment 4 as past recorded information and associate it with the control program. It can be said that the robot control system 100 associates past record information with the work corresponding to the control program.
  • the past record information may include at least part of the work environment 4 registered by the user.
  • the terminal device 9 may be configured to receive input for registering information of at least part of the work environment 4 from the user.
  • the robot control system 100 may associate the information simulated by the user with the robot control program (processing pattern) number or the like.
  • a user who can register the past record information of the first robot may be, for example, a user who has a track record of causing the first robot to perform a task in the past, or a user of another robot.
  • a user who can register the past record information of the first robot may be a manager of a factory or the like, or a worker or the like.
  • the control program can be stored in the information processing device 10 in association with the past record information, specifically in the following procedure.
  • the work 8 and the like necessary for the robot 1 to execute are prepared as shown in FIG.
  • the user places the first tray 6 and the second tray 7 in the working environment 4 .
  • a user accommodates the work 8 in the first tray 6 . That is, the workpiece 8 and the first tray 6 and the second tray 7 may be arranged by the user. As another example, the workpieces 8 and the first tray 6 and the second tray 7 may be arranged by a peripheral device.
  • the user selects a control program to be executed by the robot controller 2 using the terminal device 9.
  • the terminal device 9 outputs the selected control program to the robot controller 2 .
  • the robot controller 2 controls the robot 1 by executing the acquired control program, and causes the robot 1 to perform work.
  • the robot 1 or the robot controller 2 acquires at least part of the information of the work environment 4 by the information acquisition device 3 when the robot 1 performs the work.
  • the information acquisition device 3 may capture, for example, at least a portion of the working environment 4 and acquire a captured image of the at least a portion of the working environment 4 as information of the at least a portion of the working environment 4 .
  • the robot 1 moves the information acquisition device 3 to a predetermined imaging position, and captures at least part of the work environment 4 with the information acquisition device 3 . You can let the The information acquisition device 3 acquires not only an image of at least a part of the working environment 4 but also various information such as depth information of at least a part of the working environment 4 as information of at least a part of the working environment 4. good.
  • the information acquisition device 3 outputs at least part of the acquired information of the work environment 4 to the information processing device 10 .
  • the control unit 11 of the information processing apparatus 10 recognizes an object such as the work 8 existing in at least a part of the work environment 4 by analyzing at least a part of the information of the work environment 4 .
  • the control unit 11 may recognize each of the work 8, the first tray 6, and the second tray 7 by image analysis of at least a part of the photographed image of the working environment 4, for example.
  • the control unit 11 may recognize each of the work 8 , the first tray 6 and the second tray 7 by analyzing at least part of the depth information of the work environment 4 , for example.
  • the control unit 11 acquires attribute information and position information of each recognized object from at least part of the information of the working environment 4 .
  • the control unit 11 associates the attribute information and position information of each object with the control program stored in the storage unit 12 as past record information.
  • the control program is stored in the information processing apparatus 10 in association with the past record information.
  • the robot control system 100 can easily determine the suitability of the control program used to cause the robot 1 to perform work in the prepared work environment 4 by the following procedure. It can be said that the robot control system 100 estimates the adequacy of the candidate control program when the user selects a control program candidate to be used for causing the robot 1 to perform a task.
  • the work 8 and the like necessary for the robot 1 to execute are prepared as shown in FIG.
  • the user places the first tray 6 and the second tray 7 in the working environment 4 .
  • a user accommodates the work 8 in the first tray 6 . That is, the workpiece 8 and the first tray 6 and the second tray 7 may be arranged by the user. As another example, the workpieces 8 and the first tray 6 and the second tray 7 may be arranged by a peripheral device.
  • the user activates the robot controller 2 or the robot 1 using the terminal device 9 .
  • the user selects a control program to be executed by the robot controller 2 using the terminal device 9 .
  • the robot control system 100 acquires at least part of the information of the work environment 4 using the information acquisition device 3 .
  • the robot controller 2 moves the information acquisition device 3 to a predetermined position in order to acquire information of at least a part of the work environment 4, or moves the information acquisition device 3 to a predetermined position. It controls the motion of the robot 1 so as to direct its direction.
  • the information acquisition device 3 outputs at least part of the acquired information of the work environment 4 to the information processing device 10 .
  • the control unit 11 of the information processing apparatus 10 recognizes an object such as the work 8 existing in at least a part of the work environment 4 by analyzing at least a part of the information of the work environment 4 .
  • the control unit 11 may recognize each of the work 8, the first tray 6, and the second tray 7 by image analysis of at least a part of the photographed image of the working environment 4, for example.
  • the control unit 11 may recognize each of the work 8 , the first tray 6 and the second tray 7 by analyzing at least part of the depth information of the work environment 4 , for example.
  • the control unit 11 acquires attribute information and position information of each recognized object from at least part of the information of the work environment 4 .
  • the control unit 11 compares the attribute information and position information of each object with past record information associated with the selected control program.
  • the control unit 11 outputs the selected control program to the robot controller 2 when the past recorded information associated with the selected control program matches at least part of the information of the working environment 4 .
  • the robot controller 2 executes the selected control program and causes the robot 1 to perform work.
  • control unit 11 In order to determine whether the past recorded information associated with the selected control program matches at least part of the information of the working environment 4, the control unit 11 combines at least part of the information of the working environment 4 and the past recorded information. may be calculated. The degree of similarity is calculated as 100% when the two pieces of information completely match, and is calculated as 0% when the two pieces of information completely do not match. The control unit 11 may cause the robot controller 2 to execute the selected control program when the degree of similarity is greater than or equal to a predetermined value.
  • the control unit 11 may calculate the similarity of attribute information such as the shape or color of objects existing in at least a part of the work environment 4 as the degree of similarity. Specifically, the control unit 11 includes the attribute information of each object recognized from at least part of the information of the work environment 4 in the past recorded information associated with the control program stored in the information processing device 10. Compare with attribute information provided. The control unit 11 may calculate the degree of similarity as a numerical value using a trained model generated by template matching, machine learning, or deep learning. The control unit 11 calculates the degree of similarity for each type of recognized object. The control unit 11 calculates the degree of similarity for all types of recognized objects. Objects recognized in at least part of the work environment 4 illustrated in FIG. 3 are divided into four types.
  • the control unit 11 calculates the degree of similarity for each of the four types.
  • the control unit 11 calculates the average value and standard deviation of the four degrees of similarity calculated for each type.
  • the control unit 11 determines that the past record information is at least part of the working environment 4. It is determined that the information matches.
  • the trained model can be generated by performing machine learning on data relating to a plurality of pieces of past record information as a learning data set.
  • the control unit 11 may calculate the similarity of the positions of objects existing in at least a part of the work environment 4 as the similarity. Specifically, the control unit 11 calculates the distance between each object based on the position information of each object recognized from at least part of the information of the working environment 4 . It is assumed that the position information of each object includes the coordinates of the center of gravity of each object. The position information of each object may include the coordinates of the edges of each object (for example, the minimum or maximum values of the X, Y, and Z coordinates of the range in which each object exists). The control unit 11 calculates the distance between each object based on the coordinates of the center of gravity of each object.
  • the control unit 11 regards a plurality of objects classified into the same type as one object, calculates the coordinates of the center of gravity, and calculates the distance to other objects. When there are four types of objects in the work environment 4, there are six combinations for calculating the distance between each object. Therefore, the distance between each object is calculated in six ways.
  • the control unit 11 calculates the distance between each object included in the past record information based on the position information included in the past record information associated with the control program stored in the information processing device 10 .
  • the control unit 11 may calculate the distance between each object in the past record information including four types of objects.
  • the control unit 11 may calculate the distance between each object when the past recorded information includes an object corresponding to an object existing in at least a part of the working environment 4 . In other words, if the past record information does not include an object corresponding to an object existing in at least a part of the work environment 4, the control unit 11 does not calculate the distance between each object in the past record information. It may be determined that at least part of the information of the working environment 4 is not suitable.
  • the control unit 11 compares the distance between each object recognized from at least part of the information of the work environment 4 and the distance between each object in past recorded information associated with the control program.
  • the control unit 11 selects two objects from the objects recognized from at least part of the information on the work environment 4 . Further, the control unit 11 selects two objects corresponding to the two objects selected from the objects recognized from at least part of the information of the working environment 4 among the objects included in the past record information.
  • the control unit 11 determines the difference between the distance between two objects selected from objects recognized from at least part of the information of the working environment 4 and the distance between two objects selected from objects included in the past recording usage. Calculate the absolute value.
  • the control unit 11 calculates the distance between two objects included in at least a part of the information of the work environment 4 and the distance between the two objects included in the past record information for each of the distances calculated in the six combinations in the example of FIG. Calculate the absolute value of the difference from the distance between objects.
  • the control unit 11 calculates the average value and the standard deviation of the absolute values of the distance differences calculated for all combinations as the similarity of the positions. When the average value is equal to or greater than a preset average value determination threshold value and the standard deviation is equal to or less than a preset standard deviation determination threshold value, the control unit 11 determines that the past record information is at least part of the working environment 4. It is determined that the information matches.
  • FIG. 5 exemplifies a pattern in which the past record information does not match at least part of the information of the work environment 4.
  • the type shown as "work error” in the cell in the left column of the second row of the table in FIG. shows a case where the work 8 placed in the position is wrong.
  • the first tray 6 should contain the workpiece 8, but the first tray 6 contains the workpiece 8F as shown in the cell in the right column of the second row.
  • the type indicated as "tray set failure" in the cell in the left column of the third row of the table in FIG. indicates if Specifically, where the second tray 7 should be set, the second tray 7 is not set as indicated by the set failure tray 7F indicated by the two-dot chain line in the cell in the right column of the third row. wrong in
  • the types of cases in which the past record information does not match at least part of the information of the work environment 4 are not limited to these.
  • a pattern in which at least part of the work environment 4 is arranged with extra objects may be included.
  • a pattern in which the work 8 is not arranged may be included.
  • the control unit 11 determines that the past record information does not match at least part of the information of the working environment 4. .
  • the control unit 11 may calculate the degree of similarity and determine that the past recorded information does not match at least part of the information of the working environment 4 based on the degree of similarity. If the similarity of attribute information or the similarity of position calculated as the similarity between the past record information and at least a part of information of the work environment 4 is less than a predetermined value, the control unit 11 determines that the past record information is It may be determined that at least part of the information of the working environment 4 is not suitable. The control unit 11 does not output the control program selected by the user to the robot controller 2 when determining that the past recorded information does not match at least part of the information of the working environment 4 .
  • the control unit 11 determines that there is a control program whose past record information matches at least part of the work environment 4 among other control programs. Exists or searches. If the control program 11 finds a control program whose past record information matches at least part of the information of the work environment 4, the control unit 11 may output the found control program as a selection candidate to the terminal device 9 and present it to the user. In this case, the user may have made a mistake in selecting the control program. The user's wrong selection of the control program is also referred to as an improper selection of the control program. When the user selects a control program candidate, the control unit 11 may estimate that the user has selected a control program improperly. The control unit 11 may estimate whether the control program selection by the user is inadequate based on the estimation result of the suitability of the control program candidate.
  • the control unit 11 receives the input selected by the user from the control program selection candidates presented to the user at the terminal device 9 and acquires information specifying the control program selected by the user from the terminal device 9 .
  • the control unit 11 learns the user's selection result as teacher data so as to increase the accuracy of finding a control program candidate whose past record information matches at least part of the information in the work environment 4, and learns the result of the user's selection as teacher data. 4 may be modified.
  • the model may be modified to improve the accuracy of recognizing objects from images taken of at least part of the working environment 4 .
  • the threshold used for the determination based on the degree of similarity may be adjusted according to the selection result of the user.
  • control unit 11 assumes that the user has correctly selected the control program, and finds at least part of the information of the work environment 4. Estimate the items that are incorrectly prepared in the The presence of an incorrectly prepared item in at least part of the work environment 4 is also referred to as an environment deficiency.
  • the control unit 11 may estimate at least part of the work environment 4 to be inadequate.
  • the control unit 11 may estimate the environmental inadequacy of at least a part of the working environment 4 based on the estimation result of the adequacy of the control program candidate.
  • the control unit 11 may detect the shortage or excess of objects to be prepared in at least a part of the work environment 4, the difference in the arrangement of objects, or the difference in the attribute information of objects as factors of the environment inadequacy. For example, assuming that the control program selected by the user is correct, the control unit 11 determines the past record information associated with the number of the control program selected by the user and the work environment 4 including the object placed by the user. You can compare with the image. Based on the comparison result, the control unit 11 may detect a shortage or excess of objects to be arranged in at least a part of the working environment 4, a difference in arrangement of objects, or a difference in attribute information of objects. The control unit 11 may presume that an item for which a difference has been detected is an item that has been erroneously prepared.
  • the control unit 11 may detect objects appearing in the image of the working environment 4 by analyzing the image of the working environment 4 .
  • the control unit 11 may calculate the degree of similarity between the object included in the past recorded information and the object placed in the work environment 4 by the user and detected from the image of the work environment 4 .
  • the control unit 11 may calculate the degree of similarity between the object attribute information included in the past record information and the object attribute information detected from the image of the work environment 4 .
  • the control unit 11 may calculate the degree of similarity as a numerical value using a trained model generated by template matching, machine learning, or deep learning. Note that when using a trained model, the trained model can be generated by performing machine learning using attribute data of each object in a plurality of pieces of past record information as a learning data set.
  • the control unit 11 may calculate the degree of similarity such as the outline or size of the object.
  • the control unit 11 determines that a correct object is prepared in the working environment 4 when the similarity between the contour and size of the object is a value indicating a high degree of similarity, such as a predetermined threshold value or more. It can be determined that there are The control unit 11 may determine that the correct object is prepared in the work environment 4 where the degree of similarity in outline and size of the object is equal to or greater than a predetermined threshold.
  • the control unit 11 determines between each object included in the past record information and the object detected from the image of the work environment 4 . , similarity measures such as contours or sizes of objects may be calculated. When a plurality of objects are detected from the image of the work environment 4, the control unit 11 determines the contours of the objects or the A degree of similarity such as size may be calculated. The control unit 11 calculates the degree of similarity, such as outlines or sizes, of the objects in at least a partial combination of each object included in the past record information and each object detected from the image of the work environment 4. good. When an object having a high degree of similarity with an object detected from the image of the work environment 4 is included in the plurality of objects included in the past record information, the control unit 11 determines that the correct object is prepared in the work environment 4. It may be determined that
  • the control unit 11 may estimate factors that reduce the degree of similarity as the cause of the inadequate environment. For example, the control unit 11 may estimate whether the similarity is lowered due to an error in the position of each object placed in at least a part of the working environment 4 . For example, the control unit 11 may estimate whether the degree of similarity is lowered because an object that should be prepared in at least a part of the working environment 4 is not prepared. For example, the control unit 11 may estimate whether the similarity is lowered due to unnecessary objects being placed in at least part of the work environment 4 .
  • control unit 11 compares past record information associated with the number of the control program selected by the user and the image of the object existing in the work environment arranged by the user, based on the result of comparison, The lack of necessary objects or the presence of unnecessary objects may be detected, and the detected items may be presumed as the cause of the decrease in similarity.
  • the control unit 11 may estimate how at least part of the work environment 4 should be changed so as to increase the similarity, and output points to be changed to the terminal device 9 to present them to the user.
  • the user may change at least part of the working environment 4 based on the presented information, or may instruct the robot controller 2 to execute the selected control program as it is.
  • the control unit 11 may learn user instruction content as teacher data and change the model for determining whether the past record information matches at least part of the information of the work environment 4 .
  • the threshold used for determination based on the degree of similarity may be adjusted depending on the content of the user's instruction.
  • the control unit 11 may estimate the type of deficiency based on the degree of similarity. Specifically, the control unit 11 may estimate whether the cause of the low similarity is poor selection or poor environment. For example, the control unit 11 may estimate that the environment is insufficient when the similarity is 90% or more, and that the selection is insufficient when the similarity is less than 50%.
  • the control unit 11 may determine whether or not the positions of objects that are determined to be correct objects to be prepared in the work environment 4 among the objects detected from the image of the work environment 4 are correct. The control unit 11 performs work based on the position where the object should be placed in the work environment 4 and the position of the object shown in the image of the work environment 4, which are specified by the data stored in the information processing device 10. The distance from the position where the object should originally be placed in the environment 4 to the position where the object is actually placed may be calculated. The control unit 11 may calculate the distance between the position of an object with a high degree of similarity included in the past record information and the position of the object appearing in the image of the working environment 4 .
  • the control unit 11 may determine that the object is placed at the correct position when the calculated distance is less than a predetermined threshold. The control unit 11 may determine that the object is placed at the correct position when the calculated distance is less than a predetermined threshold. The control unit 11 may determine that the position of the object placed in the work environment 4 by the user is incorrect when the calculated distance is equal to or greater than a predetermined threshold.
  • the control unit 11 may determine whether the number of objects placed in the working environment 4 by the user is correct.
  • the control unit 11 selects an object that has a high degree of similarity among the objects placed in the working environment 4 by the user and that has one object in the data stored in the information processing device 10 and that is specified by the data.
  • the number of identical objects may be compared to determine whether they are properly prepared.
  • the control unit 11 determines whether the user actually operates the work environment 4 . Among the objects placed in the work environment 4, if there is one object that is determined to be the same object as the first type of object, the number of the first type of objects placed in the work environment 4 is correct. can be determined.
  • the control unit 11 may calculate the number of objects having a high degree of similarity to the first type of objects among the objects actually arranged in the work environment 4 by the user. When the calculated number matches the number of first type objects to be placed in the work environment 4, the control unit 11 determines that the number of the first type objects placed in the work environment 4 is correct. You can judge. If the calculated number is greater than the number of first type objects to be placed in the work environment 4, the control unit 11 determines that the number of the first type objects placed in the work environment 4 is correct, or , it may be determined that the working environment 4 is sufficiently populated with objects of the first type.
  • the control unit 11 determines that the degree of similarity of each piece of attribute information is less than a predetermined threshold. , or if it is not similar to the first type of object, it is determined that the user has not placed the first type of object in the work environment 4 or that the user has placed the wrong object in the work environment 4. you can
  • the control unit 11 may calculate the number of objects determined to be correct objects to be prepared in the work environment 4, regardless of the type of object.
  • the control unit 11 determines the number of objects determined to be correct objects to be placed in the work environment 4 by the data stored in the information processing device 10, and the number of objects to be prepared in the work environment 4. , it may be determined that the number of prepared objects is correct. If the number of objects determined to be correct objects to be placed in the work environment 4 is larger than the number of objects to be placed in the work environment 4, the control unit 11 determines that the number of the placed objects is correct, or , it may be determined that the objects are sufficiently arranged in the work environment 4 .
  • control unit 11 may notify the terminal device 9 that the objects placed in the work environment 4 are incorrect.
  • the control unit 11 may notify the terminal device 9 of correct information on the positions or the number of objects to be placed in the work environment 4 and prompt the user to change the placement to the correct one.
  • the terminal device 9 receives confirmation from the user that the positions and the number of the objects placed in the work environment 4 by the user are correct.
  • the information may be acquired and the information processing apparatus 10 may be notified that the positions and the number of objects placed in the work environment 4 by the user are correct.
  • the information processing apparatus 10 may update the data by setting the positions and the number of objects placed in the working environment 4 by the user as correct data.
  • the robot control system 100 may execute an information processing method including the procedures of the flowcharts illustrated in FIGS. 6 and 7 .
  • the information processing method may be implemented as an information processing program to be executed by a processor constituting the controller 11 of the information processing device 10, the robot controller 2, or the terminal device 9.
  • the information processing program may be stored in a non-transitory computer-readable medium.
  • the control unit 11 may associate past record information with the control program by executing the procedure of the flowchart of FIG.
  • the control unit 11 acquires the selection of the control program from the user (step S1). Specifically, the terminal device 9 receives an input for selecting a control program from the user. The control unit 11 acquires the selection of the control program from the terminal device 9 .
  • the control unit 11 acquires an image of at least part of the working environment 4 (step S2). Specifically, the information acquisition device 3 acquires an image of at least part of the work environment 4 or depth information of at least part of the work environment 4 as at least part of the work environment 4 . The control unit 11 acquires at least part of the information on the work environment 4 from the information acquisition device 3 .
  • the control unit 11 recognizes objects in at least part of the work environment 4 (step S3).
  • the control unit 11 acquires attribute information and position information of the recognized object (step S4).
  • the control unit 11 associates the control program selected by the user with past recorded information including attribute information and position information of objects recognized in at least a part of the work environment 4, and stores the information in the storage unit 12 (step S5). After executing the procedure of step S5, the control unit 11 ends the execution of the procedure of the flowchart of FIG.
  • the control unit 11 may determine the suitability of the control program selected by the user by executing the procedure of the flowchart of FIG.
  • the control unit 11 acquires the control program selection from the user (step S11). Specifically, the terminal device 9 receives an input for selecting a control program from the user. The control unit 11 acquires the selection of the control program from the terminal device 9 .
  • the control unit 11 acquires an image of at least part of the work environment 4 (step S12).
  • the control unit 11 recognizes objects in at least part of the working environment 4 (step S13).
  • the control unit 11 acquires attribute information and position information of the recognized object (step S14).
  • the control unit 11 calculates the degree of similarity between the past record information associated with the control program and the attribute information and position information of the recognized object (step S15). The control unit 11 determines whether the calculated similarity is greater than or equal to the threshold (step S16). If the calculated similarity is greater than or equal to the threshold (step S16: YES), the control unit 11 causes the robot controller 2 to execute the control program selected by the user (step S17). After executing the procedure of step S17, the control unit 11 ends the execution of the procedure of the flowchart of FIG.
  • step S16 determines whether there is a control program similar to the work that the user wants to execute.
  • Step S18 determines that, among the control programs stored in the storage unit 12, there is a control program associated with past record information similar to at least part of the information of the work environment 4 prepared by the user. decide whether to The control unit 11 may determine whether or not there is a control program associated with past record information that has a high degree of similarity with at least part of the information on the work environment 4 prepared by the user.
  • step S18 If a similar control program exists (step S18: YES), the control unit 11 extracts the similar control program (step S19).
  • the control unit 11 outputs information specifying the extracted control program to the terminal device 9 and presents it to the user.
  • the control unit 11 acquires the selection of the control program from the user (step S20).
  • the terminal device 9 receives an input for selecting a control program from the user.
  • the control unit 11 acquires the selection of the control program from the terminal device 9 .
  • the control unit 11 proceeds to the procedure of step S17, and causes the robot controller 2 to execute the control program selected by the user.
  • step S18 If a similar control program does not exist (step S18: NO), the control unit 11 estimates a factor that reduces the degree of similarity with the existing past record information in at least a part of the work environment 4, and outputs it to the terminal device 9. is displayed (step S21). After executing the procedure of step S21, the control unit 11 ends the execution of the procedure of the flowchart of FIG. After executing the procedure of step S21, the control unit 11 may confirm that the user has changed at least part of the work environment 4 and return to the determination procedure of step S18.
  • suitability of a candidate control program for controlling the robot 1 is estimated from a plurality of control programs. obtain. It may also be determined whether the control program selected by the user is compatible with at least part of the working environment 4 . By doing so, the user is less likely to make a mistake in selecting the control program. In addition, it is possible to avoid starting work of the robot 1 while preparing at least part of the work environment 4 or selecting the control program incorrectly. As a result, the burden on the user can be reduced.
  • the work of the robot 1 may not be successful with the selected control program. Conversely, even if the work 8 and the like are not available in the working environment 4, the robot 1 may successfully perform the work with the selected control program. Comparing at least part of the information of the work environment 4 with past record information, or making determinations based on the degree of similarity, allows flexibility in the work of the robot 1 . As a result, the work of the robot 1 can be efficiently performed.
  • the suitability of the control program may be determined for each task to be performed by the robot 1, or may be performed collectively for a series of multiple tasks to be performed by the robot 1.
  • Cooperative work includes work performed in cooperation between a person and the robot 1, work performed in cooperation between the robots 1, or work performed in cooperation between the robot 1 and another machine.
  • Collaborative work includes collaborative work performed by a person and the robot 1 in cooperation with each other.
  • the robot control system 100 can be deployed at industrial product production sites, food processing sites that handle ingredients, or cosmetics or pharmaceutical manufacturing sites.
  • the robot control system 100 can be used to control the robot 1 such as a communication robot or a service robot that interacts with a user.
  • the robot 1 has an information acquisition device 3 such as a camera
  • the robot control system 100 determines whether or not to select a control program based on the result of recognizing the face of the user who is the target of interaction or service provision. good.
  • the robot control system 100 can be applied, for example, when it is necessary to change the control program according to a change in the situation, such as when the user is wearing glasses or wearing a mask.
  • the control unit 11 of the information processing device 10 may acquire usage information indicating the usage status of the control program candidate selected by the user, and estimate suitability of the candidate based on the degree of similarity and the usage information.
  • the usage information may represent working hours in a day, or work periods in a year or a month. Further, the usage information may represent the user who causes the robot 1 to perform the work, or may represent the location where the robot 1 is caused to perform the work. The usage information may also represent the most recent time at which the robot 1 was caused to perform a task, or the most recent frequency at which the robot 1 was caused to perform a task.
  • the control unit 11 When the control program to be executed by the robot controller 2 is determined, the control unit 11 newly associates the preparation environment information corresponding to at least a part of the work environment 4 of the robot 1 with the determined control program as past record information. you can
  • the new past record information associated with the determined control program is also called third environmental information.
  • Associating third environment information with the control program is also referred to as recording third environment information.
  • the recording of the third environment information may be performed regardless of whether the candidate selected by the user is determined to be appropriate or is estimated to be inappropriate.
  • the recording of the third environment information may be executed when the candidate selected by the user is instructed to be used as it is even though the control unit 11 determines that the candidate selected by the user is inappropriate. .
  • the recording of the third environment information may be executed when the user instructs to use another candidate.
  • the control unit 11 can improve the accuracy of estimating the suitability of the control program candidate.
  • AI Artificial Intelligence
  • the environment information to which the control program is applied may be updated by re-learning, or the control unit 11 estimates the adequacy of the candidate for the control program. model may be updated by retraining.
  • the control unit 11 may modify the criteria for judging the suitability of the control program candidate. For example, when the control unit 11 estimates that the candidate is appropriate when the similarity is 90% or more, the candidate is appropriate even when the similarity is 80% or more based on the third environment information. The criteria may be modified to estimate that Further, when the control unit 11 estimates that the environment is inadequate when the similarity is 80% or more and less than 90%, the environment is inadequate when the similarity is 75% or more and less than 90%. The criteria may be modified to assume that there is, and assume that if the similarity is less than 75%, then the selection is poor.
  • the allowable range for judging that the similarity is 90% may change.
  • the relationship between at least a part of the information of the work environment 4 and the past recorded information whose similarity was less than 90% before the revision of the criterion may have a similarity of 90% or more after the revision of the criterion.
  • the control unit 11 may detect an insufficient environment or insufficient selection based on the positional relationship of multiple objects.
  • the control unit 11 may detect an insufficient environment or an insufficient selection further based on information that associates the past record information with the control program.
  • the robot control system 100 determines whether the control program selected by the user is appropriate.
  • the robot control system 100 may display control program candidates on the terminal device 9 when the user selects a control program.
  • the control unit 11 of the robot control system 100 through the robot controller 2, the terminal device 9, the information acquisition device 3, or the like, for example, the attribute information of the robot 1 that the user plans to use, the information of the user who plans to use the robot 1,
  • information such as the recognized work target that is scheduled to be worked is acquired, and as a control program candidate, for example, the robot 1 that the user plans to use, the robot that is the same type as the robot 1 that is scheduled to be used, or the work target that is scheduled to be worked,
  • a control program frequently executed within a predetermined period or a control program frequently executed by a specific user may be extracted.
  • the robot control system 100 may calculate an index called a degree of association.
  • the control unit 11 of the information processing device 10 may calculate the degree of association so that, for example, a control program that has been executed with a high frequency in a predetermined period before the present is given a high priority.
  • the control unit 11 may calculate the degree of association so that the most recently executed control program is given a higher priority.
  • the control unit 11 assigns high priority to the control program executed most recently by the user who prepares the work environment 4 and causes the robot 1 to execute the work, or the control program executed frequently by the user. , the relevance may be calculated.
  • the control unit 11 may extract the control program based on the calculated value of the index such as the degree of association.
  • a storage medium on which the program is recorded for example, an optical disk, Magneto-optical disk, CD-ROM, CD-R, CD-RW, magnetic tape, hard disk, memory card, etc.
  • the program may be stored on a non-transitory computer-readable medium.
  • the implementation form of the program is not limited to an application program such as an object code compiled by a compiler or a program code executed by an interpreter. good.
  • the program may or may not be configured so that all processing is performed only in the CPU on the control board.
  • the program may be configured to be partially or wholly executed by another processing unit mounted on an expansion board or expansion unit added to the board as required.
  • Embodiments according to the present disclosure are not limited to any specific configuration of the embodiments described above. Embodiments of the present disclosure extend to any novel feature or combination thereof described in the present disclosure or any novel method or process step or combination thereof described. be able to.
  • Descriptions such as “first” and “second” in this disclosure are identifiers for distinguishing the configurations. Configurations that are differentiated in descriptions such as “first” and “second” in this disclosure may interchange the numbers in that configuration.
  • the first tray 6 can exchange the identifiers “first” and “second” with the second tray 7 . The exchange of identifiers is done simultaneously. The configurations are still distinct after the exchange of identifiers. Identifiers may be deleted. Configurations from which identifiers have been deleted are distinguished by codes. The description of identifiers such as “first” and “second” in this disclosure should not be used as a basis for interpreting the order of the configuration or the existence of lower numbered identifiers.
  • robot control system (1: robot, 2: robot controller, 3: information acquisition device, 4: work environment, 5: workbench, 6: first tray, 7: second tray, 8: work) 9 terminal device 10 information processing device (11: control unit, 12: storage unit)) 80 network

Landscapes

  • Engineering & Computer Science (AREA)
  • Robotics (AREA)
  • Mechanical Engineering (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Automation & Control Theory (AREA)
  • Manipulator (AREA)
PCT/JP2022/036312 2021-09-28 2022-09-28 情報処理装置、ロボットコントローラ、ロボット制御システム、及び情報処理方法 Ceased WO2023054535A1 (ja)

Priority Applications (4)

Application Number Priority Date Filing Date Title
US18/696,301 US20240391105A1 (en) 2021-09-28 2022-09-28 Information processing device, robot controller, robot control system, and information processing method
CN202280065536.4A CN118019623A (zh) 2021-09-28 2022-09-28 信息处理设备、机器人控制器、机器人控制系统和信息处理方法
EP22876401.5A EP4410500A4 (en) 2021-09-28 2022-09-28 INFORMATION PROCESSING DEVICE, ROBOT CONTROL DEVICE, ROBOT CONTROL SYSTEM, AND INFORMATION PROCESSING METHOD
JP2023551815A JP7678121B2 (ja) 2021-09-28 2022-09-28 情報処理装置、ロボットコントローラ、ロボット制御システム、及び情報処理方法

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2021158480 2021-09-28
JP2021-158480 2021-09-28

Publications (1)

Publication Number Publication Date
WO2023054535A1 true WO2023054535A1 (ja) 2023-04-06

Family

ID=85780758

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2022/036312 Ceased WO2023054535A1 (ja) 2021-09-28 2022-09-28 情報処理装置、ロボットコントローラ、ロボット制御システム、及び情報処理方法

Country Status (5)

Country Link
US (1) US20240391105A1 (https=)
EP (1) EP4410500A4 (https=)
JP (1) JP7678121B2 (https=)
CN (1) CN118019623A (https=)
WO (1) WO2023054535A1 (https=)

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2005138245A (ja) * 2003-11-07 2005-06-02 Yaskawa Electric Corp 人間介入型ロボットの制御装置
JP2006343828A (ja) * 2005-06-07 2006-12-21 Fanuc Ltd ロボット制御装置及びロボット制御方法
JP2018051652A (ja) * 2016-09-27 2018-04-05 株式会社日立製作所 ロボットシステム
JP2020016446A (ja) * 2018-07-23 2020-01-30 キヤノン株式会社 情報処理装置、情報処理装置の制御方法、プログラム、計測装置、及び物品製造方法
JP2020099976A (ja) * 2018-12-25 2020-07-02 セイコーエプソン株式会社 ロボット管理システム
JP2020135045A (ja) * 2019-02-13 2020-08-31 ファナック株式会社 機械の制御装置及び制御方法
JP2021030407A (ja) 2019-08-29 2021-03-01 キヤノン株式会社 情報処理装置、情報処理方法及びプログラム
US20210122051A1 (en) * 2019-10-24 2021-04-29 Samsung Sds Co., Ltd. Robot process automation apparatus and method for detecting changes thereof
JP2021158480A (ja) 2020-03-26 2021-10-07 セイコーエプソン株式会社 画像読取装置

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPS5822412A (ja) * 1981-08-04 1983-02-09 Fanuc Ltd 工業用ロボツト制御方式
JP6754883B1 (ja) * 2019-11-27 2020-09-16 株式会社安川電機 制御システム、ローカルコントローラ及び制御方法

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2005138245A (ja) * 2003-11-07 2005-06-02 Yaskawa Electric Corp 人間介入型ロボットの制御装置
JP2006343828A (ja) * 2005-06-07 2006-12-21 Fanuc Ltd ロボット制御装置及びロボット制御方法
JP2018051652A (ja) * 2016-09-27 2018-04-05 株式会社日立製作所 ロボットシステム
JP2020016446A (ja) * 2018-07-23 2020-01-30 キヤノン株式会社 情報処理装置、情報処理装置の制御方法、プログラム、計測装置、及び物品製造方法
JP2020099976A (ja) * 2018-12-25 2020-07-02 セイコーエプソン株式会社 ロボット管理システム
JP2020135045A (ja) * 2019-02-13 2020-08-31 ファナック株式会社 機械の制御装置及び制御方法
JP2021030407A (ja) 2019-08-29 2021-03-01 キヤノン株式会社 情報処理装置、情報処理方法及びプログラム
US20210122051A1 (en) * 2019-10-24 2021-04-29 Samsung Sds Co., Ltd. Robot process automation apparatus and method for detecting changes thereof
JP2021158480A (ja) 2020-03-26 2021-10-07 セイコーエプソン株式会社 画像読取装置

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP4410500A4

Also Published As

Publication number Publication date
EP4410500A4 (en) 2025-10-08
JP7678121B2 (ja) 2025-05-15
EP4410500A1 (en) 2024-08-07
CN118019623A (zh) 2024-05-10
US20240391105A1 (en) 2024-11-28
JPWO2023054535A1 (https=) 2023-04-06

Similar Documents

Publication Publication Date Title
WO2023054539A1 (ja) 情報処理装置、ロボットコントローラ、情報処理システム、及び情報処理方法
US10857673B2 (en) Device, method, program and recording medium, for simulation of article arraying operation performed by robot
JP7495688B2 (ja) ロボットシステムの制御方法及び制御装置
JP2020055096A (ja) 情報処理装置、情報処理方法及びシステム
US20210162600A1 (en) Method of programming an industrial robot
US12263598B2 (en) Control device, picking system, distribution system, program, control method and production method
CN110520259B (zh) 控制装置、拾取系统、物流系统、存储介质以及控制方法
TWI649169B (zh) 把持位置姿勢教導裝置、把持位置姿勢教導方法及機器人系統
US20200030976A1 (en) Control device, picking system, distribution system, program, control method and production method
JP2021030407A (ja) 情報処理装置、情報処理方法及びプログラム
CN114670189B (zh) 存储介质、以及生成机器人的控制程序的方法及系统
US20240051135A1 (en) Interference determination device, robot control system, and method for determining interference
CN114080590A (zh) 使用先进扫描技术的机器人料箱拾取系统和方法
CN116940448A (zh) 程序管理装置、机器人控制系统和用于管理程序的方法
CN117794709A (zh) 握持参数估计设备和握持参数估计方法
JP7678121B2 (ja) 情報処理装置、ロボットコントローラ、ロボット制御システム、及び情報処理方法
CN115063670A (zh) 一种自动分拣方法、装置及系统
CN111470244B (zh) 机器人系统的控制方法以及控制装置
CN111899629A (zh) 柔性机器人教学系统和方法
WO2023120728A1 (ja) ロボット制御装置及びロボット制御方法
WO2022250152A1 (ja) 保持位置決定装置、及び保持位置決定方法
EP4703096A1 (en) Robot control device, robot control system, trained model, and method for generating trained model
CN118139730A (zh) 信息处理设备、机器人控制器、机器人控制系统和信息处理方法
US20240265669A1 (en) Trained model generating device, trained model generating method, and recognition device
EP4491353A1 (en) Robot control device and robot control method

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22876401

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 18696301

Country of ref document: US

Ref document number: 202280065536.4

Country of ref document: CN

Ref document number: 2023551815

Country of ref document: JP

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2022876401

Country of ref document: EP

Effective date: 20240429