WO2023167324A1 - Production system and reproduction method - Google Patents

Abstract

A production system 1 comprises: a simulator 214 with which, on the basis of transition information representing transition of a control signal collected from a controller 3 that controls a production apparatus 2 including a robot 5B and model information about the production apparatus 2, a virtual production apparatus V2 corresponding to the production apparatus 2 is operated in a virtual space VS; and a display part 216 for displaying, on a monitor, the virtual space VS in which the virtual production apparatus V2 is operated.

Description

Production system and reproduction method

This disclosure relates to a production system and a reproduction method.

Patent Document 1 describes a controller that operates one or more robots in a real space based on an operation program, and a virtual robot that operates one or more virtual robots corresponding to the one or more robots in a virtual space based on an operation program. A controller, an action interrupting unit that interrupts the motion of one or more robots by the controller, an interrupted state reproducing unit that reproduces in a virtual space the state of the real space in which the motion of the one or more robots is interrupted, and an interrupted state reproducing unit. a restart simulator for restarting the motion of one or more virtual robots based on a motion program by a virtual controller in a virtual space that reproduces the state of the real space.

Patent No. 6898506

The present disclosure provides an effective system for post-verification of the operation of production equipment.

A production system according to one aspect of the present disclosure provides a virtual machine corresponding to a production apparatus based on transition information representing transition of control signals collected from a controller that controls a production apparatus including a robot, and model information of the production apparatus. A simulator for operating a production device in a virtual space, and a display section for displaying a virtual space in which the virtual production device operates on a monitor.

A reproduction method according to another aspect of the present disclosure is a virtual machine corresponding to a production device based on transition information representing transitions of control signals collected from a controller that controls a production device including a robot, and a model of the production device. It includes operating the production equipment in the virtual space and displaying the virtual space in which the virtual production equipment operates on the monitor.

According to the present disclosure, it is possible to provide an effective system for post-verification of the operation of production equipment.

1 is a schematic diagram illustrating the configuration of a production system; FIG. 1 is a schematic diagram illustrating the configuration of a robot; FIG. 2 is a block diagram illustrating functional configurations of a data collection device and a simulation device; FIG. FIG. 10 is a diagram illustrating extraction of state changes of a virtual peripheral object; FIG. 10 is a diagram illustrating extraction of state changes of a virtual peripheral object; It is a block diagram which shows the modification of a simulation apparatus. FIG. 11 is a block diagram showing a further modified example of the simulation device; FIG. 4 is a diagram illustrating a display screen; 2 is a block diagram illustrating hardware configurations of a data collection device, a simulation device, a host controller, and a local controller; FIG. 9 is a flowchart illustrating a procedure for displaying an interval setting screen; 8 is a flowchart illustrating a procedure for displaying an initial screen; 12 is a flowchart illustrating a simulation procedure in FIG. 11; 5 is a flowchart illustrating a display update procedure; 14 is a flowchart illustrating a screen transition procedure in FIG. 13; FIG. 14 is a flowchart illustrating a reproduction procedure in FIG. 13; FIG.

Hereinafter, embodiments will be described in detail with reference to the drawings. In the description, the same reference numerals are given to the same elements or elements having the same function, and overlapping descriptions are omitted.

[Production system]
A production system 1 shown in FIG. 1 is a system for producing a product in a physical space. A product may be any tangible object produced by mechanical processing and assembly of one or more parts. Real space is the space in which tangible objects actually exist.

The production system 1 includes a production device 2 and a controller 3. production equipment 2. A product is produced by executing a processing process on the work W in the real space. The work W is a tangible object handled by the production apparatus 2 to constitute at least part of the product. For example, the work W may be a part to be assembled into a product, an intermediate product formed by assembling parts or the like, or a final finished product itself.

The processing process for work W includes multiple tasks. A task is a pre-ordered set of actions for a given work purpose. At least one of the plurality of tasks can be shared by two or more different processing steps. Depending on the content of the processing steps, the execution order of multiple tasks can be changed, but the order of operations within one task cannot be changed. Examples of multiple tasks include bringing the base part into the work area, assembling the part to the base part, fixing the part to the base part by fastening or welding, etc., and assembling one or more parts to the base part. Carrying out the completed product from the work area, and the like.

For example, the production equipment 2 includes a plurality of machines 5. The plurality of machines 5 includes at least robots. The plurality of machines 5 may further include machines other than robots. Examples of machines other than robots include transfer devices that transfer workpieces W, devices that adjust the position and orientation of workpieces W to be worked by robots, machine tools that process workpieces W, and the like. is not limited to A plurality of machines 5 can have any structure as long as they can execute a task for the work W. FIG.

A plurality of tasks may include two or more tasks to be executed sequentially in series, or may include two or more tasks that can be executed concurrently by a plurality of machines 5 . Examples of tasks executed by the robot include transportation of the work W, assembly of the work W, fixation of the work W by fastening, loading/unloading of the work W to/from peripheral machines such as machine tools, and the like. Examples of tasks executed by the machine tool include opening and closing the door, chucking the loaded workpiece W, rotating/moving the workpiece W, changing tools, arranging/moving the tool relative to the workpiece W, and releasing the chuck after machining. is mentioned.

The plurality of machines 5 shown in FIG. 1 includes a transfer device 5A, robots 5B, 5C, and 5D, and a machine tool 5E, but is not limited to this. The number and type of machines 5 can vary in any way, at least as long as they include robots.

The transport device 5A transports the work W by being driven by, for example, an electric motor. Examples of the conveying device 5A include belt conveyors, roller conveyors, carousels, and the like.

The robots 5B and 5C are vertically articulated robots, and have an articulated arm 10 and an end effector 50, as shown in FIG. The end effector 50 acts on the workpiece W. As shown in FIG. Examples of the end effector 50 include a hand for gripping the work W, a suction nozzle for sucking the work W, a welding torch for welding the work W, a screw tightening tool for screwing the work W, and the like.

The end effector 50 is connected to the articulated arm 10. The multi-joint arm 10 changes the position and orientation of the end effector 50 by multi-joint motion. For example, the articulated arm 10 includes a base portion 11, a turning portion 12, a first arm 13, a second arm 14, a swing portion 15, a third arm 17, a tip portion 18, actuators 41, 42, 43, 44, 45, 46. The base 11 is installed around the transport device 5A. The swivel part 12 is provided on the base part 11 so as to swivel around a vertical axis 21 . The first arm 13 is connected to the swivel portion 12 so as to swing about an axis 22 that intersects (for example, is perpendicular to) the axis 21 . Crossing includes a twisted relationship such as a so-called overpass. The second arm 14 is connected to the tip of the first arm 13 so as to swing about an axis 23 substantially parallel to the axis 22 . The second arm 14 includes a swinging portion 15 and a turning portion 16 . The swinging portion 15 is connected to the tip of the first arm 13 and extends along an axis 24 that intersects (for example, orthogonally) the axis 23 . The swivel portion 16 is connected to the distal end portion of the swing portion 15 so as to swivel around the axis 24 and further extends along the axis 24 . The third arm 17 is connected to the distal end of the turning section 16 so as to swing about an axis 25 intersecting (for example, perpendicular to) the axis 24 . The distal end portion 18 is connected to the distal end portion of the third arm 17 so as to pivot about an axis 26 that intersects (for example, is perpendicular to) the axis 25 . An end effector 50 is attached to the distal end portion 18 .

Thus, the articulated arm 10 includes a joint 31 connecting the base portion 11 and the swivel portion 12, a joint 32 connecting the swivel portion 12 and the first arm 13, the first arm 13 and the second arm . a joint 34 connecting the swinging portion 15 and the turning portion 16 in the second arm 14; a joint 35 connecting the turning portion 16 and the third arm 17; and a joint 36 connecting with the portion 18 .

The actuators 41, 42, 43, 44, 45, 46 include, for example, electric motors and speed reducers, and drive the joints 31, 32, 33, 34, 35, 36, respectively. For example, the actuator 41 swings the swivel unit 12 about the axis 21, the actuator 42 swings the first arm 13 about the axis 22, the actuator 43 swings the second arm 14 about the axis 23, The actuator 44 pivots the pivot 16 about the axis 24 , the actuator 45 pivots the third arm 17 about the axis 25 , and the actuator 46 pivots the tip 18 about the axis 26 .

The configuration of the articulated arm 10 can be changed as appropriate. For example, the articulated arm 10 may be a 7-axis redundant robot in which a 1-axis joint is added to the 6-axis configuration described above, or a so-called scalar type articulated robot.

Returning to FIG. 1, the robot 5D is a robot capable of autonomous travel. For example, the robot 5D is similar to the robots 5B and 5C, but the base 11 of the articulated arm 10 is self-propelled. An example of the self-propellable base 11 is an electric automatic guided vehicle (AGV: Automated Guided Vehicle).

The machine tool 5E performs cutting and the like on the work W that is carried in and out by the robots 5B and 5C. Examples of the machine tool 5E include an NC lathe, an NC milling machine, a machining center, and the like. The production device 2 may further include peripheral devices whose movable ranges overlap with any of the robots 5B, 5C, and 5D. For example, the movable range of the robot 5C (peripheral device) may overlap with that of the robot 5B.

The controller 3 controls a plurality of machines 5. For example, the controller 3 has a plurality of local controllers 400 each controlling a plurality of machines 5 and a host controller 300 . For example, the plurality of local controllers 400 include a local controller 400A that controls the transfer device 5A, a local controller 400B that controls the robot 5B, a local controller 400C that controls the robot 5C, a local controller 400D that controls the robot 5D, and a machine tool. and a local controller 400E that controls the machine 5E.

The host controller 300 collects state information of the physical space, and outputs task execution commands to a plurality of local controllers 400 based on a predetermined control program and the collected state information. Each of the multiple local controllers 400 controls the machine 5 based on execution instructions output from the host controller 300 .

For example, the local controller 400 determines the output to be supplied to the machine 5 in order to perform the task corresponding to the execution command, and supplies the determined output to the machine 5 . To determine the output, local controller 400 acquires, generates, or outputs one or more control signals.

For example, the local controller 400B determines the output to be supplied to the robot 5B in order to perform the task corresponding to the execution command, and supplies the determined output to the robot 5B at a predetermined control cycle from the start to the completion of the task. Execute repeatedly. In order to supply the above outputs to the robot 5B, the local controller 400B acquires, generates or outputs one or more control signals in each control cycle. The execution command may include designation of one of a plurality of motion programs held by local controller 400B. For example, the local controller 400B reads out a motion program corresponding to the execution command, calculates the target position/target orientation of the end effector 50 for performing the task based on the read motion program, and calculates the target position/target orientation. Calculating the target angles of the joints 31, 32, 33, 34, 35, and 36 corresponding to , obtaining the feedback values of the angles of the joints 31, 32, 33, 34, 35, and 36, and the target angles and Based on the deviation from the feedback value, target currents (examples of the above outputs) to be supplied to the actuators 41, 42, 43, 44, 45, and 46 are calculated; , 43, 44, 45, and 46 are repeatedly executed at the control period. the execution command described above, the target position/target orientation of the end effector 50, the target angles of the joints 31, 32, 33, 34, 35, and 36, the feedback values of the angles of the joints 31, 32, 33, 34, 35, and 36; All target currents are included in one or more control signals. The one or more control signals also include a program execution history indicating which part (eg, line or block) of the motion program has been executed.

The local controller 400B may output commands to the end effector 50 placed at the target position/target posture. For example, when the end effector 50 is a hand, the local controller 400B may output a workpiece W gripping command to the end effector 50 . The local controller 400B may output a screw tightening command to the end effector 50 when the end effector 50 is a screw tightening tool. Both the grip command and the screw tightening command are included in one or more control signals.

The robot 5B may have cameras 51 and 52 on the tip portion 18 or the turning portion 12 or the like. In this case, the local controller 400B may recognize the task execution target based on the imaging data acquired from the cameras 51 and 52, and calculate the target position/target orientation of the end effector 50 based on the recognition result. good. The recognition result of the task execution target is included in the control signal of the production system 1 .

The local controller 400C also acquires, generates, or outputs one or more control signals similar to those of the local controller 400B in each control cycle. The local controller 400D of the robot 5D whose base 11 is movable has, in addition to one or more control signals similar to those of the local controller 400B, a target position of the base 11, a current position of the base 11, a target position of the base 11 and a current position of the base 11. A target speed or the like of the base 11 based on the deviation from is obtained, generated, or output for each control cycle. The target position of the base 11, the current position of the base 11, and the target speed of the base 11 are also included in the one or more control signals.

The local controller 400E sequentially outputs operation commands to the machine tool 5E for performing tasks corresponding to the execution commands. The execution command may include designation of one of a plurality of machining programs held by the local controller 400E. For example, the local controller 400E reads a machining program corresponding to an execution command, and based on the read machining program, a door opening command, a work W chucking command, a tool setting command, a tool placement command, a tool or work rotation command. A drive command, a tool or work feed command, etc. are sequentially output to the machine tool 5E. The local controller 400E may acquire a notification of completion of the corresponding operation from the machine tool 5E for each operation command, and output the next operation command to the machine tool 5E after receiving the completion notification. All of the above execution commands, each motion command, and completion notification for each motion command are included in one or more control signals. The one or more control signals also include a program execution history indicating which part (eg, line or block) of the machining program has been executed.

The control signal acquired, generated or output by each of the plurality of local controllers 400 may further include one or more status signals representing the status of at least one of the corresponding machine 5 and local controller 400. For example, each of the multiple local controllers 400 may generate a completion notification indicating that the task has been completed normally, and output it to the upper controller 300 . Further, each of the plurality of local controllers 400 may generate an alarm to report an abnormality in the machine 5 or an abnormality in the local controller 400 and output it to the host controller 300 . For example, the local controllers 400B, 400C, 400D may generate an alarm notifying the occurrence of overload or overcurrent in at least one of the actuators 41, 42, 43, 44, 45, 46 and output it to the host controller 300. good. All of the completion notifications and alarms described above are included in one or more status signals.

The upper controller 300 collects one or more control signals from multiple local controllers 400 . For example, the host controller 300 performs wired or wireless synchronous communication with a plurality of local controllers 400 to collect one or more control signals. Synchronous communication means that a plurality of machines 5 also communicate with the local controller 400 every cycle in synchronization with a synchronous frame of a constant communication cycle. The host controller 300 may store state records in which one or more control signals collected in the same cycle are associated with the same time.

The host controller 300 may further collect position information and the like of the work W based on the detection results of the environment sensor 6 provided in the physical space, and store the information as part of the state record. The positional information of the workpiece W and the like collected by the host controller 300 may be obtained as one or more control signals by a plurality of local controllers 400 .

Examples of the environment sensor 6 include a load sensor provided at a position where the work W can be placed, or a camera that captures the position where the work W can be placed. If the environment sensor 6 is a camera, the host controller 300 may generate position information and the like of the workpiece W based on the image processing result of image data captured by the environment sensor 6 . The host controller 300 may further collect data captured by the environment sensor 6 and the cameras 51 and 52, and store the collected captured data in the state record.

The upper controller 300 generates task execution commands for the multiple local controllers 400 based on the stored status records and production instructions based on the production plan.

At least one of the plurality of local controllers 400 (for example, local controllers 400B, 400C, and 400D) may autonomously determine the execution timing of the task corresponding to the execution instruction based on the status record stored by the host controller 300. good.

The data collection device 100 accumulates transition information representing transitions of the multiple types of control signals collected by the multiple local controllers 400 . For example, the data collection device 100 acquires and sequentially accumulates state records stored by the host controller 300 . As a result, transition information representing transitions of the plurality of control signals is accumulated. Transition means change over time, and is represented by a plurality of control signals arranged in time series. The plurality of control signals need only be stored in association with time so that they can be arranged in chronological order, and they do not necessarily have to be arranged in chronological order within the data collection device 100 . Note that the data collection device 100 may acquire transition information about at least one of the plurality of types of control signals from any one of the plurality of local controllers 400 without going through the host controller 300 . At least one of the plurality of machines 5 may be directly controlled by the host controller 300 without going through the local controller 400 . Control signals for the machine 5 directly controlled by the host controller 300 are collected by the host controller 300 . All machines 5 may be directly controlled by the host controller 300 (the controller 3 may not be divided into the host controller 300 and a plurality of local controllers 400).

The simulation apparatus 200 executes a simulation of operating a virtual production apparatus V2 (a model of the production apparatus 2) corresponding to the production apparatus 2 in the virtual space VS for various purposes such as prior verification of the operation of the production apparatus 2. is configured as Operating the virtual production apparatus V2 in the virtual space VS is based on the model information of the production apparatus 2, without actually operating the plurality of machines 5. Simulation data representing the state of the plurality of machines 5 after operation is performed. means to calculate The model information of the production equipment 2 includes model information of a plurality of machines 5 . Each of the model information of the plurality of machines 5 includes, for example, information on the arrangement, structure, shape, size, etc. of the corresponding machine 5 in the real space. The shape and size of the machine 5 in the physical space are represented by three-dimensional data such as polygon data or voxel data.

It can also be said that the simulation data calculated based on the model information of the production device 2 is the virtual production device V2 (model of the production device 2), and the coordinate system on which the simulation data is based is the virtual space VS. The configurations of the data collection device 100 and the simulation device 200 are illustrated in detail below.

[Data collection device and simulation device]
As illustrated in FIG. 3, the data collection device 100 includes a collection unit 111 and a database 113 as functional components (hereinafter referred to as "function blocks"). The collection unit 111 acquires status records from the host controller 300 and sequentially accumulates them in the database 113 .

The simulation device 200 has a program storage unit 211, a virtual controller 212, a model holding unit 213, a simulator 214, a result holding unit 215, and a display unit 216 as functional blocks. The program storage unit 211 stores programs executed by each of the host controller 300 and the plurality of local controllers 400 to control the plurality of machines 5 (for example, the above control program, a plurality of motion programs, a plurality of machining programs, etc.). are doing. The virtual controller 212 executes programs stored in the program storage unit 211 in the same procedure as when the host controller 300 and the plurality of local controllers 400 control the plurality of machines 5 . The model holding unit 213 stores model information of one or more models included in the virtual space VS. The one or more models include at least the models of the production device 2 (models of the plurality of machines 5) described above. The simulator 214 causes the virtual production apparatus V2 to operate in the virtual space VS based on the model information stored by the model holding unit 213 and the execution result of the program by the virtual controller 212 .

As an example, the simulator 214 virtualizes the model of the robot 5B by forward kinematics calculation based on the motion angles of the joints 31, 32, 33, 34, 35, and 36 calculated by the virtual controller 212 and the model information of the robot 5B. Operate in space VS. The same applies to robots 5C and 5D.

The simulator 214 operates the virtual production apparatus V2 in the virtual space VS each time the execution result of the program by the virtual controller 212 is updated, and accumulates the simulation results including the virtual production apparatus V2 after the operation in the result holding unit 215. . The simulation results include models of multiple machines 5 after operation. The models of the machines 5 include the three-dimensional data representing the appearance of the machines 5 after operation.

The display unit 216 displays the virtual space VS in which the virtual production device V2 is operating on a monitor (for example, the monitor of the user interface 295 described later) based on the simulation results accumulated in the result holding unit 215 in chronological order. For example, the display unit 216 reads the simulation results accumulated in the result holding unit 215 in chronological order, converts the three-dimensional data included in the simulation results into two-dimensional data from a predetermined viewpoint, and displays the data on the monitor. As a result, the moving image of the virtual space VS in which the virtual production device V2 is operating is reproduced on the monitor.

By using the simulation, it is possible to construct the production apparatus 2 efficiently while verifying the operation of the virtual production apparatus V2 in the virtual space VS. It may be necessary to verify the operation of the after the fact. For example, when an unexpected event occurs in the operation of the production device 2 in the real space, it may be necessary to verify the cause of the event after the fact.

In order to verify the operation of the production device 2 in the real space after the fact, it is conceivable to reproduce the photographed data photographed in the real space. However, it is difficult to photograph the entire production apparatus 2 including a plurality of machines 5, and there is a limit to post facto verification. Therefore, the simulation device 200 operates the virtual production device V2 in the virtual space VS based on the transition information representing the transition of the control signal collected from the controller 3 and the model information of the production device 2, and the transition information and displaying on a monitor a virtual space VS in which the virtual production apparatus V2 operates based on the above.

The control signals are, for example, past control signals acquired, generated, or output by the plurality of local controllers 400 when operating the plurality of machines 5 in the physical space, as described above. As described above, transition means change over time and is represented by a plurality of control signals arranged in time series.

Hereinafter, displaying on the monitor the virtual space VS in which the virtual production apparatus V2 operates based on the transition information of the past control signal is referred to as "log reproduction". According to the log reproduction, even if the entirety of the production apparatus 2 having a plurality of machines 5 is not photographed thoroughly, phenomena occurring in the real space can be reproduced. Reproduction results can also be displayed from any viewpoint. Therefore, it is effective for ex-post verification of system operation.

For example, the simulation device 200 further includes a transition information extraction unit 217 and a transition information storage unit 218 as functional blocks. The transition information extraction unit 217 extracts the transition information of the target section of log reproduction from the database 113 and stores it in the transition information storage unit 218 . The target section means a section on the time axis of transition information. For example, the transition information extraction unit 217 extracts a plurality of types of control signals included in the state record corresponding to the target section from the database 113 and stores them in the transition information storage unit 218 in association with the time included in the state record.

When executing log reproduction, the simulator 214 causes the virtual production apparatus V2 to operate in the virtual space VS based on transition information stored in the transition information storage unit 218, instead of the result of program execution by the virtual controller 212. For example, the simulator 214 reads multiple types of control signals from the transition information storage unit 218 in chronological order, and performs virtual production based on the read multiple types of control signals and the model of the production apparatus 2 stored in the model holding unit 213. Device V2 is operated in virtual space VS.

As an example, the simulator 214 corresponds to the robot 5B by forward kinematics calculation based on the angles of the joints 31, 32, 33, 34, 35, and 36 read from the transition information storage unit 218 and the model information of the robot 5B. The virtual robot V5B is operated in the virtual space VS. The same applies to robots 5C and 5D.

Every time one or more control signals are read from the transition information storage unit 218, the simulator 214 causes the virtual production apparatus V2 to operate in the virtual space VS, and accumulates the simulation results including the virtual production apparatus V2 after operation in the result holding unit 215. do.

The display unit 216 causes the monitor to display the virtual space VS in which the virtual production device V2 is operating, based on the simulation results accumulated in the result holding unit 215 in time series corresponding to the transition information of the target section. For example, the display unit 216 reads a plurality of simulation results corresponding to the transition information of the target section in chronological order, converts the three-dimensional data included in the simulation results into two-dimensional data from a predetermined viewpoint, and displays the data on the monitor. . As a result, a moving image of the virtual space VS in which the virtual production apparatus V2 is operating is reproduced on the monitor based on the transition information of the target section.

As described above, the operation of the production apparatus 2 in the past can be reproduced in the virtual space VS. cannot be reproduced in the virtual space VS. Examples of peripheral objects include a work W, a work tool held by the end effector 50, and the like.

Therefore, the simulation apparatus 200 further extracts the state change of the virtual space VS based on the operation of the virtual production apparatus V2 in the virtual space VS, and displays the extracted state change of the virtual space VS on the monitor. may be configured to run. The transition of the control signal stored in the database 113 can also be effectively used to reproduce the state change of surrounding objects. By including state changes of virtual surrounding objects in the display, phenomena occurring in the real space can be verified more easily.

For example, the model holding unit 213 stores model information of one or more peripheral objects in addition to the model information of the production device 2 . The model information of the surrounding objects includes information such as the arrangement, structure, shape and size of the surrounding objects. Based on the model information of one or more surrounding objects, the simulator 214 accumulates, in the result holding unit 215, simulation results further including one or more virtual surrounding objects (one or more surrounding object models) arranged in the virtual space VS. do.

For example, the simulation device 200 further has an extraction unit 219 as a functional block. The extraction unit 219 extracts state changes in the virtual space VS based on the operation of the virtual production apparatus V2 in the virtual space VS.

For example, the extraction unit 219 extracts the state change of the virtual space VS in the portion not operated by the control signal, based on the operation of the virtual production apparatus V2 by the control signal. For example, when one virtual machine V5 (model of machine 5) operates according to the control signal, the extraction unit 219 extracts state changes of virtual peripheral objects (models of peripheral objects) that do not operate according to the control signal. The virtual peripheral object may be a work W or work tool, or may be another virtual machine V5 (a model of another machine 5).

The extraction unit 219 may extract at least one state change of a plurality of models based on the positional relationship between the plurality of models (the plurality of models included in the virtual space VS) stored in the model holding unit 213. Based on the positional relationship, state changes can be easily extracted.

A state change may be a change in the position of the model, a change in the shape of the model, or a change in the size of the model. As an example, when a plurality of machines 5 includes a grinder for grinding, and a grindstone model in the grinder interferes with a model of the work W in the virtual space VS, the extraction unit 219 extracts a part of the work W that interfered with the model of the grindstone. We may extract the changes to remove from the model of .

The state change is from a state in which the position of the model in the production system 1 is independent of the positions of the other models (hereinafter referred to as "independent state") to a state in which the position of the model in the production system 1 is related to the positions of the other models. It may be a change to a state (hereinafter referred to as "subordinate state"). As an example, when the end effector 50 of the robot 5B is a suction nozzle and the model of the end effector 50 contacts the model of the work W in the virtual space VS, the extraction unit 219 extracts the model of the work W from the model of the end effector 50. has changed from the independent state to the dependent state.

When the change from the independent state to the dependent state is extracted, the extraction unit 219 converts the position/orientation of the model of the work W in the virtual space VS into a relative position/orientation based on the position/orientation of the model of the end effector 50 . You can rewrite the posture. After the rewriting, when the simulator 214 operates the virtual production device V2 in the virtual space VS, the model of the work W moves together with the model of the end effector 50 based on the relative position/relative orientation. Since the movement of the model of the work W accompanying the movement of the model of the end effector 50 is displayed on the monitor by the display unit 216, the change from the independent state to the dependent state is displayed on the monitor.

It is conceivable that it may not be possible to accurately extract state changes from only the positional relationships of multiple models. For example, when the end effector 50 is a hand and the model of the end effector 50 approaches the model of the peripheral object in the virtual space VS, the extracted The state changes that should occur can be different.

For example, if the peripheral object is a peripheral device having a button, and the purpose of the approach is to press the button with the end effector 50, the state change to be extracted is from before the button is pressed to after the button is pressed. Change. When the surrounding object is the work W and the purpose of the approach is to grip the work W with the end effector 50, the change to be extracted is that the state of the model of the work W with respect to the model of the end effector 50 is the independent state to the above dependent state.

Therefore, the extraction unit 219 may extract at least the state change of the model of the production system 1 included in the virtual space VS based on the control signal and the purpose information associated with the control signal. Based on the purpose information, it is possible to more accurately extract the state change of the virtual peripheral object.

The purpose information represents, for example, the work purpose of the task corresponding to the execution command, which is an example of the control signal. Examples of work purposes include "grasping", "conveying", "joining", and "fastening".

For example, in the program storage unit 211, purpose information may be associated with at least one of a plurality of programs (a plurality of motion programs and a plurality of machining programs). In this case, by associating the program with the purpose information, at least one of the plurality of types of control signals is associated with the purpose information via the program. For example, control signals representing program execution history are associated with the purpose information. The extracting unit 219 identifies a program executed when acquiring, generating, or outputting the control signal based on the control signal representing the execution history of the program, and associates the purpose information associated with the identified program with the control signal. It acquires from the program storage unit 211 as the purpose information.

The extraction unit 219 identifies the virtual peripheral object on which the virtual end effector V50 acts based on the positional relationship between one or more virtual peripheral objects included in the virtual space VS and the virtual end effector V50 (model of the end effector 50). , the change in state of the virtual peripheral object may be extracted based on the control signal and the purpose information relating to the operation of the virtual end effector V50.

For example, the extraction unit 219 recognizes that the program being executed when the control signal for gripping the workpiece W by the end effector 50 (hand) is generated is associated with the purpose information "gripping". Based on the recognition result that "grasping" is associated with the program, the extraction unit 219 extracts the position/orientation of the virtual work VW (model of the work W) in the virtual space VS with the position of the virtual end effector V50 as a reference. Rewrite to the relative position/relative orientation. After rewriting, when the simulator 214 moves the virtual end effector V50 in the virtual space VS, the virtual work VW moves together with the virtual end effector V50 based on the relative position/relative orientation.

　Referring to FIGS. 4 and 5, the extraction and display of state changes are further illustrated. 4 and 5 include virtual robot V5B and virtual robot V5C operating in virtual space VS. Virtual robots V5B and V5C are models of robots 5B and 5C, each having a virtual end effector V50 (model of end effector 50). As an example, the end effector 50 of the robot 5B is a hand, and the end effector 50 of the robot 5C is a joining tool such as welding. The virtual space VS includes virtual tables VT1 and VT2, which are virtual peripheral objects, in addition to the virtual robots V5B and V5C. Virtual works VW1 and VW2, which are virtual peripheral objects, are arranged on the virtual table VT1.

As shown in FIG. 4, the simulator 214, based on the transition information and the model information of the production apparatus 2, after the virtual robot V5B moves the virtual end effector V50 to the position for gripping the virtual workpiece VW1 ( (see FIG. 4(a)), the virtual end effector V50 is operated so as to grip the virtual work VW1.

The extraction unit 219 recognizes that the control signal for operating the virtual end effector V50 to grip the virtual work VW1 is associated with the purpose information "gripping", and extracts the position/orientation of the virtual work VW1 in the virtual space VS. to a relative position/orientation based on the position of the virtual end effector V50. The simulator 214 causes the virtual robot V5B to move the virtual end effector V50 to a position where the virtual work VW1 is placed on the virtual work VW2 based on the transition information and the model information of the production apparatus 2 . The simulator 214 moves the virtual work VW1 together with the virtual end effector V50 based on the movement of the virtual end effector V50 and the relative position/orientation, and places it on the virtual work VW2 ((b) in FIG. 4). reference).

After that, the simulator 214 operates the virtual end effector V50 to release the grip of the virtual work VW1. The extracting unit 219 recognizes that the control signal for operating the virtual end effector V50 to release the grip of the virtual work VW1 is associated with the purpose information "grip release", and the position of the virtual end effector V50 is used as a reference. The relative position/relative orientation of the virtual work VW1 to be is rewritten to the position/orientation in the virtual space VS. After that, the simulator 214 causes the virtual robot V5B to withdraw the virtual end effector V50 from the virtual work VW1 based on the transition information and the model information of the production apparatus 2 . As described above, since the position of the virtual work VW1 has been rewritten to the position and orientation in the virtual space VS, the virtual work VW1 remains on the virtual work VW2 without moving together with the virtual end effector V50 (see FIG. 4). (c)).

As shown in FIG. 5, the simulator 214 moves the virtual end effector V50, which is the joining tool, between the virtual work VW1 and the virtual work VW2 using the virtual robot V5C based on the transition information and the model information of the production apparatus 2. contact between them (see FIG. 5(a)). The extraction unit 219 recognizes that the control signal for bringing the virtual end effector V50 into contact with the virtual work VW1 and the virtual work VW2 is associated with the purpose information of "joining", and recognizes that the purpose information "bonding" is associated with the virtual work VW2 in the virtual space VS. The position/orientation is rewritten to a relative position/orientation based on the position of the virtual work VW1. After that, the simulator 214 causes the virtual robot V5C to withdraw the virtual end effector V50 from the virtual works VW1 and VW2 based on the transition information and the model information of the production apparatus 2, and moves the virtual end effector V50 to a position for gripping the virtual work VW1. The virtual end effector V50 is moved by the robot V5B (see (b) of FIG. 5). Furthermore, the simulator 214 operates the virtual end effector V50 so as to grip the virtual work VW1.

The extraction unit 219 recognizes in the program storage unit 211 that the control signal for operating the virtual end effector V50 so as to grip the virtual work VW1 is associated with the purpose information "gripping the work W", The position/orientation of the virtual work VW1 in VS is rewritten to a relative position/orientation based on the position of the virtual end effector V50. After that, the simulator 214 causes the virtual robot V5B to move the virtual end effector V50 to a position where the virtual work VW3 is arranged on the virtual table VT2 based on the transition information and the model information of the production apparatus 2 . The simulator 214 moves the virtual work VW1 based on the movement of the virtual end effector V50, the relative position/relative orientation of the virtual work VW1 with respect to the virtual end effector V50, and the relative position/relative orientation of the virtual work VW2 with respect to the virtual work VW1. VW1 is moved together with the virtual end effector V50, and the virtual work VW2 is moved together with the virtual work VW1. As a result, the virtual work VW3 moves onto the virtual table VT2 (see (c) in FIG. 5).

Returning to FIG. 3, the simulation device 200 may be configured to further display a graph representing the transition of the control signal on the monitor. For example, the simulation device 200 may display the graph on the monitor so that it can be visually recognized together with the virtual space VS. The simulation device 200 may display a graph in association with a point-in-time indicator indicating a point in time in the operation of the virtual production apparatus V2 displayed on the monitor. The value of the control signal in the graph can be instantly associated with the display of the virtual space VS.

For example, the simulation device 200 further includes a graph generator 221 and a synchronizer 222 as functional blocks. The graph generation unit 221 reads the transition information of the target section from the transition information storage unit 218 and generates the above graph based on the read transition information.

The synchronization unit 222 synchronizes the time point in the operation of the virtual production apparatus V2 displayed on the monitor with the time point indicated by the time point index in the graph. For example, the synchronization unit 222 identifies the position on the graph corresponding to the point in the operation of the virtual production apparatus V2 displayed on the monitor. For example, the synchronizing unit 222 selects a point in time corresponding to the operation of the virtual production apparatus V2 displayed on the monitor (a point in time to be reproduced by the operation of the virtual production apparatus V2 displayed on the monitor) on the time axis of the transition information. Identify and identify the location on the graph that corresponds to the identified time point. The display unit 216 displays the graph generated by the graph generation unit 221 on the monitor, and causes the monitor to display the time index in association with the position specified by the synchronization unit 222 .

As long as the position specified by the synchronization unit 222 can be visually recognized, there is no particular limitation on the display form of the time point index. Examples of the time point index include a line drawn on the graph passing through the position specified by the synchronization unit 222, a point displayed at the position specified by the synchronization unit 222, and a position specified by the synchronization unit 222. A pointing arrow or the like can be used.

The display unit 216 may cause the monitor to display an event index representing the event in association with the position on the graph corresponding to the time represented by the event information, based on the event information representing the time when the specific event occurred. By making it possible to instantly grasp how the production apparatus operates and what the value of the control signal is when an event occurs, it is possible to quickly identify the cause of the event.

As long as the time points represented by the event information can be visually recognized on the graph, there are no particular restrictions on the display format of the event indicators. An example of an event index is a combination of a position index indicating a position on a graph and an icon indicating occurrence of an event.

The display unit 216 displays on the monitor a notification indicating that the event has occurred in association with the operation of the virtual production apparatus V2 corresponding to the time indicated by the event information, based on the event information indicating the time when the specific event occurred. You may let For example, the display unit 216 may display the notification at a position that can be visually recognized together with the operation of the virtual production apparatus V2 at the timing when the operation of the virtual production apparatus V2 corresponding to the event information is displayed on the monitor. It is possible to quickly recognize how the production device 2 was operating when the event occurred.

As long as it can be associated with the operation of the virtual production equipment V2 corresponding to the time represented by the event information, there is no particular limitation on the display form of the notification. Examples of the form of notification display include displaying text or a diagram notifying the occurrence of an event together with the operation of the virtual production apparatus V2, or at least partially displaying the operation of the virtual production apparatus V2 corresponding to the time represented by the event information. highlighting by changing the color, etc.

The simulation device 200 may further include a detection unit 223, an event storage unit 225, and an event notification unit 226. The detection unit 223 detects the occurrence of an event based on the control signal and generates the event information. For example, the detection unit 223 may detect the occurrence of an event based on the status signal such as the alarm described above, or may detect the occurrence of the event based on the magnitude of the deviation described above. Since the detection unit 223 detects an event based on the control signal, convenience is improved.

Based on the control signal, the detection unit 223 may detect that the robots 5B, 5C, and 5D are overloaded as an event. The detector 223 may detect overload based on the status signal, or may detect overload based on the deviation between the target angle and the feedback value.

For example, the detection unit 223 sequentially reads out a plurality of status records stored in the database 113, checks the presence or absence of an event based on the read status records, associates the detected event with the time common to the status record, and stores the event. Stored in unit 225 . As a result, the event information is accumulated in the event storage unit 225 .

The event notification unit 226 confirms, based on the event information stored in the event storage unit 225, whether there is an event occurring at the time corresponding to the operation of the virtual production apparatus V2 displayed on the monitor by the display unit 216. When the event notification unit 226 identifies an event that occurred at the time corresponding to the operation of the virtual production apparatus V2, the event indicator and notification of the identified event are displayed on the display unit 216. FIG.

The simulation device 200 may be configured to set the target section based on user input. For example, the simulation device 200 may further have an interval setting section 227 . The section setting unit 227 sets the target section based on user input. For example, the section setting unit 227 displays an input screen on which the start time and the end time can be individually input, and sets the inputted start time to the end time as the target section.

The simulation device 200 may further include an event identification unit 228. The event identification unit 228 identifies at least an event of the production system 1 based on user input. The section setting unit 227 may set the target section so as to include the point in time when the event specified by the event specifying unit 228 occurs.

For example, the event identification unit 228 displays a list of events detected by the detection unit 223 based on the event information stored in the event storage unit 225, and identifies an event selected from the list by user input as event 1 above. do.

The interval setting unit 227 sets the point in time before the occurrence of the event specified by the event specifying unit 228 by a predetermined period as the start time, the point in time at which the event occurs as the end time, and the interval from the start time to the end time as the target interval. set. The interval setting unit 227 may set the end time to a point in time past the point at which the event occurred.

The display unit 216 causes the monitor to display, based on the photographed data of the production apparatus 2 photographed in the real space, the photographing index representing the photographed data in association with the position on the graph corresponding to the time when the photographed data was photographed. may It is possible to facilitate association between the photographed data of the physical space and the control signal.

The display unit 216 may display on the monitor the frames captured in the real space at the time indicated by the operation of the virtual production apparatus V2. It is possible to easily associate the frames captured in the real space with the virtual space. A frame may be a still image or one frame of a moving image.

For example, as shown in FIG. 6, the simulation device 200 may further include a photographed data extraction unit 231, a photographed data storage unit 232, and a frame identification unit 233. The photographed data extraction unit 231 extracts the photographed data included in the state record corresponding to the target section from the database 113, and stores the photographed data storage unit 231 in association with the time when the photographed data was photographed (for example, the time included in the state record). 232. The display unit 216 causes the monitor to display the shooting index based on the time points associated with the shooting data in the shooting data storage unit 232 .

There are no particular restrictions on the display format of the shooting index as long as the time point at which the shooting data was shot can be visually recognized on the graph. An example of an imaging index is a combination of a position index indicating a position on a graph and an icon indicating existence of imaging data.

The frame identification unit 233 identifies a frame captured in the real space at the time indicated by the operation of the virtual production apparatus V2 based on the time associated with the captured data stored in the captured data storage unit 232. The display unit 216 displays the virtual space VS and the frame specified by the frame specifying unit 233 on the monitor.

The simulation device 200 may be configured to change the viewpoint based on user input. Calculation results from the simulator can be effectively used to reproduce moving images from arbitrary viewpoints. For example, as shown in FIG. 7, the simulation device 200 further has a viewpoint setting section 241 and a viewpoint storage section 242 . The viewpoint setting unit 241 sets the viewpoint for the virtual space VS based on user input. For example, the viewpoint setting unit 241 sets the position and direction of the viewpoint in the virtual space VS. The display unit 216 displays the virtual space VS viewed from the viewpoint set by the viewpoint setting unit 241 .

The viewpoint storage unit 242 stores the viewpoint change history by the viewpoint setting unit 241 . For example, the viewpoint storage unit 242 stores a change history of the viewpoint on the time axis of the transition information. For example, when the viewpoint setting unit 241 changes the viewpoint, the viewpoint storage unit 242 associates and stores the point in time (point in time on the time axis) displayed by the display unit 216 and the changed viewpoint. The display unit 216 changes the viewpoint based on the viewpoint change history stored in the viewpoint storage unit 242 when displaying the operation of the virtual production apparatus V2. Viewpoint change history can be reused to improve convenience.

The simulation device 200 may further have a confirmation unit 245 . The confirmation unit 245 checks the configuration of the production apparatus 2 and the configuration of the virtual production apparatus V2 based on transition information (for example, a plurality of state records) stored in the database 113 and model information stored in the model holding unit 213. Check if they match. If the confirmation unit 245 determines that the configuration of the production apparatus 2 and the configuration of the virtual production apparatus V2 do not match, the confirmation unit 245 may display a mismatch notification on the monitor. Inappropriate verification due to mismatches can be easily prevented.

The configuration here includes at least the type and number of machines. For example, the confirmation unit 245 identifies the configuration of the production device 2 based on the data item label (column header or key, etc.) in the transition information, and identifies the configuration of the virtual production device V2 based on the model information of the production device 2. , the configuration of the specified production apparatus 2 and the configuration of the virtual production apparatus V2 are compared.

FIG. 8 is a schematic diagram illustrating an image that the display unit 216 causes the monitor to display. The image illustrated in FIG. 8 includes a simulation area 250, a graph 260, a slide bar 261, a play button 262, a time indicator 263, an event indicator 264, and a plurality of shooting indicators 265. The simulation area 250 is a two-dimensional simulation image showing the virtual space VS seen from the above viewpoint, and includes a virtual transport device V5A, virtual robots V5B, V5C, and V5D, a virtual machine tool V5E, and a virtual work VW. .

A graph 260 represents temporal changes in target sections of a plurality of control signals. In the illustrated example, graph 260 classifies the plurality of control signals into two or more groups and includes multiple graphs for each group. Multiple graphs are arranged vertically. In each of the plurality of graphs, the horizontal axis represents time on the time axis of the transition information, and the vertical axis represents the value of the control signal.

The time index 263 is displayed in association with the position on the graph 260 corresponding to the time displayed by the simulation image on the time axis of the transition information. For example, the time indicator 263 is linearly displayed on the graph 260 across multiple graphs.

The event index 264 is displayed at the position on the graph 260 corresponding to the time when the event occurred.

The shooting index 265 includes a still image shooting index 266 and a moving image shooting index 267. The still image shooting index 266 indicates the shooting time point of the still image included in the shooting data. The moving image shooting index 267 indicates the shooting section of the moving image included in the shooting data.

The slide bar 261 is an object that graphically represents the time on the time axis of transition information. Slide bar 261 has a laterally extending track 268 and a thumb 269 that moves along track 268 . The position of thumb 269 on track 268 represents the time on the time axis of the transition information.

By changing the position of the thumb 269 by a drag operation or the like, it is possible to specify the time on the time axis of the transition information. Synchronization unit 222 identifies the display position of time indicator 263 based on the specified time. The display unit 216 changes the display of the simulation area 250 based on the simulation result at the designated time, and moves the time indicator 263 to the display position specified by the synchronization unit 222 .

The play button 262 is a button for instructing to operate the virtual production apparatus V2 in the simulation area 250 based on the transition information. When the play button 262 is pressed, the display unit 216 reads the simulation results accumulated in the result holding unit 215 in chronological order, and displays the thumb of the slide bar 261 at the position representing the time associated with the read simulation result. 269 to change the display of the simulation area 250 based on the read simulation result. The synchronization unit 222 specifies the display position of the time indicator 263 based on the time associated with the simulation result read by the display unit 216, and the display unit 216 displays the time indicator 263 at the display position specified by the synchronization unit 222. move.

In FIG. 8, the time index 263 overlaps the still image shooting index 266 and the moving image shooting index 267 . Therefore, an imaging data window 271 and an imaging data window 272 are additionally displayed. The photographed data window 271 displays a still image photographed by the camera 51 in a state where the end effector 50, which is a hand, is arranged at a position where the workpiece W can be grasped. The captured data window 272 displays a frame corresponding to the point in time indicated by the time indicator 263 in the moving image captured by the environment sensor 6 .

In the above, the case where the production device 2 executes the processing process on the work W in the real space is illustrated, but the production device 2 may be a virtual production device that executes the processing process on the work W in the virtual space. [Hardware Configuration] FIG. 9 is a block diagram illustrating the hardware configuration of the local controller 400, the host controller 300, the data collection device 100, and the simulation device 200. As shown in FIG. As shown in FIG. 9, local controller 400 includes circuitry 490 . Circuitry 490 includes processor 491 , memory 492 , storage 493 , driver circuitry 494 and communication port 495 . The storage 493 is composed of one or more non-volatile memory devices such as flash memory or hard disk. The storage 493 stores programs for causing the local controller 400 to control the machine 5 .

The memory 492 is composed of one or more volatile memory devices such as random access memory. Memory 492 temporarily stores programs loaded from storage 493 . The processor 491 is composed of one or more computing devices such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). The processor 491 executes a program loaded in the memory 492 to configure each functional block described above in the local controller 400 . A calculation result by the processor 491 is temporarily stored in the memory 492 .

The driver circuit 494 operates the machine 5 according to requests from the processor 491 . The communication port 495 communicates with the host controller 300 via the communication network NW1 for control in response to requests from the processor 491 .

The upper controller 300 has a circuit 390. Circuit 390 has processor 391 , memory 392 , storage 393 , input/output port 394 , communication port 395 and communication port 396 . The storage 393 is composed of one or more non-volatile memory devices such as flash memory or hard disk. The storage 393 stores a program for causing the host controller 300 to collect the status records and generate the execution command described above.

The memory 392 is composed of one or more volatile memory devices such as random access memory. Memory 392 temporarily stores programs loaded from storage 393 . The processor 391 is composed of one or more computing devices such as CPU or GPU. The processor 391 executes a program loaded in the memory 392 to configure each functional block described above in the host controller 300 . A calculation result by the processor 391 is temporarily stored in the memory 392 .

The input/output port 394 exchanges information with the environment sensor 6 in response to requests from the processor 391 . Communication port 395 communicates with local controller 400 via communication network NW1 in response to requests from processor 391 . The communication port 396 communicates with the data collection device 100 and the simulation device 200 via the communication network NW2 in response to requests from the processor 391 . The communication network NW2 may be a network different from the communication network NW1, or may be the same network as the communication network NW1.

The data collection device 100 has a circuit 190. Circuitry 190 includes processor 191 , memory 192 , storage 193 , communication port 194 and user interface 195 . The storage 193 is composed of one or more non-volatile memory devices such as flash memory or hard disk. The storage 193 stores a program for configuring the data collection device 100 with the functional blocks described above.

The memory 192 is composed of one or more volatile memory devices such as random access memory. Memory 192 temporarily stores programs loaded from storage 193 . The processor 191 is composed of one or more arithmetic devices such as CPU or GPU. The processor 191 executes the program loaded in the memory 192 to configure the data collection device 100 with each functional block described above. A calculation result by the processor 191 is temporarily stored in the memory 192 .

The communication port 194 communicates with the simulation device 200 and the upper controller 300 via the communication network NW2 in response to requests from the processor 191. A user interface 195 inputs and outputs information to and from an operator in response to requests from the processor 191 . For example, the user interface 195 has a display device such as a liquid crystal monitor or an organic EL (Electro-Luminescence) monitor and an input device such as a keyboard or mouse. The input device may be integrated with the display device as a touch panel.

The simulation device 200 has a circuit 290. Circuitry 290 includes processor 291 , memory 292 , storage 293 , communication port 294 and user interface 295 . The storage 293 is composed of one or more non-volatile memory devices such as flash memory or hard disk. The storage 293 stores a program for causing the simulation device 200 to configure the functional blocks described above.

The memory 292 is composed of one or more volatile memory devices such as random access memory. Memory 292 temporarily stores programs loaded from storage 293 . The processor 291 is composed of one or more computing devices such as CPU or GPU. The processor 291 executes the program loaded in the memory 292 to configure the above functional blocks in the simulation device 200 . A calculation result by the processor 291 is temporarily stored in the memory 292 .

The communication port 294 communicates with the data collection device 100 and the upper controller 300 via the communication network NW2 in response to requests from the processor 291. A user interface 295 inputs and outputs information to and from an operator in response to requests from the processor 291 . For example, the user interface 295 has a display device such as a liquid crystal monitor or an organic EL (Electro-Luminescence) monitor and an input device such as a keyboard or mouse. The input device may be integrated with the display device as a touch panel.

The above hardware configuration is just an example and can be changed as appropriate. For example, simulation device 200 may be incorporated in data collection device 100 . Also, the simulation device 200 and the data collection device 100 may be incorporated in the host controller 300 .

[Log playback procedure]
As an example of the reproduction method, a log reproduction procedure executed by the simulation apparatus 200 is illustrated. The log reproduction procedure is to operate the virtual production device V2 corresponding to the production device 2 in the virtual space VS based on the transition information representing the transition of the control signal collected from the controller 3 and the model information of the production device 2. and causing the monitor to display the virtual space VS in which the virtual production apparatus V2 operates.

As an example, the log playback procedure includes, in order, a section setting screen display procedure, an initial screen display procedure, and a display update procedure. These procedures are exemplified below.

(Display procedure of section setting screen)
As shown in FIG. 10, the simulation device 200 first executes steps S01 and S02. In step S01, the confirmation unit 245 waits for the call of the log reproduction function. In step S02, the confirmation unit 245 acquires the configuration information of the production device 2 and the configuration information of the virtual production device V2. For example, the confirmation unit 245 acquires configuration information of the production apparatus 2 based on a plurality of state records stored in the database 113, and acquires configuration information of the virtual production apparatus V2 based on model information stored in the model holding unit 213. .

Next, the simulation device 200 executes step S03. In step S03, the confirmation unit 245 confirms whether or not the configuration of the production apparatus 2 and the configuration of the virtual production apparatus V2 match based on the configuration information acquired in step S02. If it is determined in step S03 that the configuration of production device 2 and the configuration of virtual production device V2 do not match, simulation device 200 executes step S04. In step S04, the confirmation unit 245 causes the monitor to display the inconsistency notification. After that, the simulation device 200 returns the processing to step S01.

When it is determined in step S03 that the configuration of the production device 2 and the configuration of the virtual production device V2 match, the simulation device 200 executes steps S05 and S06. In step S05, the detection unit 223 sequentially reads a plurality of state records from the database 113, checks the presence or absence of an event based on the read state records, and stores the event information in the event storage unit 225 based on the confirmation result. .

In step S06, the section setting unit 227 displays a target section setting screen. For example, the section setting unit 227 displays a setting screen including input boxes for the start time and the end time and a list of the events. This completes the procedure for displaying the section setting screen.

(Initial screen display procedure)
As shown in FIG. 11, the simulation apparatus 200 first executes step S11. In step S11, the section setting unit 227 confirms whether or not a section is specified by inputting the start time and the end time on the section setting screen. When it is determined in step S11 that the section is not specified, the simulation device 200 executes step S12. In step S12, the event specifying unit 228 checks whether or not the event of the production system 1 has been selected from the event list on the section setting screen. If it is determined in step S12 that no event has been selected, the simulation device 200 returns the process to step S11.

When it is determined in step S11 that a section has been designated, the simulation device 200 executes step S13. In step S13, the section setting unit 227 sets the target section from the start time to the end time.

If it is determined in step S12 that an event has been selected, the simulation device 200 executes step S14. In step S14, the event specifying unit 228 specifies the selected event, and the interval setting unit 227 sets the target interval so as to include the point in time when the specified event occurs.

After steps S13 and S14, the simulation device 200 executes steps S15, S16 and S17. In step S<b>15 , the transition information extraction unit 217 extracts the transition information of the log reproduction target section from the database 113 and stores it in the transition information storage unit 218 . In step S16, the graph generation unit 221 reads the transition information of the target section from the transition information storage unit 218, and generates the graph 260 based on the read transition information. In step S17, the photographed data extracting unit 231 extracts the photographed data included in the state record corresponding to the target section from the database 113, and associates the photographed data with the time when the photographed data was photographed (for example, the time included in the state record). It is stored in the photographed data storage unit 232 .

Next, the simulation device 200 executes steps S18 and S19. In step S18, the simulator 214 operates the virtual production apparatus V2 in the virtual space VS based on the transition information stored in the transition information storage unit 218. FIG.

In step S19, the display unit 216 sets the start time of the target section as the display target time, reads the simulation result corresponding to the display target time from the transition information storage unit 218, and based on the read simulation result, a predetermined viewpoint is displayed. generates a two-dimensional simulation image in which the virtual space VS is viewed from. The display unit 216 causes the monitor to display the simulation area 250 including the generated simulation image, the graph 260 generated in step S16, and the imaging index 265 based on the imaging data extracted in step S17. This completes the procedure for displaying the initial screen.

FIG. 12 is a flowchart illustrating the simulation procedure in step S18. As shown in FIG. 12, the simulation apparatus 200 first executes steps S21 and S22.

In step S21, the simulator 214 extracts a plurality of types of control signals corresponding to the start time of the target section from transition information stored in the transition information storage unit 218. In step S22, the simulator 214 reproduces the virtual space VS immediately before the virtual production apparatus V2 begins to operate based on the extracted multiple types of control signals and the model information of the production apparatus 2 stored in the model holding unit 213. Generate.

Next, the simulation device 200 executes steps S23 and S24. In step S<b>23 , the simulator 214 extracts the following multiple types of control signals from the transition information stored in the transition information storage unit 218 . In step S24, the simulator 214 operates the virtual production apparatus V2 in the virtual space VS based on the following multiple types of control signals and the model information of the production apparatus 2 stored in the model holding unit 213.

Next, the simulation device 200 executes step S25. In step S25, the extraction unit 219 confirms whether or not the virtual end effector V50 is positioned at the start position of the work on the virtual peripheral object based on the control signal related to the operation of the virtual end effector V50 and the purpose information. . If it is determined in step S25 that the virtual end effector V50 is positioned at the work start position, the simulation device 200 executes step S26. In step S26, the extraction unit 219 extracts a virtual peripheral object on which the virtual end effector V50 acts based on the positional relationship between one or more virtual peripheral objects included in the virtual space VS and the virtual end effector V50 (model of the end effector 50). Identify objects.

If it is determined in step S25 that the virtual end effector V50 is not placed at the work start position, the simulation device 200 executes step S27. In step S27, the extraction unit 219 confirms whether or not the virtual end effector V50 is placed at the position where the work on the virtual peripheral object is completed, based on the control signal related to the operation of the virtual end effector V50 and the purpose information. do. If it is determined in step S27 that the virtual end effector V50 is positioned at the work completion position, the simulation device 200 executes step S28. In step S28, the extraction unit 219 extracts the state change of the virtual peripheral object based on the purpose information, and changes the state of the virtual peripheral object in the virtual space VS based on the extracted state change.

After steps S26 and S28, the simulation device 200 executes step S29. In step S29, the simulator 214 confirms whether or not the operation of the virtual production apparatus V2 based on the transition information has been completed. If it is determined in step S29 that the operation of the virtual production apparatus V2 based on the transition information has not been completed, the simulation apparatus 200 returns the process to step S23. If it is determined in step S29 that the operation of the virtual production apparatus V2 based on the transition information has been completed, the simulation apparatus 200 completes the simulation procedure.

(Display update procedure)
As shown in FIG. 13, the simulation device 200 executes step S31. In step S31, the display unit 216 confirms whether or not the slide bar 261 has been operated (whether or not the thumb 269 has been moved). When it is determined in step S31 that the slide bar 261 has been operated, the simulation device 200 executes steps S32 and S33. In step S<b>32 , the display unit 216 identifies the display target time on the time axis of the transition information based on the position of the thumb 269 . In step S33, the display unit 216 executes screen transition processing to the display target time. The contents of the screen transition processing will be described later.

If it is determined in step S31 that the slide bar 261 has not been operated, the simulation device 200 executes step S34. In step S<b>34 , the viewpoint setting unit 241 confirms whether or not an operation to change the viewpoint has been performed, such as by dragging the simulation area 250 . If it is determined in step S34 that an operation for changing the viewpoint has been performed, the simulation device 200 executes steps S35 and S36. In step S35, the viewpoint setting unit 241 changes the viewpoint with respect to the virtual space VS, and causes the viewpoint storage unit 242 to store the viewpoint change history. In step S36, the display unit 216 causes the simulation area 250 to display the virtual space VS viewed from the changed viewpoint.

If it is determined in step S34 that an operation to change the viewpoint has not been performed, the simulation device 200 executes step S37. In step S37, the display unit 216 confirms whether or not a reproduction operation has been performed using the reproduction button 262. FIG. If it is determined in step S37 that the reproduction operation has been performed, the simulation apparatus 200 executes step S38. In step S38, the display unit 216 executes reproduction processing. Details of the reproduction process will be described later.

After steps S33, S36, and S38, the simulation device 200 executes step S39. If it is determined in step S37 that the reproduction operation has not been performed, the simulation apparatus 200 executes step S39 without executing steps S33, S36, and S38. In step S39, the display unit 216 confirms whether or not an instruction to end the log reproduction function has been given. If it is determined in step S39 that the termination of the log reproduction function has not been instructed, the simulation device 200 returns the process to step S31. If it is determined in step S39 that the termination of the log reproduction function has been instructed, the simulation apparatus 200 completes the display update procedure.

FIG. 14 is a flowchart illustrating the screen transition procedure in step S33. As shown in FIG. 14, the simulation device 200 executes steps S41, S42, S43, and S44. In step S<b>41 , the display unit 216 reads the simulation result corresponding to the display target time from the result holding unit 215 . In step S<b>42 , based on the change history stored in the viewpoint storage unit 242 , the viewpoint setting unit 241 sets the viewpoint corresponding to the display target time. In step S43, the display unit 216 generates a simulation image of the virtual space VS viewed from the set viewpoint based on the read simulation result. In step S44, the synchronization unit 222 identifies the display position of the time indicator 263 based on the display target time.

Next, the simulation device 200 executes step S45. In step S45, the event notification unit 226 checks whether an event has occurred at the display target time. If it is determined in step S45 that an event has occurred, the simulation device 200 executes step S46. In step S46, the display unit 216 adds an event occurrence notification to the generated simulation image.

Next, the simulation device 200 executes step S47. If it is determined in step S45 that no event has occurred, the simulation apparatus 200 executes step S47 without executing step S46. In step S47, the frame identification unit 233 confirms whether or not there is photographed data photographed at the display target time. If it is determined in step S47 that there is shooting data, the simulation device 200 executes step S48. In step S48, the frame identification unit 233 identifies the frame captured in the physical space at the display target time based on the time associated with the captured data. The display unit 216 generates captured data windows 271 and 272 for displaying the frames specified by the frame specifying unit 233 .

Next, the simulation device 200 executes step S49. If it is determined in step S47 that there is no shooting data, the simulation device 200 executes step S49 without executing step S48. In step S49, the display unit 216 updates the display of the simulation area 250 based on the generated simulation image, moves the time index 263 to the specified display position, and displays the generated captured data windows 271 and 272. This completes the screen transition procedure.

FIG. 15 is a flowchart illustrating the reproduction procedure in step S38. As shown in FIG. 15, the simulation device 200 executes steps S51, S52, and S53. In step S51, the display unit 216 adds one cycle (for example, the communication cycle described above) to update the display target time. In step S52, the display unit 216 executes screen transition processing to the updated display target time. The procedure of step S52 is the same as the screen transition procedure described above. In step S53, the display unit 216 confirms whether the display target time is the end time of the target section.

If it is determined in step S53 that the display target time is not the end time, the simulation device 200 executes step S54. In step S54, the viewpoint setting unit 241 confirms whether or not an operation for changing the viewpoint has been performed. If it is determined in step S54 that an operation for changing the viewpoint has been performed, the simulation device 200 executes step S55. In step S55, the viewpoint setting unit 241 changes the viewpoint with respect to the virtual space VS, and causes the viewpoint storage unit 242 to store the viewpoint change history. The result of changing the viewpoint is reflected in the display of the simulation area 250 in the next screen transition process.

Next, the simulation device 200 executes step S56. In step S56, the display unit 216 confirms whether or not a stop operation has been performed using the playback button 262. FIG. If it is determined in step S56 that the stop operation has not been performed, the simulation device 200 executes step S57. In step S57, the display unit 216 confirms whether or not one cycle has passed since the execution of step S51. If it is determined in step S57 that one period has not elapsed, the simulation device 200 returns the process to step S54. If it is determined in step S57 that one cycle has passed, the simulation device 200 returns the process to step S51.

If it is determined in step S53 that the display target time is the end time of the target section, or if it is determined that a stop operation has been performed in step S56, the simulation device 200 completes the reproduction procedure.

〔summary〕
The embodiments shown above include the following configurations.
(1) Create a virtual production device V2 corresponding to the production device 2 based on the transition information representing the transition of control signals collected from the controller 3 that controls the production device 2 including the robot 5B and the model information of the production device 2. in a virtual space VS, and a display unit 216 for displaying on a monitor the virtual space VS in which the virtual production device V2 operates.
Even if the real space is not fully photographed, it is possible to reproduce the phenomenon that occurred in the real space. Reproduction results can also be displayed from any viewpoint. Therefore, it is effective for ex-post verification of system operation.

(2) It further comprises an extraction unit 219 for extracting state changes in the virtual space VS based on the operation of the virtual production apparatus V2 in the virtual space VS, and the display unit 216 displays the extracted state changes in the virtual space VS on a monitor. The production system 1 according to (1), which is displayed.
The transition of the control signal stored in the database can also be effectively used to reproduce the state change of surrounding objects. By including state changes of virtual surrounding objects in the display, phenomena occurring in the real space can be verified more easily.

(3) The production system 1 according to (2), wherein the extraction unit 219 extracts at least one state change of the plurality of models based on the positional relationship between the plurality of models included in the virtual space VS.
Based on the positional relationship, state changes can be easily extracted.

(4) (2) or (3), wherein the extraction unit 219 extracts state changes of at least one model included in the virtual space VS based on the control signal and purpose information associated with the control signal. production system 1.
It is possible to more accurately extract state changes of virtual peripheral objects.

(5) The virtual production apparatus V2 includes a virtual robot V5B having a virtual end effector V50 corresponding to the end effector 50 of the robot 5B. A virtual peripheral object on which the virtual end effector V50 acts is specified based on the positional relationship with the end effector V50, and a state change of the virtual peripheral object is detected based on the control signal and purpose information relating to the operation of the virtual end effector V50. The production system 1 according to (4), which extracts.
Calculations for changing the state of the virtual peripheral object based on the operation of the virtual production device V2 can be performed more easily.

(6) A graph generation unit 221 that generates a graph representing the transition of the control signal, a synchronization unit 222 that synchronizes the time point in the operation of the virtual production apparatus V2 displayed on the monitor with the time point indicated by the time point index in the graph, The production system 1 according to any one of (1) to (5), wherein the display unit 216 causes the monitor to display the virtual space VS and the graph including the time point index.
The value of the control signal in the graph can be instantly associated with the virtual space VS being displayed.

(7) The display unit 216 causes the monitor to display an event indicator representing the event based on the event information indicating the time when the specific event occurred, in association with the position on the graph corresponding to the time indicated by the event information. (6) The production system 1 described.
By making it possible to instantly grasp how the production apparatus 2 operates and what the value of the control signal is when an event occurs, it is possible to quickly identify the cause of the event.

(8) The display unit 216, based on the event information indicating the time when the specific event occurred, displays a notification indicating the occurrence of the event in association with the operation of the virtual production apparatus V2 corresponding to the time indicated by the event information. , is displayed on the monitor, the production system 1 according to (6) or (7).
It is possible to quickly recognize how the production device 2 was operating when the event occurred.

(9) It further has a detection unit 223 that detects the occurrence of a specific event based on the control signal and generates event information about the event, and the display unit 216 detects the occurrence of the event based on the event information. The production system 1 according to any one of (6) to (8), wherein a notification representing is displayed on the monitor.
Since the detection unit 223 detects an event based on the control signal, convenience is improved.

(10) The production system 1 according to (9), wherein the robot 5B has the articulated arm 10, and the detection unit 223 detects that the robot 5B is overloaded as an event based on the control signal.
It is possible to quickly recognize the cause of the overload.

(11) The production system 1 according to any one of (10), wherein the production device 2 further includes a peripheral device whose movable range overlaps with that of the robot 5B.
Since a collision between the peripheral device and the robot 5B may occur, it is more beneficial to quickly recognize the cause of the overload.

(12) Based on the photographed data of the production apparatus 2 photographed in the real space, the display unit 216 monitors the photographing index representing the photographed data in association with the position on the graph corresponding to the time when the photographed data was photographed. , the production system 1 according to any one of (6) to (11).
It is possible to easily associate the photographed data of the physical space with the control signal.

(13) The display unit 216 further includes a frame identification unit 233 that identifies a frame captured in the physical space at the time represented by the operation of the virtual production device V2 based on the captured data of the production device 2 captured in the physical space. displays the virtual space VS and the specified frame on the monitor, the production system 1 according to any one of (1) to (12).
It is possible to easily associate the frames captured in the physical space with the virtual space VS.

(14) Further comprising a viewpoint setting unit 241 for setting a viewpoint for the virtual space VS based on user input, the display unit 216 displays the virtual space VS seen from the set viewpoint, (1) to (13) ).
The calculation result by the simulator 214 can be effectively used for display from any viewpoint.

(15) The display unit 216 further includes a viewpoint storage unit 242 that stores a viewpoint change history, and the display unit 216 changes the viewpoint based on the viewpoint change history when displaying the operation of the virtual production apparatus V2. ) production system 1 described.
Viewpoint change history can be reused to improve convenience.

(16) The simulator 214 further includes an event identification unit 228 that identifies at least one event based on user input, and an interval setting unit 227 that sets an interval including the point in time when the event occurs. The production system 1 according to any one of (1) to (15), wherein the virtual production device V2 is operated based on transition information in the section set by the unit 227.
If an event that can be verified is specified, the period for which the simulation should be performed is set, which improves convenience.

(17) Further comprising a confirmation unit 245 for confirming whether or not the configuration of the production device 2 and the configuration of the virtual production device V2 match based on the transition information and the model, (1) to (16) Production system 1 according to any one of
Inappropriate verification due to mismatches can be easily prevented.

(18) The production equipment 2 includes a plurality of machines 5 including robots 5B, the controller 3 includes a plurality of local controllers 400 that respectively control the plurality of machines 5, and the simulator 214 collects data from the plurality of local controllers 400. (1) to (17), wherein the virtual production apparatus V2 is operated in the virtual space VS based on transition information representing transitions of the plurality of types of control signals obtained and the models of the plurality of machines 5. production system 1;
Based on the control signals of a plurality of machines 5, the operation of the production device 2 can be reproduced in detail.

(19) A virtual production device V2 corresponding to the production device 2 is created based on the transition information representing the transition of control signals collected from the controller 3 that controls the production device 2 including the robot 5B and the model of the production device 2. A reproduction method including operating in a virtual space VS and displaying on a monitor the virtual space VS in which the virtual production device V2 operates.
Although the embodiments have been described above, the present invention is not necessarily limited to the above-described embodiments, and various modifications can be made without departing from the scope of the invention.

DESCRIPTION OF SYMBOLS 1... Production system, 2... Production apparatus, W... Work, 5... Machine, 5B... Robot, 10... Articulated arm, 50... End effector, 3... Controller, 400... Local controller, 214... Simulator, 216... Display part , 219... Extraction unit, VW... Virtual work, VS... Virtual space, V5B... Virtual robot, V50... Virtual end effector, 221... Graph generation unit, 222... Synchronization unit, 223... Detection unit, 227... Section setting unit, 228 ... event identification unit, 233 ... frame identification unit, 241 ... viewpoint setting unit, 242 ... viewpoint storage unit, 245 ... confirmation unit.

Claims (19)

1. A simulator that operates a virtual production device corresponding to the production device in a virtual space based on transition information representing the transition of control signals collected from a controller that controls the production device including the robot and model information of the production device. and,
   a display unit for displaying on a monitor the virtual space in which the virtual production apparatus operates;
   production system with
2. further comprising an extraction unit for extracting state changes in the virtual space based on the operation of the virtual production device in the virtual space;
   The display unit causes the monitor to display the extracted state change of the virtual space.
   The production system according to claim 1.
3. The production system according to claim 2, wherein the extraction unit extracts the state change of at least one of the plurality of models based on the positional relationship between the plurality of models included in the virtual space.
4. The extraction unit extracts state changes of at least one model included in the virtual space based on the control signal and purpose information associated with the control signal.
   3. The production system according to claim 2.
5. The virtual production device includes a virtual robot having a virtual end effector corresponding to the end effector of the robot,
   The extractor is
   identifying a virtual peripheral object on which the virtual end effector acts based on the positional relationship between one or more virtual peripheral objects included in the virtual space and the virtual end effector;
   extracting a state change of the virtual peripheral object based on the control signal and the objective information related to the operation of the virtual end effector;
   The production system according to claim 4.
6. a graph generator that generates a graph representing the transition of the control signal;
   a synchronizing unit for synchronizing the time point of the operation of the virtual production apparatus displayed on the monitor with the time point indicated by the time point index in the graph;
   further comprising
   The display unit causes the monitor to display the virtual space and the graph including the time point index.
   The production system according to any one of claims 1-5.
7. The display unit causes the monitor to display an event indicator representing the event in association with a position on the graph corresponding to the time represented by the event information based on event information representing a time at which a specific event occurs. ,
   The production system according to claim 6.
8. The display unit, based on event information indicating a time point at which a specific event occurs, in association with the operation of the virtual production apparatus corresponding to the time point indicated by the event information, displays a notification indicating the occurrence of the event, displaying on the monitor;
   The production system according to claim 6.
9. further comprising a detection unit that detects occurrence of a specific event based on the control signal and generates event information about the event;
   The display unit causes the monitor to display a notification indicating that the event has occurred, based on the event information.
   The production system according to claim 6.
10. The robot has an articulated arm,
    The detection unit detects that the robot is overloaded as the event based on the control signal.
    The production system according to claim 9.
11. The production device further includes a peripheral device whose movable range overlaps with that of the robot,
    The production system according to claim 10.
12. Based on the photographed data of the production apparatus photographed in real space, the display unit associates a photographing index representing the photographed data with a position on the graph corresponding to the time when the photographed data was photographed. display on the monitor
    The production system according to claim 6.
13. further comprising a frame identification unit that identifies a frame captured in the physical space at a point in time represented by the operation of the virtual production device based on the captured data of the production device captured in the physical space;
    The display unit causes the monitor to display the virtual space and the specified frame.
    The production system according to any one of claims 1-5.
14. further comprising a viewpoint setting unit that sets a viewpoint for the virtual space based on user input;
    the display unit displays the virtual space viewed from the set viewpoint;
    The production system according to any one of claims 1-5.
15. further comprising a viewpoint storage unit that stores a change history of the viewpoint;
    The display unit changes the viewpoint based on the change history of the viewpoint when displaying the operation of the virtual production apparatus.
    15. The production system according to claim 14.
16. an event identifier that identifies at least one event based on user input;
    an interval setting unit that sets an interval including the point in time when the event occurs;
    further having
    6. The production system according to any one of claims 1 to 5, wherein said simulator operates said virtual production apparatus based on said transition information in the section set by said section setting unit.
17. further comprising a confirmation unit that confirms whether the configuration of the production apparatus and the configuration of the virtual production apparatus match based on the transition information and the model;
    The production system according to any one of claims 1-5.
18. The production equipment includes a plurality of machines including the robot,
    the controller includes a plurality of local controllers respectively controlling the plurality of machines;
    The simulator causes the virtual production apparatus to operate in the virtual space based on the transition information representing transitions of the plurality of types of control signals collected from the plurality of local controllers and the models of the plurality of machines.
    The production system according to any one of claims 1-5.
19. Operating a virtual production device corresponding to the production device in a virtual space based on transition information representing transition of control signals collected from a controller that controls the production device including the robot and a model of the production device. ,
    displaying on a monitor the virtual space in which the virtual production device operates;
    How to reproduce, including

Images