KR20220089998A - Collaboration system of smart factory in argumented reality - Google Patents

Abstract

The present invention relates to a smart factory collaboration system using augmented reality, wherein any one or more selected robots among robots of the smart factory are displayed on an augmented reality (AR) screen, and a worker clicks a menu button displayed in the augmented reality screen When the operator terminal to check the menu information on the augmented reality screen, an interface unit configured to allow the operator to remotely control the robot, and to recognize the operation taken by the operator when the operator controls the interface unit A recognition unit communicating with the interface unit, a control unit communicating with the recognition unit, an operation unit of a robot controlled by communicating with the control unit, and a management server for managing collaboration of a smart factory, wherein the augmented reality screen includes a plurality of workers Any one or more of a list, a work process solution, and work demonstration information of the robot is displayed, and the recognition unit recognizes an action taken by the operator while the operator manipulates the interface unit based on a work model of a multi-step operation, , The recognition unit recognizes the start and completion of manual and automatic operations, and the recognition unit instructs the control unit to perform an automatic operation by the operation unit of the robot based on the work model of the multi-step operation when the manual operation is completed characterized.

Description

A smart factory collaboration system using augmented reality. {COLLABORATION SYSTEM OF SMART FACTORY IN ARGUMENTED REALITY}

The present invention relates to a smart factory collaboration system using augmented reality, and to a system in which a plurality of workers can remotely collaborate in a smart factory through augmented reality.

In the smart factory production site, efficient operation and management is possible when using a human-machine collaboration package. To this end, big data is generated through 4M1E (Man, Machine, Material, Method, Environment) analysis, and through this, automation and optimization through FOM (Factory Operation Management) data-based informatization and CPS (Cyber Physical System)-based process analysis It is necessary to achieve energy efficiency through FEM (Factory Energy Management)-based process analysis. Through such comprehensive management, it is possible to manage the convergence of each module included in the smart factory, thereby enabling the operation of an advanced MI-NPS (Meta Intelligent-New Production System) type smart factory.

For example, in Republic of Korea Patent Publication No. 10-1550740, productivity is improved through individual manpower management and production performance management at the production site, and production performance is managed based on the achievement rate and share to manage 4M (Man, Machine, Material, Method). ), a management system that can easily perform management is disclosed, and a digital factory is operated based on 4M data in this system.

However, in order to apply to the MI-NPS type smart factory advanced by human-machine collaboration with human-machine collaboration process and manufacturing process simulation analysis technology in the smart factory, these data need to be applied to the process in real time.

Recently, in a collaborative process involving a large number of workers, a technology utilizing augmented reality (AR) has been developed for the smooth operation of such a smart factory.

As an example of this, Korean Patent No. 10-2086451 has full manual information on a number of equipment and a smart factory remote collaboration CMS platform, and it is understood that the communication terminal for field workers automatically recognizes the unique identification mark or the real thing. Accordingly, the smart factory remote collaboration CMS platform is provided to the communication terminal for field workers in real time so that the menu button appears on the AR screen, and the unique identification mark or equipment identification information recognized from the real thing and the field worker For example, a remote collaboration CMS management server that supports the implementation of field work on equipment in charge of the communication terminal for field workers by determining whether to approve the use access through 1:1 matching between the unique identification information of the communication terminal.

However, in order to improve the overall work efficiency of the smart factory, more efficient use of AR is required.

The present invention has been devised in consideration of the problems of the prior art as described above, and smart factory collaboration using augmented reality (AR) enables collaboration between a plurality of workers during a human-machine collaboration process in a smart factory using augmented reality (AR). Its purpose is to provide a system.

The smart factory collaboration system using augmented reality of the present invention for achieving the above object displays any one or more robots selected among robots of the smart factory on an augmented reality (AR) screen, and a menu displayed in the augmented reality screen When the operator clicks a button, an operator terminal for checking the menu information on the augmented reality screen, an interface unit configured to allow the operator to remotely control the robot, and an interface that the operator takes when the operator controls the interface unit A recognition unit communicating with the interface unit to recognize a motion, a control unit communicating with the recognition unit, an operation unit of a robot controlled by communication with the control unit, and a management server for managing collaboration of a smart factory, the augmented reality Any one or more of a plurality of operator lists, work process solutions, and work demonstration information of the robot are displayed on the screen, and the recognition unit allows the operator to control the interface unit while the operator controls the interface unit based on the multi-step work model Recognizes the action taken, the recognition unit recognizes the start and completion of manual and automatic operations, and when the manual operation is completed, the operation unit of the robot performs an automatic operation based on the work model of the multi-step operation. It is characterized by instructing the control unit.

At this time, the management server stores the robot's manual information, operator list, work process solution, and work demonstration information of the robot, and when the operator terminal recognizes the robot's unique identification mark or the real thing, the menu is displayed on the augmented reality screen. A button may appear, match the robot's unique identification mark or real thing with one or more workers selected from the worker list, and determine whether to approve the use access and display it on the worker terminal.

In addition, the recognition unit stores a work model of a multi-step operation through CPS (Cyber Physical System)-based process analysis, and the CPS-based process analysis receives data from a sensor provided in the operation unit of the robot, and the recognition unit It is possible to create analysis data by analyzing the data received from the sensor, and to update the working model of the multi-step task by simulating the analysis data.

The smart factory collaboration system according to the present invention exhibits the effect of enabling collaboration between a plurality of workers during a human-machine collaboration process in a smart factory using augmented reality (AR).

1 is a conceptual diagram schematically showing a smart factory collaboration system of the present invention.

Hereinafter, the present invention will be described in detail.

Smart factory is the application of digital technology to the production system of the manufacturing process, and includes the possibility of utilizing various advanced technologies in information utilization, automation, monitoring, sensing, modeling, and networking fields. In the process, connectivity that enables information exchange by connecting all objects and participants to each other, and collaboration between interconnected components sharing information and resources, and performing independent decision-making and collaboration based on this are important factors.

In particular, providing visual information so that the operator can intuitively grasp and judge the overall state and enabling control of each device through this is a factor that determines the efficiency of the smart factory.

In the smart factory, the process model technology field can be divided into element technologies such as application application technology, platform technology, device/network technology, and manufacturing security technology.

The application technology is the top-level software system of the smart factory ICT solution, and it performs various manufacturing executions on platforms such as MES, ERP, PLM, and SCM. It is an intermediate software system that plays a role in delivering, analyzing the data collected by the device, and providing optimization information through modeling and virtual physics simulation. In addition, the device/network technology is the lowest hardware system of the smart factory ICT solution. It detects the location, environment and energy through smart sensors and recognizes the positions of workers and workpieces through robots. It consists of a system that can transmit data to the platform. The manufacturing security technology relates to information protection and industrial confidentiality protection technology and countermeasures that can safely protect various data, systems, manufacturing facilities, etc. for all fields from sensors to applications.

The present invention relates to a smart factory collaboration system that applies augmented reality (AR) to improve factory efficiency by improving the platform and device/network of the smart factory, and is schematically configured as shown in FIG.

That is, the smart factory collaboration system of the present invention is configured so that a plurality of workers can train, share information, simulate, collaborate, etc. through AR,

It is composed by combining a system related to augmented reality and a system related to a collaborative robot.

First, the augmented reality-related system includes a management server that allows the operator terminal owned by each worker and the cooperative robot-related system to be organically combined and operated. In addition, the cooperative robot-related system is configured to include an interface unit, a recognition unit, a control unit, and an operation unit of the robot. Specifically, the smart factory collaboration system displays any one or more selected robots among robots of the smart factory on the augmented reality (AR) screen, and when the operator clicks the menu button displayed in the augmented reality screen, the menu An operator terminal for checking information on the augmented reality screen, an interface unit configured to allow the operator to remotely control the robot, the interface unit to recognize an action taken by the operator when the operator manipulates the interface unit, and A recognition unit communicating with the recognition unit, a control unit communicating with the recognition unit, an operation unit of a robot controlled by communicating with the control unit, and a management server for managing collaboration of a smart factory, the augmented reality screen includes a plurality of worker lists, tasks Any one or more of a process solution and operation demonstration information of the robot are displayed, and the recognition unit recognizes an operation taken by the operator while the operator manipulates the interface unit based on the operation model of the multi-step operation, and the The recognition unit recognizes the start and completion of manual and automatic operations, and the recognition unit instructs the control unit to perform the automatic operation by the operation unit of the robot based on the work model of the multi-step operation when the manual operation is completed. do.

In the worker terminal, information related to the work robot of the smart factory implemented in augmented reality, as well as other worker information for collaboration, work process solutions for education and simulation, and work demonstration information of the robot are displayed one by one or at the same time. configured to be able to In addition, the worker is configured to connect to the worker terminal and drive the work robot of the smart factory through augmented reality to enable cooperative work with the robot.

The worker terminal can use a personal computer, tablet, smart phone, etc. to enable collaboration through augmented reality, and is configured so that the worker wears AR glasses and can work in conjunction.

In addition, by providing an augmented reality manual, it is possible for workers to easily manipulate and implement augmented reality. To this end, in a smart factory, an augmented reality manual related to a specific process using a robot, a video manual of the process, etc. can be prepared and displayed on the augmented reality screen.

In addition, for simulation using augmented reality, overall information related to the smart factory, average data of the process for each unit, etc. may be displayed on the augmented reality screen so that the operator can intuitively check it.

In addition, the management server is to manage the collaboration of the smart factory as a whole, is connected remotely with a worker terminal for augmented reality, and is configured to manage and build an augmented reality solution. This solution is built on a web-based basis so that the operator's terminal directly accesses the Internet and displays the information on the screen so that the operator can easily acquire information.

That is, the management server may store the manual information of the robot for work, the operator list, the work process solution, and the work demonstration information of the robot, and may retrieve the information stored in the web base and transmit it to the operator terminal.

In addition, in each process, it is necessary to match a specific worker with a specific work robot. In this case, a worker with a worker terminal can access the robot for a specific job at the site, and can also remotely designate a robot for a specific job. By making the menu button appear on the real screen, the operator can operate it.

In addition, each work robot may be given a unique identification mark of the robot, and the worker information corresponding to the unique identification mark can be matched with the worker information stored in the management server when each worker logs in to the worker terminal. have. Through this, the unique identification mark or real object and the operator selected from the operator list are matched, and a plurality of workers may be matched to one working robot, or only one worker may be matched to one robot. To this end, the management server may determine whether to approve the use access, and display it on the worker terminal so that a specific worker can access the robot for a specific job and operate it.

In addition, in the smart factory collaboration system, the system related to the cooperative robot includes an interface unit and a recognition unit communicating with the interface unit to recognize the operation taken by the operator when the operator manipulates the interface unit, as shown in FIG. 1, the It is configured to perform a multi-step operation by including a control unit communicating with the recognition unit, and an operation unit of the robot controlled by communication with the control unit.

In addition, a series of operations and manipulation processes related to the cooperative robot may be displayed in augmented reality on the screen of the worker terminal, so that the operator can smoothly perform cooperative work with the robot through the augmented reality.

In addition, the recognition unit recognizes the operation taken by the operator while the operator manipulates the interface unit based on the work model of the multi-step task, the recognition unit recognizes the start and completion of manual and automatic operations, the recognition unit When the manual operation is completed, the operation unit of the robot instructs the control unit to perform an automatic operation based on the work model of the multi-step operation.

The interface unit may include a control device capable of controlling the robot, and an operator may control each part of the robot to control the robot to perform a specific operation. In addition, the interface unit may be configured to include a display unit so that the operator can observe the operation state of the robot and apply a change through this.

In addition, by configuring a system using augmented reality (AR) or virtual reality (VR) in the display unit, the operator may improve proficiency through the interface unit and increase work efficiency through collaboration with the robot. In this case, the display unit may select and set various images, and may overlap or adjust images.

In addition, the recognition unit may use a computer, a smart device, or other data processing and storage device. The recognition unit may include a program stored in a computer that performs recognition and other procedures, and may be operated through an application installed in a smartphone or tablet PC. In addition, a device located at a remote location connected through a network may be used, and in this case, the recognition unit may be driven by utilizing an Internet of Things (IoT) system. That is, since the interface unit and the recognition unit are connected through communication, there is no need to be physically coupled.

In addition, the recognition unit may include a device using robot kinematics to recognize specific movements of the operator. In this case, it is possible to recognize the operation taken by the operator while the operator manipulates the interface unit based on the multi-step operation model. The operation of this recognition unit enables cooperative work of the entire robot system through mutual information exchange in the process of performing multi-function tasks.

In addition, the robot system including the interface unit and the sensor may be configured to perform image processing by observing or recording the movement of the robot and the working environment in real time based on the motion of the operator and the signal received from the sensor. Even in this case, it is possible to improve the information acquisition and response efficiency of the operator by using the AR or VR system.

The recognition unit recognizes the start and completion of manual and automatic operations, and the recognition unit is configured to instruct the control unit to perform an automatic operation by the operation unit of the robot based on the work model of the multi-step operation when the manual operation is completed. .

That is, the recognition unit determines an operation including a manual operation, an automatic operation, and, if necessary, a semi-automatic operation by the robot system including a sensor under the control of the controller.

The manual operation is performed similarly or identically to the operation performed in a conventional remote control robot system. In addition, the automatic operation is performed entirely under the control of the robot, and there is no operator's involvement in the operation.

That is, the operator can move a part of the system including the sensor to a position where one step and subsequent steps are to be performed in each step of the multi-step operation, and in this case, the robot system is driven by manual operation.

In this case, the recognition unit may recognize that the step is a manual operation state performed by an operator, and may instruct the automatic operation to be performed at the time when the manual operation is terminated based on the work model. For example, the recognition unit may transmit, through communication, an instruction for moving the robot arm back to a position where it can pick up and return a part in a state where the robot arm is moved by a manual operation.

In addition, the recognition unit may recognize that the operator moves a part of the robot system to a position for performing a specific step during a multi-step operation. In this case, the recognition unit may instruct the control unit to perform another step to assist the operator based on the work model. For example, there is a case in which the operator has to operate one robot arm in order to instruct the operation of the robot arm. In this case, it is possible to maintain a state in which the object is fixed for the operator by automatic operation.

The work model is a work model for performing the multi-step task of the present invention, which is 4M1E (Man, Machine, Material, Method, Environment) analysis, ABL (Actual task Based Learning) based analysis, FOM (Factory Operation Management) data based analysis , CPS (Cyber Physical System)-based process analysis, and FEM (Factory Energy Management)-based process analysis results are derived and constructed.

The 4M1E is a resource related to the production activity of the smart factory and includes the environment in addition to the 4M resources, such as a worker (Man), a machine facility (Machine), a material (Material), and a product production method (Method). This 4M1E can be used as a database, and it can be used to establish a production plan and predict production performance through pre-simulation according to work scenarios, resource management such as personnel, accounting, material and order management, and digital factory work. It can include both on-site management, which grasps the situation in real time and collects and manages work performance by resource, and the process of manufacturing a production site database using various data collected from the field. Therefore, the data collected through the 4M1E analysis enables modeling of various states predicted in the process using a collaborative robot, and through this, it is possible to establish a work model.

In addition, it is possible to revise and manage the work model by analyzing it based on FOM (Factory Operation Management) data. To this end, the cooperative robot system may separately include a FOM analysis unit. The FOM analysis unit generates a smart factory operation management analysis result including key performance indicator (KPI) or LOB analysis based on the big data module, which may be performed based on the work data of the cooperative robot system. In addition, the big data analysis data and the smart factory operation management analysis result may be provided to the customer company, and the work model may be modified by reflecting the feedback on the provided result.

In addition, CPS (Cyber Physical System)-based process analysis is a system for performing smart factory processes by interconnecting physical and virtual systems. In an embodiment, the CPS modifies sensor data such as temperature, humidity, pressure, and voltage from an external sensor, and receives and stores equipment data from various external equipment such as a robot, an actuator, a lathe, a grinder, and a cutting machine. Through this, it is possible to manage management commands including controls, settings, inquiries, etc. entered by the administrator and a work model based on them. For example, the priority of the management command may be determined, and the work model may be set to perform a role corresponding to the management command according to the priority.

To this end, the cooperative robot system may separately include a CPS analysis unit. The CPS analysis unit registers, manages, and modifies the process and attribute information of the smart factory, stores information for simulation based on data, and verifies the validity of the obtained data to change the priority of previously entered management commands or You can change the working model. In addition, for verification of the validity, deep learning through machine learning or an internal/external artificial intelligence module may be used based on a reference algorithm. In addition, when the CPS analysis unit performs the simulation, a simulation including a change in the production quantity of a product, a change in a product model, a bypass process in case of a specific process failure, etc. may be performed. Data processed by the CPS analyzer may be virtual data.

For the CPS analysis, the process of producing an air intake duct_ capable of reducing automobile exhaust is as follows as an example. First, 3D modeling is produced using Inventor for parts necessary for CPS analysis. Change the 3D modeled parts into a file in a format applicable to the CPS program, and use it to establish the CPS manufacturing process Next, arrange equipment and logistics according to the above process and create a layout. Simulation through these operations In accordance with the above simulation, the operation of the robot is performed, and when the robot is operated based on this, the operation performance of the vibration welding robot and the operator is reduced by 58.8% on average compared to before the application of CPS-based process analysis. It was found to be significantly improved by 89.8% In addition, in the result of analyzing the output per hour (UPH), it was confirmed that the productivity improved by 43.3% compared to before the application of the CPS-based process analysis.

A result of performing the CPS-based process analysis is transmitted to the recognition unit, and the recognition unit may modify the work model of the multi-step job through the CPS-based process analysis and store it. In addition, the CPS-based process analysis is performed by receiving data from a sensor provided in the operating unit of the robot in the robot system. Thus, it is possible to perform process analysis by collecting them. In addition, in the process analysis, the recognition unit analyzes all data received from the sensor, equipment data, and data obtained from an external sensor, thereby creating analysis data. The analysis data may be simulated by the recognition unit, and if the size of the data is too large, the analysis data may be transmitted to an external computer to process and simulate the analysis data in the external computer. Such a simulation can be performed using deep learning through machine learning or an internal or external artificial intelligence module, and the work model of the multi-step task can be automatically updated through the use of such machine learning or artificial intelligence module. In addition, the updated working model may be additionally stored in the recognition unit.

In addition, it is possible to modify or manage the work model through FEM (Factory Energy Management)-based process analysis. The FEM is a factory energy management system that can manage the efficient use of energy, and makes it possible to efficiently manage factors affecting the production efficiency of the smart factory. To this end, the cooperative robot system may separately include an FEM analysis unit. The FEM analysis unit generates an energy management analysis result of a smart factory based on a big data module like the FOM, which may be generated based on the work data of the cooperative robot system. In addition, it is possible to modify the work model based on the big data analysis data and energy management analysis results.

In addition, the control unit is operated in communication with the recognition unit, and the control unit receives a signal from the recognition unit and controls the operation unit of the robot to perform a multi-step operation.

The present invention described above is not limited by the above-described embodiments, and various substitutions, modifications, and changes are possible within the scope without departing from the technical spirit of the present invention. it is clear to one

Claims (3)

A worker terminal that displays any one or more selected robots among robots of the smart factory on an augmented reality (AR) screen, and checks the menu information on the augmented reality screen when the operator clicks a menu button displayed in the augmented reality screen ;
an interface unit configured to allow the operator to remotely control the robot;
a recognition unit communicating with the interface unit to recognize the operation taken by the operator when the operator manipulates the interface unit;
a control unit communicating with the recognition unit;
an operation unit of the robot controlled by communication with the control unit;
And a management server for managing collaboration of the smart factory;
includes,
Any one or more of a plurality of worker lists, work process solutions, and work demonstration information of the robot is displayed on the augmented reality screen,
The recognition unit recognizes the action taken by the operator while the operator manipulates the interface unit based on the work model of the multi-step operation,
The recognition unit recognizes the start and completion of manual and automatic operations,
The recognition unit smart factory collaboration system using augmented reality, characterized in that when the manual operation is completed, instructing the control unit to perform an automatic operation of the operation unit of the robot based on the work model of the multi-step task.
The method according to claim 1,
The management server stores manual information of the robot, a list of workers, a work process solution, and work demonstration information of the robot,
When the operator terminal recognizes the robot's unique identification mark or real, a menu button appears on the augmented reality screen,
A smart factory collaboration system using augmented reality, characterized in that the robot's unique identification mark or real object is matched with one or more workers selected from the worker list, and whether or not to approve the use access is determined and displayed on the worker terminal.
The method according to claim 1,
The recognition unit stores the work model of multi-step work through CPS (Cyber Physical System)-based process analysis,
The CPS-based process analysis receives data from a sensor provided in the operating unit of the robot,
The recognition unit creates analysis data by analyzing the data received from the sensor, and by simulating the analysis data to update the work model of the multi-step work Smart factory collaboration system using augmented reality.

Images