WO2022097775A1

WO2022097775A1 - 5g-based production, logistics management, and cloud-oriented machine vision service providing method

Info

Publication number: WO2022097775A1
Application number: PCT/KR2020/015384
Authority: WO
Inventors: 박덕근; 김유진; 윤종필
Original assignee: 위즈코어 주식회사; 에스케이텔레콤 주식회사
Priority date: 2020-11-05
Filing date: 2020-11-05
Publication date: 2022-05-12

Abstract

A 5G-based production, logistics management, and cloud-oriented machine vision service providing method according to an embodiment of the present invention comprises the steps of: specializing, through an ultralow latency ultrahigh speed large capacity network, large volume data and control data, the real-time property of which matters, and processing a service such as a robot control and a quality check through machine learning; capturing an image of a product by a high-resolution camera connected to the network, and determining whether the product is defective, by using an algorithm trained through machine learning; providing a factory automation service by using a multi-function robot in which the network, a robot vision system, a 6-axis collaborative robot arm, and an AMR are integrated; augmenting, in real time, in-plant installation state and sensor information in an AR device, and providing a manual support augmentation service; and controlling a 5G-based flexible production testbed.

Description

How to provide 5G-based production, logistics management and cloud-oriented machine vision services

The present invention relates to a 5G-based production, logistics management, and cloud-oriented machine vision service providing method.

In the era of the 4th industrial revolution, it is urgent to develop smart production and logistics services and intelligent solutions based on ICT technology to solve the problem of labor shortage due to aging and to improve production and labor efficiency. In this era of the 4th industrial revolution, the survival of a company can be guaranteed only when things necessary for production are fused with ICT technology and new values are created through intelligence.

Although the manufacturing sector accounts for a high proportion of the gross domestic product, the proportion of labor-intensive manufacturers is high, and only a small fraction of manufacturers can improve productivity and create high value-added products by introducing ICT technologies such as artificial intelligence.

In addition, since response speed and stability of production facility control are directly related to production yield, they have relied on industrial wired networks of foreign companies despite their high prices. The introduction of wireless networks has not been expanded due to problems such as response delay and access terminal restrictions.

In addition, the industrial field application of artificial intelligence technology is mainly attempted by large enterprises, and there is a problem in that it is difficult to introduce it in small and medium-sized manufacturing industries due to the excessive installation cost and learning period.

Accordingly, 5G-based production that can transmit and receive ultra-low latency, ultra-capacity, and ultra-realistic data between devices using the 5G network, collects data from the manufacturing site and analyzes the collected data with artificial intelligence under the cloud infrastructure, A method for providing logistics management and cloud-oriented machine vision services is required.

The problem to be solved by the present invention is to provide a method for providing a 5G-based production, logistics management, and cloud-oriented machine vision service.

Specifically, the problem to be solved by the present invention is to be able to transmit and receive ultra-low delay, ultra-capacity, and ultra-realistic data between devices using a 5G network, and collect data from the manufacturing site through this, and use the collected data under the cloud infrastructure. It is to provide a method for providing 5G-based production, logistics management, and cloud-oriented machine vision services analyzed by artificial intelligence.

The problems of the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the following description.

In order to solve the above problems, a method for providing a 5G-based production, logistics management and cloud-oriented machine vision service according to an embodiment of the present invention is provided.

5G-based production, logistics management, and cloud-oriented machine vision service provision method according to an embodiment of the present invention majors in control data and large-capacity data in which real-time is important through an ultra-low latency, ultra-high-speed large-capacity network, and quality through machine learning processing services such as inspection and robot control; photographing a product with a high-resolution camera connected to the network, and determining whether the product is defective using an algorithm learned through machine learning; providing a factory automation service using a multi-function robot in which the network, the robot vision system, a 6-axis collaborative robot arm, and an AMR are integrated; Augmenting the facility status and sensor information in the factory in real time to the AR device, and providing a manual support augmentation service; and controlling the 5G-based flexible production test bed.

Details of other embodiments are included in the detailed description and drawings.

According to the present invention, interworking/compatibility can be secured in advance by applying step-by-step technology in consideration of domestic and foreign 5G technology standardization and commercialization time.

The present invention can minimize the technical limitations of service implementation by applying ultra-low delay technology for each network section.

The present invention can secure technical versatility by designing for a commercial network structure.

The present invention can provide communication and overall environment verification for cloud application.

The present invention can stably store and process various heterogeneous data.

The present invention can promote efficient service development through role sharing among consortium participating companies.

The present invention enables technology recycling and rapid system construction through machine vision service.

The present invention makes it possible to derive concrete results through close collaboration with the demonstration participating organizations.

The present invention enables selection of a collaborative robot for AMR installation and selection of a 3D vision sensor through a joint test.

The present invention enables the development of a dedicated AMR design for mounting a collaborative robot and a 3D vision sensor electronic unit.

The present invention enables sharing of real-time status and map information of a manufacturing robot.

The present invention can implement network ultra-latency through connection with an edge computing device.

The present invention can utilize a glass-type HMD device in consideration of the operator's convenience of use in the field.

The present invention can provide the augmentation service of the status information and the manual of the manufacturing robot.

The present invention displays sensor data by grafting IoT technology, and it is also possible to inquire about logistics loading information.

According to the present invention, an integrated test environment and equipment interlocking test are possible, and the VoC of the field can be reflected early by applying machine vision and a robot to an actual operation line.

The effect according to the present invention is not limited by the contents exemplified above, and more various effects are included in the present specification.

1 is an exemplary diagram for explaining a 5G-based smart factory according to an embodiment of the present invention.

2 is an exemplary diagram for explaining various 5G-based smart factory systems according to an embodiment of the present invention.

3 is an exemplary diagram for explaining a 5G network according to an embodiment of the present invention.

4 to 6 are exemplary views for explaining a machine learning platform according to an embodiment of the present invention.

7 and 8 are exemplary views for explaining a machine vision system according to an embodiment of the present invention.

9 is an exemplary view for explaining a multi-function robot system according to an embodiment of the present invention.

10 and 11 are exemplary views for explaining a manufacturing facility management AR system according to an embodiment of the present invention.

12 is an exemplary view for explaining a flexible production line system according to an embodiment of the present invention.

13 is a block diagram illustrating an apparatus for providing a 5G-based production, logistics management, and cloud-oriented machine vision service according to an embodiment of the present invention.

14 is a flowchart illustrating a 5G-based production, logistics management, and cloud-oriented machine vision service providing method according to an embodiment of the present invention.

Advantages and features of the present invention and methods of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be embodied in various different forms, and only these embodiments allow the disclosure of the present invention to be complete, and common knowledge in the art to which the present invention pertains It is provided to fully inform those who have the scope of the invention, and the present invention is only defined by the scope of the claims. In connection with the description of the drawings, like reference numerals may be used for like components.

In this document, expressions such as "have," "may have," "includes," or "may include" refer to the presence of a corresponding characteristic (eg, a numerical value, function, operation, or component such as a part). and does not exclude the presence of additional features.

In this document, expressions such as “A or B,” “at least one of A or/and B,” or “one or more of A or/and B” may include all possible combinations of the items listed together. . For example, "A or B," "at least one of A and B," or "at least one of A or B" means (1) includes at least one A, (2) includes at least one B; Or (3) it may refer to all cases including both at least one A and at least one B.

As used herein, expressions such as "first," "second," "first," or "second," may modify various elements, regardless of order and/or importance, and refer to one element. It is used only to distinguish it from other components, and does not limit the components. For example, the first user equipment and the second user equipment may represent different user equipment regardless of order or importance. For example, without departing from the scope of the rights described in this document, the first component may be named as the second component, and similarly, the second component may also be renamed as the first component.

A component (eg, a first component) is "coupled with/to (operatively or communicatively)" to another component (eg, a second component); When referring to "connected to", it will be understood that the certain element may be directly connected to the other element or may be connected through another element (eg, a third element). On the other hand, when it is said that a component (eg, a first component) is "directly connected" or "directly connected" to another component (eg, a second component), the component and the It may be understood that other components (eg, a third component) do not exist between other components.

The expression "configured to (or configured to)" as used in this document, depending on the context, for example, "suitable for," "having the capacity to ," "designed to," "adapted to," "made to," or "capable of." The term “configured (or configured to)” may not necessarily mean only “specifically designed to” in hardware. Instead, in some circumstances, the expression “a device configured to” may mean that the device is “capable of” with other devices or parts. For example, the phrase "a processor configured (or configured to perform) A, B, and C" refers to a dedicated processor (eg, an embedded processor) for performing the corresponding operations, or by executing one or more software programs stored in a memory device. , may mean a generic-purpose processor (eg, a CPU or an application processor) capable of performing corresponding operations.

Terms used in this document are only used to describe specific embodiments, and may not be intended to limit the scope of other embodiments. The singular expression may include the plural expression unless the context clearly dictates otherwise. Terms used herein, including technical or scientific terms, may have the same meanings as commonly understood by one of ordinary skill in the art described in this document. Among terms used in this document, terms defined in a general dictionary may be interpreted with the same or similar meaning as the meaning in the context of the related art, and unless explicitly defined in this document, ideal or excessively formal meanings is not interpreted as In some cases, even terms defined in this document cannot be construed to exclude embodiments of this document.

Each feature of the various embodiments of the present invention may be partially or wholly combined or combined with each other, and as those skilled in the art will fully understand, technically various interlocking and driving are possible, and each embodiment may be implemented independently of each other, It may be possible to implement together in a related relationship.

Hereinafter, various embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Referring to Figure 1, the 5G-based smart factory builds a manufacturing patent network infrastructure that combines 5G (Fifth Generation) and TSN (Time Sensitive Network) technologies, and based on this, a cloud-oriented machine learning platform and machine vision for technology verification ( Machine Vision), Multi-Function Robot, and AR service will be built and demonstrated. Industry 5G is a factory-specific industrial 5G for ultra-low latency, ultra-high speed, and ultra-connection, and the ML (Machine Learning) Cloud analyzes numerous data required for the manufacturing process based on machine learning. Demo site enables education, joint research, and demonstration, and Machine Vision enables automatic quality analysis through image analysis. Multi-Function Robot enables work/logistics collaboration within a factory, and AR service enables work order, information delivery, and manual confirmation.

Referring to FIG. 2 , the 5G-based smart factory system includes an ultra-low latency/large-capacity network, a cloud machine learning platform (Cloud ML Platform), and three types of demonstration services.

Ultra-low latency/high-capacity networks include Pre-5G and Multi-access Edge Computing. Pre-5G includes Pre-5G application, commercial 5G verification, and 5G facility production. Multi access Edge Computing includes TSN development, 5G MEC interworking and 5G standardization facility interworking.

Cloud machine learning platforms include Edge Cloud and 5G Industry Cloud. 5G Industry Cloud includes GPU infrastructure construction, UI/UX development, sensor data interworking and integrated cloud.

The three types of demonstration services include Vision, Robot, and AR. Vision includes vision construction, 3D vision development, and general-purpose core development. Robot includes AMR production, vision + AMR integration and 5G interworking facilities. AR includes primary prototyping, machine location identification, and AR + AMR.

Referring to FIG. 3 , the 5G network combines wireless 5G and wired TSN to form an ultra-low-latency, ultra-high-capacity data transmission network infrastructure. Through this, the present invention can be utilized for control data, which is important in real-time, and image information transmission service for vision inspection that requires large-capacity data transmission.

The 5G network includes a Pre 5G (60GHz band) transmitting terminal that interworks with production/logistics equipment such as AMR, Bin Picking Robot, and Machine Vision, and a SW interface for interworking between Pre 5G receivers and MEC facilities is provided. In addition, an industrial TSN prototype is provided.

The 5G network provides external interfaces for AMR, Bin Picking Robot, Machine Vision facilities, and commercial 5G modems, and a computing node for MEC implementation of the service end in the 5G base station (concentrator) is built. In addition, TSN-based industrial profiles such as improved forwarding, time synchronization, and queue control functions are provided.

The 5G network includes 5G wireless communication-based production/logistics (AMR, Bin Picking Robot, MachineVision) facilities, a commercial 5G modem is embedded in the production/logistics facility, and interworking verification between TSN equipment and commercial 5G standard equipment for telecommunication companies is provided. do.

Through such a 5G network, a low-latency packet forwarding technology, a low-latency transmission technology for supporting mission critical services, and a time synchronization function between TSN devices can be provided. In addition, a reliable packet forwarding function for services sensitive to Packet Loss Ratio can be implemented, and a transmission quality management solution for low-delay service support can be built. In addition, service characteristics are discovered through quality measurement of Control Packet and Data Packet for each Smart Factory Application, and a commercial Network Switching Chipset-based TSN switch prototype can be manufactured.

Through this, the present invention enables the introduction of high-speed/ultra-low latency network infrastructure in small and medium-sized factories by applying commercial network-based 5G wireless communication and TSN-based wired communication technology, and various smart factory services based thereon.

The present invention is 5G wireless Infra. It is possible to develop and demonstrate an End-to-End Real Time Network in the form of commercial service by combining wired TSN technology and Ethernet-based industrial TSN Profile development to respond to remote control services such as AMR control.

The present invention makes it easy to secure a high-speed/ultra-low delay network base for small and medium-sized factories by utilizing commercial communication infrastructure, and replaces the wired-based domestic manufacturing facility control network dominated by global companies with a 5G wireless network to reduce dependence on foreign solutions. It is minimized, and it is easy to spread and apply by interworking with global standard communication technology with high versatility.

Referring to Figure 4, the machine learning platform is specialized in an industry where realtime is important and provides real-time processing through the 5G network-based Edge Cloud located in the factory and the machine learning learning cloud located in the central IDC in the form of SaaS. Through this, integrated process analysis of data generated within the plant is possible.

The machine learning platform provides a machine learning platform based on GPU development verification and enables cloud design suitable for manufacturing environments. The machine learning platform can provide analysis of the manufacturing environment by major target customers.

The machine learning platform can design and develop a cloud portal, and it is possible to develop a portal UI/GUI that can check user data.

The machine learning platform can provide an integrated analysis monitoring platform based on standard OPC application and large data processing.

The machine learning platform can be linked with additional three types of interlocking equipment. It is possible to select 3 types of additional interlocking data (vibration sensor, temperature sensor, thermal image) and to link REST API with solutions already owned by participating companies.

The machine learning platform can apply a filtering algorithm by classifying meaningful data among additional input data. It is possible to analyze the structure and function of the existing S/W, derive improvements and priorities for the existing S/W, and review the UI scenario according to the addition of new functions.

The machine learning platform can optimize and verify cloud system performance, and perform verification comparisons for stability and performance optimization.

The machine learning platform can develop and verify failure analysis algorithms. The machine learning platform enables failure analysis algorithm design and platform development.

The machine learning platform is capable of integrating and analyzing two or more types of factory data. The machine learning platform enables linkage of two existing legacy systems (MES, ERP, etc.), development of correlation analysis algorithms, and linkage analysis of quality data and integrated process data.

Referring to Figure 5, the machine learning platform can provide a scale-out function of the platform infrastructure for a large data processing function, a Tensor Flow provision function, a GPU-based machine learning platform, a VM provision and monitoring function, and an SQS compatible queue service. there is.

Referring to FIG. 6 , the machine learning platform may build a cloud platform development prototype SMIC and provide a cloud portal configuration screen UI/UX.

Accordingly, the machine learning platform is able to move away from the existing wired-based data interlocking and storage and simple statistical prediction systems, and processing large-scale real-time data required for the manufacturing process based on 5G, service quality inspection using machine learning results, and robot control. It is possible.

Through this, the present invention can expand the types of data that can be collected through 5G interworking and improve the collection speed.

The present invention can expand the range of data that can be analyzed in the manufacturing field based on an infrastructure capable of processing large-capacity/real-time data.

The present invention can support the application of customized S/W package for each function for interworking with additional applications, and can lay the foundation for various algorithms and correlation analysis through integrated analysis of image data and production processes that can determine quality.

The present invention can establish interlocking standardization for general-purpose machine learning in the future, and reduce infrastructure costs for processing and building large-scale manufacturing data, so that the smart factory project aimed at the nation can be smoothly carried out.

The present invention blocks domestic technology leakage to a foreign cloud, and enables regular audits through the certification examination of related service companies that can be managed and supervised under the leadership of the government.

The present invention makes it possible to create domestic jobs and secure SW competitiveness for related businesses, secure domestic cloud technology suitable for the domestic environment and provide services, and secure source technology and competitiveness for smart factory solutions.

Referring to FIG. 7 , the machine vision system can provide a quality inspection automation solution that photographs a product with a high-resolution camera connected to 5G, and determines in real time whether a mass-produced product is defective using an algorithm learned through machine learning.

The machine vision system can provide machine vision service based on 5G Machine Learning Cloud Platform.

The machine vision system can provide related services by installing the machine vision solution on the cloud industry platform. This service may be a service for demonstrating automation of quality control by collecting large-capacity, high-quality quality control image data and applying it to the automobile parts production process.

The machine vision system can be applied to the automotive parts industry to perform performance verification. Such a machine vision system can perform performance verification by identifying the required model and applying the As Is, To Be model to the sample at the model factory. The machine vision system can consider the sensor and data collection status applied to the facility, environmental variables, etc., and prepare a verification basis.

Machine vision systems can provide high-volume 3D image inspection algorithms. The machine vision system can provide high-speed pre-processing technology for large-capacity, high-quality 3D image data and high-accuracy machine vision 3D image quality inspection algorithms.

The machine vision system can be applied to the empirical model to perform large-capacity 3D image machine vision performance verification. The machine vision system can identify the verification target that requires high-capacity image-based quality inspection and perform performance verification by applying the As Is, To Be model to the sample in the factory.

A machine vision system can provide a general-purpose Industry Machine Vision core platform. The machine vision system can collect the relevant data step by step and advance the image detection algorithm. The machine vision system may provide a general-purpose algorithm capable of determining the quality of defects other than the previously learned parts. Machine vision systems provide a way to build an initial model with a small amount of data and then continuously learn from new (incremental) data. Machine vision systems have short learning times and can have similar accuracies.

Therefore, the machine vision system can provide a general-purpose Machine Vision Core that can be broadly applied to all areas of the manufacturing industry by standardizing and lightening expensive high-tech solutions such as vision inspection in the semiconductor area and developing it in the form of SaaS.

Through this, the present invention can provide an optimized deep learning analysis technique by applying ultra-expensive vision inspection, which was limited to semiconductors, to the automotive parts quality inspection process, which is the basis of the domestic manufacturing industry.

The present invention can improve the processing speed of a solution using a 5G network, and can reduce dependence on quality experts by automating quality control through Machine Vision.

The present invention makes it possible to learn with a small amount of data through transfer continuous learning, so it is easy to introduce technology for small and medium-sized enterprises (SMEs), and it is possible to provide Industry Machine Vision cloud service connected by 5G network.

The present invention lowers entry barriers for initial construction, enables rapid service use, enables basic analysis improvement without having an ICT professional manpower, and can be expected to improve productivity and quality competitiveness of small and medium-sized manufacturers.

Looking at this in more detail, when a product is shot with a high-resolution camera connected to 5G in the smart factory, the high-resolution image taken is transmitted to the edge cloud through 5G. In this case, a high-speed, ultra-low latency 5G wireless network can be built inside the smart factory.

The machine learning-based vision cloud (ML Vision Cloud) receives a large-capacity product image from the edge cloud, and learns the machine learning algorithm as an input to advance it. In other words, real-time verification/real-time quality measurement is possible using the edge cloud. Here, the machine learning algorithm may be updated through learning using such product images as an algorithm for defect determination. The machine learning-based vision cloud determines whether a product is defective through such an algorithm and delivers normal quantity/information data and defect cause data to the smart factory.

In other words, the machine vision system collects large-capacity, high-quality 3D image data of the production process and stores the collected data in a distributed storage environment. Here, distributed storage is based on GPU-based distributed machine learning, is storage for scale out, and can perform computing resource scheduling.

The machine vision system uses the above-mentioned algorithm to receive the product image on the production line and determine the amount required for the total vision judgment time, and the number of cases where the item with actual quantity among all judged products is judged as defective. It is possible to determine classification accuracy (sensitivity, true positive rate), which is a percentage, and classification accuracy (specificity) (specificity, true negative rate), which is a ratio of predicting that there is no actual defect that does not exist. In other words, the total time required for determination may mean a total time required based on a difference between a time stamp at which a product photo is taken and a time at which a final determination is made through the machine vision service. In addition, the sensitivity may mean the number of actual defective products compared to the total number of machine vision defective products.

Through this, the present invention collects large-capacity, high-quality quality control image data, and applies the machine vision service using the same to the automobile parts generation process to automate the quality management. In addition, the present invention can provide a general-purpose algorithm capable of quality determination for defects other than previously learned parts by collecting relevant data in stages and upgrading the image detection algorithm, and after making an initial model with a small amount of data By continuously learning from new data, it is possible to provide an algorithm with a short learning time and high accuracy.

In various embodiments, the present invention can provide an optimized deep learning analysis technique by applying the ultra-expensive vision audit, which was limited to semiconductors, to the automotive parts quality inspection process, which is the basis of the domestic manufacturing industry.

In various embodiments, the present invention may improve the processing speed of a solution utilizing a 5G network.

In various embodiments, the present invention may reduce reliance on quality experts by automating quality control through machine vision.

In various embodiments, the present invention uses an algorithm capable of learning with a small amount of data through transfer/continuous learning, thereby facilitating the introduction of technology by small and medium-sized enterprises (SMEs).

In various embodiments, the present invention uses a machine vision cloud connected by a 5G network to lower the barrier to entry for initial deployment and to use the service quickly.

In various embodiments, basic analysis/improvement of the present invention is possible without having an ICT professional manpower.

In various embodiments, the present invention can improve the productivity and quality competitiveness of small and medium-sized manufacturers.

In various embodiments, the present invention makes it possible to easily introduce and use advanced facilities by demonstrating cloud-type machine vision services to small and medium-sized auto parts manufacturers who lack IT professional manpower.

In various embodiments, the present invention can increase productivity and price competitiveness through automation of quality inspection, which is a simple repetitive task, and can focus on eco-friendly and autonomous vehicle technologies that are currently being activated, and produce small quantities of various types through flexible production demonstration It is possible to take a lead in this necessary future automobile correction.

Referring to FIG. 8 , the machine vision system performs problem definitions such as whether scratches, air bubbles, stamping/pressing, foreign substances/chips, etc., occur due to physical impact during the process, and learns a model for each segmented area. The detection model is established by performing problem analysis, such as reducing the amount of information to be learned through a similar background, and increasing the data set through part division.

The machine vision system performs defect learning using a DenseNet-based CNN model.

The machine vision system performs defect determination using this pre-trained defect determination model.

In various embodiments, the machine vision system may provide an interface module for a 5G-based machine vision service. Such a machine vision service can be built with a wire-based machine vision solution mounted on a cloud machine learning platform.

Through this, the present invention can verify the performance by designing a decision model and applying the to-be model to the demonstration plant. In addition, the present invention can provide a basis for verification in consideration of the sensor and data collection status, surrounding environment variables applied to the facility, and the like. In addition, the present invention can minimize variations in the existing production line, select a target centering on a line with a high automated quality inspection effect, and support improvement effect information by providing a performance report.

Referring to FIG. 9 , the multi-function robot system uses a 5G wireless communication 3D robot vision system, a 6-axis collaborative robot arm, and a multi-function robot integrated with AMR to recognize various atypical objects and perform bin picking and logistics transfer. can do.

This multi-function robotic system includes a 3D vision sensor and 6-axis collaborative robotic arm integrated hardware. The multi-function robot system enables detailed design and implementation of AMR for transport and operation for factory automation. Multi-function robotic systems can provide Edge Computing module design and prototyping.

The multi-function robot system provides a 5G module, a 3D vision sensor, a 6-axis collaborative robot arm, and an AMR integrated system. The multi-function robot system can be applied with transport and operation AMR testbeds for factory automation.

The multi-function robot system can upgrade the 5G module, 3D vision sensor, 6-axis collaborative robot arm and AMR integrated system, and can be applied to AMR mass production sites for transport and work for factory automation.

Accordingly, the multi-function robot system can expand the application area of 3D Vision solution by combining an advanced industrial 3D scanner with AMR and robot arm, and preoccupy the multifunctional robot market in the form of ‘AMR + robot arm + various solutions’.

Through this, the present invention can eliminate the physical scan operation by applying structured light irradiation to minimize the 3D image acquisition time.

The present invention can transmit large-capacity images through 5G and perform related image processing in the cloud.

The present invention is the development of a mobile solution that combines 6-axis collaborative robot & AMR in the existing fixed solution, and it can secure flexibility to be applied to various field environments.

Referring to FIG. 10 , the manufacturing facility management AR system uses 5G to augment the facility status and IoT sensor information in the factory in real time to the AR device, and establishes a manual support augmentation service to quickly check information and support remote augmentation through field personnel service can be provided.

The manufacturing facility management AR system uses real-time information augmentation technology linked with on-site manufacturing robots. Manufacturing facility management AR system can augment the manufacturing robot remote control manual with mobile devices. The manufacturing facility management AR system can augment the status information of the edge server-linked real-time robot. The manufacturing facility management AR system is capable of analyzing the location of equipment based on the manufacturing robot map. The manufacturing facility management AR system can analyze and organize the management location map data in the manufacturing robot.

In the manufacturing facility management AR system, real-time information augmentation technology linked with on-site manufacturing robots may be used. The manufacturing facility management AR system can augment the manufacturing robot remote control manual with a glass device. The manufacturing facility management AR system can augment the vibration state information of the Edge Server-linked robot. Manufacturing facility management AR system may use manufacturing robot map-based equipment location identification and augmentation technology. The manufacturing facility management AR system may provide a location map and linkage augmentation service from the manufacturing robot.

Manufacturing facility management AR system can use on-site logistics information augmented inquiry and manufacturing robot remote maintenance support technology. The manufacturing facility management AR system can provide a real-time logistics information inquiry augmentation function. Manufacturing facility management AR system can provide 5G demonstration video call solution utilization and remote interworking support technology service.

Therefore, attempts to utilize AR system for manufacturing facility management at home and abroad by using augmented reality technology for manufacturing and maintenance of industrial sites are continuing, but have not been universalized. It can be expanded to the field of inquiry of logistics loading information.

More specifically, referring to FIG. 11 , the manufacturing facility management AR system enables recognition/tracking based on marker recognition by mounting a unique marker (AR Tag) on the manufacturing robot device through the manufacturing robot device identification recognition technology.

Manufacturing facility management AR system can provide real-time inquiry and augmentation of connected equipment status values of Edge Cloud server through manufacturing robot status information inquiry and augmentation/tracking technology.

Manufacturing facility management AR system can provide manufacturing robot augmented manual production and interaction to provide vision recognition, tracking-based manual augmentation and interface of manufacturing robot remote control.

Manufacturing facility management AR system enables Edge-linked inquiry and rendering of manufacturing robot vibration sensor data through manufacturing robot vibration sensor inquiry and augmentation technology.

Through this, the present invention can configure a service by utilizing marker-based image recognition and the latest Android device's ARCore face recognition technology, rather than spatial recognition-based augmented authoring and face recognition technology.

In the present invention, the main device utilizes a tablet equipped with the latest Google OS, not Google Tango, to secure universality, and UI charts necessary for augmentation can be produced according to the field and use.

The present invention facilitates market expansion by developing a highly versatile OS, and can be expanded to fields requiring remote equipment maintenance (elevators, tractors, etc.) and logistics loading information inquiry fields.

The present invention makes it possible to create new jobs using non-professional personnel according to the accumulation/transmission of Know How.

Referring to FIG. 12 , the flexible production line system can establish a 5G network-based flexible production test bed by developing industrial interface linkages such as 5G, OPC UA, and industrial Ethernet, and developing an integrated process control system.

The flexible production line system uses Pre 5G and Industrial Ethernet linkage technology, and Server client OPC UA can be improved with PubSub structure. The flexible production line system can provide process-integrated connection technology that delivers Pre 5G-based process data.

The flexible production line system provides a 5G interface for smart process equipment and can operate a demonstration line for smart process equipment 5G commercial network verification.

The flexible production line system can provide an interoperability test bed for flexible production of smart factories, and can provide a control integration system and control system for flexible production.

Accordingly, the flexible production line system can strengthen the domestic competitiveness of the modular factory facility occupied by global companies by developing a mobile module combination type field automation line technology by grafting 5G technology to the SBB (Smart Base Block) platform.

Through this, the present invention applies 5G to SBB core technology to provide a moving module combination field automation test bed line technology by supporting wireless-based low latency, and OPC UA to secure real-time of factory device control data by applying 5G technology of high-speed streaming connection technology can be provided.

The present invention makes it possible to manufacture intelligent equipment, devices, and solutions that meet the requirements of the Industrial Internet and Industry 4.0 by using OPC UA and 5G technology in equipment or device manufacturers, engineering companies, and system providing companies.

The present invention can secure compatibility of manufacturing equipment information by automating the tasks of mapping and symbol conversion for applying information control logic, symbols, etc. for each manufacturer of existing equipment/controllers to IT solutions.

Referring to FIG. 13 , an apparatus 1300 for providing a 5G-based production, logistics management and cloud-oriented machine vision service includes a communication unit 1310 , a user input unit 1320 , an output unit 1330 , a memory 1340 , and an interface unit 1350 . ), a control unit 1360 and a power supply unit 1370 , and the like. Since the components shown in FIG. 13 are not essential, an apparatus having more or fewer components may be implemented.

Hereinafter, the components will be described in turn.

The communication unit 1310 may include one or more modules that enable wired/wireless communication between a device and a network in which the device is located. The communication unit 1310 transmits/receives a signal to and from at least one of an external device and a server on a communication network such as the Internet. The signal may include various types of data. The communication unit 210 may receive a product image file photographed from a high-resolution camera.

The user input unit 1320 generates input data for the user to control the operation of the device. The user input unit 220 may include a keypad, a dome switch, a touch pad (static pressure/capacitance), a jog wheel, a jog switch, and the like.

The output unit 1330 is for generating an output related to sight, hearing, or touch, and this may include a display unit 1331 , a sound output module 1332 , and the like.

The display unit 1331 displays (outputs) information processed by the device. For example, the device displays a user interface (UI) or graphic user interface (GUI) related to the system.

The display unit 1331 is a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), a flexible display (flexible). display) and at least one of a three-dimensional display (3D display).

The sound output module 1332 may output audio data received from the communication unit 110 or stored in the memory 1340 . The sound output module 1332 also outputs a sound signal related to a function performed by the device.

The memory unit 1340 may store a program for processing and control of the controller 260 , and may perform a function for temporarily storing input/output data. The memory 1340 may include a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (eg, SD or XD memory), and a RAM. (Random Access Memory, RAM), SRAM (Static Random Access Memory), ROM (Read-Only Memory, ROM), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic It may include at least one type of storage medium among a disk and an optical disk. The device may operate in relation to a web storage that performs a storage function of the memory 240 on the Internet.

The interface unit 1350 serves as a passage with all external devices connected to the device. The interface unit 1350 receives data from an external device, receives power and transmits it to each component inside the device, or allows data inside the device to be transmitted to an external device. For example, wired/wireless headset ports, external charger ports, wired/wireless data ports, memory card ports, ports for connecting devices equipped with identification modules, audio input/output (I/O) ports, A video input/output (I/O) port, an earphone port, etc. may be included in the interface unit 150 .

A controller 1360 typically controls the overall operation of the device. For example, the controller 1360 may major in control data and large-capacity data in which real-time is important through an ultra-low latency, ultra-high-capacity network, and may process services such as quality inspection and robot control through machine learning. The controller 1360 may photograph a product with a high-resolution camera connected to a network, and determine whether the product is defective by using an algorithm learned through machine learning. The controller 1360 may provide a factory automation service using a multi-function robot in which a network, a robot vision system, a 6-axis collaborative robot arm, and an AMR are integrated. The control unit 1360 may augment the facility state and sensor information in the factory in real time to the AR device, and may provide a manual support augmentation service. The controller 1360 may control the 5G-based flexible production test bed.

Also, the controller 1360 may include a graphic module 1361 for parallel data processing. The graphic module 261 may be implemented within the control unit 1360 or may be implemented separately from the control unit 260 .

The power supply unit 1370 receives external power and internal power under the control of the control unit 1360 to supply power required for operation of each component.

Various embodiments described herein may be implemented in a computer-readable recording medium using, for example, software, hardware, or a combination thereof.

According to the hardware implementation, the embodiments described herein include application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), It may be implemented using at least one of processors, controllers, micro-controllers, microprocessors, and other electrical units for performing functions. The described embodiments may be implemented by the controller 1360 itself.

According to the software implementation, embodiments such as the procedures and functions described in this specification may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described herein. The software code may be implemented as a software application written in a suitable programming language. The software code may be stored in the memory 1340 and executed by the controller 1360 .

Referring to Figure 14, the 5G-based production, logistics management, and cloud-oriented machine vision service providing device majors in control data and large-capacity data where real-time is important through an ultra-low latency, ultra-high-speed large-capacity network, and performs quality inspection and A service such as robot control is processed (S1400).

The 5G-based production, logistics management and cloud-oriented machine vision service providing device shoots a product with a high-resolution camera connected to a network, and uses an algorithm learned through machine learning to determine whether the product is defective (S1410).

The 5G-based production, logistics management and cloud-oriented machine vision service providing device provides factory automation services using a multi-function robot that integrates a network, a robot vision system, a 6-axis collaborative robot arm, and AMR (S1420) .

The 5G-based production, logistics management, and cloud-oriented machine vision service providing device augments the facility status and sensor information in the factory in real time to the AR device, and provides a manual support augmentation service (S1430).

The 5G-based production, logistics management and cloud-oriented machine vision service providing device controls the 5G-based flexible production test bed (S1440).

Through this, the present invention can secure interworking/compatibility in advance by applying the technology step-by-step considering the time of domestic and overseas 5G technology standardization and commercialization.

The present invention can stably store and process various heterogeneous data.

The apparatus and method according to an embodiment of the present invention may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination.

The program instructions recorded on the computer readable medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the computer software field. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - Includes magneto-optical media and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

Although embodiments of the present invention have been described in more detail with reference to the accompanying drawings, the present invention is not necessarily limited to these embodiments, and various modifications may be made within the scope without departing from the technical spirit of the present invention. . Accordingly, the embodiments disclosed in the present invention are not intended to limit the technical spirit of the present invention, but to explain, and the scope of the technical spirit of the present invention is not limited by these embodiments. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. The protection scope of the present invention should be construed by the following claims, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of the present invention.

[National R&D project supporting this invention]

[Project unique number] 1711119155

[task number] GK20P0700

[Name of Ministry] Ministry of Science and Technology Information and Communication

[Name of project management (specialized) organization] GiGA Korea Project Group

[Research project name] Pan-ministerial GigaKOREA project (R&D)

[Research project name] 5G-based production/logistics management service and cloud-oriented manufacturing specialized ML platform development

[Contribution rate] 1/1

[Name of project performing organization] SK Telecom Co., Ltd.

[Research Period] 2020-01-01 ~ 2020-12-31

Claims

In the method of providing 5G-based production, logistics management and cloud-oriented machine vision service,

majoring in control data and large-capacity data in which real-time is important through an ultra-low latency, high-speed, large-capacity network, and processing services such as quality inspection and robot control through machine learning;

photographing a product with a high-resolution camera connected to the network, and determining whether the product is defective using an algorithm learned through machine learning;

providing a factory automation service using a multi-function robot in which the network, the robot vision system, a 6-axis collaborative robot arm, and an AMR are integrated;

Augmenting the facility status and sensor information in the factory in real time to the AR device, and providing a manual support augmentation service; and

A method of providing machine vision services for 5G-based production, logistics management and cloud, comprising controlling a 5G-based flexible production test bed.