KR20220099284A - Industrial augmented reality content authoring system and work guide platform including the same - Google Patents

Abstract

The present invention provides an industrial augmented reality content authoring system capable of enabling workers to author augmented reality content for equipment-related augmented reality-based work guides in real time, and furthermore, to guide the work in real time, and a work guide platform including the same. The system enables workers without specialized knowledge to easily create augmented reality content intuitively even at industrial sites.

Description

Industrial augmented reality content authoring system and work guide platform including the same}

The present invention provides an industrial augmented reality content authoring system capable of authoring augmented reality content for an augmented reality-based work guide related to equipment in real time, and further, guiding work by workers in an industrial field, and a work guide platform including the same will provide

Augmented reality (hereinafter referred to as AR) technology, unlike virtual reality (hereinafter referred to as VR), allows users to directly and intuitively interact with content or services in a virtual space in the real world, and There is an advantage that various information can be provided by overlaying virtual information. Industrial AR content, an example of AR technology applied to industrial sites, provides data to workers and is used in assembly, repair, verification, monitoring, quality control, and education fields to improve work efficiency (work efficiency) and improve productivity.

In this way, in order to use industrial AR contents in industrial sites, it recognizes geometric markers, synthesizes virtual contents based on the recognized markers on the real image, and in order to realize augmented reality that provides interaction with virtual contents, virtual contents Since a lot of development work such as designing in many parts of the manipulations for interaction with the user is required, the development cost is high and the development period is long, so there is a great difficulty in commercialization.

However, since conventional technologies are focused on authoring AR content on a PC or mobile device, intuitive AR content authoring is possible by providing HMD (Head-Mounted Display)-based immersive interaction in front of industrial equipment. The need for technology has emerged.

Republic of Korea Patent Publication No. 10-0834904 (published on Jun. 3, 2008), Republic of Korea Patent Publication No. 10-1483054 (announced on January 16, 2015), Republic of Korea Patent Publication No. 10-1940720 (2019.04.17. Announcement)

An object of the present invention is to provide an authoring system for a user (or a worker) wearing an AR device including an HMD to create augmented reality content in an intuitive and real-time manner without PC and virtual augmented reality expertise in an industrial field. .

Another object of the present invention is to enable authoring based on an open industry standard architecture (Open Platform Communication, OPC) when authoring the augmented reality content, so that when authoring augmented reality content related to industrial equipment, several of the industrial equipment The purpose is to allow data to be reflected in the augmented reality content.

Another object of the present invention is to provide an appropriate security means when transmitting and receiving data related to the open industry standard architecture, so as to be robust against data hacking from the outside without affecting performance.

Augmented reality content authoring system according to an embodiment of the present invention, based on information obtained from a worker wearing a head-mounted display (HMD), a workspace creation module for generating an augmented reality-based workspace, the generation An immersive augmented reality content authoring module that produces augmented reality content through interaction with a worker in an established workspace, and communication with the outside to support the standard of an open industry standard architecture when producing the augmented reality content Including a supporting open industry standard architecture communication module, it is characterized in that the worker can directly author the augmented reality content in real time in the industrial field linked to the open industry standard.

According to an embodiment of the present invention, the workspace generating module further includes an image capturing member, collecting a 2D image and a 3D depth image from the image capturing member, generating work space information and marker information, and generating the generated work It is characterized in that the spatial information is transmitted to the immersive augmented reality content authoring module.

According to an embodiment of the present invention, the immersive augmented reality content authoring module uses the generated workspace information and marker information to author immersive augmented reality content, and inserts virtual content when authoring the content. Or, by inserting a user interface, or in connection with the open industry standard architecture, characterized in that the authoring of the content.

According to an embodiment of the present invention, the open industry standard architecture communication module is characterized in that when the content is authored, the industry standard architecture data related to the equipment in the industrial site is received from the outside.

The work guide platform based on the augmented reality content of the present invention includes the above-described augmented reality content authoring system, and while the worker authoring the augmented reality content, the necessary industry standard architecture data is transmitted and received through the open industry standard architecture communication module an open industry standard architecture server; And it characterized in that it further comprises an industrial augmented reality content viewer showing the augmented reality content authored by the worker.

The present invention has the effect of providing the convenience of producing augmented reality content intuitively and easily through immersive interaction in the industrial field.

In addition, the present invention has the effect of authoring augmented reality content suitable for an open industry standard architecture, intuitively and easily in the industrial field by workers without specialized knowledge about augmented reality.

In addition, the present invention has an economic effect of reducing the time and cost required for the production of augmented reality content required in the industrial field because the author does not need to create the augmented reality content in a separate space.

As a result, the present invention has the effect of allowing a small number of factory workers to easily learn training related to equipment in the industrial field through augmented reality contents in an environment of automation / unmanned like a smart factory, and in the field of smart factory, It has the effect of enabling the commercialization of augmented reality technology.

1 shows a block diagram of an augmented reality content authoring system.
Figure 2 shows a block diagram of the work guide platform.
3 is a diagram illustrating a process of creating a workspace.
4 is a flowchart illustrating an authoring module of an immersive augmented reality content authoring module.
5 is a flowchart illustrating an operation of an industrial augmented reality content viewer.
6 is an exemplary diagram of a scenario created in the industrial augmented reality content authoring module.
7 and 8 show an embodiment in which an operator authorizes augmented reality content related to equipment in the field.

Hereinafter, specific embodiments of the present invention will be described in detail with reference to the drawings. However, the spirit of the present invention is not limited to the presented embodiment, and those skilled in the art who understand the spirit of the present invention may add, change, delete, etc. other components within the scope of the same spirit, through addition, change, deletion, etc. Other embodiments included within the scope of the invention may be easily proposed, but this will also be included within the scope of the invention. In addition, components having the same function within the scope of the same idea shown in the drawings of each embodiment will be described using the same reference numerals.

The open industry standard architecture may specifically be a Unified Architecture (UA) of Open Platform Communication (OPC).

In the case of the above OPC UA, it can be seen as a standard communication standard for smart factories. It is an interoperability standard for safe and stable data exchange in industrial automation spaces and other industries. It has the characteristics of ensuring the smooth flow of information in the manufacturing system, and it can solve new requirements for security, data modeling, and multi-platform support, which have become issues in manufacturing systems, and has a high scalability feature.

In addition, the content authoring system 10000 may include a storage medium such as a memory, a hard disk, etc. for storing the modules and a processor for performing a detailed method of providing the modules, and an executable code and result of the modules. .

In addition, when performing detailed functions of the detailed configurations (open industry standard architecture server, industrial augmented reality (AR) content viewer) of the work guide platform 100000 based on the augmented reality content including the content authoring system 10000 , as described above, may include a separate processor and a storage medium.

More specifically, the work guide platform 10000 provides platform characteristics provided by combining hardware and software, including the authoring system and additional external components (server and viewer, etc.).

1 shows a block diagram including the overall configuration of an augmented reality content authoring system 10000 according to an embodiment of the present invention.

The workspace generating module 1000 further includes an image capturing member, as shown in FIGS. 1 to 3, and collects a 2D image and a 3D depth image from the image capturing member to generate work space information, The generated workspace information is transmitted to an immersive augmented reality (AR) content authoring module, which will be described later.

Specifically, the image pickup member may use an RGB-D camera to capture depth information together.

As a result, the workspace generating module 1000 generates a 3D image and a point cloud (PointCloud, data composed of points having three-dimensional coordinates) for recognizing a marker through an image capturing member such as the above-described RGB-D camera. Collect 3D depth images for

At this time, in the workspace creation module, the 2D image calculates the relative position (compared to the position of the camera) and the orientation of the marker attached to the equipment and converts it into the origin coordinate system.

In addition, from the 3D depth image, the features of the workspace are extracted and converted into point cloud information so that it can be used in an authoring module to be described later.

Afterwards, the calculated marker-based origin coordinate system and point cloud-based spatial information are combined so that a large number of virtual information can be displayed in an accurate position on a wide work space.

Thereafter, by the workspace creation module 1000 , the workspace for immersive augmented reality content authoring is created by updating the coordinate information of the point cloud information based on the origin coordinate system.

Finally, the workspace information, marker information, and point cloud information generated by the workspace creation module 1000 are transmitted to the immersive augmented reality (AR) content authoring module.

3 and 8 are diagrams to show a theoretical example and an actual creation process for the process of creating a workspace based on the above-described markers and point clouds, respectively.

The immersive augmented reality content authoring module 2000 uses the marker information and the workspace information received from the workspace creation module 1000 to author immersive augmented reality content, but when authoring the content It is characterized in that the content is authored in real time in an industrial field by inserting virtual content, inserting a user interface, or linking the open industry standard architecture.

In other words, the effect of enabling more intuitive AR content authoring activity in real time by providing HMD-based interaction with equipment in industrial sites in the process of authoring immersive content by a worker wearing an HMD. provides

This provides an effect that immersive content cannot be authored in real time at the work site, and can be authored more easily than conventional techniques that require authoring with separate equipment in a separate authoring space at different times.

A more detailed authoring flowchart of the immersive augmented reality content authoring module 200 is as shown in FIG. 4 .

That is, the worker creates a project related to the content to be authored through interaction with the HMD, and creates a scenario based on the workspace information received from the workspace creation module 1000, marker information, point cloud information, etc. can do.

Specifically, from project creation to scenario creation, it means creating a three-dimensional workspace for AR content authoring on the device.

Here, inserting the marker and the workspace means applying the marker information and the workspace information transmitted from the workspace creation module 1000 to the project.

In addition, the insertion of the virtual content means applying a virtual object such as a 3D model, a document, a text label, an audio, a video, etc. to the scenario.

More specifically, as shown in FIGS. 7 and 8 , a 3D model means virtual models such as arrows, a document means read-only documents such as a PDF, and a text label is a 2D input in a 3D workspace. means characters.

In addition, audio refers to a music file that is connected to a user interface and can be played at a user's desired time.

In addition, a video refers to a 2D image that can be reproduced by being placed in a 3D workspace.

In addition, the user interface insertion means applying a button, a slide bar, a graph, and the like to a scenario.

More specifically, the button may be implemented to provide an action corresponding to on/off in response to a user's selection gesture.

In addition, the slide bar may be implemented to provide an action to transform the interface window in response to the user's drag gesture, and the graph provides an action to visualize data transmitted from the open industry standard architecture server 20000 to be described later as a graph. can do.

Here, as shown in FIG. 7 , the location designation in the workspace is characterized in that it is controlled through HMD-based interaction based on the user's hand gesture.

The HMD-based interaction may specifically include functions of selection, movement, rotation, and zooming.

As a result, the inserted virtual content, the user interface, and scenarios connecting the relationship between the work information are stored in the form of a file and then transmitted to the industrial augmented reality (AR) content viewer 3000 to be described later. More specifically, the file in which the scenarios are stored is preferably implemented in the form of an AssetBundle that can be run at runtime on an open industry standard architecture platform.

6 is an exemplary diagram of a scenario authored by the worker in the immersive augmented reality content authoring module 200 as described above.

1 to 2, the open industry standard architecture communication module 3000 receives the industry standard architecture data related to the equipment of the industrial site from the outside when authoring the immersive augmented reality content. characterized.

In this case, the industry standard architecture data includes work information related to equipment in the industrial site, and specifically, may include information such as the remaining amount of work or the current amount of work performed.

In addition, while the worker authoring the augmented reality content, the open industry standard architecture server 20000 that transmits and receives necessary industry standard architecture data through the open industry standard architecture communication module and the augmented reality content authored by the worker Showing It provides a platform (100000) that can provide a guide for work related to the equipment of the worker further comprising an industrial augmented reality content viewer (30000).

Specifically, by using the work guide platform 100000, it is possible to provide augmented reality content for training related to equipment of workers.

In one embodiment, for new workers who deal with specific equipment for the first time in an industrial field, augmented reality content is provided in relation to the operation or operation of equipment, and these new workers wear HMDs and the like and use the augmented reality content. Enable training related to the operation of the equipment.

Specifically, the open industry standard architecture communication module 3000 transmits the identification number of the equipment actually built in the field to the open industry standard architecture server (20000, OPC UA server) built in the industrial site, and confirms with the identification number It can be implemented as a one-way data transmission device (OPC-UA client) that receives work information (remaining work amount, current work performance amount, etc.) of the used equipment.

More specifically, the open industry standard architecture communication module 3000 receives the industry standard architecture real-time data and historical data corresponding to the equipment from the open industry standard architecture server 20000. The open industry standard architecture receiving unit (OPC UA) Subscriber, reference numeral not shown).

And extract the work information of the equipment from the open industry standard architecture receiver, and use the real-time data and the history in the immersive augmented reality (AR) content authoring module 2000 and the industrial augmented reality (AR) content viewer 30000 Update the curl data.

In addition, the open industry standard architecture communication module 3000 is an industrial augmented reality (AR) content server and resources for augmented reality (AR) content (marker information, workspace information, virtual content, user interface, work information, scenario) etc.) and serves as an augmented reality content resource manager (AR Content Resource Manager, not shown).

In addition, the industrial augmented reality (AR) content server uploads and stores the scenario file created in the immersive augmented reality (AR) content authoring module 2000 through a wired/wireless network, and authoring immersive augmented reality (AR) content By controlling access to augmented reality (AR) content resources requested from the module 2000, the path and data of the requested file are configured to be downloaded.

For this function, the open industry standard architecture communication module 3000 is preferably implemented as a toolkit supporting the open industry standard architecture (OPC UA).

In addition, in communication between the open industry standard architecture communication module 3000 and the open industry standard architecture server 20000, when communicating sensitive data (such as work information) of equipment operating in an industrial field, the data is efficiently In addition to sending and receiving, it is also important to ensure the security and integrity of data.

Preferably, the battery consumption or computing power of the device (eg, HMD) equipped with the open industry standard architecture communication module 3000 is lower than that of a general server environment, and the time required for encryption, etc. Considering that, it is preferable to use a lightweight hash function suitable for the encryption in such an embedded computing environment.

The lightweight hash function is a hash that consumes relatively low computing power designed to ensure the integrity of transmitted or received data, excluding features that require high computing power in standard cryptographic hash algorithms such as SHA-3. It is a function (one-way function).

More specifically, it is preferable to use a sponge algorithm that enables permutation of data without a key (unkeyed) among these lightweight hash functions.

More specifically, the sponge makes the original message (here, the original data of the random key) to a certain size (padding), and then has a specific standard size that only the creator of the key can know (for example, the original message divided into specific bit sizes). After dividing the data into multiple pieces with , the random data is exchanged using several update functions at the end of the data divided into multiple pieces (the split original message), and the other side is implemented to decode using a known reference size.

That is, by utilizing such a lightweight hash function, while securing the security of the hash function, relatively less computing power is required than the use of a general hash function, resulting in the open industry standard architecture communication module 3000 and the industrial field. It is possible to reduce the power consumption of the equipment and to use it for a long time.

Meanwhile, the present embodiments can be implemented as computer-readable codes on a computer-readable recording medium. The computer-readable recording medium includes all types of recording devices in which data readable by a computer system is stored.

In addition, examples of the computer-readable recording medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc., and also implemented in the form of a carrier wave (eg, transmission through the Internet). includes doing In addition, the computer-readable recording medium may be distributed in a network-connected computer system, and the computer-readable code may be stored and executed in a distributed manner. In addition, functional programs, codes, and code segments for implementing the present embodiments can be easily inferred by programmers in the technical field to which the present invention pertains.

Those of ordinary skill in the art to which the present disclosure pertains will be able to understand that the disclosure may be implemented in other specific forms without changing the technical spirit or essential features. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive.

10000: Augmented reality (AR) content authoring system
1000: workspace creation module
2000: Realistic augmented reality (AR) content authoring module
3000: open industry standard architecture communication module
20000 : Open Industry Standard Architecture Server (OPC UA Server, OPC UA Publisher)
21000 : Open Industry Standard Architecture Publisher (Publisher)
30000 : Industrial Augmented Reality (AR) Content Viewer
100000: A work guide platform based on augmented reality content

Claims (5)

In a system for authoring augmented reality (AR) content related to equipment in an industrial field,
The system includes: a workspace creation module for generating an augmented reality-based workspace based on information obtained from a worker wearing a head-mounted display (HMD);
an immersive augmented reality content authoring module for creating augmented reality content through interaction with a worker in the created workspace; and
When producing the augmented reality content, including an open industry standard architecture communication module that supports communication with the outside to support the standard of the open industry standard architecture,
Augmented reality content authoring system, characterized in that the worker can directly author augmented reality content in real time at an industrial site linked to an open industry standard. The method according to claim 1,
The workspace generating module further includes an image capturing member, collecting a 2D image and a 3D depth image from the image capturing member, generating work space information and marker information, and using the generated work space information as the immersive augmented reality content Augmented reality content authoring system, characterized in that it is transmitted to the authoring module. 3. The method according to claim 2,
The immersive augmented reality content authoring module uses the generated workspace information and marker information to author immersive augmented reality content, and inserts virtual content, inserts a user interface, or inserts a user interface when authoring the content. Augmented reality content authoring system, characterized in that the content is authored in connection with an open industry standard architecture. 4. The method of claim 3,
The open industry standard architecture communication module is an augmented reality content authoring system, characterized in that when authoring the content, the industry standard architecture data related to the equipment of the industrial site is received from the outside. Including the augmented reality content authoring system of any one of claims 1 to 4,
an open industry standard architecture server for transmitting and receiving necessary industry standard architecture data through the open industry standard architecture communication module while the worker authoring the augmented reality content; and
A work guide platform based on augmented reality content, characterized in that it further comprises an industrial augmented reality content viewer showing the augmented reality content authored by the worker.

Images