WO2024043897A1

WO2024043897A1 - Industrial tasks supported by augmented virtuality

Info

Publication number: WO2024043897A1
Application number: PCT/US2022/041612
Authority: WO
Inventors: Holly FERGUSON; Ratnadeep Paul; Mareike KRITZLER
Original assignee: Siemens Corporation
Priority date: 2022-08-26
Filing date: 2022-08-26
Publication date: 2024-02-29

Abstract

A method of creating instructions for performing an industrial task in an augmented reality (AR) application includes querying a knowledge base constructed from existing data resources relating to the industrial task. An instruction set for the industrial task is generated and formatted as input for an AR application. The formatted instructions correspond to objects represented in a CAD model. Information relating to the formatted instructions may be displayed along the objects from the CAD model file. Objects represented in the CAD model file may correspond to real-world objects in the user's field of view. In an AR application, the formatted instructions may be overlaid on a real-world object in the user's field of view. The order of relevant objects to display from the CAD model may be obtained by the AR application by constructing the knowledge base in part on the CAD model file and relationships about the components.

Description

INDUSTRIAL TASKS SUPPORTED BY AUGMENTED VIRTUALITY

TECHNICAL FIELD

[0001] This application relates to industrial instructions. More particularly, this application relates to provision of instructions using augmented reality technologies.

BACKGROUND

[0002] In certain factories or field operations that cannot be or are not highly automated, there are certain manufacturing tasks that rely on execution by human workers. In many cases, the tasks requiring the person to view products or procedures represented in a 3D model, like that drawn in a computer-aided design (CAD) program, and to perform steps that result in the products or procedures to match the depiction in the CAD rendering. In addition, such tasks depend not only on information found in the drawings but may also be derived from information passed down as verbal notes, (e.g., by a supervisor or the creator of the workflow in the back office), from known heuristics, or a variety of other data formats such as PDF manuals. Workflows that emerge based on these types of mixed knowledge sources happen frequently in factory and warehouse settings as well as in the field and occur in different levels of tasking and where there may or may not be other workers present to fill in knowledge gaps. For example, one such use case is industrial cart set up and packing where combinations of products must be loaded, secured, and verified to be on a cart prior to being transferred from one point to another.

[0003] A problem arises that there is no standard workflow or manual that a worker can follow explicitly. This causes increased concern where the order of execution, the orientation of parts and the precision of component assembly and packaging is very important. By not following precise instructions, errors can result as damaged goods, delay in production, and higher operating costs. Further, helpful data that is originally available in a procurement system may not be available, known, or used during the assembly.

SUMMARY

[0004] Embodiments described in this application include a method of creating instructions for performing an industrial task in an augmented reality (AR) application, including the steps of querying a knowledge base constructed from a plurality of data resources relating to the industrial task to generate an instruction set for performing the industrial task, formatting the generated instruction set as an input for an AR creation software application, linking related information to the AR creation software application, mapping the formatted instructions to objects represented in a computer aided design (CAD) model file, displaying information relating to the formatted instructions in combination with represented objects from the CAD model file. The display may be an AR enabled head-mounted display. In addition, the objects represented in the CAD model file may correspond to real-world objects in the user’s field of view. In an AR application, the formatted instructions may be overlaid on a real-world object in the user’s field of view. The objects in the CAD model may be obtained by the AR application by constructing the knowledge base at least in part based on the CAD model file.

[0005] Mapping the generated instructions to the objects in the CAD model file may include matching each generated instruction to a one or more related components in the CAD model file and highlighting the mapped one or more related components when the corresponding instruction is displayed. The mapping may be achieved based on physical component identifiers for each related knowledge graph data point, some of which are observable from the initial CAD model file. In some embodiments, the display is an AR enabled head-mounted display.

In some embodiments, the generated instruction set is formatted as an input for an AR creation software application by identifying context in the knowledge graph for each instruction generated for the user and formatting the instruction plus context in a form that is readable by the AR content creation software application, such as a javascript object notation (JSON) format or some other The method of Claim 9, wherein the form that is readable by the AR content creation or other interoperable data exchange format.

[0006] Instructions can also be generated for an automated system to execute with a human using the AR content creation software application to verify that the machine is functioning correctly, as in a human supervisor. In some embodiments, the industrial task may be packing a plurality of objects on an industrial cart. In this case, the CAD model object may represent a configuration of the plurality of objects to be placed on the industrial cart.

[0007] In other embodiments, the industrial task may include a maintenance procedure on an industrial machine or an assembly procedure for a manufactured product. According to embodiments herein the displayed objects in the AR application are the exact same object representations in the CAD model file. The industrial task may be characterized as a set of instruction steps that result in successful performance of the industrial task.

[0008] In other embodiments a computer-based system is provided for creating instructions for performing an industrial task for use in an augmented reality (AR) application. The system includes a computer processor in communication with a non- transitory memory. The non-transitory memory storing instructions that when executed by the computer processor cause the computer processor to query a knowledge base constructed from a plurality of data resources relating to the industrial task to generate an instruction set for performing the industrial task, format the generated instruction set as an input for an AR creation software application, map the formatted instructions to objects represented in a computer aided design (CAD) model file, and provide the mapped information relating to the formatted instructions in combination with represented objects from the CAD model file to an AR content creation application for display to a user.

[0009] Further, the non-transitory memory storing instructions that when executed by the computer processor, cause the computer processor to provide the object representations from the CAD model file to the AR content creation application. The computer processor may further process instructions for causing the computer processor to map each generated instruction to a one or more related components in the CAD model file and cause the AR content creation application to highlight the mapped one or more related components when the corresponding instruction is displayed. The mapping may be achieved based on part identifiers for each related component of the CAD model file. Context for the instructions generated for the user may be extracted from the semantic information stored in the knowledge graph and converted to a format readable by an AR content creation application.

[0010] The invention presented here utilize a knowledge graph containing representations of semantic information by taking as inputs 3D CAD models, written and verbal instructions, associated images, and video resources. Knowledge extraction techniques are performed on these inputs to infer packing instruction order and ultimately generate Augmented Reality enabled instructions and tasks that may be displayed directly in the field of view (FOV) of the operators. The AR instructions based on the CAD file ensure the correct order and fit of the parts are achieved. AR applications that are built from scratch as standalone apps need to recreate information that might be already available in CAD and other data. Directly reusing existing engineering specifications and encoded heuristics to recreate scenarios ensures the correctness of procedure and makes the process faster. In the disclosed embodiments, data used in creating instructions is pre-processed from a common ontology-based knowledge graph that is queried for instructions to execute in real space. Based in context and semantic rules, instructions are populated based on both implicit and explicit domain knowledge. The human performing the task may use AR on a head mounted device to map each of the individual work steps to the parts of tasks (e.g., 3D CAD model to real life product packing/handling/maintenance/assembly/disassembly). Visual representations of the results are displayed into the field of view of a worker demonstrating how/when to put parts together for packing tasks.

[0011] Knowledge graph modeling is utilized to inform recommendations for procedural steps sent to a head mounted device such as HoloLens to be represented visually in AR space which are needed for guiding users through real-world interactions required for work tasks. Semi-automated instruction handling plus directions can also help technicians perform work with less errors and even upskill faster. Also, this approach can teach several workers in parallel and keeps the quality of instructions constant. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The foregoing and other aspects of the present invention are best understood from the following detailed description when read in connection with the accompanying drawings. For the purpose of illustrating the invention, there is shown in the drawings embodiments that are presently preferred, it being understood, however, that the invention is not limited to the specific instrumentalities disclosed. Included in the drawings are the following Figures:

[0013] FIG. 1 is a high-level view of an architecture for systems and methods according to aspects of embodiments discussed in this disclosure.

[0014] FIG. 2 is diagram for creating AR instructions from semantic information in a knowledge graph according to aspects of embodiments of this disclosure.

[0015] FIG. 3 is a functional diagram of an AR content creation application for creating AR based instructions for performing an industrial task according to aspects of embodiments described in this disclosure.

[0016] FIG. 4 is a process flow diagram of a method for creating AR-based instructions for an industrial task according to aspects of embodiments of this disclosure.

[0017] FIG. 5 is a block diagram of a computer-based system which may be used to implement systems and methods for generating AR-base instructions for an industrial task according to aspects of embodiments of this disclosure.

DETAILED DESCRIPTION

[0018] There continues to be numerous challenges with performing industrial tasks where human workers are required to perform assembly, maintenance, packing, or other steps or even where humans act as supervisor for quality assurance tasks where machines execute tasks with instructions. Individuals have a wide range of background, experience, and knowledge regarding a particular line of work. Therefore, any gaps in knowledge or experience must be learned quickly and thoroughly. This presents a particular challenge if employees are at a novice level, are seasonal employees in terms of time on the job, or even if they are experienced and need refreshers or updated training as policies change over time (i.e. OSHA sets new rules they must follow). In addition to skill levels, knowledge is gathered to perform tasks through a variety of methods. These methods may include spending substantial amounts of time reading text-based manuals, having to imagine and conceptualize directions based on partial knowledge, memorizing steps based on verbal training, or making assumptions for steps where no explicit answer or verification strategy is provided. In some cases, training can only be performed via word of mouth and taking a different route for the same task may resulting in longer process times or faulty execution.

[0019] These challenges may lead to error prone performance, and even in cases where the answer exists among the various knowledge sources, it can be tedious and inefficient to find the answers. For example, if there is a question about an ambiguous packing orientation, the worker may make an incorrect assumption if there is no way to ask follow-up questions or validate placements or products. While AR technologies are designed to solve many of these challenges, especially when paired with structured semantic data used to formulate instructions, these are not yet mainstream enough to impact workflows as needed and instructions are not automated enough to be applicable to workers of many skill ranges. Embodiments herein combine both semantic knowledge graph encodings and AR layers to utilize implicit knowledge combined with explicit knowledge needed to solve these challenges. By using the engineering data, correct AR enabled instructions can be created to increase efficiency and accuracy.

[0020] Tasks like executing steps for cart set up and packing or other manual assembly-type tasks can be tedious, time consuming, and require high expertise for success; simultaneously, user-assisted solutions for AR typically need to use complex knowledge from siloed data sources to draw conclusions for interpreting instructions and determining next steps. If knowledge engineering techniques can capture instruction set sequences with dependencies and enrich that with domain context (information only domain experts may know to complete their jobs), then those pieces of knowledge can be used with semantic enrichment to automatically generate task descriptions such as actions to take to complete packing instructions. Completing this workflow is unfortunately not how work is being performed today. Certain aspects do use the current state-of-the- art to improve an overall process; for example, using the latest 3D modeling software to represent how a cart should be packed for technicians to analyze. However, the described invention pushes the current state in terms of automating the interpretation steps by generating instructions and presenting them with 3D AR overlays seen through a device such as HoloLens glasses to help train employees, perform packing tasks with fewer errors, and providing demonstrations of steps to follow.

[0021] This invention achieves AR instruction creation by using recommendations and sentence generation from query results aligned (or to be used in combination with) 3D model-based visualizations. There are five main phases to the invented workflow getting from KG to instruction sentence generation to augmented reality display, here shown along an example in the mobility domain.

[0022] FIG. 1 shows a high-level illustration of an architecture for generating AR-based instructions for an industrial task according to embodiments of this disclosure. Methods and systems described herein utilize a pre-processed semantic knowledge graph 105: A pre-consolidated set of disparate data sources 103, 101 provide inputs for capturing human knowledge relating to the industrial task Sources 103 may include written instructions or manuals. These text-based sources may be in any format, such as a PDF file format or the like. Other sources may be pre-existing human heuristics providing recognized solutions. Further, verbal instructions may be provided that instruct on how to perform the task. From these sources, tasks 104 may be identified along with appropriate actions 106A based the input sources 103 In addition, models constructed in a computer aided design application (CAD) 101 may be used as inputs to the knowledge graph 105. In one example for a cart packing task, CAD model 101 may include packing steps 102 for performing the packing task while accounting for available packing space, product materials (e.g., weight) and for other considerations including safety and efficiency. Actions relating to the packing 106B may be extracted from the CAD model 101 for use in the knowledge graph 105. The knowledge graph 105 allows for further analysis or inference of additional steps for automatically creating instructions for performing the task. CAD model 101 includes information describing the components that make up the task and may include information relating to materials, dependencies, and orientations for products, such as orientation of products in a packing example, which can provide semantic information relating to the completion of the task at hand. These sources 101 , 103 are aligned to industry standard semantics and so represented in the knowledge graph 105. The ontologies which may be extended for use in this application include W3C and OWL ontologies, Functional Ontology for Naive Mechanics (FONM), Sensor Networks (SSN), Product Models, and other structures and models.

[0023] When the knowledge graph is constructed, use semantics may be used to create Instructions. SPARQL queries may be used to extract instruction sets based on learned dependencies and procedural descriptions that are themselves assembled from base patterns. Embodiments may use SPIN Rules and inferencing with SPARQL queries to generate instruction set data in a particular order that the AR content creation software needs in order to match to visual cues.

[0024] The instructions are formatted as an input for AR consumption 107. Once the instruction set is formulated with context required for the user, it is formatted in order such that is readable by the AR content creation software. Also, the CAD file 101 that was the input to generate the instruction set is the same one that is used to highlight components in AR engine 107 which will be the overlaid in the real world as part of the AR animation 109.

[0025] Instructions generated from the knowledge graph 105 may be mapped to correspond to their CAD-based counterpart components. Based on part IDs or other criteria, each relevant text-generated instruction is mapped to its related visual component(s) that may be highlighted from the 3D model 101. Some of the steps will map directly to CAD elements, others will pause the step progression allowing for other instruction types such as safety, tool acquisition, validations, etc.

-I Q- [0026] The AR content creator 107 generates AR components related to the task through the AR device. 109. A currently used component is matched to its instruction and displayed to the technician to follow the step. This can be instructions such as to pack an object by placing it where/how an overlay is displaying through an AR device such as HoloLens glasses from MICROSOFT Corporation along with any contextual information that is available. Alternatively, it can include other types of instruction such as following a certain safety protocol finding a particular tool required to complete the upcoming step.

[0027] Commercial software is available to build AR instructions. Re-flekt one is a no code platform that allows for the creation of work instructions. These instructions will need to be built from scratch by a person who knows the workflow exactly. This tool does not support the creation of instructions from existing data sources. Similar is a tool called Worklink. This tool also allows for the creation of AR work instructions but does not take the existing engineering or procurement knowledge into consideration. CareAR is another commercial solution which displays AR instructions; however, the AR content creation is standalone and not integrated with KG based information extraction from existing instruction manuals.

[0028] Referring now to FIG. 2, a process for generating AR-based instructions according to embodiments of this disclosure. Data sources relating to the task to be performed 201 are imported into a knowledge graph 105 including semantic information relating to components of the system represented in the knowledge graph 105. A user interface 205 is provided for interaction with the user. The knowledge graph 105 may be queried using contextual input 207 to request instructions 203 that are provided to the user via the Ul 205. The instructions 203 will include inferred information for performing the tasks that are derived from the semantic relationships contained in the knowledge graph 105. The instructions 203 are then formatted (e.g., as a JSON file) 209 to be compliant with inputs to an AR content creation application 211 . Information from the AR content creation application 211 such as sample use cases, parsed sample text, or sample language may be provided as contextual input 213 to the user for review and update if desired.

[0029] The AR content creation application 211 provides the generated content to an AR device 215. The AR device may be a head-mounted display or glasses or may be implemented on a mobile display device such as a mobile phone or a tablet. The AR de3vice 215 allows a user 217 to view his/her surroundings and allow the generated instructions to be combined with the real-world view of the user 217. Real world objects 219 in the user’s 217 environment may be leveraged by the AR content to provide the user 217 with integral steps for performing a packing procedure, for identifying parts within the environment of the user 217 by highlighting those parts in the AR device 215, or for performing troubleshooting by displaying help or reference information in the view of the user 217 while performing a task.

[0030] FIG. 3 illustrates an AR content creation application 211 . Information provided as input to the AR content creation application 211 may include paper of verbal instructions 302, video files or images of the process in the task 303 and CAD model files 101. The AR content creation application 211 includes modules that allow for the editing and format of work instructions or text in editor 312, a CAD editor 311 for displaying the components in the CAD file model 101s, a video editor 313 for formatting the video files 303 for display in an AR context, and an image editor 323 for formatting image files 303 provided. The AR content creation application 211 uses instructions generated by the knowledge graph 105 in response to a user query and incorporates the other media formats into the instructions for augmented display to the user. The user will see a current instruction step in combination with related media content including text, CAD components, videos, images or animations. These media are formatted in their respective editors for effective display to the user along with an associated instruction.

[0031] FIG. 4 is a process flow diagram for a method of generating AR-based instructions for performing an industrial task according to aspects of embodiment of this disclosure. A pre-established knowledge base containing semantic information extracted from a variety of sources, contains information reflecting the human knowledge of the system and task to be performed. This knowledge base is queried to generated instructions relating to a task to be performed 401. The knowledge base provides instructions for how to perform the task based on information inferred from the semantic relationships stored in the knowledge base 403. The received instructions are then formatted to a format that is acceptable as an input to an AR content creation application 405. The formatted inputs are provided to the AR content creation application along with components from the CAD model to create content to augment the procedural instructions 407. The additional AR content is displayed to the user in combination with the procedural instructions within the user’s field of view 409. The user is provided visual cues for performing each step of the procedure for following the instructions for completing the desired task efficiently and safely. or supervise a machine and see if the automated process is executed as planned. [0032] FIG. 5 illustrates an exemplary computing environment 500 within which embodiments of the invention may be implemented. Computers and computing environments, such as computer system 510 and computing environment 500, are known to those of skill in the art and thus are described briefly here.

[0033] As shown in FIG. 5, the computer system 510 may include a communication mechanism such as a system bus 521 or other communication mechanism for communicating information within the computer system 510. The computer system 510 further includes one or more processors 520 coupled with the system bus 521 for processing the information.

[0034] The processors 520 may include one or more central processing units (CPUs), graphical processing units (GPUs), or any other processor known in the art. More generally, a processor as used herein is a device for executing machine-readable instructions stored on a computer readable medium, for performing tasks and may comprise any one or combination of, hardware and firmware. A processor may also comprise memory storing machine-readable instructions executable for performing tasks. A processor acts upon information by manipulating, analyzing, modifying, converting, or transmitting information for use by an executable procedure or an information device, and/or by routing the information to an output device. A processor may use or comprise the capabilities of a computer, controller, or microprocessor, for example, and be conditioned using executable instructions to perform special purpose functions not performed by a general-purpose computer. A processor may be coupled (electrically and/or as comprising executable components) with any other processor enabling interaction and/or communication there-between. A user interface processor or generator is a known element comprising electronic circuitry or software or a combination of both for generating display images or portions thereof. A user interface comprises one or more display images enabling user interaction with a processor or other device.

[0035] Continuing with reference to FIG. 5, the computer system 510 also includes a system memory 530 coupled to the system bus 521 for storing information and instructions to be executed by processors 520. The system memory 530 may include computer readable storage media in the form of volatile and/or nonvolatile memory, such as read only memory (ROM) 531 and/or random-access memory (RAM) 532. The RAM 532 may include other dynamic storage device(s) (e.g., dynamic RAM, static RAM, and synchronous DRAM). The ROM 531 may include other static storage device(s) (e.g., programmable ROM, erasable PROM, and electrically erasable PROM). In addition, the system memory 530 may be used for storing temporary variables or other intermediate information during the execution of instructions by the processors 520. A basic input/output system 533 (BIOS) containing the basic routines that help to transfer information between elements within computer system 510, such as during start-up, may be stored in the ROM 531 . RAM 532 may contain data and/or program modules that are immediately accessible to and/or presently being operated on by the processors 520. System memory 530 may additionally include, for example, operating system 534, application programs 535, other program modules 536 and program data 535.

[0036] The computer system 510 also includes a disk controller 540 coupled to the system bus 521 to control one or more storage devices for storing information and instructions, such as a magnetic hard disk 541 and a removable media drive 542 (e.g., floppy disk drive, compact disc drive, tape drive, and/or solid-state drive). Storage devices may be added to the computer system 510 using an appropriate device interface (e.g., a small computer system interface (SCSI), integrated device electronics (IDE), Universal Serial Bus (USB), or FireWire).

[0037] The computer system 510 may also include a display controller 565 coupled to the system bus 521 to control a display or monitor 566, such as a cathode ray tube (CRT) or liquid crystal display (LCD), for displaying information to a computer user. The computer system includes an input interface 560 and one or more input devices, such as a keyboard 562 and a pointing device 561 , for interacting with a computer user and providing information to the processors 520. The pointing device 561 , for example, may be a mouse, a light pen, a trackball, or a pointing stick for communicating direction information and command selections to the processors 520 and for controlling cursor movement on the display 566. The display 566 may provide a touch screen interface which allows input to supplement or replace the communication of direction information and command selections by the pointing device 561. In some embodiments, an augmented reality device 565 that is wearable by a user, may provide input/output functionality allowing a user to interact with both a physical and virtual world. The augmented reality device 565 is in communication with the display controller 565 and the user input interface 560 allowing a user to interact with virtual items generated in the augmented reality device 565 by the display controller 565. The user may also provide gestures that are detected by the augmented reality device 565 and transmitted to the user input interface 560 as input signals.

[0038] The computer system 510 may perform a portion or all of the processing steps of embodiments of the invention in response to the processors 520 executing one or more sequences of one or more instructions contained in a memory, such as the system memory 530. Such instructions may be read into the system memory 530 from another computer readable medium, such as a magnetic hard disk 541 or a removable media drive 542. The magnetic hard disk 541 may contain one or more datastores and data files used by embodiments of the present invention. Datastore contents and data files may be encrypted to improve security. The processors 520 may also be employed in a multiprocessing arrangement to execute the one or more sequences of instructions contained in system memory 530. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions. Thus, embodiments are not limited to any specific combination of hardware circuitry and software.

[0039] As stated above, the computer system 510 may include at least one computer readable medium or memory for holding instructions programmed according to embodiments of the invention and for containing data structures, tables, records, or other data described herein. The term “computer readable medium” as used herein refers to any medium that participates in providing instructions to the processors 520 for execution. A computer readable medium may take many forms including, but not limited to, non- transitory, non-volatile media, volatile media, and transmission media. Non-limiting examples of non-volatile media include optical disks, solid state drives, magnetic disks, and magneto-optical disks, such as magnetic hard disk 541 or removable media drive 542. Non-limiting examples of volatile media include dynamic memory, such as system memory 530. Non-limiting examples of transmission media include coaxial cables, copper wire, and fiber optics, including the wires that make up the system bus 521 . Transmission media may also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

[0040] The computing environment 500 may further include the computer system 510 operating in a networked environment using logical connections to one or more remote computers, such as remote computing device 580. Remote computing device 580 may be a personal computer (laptop or desktop), a mobile device, a server, a router, a network PC, a peer device, or other common network node, and typically includes many or all of the elements described above relative to computer system 510. When used in a networking environment, computer system 510 may include modem 552 for establishing communications over a network 551 , such as the Internet. Modem 552 may be connected to system bus 521 via user network interface 550, or via another appropriate mechanism.

[0041] Network 551 may be any network or system generally known in the art, including the Internet, an intranet, a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a direct connection or series of connections, a cellular telephone network, or any other network or medium capable of facilitating communication between computer system 510 and other computers (e.g., remote computing device 580). The network 551 may be wired, wireless or a combination thereof. Wired connections may be implemented using Ethernet, Universal Serial Bus (USB), RJ-6, or any other wired connection generally known in the art. Wireless connections may be implemented using Wi-Fi, WiMAX, and Bluetooth, infrared, cellular networks, satellite, or any other wireless connection methodology generally known in the art. Additionally, several networks may work alone or in communication with each other to facilitate communication in the network 551. [0042] An executable application, as used herein, comprises code or machine- readable instructions for conditioning the processor to implement predetermined functions, such as those of an operating system, a context data acquisition system or other information processing system, for example, in response to user command or input. An executable procedure is a segment of code or machine-readable instruction, subroutine, or other distinct section of code or portion of an executable application for performing one or more particular processes. These processes may include receiving input data and/or parameters, performing operations on received input data and/or performing functions in response to received input parameters, and providing resulting output data and/or parameters.

[0043] A graphical user interface (GUI), as used herein, comprises one or more display images, generated by a display processor and enabling user interaction with a processor or other device and associated data acquisition and processing functions. The GUI also includes an executable procedure or executable application. The executable procedure or executable application conditions the display processor to generate signals representing the GUI display images. These signals are supplied to a display device which displays the image for viewing by the user. The processor, under control of an executable procedure or executable application, manipulates the GUI display images in response to signals received from the input devices. In this way, the user may interact with the display image using the input devices, enabling user interaction with the processor or other device.

[0044] The functions and process steps herein may be performed automatically or wholly or partially in response to user command. An activity (including a step) performed automatically is performed in response to one or more executable instructions or device operation without user direct initiation of the activity.

[0045] The system and processes of the figures are not exclusive. Other systems, processes and menus may be derived in accordance with the principles of the invention to accomplish the same objectives. Although this invention has been described with reference to particular embodiments, it is to be understood that the embodiments and variations shown and described herein are for illustration purposes only. Modifications to the current design may be implemented by those skilled in the art, without departing from the scope of the invention. As described herein, the various systems, subsystems, agents, managers, and processes can be implemented using hardware components, software components, and/or combinations thereof. No claim element herein is to be construed under the provisions of 35 U.S.C. 112, sixth paragraph, unless the element is expressly recited using the phrase “means for.”

Claims

CLAIMS What is claimed is:

1 . A method of creating instructions for performing an industrial task in an augmented reality (AR) application, comprising: querying a semantically structured knowledge base constructed from a plurality of data resources relating to the industrial task to generate an ordered instruction set for performing the industrial task; formatting the generated instruction set as an input for an AR creation software application; mapping the formatted instructions to objects represented in a computer aided design (CAD) or other 3D model file; displaying supplemental information relating to the formatted instructions in combination with represented objects from the CAD model file.

2. The method of Claim 1 , wherein the display is an AR enabled head-mounted display.

3. The method of Claim 1 , wherein the objects represented in the CAD model file correspond to real-world objects in the user’s field of view.

4. The method of Claim 1 , further comprising: overlaying the formatted instructions on a real-world object in the user’s field of view.

5. The method of Claim 1 , further comprising: constructing the knowledge base at least in part based on the CAD model file and other existing information .

6. The method of Claim 1 , wherein mapping the generated instructions to the objects in the CAD model file further comprises: matching each generated instruction to a one or more related components in the CAD model file; and highlighting the mapped one or more related components when the corresponding instruction is displayed.

7 The method of Claim 6, wherein the matching is achieved based on knowledge base-to-AR mapping identifiers for each related component of the CAD model file.

8. The method of Claim 7, wherein the display is an AR enabled head-mounted display.

9. The method of Claim 1 , wherein formatting the generated instruction set as an input for an AR creation software application further comprises:

Identifying context in the knowledge graph for an instruction generated for the user or an automated system; formatting the instruction and context in a form that is readable by the AR content creation software application.

10. The method of Claim 9, wherein the form that is readable by the AR content creation or other software application is embodied in an interoperable data exchange format.

11 . The method of Claim 1 , wherein the industrial task comprises an order set of instruction steps that result in successful performance of the industrial task.

12. The method of Claim 11 , wherein the CAD model object represents a configuration of the plurality of objects, steps, and verifications to be considered for a given task.

13. The method of Claim 1 , wherein the industrial task comprises of a maintenance procedure on an industrial machine.

14. The method of Claim 1 , wherein the industrial task comprises an assembly, transportation, and/or disassembly procedure for a manufactured product or service.

15. The method of Claim 1 , wherein the displayed objects are the exact same object representations in the CAD model file.

16. A computer-based system for creating instructions for performing an industrial task for use in an augmented reality (AR) application, comprising: a computer processor in communication with a non-transitory memory, the non- transitory memory storing instructions that when executed by the computer processor cause the computer processor to: query a knowledge base constructed from a plurality of data resources relating to the industrial task to generate an instruction set for performing the industrial task; format the generated instruction set as an input for an AR creation software application; map the formatted instructions to objects represented in a computer aided design (CAD) model file; provide the mapped information relating to the formatted instructions in combination with represented objects from the CAD model file to an AR content creation application for display to a user.

17. The system of Claim 16, the non-transitory memory storing instructions that when executed by the computer processor, cause the computer processor to further: provide the object representations from the CAD model file to the AR content creation application.

18. The system of Claim 16, the non-transitory memory storing instructions that when executed by the computer processor, cause the computer processor to further: match each generated instruction to a one or more related components in the CAD model file; and cause the AR content creation application to highlight the mapped one or more related components when the corresponding instruction is displayed.

19. The system of Claim 16, the non-transitory memory storing instructions that when executed by the computer processor, cause the computer processor to further: map the formatted instructions to objects represented in a computer aided design (CAD) model file based on part identifiers for each related component of the CAD model file.

20. The system of Claim 16, the non-transitory memory storing instructions that when executed by the computer processor, cause the computer processor to further: identify context in the knowledge graph for an instruction generated for the user; format the instruction and context in a form that is readable by the AR content creation software application.