WO2015125066A1 - Système et procédé destinés à faciliter la maintenance d'équipement à l'aide de lunettes intelligentes - Google Patents

Système et procédé destinés à faciliter la maintenance d'équipement à l'aide de lunettes intelligentes Download PDF

Info

Publication number
WO2015125066A1
WO2015125066A1 PCT/IB2015/051153 IB2015051153W WO2015125066A1 WO 2015125066 A1 WO2015125066 A1 WO 2015125066A1 IB 2015051153 W IB2015051153 W IB 2015051153W WO 2015125066 A1 WO2015125066 A1 WO 2015125066A1
Authority
WO
WIPO (PCT)
Prior art keywords
protocol
equipment
protocols
maintenance
server
Prior art date
Application number
PCT/IB2015/051153
Other languages
English (en)
Inventor
Evyatar MEIRON
Alex Rapoport
Original Assignee
Fieldbit Ltd.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Fieldbit Ltd. filed Critical Fieldbit Ltd.
Publication of WO2015125066A1 publication Critical patent/WO2015125066A1/fr

Links

Classifications

    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
    • G06F3/1423—Digital output to display device ; Cooperation and interconnection of the display device with other functional units controlling a plurality of local displays, e.g. CRT and flat panel display
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G02B2027/0178—Eyeglass type
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2370/00—Aspects of data communication
    • G09G2370/02—Networking aspects
    • G09G2370/022—Centralised management of display operation, e.g. in a server instead of locally
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G3/00—Control arrangements or circuits, of interest only in connection with visual indicators other than cathode-ray tubes
    • G09G3/001—Control arrangements or circuits, of interest only in connection with visual indicators other than cathode-ray tubes using specific devices not provided for in groups G09G3/02 - G09G3/36, e.g. using an intermediate record carrier such as a film slide; Projection systems; Display of non-alphanumerical information, solely or in combination with alphanumerical information, e.g. digital display on projected diapositive as background
    • G09G3/003—Control arrangements or circuits, of interest only in connection with visual indicators other than cathode-ray tubes using specific devices not provided for in groups G09G3/02 - G09G3/36, e.g. using an intermediate record carrier such as a film slide; Projection systems; Display of non-alphanumerical information, solely or in combination with alphanumerical information, e.g. digital display on projected diapositive as background to produce spatial visual effects

Definitions

  • the present invention is in the field of equipment maintenance and more specifically facilitating equipment maintenance using smartglasses and augmented reality.
  • Downtime is the major cost factor in process industries, accounting for an equivalent of 20% of all production costs. In the utility and manufacturing sectors, downtime cost is estimated even higher. Enterprises lose valuable direct production and labor hours cost to unplanned downtime, which also negatively impacts the quality of goods or services, delivery time and company image. Enterprises spend billions of dollars on maintenance of field equipment to prevent failure and reduce downtime, with limited success.
  • a well trained technician and a well-defined maintenance routine achieve better quality, longer life of the handled system and more satisfied customers.
  • Augmented reality is a live, direct or indirect, view of a physical, real- world environment whose elements are augmented (or supplemented) by computer-generated sensory input such as text, image, sound, video, graphics or GPS data. Augmentation is conventionally in real-time and in semantic context with environmental elements, such as sports scores on TV during a match. With the help of advanced AR technology (e.g. adding computer vision and object recognition) the information about the surrounding real world of the user becomes interactive and digitally manipulate. Artificial information about the environment and its objects can be overlaid on the real world.
  • advanced AR technology e.g. adding computer vision and object recognition
  • Hardware components for augmented reality are: processor, display, sensors and input devices (e.g. camera).
  • Modern mobile computing devices like smartphones and tablet computers contain these elements which often include a camera and MEMS sensors such as accelerometer, GPS, and solid state compass, making them suitable AR platforms.
  • a head-mounted display is a display device paired to a headset such as a harness or helmet.
  • HMDs place images of both the physical world and digital objects over the user's field of view.
  • Modern HMDs often employ sensors for six degrees of freedom monitoring that allow the system to align virtual information to the physical world and adjust accordingly with the user's head movements.
  • HMDs can provide users immersive, mobile and
  • AR displays can also be rendered on devices resembling eyeglasses, generally referred to as smartglasses. Versions include eye wear that employ cameras to intercept the real world view and re-display its augmented view through the eye pieces and devices in which the AR imagery is projected through or reflected off the surfaces of the eye wear lens pieces.
  • smartglasses Various alternative smartglasses are provided such as Google Glass
  • a system for facilitating equipment maintenance comprising: a server configured to store equipment configurations and maintenance protocols; at least one equipment maintenance protocols creation facility communicating with the server, said protocols creation facility configured to automatically create maintenance protocols for equipment comprising predefined components; and a plurality of smartglasses wearable by field technicians, the smartglasses running a smartglasses application configured to execute equipment maintenance protocols.
  • the system may further comprise an adviser application running on at least one adviser computer, the adviser application configured to communicate in real time with the field technicians via the system server to provide Over the Shoulder Coaching (OTSC).
  • OTC Over the Shoulder Coaching
  • the smartglasses application may be configured to provide visual and audio maintenance instructions communicated from the server to the field technician and to communicate back to the server information pertaining to a currently ongoing maintenance.
  • the server may be further configured to store recorded or reported equipment maintenance history, compliance reports including information about performed actions, measured field data, captured images and voice
  • the server may additionally communicate with at least one of: equipment manufacturers; enterprise management; and the professional community.
  • the equipment maintenance protocols may be stored in a script format designed to facilitate identification of various equipment parts, displaying the identified parts or images captured by the smartglasses overlaid with textual or audio instructions and deviation from the protocol.
  • the at least one equipment maintenance protocols creation facility may comprise: protocol building materials comprising protocol templates, digital materials describing various equipment parts, operations pertaining to said equipment parts, assembly and maintenance procedures; an interactive protocol building application configured to build protocols from said protocol building materials; and means for converting the resulting protocols into proprietary format and storing them in a protocols database on the server.
  • the digital materials may be selected from text, images, video clips and audio recordings.
  • the system may further comprise text to speech conversion means.
  • the equipment maintenance protocol may comprise a common header followed by a list of one or more steps.
  • Each step may have a common step header followed by a specific step data, according to the step type.
  • the equipment configuration may be declared as object oriented templates for re-use of common components and field service actions for defining specific instances of physical equipment.
  • the protocol templates may be declared and re-used for creating a dedicated field service protocol for a specific physical equipment.
  • the OTSC may comprise optical see-through technology.
  • a method of facilitating equipment maintenance comprising: automatically creating equipment maintenance protocols for equipment comprising predefined components; storing said protocols on a server; and running a smartglasses application configured to execute said equipment maintenance protocols.
  • Running the smartglasses application may comprise providing visual and audio maintenance instructions from a protocol stored on the server to smartglasses worn by a field technician and communicating back to the server information pertaining to a currently ongoing maintenance.
  • the method may further comprise storing recorded or reported equipment maintenance history, compliance reports including information about performed actions, measured field data, captured images and voice
  • Storing said protocols may comprise storing in a script format designed to facilitate identification of various equipment parts, displaying the identified parts or images captured by the smartglasses overlaid with textual or audio instructions and deviation from the protocol.
  • Creating equipment maintenance protocols may comprise: providing protocol building materials comprising protocol templates, digital materials describing various equipment parts, operations pertaining to said equipment parts, assembly and maintenance procedures; interactively building protocols from said protocol building materials; converting the resulting protocols into proprietary format; and storing them in a protocols database on the server.
  • the digital materials may be selected from text, images, video clips and audio recordings.
  • the method may further comprise text to speech conversion.
  • the equipment maintenance protocol may comprise a common header followed by a list of one or more steps.
  • Each step may have a common step header followed by a specific step data, according to the step type.
  • the method may further comprise declaring said equipment configuration as object oriented templates for re-use of common components and field service actions for defining specific instances of physical equipment.
  • the method may further comprise declaring and re-using said protocol templates for creating a dedicated field service protocol for a specific physical equipment.
  • the method further comprise communicating in real time with a field technicians via the system server to provide Over the Shoulder Coaching (OTSC).
  • OTSC Over the Shoulder Coaching
  • the OTSC may comprise optical see-through technology.
  • Communicating in real time may comprise using an adviser application running on an adviser computer.
  • a method of Over the Shoulder Coaching (OTSC) of a field technician in relation with equipment to be repaired or maintained comprising: capturing by the field technician an image the equipment; uploading by the field technician said captured image to a system server application and requesting assistance related to said equipment repair or maintenance; publishing by said system server application the assistance request to at least one predefined adviser; receiving by the field technician assistance confirmation from one of said at least one predefined adviser; and conducting an OTSC session between said field technician and said adviser using optical see-through technology.
  • OTSC Over the Shoulder Coaching
  • Capturing may comprise capturing by one of a smartphone and smartglasses.
  • Uploading the captured image may further comprise annotating said uploaded image.
  • Uploading the captured image may further comprise adding text related to said captured image.
  • Fig. 1 is a schematic drawing of the system according to the present invention.
  • Fig. 2 is a schematic drawing showing the various software modules of the system according to the present invention.
  • Fig. 3 is a schematic drawing of the protocols flow according to the present invention.
  • Fig. 4 is a schematic block diagram of the studio, which is the system module in charge of producing equipment maintenance protocols;
  • Fig. 5 is a schematic graphic representation of the assets and components templates
  • Fig. 6 is a schematic graphic representation of the asset object
  • Fig. 8 shows an exemplary MEDIA step
  • Fig. 9 shows the TEXT_READ_MORE step
  • Fig. 10 shows the C U STO M_HTM L step
  • Fig. 11 shows the AR_1 step
  • Fig. 12 shows the GET_VOICE_INPUT step
  • Fig. 13 shows the GET_ INPUT_SELECTION step
  • Fig. 14 is a block diagram showing the main smartglasses application modules and the data flow between them;
  • Fig. 5 is a flowchart showing the general steps taken by an exemplary equipment maintenance smartglass application
  • Fig. 16 is a schematic block diagram showing the server application modules and connectivity
  • Fig. 17 is a flowchart showing the OTSC process
  • Fig. 18 is a schematic block diagram of the various modules of adviser application.
  • Fig. 19 shows the process of OTSC on the technician's side
  • Fig. 20 shows the process of OTSC on the adviser's side; and Fig. 21 shows a picture sent from the technician's smartglasses to the adviser, displayed in the adviser application screen, annotated by the adviser and sent back to the technician.
  • the system according to the present invention is an equipment maintenance protocols generator, designed to automatically create a maintenance protocol for every type of equipment given the equipment components and their attributes.
  • the main goals of the system are: 1. To provide a field technician with interactive guidelines which are
  • smartglasses such as EPSON BT-200.
  • Fig. 1 is a schematic drawing of the system 100 according to the present invention, comprising:
  • a central internet server 1 10 storing equipment configurations 1 18, maintenance protocols 1 15 and recorded or reported equipment maintenance history 120;
  • At least one maintenance protocols creation facility (studio) 125 At least one maintenance protocols creation facility (studio) 125
  • a plurality of smartglasses 145 wearable by field technicians 140.
  • the smartglasses 145 provide visual and audio maintenance instructions to the field technician, communicated from the server 1 10, and
  • the central server 1 10 may also communicate with various stake holders such as, for example: - Equipment manufacturers 170 - for providing information regarding equipment functionality, quality of maintenance, and visibility to technicians;
  • the professional community 180 - for providing additional maintenance knowhow and information regarding functionality of various equipment types.
  • Fig. 2 is a schematic drawing showing the various software modules of the system:
  • Fig. 3 is a schematic drawing of the protocols flow according to the present invention.
  • Fig. 4 is a schematic block diagram of the studio 125, which is the system module in charge of producing equipment maintenance protocols.
  • the protocols are stored in a proprietary script format which is designed to facilitate identification of various equipment parts, display of the identified parts or of images captured by the smartglasses overlaid with textual or audio instructions, deviation from the protocol if necessary, e.g. when an operation has to be repeated, automatic or manual advancement to the next instruction, etc.
  • the studio 125 functionality begins with digital materials such as text files 210, image files 215, audio recordings 218, video files 220 and 2D or 3D graphics files (225, 230 respectively), primarily received from the equipment
  • the equipments configuration database 1 18 is designed with an object- oriented approach.
  • the database holds two types of definitions: Templates (Classes) and Physical objects (Instances)
  • Asset Templates is a system or equipment which is a subject for a field service (e.g. electric panel, conveyor, water quality measurements system).
  • the asset consists of components.
  • a Conveyor is an asset and it is composed of components such as Motor, Belt, Gears and Controller.
  • a protocol defines a field service procedures applied to an asset.
  • Asset Template declares an asset class and defines all components which compose this class. Each component has a number of general attributes (manufacture, model, etc) and definition of service actions which can be applied to this component. Actions are used to define an actual protocol downloaded to the glass.
  • the tables below define component and asset templates:
  • Components are used to define assets templates. Components are common objects used in different assets and assets types.
  • Fig. 5 is a schematic graphic representation of the assets and components templates.
  • Asset Definition An asset is created from a relevant asset template. Once the new asset is created the user should define specific attributes such as Location, Address, Contact person and Dynamic data associated with this asset. The asset inherits all protocols and tags definitions from the associated asset template. It is possible to modify inherited protocols and create new protocols. The asset also holds the reference to all compliance reports.
  • ReportsList List of all compliance reports IDs
  • Fig. 6 is a schematic graphic representation of the asset object. Protocol Templates (Class)
  • a protocol template is the sequence of actions which are already defined for the asset components. To create the Protocol template the user should set the sequence of actions. It is possible to define new actions which were not defined for components.
  • the protocol associated with the asset can be modified or a new protocol can be created.
  • the table below shows the protocol structure.
  • the actions defined in the protocol database should include all information for creating an actual protocol step as defined below.
  • Protocol defines a sequence of operations to be applied to modules, components and parts, as well as equipment specific specifications.
  • a protocol consists of metadata and data.
  • the metadata represents basically the logic of the protocol, the steps definitions, the transitions, menus and more.
  • the data is the real resources used by the protocol - images, videos, texts and any other user given pre-made inputs.
  • the protocol must contain all the information needed in order to allow the smartglasses application located in the smartglasses to execute it, thus allowing the application to be data independent.
  • the protocol supports multiple languages and also multiple hardware devices.
  • the protocol is the common language of the studio application and the smartglasses application; the Studio product is the smartglasses application input. Keeping the protocol as a standard is mandatory in order to make a successful integration.
  • Protocol data is known when one creates it. Some data can be supplied only at runtime; hence the protocol supports receiving online information.
  • the protocol supports a non-linear progress, where inputs from the field can change the execution flow.
  • the protocol is a set of steps comprising: initial step, final step (can be more than one), transitions between the steps, type of each step, menu items, gesture items, online /offline requirements, rank.
  • the system instructs the user not only what to do but also how to do it.
  • the design of the system takes into account the fact that not all the technicians have the same level of expertise. Some technicians are less trained and require more assistance and some are veterans with far more experience and knowledge.
  • the Ranks concept was introduced. Every user has a rank and every step has a minimum rank level. Also, the application can execute the protocol in two modes - "Sequential" or
  • the application when the user chooses to proceed to the next step, the application skips steps having a rank lower than the technician's rank. This way, the protocol can be divided to critical and non-critical paths according to the user's level.
  • Layouts define the visual structure of a user interface. Basically it determines how we see something and not what we see.
  • Android framework has developed an extensive language in order to decouple the logic (code) from the Ul. Using the correct practice, one can develop one layout to multiple devices. Nonetheless, sometimes the generic approach is not the optimal one, and some adjustments will be needed to support a new device.
  • the protocol can stay "thin", holding only the real data, while the layouts are responsible for holding the look and feel of it.
  • the smartglasses application holds a set of predefined layouts, made once by the development team. The same layouts with same IDs are also known to the Studio application, where the user can "play” with the different layouts.
  • the layoutjd is written in the step metadata allowing the smartglasses application to visualize the data as the user requested.
  • Getting user inputs is mandatory in order to complete a maintenance execution.
  • the technician may choose to skip to the next protocol once he completed the current one; he can decide to go back to a previous step. He can also perform some actions which are not part of the specific protocol, for example sending to his manager a picture of the device he is working on because he wants to consult him.
  • the protocol of the present invention supports different types of input. The protocol designer has the freedom to define the required user inputs and their corresponding actions.
  • the user can record an audio TAKE VIDEO
  • the user can record a video READ ALOUD For steps with texts, this can be used to read them to the user
  • the transmission type is derived from the contact definition.
  • OPEN MENU This action displays the options menu.
  • Every gesture item is a pair of type and action.
  • AssetID ID of the asset of the protocol 1234 description More information about the Regular protocol maintenanc e protocol requires_online .
  • Some protocols may require TRUE nnectivity connectivity as a pre-condition user_gestures[j An array of common user See user gestures gestures section menu_items[] An array of common menu items See menu which are presented to the user. items
  • a list of one or more steps each has a common step header followed by a specific step data, according to the step type.
  • Purpose Example type Defines the step parsing and STEP_TYPE_ analyzing TEXTJMAGE id Every step has a unique id In ⁇ 2
  • step identification requires_onli Overrides the global definition False ne_connectivi
  • menu_items[] A list of menu items for this See menu
  • Rank Defines the rank of the step See Ranks
  • step is the last step
  • MEDIA (Fig. 8): Displays text(s), image(s), video(s) and sound(s) to the user. How the data is represented is determined by the layout id, which must match the actual data. e.g. - if there are 4 images defined in the step, the layout must have at least 4 images.
  • TEXT_READ_MORE (Fig. 9): Displays a long text which cannot be shown in one screen, or can be shown but it is not clear. The user can scroll to see all the text.
  • TEXT_CUSTOM_HTML (Fig. 10): This step displays custom html view.
  • AR_1 (Fig. 1 1 ): This step is responsible for Augmented Reality scene.
  • the screen is divided into the camera view and a fixed static view with image and text. This steps used for smartgalsses that do not support optical see- through.
  • GET_VOICE_INPUT (Fig. 12): This step gets voice input from the user.
  • GET_ INPUT_SELECTION (Fig. 13): This step displays one or more options the user can choose from. The selection may be done by sliding backward and forward and tapping twice to select, (tap once opens the menu item).
  • the smartglasses application supports the protocol as defined above.
  • the application also supports remote locations where no internet connectivity is available ("offline”), specifically: - Except for functionalities which by definition require online connection , e.g. real-time video call, the application is fully functional.
  • the application assists the technician by instructing him what and how to do the task.
  • the application doesn't disturb an experienced technician, and assists him only when requested.
  • Fig. 14 is a block diagram showing the main application modules and the data flow between them.
  • Main Entry 1000 When the user launches the application, the first loaded module is the Main Entry. This module is responsible for initializing basic configurations, the welcome screen and the primary menu.
  • the Main entry also looks for an abnormal exit state from the last running protocol. If such is found, the module gives the user the option to continue from where he stopped (specific step). If the user wishes to do so, the Main Entry continues immediately to the Step Controller phase 1020, where the flow continues from the last step.
  • the main menu contains the following items (actions):
  • START_WORK Starts the maintenance protocol by launching the Tasks Updater module 1010.
  • SETTINGS Launches activity which can configure various settings. For example:
  • UPDATE_APPLICATION A request to upgrade the application (if needed).
  • Tasks Updater 1010 This module is responsible for getting the list of the tasks ids (protocols), which the technician is required to perform in the upcoming day. The module connects to the system server and retrieves the list of protocols according to the user credentials and the smartglasses application version.
  • the list of protocol IDs is saved, together with the last update timestamp.
  • the task list must also include the Hash (e.g. md5) of the protocol.
  • the list has to match the application version.
  • a possible option is that the specific protocol has some versions, where some of them are not supported by the current smartglasses application version.
  • Optional data if exists in the server, is also attached (e.g. customer name and address).
  • Advanced feature is that the GPS address of the device will also be attached.
  • Protocol Selector 1030 This module is responsible for selecting the active protocol, the one which the technician needs to use now. There are two ways to select the correct protocol - "Data Oriented” and "Image Oriented”. a. Data Oriented Protocol Selector
  • the application displays the list of protocols waiting for the user, received by the Tasks Updater module 1010. The user can cycle through them, and select the one he needs. The application can also show the user the additional information regarding each protocol, (e.g. customer name) to assist him with the selection. Optimizations:
  • the list is sorted according to the scheduler received from the server, thus the first protocol is supposed to be the first task the technician needs to perform.
  • the list can also be sorted by proximity and alert the user if he is on-site.
  • the system can always select the right protocol for the user.
  • the system is designed to allow the user to select it manually if something unpredictable happens or if the matching suggested by the system is wrong. (Semi-auto).
  • a good solution can be to suggest the protocol to the user and continue if he accepts the suggestion. If he doesn't, show him the available protocols list.
  • the camera When the module loads, the camera is operated, and the user is required to look at the device.
  • the image is compared with the repository of images of the protocols. If it finds a match, the protocol is selected.
  • the task updater is also responsible for fetching the protocol image from the server.
  • the image will be saved for future use under /data/protocols/identification/exists, so it needs to do it only once.
  • protocol (123) is selected. If no matching was found - the system automatically reverts to the "Data Oriented” selector.
  • server search is the solution, which needs online connectivity.
  • Protocol Loader 1040 This module is responsible for configuring the working folder properly with the required protocol. Following this phase, the assumption is that all the data needed for executing the protocol is ready. Notes:
  • Protocols are considered as external data which are not part of the application distribution (APK).
  • Every protocol has a unique ID.
  • the Selector module gives the Loader module the ID of the required protocol.
  • the Loader requests from the updater module to get the protocol from the server. - If the protocol exists in the current folder and no new version was needed, then the loader has nothing to do. - If the protocol doesn't exist in the current folder or it has just been updated, the Loader extracts the protocol to the current folder.
  • Protocol Updater 1050 This module is responsible for retrieving a specific protocol from the server. The module gets as input the required protocol ID.
  • the protocol file is saved under /data/protocols/store/ ⁇ ID>.
  • the Updater also updates a file in the folder called "protocols_hash.xml". This file holds the mapping of every protocol to its hash. This file is an optimization in order not to calculate the hash every execution. (The assumption is that the protocol doesn't change every day).
  • This module is also responsible for keeping the protocols folder (store/) size under some thresholds, and to delete a file if the limit was crossed.
  • the module will delete the minimum protocols (files) needed in order not to cross the limit.
  • the oldest(s) file will be deleted first, as long as they are not in the task list of the day.
  • the Uploader connects to the server in a known URL, giving as input the required ID and the application version.
  • the Server's responsibility is to supply the required protocol.
  • Step Controller 1020 This module is responsible for handling the flow of steps in the protocol.
  • the controller controls the flow of the protocol. It initiates a step according to the first step defined in the protocol. It then executes it and waits for the result. According to the result and the next step ID, the controller initiates the next step.
  • the Controller has two phases - Preprocess and Process.
  • the controller is responsible for launching every step (Android Activity) according to the flow, and to wait for the response. According to the response, the controller realizes what the next step is and activates the appropriate activity. It is also responsible for sending the required metadata for every activity. For instance, let's assume we have a short protocol of two instruction steps, each showing image and text. The controller, initiates an activity of type "TEXTJMAGE", and passes it relevant metadata object, containing the image name and text id for this step.
  • the Activity displays the information needed, when pressed “Next” it terminates with positive result code.
  • the controller 'knows' the next step on the positive path is "5" and launches it, this time with the relevant metadata for it.
  • the controller should not parse the step data, only send the specific section, ⁇ step> to ⁇ /step>, to the relevant activity as a simple string. This way, every step handles its own parsing (encapsulation). Moreover, the controller code won't change if some step will be changed.
  • the controller also saves an 'in progress' state together with the current step until the protocol ends (either it is really ended or the user asks explicitly to exit/abort). This allows recovering from abnormal termination, e.g. drained battery.
  • Step 1060 Each type of step defined in the protocol has a corresponding data structure.
  • Step types All activities (Step types) inherit from the Step abstract type which contains the common data.
  • Compliance 1070 One of the biggest problems solved by the product of the present invention is the absence of automatic compliance and validation in prevailing maintenance practices.
  • the responsibility of the Compliance module 1070 is to log every step in the protocol execution, save important data defined by the protocol, supply
  • the compliance module requirements are:
  • the module can operate in two modes - Step-mode or Protocol-mode.
  • Protocol mode (default), all the data is collected during the execution of the protocol and kept locally. Only when the protocol ends, the data is packed into a single report file and uploaded to the server. If no on-line connectivity exists, the uploading is postponed. The application stores all the data in the glass local storage, and a log of all the reports waiting for upload. Upload will be done when there will be connectivity. Step-mode on the other hand, requires online connectivity. The module uploads every step result, once the step is completed (or fails). This may help when remote monitoring is needed.
  • the report file must have a structure which will give strong and clear evidence to the maintenance procedure.
  • Network Controller 1080 This module monitors the network state and holds the current state (Online/Offline). It may hold additional information related to the network, for example, if the system server is accessible or the connection method (Wi-Fi/ Bluetooth).
  • the module listens to any change in the network and updates the state.
  • Fig.15 is a flowchart showing the general steps taken by an exemplary equipment maintenance smartglasses application.
  • the equipment is identified by the application. Identification may be achieved in various ways, such as for example:
  • step 1 1 10 the maintenance protocol for the identified equipment is downloaded from the server repository to the smartglasses.
  • the equipment identification step may not be necessary if the smartglasses have been preloaded with the equipment protocol.
  • the next instruction is read from the protocol.
  • the instruction may be an instruction to be performed by the smartglasses, e.g. "Take picture” 1 140 of the equipment or part thereof or "Read value” 1 130 off a gauge or panel.
  • the output of these instructions (e.g. value read) may be analyzed by the smartglasses application to determine the next instruction to be
  • the instruction may be a textual instruction displayed to the user 1 120.
  • step 1 150 the smartglasses application may upload feedback to the server.
  • the feedback may comprise for example: - Acknowledgement of receiving the instruction;
  • the feedback type may be selected from a menu.
  • the system may provide the technician with online help by connecting him to an appropriate expert, or by querying the historical troubleshooting database for similar reported problems and their solutions.
  • the system's historical database is updated 1 170 with pertinent data such as for example:
  • the system server 1 10 is a critical node in the system of the present invention. It is responsible for executing a group of services (applications), which live side by side and can interact with each other, and together establish the overall server solution.
  • applications applications
  • the server application is combined of the following main services:
  • Fig. 16 is a schematic block diagram showing the server application modules and connectivity.
  • Glass Service 1220 This web service is responsible for communicating with the SmartGlass application.
  • the service address is known to the SmartGlass application which can initiate a request or post any update through this channel. Every request arriving from the SmartGlass application is first authenticated by this module. If the glass user is known, the request is handled according to its type.
  • GET EXTERNAL DATA A request to get external data. A known label will be supplied as input.
  • the data can be data stored in the Database or a request from external interface.
  • This service is responsible for maintaining the various protocols. It can receive new or updated protocols, assets and components from the Studio application and allows the Studio to use them and also the Glass application to download the published protocols.
  • Another major responsibility of this module is to analyze the various reports according to pre-defined criteria and allow easy exporting of the results to any external "customer”.
  • Some protocols may require some negotiation with external entities like ERP or SCADA.
  • Front end 1250 is responsible for communicating with the various entities.
  • This module is the web access point to the server application.
  • the module allows the user (manager, supervisor) higher level services which are based on the data located in the database.
  • the module supports the following exemplary functionalities:
  • Studio 125 The Studio is a standalone application which allows any user to produce equipment maintenance protocols without any programming knowledge.
  • the Studio is a WYSIWYG ("What You See Is What You Get") protocol editor.
  • the Company XYZ has 10 conveyers. Each conveyer consists of AC Motor, Belt, Gear and Controller. The field service should be performed for each conveyer and it includes lubrication of gears, checking motor performance and fastening the belt. The following process describes how this procedure is implemented in the system. 1 .
  • the user logs-in to the studio and goes to Components definitions and defines 4 components (Motor, Belt, Gear, and Controller). For Each component the user defines a set of actions. These actions represent steps in the smartglass protocol. Below are a few examples of actions: Motor->Measure Startup current
  • Controller component Assuming that a Controller component was already defined in the system, it can be reused.
  • the controller component has two possible actions:
  • the user can declare the Conveyer Class or template by selecting the components.
  • the user defines general attributes of the Conveyer (Name, Description) and defines collection of components which compose the Conveyer Template.
  • Protocol Template Definition Once the Asset template is defined, the user can construct the protocol by arranging the sequence of actions defined for each asset components. For example, the following protocol can be defined:
  • Asset Object Creation After the Asset Template and Protocols are declared, the user can create physical objects using this template. The user should fill in specific attributes: location, serial number, contact person and can check and modify the protocols inherited from the template.
  • Example of the Asset protocol :
  • Example - The user logs in to the web server application, fills (once) his details.
  • the application stores the user credentials and from that moment it can interact with him.
  • the user may receive alerts (push notifications) at all times, even when he is not using the application.
  • Example - The user is notified by a static card in his timeline that there is a work order /new Glass version / Message waiting for him. User actions - The card has some fixed menu options: "Accept”, “Delete”, “Get Protocol”, “Update Application” according to the notification.
  • the OTSC workflow enables collaboration between a field technician and other authorized advisers in the organization (e.g. supervisor or expert).
  • OTSC is used when a field technician requires support in the course of his repair or maintenance work.
  • the support may be received by sharing visual information with the adviser and receiving Augmented Reality (AR) feedback.
  • AR Augmented Reality
  • the OTSC workflow is implemented as a technician application running on both smartphone and smartglasses and an adviser application running on a tablet, PC, smartphone etc.
  • Fig. 17 is a flowchart showing the OTSC process.
  • the technician starts the OTSC application on his smartphone or (on his smartglasses). The reason for using the smartphone in this stage is that the smartphone is readily available for use, while the smartglasses have to be started and worn.
  • the technician captures the equipment or a part thereof to be sent to the adviser.
  • the OTSC application provides annotation tools so that the technician may add textual description and graphic markers on top of the captured image, to highlight the problem.
  • the technician may also provide text describing the problem.
  • the captured image plus annotations and text is distributed by the system server to all the available relevant advisers or to a selected group of advisers. The advisers have been previously defined and saved in the system server's database according to region, site and equipment category.
  • step 1730 the technician who now waits for feedback wears his
  • step 1740 one of the approached advisers confirms his availability for an OTSC session and in step 1750 a connection is established between the technician and the adviser. Once the connection is established, they can exchange information and perform different actions using the adviser application 190.
  • Fig. 18 is a schematic block diagram of the various modules of adviser application 190:
  • Picture strip 1800 - A space where all the pictures captured in the current session are displayed.
  • the pictures may be marked by events such as "after sending Change POV command” or "after sending annotations”.
  • Annotation editor 1810 Enables the adviser to edit pictures (e.g. crop) and add textual annotations, markers, data etc.
  • AR scene repository 1820 - Adviser may save the annotation elements and the pictures as an AR scene and provide names, description and tags to these scenes.
  • the actions enabled between the technician and the adviser are, for example:
  • the annotation editor 1810 enables the adviser to do the following exemplary actions: Action Description
  • AR Layer Can add the follwoing annotations and AR elements:
  • measurement data is received, it is dispayed in the appropriuate list of measurements.
  • the advisor can login to the AR scenes repository 1820 and look for
  • the scenes in the repository 1820 may be sorted and filtered by equipment name, component name, tag etc.
  • Fig. 19 shows the process of OTSC on the technician's side, including:
  • Transmission mode 1900 in which: a. The technician walks around the equipment site and the smartglasses
  • the technician may aaugent it by lowering the
  • Viewing mode 1910 in which: a. The digital content created by the adviser is displayed to the technician on the smartglasses in combination with the physical content.
  • the technician may 'browse' additional digital content sent by the
  • the technician may query the server for historical data which is related to the specific equipment and to similar equipments on other sites. Search query can be influenced by his current location (GPS) , the type of system he is working on, model name, customer name and other relevant attributes.
  • GPS current location
  • the technician may select from a number of viewing screens, each holding different digital content for the same picture or for different pictures.
  • Archive mode 1920 in which the technician may grade his satisfaction from the extent to which the digital content has helped him solve the problem or perform the task. The grading is kept in the archive for the benefit of future technicians.
  • Fig. 20 shows the process of OTSC on the adviser's side, including: Receiving mode 2000 in which: a. The adviser receives on his tablet or smartphone a succession of
  • the adviser selects the pictures on which he wishes to focus and
  • the adviser may remotely stop the transmission mode of the technician's smartglasses.
  • Digital content mode 2010 in which: a.
  • the adviser uploads digital content on the selected picture: stock drawings, free hand drawings explanatory text, hyperlink to a page in the operator's manual, hyperlink to the exact place in the electric scheme, hyperlink to other pictures in the archive etc.
  • the adviser may create a number of digital content layers for the same pictures.
  • the system transmits the digital content to the technician, along with the serial number of the relevant picture.
  • Fig. 21 shows a picture 2100 sent 2000 from the technician's smartglasses to the adviser, displayed in the adviser application 190 screen, annotated 21 10 by the adviser and sent back 2120 to the technician to be displayed in optical see-through mode on the technician's smartglasses 2130.
  • the adviser's annotations are displayed over the actual scene seen by the technician.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • General Engineering & Computer Science (AREA)
  • Optics & Photonics (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

La présente invention concerne un système destiné à faciliter la maintenance d'équipement, comprenant : un serveur configuré pour mémoriser des configurations d'équipement et des protocoles de maintenance ; au moins une installation de création de protocoles de maintenance d'équipement communiquant avec le serveur, l'installation de création de protocoles étant configurée pour créer automatiquement des protocoles de maintenance pour un équipement comprenant des éléments prédéfinis ; et une pluralité de lunettes intelligentes pouvant être portées par des techniciens itinérants, les lunettes intelligentes exécutant une application de lunettes intelligentes configurée pour exécuter des protocoles de maintenance d'équipement.
PCT/IB2015/051153 2014-02-19 2015-02-17 Système et procédé destinés à faciliter la maintenance d'équipement à l'aide de lunettes intelligentes WO2015125066A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US201461941514P 2014-02-19 2014-02-19
US61/941,514 2014-02-19

Publications (1)

Publication Number Publication Date
WO2015125066A1 true WO2015125066A1 (fr) 2015-08-27

Family

ID=53877684

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/IB2015/051153 WO2015125066A1 (fr) 2014-02-19 2015-02-17 Système et procédé destinés à faciliter la maintenance d'équipement à l'aide de lunettes intelligentes

Country Status (1)

Country Link
WO (1) WO2015125066A1 (fr)

Cited By (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
ES2621929A1 (es) * 2016-08-31 2017-07-05 Mikel Aingeru ARDANAZ JIMENEZ Método y sistema de entrenamiento deportivo.
EP3206121A1 (fr) * 2016-02-09 2017-08-16 Nokia Technologies Oy Procédés et appareils relatifs à la manipulation d'un contenu visuel de réalité virtuelle
WO2018072899A1 (fr) * 2016-10-20 2018-04-26 Krones Ag Dispositif de terminal informatique et procédé de création de supports de documentation numériques pour une machine de l'industrie alimentaire
EP3584692A1 (fr) * 2018-06-19 2019-12-25 InterDigital CE Patent Holdings Partage de contenu virtuel dans une scène de réalité mixte
BE1026463B1 (fr) * 2018-07-10 2020-02-10 Safran Aero Boosters Sa Système de maintenance pour équipement aéronautique
EP3506192A4 (fr) * 2016-08-16 2020-03-18 Daesung Industrial Co., Ltd Procédé de gestion intégrée à base de vidéo pour la construction et la maintenance de bâtiments
US10597254B2 (en) 2017-03-30 2020-03-24 Otis Elevator Company Automated conveyance system maintenance
US11094220B2 (en) 2018-10-23 2021-08-17 International Business Machines Corporation Intelligent augmented reality for technical support engineers
US11105528B2 (en) 2017-11-15 2021-08-31 Johnson Controls Tyco IP Holdings LLP Building management system with automatic synchronization of point read frequency
CN113741687A (zh) * 2021-08-10 2021-12-03 广东工业大学 基于ar眼镜的工业空调控制通讯方法、系统及存储介质

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20020067372A1 (en) * 1999-03-02 2002-06-06 Wolfgang Friedrich Utilizing augmented reality-based technologies to provide situation-related assistance to a skilled operator from remote experts
US20070273610A1 (en) * 2006-05-26 2007-11-29 Itt Manufacturing Enterprises, Inc. System and method to display maintenance and operational instructions of an apparatus using augmented reality

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20020067372A1 (en) * 1999-03-02 2002-06-06 Wolfgang Friedrich Utilizing augmented reality-based technologies to provide situation-related assistance to a skilled operator from remote experts
US20070273610A1 (en) * 2006-05-26 2007-11-29 Itt Manufacturing Enterprises, Inc. System and method to display maintenance and operational instructions of an apparatus using augmented reality

Cited By (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3206121A1 (fr) * 2016-02-09 2017-08-16 Nokia Technologies Oy Procédés et appareils relatifs à la manipulation d'un contenu visuel de réalité virtuelle
WO2017137658A1 (fr) * 2016-02-09 2017-08-17 Nokia Technologies Oy Procédés et appareils concernant la gestion de contenu de la réalité virtuelle visuelle
US11347302B2 (en) 2016-02-09 2022-05-31 Nokia Technologies Oy Methods and apparatuses relating to the handling of visual virtual reality content
EP3506192A4 (fr) * 2016-08-16 2020-03-18 Daesung Industrial Co., Ltd Procédé de gestion intégrée à base de vidéo pour la construction et la maintenance de bâtiments
ES2621929A1 (es) * 2016-08-31 2017-07-05 Mikel Aingeru ARDANAZ JIMENEZ Método y sistema de entrenamiento deportivo.
WO2018072899A1 (fr) * 2016-10-20 2018-04-26 Krones Ag Dispositif de terminal informatique et procédé de création de supports de documentation numériques pour une machine de l'industrie alimentaire
US10597254B2 (en) 2017-03-30 2020-03-24 Otis Elevator Company Automated conveyance system maintenance
US11105528B2 (en) 2017-11-15 2021-08-31 Johnson Controls Tyco IP Holdings LLP Building management system with automatic synchronization of point read frequency
EP3584692A1 (fr) * 2018-06-19 2019-12-25 InterDigital CE Patent Holdings Partage de contenu virtuel dans une scène de réalité mixte
WO2019243160A1 (fr) * 2018-06-19 2019-12-26 Interdigital Ce Patent Holdings Partage de contenu virtuel dans une scène de réalité mixte
US11443492B2 (en) 2018-06-19 2022-09-13 Interdigital Ce Patent Holdings Sharing virtual content in a mixed reality scene
US11651576B2 (en) 2018-06-19 2023-05-16 Interdigital Ce Patent Holdings Sharing virtual content in a mixed reality scene
EP3599453A3 (fr) * 2018-07-10 2020-05-13 Safran Aero Boosters SA Banc d'essai de turbomachine
BE1026463B1 (fr) * 2018-07-10 2020-02-10 Safran Aero Boosters Sa Système de maintenance pour équipement aéronautique
US11094220B2 (en) 2018-10-23 2021-08-17 International Business Machines Corporation Intelligent augmented reality for technical support engineers
CN113741687A (zh) * 2021-08-10 2021-12-03 广东工业大学 基于ar眼镜的工业空调控制通讯方法、系统及存储介质

Similar Documents

Publication Publication Date Title
WO2015125066A1 (fr) Système et procédé destinés à faciliter la maintenance d'équipement à l'aide de lunettes intelligentes
US10878634B2 (en) Methods for augmented reality applications
Klinker et al. Augmented maintenance of powerplants: A prototyping case study of a mobile AR system
US9407861B2 (en) User interface tool kit for mobile devices
US10534584B2 (en) Enhanced software application ecosystem
US10289390B2 (en) Interactive multimodal display platform
CN1573762A (zh) 维护与检查系统和方法
US20160349978A1 (en) Knowledge base studio
EP2891946A1 (fr) Procédé d'interaction et dispositif d'interaction permettant d'intégrer la technologie de réalité augmentée et des données en masse
US20220301305A1 (en) Augmented Reality Support Platform
US20140365946A1 (en) Systems and methods for operating and managing enterprise systems on a mobile electronic device
US11380434B2 (en) Telehealth platform
US20180349932A1 (en) Methods and systems for determining persona of participants by the participant use of a software product
US11978018B2 (en) Project management system with capture review transmission function and method thereof
US11721075B2 (en) Systems and methods for integrating and using augmented reality technologies
US20200250810A1 (en) Reporting Method
Marreiros et al. From kpi dashboards to advanced visualization
US10796108B2 (en) Integrated machine information management with application interactive user interface
KR102355791B1 (ko) 화면 요구사항 스펙 작성 도구
US20240169286A1 (en) Video markup system
US20230367286A1 (en) Intelligent cognitive assistant system and method
US20210407109A1 (en) Visual product identification
George Using Object Recognition on Hololens 2 for Assembly
JIANG A Novel Augmented Reality Based System for Providing Maintenance Assistance
Preda et al. Augmented Reality Training in Manufacturing Sectors

Legal Events

Date Code Title Description
DPE2 Request for preliminary examination filed before expiration of 19th month from priority date (pct application filed from 20040101)
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 15751888

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 15751888

Country of ref document: EP

Kind code of ref document: A1