CN115437507B - AR (augmented reality) -glasses-based equipment maintenance guiding method and system and related equipment - Google Patents

AR (augmented reality) -glasses-based equipment maintenance guiding method and system and related equipment Download PDF

Info

Publication number
CN115437507B
CN115437507B CN202211384772.0A CN202211384772A CN115437507B CN 115437507 B CN115437507 B CN 115437507B CN 202211384772 A CN202211384772 A CN 202211384772A CN 115437507 B CN115437507 B CN 115437507B
Authority
CN
China
Prior art keywords
target object
equipment
model
information
maintenance
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202211384772.0A
Other languages
Chinese (zh)
Other versions
CN115437507A (en
Inventor
杨之乐
刘祥飞
饶建波
郭媛君
谭勇
朱俊丞
余发国
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Zhongke Hangmai CNC Software Shenzhen Co Ltd
Original Assignee
Zhongke Hangmai CNC Software Shenzhen Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Zhongke Hangmai CNC Software Shenzhen Co Ltd filed Critical Zhongke Hangmai CNC Software Shenzhen Co Ltd
Priority to CN202211384772.0A priority Critical patent/CN115437507B/en
Publication of CN115437507A publication Critical patent/CN115437507A/en
Application granted granted Critical
Publication of CN115437507B publication Critical patent/CN115437507B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/20—Administration of product repair or maintenance
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/20—Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/20—Scenes; Scene-specific elements in augmented reality scenes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
    • G06F2203/01—Indexing scheme relating to G06F3/01
    • G06F2203/012—Walk-in-place systems for allowing a user to walk in a virtual environment while constraining him to a given position in the physical environment

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Business, Economics & Management (AREA)
  • Human Resources & Organizations (AREA)
  • Marketing (AREA)
  • Tourism & Hospitality (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Human Computer Interaction (AREA)
  • Operations Research (AREA)
  • Quality & Reliability (AREA)
  • Strategic Management (AREA)
  • Economics (AREA)
  • General Business, Economics & Management (AREA)
  • Architecture (AREA)
  • Computer Graphics (AREA)
  • Computer Hardware Design (AREA)
  • Software Systems (AREA)
  • Multimedia (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

The invention discloses an AR glasses-based equipment maintenance guiding method, an AR glasses-based equipment maintenance guiding system and related equipment, wherein the method comprises the following steps: scanning and obtaining equipment to be maintained through AR glasses corresponding to the first target object; acquiring static information and a three-dimensional virtual model corresponding to equipment to be maintained; establishing a communication channel for the first target object and the second target object when a remote contact instruction input by the first target object is received; acquiring dynamic information and description information corresponding to equipment to be maintained, and transmitting static information, a three-dimensional virtual model, dynamic information and description information corresponding to the equipment to be maintained to a second target object through a communication channel to trigger the second target object to return to a target maintenance guidance scheme; and acquiring a target maintenance guidance scheme, and outputting the target maintenance guidance scheme to the first target object through AR glasses to trigger the first target object to maintain the equipment to be maintained according to the target maintenance guidance scheme. The scheme of the invention is beneficial to improving the efficiency and the effect of equipment maintenance.

Description

AR (augmented reality) -glasses-based equipment maintenance guiding method and system and related equipment
Technical Field
The invention relates to the technical field of equipment maintenance guidance, in particular to an AR (augmented reality) glasses-based equipment maintenance guidance method, an AR glasses-based equipment maintenance guidance system and related equipment.
Background
With the development of science and technology and the wide application of various devices, more and more attention is paid to the maintenance of the devices. The equipment maintenance is to maintain and repair the equipment, specifically, the equipment which may have a fault is checked by a technical means and the fault reason is found out, so that the maintenance is performed according to the fault reason.
In the prior art, a maintenance person usually goes to the field to actually maintain the equipment, so that the maintenance process only depends on the personal experience of the field maintenance person, the field maintenance person is required to independently identify and independently maintain each fault, and the maintenance effect (such as whether the maintenance of all faults existing in the equipment can be well completed) and the efficiency (i.e. the rate of completing the fault maintenance) are limited by the experience of the maintenance person. The problem of the prior art is that the equipment maintenance can only depend on the personal experience of field maintenance personnel, which is not beneficial to improving the efficiency and the effect of the equipment maintenance.
Thus, there is still a need for improvement and development of the prior art.
Disclosure of Invention
The invention mainly aims to provide an AR (augmented reality) glasses-based equipment maintenance guiding method, an AR glasses-based equipment maintenance guiding system and related equipment, and aims to solve the problems that in the prior art, equipment maintenance can only depend on personal experience of field maintenance personnel, and the efficiency and the effect of equipment maintenance are not improved.
In order to achieve the above object, a first aspect of the present invention provides an AR glasses-based device maintenance guiding method, wherein the AR glasses-based device maintenance guiding method includes:
scanning and acquiring equipment to be maintained through AR glasses corresponding to the first target object;
acquiring static information and a three-dimensional virtual model corresponding to the equipment to be maintained, and outputting the static information and the three-dimensional virtual model to the first target object through the AR glasses, wherein the static information comprises an equipment model, an equipment type and equipment safe operation limiting information;
when a remote contact instruction input by the first target object is received, establishing a communication channel for the first target object and a second target object;
acquiring dynamic information and description information corresponding to the equipment to be maintained, and transmitting static information, a three-dimensional virtual model, dynamic information and description information corresponding to the equipment to be maintained to the second target object through the communication channel to trigger the second target object to return a target maintenance guidance scheme, wherein the dynamic information comprises an equipment image and/or an equipment video, and the description information comprises text information and/or voice information which is input by the first target object and used for describing the fault state of the equipment to be maintained;
and acquiring a target maintenance guidance scheme returned by the second target object through the communication channel, and outputting the target maintenance guidance scheme to the first target object through the AR glasses to trigger the first target object to maintain the equipment to be maintained according to the target maintenance guidance scheme.
Optionally, before the obtaining of the static information and the three-dimensional virtual model corresponding to the device to be maintained, the method further includes:
inputting the static information of each device to be scanned in the area to be scanned into the AR glasses;
constructing a three-dimensional visual model of each device to be scanned through preset three-dimensional model processing software, respectively labeling information of parts of each three-dimensional visual model, rendering the labeled three-dimensional visual model to obtain a three-dimensional virtual model corresponding to each device to be scanned, and storing each three-dimensional virtual model into the AR glasses;
wherein, the equipment to be maintained is the equipment to be scanned in the area to be scanned.
Optionally, the obtaining of the static information and the three-dimensional virtual model corresponding to the device to be maintained includes:
when the AR glasses scan the equipment to be maintained, acquiring a plurality of identification images corresponding to the equipment to be maintained, inputting the plurality of identification images into a trained equipment identification model and acquiring an equipment model corresponding to the equipment to be maintained, wherein the acquisition angles corresponding to the identification images in the plurality of identification images are different;
and searching and obtaining a three-dimensional visual model, equipment type and equipment safe operation limiting information corresponding to the equipment to be maintained from the AR glasses according to the equipment model.
Optionally, the establishing a communication channel for the first target object and the second target object when receiving the remote contact instruction input by the first target object includes:
when a remote contact instruction input by the first target object is received, acquiring the online state of each remote guidance object according to a preset remote object library;
determining at least one second target object from the remote object library according to the device model, the device type and the online status of the remote object;
and establishing a communication channel for the first target object and the second target object.
Optionally, the method further includes:
acquiring a model updating instruction in real time, updating the three-dimensional virtual model in real time according to the model updating instruction, and synchronously displaying the updated three-dimensional virtual model to the first target object and the second target object;
the model updating instruction comprises model operation instructions input by the first target object and the second target object, the model operation instructions comprise a disassembling instruction, a dragging instruction, a combining instruction and a labeling instruction, and the model operation instructions are input by the first target object through the AR glasses in a gesture control and/or voice control mode.
Optionally, the communication channel includes an equipment information communication channel, a guidance scheme communication channel, and a model operation communication channel that are independent of each other;
the equipment information communication channel is used for transmitting static information, dynamic information and description information corresponding to the equipment to be maintained;
the guidance scheme communication channel is used for transmitting the target maintenance guidance scheme;
the model operation communication channel is used for transmitting the model updating command.
Optionally, the method further includes:
when the first target object maintains the equipment to be maintained according to a target step in the target maintenance guide scheme, acquiring a maintenance operation image of the first target object, sending the maintenance operation image to the second target object and/or a trained operation judgment model to trigger the second target object and/or the trained operation judgment model to perform misoperation judgment, wherein the trained operation judgment model is used for performing feature recognition on the maintenance operation image to acquire an operation feature, and comparing the operation feature with a step feature corresponding to the target step to perform misoperation judgment;
when receiving misoperation indicating information, carrying out misoperation warning prompt on the first target object, and limiting the AR glasses not to output the steps after the target step in the target maintenance guidance scheme until the first target object corrects the misoperation corresponding to the target step;
the misoperation instruction information is either one of model determination information acquired by the trained operation determination model and guidance object determination information input by the second target object.
Optionally, the method further includes:
when misoperation indicating information is received, acquiring an error grade corresponding to the misoperation indicating information according to the misoperation indicating information, and recording misoperation duration time corresponding to the target step;
and when the error grade exceeds a preset grade threshold or the misoperation duration time exceeds a preset time threshold, locking the maintenance operation permission of the first target object, and sending permission locking alarm information to preset mobile equipment.
A second aspect of the present invention provides an AR glasses-based equipment maintenance guidance system, wherein the AR glasses-based equipment maintenance guidance system includes:
the device scanning module is used for scanning and acquiring the device to be maintained through the AR glasses corresponding to the first target object;
an information obtaining module, configured to obtain static information and a three-dimensional virtual model corresponding to the device to be maintained, and output the static information and the three-dimensional virtual model to the first target object through the AR glasses, where the static information includes a device model, a device type, and device safe operation restriction information;
the communication module is used for establishing a communication channel for the first target object and the second target object when receiving a remote contact instruction input by the first target object;
an information processing module, configured to obtain dynamic information and description information corresponding to the to-be-maintained device, and transmit static information, a three-dimensional virtual model, dynamic information, and description information corresponding to the to-be-maintained device to the second target object through the communication channel to trigger the second target object to return to a target maintenance guidance scheme, where the dynamic information includes a device image and/or a device video, and the description information includes text information and/or voice information that is input by the first target object and used for describing a fault state of the to-be-maintained device;
and the maintenance guidance module is used for acquiring a target maintenance guidance scheme returned by the second target object through the communication channel, and outputting the target maintenance guidance scheme to the first target object through the AR glasses to trigger the first target object to maintain the equipment to be maintained according to the target maintenance guidance scheme.
A third aspect of the present invention provides an intelligent terminal, where the intelligent terminal includes a memory, a processor, and an AR glasses-based device maintenance guidance program stored in the memory and executable on the processor, and the AR glasses-based device maintenance guidance program implements any one of the AR glasses-based device maintenance guidance methods when executed by the processor.
As can be seen from the above, in the scheme of the present invention, the device to be maintained is obtained by scanning the AR glasses corresponding to the first target object; obtaining static information and a three-dimensional virtual model corresponding to the equipment to be maintained, and outputting the static information and the three-dimensional virtual model to the first target object through the AR glasses, wherein the static information comprises equipment model, equipment type and equipment safe operation limiting information; when a remote contact instruction input by the first target object is received, establishing a communication channel for the first target object and a second target object; acquiring dynamic information and description information corresponding to the equipment to be maintained, and transmitting static information, a three-dimensional virtual model, dynamic information and description information corresponding to the equipment to be maintained to the second target object through the communication channel to trigger the second target object to return a target maintenance guidance scheme, wherein the dynamic information comprises an equipment image and/or an equipment video, and the description information comprises text information and/or voice information which is input by the first target object and used for describing the fault state of the equipment to be maintained; and acquiring a target maintenance guidance scheme returned by the second target object through the communication channel, and outputting the target maintenance guidance scheme to the first target object through the AR glasses to trigger the first target object to maintain the equipment to be maintained according to the target maintenance guidance scheme.
Compared with the prior art, in the scheme of the invention, based on the AR technology, the equipment to be maintained is obtained through the scanning of the AR glasses, so that the static information and the three-dimensional virtual model corresponding to the equipment to be scanned are obtained, and the static information and the three-dimensional virtual model are output to the first target object (namely, on-site maintenance personnel) through the AR glasses, so that the equipment maintenance of the first target object can be assisted, and the equipment maintenance efficiency is improved. Meanwhile, a communication channel between the first target object and the second target object (namely, a remote instructor) can be established according to a remote contact instruction input by the first target object, the relevant information of the equipment to be maintained is transmitted to the second target object through the communication channel, a target maintenance guidance scheme provided by the second target object is fed back to the first target object through the communication channel, and the target maintenance guidance scheme is output to the first target object through the AR glasses, so that the first target object is guided to carry out equipment maintenance. Therefore, the first target object can be combined with the guidance of the second target object to perform equipment maintenance, the equipment maintenance only depending on the experience of the first target object is avoided, and the efficiency and the effect of the equipment maintenance are improved.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the embodiments or the prior art descriptions will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive exercise.
Fig. 1 is a schematic flowchart of an AR glasses-based device maintenance guidance method according to an embodiment of the present invention;
FIG. 2 is a schematic structural diagram of an AR glasses-based equipment maintenance guidance system according to an embodiment of the present invention;
fig. 3 is a schematic block diagram of an internal structure of an intelligent terminal according to an embodiment of the present invention.
Detailed Description
In the following description, for purposes of explanation and not limitation, specific details are set forth, such as particular system structures, techniques, etc. in order to provide a thorough understanding of the embodiments of the invention. It will be apparent, however, to one skilled in the art that the present invention may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present invention with unnecessary detail.
It will be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It is also to be understood that the terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in the specification of the present invention and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
It should be further understood that the term "and/or" as used in this specification and the appended claims refers to any and all possible combinations of one or more of the associated listed items and includes such combinations.
As used in this specification and the appended claims, the term "if" may be interpreted contextually as "when …" or "upon" or "in response to a determination" or "in response to a classification". Similarly, the phrase "if it is determined" or "if it is classified to [ a described condition or event ]" may be interpreted contextually to mean "upon determining" or "in response to determining" or "upon classifying to [ a described condition or event ]" or "in response to classifying to [ a described condition or event ]".
The technical solutions in the embodiments of the present invention are clearly and completely described below with reference to the drawings of the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention, but the present invention may be practiced in other ways than those specifically described and will be readily apparent to those of ordinary skill in the art without departing from the spirit of the present invention, and therefore the present invention is not limited to the specific embodiments disclosed below.
With the development of science and technology and the wide application of various devices, the maintenance of the devices is also more and more emphasized. The equipment maintenance is to maintain and repair the equipment, specifically, the equipment which may have a fault is checked by a technical means and the fault reason is found out, so that the maintenance is performed according to the fault reason.
In the prior art, a maintenance person usually goes to the field to actually maintain the equipment, so that the maintenance process only depends on the personal experience of the field maintenance person, the field maintenance person is required to independently identify and independently maintain each fault, and the maintenance effect (such as whether the maintenance of all faults existing in the equipment can be well completed) and the efficiency (i.e. the rate of completing the fault maintenance) are limited by the experience of the maintenance person. The problem of the prior art is that the equipment maintenance can only depend on the personal experience of field maintenance personnel, which is not beneficial to improving the efficiency and the effect of the equipment maintenance. And the field maintenance personnel may have the condition of operation judgment errors or operation errors, so that the equipment may further break down or even be damaged due to the judgment errors or the operation errors when the equipment is maintained only by the field maintenance personnel, and the maintenance effect and the use safety of the equipment are influenced.
In order to solve at least one of the problems, in the scheme of the invention, the equipment to be maintained is obtained by scanning the AR glasses corresponding to the first target object; obtaining static information and a three-dimensional virtual model corresponding to the equipment to be maintained, and outputting the static information and the three-dimensional virtual model to the first target object through the AR glasses, wherein the static information comprises equipment model, equipment type and equipment safe operation limiting information; when a remote contact instruction input by the first target object is received, establishing a communication channel for the first target object and a second target object; acquiring dynamic information and description information corresponding to the equipment to be maintained, and transmitting static information, a three-dimensional virtual model, dynamic information and description information corresponding to the equipment to be maintained to the second target object through the communication channel to trigger the second target object to return a target maintenance guidance scheme, wherein the dynamic information comprises an equipment image and/or an equipment video, and the description information comprises text information and/or voice information which is input by the first target object and used for describing the fault state of the equipment to be maintained; and acquiring a target maintenance guidance scheme returned by the second target object through the communication channel, and outputting the target maintenance guidance scheme to the first target object through the AR glasses to trigger the first target object to maintain the equipment to be maintained according to the target maintenance guidance scheme.
Compared with the prior art, in the scheme of the invention, based on the AR technology, the equipment to be maintained is obtained through the scanning of the AR glasses, so that the static information and the three-dimensional virtual model corresponding to the equipment to be scanned are obtained, and the static information and the three-dimensional virtual model are output to the first target object (namely, on-site maintenance personnel) through the AR glasses, so that the equipment maintenance of the first target object can be assisted, and the equipment maintenance efficiency is improved. Meanwhile, a communication channel between the first target object and the second target object (namely, a remote instructor) can be established according to a remote contact instruction input by the first target object, the relevant information of the equipment to be maintained is transmitted to the second target object through the communication channel, a target maintenance instruction scheme provided by the second target object is fed back to the first target object through the communication channel, and the target maintenance instruction scheme is output to the first target object through AR glasses, so that the first target object is instructed to maintain the equipment. Therefore, the first target object can be combined with the guidance of the second target object to perform equipment maintenance, the equipment maintenance only depending on the experience of the first target object is avoided, and the efficiency and the effect of the equipment maintenance are improved.
Meanwhile, according to the AR glasses-based equipment maintenance guiding method, equipment maintenance can be carried out by combining opinions of the first target object for field maintenance and the second target object for remote guidance, misoperation and misjudgment caused by single personnel error in field maintenance can be avoided, and therefore the equipment maintenance effect and the equipment use safety can be further improved.
Exemplary method
As shown in fig. 1, an embodiment of the present invention provides an AR glasses-based device maintenance guiding method, and specifically, the method includes the following steps:
and S100, scanning and obtaining the equipment to be maintained through the AR glasses corresponding to the first target object.
The first target object is a person or an intelligent device (e.g., an intelligent robot) that performs field maintenance on the device to be maintained, and in this embodiment, the first target object is a field maintenance person, which is specifically described as an example. The AR glasses corresponding to the first target object are the AR glasses worn by the first target object, and it should be noted that, in this embodiment, the AR glasses are taken as an example for specific description, and other devices having similar functions to the AR glasses, such as an AR helmet and the like, may also be used in the actual use process, and are not specifically described here. The device to be maintained is determined by scanning of the AR glasses, and needs to be maintained, in an application scene, the first target object wears the AR glasses to scan a specified area (for example, a target area), and the AR glasses take the device identified by scanning as the device to be maintained.
Step S200, obtaining static information and a three-dimensional virtual model corresponding to the equipment to be maintained, and outputting the static information and the three-dimensional virtual model to the first target object through the AR glasses, wherein the static information comprises equipment model, equipment type and equipment safe operation limiting information.
The static information is information indicating that the device to be repaired is corresponding to a fixed state (unchanged with the device use state or the use time), and may include, for example, a device model, a device type, and device safe operation restriction information of the corresponding device. The device safe operation limiting information is preset information for limiting various parameter ranges of the device to be repaired in normal use, and may include, for example, a rated power range, a rated voltage range, a rated current range of the device, temperature limiting ranges corresponding to different components in the device, a rotation angle range of a component, and the like, which are not specifically limited herein.
In an application scenario, the static information may further include a device unique code of the corresponding device. The unique equipment code can be used for indicating the corresponding equipment identity, and the unique equipment code of each equipment is different, so that even for the equipment with the same equipment model, specific safe equipment operation limiting information can be set for different equipment in advance according to different actual use requirements, different use environments and other factors, a targeted maintenance guidance scheme is further determined, and the equipment maintenance effect is improved.
In this embodiment, the to-be-maintained equipment further has a corresponding three-dimensional virtual model, the three-dimensional virtual model is used for displaying a specific structure of the to-be-maintained equipment, and each component of the three-dimensional virtual model can be disassembled, dragged, combined and labeled, so that a user can better know the internal structure of the corresponding to-be-maintained equipment.
In this embodiment, before the obtaining of the static information and the three-dimensional virtual model corresponding to the device to be maintained, the method further includes: inputting the static information of each device to be scanned in the area to be scanned into the AR glasses; constructing a three-dimensional visual model of each device to be scanned through preset three-dimensional model processing software, respectively labeling information of parts of each three-dimensional visual model, rendering the labeled three-dimensional visual models to obtain three-dimensional virtual models corresponding to each device to be scanned, and storing each three-dimensional virtual model into the AR glasses; wherein the equipment to be maintained is the equipment to be scanned in the area to be scanned.
In this embodiment, maintenance guidance may be performed on all devices to be scanned in the region to be scanned according to the device maintenance guidance method based on the AR glasses. The area to be scanned is a predefined area which needs to be scanned and maintained by the equipment, and can be determined according to actual requirements. The equipment to be scanned comprises all numerical control machine tool equipment in the area to be scanned. It should be noted that, in the embodiment, a numerical control machine is taken as an example for description, and other types of devices may be included in an actual use process, which is not specifically limited herein.
The preset three-dimensional model processing software is preset software for constructing and processing a three-dimensional model, and may be, for example, unity software, or other software, which is not limited herein. In this embodiment, a user inputs corresponding static information in advance, unity software is used to build a three-dimensional visual model of the numerical control machine tool in advance, static information is labeled on each part of the model, the model is rendered, and the rendered virtual model of the machine tool is placed in the AR glasses. It should be noted that, in this embodiment, the static information and the three-dimensional virtual model are directly stored in the AR glasses, so that the static information and the three-dimensional virtual model can be searched and called more conveniently and quickly, and the improvement of the equipment maintenance efficiency is facilitated. In the practical application process, the static information and the three-dimensional virtual model can be stored or backed up in the cloud server to prevent loss, different users can conveniently find and call the virtual model at any time, and the AR glasses are not required to be worn.
In an application scenario, when a first target object is scanned by using AR glasses, an identification image of a device to be maintained is acquired first, then a device model and/or a device unique code corresponding to the device to be maintained are acquired according to the identification image, and then corresponding device type and device safety operation limiting information are searched in the AR glasses according to the device model and/or the device unique code.
Specifically, when the surface of the device is provided with a mark (e.g., a two-dimensional code, a barcode, etc.) corresponding to the device unique code, the device unique code in the identification image can be directly recognized, thereby determining the device model, the device type, and the device security operation restriction information.
In this embodiment, the step S200 specifically includes the following steps: when the AR glasses scan the equipment to be maintained, acquiring a plurality of identification images corresponding to the equipment to be maintained, inputting the plurality of identification images into a trained equipment identification model and acquiring an equipment model corresponding to the equipment to be maintained, wherein the acquisition angles corresponding to the identification images in the plurality of identification images are different; and searching and obtaining a three-dimensional visual model, equipment type and equipment safe operation limiting information corresponding to the equipment to be maintained from the AR glasses according to the equipment model.
When the AR glasses scan an equipment entity, voice prompt information can be sent out, the first target object is prompted to move in position so as to carry out multi-angle scanning and image acquisition on the equipment to be scanned, therefore, clearer and more comprehensive identification images can be obtained, and the equipment model identification efficiency and accuracy can be improved.
The trained device recognition model is a pre-trained neural network model for recognizing and acquiring a device model corresponding to a device in an image according to an input image.
The equipment recognition model is trained in advance according to the following steps: inputting a training image in first training data into an equipment recognition model, and acquiring a predicted equipment model of each training equipment in the training image through the equipment recognition model, wherein the first training data comprises a plurality of groups of training image groups, and each group of training image group comprises a training image and a marking equipment model; and adjusting model parameters of the equipment recognition model according to the predicted equipment model and the marked equipment model corresponding to the training image, and continuously executing the step of inputting the training image in the first training data into the equipment recognition model until a preset first training condition is met to obtain the trained equipment recognition model.
Each training device in the training image corresponds to one marking device model. The preset first training condition includes that a loss value corresponding to the device identification model is not less than a first loss threshold, or that the iterative training number corresponding to the device identification model is not less than a preset first iterative threshold, and may further include other conditions, which are not specifically limited herein. The corresponding loss value may be calculated through a preset first loss function, and the first loss threshold may be preset and adjusted according to an actual requirement, which is not specifically limited herein.
Step S300, when receiving a remote contact command input by the first target object, establishing a communication channel between the first target object and a second target object.
Wherein, the second target object is a remote instructor (i.e. a remote expert) or a person (or a smart robot) for performing remote instruction on the smart robot, that is, a person (or a smart robot) who does not perform equipment maintenance on site but can provide maintenance instruction for the first target object through a preset remote expert instruction system, an expert platform or a mobile device. The remote contact instruction is input by the first target object according to requirements, captured and processed through the AR glasses and obtained for the second target object (namely, the remote instructor). Specifically, the first target object may interact with the AR glasses, for example, perform voice interaction or gesture interaction, so as to input a corresponding instruction.
Specifically, when receiving a remote contact command input by the first target object, the method for establishing a communication channel between the first target object and the second target object includes: when a remote contact instruction input by the first target object is received, acquiring the online state of each remote guidance object according to a preset remote object library; determining at least one second target object from the remote object library according to the device model, the device type and the online status of the remote object; and establishing a communication channel for the first target object and the second target object.
The remote object library is a preset database and is used for storing information of all remote guidance objects which can provide guidance for equipment maintenance. The remote object library may be preset and adjusted in real time according to actual requirements, which is not limited herein.
In this embodiment, when a first target object (i.e., a field maintenance worker) encounters difficulty during maintenance, a second target object (i.e., a remote instructor) to be contacted may be determined from a preset remote object library, a communication channel is then established, an expert is called according to a remote expert instruction system through the communication channel, the remote expert instruction system integrates a voice interaction function, an image and video transmission function, and a spatial positioning and labeling function, and the remote object library may also be integrated into the remote expert instruction system and used to store information of multiple experts, such as names, technical grades, technical categories, contact manners, online states, adept device models, and the like, where the online states are used to reflect whether the corresponding second target object (i.e., an expert) is convenient to access and guide. Maintenance personal can carry out the question-answering and exchange of problem with the expert through voice interaction, video interaction's mode, and the expert can annotate the lathe part position in remote expert system, and the information of mark can be transmitted in maintenance personal's eyes. In an application scenario, the corresponding label may be labeled to the position of the corresponding device entity in the display screen of the AR glasses based on the AR technology, and the label has a function of spatial positioning, and when the glasses frame moves, the relative position between the label and the device entity in the display screen does not change. Specifically, the association correspondence between the label and the device entity may be established, the relative position of the label and the device entity in the picture may be recorded, and after the AR glasses move, the corresponding label is moved to keep the relative position of the label and the device entity unchanged in time.
In another application scenario, the label may be further labeled into a corresponding three-dimensional virtual model, so that the first target object may further know the structure and the maintenance scheme of the equipment to be maintained by combining with the three-dimensional virtual model. In the remote expert guidance system, the maintenance personnel and the experts can also communicate text information and transmit guidance maintenance files and pictures.
Specifically, in this embodiment, when the AR glasses recognize the gesture motion of the first target object (or recognize the voice) and generate the remote contact instruction after processing, the AR glasses connect to a preset remote object library (that is, establish a communication channel between the AR glasses and the remote object library) and acquire the online status of each remote guidance object in the remote object library. And quickly screening all remote guidance objects capable of processing the type according to the type of the current equipment to be maintained, then selecting all remote guidance objects with online states in all the remote guidance objects corresponding to the type, and further traversing to select (one or more) remote guidance objects with the same adequacy equipment model as the equipment model of the equipment to be maintained as second target objects. Therefore, the remote object library is firstly screened according to the equipment type and the online state, then traversed and matched according to the equipment model, and all remote guidance objects in the remote object library do not need to be traversed during each use, so that the screening rate of the second target object is favorably improved, and the equipment maintenance efficiency is further improved.
It should be noted that, in the actual use process, one or more second target objects may be selected, and guidance may be performed by combining with more expert opinions when a plurality of second target objects are selected, so as to improve the equipment maintenance effect. In this embodiment, an example of selecting one second target object is specifically described, but the second target object is not specifically limited.
Step S400, obtaining dynamic information and description information corresponding to the device to be maintained, and transmitting static information, a three-dimensional virtual model, dynamic information and description information corresponding to the device to be maintained to the second target object through the communication channel to trigger the second target object to return to a target maintenance guidance scheme, where the dynamic information includes a device image and/or a device video, and the description information includes text information and/or voice information that is input by the first target object and used for describing a fault state of the device to be maintained.
The dynamic information is running state information of the to-be-maintained device at the current time, which is acquired by collecting the to-be-maintained device, and may include a device image and/or a device video corresponding to the device, and may also include parameters of the to-be-maintained device at the current time, such as voltage, current, power, and component temperature, which are not specifically limited herein. Further, the dynamic information may further include a fault code (or a fault code for indicating a fault) corresponding to the device to be maintained, where the fault code may be acquired from a system of the device to be maintained or acquired after identifying display content on a display screen of the device to be maintained, and other acquisition manners may also be used, which are not specifically limited herein. Further, in this embodiment, the description information may include text information and voice information input by the first target object and used for describing the fault state of the device to be repaired.
And the second target object performs fault judgment and determines a corresponding maintenance scheme according to the received static information, the three-dimensional virtual model, the dynamic information and the description information, so as to generate (or input) a target maintenance guidance scheme for guiding the first target object to perform equipment fault maintenance.
Step S500, obtaining a target maintenance guidance scheme returned by the second target object through the communication channel, and outputting the target maintenance guidance scheme to the first target object through the AR glasses to trigger the first target object to maintain the device to be maintained according to the target maintenance guidance scheme.
In this embodiment, the target maintenance guidance scheme is composed of a plurality of steps arranged in sequence, and each step may be output for the first target object in sequence through the AR glasses, so that the first target object performs equipment maintenance according to guidance, and the equipment maintenance efficiency is improved.
It should be noted that, in this embodiment, the AR glasses may display, in addition to the steps in the target maintenance guidance scheme, the static information, the three-dimensional virtual model, the dynamic information, the fault code, and the like of the device to be maintained in real time, so that the first target object can timely know the information, the operating condition, and the fault condition of the device to be maintained, and can help the first target object to further know the structure of the device to be maintained by combining with the three-dimensional virtual model. For example, the first target object can further know the internal structure of the equipment to be repaired through dragging, disassembling, combining and the like aiming at the three-dimensional virtual model, so that the repair efficiency and effect are improved.
Further, in this embodiment, the method for guiding maintenance of equipment based on AR glasses further includes: acquiring a model updating instruction in real time, updating the three-dimensional virtual model in real time according to the model updating instruction, and synchronously displaying the updated three-dimensional virtual model to the first target object and the second target object; the model updating instruction comprises model operation instructions input by the first target object and the second target object, the model operation instructions comprise a disassembling instruction, a dragging instruction, a combining instruction and a labeling instruction, and the model operation instructions are input by the first target object through the AR glasses in a gesture control and/or voice control mode.
It should be noted that, in this embodiment, the corresponding three-dimensional virtual models are synchronously displayed in real time for the first target object and the second target object, and when the three-dimensional virtual models are updated, the three-dimensional virtual models are also synchronously updated and displayed, so that the three-dimensional virtual models seen by the first target object and the second target object are completely the same, so as to improve the communication efficiency and accuracy between the first target object and the second target object, and thus improve the equipment maintenance efficiency.
Specifically, when a maintenance person (i.e., a first target object) scans and maintains the machine tool at a maintenance site by using the AR glasses, and when the AR glasses identify the physical machine tool (i.e., the device to be maintained), the three-dimensional virtual model and the static information of the corresponding machine tool are displayed through the display screen of the AR glasses. In this case, the first target object may interact with the three-dimensional virtual model, for example, the AR glasses have functions of gesture recognition and voice recognition, and the operations such as disassembling, dragging, and combining the three-dimensional virtual model are performed by recognizing a gesture motion or a voice command of the first target object. After the first target object and the second target object establish a communication channel, the first target object and the second target object synchronously display the identical three-dimensional virtual model. The second target object may also input a corresponding model update instruction through the remote expert guidance system to control the three-dimensional virtual model, and a specific input mode of the second target object may be determined according to actual requirements, for example, typing input or gesture control input, which is not specifically limited herein.
Further, the communication channel comprises an equipment information communication channel, a guidance scheme communication channel and a model operation communication channel which are independent of each other; the equipment information communication channel is used for transmitting static information, dynamic information and description information corresponding to the equipment to be maintained; the guidance scheme communication channel is used for transmitting the target maintenance guidance scheme; the model operation communication channel is used for transmitting the model updating command.
In this embodiment, for the first target object and the second target object, three mutually independent device information communication channels, guidance scheme communication channels, and model operation communication channels are respectively established to transmit different data, so as to set up bandwidths and the like corresponding to different channels according to data transmission requirements, thereby ensuring data transmission quality, increasing data transmission speed, and meeting real-time requirements. Specifically, the device information communication channel is used for transmitting static information, dynamic information and description information corresponding to the device to be maintained, and the static information, the dynamic information and the description information may include a large number of videos and pictures which need to be continuously transmitted, so that the device information communication channel sets the highest bandwidth, but has a lower requirement on real-time performance, and thus the time delay can be set to be higher. The guidance scheme communication channel is used for transmitting the corresponding target maintenance guidance scheme, and in order to avoid the maintenance scheme from making mistakes due to data transmission mistakes, the data transmission safety and stability of the channel are set to be higher. The model operation communication channel is used for transmitting model updating instructions, which have higher requirements on real-time performance (to realize real-time updating), but have less data volume, so that the bandwidth can be less, but the time delay is set to be lower. Therefore, according to the data characteristics, the data importance and the data real-time requirement, the time delay and the bandwidth of each communication channel can be flexibly set and adjusted, the stability is maintained, the maintenance guidance effect is improved, and the equipment maintenance efficiency, the accuracy and the real-time performance are improved.
In another application scenario, the communication channel may include a text communication channel, a voice communication channel, and an image communication channel, and different communication channels are dedicated to processing data of corresponding types, so as to fully utilize the communication capability thereof and improve the data communication efficiency.
In this embodiment, the method for guiding maintenance of equipment based on AR glasses further includes: when the first target object maintains the equipment to be maintained according to a target step in the target maintenance guide scheme, acquiring a maintenance operation image of the first target object, sending the maintenance operation image to the second target object and/or a trained operation judgment model to trigger the second target object and/or the trained operation judgment model to perform misoperation judgment, wherein the trained operation judgment model is used for performing feature recognition on the maintenance operation image to acquire an operation feature, and comparing the operation feature with a step feature corresponding to the target step to perform misoperation judgment; when misoperation indicating information is received, carrying out misoperation warning prompting on the first target object, and limiting the AR glasses not to output the steps after the target step in the target maintenance guidance scheme until the first target object corrects the misoperation corresponding to the target step; wherein the misoperation instruction information is one of model judgment information acquired by the trained operation judgment model and guidance object judgment information input by the second target object.
The target step is any one of the target maintenance guidance plans, specifically, a current step being executed by the first target object. When the first target object is maintained according to the target steps, the AR glasses acquire maintenance operation images of the first target object in real time and are used for judging whether the operation is wrong or not. And when the operation is wrong, an alarm prompt is given, and the steps after the target step are not output, so that the next step cannot be carried out when the error of the first target object is not corrected, and the effect and the accuracy of equipment maintenance are improved.
The judgment of the misoperation according to the maintenance operation image may be performed by a trained operation judgment model set in advance, or may be directly judged by the second target object. Specifically, the trained operation determination model is a pre-trained neural network model, and is configured to extract motion features in the input maintenance operation image, match the motion features with a step description text input into the model, and determine that no misoperation exists if the matching is successful (for example, both represent the same motion), otherwise, determine that a misoperation exists.
The operation judgment model is trained in advance according to the following steps: inputting an operation training image and an operation step description text in second training data into an operation judgment model, acquiring a prediction step description text corresponding to the operation training image through the operation judgment model, and calculating prediction similarity between the prediction step description text and the corresponding operation step description text through the operation judgment model, wherein the second training data comprises a plurality of groups of training matching data sets, and each group of training matching data set comprises the operation training image, the operation step description text and the labeling similarity; and adjusting the model parameters of the operation judgment model according to the prediction similarity and the corresponding labeling similarity, and continuing to execute the step of inputting the operation training images and the operation step description text in the second training data into the operation judgment model until a preset second training condition is met, so as to obtain the trained operation judgment model.
Wherein, the value of the labeled similarity is 0 or 1,0 represents mismatch, and 1 represents match. The result output by the operation judgment model is a prediction similarity, and the prediction similarity is a match (at this time, the model judgment information is not the misoperation indicating information) when the prediction similarity is greater than or equal to a preset threshold (for example, 0.5 or 1), or a mismatch (at this time, the model judgment information is the misoperation indicating information). The preset second training condition includes that the loss value corresponding to the operation judgment model is not less than the second loss threshold, or the iterative training times corresponding to the operation judgment model is not less than the preset second iterative threshold, and may also include other conditions, which are not specifically limited herein. The corresponding loss value may be calculated by a preset second loss function, and the second loss threshold may be preset and adjusted according to actual requirements, which is not specifically limited herein.
Further, the method for guiding the maintenance of the equipment based on the AR glasses further includes: when misoperation indicating information is received, acquiring an error grade corresponding to the misoperation indicating information according to the misoperation indicating information, and recording misoperation duration time corresponding to the target step; and when the error grade exceeds a preset grade threshold or the misoperation duration time exceeds a preset time threshold, locking the maintenance operation authority of the first target object, and sending authority locking alarm information to preset mobile equipment.
Specifically, when the AR glasses receive the misoperation indication information, the error level corresponding to the current misoperation is determined, and specifically, the error level may be determined by looking up through a preset level table. And the time during which the malfunction lasted, i.e., the time that was not corrected, is recorded. When the misoperation level is too high or the misoperation level is not corrected for a long time, the first target object is considered to have insufficient capacity, the maintenance operation authority of the first target object is locked, the first target object is stopped to be maintained, authority locking alarm information is sent to preset mobile equipment (which can be preset and adjusted, for example, can be set as a mobile phone of a manager), and new maintenance personnel are timely reminded to be dispatched. So as to avoid the first target object from causing larger errors and improve the safety of the equipment use. Specifically, the AR glasses may further acquire the identity information of the first target object in a pupil identification manner and send the identity information to the preset mobile device together.
So, in this embodiment, based on AR glasses, combine functions such as speech recognition, image recognition, space location mark, gesture recognition, combine the guidance of long-range object to maintain equipment, be favorable to realizing the accurate maintenance of instructing of digit control machine tool, improve the work efficiency of lathe maintenance, reduce maintenance personal and expert's work load. Thereby the problem of maintenance inefficiency when solving current maintenance personal unilateral maintenance.
As can be seen from the above, in the solution of the present invention, based on the AR technology, the device to be maintained is scanned and acquired through the AR glasses, so as to acquire the static information and the three-dimensional virtual model corresponding to the device to be scanned, and the static information and the three-dimensional virtual model are output to the first target object (i.e., the field maintenance staff) through the AR glasses, which can assist the first target object in performing device maintenance, thereby improving the device maintenance efficiency. Meanwhile, a communication channel between the first target object and the second target object (namely, a remote instructor) can be established according to a remote contact instruction input by the first target object, the relevant information of the equipment to be maintained is transmitted to the second target object through the communication channel, a target maintenance instruction scheme provided by the second target object is fed back to the first target object through the communication channel, and the target maintenance instruction scheme is output to the first target object through AR glasses, so that the first target object is instructed to maintain the equipment. Therefore, the first target object can be combined with the guidance of the second target object to perform equipment maintenance, the equipment maintenance only depending on the experience of the first target object is avoided, and the efficiency and the effect of the equipment maintenance are improved.
Exemplary device
As shown in fig. 2, corresponding to the method for guiding the maintenance of the device based on the AR glasses, an embodiment of the present invention further provides a system for guiding the maintenance of the device based on the AR glasses, where the system for guiding the maintenance of the device based on the AR glasses includes:
the device scanning module 610 is configured to scan and obtain a device to be maintained through the AR glasses corresponding to the first target object;
an information obtaining module 620, configured to obtain static information and a three-dimensional virtual model corresponding to the device to be maintained, and output the static information and the three-dimensional virtual model to the first target object through the AR glasses, where the static information includes a device model, a device type, and device safe operation restriction information;
a communication module 630, configured to establish a communication channel between the first target object and a second target object when receiving a remote contact instruction input by the first target object;
an information processing module 640, configured to obtain dynamic information and description information corresponding to the to-be-maintained device, and transmit static information, a three-dimensional virtual model, dynamic information, and description information corresponding to the to-be-maintained device to the second target object through the communication channel to trigger the second target object to return to a target maintenance guidance scheme, where the dynamic information includes a device image and/or a device video, and the description information includes text information and/or voice information that is input by the first target object and used for describing a fault state of the to-be-maintained device;
and a maintenance guidance module 650, configured to obtain a target maintenance guidance scheme returned by the second target object through the communication channel, and output the target maintenance guidance scheme to the first target object through the AR glasses to trigger the first target object to maintain the device to be maintained according to the target maintenance guidance scheme.
Specifically, in this embodiment, the specific functions of the AR glasses-based device maintenance guidance system and the modules thereof may refer to the corresponding descriptions in the AR glasses-based device maintenance guidance method, which are not described herein again.
The division manner of each module of the AR glasses-based equipment maintenance guidance system is not exclusive, and is not particularly limited herein.
Based on the above embodiment, the present invention further provides an intelligent terminal, and a schematic block diagram thereof may be as shown in fig. 3. The intelligent terminal comprises a processor and a memory. The memorizer of the intelligent terminal comprises an AR glasses-based equipment maintenance guide program, and the memorizer provides an environment for the operation of the AR glasses-based equipment maintenance guide program. When being executed by a processor, the AR glasses-based equipment maintenance guiding program realizes the steps of any one of the AR glasses-based equipment maintenance guiding methods. It should be noted that the above-mentioned intelligent terminal may further include other functional modules or units, which are not specifically limited herein.
It will be understood by those skilled in the art that the block diagram shown in fig. 3 is only a block diagram of a part of the structure related to the solution of the present invention, and does not constitute a limitation of the intelligent terminal to which the solution of the present invention is applied, and in particular, the intelligent terminal may include more or less components than those shown in the figure, or combine some components, or have a different arrangement of components.
The embodiment of the present invention further provides a computer-readable storage medium, where an AR glasses-based device maintenance guidance program is stored in the computer-readable storage medium, and when executed by a processor, the AR glasses-based device maintenance guidance program implements the steps of any one of the AR glasses-based device maintenance guidance methods provided in the embodiments of the present invention.
It should be understood that, the sequence numbers of the steps in the foregoing embodiments do not imply an execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-mentioned division of the functional units and modules is illustrated, and in practical applications, the above-mentioned functions may be distributed as different functional units and modules according to needs, that is, the internal structure of the system may be divided into different functional units or modules to implement all or part of the above-mentioned functions. Each functional unit and module in the embodiments may be integrated in one processing unit, or each unit may exist alone physically, or two or more units are integrated in one unit, and the integrated unit may be implemented in a form of hardware, or in a form of software functional unit. In addition, specific names of the functional units and modules are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present invention. For the specific working processes of the units and modules in the system, reference may be made to the corresponding processes in the foregoing method embodiments, which are not described herein again.
In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and reference may be made to the related descriptions of other embodiments for parts that are not described or illustrated in a certain embodiment.
Those of ordinary skill in the art would appreciate that the elements and algorithm steps of the examples described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
In the embodiments provided in the present invention, it should be understood that the disclosed system/intelligent terminal and method can be implemented in other ways. For example, the above-described system/intelligent terminal embodiments are merely illustrative, and for example, the division of the above modules or units is only one logical function division, and the actual implementation may be implemented by another division, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed.
The integrated modules/units described above, if implemented in the form of software functional units and sold or used as separate products, may be stored in a computer readable storage medium. Based on such understanding, all or part of the flow in the method according to the embodiments of the present invention may also be implemented by a computer program, which may be stored in a computer-readable storage medium, and when the computer program is executed by a processor, the steps of the method embodiments described above may be implemented. Wherein the computer program comprises computer program code, which may be in the form of source code, object code, an executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the above-mentioned computer program code, recording medium, usb disk, removable hard disk, magnetic disk, optical disk, computer Memory, read-Only Memory (ROM), random Access Memory (RAM), electrical carrier wave signal, telecommunication signal, software distribution medium, etc. It should be noted that the contents of the computer-readable storage medium can be increased or decreased as required by the legislation and patent practice in the jurisdiction.
The above-mentioned embodiments are only used for illustrating the technical solutions of the present invention, and not for limiting the same; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those skilled in the art; the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not depart from the spirit and scope of the embodiments of the present invention, and they should be construed as being included therein.

Claims (6)

1. An AR glasses-based device maintenance guidance method, the method comprising:
scanning and obtaining equipment to be maintained through AR glasses corresponding to the first target object;
acquiring static information and a three-dimensional virtual model corresponding to the equipment to be maintained, and outputting the static information and the three-dimensional virtual model to the first target object through the AR glasses, wherein the static information comprises equipment model, equipment type and equipment safe operation limiting information;
when a remote contact instruction input by the first target object is received, establishing a communication channel between the first target object and a second target object;
acquiring dynamic information and description information corresponding to the equipment to be maintained, and transmitting static information, a three-dimensional virtual model, dynamic information and description information corresponding to the equipment to be maintained to the second target object through the communication channel to trigger the second target object to return a target maintenance guidance scheme, wherein the dynamic information comprises an equipment image and/or an equipment video, and the description information comprises text information and/or voice information which is input by the first target object and used for describing the fault state of the equipment to be maintained;
acquiring a target maintenance guidance scheme returned by the second target object through the communication channel, and outputting the target maintenance guidance scheme to the first target object through the AR glasses to trigger the first target object to maintain the equipment to be maintained according to the target maintenance guidance scheme;
acquiring a model updating instruction in real time, updating the three-dimensional virtual model in real time according to the model updating instruction, and synchronously displaying the updated three-dimensional virtual model to the first target object and the second target object; the model updating instruction comprises model operation instructions input by the first target object and the second target object, the model operation instructions comprise a disassembling instruction, a dragging instruction, a combining instruction and a labeling instruction, and the first target object inputs the model operation instructions in a gesture control and/or voice control mode through the AR glasses; the communication channel comprises an equipment information communication channel, a guidance scheme communication channel and a model operation communication channel which are independent of each other; the equipment information communication channel is used for transmitting static information, dynamic information and description information corresponding to the equipment to be maintained; the guide scheme communication channel is used for transmitting the target maintenance guide scheme; the model operation communication channel is used for transmitting the model updating instruction; the bandwidth of the equipment information communication channel is highest, the data transmission safety and stability of the guidance scheme communication channel are highest, and the time delay of the model operation communication channel is lowest;
when the first target object maintains the equipment to be maintained according to one target step in the target maintenance guide scheme, acquiring a maintenance operation image of the first target object, sending the maintenance operation image to the second target object and/or a trained operation judgment model to trigger the second target object and/or the trained operation judgment model to perform misoperation judgment, wherein the trained operation judgment model is used for performing feature recognition on the maintenance operation image to acquire operation features, and comparing the operation features with step features corresponding to the target steps to perform misoperation judgment;
when misoperation indicating information is received, carrying out misoperation warning prompt on the first target object, and limiting the AR glasses not to output the steps after the target step in the target maintenance guidance scheme until the first target object corrects the misoperation corresponding to the target step; wherein the misoperation instruction information is any one of model judgment information acquired by the trained operation judgment model and guidance object judgment information input by the second target object;
when misoperation indicating information is received, acquiring an error grade corresponding to the misoperation indicating information according to the misoperation indicating information, and recording misoperation duration time corresponding to the target step; when the error grade exceeds a preset grade threshold or the misoperation duration time exceeds a preset time threshold, locking the maintenance operation authority of the first target object, and sending authority locking alarm information to preset mobile equipment;
wherein the operation judgment model is pre-trained according to the following steps: inputting an operation training image and an operation step description text in second training data into an operation judgment model, acquiring a prediction step description text corresponding to the operation training image through the operation judgment model, and calculating prediction similarity between the prediction step description text and the corresponding operation step description text through the operation judgment model, wherein the second training data comprises a plurality of groups of training matching data sets, and each group of training matching data set comprises the operation training image, the operation step description text and the labeling similarity; adjusting the model parameters of the operation judgment model according to the prediction similarity and the corresponding labeling similarity, and continuously executing the step of inputting the operation training images and the operation step description texts in the second training data into the operation judgment model until a preset second training condition is met to obtain a trained operation judgment model; the value of the labeled similarity is 0 or 1,0 represents mismatching, 1 represents matching, and the operation judgment model outputs a prediction similarity which is matched when the predicted similarity is greater than or equal to a preset threshold value, otherwise, the prediction similarity is mismatching.
2. The AR glasses-based equipment maintenance guidance method according to claim 1, wherein before the obtaining of the static information and the three-dimensional virtual model corresponding to the equipment to be maintained, the method further comprises:
inputting the static information of each device to be scanned in the area to be scanned into the AR glasses;
constructing a three-dimensional visual model of each device to be scanned through preset three-dimensional model processing software, respectively carrying out information labeling on parts of each three-dimensional visual model, rendering the labeled three-dimensional visual models to obtain three-dimensional virtual models corresponding to each device to be scanned, and storing each three-dimensional virtual model into the AR glasses;
wherein the equipment to be maintained is the equipment to be scanned in the area to be scanned.
3. The AR glasses-based equipment maintenance guiding method according to claim 2, wherein the obtaining of the static information and the three-dimensional virtual model corresponding to the equipment to be maintained comprises:
when the AR glasses scan the equipment to be maintained, acquiring a plurality of identification images corresponding to the equipment to be maintained, inputting the plurality of identification images into a trained equipment identification model and acquiring an equipment model corresponding to the equipment to be maintained, wherein the acquisition angles corresponding to the identification images in the plurality of identification images are different;
and searching the three-dimensional visual model, the equipment type and the equipment safe operation limiting information corresponding to the equipment to be maintained from the AR glasses according to the equipment model.
4. The AR glasses-based device repair guidance method of claim 1, wherein the establishing a communication channel for the first target object and a second target object upon receiving a remote contact instruction input by the first target object comprises:
when a remote contact instruction input by the first target object is received, acquiring the online state of each remote guidance object according to a preset remote object library;
determining at least one second target object from the remote object library according to the device model, the device category and the online state of the remote object;
and establishing a communication channel for the first target object and the second target object.
5. An AR glasses-based equipment maintenance guidance system, the system comprising:
the device scanning module is used for scanning and acquiring the device to be maintained through the AR glasses corresponding to the first target object;
the information acquisition module is used for acquiring static information and a three-dimensional virtual model corresponding to the equipment to be maintained and outputting the static information and the three-dimensional virtual model to the first target object through the AR glasses, wherein the static information comprises equipment model, equipment type and equipment safe operation limiting information;
the communication module is used for establishing a communication channel for the first target object and the second target object when receiving a remote contact instruction input by the first target object;
the information processing module is used for acquiring dynamic information and description information corresponding to the equipment to be maintained, and transmitting static information, a three-dimensional virtual model, dynamic information and description information corresponding to the equipment to be maintained to the second target object through the communication channel so as to trigger the second target object to return a target maintenance guidance scheme, wherein the dynamic information comprises an equipment image and/or an equipment video, and the description information comprises text information and/or voice information which is input by the first target object and used for describing the fault state of the equipment to be maintained;
the maintenance guidance module is used for acquiring a target maintenance guidance scheme returned by the second target object through the communication channel, and outputting the target maintenance guidance scheme to the first target object through the AR glasses so as to trigger the first target object to maintain the equipment to be maintained according to the target maintenance guidance scheme;
the AR glasses based device repair guidance system is further configured to: acquiring a model updating instruction in real time, updating the three-dimensional virtual model in real time according to the model updating instruction, and synchronously displaying the updated three-dimensional virtual model to the first target object and the second target object; the model updating instruction comprises model operation instructions input by the first target object and the second target object, the model operation instructions comprise a disassembling instruction, a dragging instruction, a combining instruction and a labeling instruction, and the first target object inputs the model operation instructions in a gesture control and/or voice control mode through the AR glasses; the communication channel comprises an equipment information communication channel, a guidance scheme communication channel and a model operation communication channel which are independent of each other; the equipment information communication channel is used for transmitting static information, dynamic information and description information corresponding to the equipment to be maintained; the guide scheme communication channel is used for transmitting the target maintenance guide scheme; the model operation communication channel is used for transmitting the model updating instruction; the bandwidth of the equipment information communication channel is highest, the data transmission safety and stability of the guidance scheme communication channel are highest, and the time delay of the model operation communication channel is lowest;
when the first target object maintains the equipment to be maintained according to a target step in the target maintenance guidance scheme, acquiring a maintenance operation image of the first target object, sending the maintenance operation image to the second target object and/or a trained operation judgment model to trigger the second target object and/or the trained operation judgment model to perform misoperation judgment, wherein the trained operation judgment model is used for performing feature recognition on the maintenance operation image to acquire an operation feature, and comparing the operation feature with a step feature corresponding to the target step to perform misoperation judgment;
when misoperation indicating information is received, carrying out misoperation warning prompt on the first target object, and limiting the AR glasses not to output the steps after the target step in the target maintenance guidance scheme until the first target object corrects the misoperation corresponding to the target step; wherein the misoperation instruction information is any one of model judgment information acquired by the trained operation judgment model and guidance object judgment information input by the second target object;
when misoperation indicating information is received, acquiring an error grade corresponding to the misoperation indicating information according to the misoperation indicating information, and recording misoperation duration time corresponding to the target step; when the error grade exceeds a preset grade threshold or the misoperation duration time exceeds a preset time threshold, locking the maintenance operation authority of the first target object, and sending authority locking alarm information to preset mobile equipment;
the operation judgment model is trained in advance according to the following steps: inputting an operation training image and an operation step description text in second training data into an operation judgment model, obtaining a prediction step description text corresponding to the operation training image through the operation judgment model, and calculating prediction similarity between the prediction step description text and the corresponding operation step description text through the operation judgment model, wherein the second training data comprises a plurality of groups of training matching data sets, and each group of training matching data set comprises the operation training image, the operation step description text and labeling similarity; adjusting the model parameters of the operation judgment model according to the prediction similarity and the corresponding labeling similarity, and continuously executing the step of inputting the operation training images and the operation step description texts in the second training data into the operation judgment model until a preset second training condition is met to obtain a trained operation judgment model; the value of the labeling similarity is 0 or 1,0 represents mismatching, 1 represents matching, and the operation judgment model outputs the prediction similarity which is greater than or equal to the preset threshold value, the operation judgment model outputs the prediction similarity which is matched, otherwise, the operation judgment model outputs the prediction similarity which is not matched.
6. An intelligent terminal, characterized in that the intelligent terminal comprises a memory, a processor and an AR glasses-based device maintenance guidance program stored on the memory and executable on the processor, wherein the AR glasses-based device maintenance guidance program, when executed by the processor, implements the steps of the AR glasses-based device maintenance guidance method according to any one of claims 1 to 4.
CN202211384772.0A 2022-11-07 2022-11-07 AR (augmented reality) -glasses-based equipment maintenance guiding method and system and related equipment Active CN115437507B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202211384772.0A CN115437507B (en) 2022-11-07 2022-11-07 AR (augmented reality) -glasses-based equipment maintenance guiding method and system and related equipment

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202211384772.0A CN115437507B (en) 2022-11-07 2022-11-07 AR (augmented reality) -glasses-based equipment maintenance guiding method and system and related equipment

Publications (2)

Publication Number Publication Date
CN115437507A CN115437507A (en) 2022-12-06
CN115437507B true CN115437507B (en) 2023-04-07

Family

ID=84252327

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202211384772.0A Active CN115437507B (en) 2022-11-07 2022-11-07 AR (augmented reality) -glasses-based equipment maintenance guiding method and system and related equipment

Country Status (1)

Country Link
CN (1) CN115437507B (en)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115567695B (en) * 2022-12-07 2023-04-07 国网浙江省电力有限公司宁波供电公司 Inspection method, device, system, equipment and medium based on wearable equipment
CN115576434B (en) * 2022-12-09 2023-04-25 深圳先进技术研究院 AR-based equipment operation guiding method, AR-based equipment operation guiding system and related equipment
CN115965361A (en) * 2022-12-30 2023-04-14 中国科学院空间应用工程与技术中心 Intelligent induction maintenance system for space application facilities

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1431798A2 (en) * 2002-12-19 2004-06-23 Accenture Global Services GmbH Arbitrary object tracking in augmented reality applications
CN107578110A (en) * 2017-09-19 2018-01-12 北京枭龙科技有限公司 A kind of repair message method for visualizing based on augmented reality
CN113380088A (en) * 2021-04-07 2021-09-10 上海中船船舶设计技术国家工程研究中心有限公司 Interactive simulation training support system

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9665985B2 (en) * 2014-08-15 2017-05-30 Daqri, Llc Remote expert system
CN110673726B (en) * 2019-09-23 2023-07-18 浙江赛弘众智网络科技有限公司 AR remote expert assistance method and system
US11995582B2 (en) * 2020-05-20 2024-05-28 XOEye Technologies, Inc. Automated generation of condition state information for one or more machines, devices, or systems associated with service or repair events
CN112018892B (en) * 2020-09-04 2023-02-14 南京太司德智能电气有限公司 Electric power operation and maintenance remote guidance system
CN113011723B (en) * 2021-03-04 2024-03-01 北京计算机技术及应用研究所 Remote equipment maintenance system based on augmented reality

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1431798A2 (en) * 2002-12-19 2004-06-23 Accenture Global Services GmbH Arbitrary object tracking in augmented reality applications
CN107578110A (en) * 2017-09-19 2018-01-12 北京枭龙科技有限公司 A kind of repair message method for visualizing based on augmented reality
CN113380088A (en) * 2021-04-07 2021-09-10 上海中船船舶设计技术国家工程研究中心有限公司 Interactive simulation training support system

Also Published As

Publication number Publication date
CN115437507A (en) 2022-12-06

Similar Documents

Publication Publication Date Title
CN115437507B (en) AR (augmented reality) -glasses-based equipment maintenance guiding method and system and related equipment
US20200012888A1 (en) Image annotating method and electronic device
CN110109535A (en) Augmented reality generation method and device
CN105989268A (en) Safety access method and system for human-computer identification
KR102087959B1 (en) Artificial intelligence operations system of telecommunication network, and operating method thereof
CN112560745B (en) Method for discriminating personnel on electric power operation site and related device
CN111444341B (en) User portrait construction method, device, equipment and readable storage medium
KR20200109271A (en) System for supporting parts maintenace based on mixed reality and method thereof
CN115659452B (en) Intelligent patrol method, intelligent patrol system and computer readable storage medium
CN112308006A (en) Sight line area prediction model generation method and device, storage medium and electronic equipment
CN111881740B (en) Face recognition method, device, electronic equipment and medium
WO2020065032A1 (en) Information providing method and system
CN110110320A (en) Automatic treaty review method, apparatus, medium and electronic equipment
CN115640074A (en) Service data processing method and device and intelligent counter terminal
CN110969327B (en) Work order dispatching method, device, system and data processing method
CN113190680A (en) Unstructured data marking method, device, equipment and storage medium
CN116563604A (en) End-to-end target detection model training, image target detection method and related equipment
CN111660294B (en) Augmented reality control system of hydraulic heavy-duty mechanical arm
CN114066438A (en) Model-based monitoring data display method, device, equipment and storage medium
CN114188032A (en) Digital medical information protection method based on artificial intelligence and storage medium
US20190266403A1 (en) Server, method and wearable device for supporting maintenance of military apparatus based on binary search tree in augmented reality-, virtual reality- or mixed reality-based general object recognition
CN111027429A (en) Data preprocessing method and system for intelligent identification of electrical drawings
KR102555733B1 (en) Object management for improving machine learning performance, control method thereof
CN115576434B (en) AR-based equipment operation guiding method, AR-based equipment operation guiding system and related equipment
KR102403617B1 (en) Apparatus and method for building training data

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant