CN114662714A - Machine room operation and maintenance management system and method based on AR equipment - Google Patents

Machine room operation and maintenance management system and method based on AR equipment Download PDF

Info

Publication number
CN114662714A
CN114662714A CN202210180987.4A CN202210180987A CN114662714A CN 114662714 A CN114662714 A CN 114662714A CN 202210180987 A CN202210180987 A CN 202210180987A CN 114662714 A CN114662714 A CN 114662714A
Authority
CN
China
Prior art keywords
module
maintenance
equipment
machine room
information
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210180987.4A
Other languages
Chinese (zh)
Inventor
高�浩
黄海涛
朱书良
宋巨成
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nanjing University of Posts and Telecommunications
Original Assignee
Nanjing University of Posts and Telecommunications
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nanjing University of Posts and Telecommunications filed Critical Nanjing University of Posts and Telecommunications
Priority to CN202210180987.4A priority Critical patent/CN114662714A/en
Publication of CN114662714A publication Critical patent/CN114662714A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/20—Administration of product repair or maintenance
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/006—Mixed reality
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/21—Server components or server architectures
    • H04N21/218—Source of audio or video content, e.g. local disk arrays
    • H04N21/2187—Live feed
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04W—WIRELESS COMMUNICATION NETWORKS
    • H04W4/00—Services specially adapted for wireless communication networks; Facilities therefor
    • H04W4/02—Services making use of location information
    • H04W4/024—Guidance services
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04W—WIRELESS COMMUNICATION NETWORKS
    • H04W4/00—Services specially adapted for wireless communication networks; Facilities therefor
    • H04W4/30—Services specially adapted for particular environments, situations or purposes
    • H04W4/33—Services specially adapted for particular environments, situations or purposes for indoor environments, e.g. buildings

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Business, Economics & Management (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Signal Processing (AREA)
  • Computer Graphics (AREA)
  • Human Resources & Organizations (AREA)
  • Software Systems (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Economics (AREA)
  • Strategic Management (AREA)
  • Computer Hardware Design (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Marketing (AREA)
  • Operations Research (AREA)
  • Quality & Reliability (AREA)
  • General Engineering & Computer Science (AREA)
  • Tourism & Hospitality (AREA)
  • General Business, Economics & Management (AREA)
  • Geometry (AREA)
  • Databases & Information Systems (AREA)
  • Multimedia (AREA)
  • Testing And Monitoring For Control Systems (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

The invention provides an AR machine room operation and maintenance management system and a method thereof, wherein the system comprises a plurality of modules: the broadcasting module is used for online communication and communication between operation and maintenance personnel and experts or leaders and for sending message service; the augmented reality scene generation module is used for generating a three-dimensional reconstruction point cloud model of the machine room; the indoor navigation module is used for guiding operation and maintenance personnel to quickly find equipment marked by the experts in live broadcasting for operation and maintenance; the information pushing module is used for collecting operation and maintenance operation steps sent by experts; the fault detection module is used for reporting faults when the operation and maintenance personnel find equipment faults; the remote multi-person cooperation module is used for remote watching of the visitor and guidance of the expert on one-to-many operation and maintenance personnel; the review auditing module is used for checking historical operation and maintenance records; the AR equipment is used for creating a chat room for direct broadcasting of experts and operation and maintenance personnel, displaying an indoor navigation path, a machine room scene model and pushing strategy guidance information sent by an expert terminal; the invention realizes that the operation and maintenance personnel can carry out the inspection work only by wearing the AR equipment, thereby improving the inspection efficiency.

Description

Machine room operation and maintenance management system and method based on AR equipment
Technical Field
The invention relates to an AR machine room operation and maintenance management system and method, and belongs to the technical field of augmented reality.
Background
The traditional machine room inspection mode depends on manual paperization inspection, so that the inspection quality is low, and the inspection data quality is poor.
Some computer lab are patrolled and examined and are introduced intelligence and patrol and examine the robot at present, can carry out some and examine work, nevertheless patrol and examine not only to rely on the vision, simultaneously because the influence in the aspect of mounted position, angle and the size of computer lab server, patrol and examine the phenomenon that the robot often breaks down and can't discern, patrol and read the accuracy of server data and also can't obtain guaranteeing. The general situation is that after the robot finishes the inspection, operation and maintenance personnel check the correctness of the data, the work is complicated, and the efficiency is low.
Disclosure of Invention
The invention aims to overcome the defects in the prior art, and provides an AR machine room operation and maintenance management system and method, which greatly improve the efficiency compared with the traditional manual inspection and intelligent inspection robot inspection of machine rooms.
In order to achieve the purpose, the invention is realized by adopting the following technical scheme:
in a first aspect, the present invention provides an AR device-based machine room operation and maintenance management system, including:
live module: the AR equipment is used for online communication and information sending service between the AR equipment and the expert end;
augmented reality scene generation module: the system comprises a three-dimensional reconstruction point cloud model used for generating a machine room;
the indoor navigation module: the device is used for guiding and finding equipment marked by the expert end in live broadcasting to carry out operation and maintenance;
the information pushing module: the system comprises an AR device and a server, wherein the AR device is used for collecting operation and maintenance operation steps sent by an expert terminal and pushing the operation and maintenance operation steps to the AR device;
a fault detection module: the system is used for reporting faults when the polling personnel find equipment faults;
remote multi-person collaboration module: the system is used for remote watching of the visitor and guidance of one-to-many inspection personnel by the expert;
a review audit module: used for viewing historical operation and maintenance records;
the AR device: the system is used for creating a chat room for direct broadcasting of experts and inspection personnel, displaying an indoor navigation path, a machine room scene model and pushing strategy guidance information sent by an expert terminal;
an expert end: the system is used for accessing a live broadcast module and sending strategy guide information;
a database: for storing data and records generated during the operation of the system.
Further, the AR device comprises an AR device camera and a microphone;
the live broadcast module obtains audio and video data streams from an AR equipment camera and a microphone, acquires, codes and decodes, encrypts, media files, processes images, displays, transmits over the network and performs flow control processing on the video data streams through a webRTC, and performs equipment, coding and decoding, encryption, sound files, sound processing, sound output, volume control, audio and video synchronization, network transmission and flow control processing on the audio data streams to obtain real-time audio and video, and transmits the real-time audio and video to an indoor navigation module, an information push module, a fault detection module, a remote multi-user cooperation module and a review audit module;
the augmented reality scene generation module acquires a machine room depth image from an AR equipment camera, synchronously positions and maps the machine room to obtain a three-dimensional reconstruction cloud picture of the machine room, and transmits the cloud picture to the indoor navigation module;
the indoor navigation module obtains a machine room three-dimensional reconstruction cloud picture of the augmented reality scene generation module and a navigation target point marked by a live broadcast module expert, generates a navigation planning path for operation and maintenance personnel to reach the nearest operation and maintenance equipment, and transmits the navigation planning path to an operation and maintenance personnel end of the live broadcast module;
the information pushing module obtains language explanation and video demonstration guidance of a specialist end to a fault processing process through a live broadcast module, codes are transmitted to AR equipment and a review auditing module through the live broadcast module;
the fault detection module acquires fault information of equipment to be inspected, uploads fault content on a fault alarm user interface, and transmits the fault content to the expert terminal, the review audit module and the remote multi-person cooperation module;
the remote multi-person cooperation module joins a current live broadcast room when acquiring a multi-expert remote assistance invitation of the information pushing module, guides operation and maintenance personnel in real time, obtains a solution and transmits the solution to the information pushing module, the live broadcast module and the review auditing module;
the review auditing module obtains contents of a live broadcast screen, a fault solution and fault contents from the live broadcast module, the information pushing module, the fault detection module and the remote multi-person cooperation module;
the augmented reality scene generation module provides a walkable region for the indoor navigation module for path planning.
Further, the operation method of the system comprises the following steps:
the AR device loads a display from a database; the augmented reality scene generation module transmits a three-dimensional model of a machine room to an indoor navigation module and a live broadcast module, a live broadcast module expert transmits the marked position and the operation and maintenance operation steps of operation and maintenance equipment to an information push module, the information push module transmits the position and the position of the operation and maintenance equipment to the indoor navigation module, the operation and maintenance operation steps are transmitted to AR equipment of operation and maintenance personnel, the indoor navigation module generates a navigation path and transmits the navigation path to the AR equipment of the operation and maintenance personnel, and the AR equipment transmits audio and video streams to a remote multi-person cooperation module when the operation and maintenance personnel perform equipment operation and maintenance; all data of the live broadcast module, the augmented reality scene generation module, the indoor navigation module, the information push module, the fault detection module and the remote multi-person cooperation module are transmitted to the review audit module so as to review audit.
Further, the AR device comprises a sensor module, a scene understanding module, a three-dimensional sensing module and a virtual-real fusion module;
the sensor module comprises a positioning device, an inertial sensor, an AR equipment camera and a microphone;
the scene understanding module is used for image recognition, object recognition, human body recognition and AR triggering;
the three-dimensional sensing module is used for synchronous positioning and mapping, three-dimensional reconstruction and positioning tracking;
the virtual-real fusion module is used for image rendering;
the origin of the world coordinate system of the AR equipment takes the current equipment position as the standard, and after the machine room two-dimensional code needing to be patrolled and examined is read, the corresponding machine room augmented reality model can be loaded in the AR equipment and completely attached to an actual machine room scene.
Further, the method for remote watching of the visitor and guidance of the expert on the one-to-many inspection personnel by the remote multi-person cooperation module comprises the following steps:
the remote multi-person cooperation module is accessed to AR equipment worn by an outside visitor, records all operation and audio-video information of the outside visitor after entering a machine room, and transmits the operation and audio-video information to a manager end;
and acquiring control information of a manager, and monitoring and checking one-to-one or one-to-many real-time pictures according to the control information to realize interaction of audio and video, text information and annotations of the manager and the AR equipment.
Further, the method for generating the three-dimensional reconstruction point cloud model of the machine room by the augmented reality scene generation module comprises the following steps:
the method comprises the steps of pre-acquiring point clouds of a cabinet and a server in a machine room scene to form a data set, matching the acquired data and the point clouds with objects in the data set during actual scene acquisition, removing noise points and outliers separated from a main body from the acquired data through a filtering algorithm, and selecting effective features for identification according to data characteristics;
the method comprises the steps of sampling peripheral environment information and reconstructing three-dimensional information in an inspection process by adopting a visual reconstruction technology, integrating an inspected 3D image into a real working condition environment and a peripheral environment of a machine room, processing a holographic image of the inspection by combining a data information processing and virtual visual simulation technology, and transmitting a processed reconstruction model to an indoor navigation module;
after the acquired data set is processed, three-dimensional point cloud of the whole machine room is generated; and carrying out surface mounting processing on the obtained point cloud model.
Further, the method for guiding the search of the equipment marked by the expert end in the live broadcast to carry out operation and maintenance by the indoor navigation module comprises the following steps:
after an indoor machine room map is obtained, then, the information of indoor cabinets and servers is added into the indoor map, so that the complete map construction of the machine room is completed;
self-positioning is realized by scanning the two-dimensional code; the method comprises the following steps that an indoor positioning method based on wireless signals is used in a machine room scene, and WiFi is used as basic positioning facilities; planning a navigation path after the position of the AR equipment wearer and the position of the maintenance target are known;
and carrying out real-time path guidance according to path planning, continuously receiving video frames from the AR equipment camera in the navigation process, capturing natural characteristic points in a picture, and redrawing the image according to a path planning result so as to achieve the effect of placing a 3D guide model in the real scene of the AR equipment camera.
Further, the method for collecting operation and maintenance operation steps sent by the expert and pushing the operation and maintenance operation steps to the AR equipment by the information pushing module comprises the following steps:
after a patrol inspector logs in a patrol inspection system, a monocular camera of an AR device scans a two-dimensional code on an identification device, detailed information of a set of equipment cabinet servers can be obtained from a background server, information of a virtual machine in a physical machine is obtained from a cloud management platform in an identification mode, the management platform can push a task work order, equipment information, an operation and maintenance manual and fault diagnosis to the AR device from the background, the operation and maintenance inspector checks and transfers operation and maintenance related information through gesture operation in an MRTK suite, operation and maintenance are assisted, meanwhile, the AR device is also connected with a Windows universal application platform (UMP), alarm information received in the AR device is synchronized during operation and maintenance, and real-time operation and maintenance dynamic is obtained.
Furthermore, the review and audit module is used for uploading task contents after the patrol personnel completes each operation and maintenance task, so that the patrol personnel and experts can review and examine historical operation and maintenance tasks later.
In a second aspect, the present invention provides an AR machine room operation and maintenance management method, based on the system in the first aspect, including the following steps:
s10: and acquiring a preset three-dimensional reconstruction model of a scene to be monitored, wherein the scene to be monitored comprises a plurality of objects to be monitored.
S20: acquiring operation data of an object to be monitored in real time, storing the operation data into a database, updating an integrated machine room reconstruction model of a scene to be monitored, displaying the object to be monitored and the operation data thereof on AR equipment, and generating fault equipment information when a monitoring server has an operation fault;
s30: judging whether the monitored object is in a fault state or not according to the information of the fault detection module, if the object to be monitored is in the fault state, sending an operation and maintenance task to an operation and maintenance worker, planning a path according to the position of specific fault equipment, and guiding the operation and maintenance worker to reach a server to be operated and maintained;
s40: the operation and maintenance personnel arrive at the to-be-operated and maintained server, live broadcast communication is carried out on the operation and maintenance operation of the communication equipment with the experts, the experts transmit specific operation and maintenance operation manuals to the information pushing module, and if a single expert cannot solve the operation and maintenance of the equipment, the information pushing module transmits assistance requests to the remote multi-person cooperation module;
s50: if the multiple online experts receive the information pushing invitation to perform remote assistance, the online experts can join the live broadcast module to perform multi-expert common remote cooperation to guide operation and maintenance personnel to perform operation and maintenance operation, and visit personnel obtain permission to enter the live broadcast module to view;
s60: the whole equipment operation and maintenance task process participated by operation and maintenance personnel and experts is transmitted to the review audit module, and data is uploaded to the cloud server for storage.
Compared with the prior art, the invention has the following beneficial effects:
1. the patrol personnel can be remotely live broadcast with the experts or leaders in the AR equipment, after the fault equipment is marked by the expert, the AR equipment can generate a navigation path to guide the patrol personnel, and meanwhile, the operation and maintenance steps of the fault equipment given by the experts can be displayed.
2. The live broadcast module can assist field inspectors to easily deal with sudden failures and process related problems in real time, when the related difficult problems are met, the inspectors can start a fault diagnosis remote assistance function, the inspectors wear AR equipment and establish video connection with rear technicians or experts by using a Wi-Fi network, the whole process of maintenance of the operation and maintenance tasks of the machine room is transmitted to the rear expert AR equipment in a video stream mode through the AR equipment, and technical supervision or field guidance is achieved. The operation and maintenance personnel and the expert terminal can communicate in real time through the live broadcast module, the operation and maintenance personnel obtain the position of the fault equipment and transmit the position to the indoor navigation module, and the operation and maintenance operation steps of the expert terminal guide and transmit the position to the information push module, so that the operation and maintenance efficiency is improved, and the operation and maintenance time cost and the labor cost are reduced;
3. the indoor navigation module generates a ground path to guide the operation and maintenance personnel to reach the specified operation and maintenance equipment, a shortest path is found through a Dijkstra algorithm, the operation and maintenance personnel is guided to reach an operation and maintenance point at the highest speed to carry out the next specific operation and maintenance operation, and the expert end transmits the operation and maintenance guiding operation to the information pushing module.
4. The operation and maintenance personnel can request remote expert support when the operation and maintenance personnel arrive at the equipment needing operation and maintenance to check the fault problem in time and obtain operation and maintenance operation guidance sent by the information pushing module, and can also send a request to the remote multi-person cooperation module to request multi-expert support if a single expert cannot solve the problem, so that the operation and maintenance personnel and the multi-expert can achieve the communication of the face-to-face effect in the AR equipment, and the fault problem can be solved quickly and effectively, thereby improving the operation and maintenance efficiency;
6. the invention transmits the operation and maintenance data to the review and audit module by recording the operation and maintenance tasks of the operation and maintenance personnel each time so as to achieve the purpose of reviewing and auditing the operation and maintenance behaviors. Meanwhile, the data of the machine room model is updated in real time, the three-dimensional model of the machine room can be downloaded on line through a Web end http webpage, so that inspection personnel and experts can master the poses of the server and the cabinet of the machine room to be inspected in real time, and the augmented reality scene generation module can carry out three-dimensional reconstruction and display on all the servers and the cabinet of the machine room, so that the inspection personnel can conveniently and globally monitor machine room equipment.
7. When the sensor, the monitoring software or the manual work detects that the equipment to be operated and maintained is in a fault state, the information of the fault equipment is uploaded to the fault detection module, real-time warning display is carried out on the AR equipment, operation and maintenance personnel and managers can find hidden dangers in time, and operation and maintenance efficiency is improved.
Drawings
Fig. 1 is a schematic flowchart of a machine room operation and maintenance management method based on an AR device in a second embodiment of the present invention;
fig. 2 is a schematic structural diagram of a device of a machine room operation and maintenance management method based on AR equipment in a first embodiment of the present invention;
FIG. 3 is a schematic structural diagram of an AR wearable device according to an embodiment of the present invention;
fig. 4 is a block diagram of the system architecture of the present invention.
Detailed Description
The invention is further described below with reference to the accompanying drawings. The following examples are only for illustrating the technical solutions of the present invention more clearly, and the protection scope of the present invention is not limited thereby. It should be understood that the step numbers used herein are for convenience of description only and are not intended as limitations on the order in which the steps are performed.
It is to be understood that the terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in the specification of the present invention and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
The term "and/or" refers to any and all possible combinations of one or more of the associated listed items and includes such combinations.
The first embodiment is as follows:
as shown in fig. 2, in this embodiment, the computer room operation and maintenance management system based on the AR device includes a live broadcast module, an augmented reality scene generation module, an indoor navigation module, an information push module, a fault detection module, a remote multi-person cooperation module, and a review audit module, as shown in fig. 2. The functional modules are explained in detail as follows:
live module: the system is used for online communication and message sending service between inspection personnel and experts or leaders;
augmented reality scene generation module: the system comprises a three-dimensional reconstruction point cloud model used for generating a machine room;
the indoor navigation module: the system is used for guiding operation and maintenance personnel to quickly find equipment marked by the experts in live broadcasting for operation and maintenance;
the information pushing module: an operation and maintenance operation step for collecting expert transmission;
a fault detection module: the system is used for reporting faults when the polling personnel find equipment faults;
remote multi-person collaboration module: the system is used for remote watching of the visitor and guidance of one-to-many inspection personnel by the expert;
a review audit module: used for viewing historical operation and maintenance records;
the system comprises a live broadcast module, an AR equipment camera and a microphone, a webRTC video data stream, an equipment, a codec (I420/VP8), an encryption module, a media file, an image processing module, a display module, a network transmission and flow control (RTP/RTCP) module, an audio data stream, an audio and video (iLIBC/iSAC/G722/PCM16/RED/AVT, NetEQ), an encryption module, a sound file, a sound processing module, a sound output module, a volume control module, an audio and video synchronization module, a network transmission and flow control (RTP/RTCP) module, a real-time audio and video are obtained and transmitted to an indoor navigation module, an information pushing module, a fault detection module, a remote multi-person cooperation module and a review auditing module; the augmented reality scene generation module acquires a machine room depth image from the AR equipment RGB-D camera, synchronously positions and maps the machine room to obtain a three-dimensional reconstruction cloud point map of the machine room, and transmits the cloud point map to the indoor navigation module; the indoor navigation module obtains a machine room three-dimensional reconstruction cloud picture of the augmented reality scene generation module and a navigation target point marked by a live broadcast module expert, generates a navigation planning path for operation and maintenance personnel to reach the nearest operation and maintenance equipment, and transmits the navigation planning path to an operation and maintenance personnel end of the live broadcast module; the information pushing module obtains language explanation and video demonstration guidance of a live broadcast module expert terminal on a fault handling process, codes and transmits the codes to a live broadcast module operation and maintenance personnel terminal and a review auditing module; the fault detection module acquires fault information of equipment to be inspected, uploads fault content on a fault alarm user interface, and transmits the fault content to a live broadcast module expert terminal, a review auditing module and a remote multi-person cooperation module; the remote multi-person cooperation module joins a current live broadcast room when acquiring a multi-expert remote assistance invitation of the information pushing module, guides operation and maintenance personnel in real time, and transmits a solution to the information pushing module, the live broadcast module and the review auditing module; the review auditing module obtains live broadcast screen recording contents, fault solutions, fault contents and the like from the live broadcast module, the information pushing module, the fault detection module and the remote multi-person cooperation module. The augmented reality scene generation module provides a walkable area for the indoor navigation module for path planning, the live broadcast module runs through all the modules to ensure the normal realization of the functions of the rest modules, and the modules have certain independence in function, but are interwoven as a whole.
The machine room operation and maintenance management system based on the AR equipment integrates a live broadcast module, an augmented reality scene generation module, an indoor navigation module, an information push module, a fault detection module, a remote multi-person cooperation module and a review audit module, all data of the modules are transmitted to a database, and an AR end loads and displays the data from the database; the augmented reality scene generation module transmits a machine room three-dimensional model to an indoor navigation module and a live broadcast module, a live broadcast module expert transmits the marked operation and maintenance equipment pose and operation and maintenance operation steps to an information push module, the information push module transmits the pose of the operation and maintenance equipment to the indoor navigation module, the operation and maintenance operation steps are transmitted to an operation and maintenance personnel AR end, the indoor navigation module generates a navigation path and transmits the navigation path to the operation and maintenance personnel AR end, and the operation and maintenance personnel transmit audio and video streams to a remote multi-person cooperation module during equipment operation and maintenance; all data of the live broadcast module, the augmented reality scene generation module, the indoor navigation module, the information push module, the fault detection module and the remote multi-person cooperation module are transmitted to the review audit module so as to review audit, find out errors, violations and other operations in the operation process, and improve the fault troubleshooting and service improvement capabilities.
Patrol and examine personnel and after wearing AR equipment entering operation page, the initial point of whole world coordinate system uses current equipment location as the standard, patrol and examine personnel after having scanned the computer lab two-dimensional code, the computer lab augmented reality model that corresponds can load in AR equipment and laminate completely with actual computer lab scene.
The patrol personnel can be remotely live broadcast with the experts or leaders in the AR equipment, after the fault equipment is marked by the expert, the AR equipment can generate a navigation path to guide the patrol personnel, and meanwhile, the operation and maintenance steps of the fault equipment given by the experts can be displayed.
The live broadcast module can assist field inspection personnel to easily deal with sudden faults and process related problems in real time, when related problems are encountered, the inspection personnel can start a fault diagnosis remote assistance function, obtain audio and video data streams from an AR device camera and a microphone, acquire, encode and decode (I420/VP8), encrypt, media files, process images, display, process network transmission and flow control (RTP/RTCP), perform equipment, encode and decode (iLIBC/iSAC/G722/PCM16/RED/AVT, Net), encrypt, sound files, process sound, output sound, volume control, audio and video synchronization, process network transmission and flow control (RTP/RTCP), obtain real-time audio and video displayed at an expert end and an inspection personnel end, and transmit the real-time audio and video to an indoor navigation module, an information push module, an EQ, The system comprises a fault detection module, a remote multi-person cooperation module and a review and audit module.
The inspection personnel wear the AR equipment and establish video connection with rear technicians or experts by using the Wi-Fi network, and the whole maintenance process of the operation and maintenance task of the machine room is transmitted to the AR end of the rear experts by video streaming through the AR equipment, so that technical supervision or 'field guidance' is realized.
The live broadcast module adopts SFU (selective Forwarding Unit) scheme, and selects a live broadcast server Janus-gateway supporting WebRTC protocol as media communication service.
The augmented reality scene generation module pre-acquires point clouds of a cabinet and a server in a machine room scene to form a data set, matches the acquired data and the point clouds with objects in the data set when the actual scene is acquired, removes noise points and outliers separated from a main body from the acquired data through a filtering algorithm, selects effective characteristics for a recognition algorithm according to data characteristics, for example, for eliminating the noise points of a ceiling when a machine room model is reconstructed, a BAD SLAM direct method is adopted for real-time BA + RGBD, the front end tracks the motion of an RGB-D camera in real time, initial estimation is provided for the posture and scene geometry of the camera, the rear end runs at a lower frequency, and the camera track and a scene 3D map are corrected, so that the acquired objects are well described. The module adopts a visual reconstruction technology to sample peripheral environment information and reconstruct three-dimensional information in the inspection process, and specifically shows that AR imaging is carried out by adopting AR wearable equipment, an inspected 3D image is fused in a real working condition environment and a peripheral environment of a machine room, holographic image processing of inspection is carried out by combining data information processing and a virtual visual simulation technology, and a processed reconstruction model is transmitted to an indoor navigation module;
after the acquired data set is processed, three-dimensional point cloud of the whole machine room is generated; and carrying out surface mounting processing on the obtained point cloud model, and carrying out addition, deletion, check and modification operation on a machine room and a server in the machine room model.
The indoor navigation module is used for planning an optimal route from a patrol worker to a device needing to be patrolled and generating a virtual navigation path, and if a set of complete indoor navigation module is to be built, an indoor machine room map, patrol worker positioning, path planning and path real-time guidance are required, the indoor map is a carrier of indoor space information, the display of the space information can be performed only by means of the construction of the indoor map, and after the indoor machine room map is obtained, indoor cabinet and server information is added into the indoor map, so that the complete map construction of a machine room is completed; in order to guide the route of an inspector, the accurate indoor position of the inspector must be known, when a plurality of machine rooms are provided and the environments are consistent, in order to facilitate the inspector to complete indoor positioning work, a designed two-dimensional code is placed at an inlet of each machine room, self positioning can be rapidly realized by scanning the two-dimensional code, an indoor positioning method based on wireless signals is mainly used in a machine room scene, and WiFi (WLAN based on IEEE802.11 standard) is used as an infrastructure positioning facility; after the positions of the AR equipment wearer and the positions of the overhaul targets are known, a navigation path needs to be planned, one important problem to be solved in path planning is a shortest path problem, and Dijkstra (Dijkstra) algorithm is a typical shortest path algorithm and can calculate the shortest path from one place to other places; the real-time path guidance is that after a wearer receives daily maintenance tasks or fault information, the wearer clicks task prompts, a virtual path can be generated under feet of the wearer, the wearer can be quickly navigated to a task place, an AR module takes Dijkstra path planning results as input, AR resources are loaded to a cache, an AR engine continuously receives video frames from a device camera in the navigation process, natural feature points in a picture are captured, and then the image is redrawn according to the path planning results, so that the effect of placing a 3D guide model in a camera real scene is achieved.
After the inspection personnel log in the inspection system, the information pushing module scans and identifies two-dimensional codes on the equipment through the monocular camera of the AR equipment, detailed information of the set of equipment cabinet server can be obtained from the background server, information of a virtual machine in the physical machine can be identified and obtained from the cloud management platform, the management platform can push required information such as a task work order, equipment information, an operation and maintenance manual and fault diagnosis and the like to the AR equipment from the background, the operation and maintenance personnel check and call operation and maintenance related information through gesture operation in the MRTK suite, operation and maintenance are assisted, meanwhile, the AR equipment is further connected with the UMP, alarm information received in the AR equipment is synchronized during operation and maintenance, and real-time operation and maintenance dynamic state is obtained.
When a user wears AR equipment to enter a machine room for operation and maintenance work, and when the fault is manually detected in the machine room, the fault detection module clicks a fault alarm user interface button to upload fault content.
The remote multi-person cooperation module can be used for an external visitor to wear the AR equipment to realize remote accompanying, all operation and audio and video information of the external visitor after entering the machine room can be recorded, and a manager can supervise and check one-to-one and one-to-many real-time pictures on a rear-end management platform, so that interaction of audio and video, character information and comments of the manager and the staff is realized, and real-time communication can be realized when problems are found.
And the review and audit module is used for uploading task contents after the patrol personnel completes each operation and maintenance task so that the patrol personnel and experts can review and examine historical operation and maintenance tasks later.
The AR interaction module is characterized in that AR equipment can operate in a space around the equipment, a user mainly performs information interaction in a Gaze (staring), Gesture (Gesture) and Voice (Voice) mode, the Gaze determines an interactive object in an eye staring mode, the AR equipment automatically performs positioning, the function is similar to that of a mouse, the AR equipment performs image recognition on stared equipment to be inspected during polling operation, so that equipment confirmation is performed, Gesture controls interactive contents through the Gesture, built-in gestures of the AR equipment comprise Press and release, Bloom, Hold, management, Navigation and the like, and polling information can be input or other functions can be selected and called through the Gesture during polling operation; meanwhile, the AR equipment has a voice input mode, allows diversified input modes, operates by voice, and can issue commands or requests by voice to call other functions when in inspection operation
The modules described above may be implemented in the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein, embodied entirely in software, or embodied in a combination of software and hardware.
As shown in fig. 3, an embodiment of the present invention further provides an AR wearable device, including a processor and a storage medium, where the storage medium is used to store instructions, and the processor is used to perform the following steps of the method according to the instructions:
s10: and acquiring a preset three-dimensional reconstruction model of a scene to be monitored, wherein the scene to be monitored comprises a plurality of objects to be monitored.
S20: the method comprises the steps of obtaining operation data of an object to be monitored in real time, storing the operation data into an operation and maintenance database, updating an integrated machine room reconstruction model of a scene to be monitored, displaying the object to be monitored and the operation data thereof on AR equipment, and sending fault equipment information to a fault detection module when a monitoring server has an operation fault.
S30: and judging whether the monitored object is in a fault state or not according to the information of the fault detection module, if the object to be monitored is in the fault state, receiving an operation and maintenance task by the operation and maintenance personnel, and making a path plan by the indoor navigation module according to the position of the specific fault equipment to guide the operation and maintenance personnel to reach the server to be operated and maintained.
S40: the operation and maintenance personnel arrive at the server to be operated and maintained, the operation and maintenance operation of the communication equipment is communicated with the experts in a live broadcast mode, the experts transmit specific operation and maintenance operation manuals to the information pushing module, and if the operation and maintenance of the equipment cannot be solved by a single expert, the information pushing module transmits assistance requests to the remote multi-person cooperation module.
S50: if the multiple online experts receive the information pushing invitation to perform remote assistance, the online experts can join the live broadcast module to perform multi-expert common remote cooperation, the operation and maintenance personnel are guided to perform operation and maintenance operation, and the outsiders are allowed to enter the live broadcast module to watch.
Example two:
as shown in fig. 1, an embodiment of the present invention provides a machine room operation and maintenance management method based on an AR device, including the following steps:
s10: and acquiring a preset three-dimensional reconstruction model of a scene to be monitored, wherein the scene to be monitored comprises a plurality of objects to be monitored.
In this embodiment, the scene to be monitored refers to a machine room in which an infrastructure needs to be monitored, and in this embodiment, the scene to be monitored is a machine room in which a server is placed, and in other embodiments, the scene to be monitored may also be a school, an office building, a market, a factory, or the like. The object to be monitored refers to an infrastructure to be monitored, in this embodiment, the object to be monitored is a server placed in a machine room, and in other embodiments, the object to be monitored may be a precision instrument cabinet, an IT device, a power supply and distribution device, a lighting system and the like to be monitored. The integrated machine room reconstruction model is a three-dimensional reconstruction model of all cabinets and servers in a machine room, real information of all objects in the machine room is built through a three-dimensional reconstruction technology, and the real information runs through four stages of the whole life cycle of the machine room server, namely planning and planning, surveying and designing, operating and maintaining, adding and deleting and checking and modifying. In this embodiment, the integrated machine room reconstruction model is an integration of a plurality of machine room reconstruction models of a company, and information displayed by the integrated machine room reconstruction model includes detailed information of each machine room reconstruction model, such as a machine room number, a cabinet number, a server number, and the like.
Specifically, the computer lab rebuilds the model through AR equipment camera with three-dimensional reconstruction technique when building the computer lab server from the planning, through assembling into an integrated computer lab rebuilds the model with numerous computer lab rebuilds the model, derive the three-dimensional model of OBJ form among the AR equipment holographic image, upload appointed interface with the 2D plane model and the 3D model of computer lab rebuild the model, with the integrated computer lab of convenient later stage rebuild the model heat and update, 2D plane model supplies the expert to mark the place, it is regional that the 3D model generates the accessible when supplying path planning, the computer lab rebuilds the camera and can use Kinect, AR equipment is from taking the depth camera etc. provides computer lab depth information acquisition function.
S20: the method comprises the steps of obtaining operation data of an object to be monitored in real time, storing the operation data into an operation and maintenance database, updating an integrated machine room reconstruction model of a scene to be monitored, displaying the object to be monitored and the operation data thereof on AR equipment, and sending fault equipment information to a fault detection module when a monitoring server has an operation fault.
In this embodiment, the operation data refers to data parameters generated by the object to be monitored in the operation process, for example, if the object to be monitored is a server temperature, the operation data includes data such as a boot temperature, an average temperature per hour, and an operation time. The operation and maintenance database is a database used for storing data such as real-time monitoring information, maintenance information and the like in the operation and maintenance process of the equipment, wherein the real-time monitoring information is operation data obtained in real time.
Specifically, the running data of the object to be monitored is obtained in real time through a sensor or monitoring software, then the running data is stored in a running and maintenance database, then the information in the running and maintenance database is imported into the AR equipment, a three-dimensional model of all the objects to be monitored and the basic information and the real-time running data of the objects to be monitored can be displayed, and when the running fault of the monitored server occurs, the fault equipment information is sent to a fault detection module.
S30: and judging whether the monitored object is in a fault state or not according to the information of the fault detection module, if the object to be monitored is in the fault state, receiving an operation and maintenance task by the operation and maintenance personnel, and making a path plan by the indoor navigation module according to the position of the specific fault equipment to guide the operation and maintenance personnel to reach the server to be operated and maintained.
In this embodiment, the failure state refers to that a parameter value generated by an object to be monitored during operation is abnormal, and operation and maintenance personnel are required to overhaul the object to be monitored. And (4) performing indoor navigation to plan a path and guiding the operation and maintenance personnel to reach the target monitored object to be overhauled as soon as possible.
Specifically, when the operation and maintenance personnel receive the task to be operated and maintained, the AR equipment displays a flow arrow to guide the operation and maintenance personnel to go to the operation and maintenance personnel quickly.
S40: the operation and maintenance personnel arrive at the to-be-operated and maintained server, live broadcast communication is carried out with the expert to exchange equipment operation and maintenance operation, the expert transmits a specific operation and maintenance operation manual to the information pushing module, and if a single expert cannot solve equipment operation and maintenance, the information pushing module transmits an assistance request to the remote multi-person cooperation module.
In the embodiment, when the operation and maintenance personnel arrive at the specified equipment to be overhauled, an expert calling request is sent, the expert and the operation and maintenance personnel carry out technical communication in a live broadcast room, and when the operation and maintenance personnel cannot solve the problem of equipment overhaul, the expert can provide technical support for the operation and maintenance personnel; if a single expert still cannot solve the current problem, multiple experts need to be invited to join the live broadcast room.
Specifically, if the server receives an instruction that an operation and maintenance worker clicks and checks alarm information appearing on the fault detection module on the AR device, the server displays specific alarm information on the AR device, performs fault diagnosis, and generates a maintenance work order.
Furthermore, operation and maintenance personnel or expert personnel can select the alarm information of the object to be processed in the fault detection module to carry out fault diagnosis processing, and the operation and maintenance personnel and the expert personnel can discuss on line through the live broadcast module. After the fault diagnosis is completed, a maintenance work order can be generated and sent to the information pushing module for operation and maintenance personnel or other management personnel to check.
Preferably, in this embodiment, in addition to performing fault diagnosis and generating a maintenance work order, an abnormal positioning operation may be performed on the object to be processed, the object to be processed is three-dimensionally and visually displayed through the AR device, and a specific fault component or part of the object (device) to be processed is displayed, so that an operation and maintenance person or other management person can more intuitively check the fault condition of the object to be processed.
S50: if the multiple online experts receive the information pushing invitation to perform remote assistance, the online experts can join the live broadcast room to perform multi-expert common remote cooperation to guide operation and maintenance personnel to perform operation and maintenance operation, and the outsiders can enter the live broadcast room to watch after obtaining permission.
In this embodiment, the expert receives the invitation to join the live broadcast room so as to instruct the operation and maintenance personnel to perform the equipment operation and maintenance, and the visit personnel or the leader can enter the live broadcast room in the whole course to watch the operation and maintenance flow after obtaining the authorization.
Specifically, when the operation and maintenance personnel communicate with a single expert, if the problem of the fault of the equipment to be maintained cannot be solved, the operation and maintenance personnel sends a multi-expert remote assistance request, when the multiple experts jointly guide the operation and maintenance personnel to complete the maintenance of the equipment to be maintained on line, the fault solution is stored in the operation and maintenance database, the operation and maintenance database is called through a Web terminal, the corresponding fault solution is displayed for the problem encountered in the future, and the whole operation and maintenance process is transmitted to the review audit module.
S60: and the review auditing module receives data in the whole equipment operation and maintenance process, so that operation and maintenance operation behavior auditing and operation and maintenance operation review checking are achieved.
In this embodiment, live broadcast contents of the experts and the operation and maintenance personnel, specific fault equipment numbers, fault contents, and fault solutions are uploaded to the fault detection module.
Specifically, the screen recording, log recording and system state and operation of the live broadcast module are performed to store the content. According to the returned operation records and snapshots, information backup and storage are carried out, when the system fails, information backtracking is carried out through the recorded operation contents, and the safety and controllability of the system are improved.
The invention can add, delete, check and modify data by updating the machine room model in real time, and can download the three-dimensional model of the machine room on line through the HTTP webpage of the Web end, so that the patrol personnel and experts can master the poses of the server and the cabinet of the machine room to be patrolled and examined in real time, and the augmented reality scene generation module can carry out three-dimensional reconstruction and display on all the servers and the cabinet of the machine room, thereby facilitating the patrol personnel to monitor the equipment of the machine room globally.
When the fault state of the equipment to be operated and maintained is detected through a sensor, monitoring software or manually, information of the fault equipment is uploaded to the fault detection module, real-time alarm display is carried out on the AR equipment, operation and maintenance personnel and managers can find hidden dangers in time, and operation and maintenance efficiency is improved;
according to the invention, the operation and maintenance personnel and the expert terminal can communicate in real time through the live broadcast module, the operation and maintenance personnel obtain the position of the fault equipment and transmit the position to the indoor navigation module, and the operation and maintenance operation steps of the expert terminal guide and transmit the position to the information push module, so that the operation and maintenance efficiency is improved, and the operation and maintenance time cost and the labor cost are reduced;
the indoor navigation module generates a ground path to guide operation and maintenance personnel to reach the designated operation and maintenance equipment, the shortest path is found through a Dijkstra algorithm, the operation and maintenance personnel is guided to reach an operation and maintenance point at the highest speed to carry out the next specific operation and maintenance operation, and the expert end transmits the operation and maintenance guiding operation to the information pushing module.
The method comprises the steps that when an operation and maintenance person arrives at equipment needing operation and maintenance, the problem is checked in time, operation and maintenance operation guidance sent by an information pushing module is obtained, remote expert support can be requested when a fault solution is unclear or not understood in the period, if a single expert cannot solve the problem, a request can be sent to a remote multi-person cooperation module to request multi-expert support, and the operation and maintenance person and the multiple experts can achieve communication of a face-to-face effect on AR equipment, so that the fault problem can be solved quickly and effectively, and the operation and maintenance efficiency is improved;
the operation and maintenance data are transmitted to the reviewing and auditing module by recording the data of the operation and maintenance task of each operation and maintenance personnel, so that the purpose of reviewing and auditing the operation and maintenance behavior is achieved.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The above description is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, several modifications and variations can be made without departing from the technical principle of the present invention, and these modifications and variations should also be regarded as the protection scope of the present invention.

Claims (10)

1. The utility model provides a computer lab fortune dimension management system based on AR equipment which characterized in that includes:
a live broadcast module: the AR equipment is used for online communication and information sending service between the AR equipment and the expert end;
augmented reality scene generation module: the system comprises a three-dimensional reconstruction point cloud model used for generating a machine room;
the indoor navigation module: the device is used for guiding and finding equipment marked by the expert end in live broadcasting to carry out operation and maintenance;
the information pushing module: the system comprises an AR device and a server, wherein the AR device is used for collecting operation and maintenance operation steps sent by an expert terminal and pushing the operation and maintenance operation steps to the AR device;
a fault detection module: the system is used for reporting faults when the polling personnel find equipment faults;
remote multi-person collaboration module: the system is used for remote watching of the visitor and guidance of one-to-many inspection personnel by the expert;
a review audit module: used for viewing historical operation and maintenance records;
the AR device: the system is used for creating a chat room for direct broadcasting of experts and inspection personnel, displaying an indoor navigation path, a machine room scene model and pushing strategy guidance information sent by an expert terminal;
an expert end: the system is used for accessing a live broadcast module and sending strategy guide information;
a database: for storing data and records generated during the operation of the system.
2. The machine room operation and maintenance management system according to claim 1, wherein the AR device comprises an AR device camera and a microphone;
the live broadcast module obtains audio and video data streams from an AR equipment camera and a microphone, acquires, codes and decodes, encrypts, media files, processes images, displays, transmits over the network and performs flow control processing on the video data streams through a webRTC, and performs equipment, coding and decoding, encryption, sound files, sound processing, sound output, volume control, audio and video synchronization, network transmission and flow control processing on the audio data streams to obtain real-time audio and video, and transmits the real-time audio and video to an indoor navigation module, an information push module, a fault detection module, a remote multi-user cooperation module and a review audit module;
the augmented reality scene generation module acquires a machine room depth image from the AR equipment camera, synchronously positions and maps the machine room to obtain a three-dimensional reconstruction cloud map of the machine room, and transmits the three-dimensional reconstruction cloud map to the indoor navigation module;
the indoor navigation module obtains a machine room three-dimensional reconstruction cloud picture of the augmented reality scene generation module and a navigation target point marked by a live broadcast module expert, generates a navigation planning path for operation and maintenance personnel to reach the nearest operation and maintenance equipment, and transmits the navigation planning path to an operation and maintenance personnel end of the live broadcast module;
the information pushing module obtains language explanation and video demonstration guidance of a specialist end to a fault processing process through a live broadcast module, codes are transmitted to AR equipment and a review auditing module through the live broadcast module;
the fault detection module acquires fault information of equipment to be inspected, uploads fault content on a fault alarm user interface, and transmits the fault content to the expert terminal, the review audit module and the remote multi-person cooperation module;
the remote multi-person cooperation module joins a current live broadcast room when acquiring a multi-expert remote assistance invitation of the information pushing module, guides operation and maintenance personnel in real time, obtains a solution and transmits the solution to the information pushing module, the live broadcast module and the review auditing module;
the review auditing module obtains contents of a live broadcast screen, a fault solution and fault contents from the live broadcast module, the information pushing module, the fault detection module and the remote multi-person cooperation module;
the augmented reality scene generation module provides a walkable region for the indoor navigation module for path planning.
3. The machine room operation and maintenance management system according to claim 2, wherein the operation method of the system comprises:
the AR device loads a display from a database; the augmented reality scene generation module transmits a three-dimensional model of a machine room to an indoor navigation module and a live broadcast module, a live broadcast module expert transmits the marked position and the operation and maintenance operation steps of operation and maintenance equipment to an information push module, the information push module transmits the position and the position of the operation and maintenance equipment to the indoor navigation module, the operation and maintenance operation steps are transmitted to AR equipment of operation and maintenance personnel, the indoor navigation module generates a navigation path and transmits the navigation path to the AR equipment of the operation and maintenance personnel, and the AR equipment transmits audio and video streams to a remote multi-person cooperation module when the operation and maintenance personnel perform equipment operation and maintenance; all data of the live broadcast module, the augmented reality scene generation module, the indoor navigation module, the information push module, the fault detection module and the remote multi-person cooperation module are transmitted to the review audit module so as to facilitate review audit.
4. The machine room operation and maintenance management system according to claim 2, wherein the AR device comprises a sensor module, a scene understanding module, a three-dimensional sensing module and a virtual-real fusion module;
the sensor module comprises a positioning device, an inertial sensor, an AR equipment camera and a microphone;
the scene understanding module is used for image recognition, object recognition, human body recognition and AR triggering;
the three-dimensional sensing module is used for synchronous positioning and mapping, three-dimensional reconstruction and positioning tracking;
the virtual-real fusion module is used for image rendering;
the origin of the world coordinate system of the AR equipment takes the current equipment position as the standard, and after the machine room two-dimensional code needing to be patrolled and examined is read, the corresponding machine room augmented reality model can be loaded in the AR equipment and completely attached to the actual machine room scene.
5. The machine room operation and maintenance management system according to claim 2, wherein the method for remotely watching by the visitor and guiding by the expert through one-to-many inspection by using the remote multi-person cooperation module comprises the following steps:
the remote multi-person cooperation module is accessed to AR equipment worn by an outside visitor, records all operation and audio-video information of the outside visitor after entering a machine room, and transmits the operation and audio-video information to a manager end;
and acquiring control information of a manager, and monitoring and checking one-to-one or one-to-many real-time pictures according to the control information to realize interaction of audio and video, text information and annotations of the manager and the AR equipment.
6. The machine room operation and maintenance management system according to claim 1, wherein the method for generating the three-dimensional reconstruction point cloud model of the machine room by the augmented reality scene generation module comprises the following steps:
the method comprises the steps of pre-acquiring point clouds of a cabinet and a server in a machine room scene to form a data set, matching the acquired data and the point clouds with objects in the data set during actual scene acquisition, removing noise points and outliers separated from a main body from the acquired data through a filtering algorithm, and selecting effective features for identification according to data characteristics;
the method comprises the steps of sampling peripheral environment information and reconstructing three-dimensional information in an inspection process by adopting a visual reconstruction technology, integrating an inspected 3D image into a real working condition environment and a peripheral environment of a machine room, processing a holographic image of the inspection by combining a data information processing and virtual visual simulation technology, and transmitting a processed reconstruction model to an indoor navigation module;
after the acquired data set is processed, three-dimensional point cloud of the whole machine room is generated; and carrying out surface mounting processing on the obtained point cloud model.
7. The machine room operation and maintenance management system according to claim 1, wherein the method for guiding the finding of the equipment marked by the expert in the live broadcast by the indoor navigation module to perform operation and maintenance comprises the following steps:
after an indoor machine room map is obtained, then, the information of indoor cabinets and servers is added into the indoor map, so that the complete map construction of the machine room is completed;
self-positioning is realized by scanning the two-dimensional code; the method comprises the following steps that an indoor positioning method based on wireless signals is used in a machine room scene, and WiFi is used as basic positioning facilities; planning a navigation path after the position of the AR equipment wearer and the position of the maintenance target are known;
and performing real-time path guidance according to path planning, continuously receiving video frames from the AR equipment camera in the navigation process, capturing natural feature points in a picture, and redrawing the image according to a path planning result, thereby achieving the effect of placing a 3D guide model in the real scene of the AR equipment camera.
8. The machine room operation and maintenance management system according to claim 1, wherein the method for the information pushing module to collect the operation and maintenance operation steps sent by the expert and push the operation and maintenance operation steps to the AR device comprises:
after a patrol inspector logs in a patrol inspection system, a two-dimensional code on an identification device is scanned through an AR device monocular camera, detailed information of the set of equipment cabinet servers can be obtained from a background server, information of a virtual machine in a physical machine is obtained from a cloud management platform in an identification mode, the management platform can push a task work order, equipment information, an operation and maintenance manual and fault diagnosis to the AR device from the background, and the operation and maintenance staff check and call operation and maintenance related information through gesture operation in an MRTK suite, so that operation and maintenance are assisted.
9. The machine room operation and maintenance management system according to claim 1, wherein the review and audit module uploads task contents after each operation and maintenance task is completed by an inspector so that the inspector and an expert can review and review historical operation and maintenance tasks later.
10. An AR machine room operation and maintenance management method, which is based on the system of any one of claims 1-9, and comprises the following steps:
acquiring a preset three-dimensional reconstruction model of a scene to be monitored, wherein the scene to be monitored comprises a plurality of objects to be monitored;
acquiring operation data of an object to be monitored in real time, storing the operation data into a database, updating an integrated machine room reconstruction model of a scene to be monitored, displaying the object to be monitored and the operation data thereof on AR equipment, and generating fault equipment information when a monitoring server has an operation fault;
judging whether the monitored object is in a fault state or not according to the information of the fault detection module, if the object to be monitored is in the fault state, sending an operation and maintenance task to operation and maintenance personnel, planning a path according to the position of specific fault equipment, and guiding the operation and maintenance personnel to reach a server to be operated and maintained;
the operation and maintenance personnel arrive at the to-be-operated and maintained server, live broadcast communication is carried out on the operation and maintenance operation of the communication equipment with the experts, the experts transmit specific operation and maintenance operation manuals to the information pushing module, and if a single expert cannot solve the operation and maintenance of the equipment, the information pushing module transmits assistance requests to the remote multi-person cooperation module;
if the multiple online experts receive the information push invitation to perform remote assistance, the online experts can join the live broadcast module to perform the common remote cooperation of the multiple experts, and instruct operation and maintenance personnel to perform operation and maintenance operation, and visit personnel obtain permission to enter the live broadcast module to view;
the whole equipment operation and maintenance task process participated by operation and maintenance personnel and experts is transmitted to the review audit module, and data is uploaded to the cloud server for storage.
CN202210180987.4A 2022-02-25 2022-02-25 Machine room operation and maintenance management system and method based on AR equipment Pending CN114662714A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210180987.4A CN114662714A (en) 2022-02-25 2022-02-25 Machine room operation and maintenance management system and method based on AR equipment

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210180987.4A CN114662714A (en) 2022-02-25 2022-02-25 Machine room operation and maintenance management system and method based on AR equipment

Publications (1)

Publication Number Publication Date
CN114662714A true CN114662714A (en) 2022-06-24

Family

ID=82028018

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210180987.4A Pending CN114662714A (en) 2022-02-25 2022-02-25 Machine room operation and maintenance management system and method based on AR equipment

Country Status (1)

Country Link
CN (1) CN114662714A (en)

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115314474A (en) * 2022-10-12 2022-11-08 中通服建设有限公司 Communication emergency equipment assists maintenance system based on AR
CN115314684A (en) * 2022-10-10 2022-11-08 中国科学院计算机网络信息中心 Railway bridge inspection method, system, equipment and readable storage medium
CN115334062A (en) * 2022-06-30 2022-11-11 浙江工业大学 Digital factory inspection system based on mixed reality technology
CN116524136A (en) * 2023-07-05 2023-08-01 武汉吧哒科技股份有限公司 Computer room three-dimensional visualization method, system, electronic equipment and readable storage medium
CN116894926A (en) * 2023-09-08 2023-10-17 苏州浪潮智能科技有限公司 Machine room scene display method, device, terminal, readable storage medium and system
CN117351406A (en) * 2023-12-06 2024-01-05 武汉蓝海科创技术有限公司 Expert remote visual auxiliary enabling system based on image recognition technology
CN117692610A (en) * 2024-02-02 2024-03-12 建龙西林钢铁有限公司 AR workshop inspection system

Cited By (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115334062A (en) * 2022-06-30 2022-11-11 浙江工业大学 Digital factory inspection system based on mixed reality technology
CN115334062B (en) * 2022-06-30 2024-05-03 浙江工业大学 Digital factory inspection system based on mixed reality technology
CN115314684A (en) * 2022-10-10 2022-11-08 中国科学院计算机网络信息中心 Railway bridge inspection method, system, equipment and readable storage medium
CN115314474A (en) * 2022-10-12 2022-11-08 中通服建设有限公司 Communication emergency equipment assists maintenance system based on AR
CN115314474B (en) * 2022-10-12 2023-01-17 中通服建设有限公司 Communication emergency equipment assists maintenance system based on AR
CN116524136A (en) * 2023-07-05 2023-08-01 武汉吧哒科技股份有限公司 Computer room three-dimensional visualization method, system, electronic equipment and readable storage medium
CN116524136B (en) * 2023-07-05 2023-10-10 武汉吧哒科技股份有限公司 Computer room three-dimensional visualization method, system, electronic equipment and readable storage medium
CN116894926A (en) * 2023-09-08 2023-10-17 苏州浪潮智能科技有限公司 Machine room scene display method, device, terminal, readable storage medium and system
CN116894926B (en) * 2023-09-08 2024-01-23 苏州浪潮智能科技有限公司 Machine room scene display method, device, terminal, readable storage medium and system
CN117351406A (en) * 2023-12-06 2024-01-05 武汉蓝海科创技术有限公司 Expert remote visual auxiliary enabling system based on image recognition technology
CN117692610A (en) * 2024-02-02 2024-03-12 建龙西林钢铁有限公司 AR workshop inspection system
CN117692610B (en) * 2024-02-02 2024-04-26 建龙西林钢铁有限公司 AR workshop inspection system

Similar Documents

Publication Publication Date Title
CN114662714A (en) Machine room operation and maintenance management system and method based on AR equipment
CN110047150B (en) Complex equipment operation on-site simulation system based on augmented reality
US10977868B2 (en) Remote collaboration methods and systems
CN111526118B (en) Remote operation guiding system and method based on mixed reality
KR101292463B1 (en) Augmented reality system and method that share augmented reality　service to remote
CN110865917A (en) AR technology-based electric power machine room inspection operation method, system and application
CN111612933A (en) Augmented reality intelligent inspection system based on edge cloud server
CN110728670A (en) Low-voltage equipment operation and maintenance method based on AR technology
CN110471804B (en) Equipment debugging method and device and electronic equipment
CN116911823A (en) AR (augmented reality) glasses-based substation inspection interaction method and system
WO2016135402A1 (en) Method and device for remotely inspecting the state of an aircraft engine
CN109889777A (en) The switching methods of exhibiting and system of 3D outdoor scene vision monitoring
KR20200046142A (en) Maintenance work instruction system of city gas facility using IoT platform and augmented reality
CN110070622A (en) A kind of library application system and application method based on AR technology
WO2020045186A1 (en) Information processing device, method, and program
CN112306233A (en) Inspection method, inspection system and inspection management platform
CN112887651A (en) AR-based remote command method and system
CN110992492A (en) Three-dimensional visual production management system and method
CN104092976A (en) Method for achieving environment device monitoring system warning and video monitoring system linkage
CN117424924A (en) Tour enhancement method based on meta-universe interaction
CN110751734B (en) Mixed reality assistant system suitable for job site
CN115469745A (en) Machine room equipment fault processing method and system based on AR technology
Viola et al. VR2Gather: A collaborative, social virtual reality system for adaptive, multiparty real-time communication
CN110889514A (en) Air traffic control station equipment emergency repair remote support system and method
Kleiber et al. User-centered design and evaluation of an integrated AR-VR system for tele-maintenance

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination