CN115334062B - Digital factory inspection system based on mixed reality technology - Google Patents
Digital factory inspection system based on mixed reality technology Download PDFInfo
- Publication number
- CN115334062B CN115334062B CN202210767834.XA CN202210767834A CN115334062B CN 115334062 B CN115334062 B CN 115334062B CN 202210767834 A CN202210767834 A CN 202210767834A CN 115334062 B CN115334062 B CN 115334062B
- Authority
- CN
- China
- Prior art keywords
- data
- interface
- equipment
- real
- user host
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000007689 inspection Methods 0.000 title claims abstract description 35
- 238000005516 engineering process Methods 0.000 title claims abstract description 19
- 238000004891 communication Methods 0.000 claims abstract description 130
- 238000012423 maintenance Methods 0.000 claims abstract description 36
- 238000012545 processing Methods 0.000 claims abstract description 32
- 230000003993 interaction Effects 0.000 claims abstract description 8
- 238000013500 data storage Methods 0.000 claims description 12
- 238000000034 method Methods 0.000 claims description 12
- 238000012795 verification Methods 0.000 claims description 6
- 230000005540 biological transmission Effects 0.000 claims description 3
- 230000000007 visual effect Effects 0.000 claims description 3
- 238000010586 diagram Methods 0.000 description 5
- 230000009286 beneficial effect Effects 0.000 description 1
- 208000013407 communication difficulty Diseases 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
-
- G—PHYSICS
- G07—CHECKING-DEVICES
- G07C—TIME OR ATTENDANCE REGISTERS; REGISTERING OR INDICATING THE WORKING OF MACHINES; GENERATING RANDOM NUMBERS; VOTING OR LOTTERY APPARATUS; ARRANGEMENTS, SYSTEMS OR APPARATUS FOR CHECKING NOT PROVIDED FOR ELSEWHERE
- G07C1/00—Registering, indicating or recording the time of events or elapsed time, e.g. time-recorders for work people
- G07C1/20—Checking timed patrols, e.g. of watchman
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/02—Standardisation; Integration
- H04L41/0246—Exchanging or transporting network management information using the Internet; Embedding network management web servers in network elements; Web-services-based protocols
- H04L41/0253—Exchanging or transporting network management information using the Internet; Embedding network management web servers in network elements; Web-services-based protocols using browsers or web-pages for accessing management information
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/16—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks using machine learning or artificial intelligence
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/22—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks comprising specially adapted graphical user interfaces [GUI]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/16—Implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP]
- H04L69/161—Implementation details of TCP/IP or UDP/IP stack architecture; Specification of modified or new header fields
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Databases & Information Systems (AREA)
- Evolutionary Computation (AREA)
- Medical Informatics (AREA)
- Software Systems (AREA)
- Computer Security & Cryptography (AREA)
- Human Computer Interaction (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
- Computer And Data Communications (AREA)
Abstract
A digital factory inspection system based on a mixed reality technology comprises Hololens 2 equipment, embedded equipment and a user host. The user host computer respectively establishes network connection with the HoloLens 2 equipment and the embedded equipment, receives the instruction and the data sent by the HoloLens 2 equipment on one hand, and returns a processing result according to the instruction; on the other hand, the real-time state data of the device sent by the embedded device is received, and the data is stored in a database according to a data format agreed in advance. The HoloLens 2 device obtains input information or instructions of the user through the first UI interface and the second UI interface, and generates data interaction with the user host through a TCP/IP protocol or a WebRTC protocol according to actual conditions. The embedded device is used for acquiring real-time state data of the device and sending the real-time state data to the user host for processing through a TCP/IP network protocol. The invention is convenient for maintenance personnel to find equipment problems in time according to the data query result, improves the communication efficiency between remote experts and maintenance personnel, and improves the factory digital level.
Description
Technical Field
The invention relates to the technical field of digital twinning, in particular to a digital factory inspection system based on a mixed reality technology.
Background
The digital twin technology is a computer technology for creating a virtual model of a physical entity in a digital manner and fully utilizing operation data of the physical entity. The full life cycle process of the physical entity is shown through the technologies of virtual-real interaction, data fusion and the like, and more real-time and intelligent service is provided for the industrial field. Holonens 2 is a head-mounted device which is put forward by microsoft and applies digital twin technology, and can superimpose a virtual image generated by a computer in an actual scene, so that a wearer can obtain a virtual-real combined environment experience.
The traditional method of factory inspection is realized by methods of observing an instrument panel, checking equipment states and the like by maintenance personnel, and the methods have the problems that the operation data acquisition mode is complex, the equipment states can be acquired only through an indicator lamp and the like. The digital factory can transmit the real-time operation data of the device to the remote host and store the real-time operation data, and the acquisition of the real-time operation data can be realized by using proper devices. In order to improve the working efficiency of a manual inspection mode and the level of factory digitalization, hao Fangzhou and the like, a power distribution room on-site inspection method and a system (Hao Fangzhou; zhao Hongshan; qu Yue) based on a holonens mixed reality technology are provided, the power distribution room on-site inspection method and the system [ P ] based on the holonens mixed reality technology are provided, the system performs functions of identification, data reading and the like on equipment in a power distribution room through an image acquisition technology, positioning of inspection personnel is achieved by combining SLAM technology and IMU data, and an inspection route scheme can be formulated according to indoor equipment. The method realizes the rapid and efficient acquisition of real-time operation data of equipment and the superposition of a virtual panel displaying state data and physical equipment.
Both patents propose a method capable of greatly improving the working efficiency of maintenance personnel based on HoloLens, but when maintenance personnel seek help from a remote expert, the remote expert cannot obtain the real-time condition of the current factory end, so that the communication difficulty between the maintenance personnel and the expert is greatly increased.
Disclosure of Invention
The invention provides a digital factory inspection system based on a mixed reality technology, which aims to improve the working efficiency of maintenance personnel and the digital level of a factory and realize high-efficiency communication between the maintenance personnel and a remote expert.
The invention discloses a digital factory inspection system based on a mixed reality technology, which comprises HoloLens 2 equipment, embedded equipment and a user host. The user host, the HoloLens 2 device and the embedded device respectively establish network connection, on one hand, the user host receives the instruction and the data sent by the HoloLens 2 device, and returns a processing result according to the instruction; and on the other hand, receiving and storing the real-time state data of the device sent by the embedded device. The HoloLens 2 device obtains input information or instructions of the user through the first UI interface and the second UI interface, and generates data interaction with the user host through a network protocol according to actual conditions. The embedded device is used for acquiring real-time state data of the device and transmitting the real-time state data to the user host for processing through network connection by using a TCP/IP protocol.
The HoloLens 2 device comprises a first UI interface, a network communication subsystem, a job auxiliary subsystem and a patrol auxiliary subsystem. The first UI provides visual operation interfaces and keys for maintenance personnel, and the operation auxiliary subsystem and the inspection auxiliary subsystem realize data interaction with a user host through the network communication subsystem according to actual requirements;
the first UI comprises a login interface, a registration interface and a function selection interface;
The login interface is characterized in that a maintainer inputs an account number and a password of the maintainer to an account number column and a password column of the login interface presented on the HoloLens 2, then clicks a login button, the HoloLens 2 calls a TCP/IP communication module in the network communication subsystem to send account number and password data to the user host for verification, and whether to switch into a function selection interface is judged according to a verification result returned by the user host;
The registration interface is used for inputting an account number and a password to be registered into an account number column and a password column of the registration interface presented on the HoloLens 2 by a maintainer, clicking a registration button, calling a TCP/IP communication module in a network communication subsystem by the HoloLens 2 to send account number and password data to a user host for registration, and judging whether to switch into a function selection interface according to a received registration result returned by the user host;
The function selection interface enables maintenance personnel to select to enter the operation auxiliary subsystem or the inspection auxiliary subsystem after successful login or registration;
The network communication subsystem comprises a TCP/IP communication module and a WebRTC communication module;
The TCP/IP communication module establishes network communication connection with the user host by using a TCP/IP protocol, and is used for transmitting information such as historical state data, real-time state data, equipment names, account passwords of maintenance personnel and the like of equipment. The WebRTC communication module adopts the WebRTC protocol to realize communication connection between HoloLens 2 and a user host, and is used for realizing real-time audio and video communication between a remote expert and a maintainer and realizing the function of remote expert guidance.
The operation auxiliary subsystem comprises a second UI interface, a historical data query module and a remote expert assistance module;
The second UI interface comprises a historical data query interface and a remote communication interface in the operation auxiliary subsystem;
The historical data query interface is used for inputting the historical data to be queried at a designated position of the interface by a maintainer, the historical data query interface can send the data input by the maintainer to the historical data query module for further processing, and after the historical data query processing is finished, the historical data query interface can display a query result at the designated position;
The remote communication interface is used for enabling maintenance personnel to click a start communication button on the remote communication interface, a communication instruction is issued to the remote assistance module for processing, and after the remote assistance module finishes processing, the maintenance personnel can see real-time pictures of a remote expert and a local end at a designated position on the remote communication interface so as to realize the assistance of the remote expert;
The historical data query module calls the TCP/IP communication module after receiving the data issued by the historical data query interface, acquires the data to be queried from the user host according to a data communication format appointed in advance, and uploads the data to the historical data query interface;
And the remote expert assistance module is used for calling the WebRTC communication module to establish WebRTC communication connection with the user host after receiving a communication instruction issued by the remote communication interface, and is used for transmitting the received WebRTC video stream returned by the user host to the remote communication interface for display after the connection is successful.
The inspection auxiliary subsystem comprises a second UI interface and a device real-time state query module;
The second UI comprises a real-time equipment state query interface in the inspection auxiliary subsystem;
And after entering the equipment real-time state query interface, a maintainer can select a manual input equipment name on the interface to query the real-time state data of the equipment or select an inspection auxiliary subsystem to automatically complete the real-time state data query of the current equipment. After the selection is finished, the second UI interface transmits the selected mode and the data to be queried to the equipment real-time state query module, and after the equipment real-time state query module finishes processing, the query result is displayed at the appointed position;
the device real-time state query module selects a manual mode to input a device name query state data mode, and after receiving data issued by a device real-time state query interface, the device real-time state query module calls the TCP/IP communication module to send the input data to the user host according to a preset data transmission format, and obtains the device real-time state data returned by the user host; and selecting an automatic state data inquiring mode of the inspection auxiliary subsystem, calling a TCP/IP communication module by the equipment real-time state inquiring module to send the image acquired by the HoloLens 2 to the user host, identifying the equipment name by the user host, and returning corresponding state data to the equipment real-time state inquiring module. Whichever inquiry mode is selected, the device real-time state inquiry module can upload inquiry results to a device real-time state inquiry interface for further processing.
The user host comprises a Web interface, a network communication subsystem, a data storage subsystem and an object identification subsystem;
The Web interface is used for acquiring real-time voice and images of a maintenance personnel end through the remote expert after connection is established between the Hololens 2 and the user host through a WebRTC protocol, and transmitting the real-time voice and images of the expert to the maintenance personnel to realize real-time audio and video communication between the Hololens 2 and the user host;
the network communication subsystem comprises a TCP/IP communication module and a WebRTC communication module;
The TCP/IP communication module establishes network communication connection by using a TCP/IP protocol, holoLens 2 and embedded equipment, and is used for transmitting information such as historical data, real-time state data, equipment names, account passwords of maintenance personnel and the like of the equipment. The WebRTC communication module establishes communication connection with HoloLens 2 by adopting a WebRTC protocol and is used for realizing real-time audio and video communication between a remote expert and a maintainer;
The data storage subsystem establishes TCP/IP connection between the user host and the HoloLens 2, receives an instruction sent by the HoloLens 2, and stores or reads data according to the instruction; the method comprises the steps that TCP/IP communication is established between a user host and embedded equipment, and real-time state data of the equipment are received and stored according to a preset data format;
And the object recognition subsystem establishes TCP/IP communication connection between the HoloLens 2 and the user host, and the user host inputs the received image into an object recognition algorithm for processing according to the instruction. And after the identification is finished, sending the identification result to a data storage subsystem for processing, and sending real-time state data of the equipment to HoloLens 2 by the data storage subsystem according to the identified equipment name.
The embedded equipment is arranged on the equipment, is positioned in the same network section with the user host computer through the Ethernet, and mainly acquires real-time state data of the equipment and sends the real-time state data to the user host computer through a TCP/IP communication protocol.
Preferably, the main control chip model of the embedded device is STM32F103ZET6.
Preferably, the object recognition algorithm is YOLOv's 4 algorithm.
The historical data includes equipment operation data and maintenance records.
Compared with the prior art, the invention has the beneficial effects that: the system is added with historical state data and equipment real-time state data query functions, so that maintenance personnel can find equipment problems in time according to data query results; the real-time audio and video communication between HoloLens 2 and the user host is realized by adopting the WebRTC communication protocol, so that the communication efficiency between a remote expert and maintenance personnel is improved; the system designs two methods of manually inputting the name of the equipment and identifying the equipment in real time by using an object identification algorithm, so that the flexibility of the system is improved; the state data of the equipment is sent to the user host, so that the storage capacity of the data is improved, the unified management efficiency of the data is improved, and the digital level of a factory is improved.
Drawings
Fig. 1 is a system block diagram of the present invention.
Fig. 2 is a block diagram of the holonens 2 system in the present invention.
Fig. 3 is a system block diagram of a subscriber host in the present invention.
Detailed Description
Examples of the invention are described in further detail below with reference to the accompanying drawings:
As shown in FIG. 1, the digital factory inspection system based on the mixed reality technology comprises Hololens 2 equipment, embedded equipment and a user host. The user host, the HoloLens 2 device and the embedded device respectively establish network connection, on one hand, the user host receives the instruction and the data sent by the HoloLens 2 device, and returns a processing result according to the instruction; and on the other hand, receiving and storing the real-time state data of the device sent by the embedded device. The Hololens 2 equipment acquires input information or instructions of a user through a UI interface, and generates data interaction with a user host through a TCP/IP protocol or a WebRTC protocol according to actual conditions. The embedded device is used for acquiring real-time state data of the device and transmitting the real-time state data to the user host for processing through network connection by using a TCP/IP protocol.
The HoloLens 2 system block diagram is shown in fig. 2, and the HoloLens 2 device comprises a first UI interface, a network communication subsystem, a work auxiliary subsystem and a patrol auxiliary subsystem. The first UI provides visual operation interfaces and keys for maintenance personnel, and the operation auxiliary subsystem and the inspection auxiliary subsystem realize data interaction with a user host through the network communication subsystem according to actual requirements;
the first UI comprises a login interface, a registration interface and a function selection interface;
The login interface is characterized in that a maintainer inputs an account number and a password of the maintainer to an account number column and a password column of the login interface presented on the HoloLens 2, then clicks a login button, the HoloLens 2 calls a TCP/IP communication module in the network communication subsystem to send account number and password data to the user host for verification, and whether to switch into a function selection interface is judged according to a verification result returned by the user host;
The registration interface is used for inputting an account number and a password to be registered into an account number column and a password column of the registration interface presented on the HoloLens 2 by a maintainer, clicking a registration button, calling a TCP/IP communication module in a network communication subsystem by the HoloLens 2 to send account number and password data to a user host for registration, and judging whether to switch into a function selection interface according to a received registration result returned by the user host;
The function selection interface enables maintenance personnel to select to enter the operation auxiliary subsystem or the inspection auxiliary subsystem after successful login or registration;
The network communication subsystem comprises a TCP/IP communication module and a WebRTC communication module;
The TCP/IP communication module establishes network communication connection with the user host by using a TCP/IP protocol, and is used for transmitting information such as historical state data, real-time state data, equipment names, account passwords of maintenance personnel and the like of equipment. The WebRTC communication module adopts the WebRTC protocol to realize communication connection between HoloLens 2 and a user host, and is used for realizing real-time audio and video communication between a remote expert and a maintainer and realizing the function of remote expert guidance.
The operation auxiliary subsystem comprises a second UI interface, a historical data query module and a remote expert assistance module;
The second UI interface comprises a historical data query interface and a remote communication interface in the operation auxiliary subsystem;
The historical data query interface is used for inputting the historical data to be queried at a designated position of the interface by a maintainer, the historical data query interface can send the data input by the maintainer to the historical data query module for further processing, and after the historical data query processing is finished, the historical data query interface can display a query result at the designated position;
The remote communication interface is used for enabling maintenance personnel to click a start communication button on the remote communication interface, a communication instruction is issued to the remote assistance module for processing, and after the remote assistance module finishes processing, the maintenance personnel can see real-time pictures of a remote expert and a local end at a designated position on the remote communication interface so as to realize the assistance of the remote expert;
The historical data query module calls the TCP/IP communication module after receiving the data issued by the historical data query interface, acquires the data to be queried from the user host according to a data communication format appointed in advance, and uploads the data to the historical data query interface;
And the remote expert assistance module is used for calling the WebRTC communication module to establish WebRTC communication connection with the user host after receiving a communication instruction issued by the remote communication interface, and is used for transmitting the received WebRTC video stream returned by the user host to the remote communication interface for display after the connection is successful.
The inspection auxiliary subsystem comprises a second UI interface and a device real-time state query module;
The second UI comprises a real-time equipment state query interface in the inspection auxiliary subsystem;
And after entering the equipment real-time state query interface, a maintainer can select a manual input equipment name on the interface to query the real-time state data of the equipment or select an inspection auxiliary subsystem to automatically complete the real-time state data query of the current equipment. After the selection is finished, the second UI interface transmits the selected mode and the data to be queried to the equipment real-time state query module, and after the equipment real-time state query module finishes processing, the query result is displayed at the appointed position;
the device real-time state query module selects a manual mode to input a device name query state data mode, and after receiving data issued by a device real-time state query interface, the device real-time state query module calls the TCP/IP communication module to send the input data to the user host according to a preset data transmission format, and obtains the device real-time state data returned by the user host; and selecting an automatic state data inquiring mode of the inspection auxiliary subsystem, calling a TCP/IP communication module by the equipment real-time state inquiring module to send the image acquired by the HoloLens 2 to the user host, identifying the equipment name by the user host, and returning corresponding state data to the equipment real-time state inquiring module. Whichever inquiry mode is selected, the device real-time state inquiry module can upload inquiry results to a device real-time state inquiry interface for further processing.
The system block diagram of the user host is shown in fig. 3, and comprises a Web interface, a network communication subsystem, a data storage subsystem and an object recognition subsystem;
The Web interface is used for acquiring real-time voice and images of a maintenance personnel end through the remote expert after connection is established between the Hololens 2 and the user host through a WebRTC protocol, and transmitting the real-time voice and images of the expert to the maintenance personnel to realize real-time audio and video communication between the Hololens 2 and the user host;
the network communication subsystem comprises a TCP/IP communication module and a WebRTC communication module;
The TCP/IP communication module establishes network communication connection by using a TCP/IP protocol, holoLens 2 and embedded equipment, and is used for transmitting information such as historical data, real-time state data, equipment names, account passwords of maintenance personnel and the like of the equipment. The WebRTC communication module establishes communication connection with HoloLens 2 by adopting a WebRTC protocol and is used for realizing real-time audio and video communication between a remote expert and a maintainer;
The data storage subsystem establishes TCP/IP connection between the user host and the HoloLens 2, receives an instruction sent by the HoloLens 2, and stores or reads data according to the instruction; the method comprises the steps that TCP/IP communication is established between a user host and embedded equipment, and real-time state data of the equipment are received and stored according to a preset data format;
And the object recognition subsystem establishes TCP/IP communication connection between the HoloLens 2 and the user host, and the user host inputs the received image into an object recognition algorithm for processing according to the instruction. And after the identification is finished, sending the identification result to a data storage subsystem for processing, and sending real-time state data of the equipment to HoloLens 2 by the data storage subsystem according to the identified equipment name.
The embodiments described in the present specification are merely examples of implementation forms of the inventive concept, and the scope of protection of the present invention should not be construed as being limited to the implementation of the specific forms described, but the scope of protection of the present invention and equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.
Claims (4)
1. A digital factory inspection system based on a mixed reality technology comprises Hololens 2 equipment, embedded equipment and a user host; the user host, the HoloLens 2 device and the embedded device respectively establish network connection, on one hand, the user host receives the instruction and the data sent by the HoloLens 2 device, and returns a processing result according to the instruction; on the other hand, receiving and storing the real-time state data of the equipment, which is sent by the embedded equipment; the HoloLens 2 device obtains input information or instructions of a user through the first UI interface and the second UI interface, and generates data interaction with a user host through a network protocol according to actual conditions; the embedded equipment is used for acquiring real-time state data of the equipment and transmitting the real-time state data to the user host for processing by using a TCP/IP protocol through network connection;
The HoloLens 2 device comprises a first UI interface, a network communication subsystem, a work auxiliary subsystem and a patrol auxiliary subsystem; the first UI provides visual operation interfaces and keys for maintenance personnel, and the operation auxiliary subsystem and the inspection auxiliary subsystem realize data interaction with a user host through the network communication subsystem according to actual requirements;
the first UI comprises a login interface, a registration interface and a function selection interface;
The login interface is characterized in that a maintainer inputs an account number and a password of the maintainer to an account number column and a password column of the login interface presented on the HoloLens 2, then clicks a login button, the HoloLens 2 calls a TCP/IP communication module in the network communication subsystem to send account number and password data to the user host for verification, and whether to switch into a function selection interface is judged according to a verification result returned by the user host;
The registration interface is used for inputting an account number and a password to be registered into an account number column and a password column of the registration interface presented on the HoloLens 2 by a maintainer, clicking a registration button, calling a TCP/IP communication module in a network communication subsystem by the HoloLens 2 to send account number and password data to a user host for registration, and judging whether to switch into a function selection interface according to a received registration result returned by the user host;
The function selection interface enables maintenance personnel to select to enter the operation auxiliary subsystem or the inspection auxiliary subsystem after successful login or registration;
The network communication subsystem comprises a TCP/IP communication module and a WebRTC communication module;
The TCP/IP communication module establishes network communication connection with the user host by using a TCP/IP protocol and is used for transmitting information such as historical state data, real-time state data, equipment names, account passwords of maintenance personnel and the like of the equipment; the WebRTC communication module adopts the WebRTC protocol to realize communication connection between HoloLens 2 and a user host, and is used for realizing real-time audio and video communication between a remote expert and a maintainer and realizing the function of remote expert guidance;
the operation auxiliary subsystem comprises a second UI interface, a historical data query module and a remote expert assistance module;
The second UI interface comprises a historical data query interface and a remote communication interface in the operation auxiliary subsystem;
The historical data query interface is used for inputting the historical data to be queried at a designated position of the interface by a maintainer, the historical data query interface can send the data input by the maintainer to the historical data query module for further processing, and after the historical data query processing is finished, the historical data query interface can display a query result at the designated position;
The remote communication interface is used for enabling maintenance personnel to click a start communication button on the remote communication interface, a communication instruction is issued to the remote assistance module for processing, and after the remote assistance module finishes processing, the maintenance personnel can see real-time pictures of a remote expert and a local end at a designated position on the remote communication interface so as to realize the assistance of the remote expert;
The historical data query module calls the TCP/IP communication module after receiving the data issued by the historical data query interface, acquires the data to be queried from the user host according to a data communication format appointed in advance, and uploads the data to the historical data query interface;
The remote expert assistance module calls the WebRTC communication module to establish WebRTC communication connection with the user host after receiving a communication instruction issued by the remote communication interface, and the remote expert assistance module transmits the received WebRTC video stream returned by the user host to the remote communication interface for display after the connection is successful;
the inspection auxiliary subsystem comprises a second UI interface and a device real-time state query module;
The second UI comprises a real-time equipment state query interface in the inspection auxiliary subsystem;
The equipment real-time state query interface, after entering the equipment real-time state query interface, a maintainer can select a manual input equipment name on the interface to query the real-time state data of the equipment or select an inspection auxiliary subsystem to automatically complete the real-time state data query of the current equipment; after the selection is finished, the second UI interface transmits the selected mode and the data to be queried to the equipment real-time state query module, and after the equipment real-time state query module finishes processing, the query result is displayed at the appointed position;
The device real-time state query module selects a manual mode to input a device name query state data mode, and after receiving data issued by a device real-time state query interface, the device real-time state query module calls the TCP/IP communication module to send the input data to the user host according to a preset data transmission format, and obtains the device real-time state data returned by the user host; selecting an automatic state data inquiring mode of the inspection auxiliary subsystem, calling a TCP/IP communication module by the equipment real-time state inquiring module to send an image acquired by HoloLens 2 to a user host, identifying the equipment name by the user host, and returning corresponding state data to the equipment real-time state inquiring module; whichever inquiry mode is selected, the equipment real-time state inquiry module can upload inquiry results to the equipment real-time state inquiry interface for further processing;
the user host comprises a Web interface, a network communication subsystem, a data storage subsystem and an object identification subsystem;
The Web interface is used for acquiring real-time voice and images of a maintenance personnel end through the remote expert after connection is established between the Hololens 2 and the user host through a WebRTC protocol, and transmitting the real-time voice and images of the expert to the maintenance personnel to realize real-time audio and video communication between the Hololens 2 and the user host;
the network communication subsystem comprises a TCP/IP communication module and a WebRTC communication module;
The TCP/IP communication module establishes network communication connection by using a TCP/IP protocol, holoLens 2 and embedded equipment, and is used for transmitting information such as historical data, real-time state data, equipment names, account passwords of maintenance personnel and the like of the equipment; the WebRTC communication module establishes communication connection with HoloLens 2 by adopting a WebRTC protocol and is used for realizing real-time audio and video communication between a remote expert and a maintainer;
The data storage subsystem establishes TCP/IP connection between the user host and the HoloLens 2, receives an instruction sent by the HoloLens 2, and stores or reads data according to the instruction; the method comprises the steps that TCP/IP communication is established between a user host and embedded equipment, and real-time state data of the equipment are received and stored according to a preset data format;
The object recognition subsystem establishes TCP/IP communication connection between HoloLens 2 and the user host, and the user host inputs the received image into an object recognition algorithm for processing according to the instruction; after the identification is finished, the identification result is sent to the data storage subsystem for processing, and the data storage subsystem sends the real-time state data of the equipment to HoloLens 2 according to the identified equipment name;
the embedded equipment is arranged on the equipment, is positioned in the same network section with the user host computer through the Ethernet, and mainly acquires real-time state data of the equipment and sends the real-time state data to the user host computer through a TCP/IP communication protocol.
2. The digital factory inspection system based on mixed reality technology of claim 1, wherein: the main control chip model of the embedded equipment is STM32F103ZET6.
3. The digital factory inspection system based on mixed reality technology of claim 1, wherein: the object recognition algorithm adopts YOLOv's 4 algorithm.
4. The digital factory inspection system based on mixed reality technology of claim 1, wherein: the historical data includes equipment operational data and maintenance records.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210767834.XA CN115334062B (en) | 2022-06-30 | 2022-06-30 | Digital factory inspection system based on mixed reality technology |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210767834.XA CN115334062B (en) | 2022-06-30 | 2022-06-30 | Digital factory inspection system based on mixed reality technology |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115334062A CN115334062A (en) | 2022-11-11 |
| CN115334062B true CN115334062B (en) | 2024-05-03 |
Family
ID=83918064
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210767834.XA Active CN115334062B (en) | 2022-06-30 | 2022-06-30 | Digital factory inspection system based on mixed reality technology |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115334062B (en) |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107578487A (en) * | 2017-09-19 | 2018-01-12 | 北京枭龙科技有限公司 | A kind of cruising inspection system based on augmented reality smart machine |
| WO2018187748A1 (en) * | 2017-04-07 | 2018-10-11 | Unveil, LLC | Systems and methods for mixed reality medical training |
| CN108874153A (en) * | 2018-08-23 | 2018-11-23 | 上海机器人产业技术研究院有限公司 | One kind being based on AR glasses cloud robot holography review system and method |
| CN110246235A (en) * | 2019-06-18 | 2019-09-17 | 广州供电局有限公司 | A kind of power distribution room scene method for inspecting and system based on Hololens mixed reality technology |
| KR102086451B1 (en) * | 2019-09-06 | 2020-05-29 | 주식회사 글로벌제조혁신네트웍 | Smart Factory remote collaboration CMS system in augmented reality and Drive method of the Same |
| CN112578907A (en) * | 2020-12-01 | 2021-03-30 | 国电湖南巫水水电开发有限公司 | Method and device for realizing remote guidance operation based on AR |
| CN112671898A (en) * | 2020-12-23 | 2021-04-16 | 浙江工业大学 | Remote communication control system based on embedded equipment |
| CN112732072A (en) * | 2020-12-21 | 2021-04-30 | 航天信息股份有限公司 | Equipment intelligent maintenance system based on VR/AR |
| CN112929384A (en) * | 2021-03-05 | 2021-06-08 | 瑞丰宝丽(北京)科技有限公司 | AR intelligent point inspection system based on space anchor point |
| CN113297945A (en) * | 2021-05-18 | 2021-08-24 | 北京远舢智能科技有限公司 | Indoor equipment inspection auxiliary method and system based on mixed reality space positioning |
| CN113936120A (en) * | 2021-10-12 | 2022-01-14 | 北京邮电大学 | Mark-free lightweight Web AR method and system |
| CN114662714A (en) * | 2022-02-25 | 2022-06-24 | 南京邮电大学 | Machine room operation and maintenance management system and method based on AR equipment |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180301048A1 (en) * | 2017-04-12 | 2018-10-18 | Age Of Learning, Inc. | Remote live tutoring platform |
| US20190096130A1 (en) * | 2017-09-26 | 2019-03-28 | Akn Korea Inc. | Virtual mobile terminal implementing system in mixed reality and control method thereof |
-
2022
- 2022-06-30 CN CN202210767834.XA patent/CN115334062B/en active Active
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018187748A1 (en) * | 2017-04-07 | 2018-10-11 | Unveil, LLC | Systems and methods for mixed reality medical training |
| CN107578487A (en) * | 2017-09-19 | 2018-01-12 | 北京枭龙科技有限公司 | A kind of cruising inspection system based on augmented reality smart machine |
| CN108874153A (en) * | 2018-08-23 | 2018-11-23 | 上海机器人产业技术研究院有限公司 | One kind being based on AR glasses cloud robot holography review system and method |
| CN110246235A (en) * | 2019-06-18 | 2019-09-17 | 广州供电局有限公司 | A kind of power distribution room scene method for inspecting and system based on Hololens mixed reality technology |
| KR102086451B1 (en) * | 2019-09-06 | 2020-05-29 | 주식회사 글로벌제조혁신네트웍 | Smart Factory remote collaboration CMS system in augmented reality and Drive method of the Same |
| CN112578907A (en) * | 2020-12-01 | 2021-03-30 | 国电湖南巫水水电开发有限公司 | Method and device for realizing remote guidance operation based on AR |
| CN112732072A (en) * | 2020-12-21 | 2021-04-30 | 航天信息股份有限公司 | Equipment intelligent maintenance system based on VR/AR |
| CN112671898A (en) * | 2020-12-23 | 2021-04-16 | 浙江工业大学 | Remote communication control system based on embedded equipment |
| CN112929384A (en) * | 2021-03-05 | 2021-06-08 | 瑞丰宝丽(北京)科技有限公司 | AR intelligent point inspection system based on space anchor point |
| CN113297945A (en) * | 2021-05-18 | 2021-08-24 | 北京远舢智能科技有限公司 | Indoor equipment inspection auxiliary method and system based on mixed reality space positioning |
| CN113936120A (en) * | 2021-10-12 | 2022-01-14 | 北京邮电大学 | Mark-free lightweight Web AR method and system |
| CN114662714A (en) * | 2022-02-25 | 2022-06-24 | 南京邮电大学 | Machine room operation and maintenance management system and method based on AR equipment |
Non-Patent Citations (2)
| Title |
|---|
| "Proof of Concept for Development of Collaborative Mixed Reality Applications from 3D City Models";Y. -S. Chang等;《2019 International Conference on Information and Communication Technology Convergence (ICTC)》;20191127;全文 * |
| 采用HoloLens进行变电站巡检的作业信息管理系统解决方案;潘建乔;吴迪;余方召;潘白浪;龚雁峰;郭国栋;;电力设备管理;20200125(01);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115334062A (en) | 2022-11-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN1282340C (en) | Remote service processing appts. utilizing graphic users interface in home network environment | |
| AU7853798A (en) | Camera control system controlling different types of cameras | |
| CN102438173A (en) | Method and system for remotely controlling set top box (STB) | |
| CN110909898A (en) | Zero-knowledge bank machine room fault diagnosis and maintenance guidance system and method based on AR glasses | |
| CN102158685B (en) | System, system construction method, management terminal, and program | |
| CN110210793A (en) | A kind of project sites construction remote monitoring platform, system and method | |
| CN109299908A (en) | A kind of remote visualization assistance system | |
| JP4327545B2 (en) | Field monitoring system and field monitoring method | |
| CN109936814A (en) | A kind of intercommunication terminal, speech talkback coordinated dispatching method and its system | |
| CN115334062B (en) | Digital factory inspection system based on mixed reality technology | |
| JP2002374584A (en) | Remote control, control signal distributor, control signal acquisition method, control signal distribution method, drive signal generator, and access service system | |
| JP2002034026A (en) | Remote monitor system | |
| KR101184760B1 (en) | System for delivering urgent disaster information and method | |
| CN106231260B (en) | Transformer substation video monitoring method, device and system based on threedimensional model | |
| CN110213344B (en) | Multi-center remote sign language online translation system and method | |
| JP2001344189A (en) | Computer changeover device | |
| CN211906347U (en) | Remote after-sale system | |
| JP2002009868A (en) | Information transmission system, information managing device and setting information distribution method | |
| CN104581025A (en) | Image monitoring system and method | |
| CN210181654U (en) | Teaching device for realizing cloud platform management by scanning two-dimensional code | |
| KR102166954B1 (en) | Electric power solution management method using augmented realitybased on position | |
| CN116700498A (en) | Mechanical equipment monitoring and management system based on mixed reality | |
| CN112348967A (en) | Seamless fusion method for three-dimensional model and real-time video of power equipment | |
| CN109547919B (en) | Method for remotely managing operation site | |
| CN111782160B (en) | Multi-machine multi-source common screen control device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |