CN115334062A - Digital factory inspection system based on mixed reality technology - Google Patents

Digital factory inspection system based on mixed reality technology Download PDF

Info

Publication number
CN115334062A
CN115334062A CN202210767834.XA CN202210767834A CN115334062A CN 115334062 A CN115334062 A CN 115334062A CN 202210767834 A CN202210767834 A CN 202210767834A CN 115334062 A CN115334062 A CN 115334062A
Authority
CN
China
Prior art keywords
interface
data
equipment
real
user host
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202210767834.XA
Other languages
Chinese (zh)
Other versions
CN115334062B (en
Inventor
禹鑫燚
赵嘉楠
喻志祥
欧林林
周利波
魏岩
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Zhejiang University of Technology ZJUT
Original Assignee
Zhejiang University of Technology ZJUT
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Zhejiang University of Technology ZJUT filed Critical Zhejiang University of Technology ZJUT
Priority to CN202210767834.XA priority Critical patent/CN115334062B/en
Publication of CN115334062A publication Critical patent/CN115334062A/en
Application granted granted Critical
Publication of CN115334062B publication Critical patent/CN115334062B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
    • G—PHYSICS
    • G07—CHECKING-DEVICES
    • G07C—TIME OR ATTENDANCE REGISTERS; REGISTERING OR INDICATING THE WORKING OF MACHINES; GENERATING RANDOM NUMBERS; VOTING OR LOTTERY APPARATUS; ARRANGEMENTS, SYSTEMS OR APPARATUS FOR CHECKING NOT PROVIDED FOR ELSEWHERE
    • G07C1/00—Registering, indicating or recording the time of events or elapsed time, e.g. time-recorders for work people
    • G07C1/20—Checking timed patrols, e.g. of watchman
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/02—Standardisation; Integration
    • H04L41/0246—Exchanging or transporting network management information using the Internet; Embedding network management web servers in network elements; Web-services-based protocols
    • H04L41/0253—Exchanging or transporting network management information using the Internet; Embedding network management web servers in network elements; Web-services-based protocols using browsers or web-pages for accessing management information
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/16—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks using machine learning or artificial intelligence
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/22—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks comprising specially adapted graphical user interfaces [GUI]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
    • H04L69/16—Implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP]
    • H04L69/161—Implementation details of TCP/IP or UDP/IP stack architecture; Specification of modified or new header fields

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Physics & Mathematics (AREA)
  • Artificial Intelligence (AREA)
  • Physics & Mathematics (AREA)
  • Databases & Information Systems (AREA)
  • Evolutionary Computation (AREA)
  • Medical Informatics (AREA)
  • Software Systems (AREA)
  • Computer Security & Cryptography (AREA)
  • Human Computer Interaction (AREA)
  • Data Exchanges In Wide-Area Networks (AREA)
  • Computer And Data Communications (AREA)

Abstract

A digital factory inspection system based on a mixed reality technology comprises a HoloLens 2 device, an embedded device and a user host. The user host establishes network connection with the HoloLens 2 device and the embedded device respectively, receives the instruction and data sent by the HoloLens 2 device on one hand, and returns a processing result according to the instruction; and on the other hand, receiving the real-time state data of the equipment sent by the embedded equipment, and storing the data into a database according to a predetermined data format. The HoloLens 2 device acquires input information or instructions of a user through the first UI interface and the second UI interface, and generates data interaction with a user host through a TCP/IP protocol or a WebRTC protocol according to actual conditions. The embedded device is used for acquiring real-time state data of the device and sending the real-time state data to the user host for processing through a TCP/IP network protocol. The invention is convenient for maintenance personnel to find out equipment problems in time according to data query results, improves the communication efficiency between remote experts and the maintenance personnel, and improves the digitization level of a factory.

Description

Digital factory inspection system based on mixed reality technology
Technical Field
The invention relates to the technical field of digital twins, in particular to a digital factory inspection system based on a mixed reality technology.
Background
The digital twin technology is a computer technology which creates a virtual model of a physical entity in a digital mode and fully utilizes the running data of the physical entity. The full life cycle process of the physical entity is shown through technologies such as virtual-real interaction, data fusion and the like, and real-time and intelligent service is provided for the industrial field. The HoloLens 2 is head-mounted equipment which is introduced by Microsoft and applies a digital twin technology, and virtual images generated by a computer can be superposed in an actual scene, so that a wearer can obtain a virtual-real fused environment experience.
The traditional method of factory inspection is realized by methods of observing a dashboard by a maintainer, checking the state of equipment and the like, and the methods have the problems that the operation data acquisition mode is complex, the state of the equipment can be acquired only by an indicator lamp and the like. The digital factory can transmit the real-time operation data of the equipment to a remote host computer for storage, and can acquire the real-time operation data by using proper equipment. In order to improve the working efficiency of a manual inspection mode and improve the digital level of a factory, hololens mixed reality technology-based power distribution room field inspection method and system (Hololens mixed reality technology-based power distribution room field inspection method and system [ P ] in China patent CN110246235A, 2019-09-17.) are provided by Hololens mixed reality technology, the system identifies and reads data of equipment in a power distribution room through an image acquisition technology, realizes the positioning of inspection personnel by combining an SLAM technology and IMU data, and can also formulate an inspection route scheme according to indoor equipment. The swim ice and the like provide an MR workshop tour method (swim ice, zhu-Caesareal, liu-Ying, and the like) based on HoloLens, and the MR workshop tour method [ P ] based on HoloLens is a Chinese patent CN109636920A,2019-04-16.
Both the above two patents propose methods capable of greatly improving the working efficiency of maintenance personnel based on HoloLens, but when the maintenance personnel seek help from remote experts, the remote experts cannot obtain the real-time situation of the current factory side, and the communication difficulty between the maintenance personnel and the experts is greatly increased.
Disclosure of Invention
The invention provides a digital factory inspection system based on a mixed reality technology, which aims to improve the working efficiency of maintenance personnel and the digitization level of a factory and realize high-efficiency communication between the maintenance personnel and a remote expert.
The invention discloses a digital factory inspection system based on a mixed reality technology, which comprises a HoloLens 2 device, an embedded device and a user host. The method comprises the steps that a user host, a HoloLens 2 device and an embedded device are respectively connected through a network, on one hand, an instruction and data sent by the HoloLens 2 device are received, and a processing result is returned according to the instruction; and on the other hand, the real-time state data of the equipment sent by the embedded equipment is received and stored. The HoloLens 2 device acquires input information or instructions of a user through the first UI interface and the second UI interface, and generates data interaction with a user host through a network protocol according to actual conditions. The embedded device is used for acquiring real-time state data of the device and sending the real-time state data to the user host for processing through network connection by using a TCP/IP protocol.
The HoloLens 2 device comprises a first UI interface, a network communication subsystem, a work auxiliary subsystem and a patrol auxiliary subsystem. The first UI interface provides visual operation interfaces and keys for maintenance personnel, and the operation auxiliary subsystem and the inspection auxiliary subsystem realize data interaction with the user host through the network communication subsystem according to actual needs;
the first UI interface comprises a login interface, a registration interface and a function selection interface;
the maintenance personnel inputs the account and the password of the maintenance personnel into an account column and a password column of the login interface presented on the HoloLens 2 and then clicks a login button, the HoloLens 2 calls a TCP/IP communication module in the network communication subsystem to send the account and the password data to the user host for verification, and whether the user host is switched into a function selection interface is judged according to the received verification result returned by the user host;
a registration interface, wherein maintenance personnel input an account and a password to be registered into an account column and a password column of the registration interface presented on the HoloLens 2 and then click a registration button, the HoloLens 2 calls a TCP/IP communication module in a network communication subsystem to send the account and the password data to a user host for registration, and judges whether to switch into a function selection interface or not according to a received registration result returned by the user host;
after the maintenance personnel successfully complete login or registration, the maintenance personnel can enter the operation auxiliary subsystem or the inspection auxiliary subsystem on the function selection interface;
the network communication subsystem comprises a TCP/IP communication module and a WebRTC communication module;
the TCP/IP communication module establishes network communication connection with the user host by using a TCP/IP protocol and is used for transmitting information such as historical state data, real-time state data, equipment names, account passwords of maintenance personnel and the like of the equipment. The WebRTC communication module adopts a WebRTC protocol to realize the communication connection between the HoloLens 2 and the user host, and is used for realizing the real-time audio and video communication between the remote expert and the maintenance personnel and realizing the function of remote expert guidance.
The operation auxiliary subsystem comprises a second UI interface, a historical data query module and a remote expert assistance module;
a second UI interface in the job assisting subsystem, the second UI interface comprising a historical data query interface and a remote communication interface;
the historical data query interface is used for inputting the historical data to be queried at the designated position of the interface by a maintainer, transmitting the data input by the maintainer to the historical data query module for further processing, and displaying a query result at the designated position by the historical data query interface after the historical data query is finished;
the remote communication interface is used for allowing maintenance personnel to click a communication starting button on the remote communication interface, a communication instruction is sent to the remote assistance module for processing, and after the remote assistance module is processed, the maintenance personnel can see real-time pictures of a remote expert and a local end at a designated position on the remote communication interface to realize remote expert assistance;
the historical data query module calls the TCP/IP communication module after receiving the data sent by the historical data query interface, acquires the data to be queried from the user host according to a pre-specified data communication format and uploads the data to the historical data query interface;
and the remote expert assistance module calls the WebRTC communication module to establish WebRTC communication connection with the user host after receiving a communication instruction transmitted by the remote communication interface, and delivers the received WebRTC video stream returned by the user host to the remote communication interface for display after the connection is successful.
The inspection auxiliary subsystem comprises a second UI interface and an equipment real-time state query module;
the second UI interface comprises an equipment real-time state query interface in the patrol inspection auxiliary subsystem;
after entering the real-time equipment status query interface, maintenance personnel can select to manually input the equipment name to query the real-time status data of the equipment on the interface or select to automatically complete the query of the real-time status data of the current equipment by the inspection auxiliary subsystem. After the selection is finished, the second UI interface sends the selected mode and the data to be inquired to the equipment real-time state inquiry module, and after the equipment real-time state inquiry module finishes processing, the inquiry result is displayed at the designated position;
the equipment real-time state query module selects a manual mode to input an equipment name query state data mode, and after receiving data issued by an equipment real-time state query interface, the equipment real-time state query module calls the TCP/IP communication module to send the input data to the user host according to a preset data transmission format and obtains the equipment real-time state data returned by the user host; and selecting a mode of automatically inquiring the state data by the inspection auxiliary subsystem, calling a TCP/IP communication module by the equipment real-time state inquiring module to send the image acquired by the HoloLens 2 to the user host, identifying the equipment name by the user host, and returning the corresponding state data to the equipment real-time state inquiring module. No matter which query mode is selected, the device real-time state query module uploads a query result to a device real-time state query interface for further processing.
The user host comprises a Web interface, a network communication subsystem, a data storage subsystem and an object identification subsystem;
after the HoloLens 2 and the user host are connected through the WebRTC protocol, a remote expert can acquire real-time voice and images of a maintenance personnel end through the Web interface and send the real-time voice and images of the expert to the maintenance personnel to realize real-time audio and video communication between the HoloLens 2 and the user host;
the network communication subsystem comprises a TCP/IP communication module and a WebRTC communication module;
the TCP/IP communication module establishes network communication connection with the HoloLens 2 and the embedded equipment by using a TCP/IP protocol, and is used for transmitting information such as historical data, real-time state data, equipment name, account password of maintainers and the like of the equipment. The WebRTC communication module adopts a WebRTC protocol to establish communication connection with the HoloLens 2 and is used for realizing real-time audio and video communication between a remote expert and a maintenance worker;
the data storage subsystem is used for establishing TCP/IP connection between the user host and the HoloLens 2, receiving an instruction sent by the HoloLens 2 and storing or reading data according to the instruction; TCP/IP communication is established between the user host and the embedded equipment, and real-time state data of the equipment is received and stored according to a preset data format;
and a TCP/IP communication connection is established between the object recognition subsystem, holoLens 2 and the user host, and the user host inputs the received image into an object recognition algorithm for processing according to the instruction. And after the identification is finished, sending the identification result to a data storage subsystem for processing, and sending the real-time state data of the equipment to the HoloLens 2 by the data storage subsystem according to the identified equipment name.
The embedded device is installed on the equipment, is in the same network segment with the user host through the Ethernet, and mainly acquires real-time state data of the equipment and sends the data to the user host through a TCP/IP communication protocol.
Preferably, the model of the main control chip of the embedded device is STM32F103ZET6.
Preferably, the object recognition algorithm uses the YOLOv4 algorithm.
The historical data comprises equipment operation data and maintenance records.
Compared with the prior art, the invention has the beneficial effects that: the system is added with the functions of historical state data and equipment real-time state data query, so that maintenance personnel can find equipment problems in time according to data query results; real-time audio and video communication between the HoloLens 2 and the user host is realized by adopting a WebRTC communication protocol, so that the communication efficiency between a remote expert and a maintainer is improved; the system designs two methods of manually inputting the equipment name and identifying the equipment in real time by using an object identification algorithm, so that the flexibility of the system is improved; the state data of the equipment is sent to the user host, so that the storage capacity of the data is improved, the unified management efficiency of the data is improved, and the digitization level of a factory is improved.
Drawings
FIG. 1 is a system block diagram of the present invention.
Fig. 2 is a block diagram of the HoloLens 2 system of the present invention.
Fig. 3 is a system block diagram of a subscriber host in the present invention.
Detailed Description
The following examples are further detailed in conjunction with the accompanying drawings:
as shown in fig. 1, the digital factory inspection system based on the mixed reality technology includes a HoloLens 2 device, an embedded device, and a user host. The method comprises the steps that a user host, a HoloLens 2 device and an embedded device are respectively connected through a network, on one hand, an instruction and data sent by the HoloLens 2 device are received, and a processing result is returned according to the instruction; and on the other hand, the real-time state data of the equipment sent by the embedded equipment is received and stored. The HoloLens 2 device acquires input information or instructions of a user through a UI interface, and generates data interaction with a user host through a TCP/IP protocol or a WebRTC protocol according to actual conditions. The embedded device is used for acquiring real-time state data of the device and sending the real-time state data to the user host for processing through network connection by using a TCP/IP protocol.
The block diagram of the HoloLens 2 system is shown in fig. 2, and the HoloLens 2 device comprises a first UI interface, a network communication subsystem, a work auxiliary subsystem and an inspection auxiliary subsystem. The first UI interface provides visual operation interfaces and keys for maintenance personnel, and the operation auxiliary subsystem and the inspection auxiliary subsystem realize data interaction with the user host through the network communication subsystem according to actual needs;
the first UI interface comprises a login interface, a registration interface and a function selection interface;
a login interface, wherein a maintainer inputs an account and a password into an account column and a password column of the login interface presented on a HoloLens 2 and then clicks a login button, the HoloLens 2 calls a TCP/IP communication module in a network communication subsystem to send the account and the password data to a user host for verification, and whether to enter a function selection interface is judged according to a received verification result returned by the user host;
a registration interface, wherein maintenance personnel input an account and a password to be registered into an account column and a password column of the registration interface presented on the HoloLens 2 and then click a registration button, the HoloLens 2 calls a TCP/IP communication module in a network communication subsystem to send account and password data to a user host for registration, and whether the user host is switched into a function selection interface is judged according to a received registration result returned by the user host;
after the maintenance personnel successfully complete login or registration, the maintenance personnel can enter the operation auxiliary subsystem or the inspection auxiliary subsystem on the function selection interface;
the network communication subsystem comprises a TCP/IP communication module and a WebRTC communication module;
the TCP/IP communication module establishes network communication connection with the user host by using a TCP/IP protocol and is used for transmitting information such as historical state data, real-time state data, equipment names, account passwords of maintenance personnel and the like of the equipment. The WebRTC communication module adopts a WebRTC protocol to realize the communication connection between the HoloLens 2 and the user host, and is used for realizing the real-time audio and video communication between the remote expert and the maintenance personnel and realizing the function of remote expert guidance.
The operation auxiliary subsystem comprises a second UI interface, a historical data query module and a remote expert assistance module;
the second UI interface comprises a historical data query interface and a remote communication interface in the operation auxiliary subsystem;
the historical data query interface is used for inputting the historical data to be queried at the designated position of the interface by a maintainer, transmitting the data input by the maintainer to the historical data query module for further processing, and displaying a query result at the designated position by the historical data query interface after the historical data query is finished;
the remote communication interface is used for allowing maintenance personnel to click a communication starting button on the remote communication interface, a communication instruction is issued to the remote assistance module for processing, and after the remote assistance module finishes processing, the maintenance personnel can see real-time images of a remote expert and a local end at a specified position on the remote communication interface to realize remote expert assistance;
the historical data query module calls the TCP/IP communication module after receiving the data sent by the historical data query interface, acquires the data to be queried from the user host according to a pre-specified data communication format and uploads the data to the historical data query interface;
and the remote expert assistance module calls the WebRTC communication module to establish WebRTC communication connection with the user host after receiving a communication instruction transmitted by the remote communication interface, and delivers the received WebRTC video stream returned by the user host to the remote communication interface for display after the connection is successful.
The inspection auxiliary subsystem comprises a second UI interface and an equipment real-time state query module;
the second UI interface comprises an equipment real-time state query interface in the patrol inspection auxiliary subsystem;
and after entering the equipment real-time state query interface, maintenance personnel can select a manual input equipment name on the interface to query the real-time state data of the equipment or select the polling auxiliary subsystem to automatically complete the query of the real-time state data of the current equipment. After the selection is finished, the second UI interface sends the selected mode and the data to be inquired to the equipment real-time state inquiry module, and after the equipment real-time state inquiry module finishes processing, the inquiry result is displayed at the designated position;
the equipment real-time state query module selects a manual mode to input an equipment name query state data mode, and after receiving data sent by an equipment real-time state query interface, the equipment real-time state query module calls the TCP/IP communication module to send the input data to the user host according to a preset data transmission format and obtains the equipment real-time state data returned by the user host; and selecting a mode of automatically inquiring the state data by the inspection auxiliary subsystem, calling a TCP/IP communication module by the equipment real-time state inquiry module to send the image acquired by the HoloLens 2 to the user host, identifying the equipment name by the user host, and returning the corresponding state data to the equipment real-time state inquiry module. No matter which query mode is selected, the device real-time state query module uploads a query result to a device real-time state query interface for further processing.
A system block diagram of the user host is shown in fig. 3, and includes a Web interface, a network communication subsystem, a data storage subsystem, and an object identification subsystem;
after the HoloLens 2 and the user host are connected through the WebRTC protocol, a remote expert can acquire real-time voice and images of a maintenance personnel end through the Web interface and send the real-time voice and images of the expert to the maintenance personnel to realize real-time audio and video communication between the HoloLens 2 and the user host;
the network communication subsystem comprises a TCP/IP communication module and a WebRTC communication module;
the TCP/IP communication module establishes network communication connection with the HoloLens 2 and the embedded equipment by using a TCP/IP protocol, and is used for transmitting information such as historical data, real-time state data, equipment name, account password of maintainers and the like of the equipment. The WebRTC communication module adopts a WebRTC protocol to establish communication connection with the HoloLens 2 and is used for realizing real-time audio and video communication between a remote expert and a maintenance worker;
the data storage subsystem is used for establishing TCP/IP connection between the user host and the HoloLens 2, receiving an instruction sent by the HoloLens 2 and storing or reading data according to the instruction; TCP/IP communication is established between the user host and the embedded equipment, and real-time state data of the equipment is received and stored according to a preset data format;
and a TCP/IP communication connection is established between the object recognition subsystem, holoLens 2 and the user host, and the user host inputs the received image into an object recognition algorithm for processing according to the instruction. And after the identification is finished, sending the identification result to a data storage subsystem for processing, and sending the real-time state data of the equipment to the HoloLens 2 by the data storage subsystem according to the identified equipment name.
The embodiments described in this specification are merely illustrative of implementations of the inventive concept and the scope of the invention should not be considered limited to the specific forms set forth, but rather by the equivalents thereof as may be contemplated by those skilled in the art based on the teachings of the present invention.

Claims (4)

1. A digital factory inspection system based on mixed reality technology comprises a HoloLens 2 device, an embedded device and a user host; the method comprises the steps that a user host, a HoloLens 2 device and an embedded device are respectively connected through a network, on one hand, an instruction and data sent by the HoloLens 2 device are received, and a processing result is returned according to the instruction; on the other hand, receiving and storing the real-time equipment state data sent by the embedded equipment; the method comprises the following steps that the HoloLens 2 device obtains input information or instructions of a user through a first UI interface and a second UI interface, and data interaction is generated between the HoloLens 2 device and a user host through a network protocol according to actual conditions; the embedded equipment is used for acquiring real-time state data of the equipment and sending the real-time state data to the user host for processing through network connection by using a TCP/IP protocol;
the HoloLens 2 equipment comprises a first UI interface, a network communication subsystem, an operation auxiliary subsystem and an inspection auxiliary subsystem; the first UI interface provides visual operation interfaces and keys for maintenance personnel, and the operation auxiliary subsystem and the inspection auxiliary subsystem realize data interaction with the user host through the network communication subsystem according to actual needs;
the first UI interface comprises a login interface, a registration interface and a function selection interface;
a login interface, wherein a maintainer inputs an account and a password into an account column and a password column of the login interface presented on a HoloLens 2 and then clicks a login button, the HoloLens 2 calls a TCP/IP communication module in a network communication subsystem to send the account and the password data to a user host for verification, and whether to enter a function selection interface is judged according to a received verification result returned by the user host;
a registration interface, wherein maintenance personnel input an account and a password to be registered into an account column and a password column of the registration interface presented on the HoloLens 2 and then click a registration button, the HoloLens 2 calls a TCP/IP communication module in a network communication subsystem to send account and password data to a user host for registration, and whether the user host is switched into a function selection interface is judged according to a received registration result returned by the user host;
after the maintenance personnel successfully complete login or registration, the maintenance personnel can enter the operation auxiliary subsystem or the inspection auxiliary subsystem on the function selection interface;
the network communication subsystem comprises a TCP/IP communication module and a WebRTC communication module;
the TCP/IP communication module establishes network communication connection with a user host by using a TCP/IP protocol and is used for transmitting information such as historical state data, real-time state data, equipment names, account passwords of maintenance personnel and the like of the equipment; the WebRTC communication module adopts a WebRTC protocol to realize the communication connection between the HoloLens 2 and the user host, and is used for realizing the real-time audio-video communication between the remote expert and the maintenance personnel and realizing the function of remote expert guidance;
the operation auxiliary subsystem comprises a second UI interface, a historical data query module and a remote expert assistance module;
the second UI interface comprises a historical data query interface and a remote communication interface in the operation auxiliary subsystem;
the historical data query interface is used for inputting the historical data to be queried at the designated position of the interface by a maintainer, transmitting the data input by the maintainer to the historical data query module for further processing, and displaying a query result at the designated position by the historical data query interface after the historical data query is finished;
the remote communication interface is used for allowing maintenance personnel to click a communication starting button on the remote communication interface, a communication instruction is sent to the remote assistance module for processing, and after the remote assistance module is processed, the maintenance personnel can see real-time pictures of a remote expert and a local end at a designated position on the remote communication interface to realize remote expert assistance;
the historical data query module calls the TCP/IP communication module after receiving the data sent by the historical data query interface, acquires the data to be queried from the user host according to a pre-specified data communication format and uploads the data to the historical data query interface;
the remote expert assistance module calls the WebRTC communication module to establish WebRTC communication connection with the user host after receiving a communication instruction sent by the remote communication interface, and after the connection is successful, the remote expert assistance module delivers the received WebRTC video stream returned by the user host to the remote communication interface for display;
the inspection auxiliary subsystem comprises a second UI interface and an equipment real-time state query module;
the second UI interface comprises an equipment real-time state query interface in the patrol inspection auxiliary subsystem;
after entering the real-time state query interface of the equipment, maintenance personnel can select to manually input the equipment name to query the real-time state data of the equipment on the interface or select to automatically complete the query of the real-time state data of the current equipment by the inspection auxiliary subsystem; after the selection is finished, the second UI interface sends the selected mode and the data to be inquired to the equipment real-time state inquiry module, and after the equipment real-time state inquiry module finishes processing, the inquiry result is displayed at the designated position;
the equipment real-time state query module selects a manual mode to input an equipment name query state data mode, and after receiving data issued by an equipment real-time state query interface, the equipment real-time state query module calls the TCP/IP communication module to send the input data to the user host according to a preset data transmission format and obtains the equipment real-time state data returned by the user host; selecting a mode of automatically inquiring state data by the inspection auxiliary subsystem, calling a TCP/IP communication module by the equipment real-time state inquiring module to send an image acquired by the HoloLens 2 to the user host, identifying the equipment name by the user host, and returning corresponding state data to the equipment real-time state inquiring module; no matter which query mode is selected, the equipment real-time state query module uploads a query result to an equipment real-time state query interface for further processing;
the user host comprises a Web interface, a network communication subsystem, a data storage subsystem and an object identification subsystem;
after the HoloLens 2 and the user host are connected through the WebRTC protocol, a remote expert can acquire real-time voice and images of a maintenance personnel end through the Web interface and send the real-time voice and images of the expert to the maintenance personnel to realize real-time audio and video communication between the HoloLens 2 and the user host;
the network communication subsystem comprises a TCP/IP communication module and a WebRTC communication module;
the TCP/IP communication module establishes network communication connection with the HoloLens 2 and the embedded equipment by using a TCP/IP protocol, and is used for transmitting information such as historical data, real-time state data, equipment name, account password of maintainers and the like of the equipment; the WebRTC communication module adopts a WebRTC protocol to establish communication connection with the HoloLens 2 and is used for realizing real-time audio and video communication between a remote expert and a maintenance worker;
the data storage subsystem is used for establishing TCP/IP connection between the user host and the HoloLens 2, receiving an instruction sent by the HoloLens 2 and storing or reading data according to the instruction; TCP/IP communication is established between the user host and the embedded equipment, and real-time state data of the equipment is received and stored according to a preset data format;
the object recognition subsystem, establish TCP/IP communication connection between HoloLens 2 and the user host, the user host inputs the received image into the object recognition algorithm to process according to the instruction; after the identification is finished, sending the identification result to a data storage subsystem for processing, and sending real-time state data of the equipment to the HoloLens 2 by the data storage subsystem according to the identified equipment name;
the embedded device is installed on the equipment, is in the same network segment with the user host through the Ethernet, and mainly acquires real-time state data of the equipment and sends the real-time state data to the user host through a TCP/IP communication protocol.
2. The digital factory inspection system based on the mixed reality technology according to claim 1, wherein: the model of the main control chip of the embedded device is STM32F103ZET6.
3. The digital factory inspection system based on the mixed reality technology according to claim 1, wherein: the object recognition algorithm uses the YOLOv4 algorithm.
4. The digital factory inspection system based on the mixed reality technology according to claim 1, wherein: the historical data comprises equipment operation data and maintenance records.
CN202210767834.XA 2022-06-30 2022-06-30 Digital factory inspection system based on mixed reality technology Active CN115334062B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210767834.XA CN115334062B (en) 2022-06-30 2022-06-30 Digital factory inspection system based on mixed reality technology

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210767834.XA CN115334062B (en) 2022-06-30 2022-06-30 Digital factory inspection system based on mixed reality technology

Publications (2)

Publication Number Publication Date
CN115334062A true CN115334062A (en) 2022-11-11
CN115334062B CN115334062B (en) 2024-05-03

Family

ID=83918064

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210767834.XA Active CN115334062B (en) 2022-06-30 2022-06-30 Digital factory inspection system based on mixed reality technology

Country Status (1)

Country Link
CN (1) CN115334062B (en)

Citations (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107578487A (en) * 2017-09-19 2018-01-12 北京枭龙科技有限公司 A kind of cruising inspection system based on augmented reality smart machine
WO2018187748A1 (en) * 2017-04-07 2018-10-11 Unveil, LLC Systems and methods for mixed reality medical training
US20180301048A1 (en) * 2017-04-12 2018-10-18 Age Of Learning, Inc. Remote live tutoring platform
CN108874153A (en) * 2018-08-23 2018-11-23 上海机器人产业技术研究院有限公司 One kind being based on AR glasses cloud robot holography review system and method
US20190096130A1 (en) * 2017-09-26 2019-03-28 Akn Korea Inc. Virtual mobile terminal implementing system in mixed reality and control method thereof
CN110246235A (en) * 2019-06-18 2019-09-17 广州供电局有限公司 A kind of power distribution room scene method for inspecting and system based on Hololens mixed reality technology
KR102086451B1 (en) * 2019-09-06 2020-05-29 주식회사 글로벌제조혁신네트웍 Smart Factory remote collaboration CMS system in augmented reality and Drive method of the Same
CN112578907A (en) * 2020-12-01 2021-03-30 国电湖南巫水水电开发有限公司 Method and device for realizing remote guidance operation based on AR
CN112671898A (en) * 2020-12-23 2021-04-16 浙江工业大学 Remote communication control system based on embedded equipment
CN112732072A (en) * 2020-12-21 2021-04-30 航天信息股份有限公司 Equipment intelligent maintenance system based on VR/AR
CN112929384A (en) * 2021-03-05 2021-06-08 瑞丰宝丽(北京)科技有限公司 AR intelligent point inspection system based on space anchor point
CN113297945A (en) * 2021-05-18 2021-08-24 北京远舢智能科技有限公司 Indoor equipment inspection auxiliary method and system based on mixed reality space positioning
CN113936120A (en) * 2021-10-12 2022-01-14 北京邮电大学 Mark-free lightweight Web AR method and system
CN114662714A (en) * 2022-02-25 2022-06-24 南京邮电大学 Machine room operation and maintenance management system and method based on AR equipment

Patent Citations (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2018187748A1 (en) * 2017-04-07 2018-10-11 Unveil, LLC Systems and methods for mixed reality medical training
US20180301048A1 (en) * 2017-04-12 2018-10-18 Age Of Learning, Inc. Remote live tutoring platform
CN107578487A (en) * 2017-09-19 2018-01-12 北京枭龙科技有限公司 A kind of cruising inspection system based on augmented reality smart machine
US20190096130A1 (en) * 2017-09-26 2019-03-28 Akn Korea Inc. Virtual mobile terminal implementing system in mixed reality and control method thereof
CN108874153A (en) * 2018-08-23 2018-11-23 上海机器人产业技术研究院有限公司 One kind being based on AR glasses cloud robot holography review system and method
CN110246235A (en) * 2019-06-18 2019-09-17 广州供电局有限公司 A kind of power distribution room scene method for inspecting and system based on Hololens mixed reality technology
KR102086451B1 (en) * 2019-09-06 2020-05-29 주식회사 글로벌제조혁신네트웍 Smart Factory remote collaboration CMS system in augmented reality and Drive method of the Same
CN112578907A (en) * 2020-12-01 2021-03-30 国电湖南巫水水电开发有限公司 Method and device for realizing remote guidance operation based on AR
CN112732072A (en) * 2020-12-21 2021-04-30 航天信息股份有限公司 Equipment intelligent maintenance system based on VR/AR
CN112671898A (en) * 2020-12-23 2021-04-16 浙江工业大学 Remote communication control system based on embedded equipment
CN112929384A (en) * 2021-03-05 2021-06-08 瑞丰宝丽(北京)科技有限公司 AR intelligent point inspection system based on space anchor point
CN113297945A (en) * 2021-05-18 2021-08-24 北京远舢智能科技有限公司 Indoor equipment inspection auxiliary method and system based on mixed reality space positioning
CN113936120A (en) * 2021-10-12 2022-01-14 北京邮电大学 Mark-free lightweight Web AR method and system
CN114662714A (en) * 2022-02-25 2022-06-24 南京邮电大学 Machine room operation and maintenance management system and method based on AR equipment

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
Y. -S. CHANG等: ""Proof of Concept for Development of Collaborative Mixed Reality Applications from 3D City Models"", 《2019 INTERNATIONAL CONFERENCE ON INFORMATION AND COMMUNICATION TECHNOLOGY CONVERGENCE (ICTC)》, 27 November 2019 (2019-11-27) *
潘建乔;吴迪;余方召;潘白浪;龚雁峰;郭国栋;: "采用HoloLens进行变电站巡检的作业信息管理系统解决方案", 电力设备管理, no. 01, 25 January 2020 (2020-01-25) *

Also Published As

Publication number Publication date
CN115334062B (en) 2024-05-03

Similar Documents

Publication Publication Date Title
US6088737A (en) Information processing system and control method thereof
AU2003266591B2 (en) Remote education system, course attendance check method, and course attendance check program
AU718452B2 (en) Image-transmitting remote object examination device
CN111612933A (en) Augmented reality intelligent inspection system based on edge cloud server
JP4026944B2 (en) Video transmission device and control method thereof
CN102158685B (en) System, system construction method, management terminal, and program
AU7853798A (en) Camera control system controlling different types of cameras
CN110909898A (en) Zero-knowledge bank machine room fault diagnosis and maintenance guidance system and method based on AR glasses
CN110210793A (en) A kind of project sites construction remote monitoring platform, system and method
CN101330598A (en) Method and system for managing video as well as monitoring/diagnosis method thereof
CN102520681B (en) Internet-of-things terminal intelligent configuration method based on appearance information identification and system
CN115334062A (en) Digital factory inspection system based on mixed reality technology
CN112423073B (en) Remote control method and system for screen throwing equipment
CN107846609B (en) Real-time transmission method, device and system for control room data
JP2002241060A (en) Integrated elevator monitoring system
CN112085558A (en) House watching system for house renting and working method thereof
JPH09222916A (en) Job site monitor system
CN104880997A (en) Respirator network connection control method and system
CN110213344B (en) Multi-center remote sign language online translation system and method
CN116700498A (en) Mechanical equipment monitoring and management system based on mixed reality
CN113422924A (en) Method and system for realizing intelligent instruction conversion of various video conference devices
CN107480437B (en) Data transmission method and device
CN111782160B (en) Multi-machine multi-source common screen control device
CN219740470U (en) Night patrol system based on face recognition technology
CN109547919B (en) Method for remotely managing operation site

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant