CN109218794B - Remote operation guidance method and system - Google Patents
Remote operation guidance method and system Download PDFInfo
- Publication number
- CN109218794B CN109218794B CN201710523982.6A CN201710523982A CN109218794B CN 109218794 B CN109218794 B CN 109218794B CN 201710523982 A CN201710523982 A CN 201710523982A CN 109218794 B CN109218794 B CN 109218794B
- Authority
- CN
- China
- Prior art keywords
- video
- frame
- audio
- augmented reality
- information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 34
- 230000003190 augmentative effect Effects 0.000 claims abstract description 148
- 238000002372 labelling Methods 0.000 claims abstract description 137
- 230000001360 synchronised effect Effects 0.000 claims abstract description 80
- 238000012545 processing Methods 0.000 claims description 27
- 238000004891 communication Methods 0.000 claims description 13
- 230000009467 reduction Effects 0.000 claims description 6
- 230000008030 elimination Effects 0.000 claims description 3
- 238000003379 elimination reaction Methods 0.000 claims description 3
- 239000000284 extract Substances 0.000 claims description 3
- 238000005457 optimization Methods 0.000 claims description 3
- 238000004590 computer program Methods 0.000 description 13
- 239000000872 buffer Substances 0.000 description 12
- 238000010586 diagram Methods 0.000 description 12
- 230000008569 process Effects 0.000 description 6
- 230000003139 buffering effect Effects 0.000 description 5
- 238000003860 storage Methods 0.000 description 5
- 230000006870 function Effects 0.000 description 4
- 238000009434 installation Methods 0.000 description 4
- 230000003111 delayed effect Effects 0.000 description 3
- 238000009826 distribution Methods 0.000 description 3
- 238000007689 inspection Methods 0.000 description 3
- 230000008439 repair process Effects 0.000 description 3
- 230000011664 signaling Effects 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 230000006835 compression Effects 0.000 description 2
- 238000007906 compression Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 239000011521 glass Substances 0.000 description 2
- 238000012423 maintenance Methods 0.000 description 2
- 238000009877 rendering Methods 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000012966 insertion method Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000010422 painting Methods 0.000 description 1
- 238000011946 reduction process Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/4302—Content synchronisation processes, e.g. decoder synchronisation
- H04N21/4307—Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/60—Network streaming of media packets
- H04L65/75—Media network packet handling
- H04L65/762—Media network packet handling at the source
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/60—Network streaming of media packets
- H04L65/75—Media network packet handling
- H04L65/764—Media network packet handling at the destination
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/04—Synchronising
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Computer Networks & Wireless Communication (AREA)
- Information Transfer Between Computers (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
Abstract
The invention provides a remote operation guidance method and a system, comprising the following steps: the intelligent terminal adds a first relative timestamp to the collected audio, video and augmented reality labeling information to synchronize the audio, video and augmented reality labeling information, and sends the synchronized information to the WEB terminal; the intelligent terminal acquires the video added with the second relative timestamp and the remote assistance information returned by the WEB terminal; the intelligent terminal synchronously plays back audio, video and augmented reality labeling information according to the first relative timestamp, and synchronously plays back the video and remote assistance information according to the second relative timestamp; the intelligent terminal takes the video as a main stream and displays the video, the audio, the augmented reality labeling information and the remote assistance information after synchronous playback. According to the technical scheme provided by the invention, the synchronization of the audio, the video and the augmented reality labeling information is realized by using a relative timestamp synchronization method, the synchronous display of the video, the audio, the augmented reality labeling information and the remote assistance information is realized by adopting synchronous playback, and the precise remote guidance is realized.
Description
Technical Field
The invention belongs to the field of information technology multimedia application, and particularly relates to a remote operation guidance method and a remote operation guidance system.
Background
Along with the company's electric wire netting scale enlarges increasingly and novel equipment is put into operation, electric wire netting equipment installation, operation, maintenance face the data numerous and diverse, the step is complicated, equipment dismouting precision is high and personnel's level is not the scheduling problem. The traditional power grid operation mode (infrastructure installation, inspection and overhaul, distribution network first-aid repair) adopts a detailed plan made before operation, and the detailed plan is executed in the execution process strictly according to the plan.
With the development of wearable and augmented reality technologies, power grid operators can acquire, transmit and process information in a way of wearing intelligent terminals, and can receive remote assistance of technical experts when encountering difficult problems. And related operation can not be clearly realized by each participant through video and sound, so that the remote assistance guidance method and system for the power grid overhaul scene provide visual and convenient operation guidance and cooperation support for power grid field operation personnel, and improve the work efficiency and safe operation level of power grid equipment infrastructure installation, inspection overhaul and distribution network first-aid repair.
Disclosure of Invention
In order to meet the requirement of on-site operation of power grid maintenance, the invention provides a remote operation guidance method and a remote operation guidance system, which realize accurate remote guidance in a video, sound and remote expert labeling mode.
In order to realize the purpose, the following technical scheme is adopted:
the invention provides a remote operation guidance method, which comprises the following steps:
the intelligent terminal adds a first relative timestamp to the collected audio, video and augmented reality labeling information to synchronize the audio, video and augmented reality labeling information and provides the synchronized audio, video and augmented reality labeling information to the WEB terminal;
the intelligent terminal acquires the video added with the second relative timestamp and the remote assistance information returned by the WEB terminal;
the intelligent terminal synchronously plays back the audio, the video and the augmented reality labeling information according to the first relative timestamp, and simultaneously plays back the video and the remote assistance information synchronously according to the second relative timestamp;
the intelligent terminal takes the video as a main stream and displays the video, the audio, the augmented reality labeling information and the remote assistance information after synchronous playback.
The intelligent terminal provides audio frequency, video and augmented reality mark information after the synchronization to the WEB terminal, includes:
the intelligent terminal compresses and encodes the synchronized audio, video and augmented reality labeling information, sends the compressed and encoded audio, video and augmented reality labeling information to the video server through respective independent channels simultaneously or sequentially, and then sends the compressed and encoded audio, video and augmented reality labeling information to the WEB end through the video server.
The remote assistance information includes: remote assistance audio and remote assistance annotation information.
The synchronized playback of audio, video, and augmented reality annotation information according to the first relative timestamp comprises:
and simultaneously extracting an audio frame, a video frame and an augmented reality labeling information frame for comparison, wherein the audio frame, the video frame and the augmented reality labeling information frame are provided with first relative timestamps, if the video frame is normal and the audio frame and the augmented reality labeling information frame are lagged, the audio frame and the augmented reality labeling information frame are discarded, if the audio frame and the video frame are synchronous and the augmented reality labeling information frame are lagged, the augmented reality labeling information frame is discarded, and if the video frame is lagged, the audio playing speed is slowed down or the video playing speed is accelerated.
The synchronized playback of the video and the remote assistance information according to the second relative time stamp includes:
and simultaneously extracting a video frame, a remote assistance audio frame and a remote assistance annotation information frame for comparison, if the video frame is normal and the remote assistance audio frame and the remote assistance annotation information frame are both lagged, discarding the remote assistance audio frame and the remote assistance annotation information frame, if the remote assistance audio frame and the video frame are synchronous and the remote assistance annotation information frame is lagged, discarding the remote assistance annotation information frame, and if the video frame is lagged, slowing down the playing speed of the remote assistance audio or speeding up the playing speed of the video.
The invention provides an intelligent terminal, comprising:
the first acquisition module is used for acquiring video, audio and augmented reality labeling information;
the first synchronization module is used for adding a first relative timestamp to the video, the audio and the augmented reality labeling information for synchronization;
the first communication module is used for acquiring the video added with the second relative timestamp and the remote assistance information from the WEB side;
the first synchronous playback module is used for synchronously playing back the video, the audio and the augmented reality labeling information according to the first relative timestamp; and synchronously playing back the video and the remote assistance information according to the second relative time stamp;
and the first display module is used for taking the video as a main stream and displaying the video, the audio, the augmented reality labeling information and the remote assistance information which are played back synchronously.
The remote assistance information acquired by the first communication module from the WEB side comprises remote assistance audio and remote assistance marking information.
The first synchronized playback module includes:
the first comparison unit is used for simultaneously extracting an audio frame, a video frame and an augmented reality labeling information frame according to a first relative timestamp for comparison, wherein the audio frame, the video frame and the augmented reality labeling information frame are printed with the first relative timestamp;
the first processing unit is used for discarding the audio frame and the augmented reality labeling information frame if the video frame is normal and the audio frame and the augmented reality labeling information frame are both lagged, discarding the augmented reality labeling information frame if the audio frame and the audio frame are synchronous and the augmented reality labeling information frame is lagged, and slowing down or speeding up the playing speed of the audio if the video frame is lagged.
The first synchronized playback module includes:
the second comparison unit is used for simultaneously extracting the video frame, the remote assistance audio frame and the remote assistance marking information frame according to the second relative time stamp for comparison;
and the second processing unit is used for discarding the remote assistance audio frame and the remote assistance annotation information frame if the video frame is normal and the remote assistance audio frame and the remote assistance annotation information frame are both lagged, discarding the remote assistance annotation information frame if the remote assistance audio frame and the video frame are synchronous and the remote assistance annotation information frame is lagged, and slowing down or speeding up the playing speed of the remote assistance audio if the video frame is lagged.
The invention provides a remote operation guidance method, which is characterized by comprising the following steps:
the WEB terminal acquires the audio, video and augmented reality labeling information added with the first relative timestamp from the intelligent terminal, and synchronously plays back and displays the audio, video and augmented reality labeling information according to the first relative timestamp;
a WEB terminal acquires remote assistance information and adds a second relative timestamp to the remote assistance information and the video for synchronization;
and the WEB terminal provides the synchronized video added with the second relative time stamp and the remote assistance information to the intelligent terminal.
The step of sending the synchronized remote assistance information and video to the intelligent terminal by the WEB side comprises the following steps:
and the WEB side compresses and encodes the synchronized remote assistance information and video and then sends the compressed and encoded remote assistance information and video to the intelligent terminal through the video server.
The synchronized playback of audio, video, and augmented reality annotation information according to the first relative timestamp comprises:
and simultaneously extracting an audio frame, a video frame and an augmented reality labeling information frame for comparison, wherein the audio frame, the video frame and the augmented reality labeling information frame are provided with first relative timestamps, if the video frame is normal and the audio frame and the augmented reality labeling information frame are lagged, the audio frame and the augmented reality labeling information frame are discarded, if the audio frame and the video frame are synchronous and the augmented reality labeling information frame are lagged, the augmented reality labeling information frame is discarded, and if the video frame is lagged, the audio playing speed is slowed down or the video playing speed is accelerated.
The invention provides a WEB side, which comprises:
the second acquisition module is used for acquiring remote assistance information;
the second synchronous playback module is used for synchronously playing back the video, the audio and the augmented reality labeling information according to the first relative timestamp;
the second display module is used for displaying the video, the audio and the augmented reality labeling information after synchronous playback;
the second synchronization module is used for synchronizing the remote assistance information and the added video with a second relative timestamp;
and the second communication module is used for providing the synchronized video added with the second relative timestamp and the remote assistance information for the intelligent terminal.
The second synchronous playback module extracts an audio frame, a video frame and an augmented reality labeling information frame at the same time according to the first relative timestamp for comparison, the audio frame, the video frame and the augmented reality labeling information frame are marked with the first relative timestamp, if the video frame is normal and the audio frame and the augmented reality labeling information frame are lagged, the audio frame and the augmented reality labeling information frame are discarded, if the audio frame and the video frame are synchronous and the augmented reality labeling information frame are lagged, the augmented reality labeling information frame is discarded, and if the video frame is lagged, the audio playing speed is slowed down or the video playing speed is accelerated.
The invention provides a remote operation guidance system, which comprises an intelligent terminal, a WEB terminal and a video server, wherein the video server is respectively communicated with the intelligent terminal and the WEB terminal, and comprises:
the video quality optimization module is used for performing audio and video rate control, quality compensation, echo elimination, noise reduction processing and/or congestion control on the decoded data information added with the first timestamp and/or the second timestamp;
and the third communication module is used for communicating with the intelligent terminal and/or the WEB terminal.
Compared with the closest prior art, the technical scheme provided by the invention has the following beneficial effects:
according to the technical scheme provided by the invention, the intelligent terminal uses a relative timestamp synchronization method to insert the first relative timestamps into the audio, video and augmented reality labeling information respectively, so that the synchronization of the audio, video and augmented reality labeling information is realized; the intelligent terminal acquires the video added with the second relative timestamp and the remote assistance information returned by the WEB end, and synchronously displays the video, the audio, the augmented reality labeling information and the remote assistance information by synchronous playback according to the first timestamp and the second timestamp, so that precise remote guidance is realized.
According to the technical scheme provided by the invention, the relative timestamp synchronization method is adopted by the video, the acquired remote guidance audio and the acquired remote guidance marking information at the WEB end, and the relative timestamps are respectively inserted into the video, the remote guidance audio and the remote guidance marking information, so that the synchronization of the video, the remote guidance audio and the remote guidance marking information is realized, the video, the remote guidance audio and the remote guidance marking information are sent to the video server after being synchronized, and the video, the remote guidance audio and the remote guidance marking information are sent to each wearable intelligent terminal again after being processed by the video server, so that the accurate remote guidance is realized.
According to the technical scheme provided by the invention, the WEB terminal realizes synchronous display of video, audio and augmented reality annotation information by synchronous playback according to the first relative timestamp, so that precise remote guidance is realized.
According to the technical scheme provided by the invention, the intelligent terminal not only collects video and audio, but also collects augmented reality labels of power grid field operators, and collects complex conditions of a power grid operation field from multiple directions.
The technical scheme provided by the invention provides visual and convenient operation guidance and cooperative support for power grid field operation personnel, and improves the work efficiency and safe operation level of power grid equipment infrastructure installation, inspection and overhaul and distribution network first-aid repair.
Drawings
Fig. 1 is a flowchart of a wearable intelligent terminal of a remote work guidance method of the present invention;
FIG. 2 is a flow chart of a remote operation aware method WEB side of the present invention;
FIG. 3 is a schematic diagram of audio-video and annotation information relative timestamp insertion method;
FIG. 4 is a schematic diagram of a method for synchronously playing back audio and video and annotation information according to an embodiment of the present invention;
FIG. 5 is a diagram of a video server architecture according to an embodiment of the present invention;
fig. 6 is an architecture diagram of a remote work guidance system according to an embodiment of the present invention.
Detailed Description
The invention is described in further detail below with reference to the accompanying drawings:
in order to achieve the purpose, the invention provides the following technical scheme:
as shown in fig. 1, the present invention provides a remote work guidance method, which may include the steps of:
the intelligent terminal adds a first relative timestamp to the collected audio, video and augmented reality labeling information to synchronize the audio, video and augmented reality labeling information and provides the synchronized audio, video and augmented reality labeling information to the WEB terminal;
the intelligent terminal acquires the video added with the second relative timestamp and the remote assistance information returned by the WEB terminal;
the intelligent terminal synchronously plays back audio, video and augmented reality labeling information according to the first relative timestamp, and synchronously plays back the video and remote assistance information according to the second relative timestamp;
the intelligent terminal takes the video as a main stream and displays the video, the audio, the augmented reality labeling information and the remote assistance information after synchronous playback.
The intelligent terminal provides audio frequency, video and augmented reality mark information after the synchronization to the WEB terminal, includes:
the intelligent terminal compresses and encodes the synchronized audio, video and augmented reality labeling information, sends the compressed and encoded audio, video and augmented reality labeling information to the video server through respective independent channels simultaneously or sequentially, and then sends the compressed and encoded audio, video and augmented reality labeling information to the WEB end through the video server.
The remote assistance information includes: remote assistance audio and remote assistance annotation information.
The synchronous playback of the audio, video and augmented reality annotation information according to the first relative timestamp comprises:
and simultaneously extracting an audio frame, a video frame and an augmented reality labeling information frame for comparison, wherein the audio frame, the video frame and the augmented reality labeling information frame are provided with first relative timestamps, if the video frame is normal and the audio frame and the augmented reality labeling information frame are lagged, the audio frame and the augmented reality labeling information frame are discarded, if the audio frame and the video frame are synchronous and the augmented reality labeling information frame are lagged, the augmented reality labeling information frame is discarded, and if the video frame is lagged, the audio playing speed is slowed down or the video playing speed is accelerated.
The synchronized playback of the video and the remote assistance information according to the second relative time stamp includes:
and simultaneously extracting a video frame, a remote assistance audio frame and a remote assistance annotation information frame for comparison, if the video frame is normal and the remote assistance audio frame and the remote assistance annotation information frame are both lagged, discarding the remote assistance audio frame and the remote assistance annotation information frame, if the remote assistance audio frame and the video frame are synchronous and the remote assistance annotation information frame is lagged, discarding the remote assistance annotation information frame, and if the video frame is lagged, slowing down the playing speed of the remote assistance audio or speeding up the playing speed of the video.
The present invention provides an intelligent terminal, which can comprise:
the first acquisition module is used for acquiring video, audio and augmented reality labeling information;
the first synchronization module is used for adding a first relative timestamp to the video, the audio and the augmented reality labeling information for synchronization;
the first communication module is used for acquiring the video added with the second relative timestamp and the remote assistance information from the WEB side;
the first synchronous playback module is used for synchronously playing back the video, the audio and the augmented reality labeling information according to the first relative timestamp; and synchronously playing back the video and the remote assistance information according to the second relative time stamp;
and the first display module is used for taking the video as a main stream and displaying the video, the audio, the augmented reality labeling information and the remote assistance information which are played back synchronously.
The remote assistance information acquired by the first communication module from the WEB side comprises remote assistance audio and remote assistance marking information.
The first synchronized playback module includes:
the first comparison unit is used for simultaneously extracting an audio frame, a video frame and an augmented reality labeling information frame according to a first relative timestamp for comparison, wherein the audio frame, the video frame and the augmented reality labeling information frame are printed with the first relative timestamp;
the first processing unit is used for discarding the audio frame and the augmented reality labeling information frame if the video frame is normal and the audio frame and the augmented reality labeling information frame are both lagged, discarding the augmented reality labeling information frame if the audio frame and the audio frame are synchronous and the augmented reality labeling information frame is lagged, and slowing down or speeding up the playing speed of the audio if the video frame is lagged.
The first synchronized playback module includes:
the second comparison unit is used for simultaneously extracting the video frame, the remote assistance audio frame and the remote assistance marking information frame according to the second relative time stamp for comparison;
and a second processing unit, configured to discard the remote assistance audio frame and the remote assistance markup information frame if the video frame is normal and the remote assistance audio frame and the remote assistance markup information frame are both delayed, discard the remote assistance markup information frame if the remote assistance audio frame and the video frame are synchronized and the remote assistance markup information frame is delayed, and slow down or speed up the playing speed of the remote assistance audio if the video frame is delayed.
Based on the same inventive concept, the invention also provides an intelligent terminal, which comprises a processing system, a memory and a computer program which is stored on the memory and can run on the processing system, wherein the processing system is coupled with the memory, and the processing system realizes the remote operation guidance method when executing the computer program.
Based on the same inventive concept, the present invention also provides a computer-readable storage medium having a computer program that can execute the above-described remote work guidance method.
As shown in fig. 2, the present invention provides a remote work guidance method, which may include the steps of:
the WEB terminal acquires the audio, video and augmented reality labeling information added with the first relative timestamp from the intelligent terminal, and synchronously plays back and displays the audio, video and augmented reality labeling information according to the first relative timestamp;
a WEB side acquires remote assistance information, and adds a second relative timestamp in the remote assistance information and a video for synchronization;
and the WEB terminal provides the synchronized video added with the second relative time stamp and the remote assistance information to the intelligent terminal.
The step of sending the synchronized remote assistance information and video to the intelligent terminal by the WEB side comprises the following steps:
and the WEB side compresses and encodes the synchronized remote assistance information and video and then sends the compressed and encoded remote assistance information and video to the intelligent terminal through the video server.
The synchronized playback of audio, video, and augmented reality annotation information according to the first relative timestamp comprises:
and simultaneously extracting an audio frame, a video frame and an augmented reality labeling information frame for comparison, wherein the audio frame, the video frame and the augmented reality labeling information frame are provided with first relative timestamps, if the video frame is normal and the audio frame and the augmented reality labeling information frame are lagged, the audio frame and the augmented reality labeling information frame are discarded, if the audio frame and the video frame are synchronous and the augmented reality labeling information frame are lagged, the augmented reality labeling information frame is discarded, and if the video frame is lagged, the audio playing speed is slowed down or the video playing speed is accelerated.
The invention provides a WEB side, which can comprise:
the second acquisition module is used for acquiring remote assistance information;
the second synchronous playback module is used for synchronously playing back the video, the audio and the augmented reality labeling information according to the first relative timestamp;
the second display module is used for displaying the video, the audio and the augmented reality labeling information after synchronous playback;
the second synchronization module is used for synchronizing the remote assistance information and the added video with a second relative timestamp;
and the second communication module is used for providing the synchronized video added with the second relative timestamp and the remote assistance information for the intelligent terminal.
The second synchronous playback module extracts an audio frame, a video frame and an augmented reality labeling information frame at the same time according to the first relative timestamp for comparison, the audio frame, the video frame and the augmented reality labeling information frame are marked with the first relative timestamp, if the video frame is normal and the audio frame and the augmented reality labeling information frame are lagged, the audio frame and the augmented reality labeling information frame are discarded, if the audio frame and the video frame are synchronous and the augmented reality labeling information frame are lagged, the augmented reality labeling information frame is discarded, and if the video frame is lagged, the audio playing speed is slowed down or the video playing speed is accelerated.
Based on the same inventive concept, the invention further provides a WEB side, which comprises a processing system, a memory and a computer program which is stored on the memory and can run on the processing system, wherein the processing system is coupled with the memory, and the processing system realizes the remote operation guidance method when executing the computer program.
Based on the same inventive concept, the present invention also provides a computer-readable storage medium having a computer program that can execute the above-described remote work guidance method.
The present invention provides a remote work guidance system, which may include: the intelligent terminal, WEB end and the video server with intelligent terminal and WEB end communication respectively, the video server includes:
the video quality optimization module is used for performing audio and video rate control, quality compensation, echo elimination, noise reduction processing and/or congestion control on the decoded data information added with the first timestamp and/or the second timestamp;
and the third communication module is used for communicating with the intelligent terminal and/or the WEB terminal.
The smart terminal may be a wearable smart terminal, for example: intelligent glasses, intelligent helmet, etc.
Specifically, the remote work guidance method provided by the present invention may include:
the wearable intelligent terminal collects various multimedia information of an operation site, respectively inserts a first relative timestamp, then performs compression coding together with a control signaling, and transmits the compressed coded information to the video server;
the video server encodes, decodes, buffers and processes the received data packet and transmits the data packet to a WEB terminal;
the WEB side decodes, buffers, synchronously plays back and displays the data packet, acquires the sound and the marking information added by an expert during display, inserts a second relative timestamp into the displayed video stream and the expert sound and the marking information collected by the WEB side, compresses and codes the second relative timestamp, and transmits the compressed and coded second relative timestamp to the video server;
the video server transmits the data packet transmitted by the WEB terminal to the wearable intelligent terminal after encoding, decoding, buffering and processing;
and the wearable intelligent terminal receives the data packet sent by the video server, decodes and buffers the data packet, synchronously plays back the data packet, renders the data packet and displays the data packet, and completes the precise remote guidance.
The specific method comprises the following steps:
1. data acquisition
Wearable intelligent terminal (intelligent glasses, intelligent helmet etc.) adopts camera and microphone to gather video, audio frequency, and the electric wire netting operation personnel augmented reality mark and drawing information are gathered to the depth of field camera to with audio frequency, video and mark information, insert first relative time stamp respectively at audio frequency, video and augmented reality mark information, and control signaling carries out compression coding together, form and follow network protocol's data packet and send and reach video server.
2. Data synchronization
When the collected video stream, the audio stream and the augmented reality tagging information are respectively sent to a video server, the two media streams and the augmented reality tagging information are transmitted through independent channels, and delay jitter is brought to transmission by a network, so that a receiving party needs to perform synchronous control to recover the time relation between data when the receiving party needs to realize the synchronization of the audio and the video and the tagging information. The audio stream needs to maintain its continuity, and the video stream changes in accordance with the audio information and the annotation information, so that three-way synchronization is achieved.
As shown in fig. 3, audio and video streams and annotation information have respective time axes, and a relative time stamp RTS is stamped on each frame when the audio and video streams and the annotation information are sent. And selecting the video stream as a main stream corresponding to the audio and the video and the label, continuously playing the main stream, and determining the playing of the auxiliary stream and the label information according to the playing state of the main stream so as to realize the synchronization of the three.
At a sending end, the same time stamp is marked on the audio and video and the labeling information frame collected at the same time, and the audio and video and the labeling information frame are sent out as simultaneously as possible. The sending process can be processed in the same thread, namely a video packet, an audio packet, a labeling packet and a video packet are sent in sequence, so that the synchronization of a sending end can be guaranteed. Because multiple audio and annotation frames are captured within the sample time of a video frame, there are multiple audio and annotation frames in a video packet.
As shown in fig. 4, when the RTP frame arrives at the receiving end, it is then sent to the corresponding decoder for decoding. In order to accurately control the playback time, the data packets are put into respective buffer areas before synchronization, video frames are extracted from the video buffer areas at regular time, meanwhile, audio frames and annotation frames are extracted from the audio and annotation buffer areas for comparison, if the video frames are normal and the audio frames and the annotation frames are lagged, the audio frames and the annotation frames are discarded, if the audio and video frames are synchronous and the annotation frames are lagged, the annotation frames are discarded, and if the video frames are lagged, the audio playing speed is slowed down or the video playing speed is accelerated.
3. Data processing
As shown in fig. 5, the video server is divided into an audio, video, tag information and control terminal interface, etc., where the audio, video and tag information interface receives each data stream and puts it into a buffer and decodes it, and the video processing module performs rate control, quality compensation, echo cancellation, noise reduction, and congestion control on the video stream, where the rate control controls the audio and video rate, the quality compensation compensates the video quality, the echo cancellation cancels sound surround and echo, the noise reduction processes audio noise, and congestion control is performed on the data stream, and the control information controls media stream transmission. And then, the audio, video and label information packet are put into a buffer zone, compressed and encoded, and then sent to a WEB terminal.
4. Expert remote assistance
The remote expert receives a data packet sent by the video server through the WEB terminal to decode, buffer, synchronously playback, render and display, the remote expert guides videos of power grid field operation personnel through modes such as voice, a touch screen, mouse sliding marks and a painting brush tool, video stream, expert marks and sound guide information are printed with a second relative timestamp and then compressed and coded and sent to the video server, the server sends the processed data to the wearable intelligent terminal again, and accurate remote guidance is achieved.
Based on the same inventive concept, the present invention further provides a remote work guidance system, as shown in fig. 6, the system may include:
wearable intelligent terminal: collecting various multimedia information of an operation site, buffering the collected multimedia information, respectively inserting a first relative timestamp, compressing and encoding the multimedia information and a control signaling together, and sending the multimedia information to a video server;
a video server: receiving and processing a data packet sent by the wearable intelligent terminal and then transmitting the data packet to the WEB terminal;
WEB terminal: receiving multimedia information processed by a video server, acquiring sound and annotation information added by an expert, inserting a second relative timestamp into the displayed video stream and the expert sound and annotation information collected by a WEB end, compressing and encoding the second relative timestamp, and transmitting the compressed and encoded second relative timestamp to the video server;
the video server also receives the data packet sent by the WEB terminal, processes the data packet and then pushes the data packet to the wearable intelligent terminal.
The wearable smart terminal may include: the system comprises a camera for collecting videos, a microphone for collecting audios, a depth-of-field camera for collecting augmented reality labeling information, a synchronous sending module for synchronously sending data, a synchronous playback module for synchronously receiving data, a display module for rendering and displaying videos, a buffer area for buffering data, a coder and a decoder for coding and decoding and the like.
The video server may include: the device comprises an audio interface, a video interface, an augmented reality labeling information interface, a control terminal interface, an audio and video rate control module for audio and video rate control, a video quality compensation module for video quality compensation, an echo cancellation module for echo cancellation, a noise reduction processing module for noise reduction processing, a congestion control module for congestion control of data streams, a buffer area for buffering data, a coder and a decoder for coding and decoding and the like.
The WEB terminal may include: the device comprises a buffer area for buffering data, an encoder and a decoder for encoding and decoding, a synchronous playback module for data receiving synchronization, a display for video rendering and display, a collection device for obtaining external added information and a synchronous sending module for synchronously sending the added information.
The WEB terminal can comprise a video display module and is used for collecting the voice and the labeling information of an expert.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solutions of the present invention and not for limiting the scope of protection thereof, and although the present application is described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that: numerous variations, modifications, and equivalents will occur to those skilled in the art upon reading the present application and are within the scope of the claims appended hereto.
Claims (7)
1. A remote work guidance method, comprising:
the intelligent terminal adds a first relative timestamp to the collected audio, video and augmented reality labeling information to synchronize the audio, video and augmented reality labeling information and provides the synchronized audio, video and augmented reality labeling information to the WEB terminal;
the intelligent terminal acquires the video added with the second relative timestamp and the remote assistance information returned by the WEB terminal; the intelligent terminal synchronously plays back audio, video and augmented reality labeling information according to the first relative timestamp, and synchronously plays back the video and remote assistance information according to the second relative timestamp;
the intelligent terminal takes the video as a main stream and displays the video, the audio, the augmented reality labeling information and the remote assistance information after synchronous playback;
the intelligent terminal provides audio frequency, video and augmented reality mark information after the synchronization to the WEB terminal, includes: the intelligent terminal compresses and encodes the synchronized audio, video and augmented reality labeling information, sends the compressed and encoded audio, video and augmented reality labeling information to a video server through respective independent channels simultaneously or sequentially in the order of a video packet, an audio packet and an augmented reality labeling information packet, and then sends the compressed and encoded audio, video and augmented reality labeling information to a WEB end through the video server;
the synchronized playback of audio, video, and augmented reality annotation information according to the first relative timestamp comprises: and simultaneously extracting an audio frame, a video frame and an augmented reality labeling information frame for comparison, wherein the audio frame, the video frame and the augmented reality labeling information frame are provided with first relative timestamps, if the video frame is normal and the audio frame and the augmented reality labeling information frame are lagged, the audio frame and the augmented reality labeling information frame are discarded, if the audio frame and the video frame are synchronous and the augmented reality labeling information frame are lagged, the augmented reality labeling information frame is discarded, and if the video frame is lagged, the audio playing speed is slowed down or the video playing speed is accelerated.
2. The remote work guidance method according to claim 1, wherein the remote assistance information includes: remote assistance audio and remote assistance annotation information.
3. The remote work guidance method according to claim 2, wherein the synchronized playback of the video and the remote assistance information according to the second relative time stamp includes:
and simultaneously extracting a video frame, a remote assistance audio frame and a remote assistance annotation information frame for comparison, if the video frame is normal and the remote assistance audio frame and the remote assistance annotation information frame are both lagged, discarding the remote assistance audio frame and the remote assistance annotation information frame, if the remote assistance audio frame and the video frame are synchronous and the remote assistance annotation information frame is lagged, discarding the remote assistance annotation information frame, and if the video frame is lagged, slowing down the playing speed of the remote assistance audio or speeding up the playing speed of the video.
4. An intelligent terminal, comprising:
the first acquisition module is used for acquiring video, audio and augmented reality labeling information;
the first synchronization module is used for adding a first relative timestamp to the video, the audio and the augmented reality labeling information for synchronization;
the first communication module is used for acquiring the video added with the second relative timestamp and the remote assistance information from the WEB side;
the first synchronous playback module is used for synchronously playing back the video, the audio and the augmented reality labeling information according to the first relative timestamp; and synchronously playing back the video and the remote assistance information according to the second relative time stamp; the first display module is used for displaying the video, the audio, the augmented reality labeling information and the remote assistance information which are played back synchronously by taking the video as a main stream;
the remote assistance information acquired by the first communication module from the WEB side comprises remote assistance audio and remote assistance marking information;
the first synchronized playback module includes:
the first comparison unit is used for simultaneously extracting an audio frame, a video frame and an augmented reality labeling information frame according to a first relative timestamp for comparison, wherein the audio frame, the video frame and the augmented reality labeling information frame are printed with the first relative timestamp;
the first processing unit is used for discarding the audio frame and the augmented reality labeling information frame if the video frame is normal and the audio frame and the augmented reality labeling information frame are both lagged, discarding the augmented reality labeling information frame if the audio frame and the video frame are synchronous and the augmented reality labeling information frame is lagged, and slowing down or accelerating the playing speed of the audio if the video frame is lagged;
the first synchronized playback module includes:
the second comparison unit is used for simultaneously extracting the video frame, the remote assistance audio frame and the remote assistance marking information frame according to the second relative time stamp for comparison;
and the second processing unit is used for discarding the remote assistance audio frame and the remote assistance annotation information frame if the video frame is normal and the remote assistance audio frame and the remote assistance annotation information frame are both lagged, discarding the remote assistance annotation information frame if the remote assistance audio frame and the video frame are synchronous and the remote assistance annotation information frame is lagged, and slowing down or speeding up the playing speed of the remote assistance audio if the video frame is lagged.
5. A remote work guidance method, comprising:
the WEB terminal acquires the audio, video and augmented reality labeling information added with the first relative timestamp from the intelligent terminal, and synchronously plays back and displays the audio, video and augmented reality labeling information according to the first relative timestamp;
a WEB terminal acquires remote assistance information and adds a second relative timestamp to the remote assistance information and the video for synchronization;
the WEB terminal provides the synchronized video added with the second relative timestamp and the remote assistance information to the intelligent terminal; the step of sending the synchronized remote assistance information and video to the intelligent terminal by the WEB side comprises the following steps:
the WEB side compresses and encodes the synchronized remote assistance information and video and then sends the compressed and encoded remote assistance information and video to the intelligent terminal through the video server;
the synchronized playback of audio, video, and augmented reality annotation information according to the first relative timestamp comprises: and simultaneously extracting an audio frame, a video frame and an augmented reality labeling information frame for comparison, wherein the audio frame, the video frame and the augmented reality labeling information frame are marked with first relative timestamps, if the video frame is normal and the audio frame and the augmented reality labeling information frame are lagged, the audio frame and the augmented reality labeling information frame are discarded, if the audio frame and the video frame are synchronous and the augmented reality labeling information frame are lagged, the augmented reality labeling information frame is discarded, and if the video frame is lagged, the audio playing speed is slowed down or the video playing speed is accelerated.
6. A WEB side, comprising:
the second acquisition module is used for acquiring remote assistance information;
the second synchronous playback module is used for synchronously playing back the video, the audio and the augmented reality labeling information according to the first relative timestamp;
the second display module is used for displaying the video, the audio and the augmented reality labeling information after synchronous playback;
the second synchronization module is used for synchronizing the remote assistance information and the added video with a second relative timestamp; the second communication module is used for providing the synchronized video added with the second relative timestamp and the remote assistance information for the intelligent terminal;
the second synchronous playback module extracts an audio frame, a video frame and an augmented reality labeling information frame at the same time according to the first relative timestamp for comparison, the audio frame, the video frame and the augmented reality labeling information frame are marked with the first relative timestamp, if the video frame is normal and the audio frame and the augmented reality labeling information frame are lagged, the audio frame and the augmented reality labeling information frame are discarded, if the audio frame and the video frame are synchronous and the augmented reality labeling information frame are lagged, the augmented reality labeling information frame is discarded, and if the video frame is lagged, the audio playing speed is slowed down or the video playing speed is accelerated.
7. A remote work guidance system comprising the intelligent terminal according to claim 4, the WEB terminal according to claim 6, and a video server communicating with the intelligent terminal and the WEB terminal, respectively, the video server comprising:
the video quality optimization module is used for performing audio and video rate control, quality compensation, echo elimination, noise reduction processing and/or congestion control on the decoded data information added with the first timestamp and/or the second timestamp;
and the third communication module is used for communicating with the intelligent terminal and/or the WEB terminal.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710523982.6A CN109218794B (en) | 2017-06-30 | 2017-06-30 | Remote operation guidance method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710523982.6A CN109218794B (en) | 2017-06-30 | 2017-06-30 | Remote operation guidance method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109218794A CN109218794A (en) | 2019-01-15 |
| CN109218794B true CN109218794B (en) | 2022-06-10 |
Family
ID=64961029
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710523982.6A Active CN109218794B (en) | 2017-06-30 | 2017-06-30 | Remote operation guidance method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109218794B (en) |
Families Citing this family (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110290340A (en) * | 2019-07-02 | 2019-09-27 | 江苏达实久信数字医疗科技有限公司 | A kind of medical operating teaching system and method |
| CN110740285A (en) * | 2019-10-30 | 2020-01-31 | 中车青岛四方机车车辆股份有限公司 | telematics method and device |
| CN113766178B (en) * | 2020-06-05 | 2023-04-07 | 北京字节跳动网络技术有限公司 | Video control method, device, terminal and storage medium |
| CN112804559A (en) * | 2020-12-08 | 2021-05-14 | 中国船舶重工集团公司第七0九研究所 | Video picture and keyboard and mouse synchronous transmission method and system |
| CN112887651A (en) * | 2021-01-27 | 2021-06-01 | 昭通亮风台信息科技有限公司 | AR-based remote command method and system |
| CN113891132A (en) * | 2021-10-25 | 2022-01-04 | 北京字节跳动网络技术有限公司 | Audio and video synchronization monitoring method and device, electronic equipment and storage medium |
Family Cites Families (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101127917B (en) * | 2007-09-06 | 2010-07-14 | 中兴通讯股份有限公司 | A method and system for synchronizing Internet stream media format video and audio |
| US20090103897A1 (en) * | 2007-10-22 | 2009-04-23 | Min-Shu Chen | Method for synchronzing audio and video data in avi file |
| CN101453655A (en) * | 2007-11-30 | 2009-06-10 | 深圳华为通信技术有限公司 | Method, system and device for customer controllable audio and video synchronization regulation |
| CN101827271B (en) * | 2009-03-04 | 2012-07-18 | 联芯科技有限公司 | Audio and video synchronized method and device as well as data receiving terminal |
| CN105898857B (en) * | 2009-06-23 | 2021-05-07 | 北京三星通信技术研究有限公司 | Data synchronization method and system |
| CN102957892A (en) * | 2011-08-24 | 2013-03-06 | 三星电子（中国）研发中心 | Method, system and device for realizing audio and video conference |
| CN103634621B (en) * | 2012-08-27 | 2019-04-16 | 中兴通讯股份有限公司 | Synchronisation control means and device, system are played in a kind of video recommendations business |
| US9516440B2 (en) * | 2012-10-01 | 2016-12-06 | Sonos | Providing a multi-channel and a multi-zone audio environment |
| US9030495B2 (en) * | 2012-11-21 | 2015-05-12 | Microsoft Technology Licensing, Llc | Augmented reality help |
| CN103414957A (en) * | 2013-07-30 | 2013-11-27 | 广东工业大学 | Method and device for synchronization of audio data and video data |
| US10134296B2 (en) * | 2013-10-03 | 2018-11-20 | Autodesk, Inc. | Enhancing movement training with an augmented reality mirror |
| CN104618786B (en) * | 2014-12-22 | 2018-01-05 | 深圳市腾讯计算机系统有限公司 | Audio and video synchronization method and device |
| CN104506621B (en) * | 2014-12-24 | 2018-07-31 | 北京佳讯飞鸿电气股份有限公司 | A method of remote guide is carried out using video labeling |
| WO2016117480A1 (en) * | 2015-01-19 | 2016-07-28 | シャープ株式会社 | Telecommunication system |
| CN105739704A (en) * | 2016-02-02 | 2016-07-06 | 上海尚镜信息科技有限公司 | Remote guidance method and system based on augmented reality |
| US20180284735A1 (en) * | 2016-05-09 | 2018-10-04 | StrongForce IoT Portfolio 2016, LLC | Methods and systems for industrial internet of things data collection in a network sensitive upstream oil and gas environment |
| CN106339094B (en) * | 2016-09-05 | 2019-02-26 | 山东万腾电子科技有限公司 | Interactive remote expert cooperation examination and repair system and method based on augmented reality |
| CN106780151A (en) * | 2017-01-04 | 2017-05-31 | 国网江苏省电力公司电力科学研究院 | Transformer station's Bidirectional intelligent cruising inspection system and method based on wearable augmented reality |
-
2017
- 2017-06-30 CN CN201710523982.6A patent/CN109218794B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN109218794A (en) | 2019-01-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109218794B (en) | Remote operation guidance method and system | |
| CN110868600B (en) | Target tracking video plug-flow method, display method, device and storage medium | |
| CN104618786A (en) | Audio/video synchronization method and device | |
| CN109361945A (en) | The meeting audiovisual system and its control method of a kind of quick transmission and synchronization | |
| CN107509100A (en) | Audio and video synchronization method, system, computer installation and computer-readable recording medium | |
| CN102957892A (en) | Method, system and device for realizing audio and video conference | |
| CN113163222A (en) | Video frame synchronization method, system, equipment and readable storage medium | |
| CN103702013A (en) | Frame synchronization method for multiple channels of real-time videos | |
| CN103546662A (en) | Audio and video synchronizing method in network monitoring system | |
| CN101742548B (en) | H.324M protocol-based 3G video telephone audio and video synchronization device and method thereof | |
| CN103414957A (en) | Method and device for synchronization of audio data and video data | |
| WO2012034442A1 (en) | System and method for realizing synchronous transmission and reception of scalable video coding service | |
| CN108243350A (en) | A kind of method and apparatus of audio-visual synchronization processing | |
| CN109257641B (en) | Audio and video synchronization method and system in wireless screen transmission | |
| CN101419827A (en) | Method for synchronzing audio and video data in avi file | |
| CN101729908A (en) | Synchronous multiplexing method for video and audio of transmission stream | |
| EP2276192A2 (en) | Method and apparatus for transmitting/receiving multi - channel audio signals using super frame | |
| RU2012130007A (en) | RECEIVING DEVICE, TRANSMITTING DEVICE, COMMUNICATION SYSTEM, RECEIVING DEVICE METHOD AND PROGRAM | |
| CN103049238A (en) | Method and device for transmitting image data | |
| CN109040818B (en) | Audio and video synchronization method, storage medium, electronic equipment and system during live broadcasting | |
| CN108122558A (en) | A kind of LATM AAC audio streams turn appearance implementation method and device in real time | |
| KR20130107438A (en) | Augmented broadcasting stream transmission device and method, and augmented broadcasting service providing device and method | |
| CN101218819A (en) | Method and apparatus for synchronizing data service with video service in digital multimedia broadcasting | |
| CN105187688B (en) | The method and system that a kind of real-time video and audio to mobile phone collection synchronizes | |
| CN102256128B (en) | Synchronous decoding method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |