CN116506673B - Audio-video recording and detecting equipment data synchronous linkage and sound control searching method - Google Patents
Audio-video recording and detecting equipment data synchronous linkage and sound control searching method Download PDFInfo
- Publication number
- CN116506673B CN116506673B CN202310793851.5A CN202310793851A CN116506673B CN 116506673 B CN116506673 B CN 116506673B CN 202310793851 A CN202310793851 A CN 202310793851A CN 116506673 B CN116506673 B CN 116506673B
- Authority
- CN
- China
- Prior art keywords
- video
- data
- audio
- handheld terminal
- recording
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 73
- 230000001360 synchronised effect Effects 0.000 title description 6
- 238000001514 detection method Methods 0.000 claims abstract description 79
- 238000012545 processing Methods 0.000 claims abstract description 51
- 238000004891 communication Methods 0.000 claims abstract description 19
- 238000007781 pre-processing Methods 0.000 claims abstract description 12
- 238000005457 optimization Methods 0.000 claims abstract description 9
- 230000002194 synthesizing effect Effects 0.000 claims abstract description 5
- 230000005540 biological transmission Effects 0.000 claims description 69
- 238000005516 engineering process Methods 0.000 claims description 19
- 230000008569 process Effects 0.000 claims description 15
- 230000005236 sound signal Effects 0.000 claims description 13
- 230000006870 function Effects 0.000 claims description 12
- 238000013500 data storage Methods 0.000 claims description 9
- 238000003058 natural language processing Methods 0.000 claims description 9
- 230000011218 segmentation Effects 0.000 claims description 8
- 239000008186 active pharmaceutical agent Substances 0.000 claims description 7
- 238000007689 inspection Methods 0.000 claims description 5
- 230000001133 acceleration Effects 0.000 claims description 4
- 238000006243 chemical reaction Methods 0.000 claims description 4
- 238000006073 displacement reaction Methods 0.000 claims description 4
- 238000001914 filtration Methods 0.000 claims description 4
- 230000008676 import Effects 0.000 claims description 3
- 239000006247 magnetic powder Substances 0.000 claims description 3
- 238000001228 spectrum Methods 0.000 claims description 3
- 238000001931 thermography Methods 0.000 claims description 3
- 230000000007 visual effect Effects 0.000 claims 1
- 230000010354 integration Effects 0.000 abstract description 5
- 238000004458 analytical method Methods 0.000 description 13
- 238000012546 transfer Methods 0.000 description 10
- 238000009877 rendering Methods 0.000 description 9
- 238000005070 sampling Methods 0.000 description 9
- 230000006835 compression Effects 0.000 description 3
- 238000007906 compression Methods 0.000 description 3
- 238000013523 data management Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 230000002093 peripheral effect Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 230000015572 biosynthetic process Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000004321 preservation Methods 0.000 description 1
- 238000003786 synthesis reaction Methods 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/4302—Content synchronisation processes, e.g. decoder synchronisation
- H04N21/4307—Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen
- H04N21/43076—Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen of the same content streams on multiple devices, e.g. when family members are watching the same movie on different devices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/70—Information retrieval; Database structures therefor; File system structures therefor of video data
- G06F16/73—Querying
- G06F16/735—Filtering based on additional data, e.g. user or group profiles
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/70—Information retrieval; Database structures therefor; File system structures therefor of video data
- G06F16/78—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F16/783—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
- G06F16/7844—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using original textual content or text extracted from visual content or transcript of audio data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T1/00—General purpose image data processing
- G06T1/0021—Image watermarking
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/433—Content storage operation, e.g. storage operation in response to a pause request, caching operations
- H04N21/4334—Recording operations
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/436—Interfacing a local distribution network, e.g. communicating with another STB or one or more peripheral devices inside the home
- H04N21/4363—Adapting the video stream to a specific local network, e.g. a Bluetooth® network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/439—Processing of audio elementary streams
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
- H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/83—Generation or processing of protective or descriptive data associated with content; Content structuring
- H04N21/835—Generation of protective data, e.g. certificates
- H04N21/8358—Generation of protective data, e.g. certificates involving watermark
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D30/00—Reducing energy consumption in communication networks
- Y02D30/70—Reducing energy consumption in communication networks in wireless communication networks
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- Library & Information Science (AREA)
- Computer Security & Cryptography (AREA)
- Computational Linguistics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Television Signal Processing For Recording (AREA)
- Management Or Editing Of Information On Record Carriers (AREA)
Abstract
The invention discloses a method for synchronously linking audio-video recording and detecting equipment data and retrieving by sound control, which comprises the following steps of 1) audio-video data acquisition: acquiring and processing required data by using an audio-video recorder and detection equipment; 2) After the audio-video data is transmitted wirelessly or recorded by the recording equipment in the step 1), the processed data is transmitted to the handheld terminal in a wireless communication mode; 3) Synthesizing and storing audio and video data: after the step 2), the data received by the handheld terminal is subjected to watermark processing, data preprocessing, presentation processing, conventional optimization and storage; 4) The data is exported from the handheld terminal in a wired or wireless mode; 5) Completing record retrieval by adopting a text retrieval mode or a sound control retrieval mode; by linking the audio-video recording device (audio-video recorder) with the detection data acquisition device (detection device), automatic recording and data integration can be realized, and errors and loopholes of manual operation are reduced.
Description
Technical Field
The invention relates to the technical field of detection, in particular to a method for synchronously linking audio-video recording with data of detection equipment and retrieving by sound control.
Background
In the field of specialty equipment inspection, medical equipment, and environmental protection equipment, it is often necessary to inspect and maintain equipment, and these inspection data must be recorded for subsequent analysis and processing. At the same time, field personnel need to quickly find the required data in order to perform the corresponding operation.
The traditional recording method is to record by manual recording or using a notebook computer and other devices, and the method has the problems of inaccurate recording, incomplete recording, untimely recording and the like; meanwhile, the detected data is recorded and displayed in the recorded video in real time in the detected process of the detected object, and the method solves the problem that the detected data is possibly false because the detected data and the detected audio and video are not synchronously displayed on one interface. Moreover, when it is necessary to find a specific data, it takes a lot of time and effort to search.
Therefore, a method for synchronously linking audio-video recording and detecting equipment data and retrieving sound control is needed.
Disclosure of Invention
The invention aims to design a method for synchronously linking and retrieving audio and video recording and detecting equipment data, which aims to improve the synchronism, the authenticity, the efficiency and the accuracy of the on-site audio and video recording and detecting data and facilitate the subsequent data management and retrieval. By synchronously linking the audio-video recording equipment (audio-video recorder) with the detection data acquisition equipment (detection equipment), automatic recording and data integration can be realized, and errors and loopholes of manual operation are reduced. Meanwhile, through a voice control searching mode, required records and data can be quickly found, and the working efficiency is improved.
The invention is realized by the following technical scheme: a method for synchronously linking audio-video recording and detecting equipment data and retrieving sound control comprises the following specific steps:
1) And (3) audio and video data acquisition: acquiring and processing required data by using an audio-video recorder and detection equipment;
2) And (3) audio-video data wireless transmission: after the step 1), the processed data is transmitted to the handheld terminal in a wireless communication mode; the handheld terminal can be a smart phone, a tablet computer and other devices;
3) Synthesizing and storing audio and video data: after the step 2), the data received by the handheld terminal is subjected to watermark processing (without limitation), data preprocessing, presentation processing and conventional optimization and is stored;
4) The data is exported from the handheld terminal in a wired or wireless mode;
5) And completing record retrieval by adopting a text retrieval mode or a voice control retrieval mode.
Further, in order to better realize the method for synchronously linking the audio-video recording and the data of the detection equipment and retrieving the audio-video, the invention adopts the following setting modes: the step 1) comprises the following specific steps:
1.1 Signal input: inputting an audio signal into an audio-video recorder (such as a webcam) through a recording device (such as a microphone, a headset and the like); the video signal is input into an audio-video recorder through a camera or screen capturing software; collecting various sensor signals by using a detection device (such as a multifunctional data collector); the sampling rate used by the recording device in recording audio includes, but is not limited to, 44.1khz or 48khz, with a typical number of sampling bits being 16 bits or 24 bits; resolution typically employed in video recording includes, but is not limited to, 720p or 1080p; the audio-video recorder may employ a device supporting video capture, such as an IP camera supporting the RTSP protocol, or may use the RTSP service plug-flow of the device itself by accessing the camera into a transcoding device supporting the RTSP protocol. The detection equipment (such as a multifunctional data acquisition instrument) capable of recording various physical data (sensor data) on site supports wireless communication modes such as WI-FI or Bluetooth. Therefore, at the acquisition end, the video stream can be pushed by RTSP, and the physical detection data can be sent to the handheld terminal in a wireless communication mode such as WI-FI or Bluetooth.
When audio and video data are acquired, for example, the influence of background noise is noted when the audio data are acquired, so that the problems of excessively complex scenes and the like are avoided. The problems of acquisition angle and arrangement of the position of the audio-video recorder are also required to be noted when acquiring video data. The collected audio-video data can achieve the best effect through strict specifications and requirements.
1.2 Signal conversion: converting the audio signal, the video signal and the sensor signal into an audio data digital signal, a video data digital signal and a sensor data digital signal through an analog-to-digital converter; analog-to-digital converters are key devices for acquisition, which convert input signals (audio signals, video signals, sensor signals) into digital signals, the accuracy and sampling rate of which have a decisive influence on the accuracy and recording quality of the data.
1.3 After step 1.2), storing the obtained digital signal in a storage device of an audio-video recorder, and performing processing and treatment such as editing, compressing, archiving and the like on the digital signal; the storage device comprises, but is not limited to, a Flash and an SD card in the handheld terminal.
Further, in order to better realize the method for synchronously linking the audio-video recording and the data of the detection equipment and retrieving the audio-video, the invention adopts the following setting modes: the sensor comprises one or more of a gyroscope, an acceleration sensor, a displacement sensor, noise, temperature and humidity, laser ranging, angle, infrared thermal imaging, current and voltage, rotating speed, ultrasonic waves, magnetic leakage, vortex, acoustic emission, magnetic powder, strong magnetic, weak magnetic, ray, gas, spectrum, stress, coulometer sensor and the like, and the sensor and the inspection equipment are transmitted through standard IIC, serial ports, bluetooth and WIFI.
Further, in order to better realize the method for synchronously linking the audio-video recording and the data of the detection equipment and retrieving the audio-video, the invention adopts the following setting modes: the step 2) includes two ways:
mode one: establishing WiFi communication between the audio-video recorder and the handheld terminal, and transmitting the audio and video data finally processed by the audio-video recorder into the handheld terminal based on RTSP through a WiFi transmission mode;
mode two: and establishing Bluetooth communication between the detection equipment and the handheld terminal, and transmitting the sensor data finally processed by the detection equipment into the handheld terminal based on a BLE protocol through a Bluetooth transmission mode.
In the case of wireless transmission, it is necessary to select an appropriate transmission scheme for transmitting audio/video data (audio signal, video signal, sensor signal) wirelessly. Including but not limited to Wi-Fi and bluetooth. If the network environment supports Wi-Fi connection, wi-Fi can be directly used for audio and video data transmission. If the network environment is poor or a Wi-Fi network is not available, the Bluetooth-supported device can be selected to transmit audio and video data through Bluetooth, and the specific preference is: wiFi is adopted between the audio-video recorder and the handheld terminal, and Bluetooth is adopted between the detection equipment and the handheld terminal; the audio-video recorder transmits and selects the RTSP protocol for transmission. RTSP is a real-time streaming protocol, which can realize real-time transmission of audio-video data.
The detection device data format adopts a JSON format (not limited to the JSON format), and other parameters such as device ID, sensor data, acquisition time stamp and the like are included. The Bluetooth BLE (Bluetooth Low energy) protocol is used for communication, a handheld terminal (including but not limited to a smart phone, a tablet computer and the like) is used as a central device, and a sensor is used as a peripheral. After the detection equipment collects data, a data packet is sent in a BLE broadcast mode, and the handheld terminal subscribes to the BLE broadcast, so that the detection data is received.
Further, in order to better realize the method for synchronously linking the audio-video recording and the data of the detection equipment and retrieving the audio-video, the invention adopts the following setting modes: the step 3) comprises the following specific steps:
3.1 Watermarking process:
3.1.1 Reading a video file: using a video processing library (FFmpeg) to read video data in the handheld terminal and acquire information such as the frame number, frame rate, resolution and the like of the video;
3.1.2 Watermark picture generation: after step 3.1.1), converting the information into a gray image or a color image using an image processing library (OpenCV);
3.1.3 Reading the watermark picture: using an image processing library (OpenCV) to read the watermark picture and obtaining the information such as the size and the transparency of the watermark picture;
3.1.4 Converting the watermark picture into a transparency channel: converting the watermark picture into RGBA format, and storing transparency information of the watermark picture in a transparency channel;
3.1.5 Scaling the watermark picture to a suitable size: according to the resolution of the video and the size of the watermark picture, calculating the scaling required by the watermark picture, and then scaling the watermark picture to be matched with the size of the video picture by using an image processing library;
3.1.6 Superimposing the watermark picture onto the video frame: for each frame of video, an image processing library is used for superposing watermark pictures on the video frames, and a transparency channel can be used for controlling the transparency of the watermark;
3.1.7 Outputting a video file: writing the processed video frames into a new video file to generate a video with a watermark;
3.2 Data preprocessing: processing the sensor data by using a filtering technology (the sensor data can be more accurate and reliable) and synchronizing the sensor data with the video data (realized by adopting a time stamp algorithm), so that watermark information is synchronized with the video data;
3.3 After the watermark is added, encoding and compressing the video data added with the watermark; wherein the watermark is visualized by an image processing library (Opencv); i.e. after the watermarking is completed, a rendering process is required to enable normal display in the new video file. The rendering process requires encoding and compression of the watermarked video, enabling it to reduce the size of the video file without affecting the video quality. The rendering process may also visualize the watermark through an image processing library (Opencv).
3.4 After the step 3.3), the video data is subjected to conventional optimization by using an optimized encryption algorithm, a sensor data preprocessing algorithm and a synchronization algorithm to form a video stream file;
3.5 Using a storage API in the handheld terminal (e.g., sharedreferences, etc.) to save the video stream file into a local audio-visual folder for later offline viewing.
Further, in order to better realize the method for synchronously linking the audio-video recording and the data of the detection equipment and retrieving the audio-video, the invention adopts the following setting modes: said step 4) comprises the steps of:
4.1 Using USB to import the local video stream file stored in the handheld terminal into the computer or other storage devices and exit;
4.2 Using bluetooth or WiFi to save the video stream file stored locally in the handheld terminal in the local device memory.
Further, in order to better realize the method for synchronously linking the audio-video recording and the data of the detection equipment and retrieving the audio-video, the invention adopts the following setting modes: said step 4.1) comprises the steps of:
4.1.1 Connected to the handheld terminal): connecting the handheld terminal to a computer through a USB; if the handheld terminal supports the USB data storage mode, directly accessing the storage of the handheld terminal or the file in the SD card; if the handheld terminal does not start the USB data storage mode, the function needs to be started in the setting of the handheld terminal;
4.1.2 Export file): opening a file manager on a computer, finding an audio/video folder of a video stream file stored in a handheld terminal or an SD card, selecting the video stream file to be exported, and copying the video stream file to a local disk or other storage devices of the computer;
4.1.3 After the file is exported, the handheld terminal is safely pulled out.
Further, in order to better realize the method for synchronously linking the audio-video recording and the data of the detection equipment and retrieving the audio-video, the invention adopts the following setting modes: said step 4.2) comprises the steps of:
4.2.1 Installing an application program supporting Bluetooth or WiFi file transfer on the handheld terminal, such as a common file manager application or a special Bluetooth file transfer application;
4.2.2 Starting Bluetooth or WiFi functions on the handheld terminal and pairing with a computer to be transmitted or other equipment supporting Bluetooth or WiFi transmission;
4.2.3 Opening an application program of Bluetooth or WiFi file transmission, selecting a video stream file to be exported, and adding the video stream file to a transmission list; ensuring that a connection between the handheld terminal and a computer or other bluetooth device is established.
4.2.4 Clicking the video stream file in the transmission list to start video stream file transmission; the transmission status and progress can be viewed through a transmission progress bar in the application.
4.2.5 A computer or other device supporting bluetooth or WiFi transmission receives the transmitted video stream file and stores it in device memory.
Further, in order to better realize the method for synchronously linking the audio-video recording and the data of the detection equipment and retrieving the audio-video, the invention adopts the following setting modes: the method for recording and searching by adopting the text searching concretely comprises the following steps: splitting a segment of words (how to split the words needs to be considered when the word segmentation algorithm is selected) into a plurality of meaningful words based on a word segmentation algorithm mode to form segmented words; and then establishing an inverted index relation for the segmented words in the text through a full text index technology, so that efficient text query is realized. The elastiscearch is a distributed search engine developed based on the Lucene search engine library, and can realize higher performance and scalability.
Further, in order to better realize the method for synchronously linking the audio-video recording and the data of the detection equipment and retrieving the audio-video, the invention adopts the following setting modes: the method for completing record retrieval by adopting the voice control retrieval mode comprises the following specific steps:
5.2.1 Voice recognition techniques are used to convert the sound recordings into text to form text data for subsequent processing. At present, there are several APIs for providing speech recognition services, such as hundred-degree speech recognition, ali cloud speech recognition, etc. on the market;
5.2.2 For the converted text data, natural language processing technology can be used for analysis, and related information including time, detection items, detection values and detection places in the converted text data is extracted; natural language processing techniques, including lexical analysis, grammatical analysis, semantic analysis, and the like, may be implemented with associated tools and libraries.
5.2.3 The extracted related information is stored in a database, so that subsequent inquiry and retrieval are facilitated. For example, a relational database such as MySQL may be used. In the case of querying, the SQL language may be used to query records within a specified time range, or query may be performed according to conditions such as a detection item or a detection location.
5.2.4 Developing an application program to provide the functions of voice recording and retrieving related records; during recording, a user can input related information through voice, and the handheld terminal automatically recognizes and stores the related information into a database; during retrieval, the user can input the query condition through voice, and the handheld terminal automatically retrieves related records from the database and returns the related records to the user.
Compared with the prior art, the invention has the following advantages:
the invention can carry out synchronous linkage recording of data by using the audio-video recording equipment (audio-video recorder) and the detection equipment, thereby improving the accuracy and the comprehensiveness of recording.
Compared with the prior art, the invention can reduce errors and loopholes of manual operation and improve the working efficiency through automatic recording and data integration.
Compared with the prior art, the invention can realize automatic integration of data, reduce errors and loopholes in the data processing process and improve the data accuracy by linking the audio-video recording equipment and the detection data acquisition equipment.
And the data management and retrieval are convenient: by means of voice control searching, needed records and data can be quickly found, and data management and searching are facilitated.
Compared with the prior art, the invention can effectively prevent data from being tampered and forged by adopting the technologies such as hash algorithm and the like, and improves the data security.
Compared with the prior art, the invention can reduce the cost of manual operation and lower the cost through automatic recording and data integration.
Drawings
Fig. 1 is a general flow chart of the present invention.
Fig. 2 is a diagram of audio/video and sensor data sources according to the present invention.
Fig. 3 is a wireless transmission diagram of audio and video data according to the present invention.
Fig. 4 is a flowchart of audio/video data synthesis and storage according to the present invention.
Fig. 5 is a flowchart of completing a record search by using a voice control search method according to the present invention.
Detailed Description
The present invention will be described in further detail with reference to examples, but embodiments of the present invention are not limited thereto.
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments. All other embodiments, based on the embodiments of the invention, which are apparent to those of ordinary skill in the art without inventive faculty, are intended to be within the scope of the invention. Thus, the following detailed description of the embodiments of the invention, as presented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention.
In the description of the present invention, it should be understood that the orientation or positional relationship indicated by the terms and the like is based on the orientation or positional relationship shown in the drawings, and is merely for convenience in describing the present invention and simplifying the description, and does not indicate or imply that the apparatus or element to be referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus should not be construed as limiting the present invention.
Furthermore, the terms "first," "second," and the like, are used for descriptive purposes only and are not to be construed as indicating or implying a relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include one or more such feature. In the description of the present invention, the meaning of "a plurality" is two or more, unless explicitly defined otherwise.
In the present invention, unless explicitly specified and limited otherwise, the terms "mounted," "connected," "disposed," "deployed," "secured," and the like are to be construed broadly, and may be, for example, fixedly connected, detachably connected, or integrally formed, particularly by means other than by screwing, interference fit, riveting, screw-assisted connection, and the like, in any of a variety of conventional mechanical connection means. The specific meaning of the above terms in the present invention can be understood by those of ordinary skill in the art according to the specific circumstances.
In the present invention, unless expressly stated or limited otherwise, a first feature "above" or "below" a second feature may include both the first and second features being in direct contact, as well as the first and second features not being in direct contact but being in contact with each other through additional features therebetween. Moreover, a first feature being "above," "over" and "on" a second feature includes the first feature being directly above and obliquely above the second feature, or simply indicating that the first feature is higher in level than the second feature. The first feature being "under", "below" and "beneath" the second feature includes the first feature being directly under and obliquely below the second feature, or simply means that the first feature is less level than the second feature.
Example 1:
a method for synchronously linking audio-video recording and detecting equipment data and retrieving sound control comprises the following specific steps:
1) And (3) audio and video data acquisition: acquiring and processing required data by using an audio-video recorder and detection equipment;
2) And (3) audio-video data wireless transmission: after the step 1), the processed data is transmitted to the handheld terminal in a wireless communication mode; the handheld terminal can be a smart phone, a tablet computer and other devices;
3) Synthesizing and storing audio and video data: after the step 2), the data received by the handheld terminal is subjected to watermark processing (without limitation), data preprocessing, presentation processing and conventional optimization and is stored;
4) The data is exported from the handheld terminal in a wired or wireless mode;
5) And completing record retrieval by adopting a text retrieval mode or a voice control retrieval mode.
Example 2:
the embodiment is further optimized based on the above embodiment, and the same features as the foregoing technical solutions are not repeated herein, so as to further better implement the method for synchronously linking audio-video recording and detecting equipment data and retrieving voice control according to the present invention, and particularly adopts the following setting manner: the step 1) comprises the following specific steps:
1.1 Signal input: inputting an audio signal into an audio-video recorder (such as a webcam) through a recording device (such as a microphone, a headset and the like); the video signal is input into an audio-video recorder through a camera or screen capturing software; collecting various sensor signals by using a detection device (such as a multifunctional data collector); the sampling rate used by the recording device in recording audio includes, but is not limited to, 44.1khz or 48khz, with a typical number of sampling bits being 16 bits or 24 bits; resolution typically employed in video recording includes, but is not limited to, 720p or 1080p; the audio-video recorder may employ a device supporting video capture, such as an IP camera supporting the RTSP protocol, or may use the RTSP service plug-flow of the device itself by accessing the camera into a transcoding device supporting the RTSP protocol. The detection equipment (such as a multifunctional data acquisition instrument) capable of recording various physical data (sensor data) on site supports wireless communication modes such as WI-FI or Bluetooth. Therefore, at the acquisition end, the video stream can be pushed by RTSP, and the physical detection data can be sent to the handheld terminal in a wireless communication mode such as WI-FI or Bluetooth.
When audio and video data are acquired, for example, the influence of background noise is noted when the audio data are acquired, so that the problems of excessively complex scenes and the like are avoided. The problems of acquisition angle and arrangement of the position of the audio-video recorder are also required to be noted when acquiring video data. The collected audio-video data can achieve the best effect through strict specifications and requirements.
1.2 Signal conversion: converting the audio signal, the video signal and the sensor signal into an audio data digital signal, a video data digital signal and a sensor data digital signal through an analog-to-digital converter; analog-to-digital converters are key devices for acquisition, which convert input signals (audio signals, video signals, sensor signals) into digital signals, the accuracy and sampling rate of which have a decisive influence on the accuracy and recording quality of the data.
1.3 After step 1.2), storing the obtained digital signal in a storage device of an audio-video recorder, and performing processing and treatment such as editing, compressing, archiving and the like on the digital signal; the storage device comprises, but is not limited to, a Flash and an SD card in the handheld terminal.
Example 3:
the embodiment is further optimized on the basis of any one of the embodiments, and the same features as the foregoing technical solutions are not repeated herein, so as to further better implement the method for synchronously linking audio-video recording with data of the detection device and retrieving the audio-video data, and particularly adopt the following setting modes: the sensor comprises one or more of a gyroscope, an acceleration sensor, a displacement sensor, noise, temperature and humidity, laser ranging, angle, infrared thermal imaging, current and voltage, rotating speed, ultrasonic waves, magnetic leakage, vortex, acoustic emission, magnetic powder, strong magnetic, weak magnetic, ray, gas, spectrum, stress, coulometer sensor and the like, and the sensor and the inspection equipment are transmitted through standard IIC, serial ports, bluetooth and WIFI.
Example 4:
the embodiment is further optimized on the basis of any one of the embodiments, and the same features as the foregoing technical solutions are not repeated herein, so as to further better implement the method for synchronously linking audio-video recording with data of the detection device and retrieving the audio-video data, and particularly adopt the following setting modes: the step 2) includes two ways:
mode one: establishing WiFi communication between the audio-video recorder and the handheld terminal, and transmitting the audio and video data finally processed by the audio-video recorder into the handheld terminal based on RTSP through a WiFi transmission mode;
mode two: and establishing Bluetooth communication between the detection equipment and the handheld terminal, and transmitting the sensor data finally processed by the detection equipment into the handheld terminal based on a BLE protocol through a Bluetooth transmission mode.
In the case of wireless transmission, it is necessary to select an appropriate transmission scheme for transmitting audio/video data (audio signal, video signal, sensor signal) wirelessly. Including but not limited to Wi-Fi and bluetooth. If the network environment supports Wi-Fi connection, wi-Fi can be directly used for audio and video data transmission. If the network environment is poor or a Wi-Fi network is not available, the Bluetooth-supported device can be selected to transmit audio and video data through Bluetooth, and the specific preference is: wiFi is adopted between the audio-video recorder and the handheld terminal, and Bluetooth is adopted between the detection equipment and the handheld terminal; the audio-video recorder transmits and selects the RTSP protocol for transmission. RTSP is a real-time streaming protocol, which can realize real-time transmission of audio-video data.
The detection device data format adopts a JSON format (not limited to the JSON format), and other parameters such as device ID, sensor data, acquisition time stamp and the like are included. The Bluetooth BLE (Bluetooth Low energy) protocol is used for communication, a handheld terminal (including but not limited to a smart phone, a tablet computer and the like) is used as a central device, and a sensor is used as a peripheral. After the detection equipment collects data, a data packet is sent in a BLE broadcast mode, and the handheld terminal subscribes to the BLE broadcast, so that the detection data is received.
Example 5:
the embodiment is further optimized on the basis of any one of the embodiments, and the same features as the foregoing technical solutions are not repeated herein, so as to further better implement the method for synchronously linking audio-video recording with data of the detection device and retrieving the audio-video data, and particularly adopt the following setting modes: the step 3) comprises the following specific steps:
3.1 Watermarking process:
3.1.1 Reading a video file: using a video processing library (FFmpeg) to read video data in the handheld terminal and acquire information such as the frame number, frame rate, resolution and the like of the video;
3.1.2 Watermark picture generation: after step 3.1.1), converting the information into a gray image or a color image using an image processing library (OpenCV);
3.1.3 Reading the watermark picture: using an image processing library (OpenCV) to read the watermark picture and obtaining the information such as the size and the transparency of the watermark picture;
3.1.4 Converting the watermark picture into a transparency channel: converting the watermark picture into RGBA format, and storing transparency information of the watermark picture in a transparency channel;
3.1.5 Scaling the watermark picture to a suitable size: according to the resolution of the video and the size of the watermark picture, calculating the scaling required by the watermark picture, and then scaling the watermark picture to be matched with the size of the video picture by using an image processing library;
3.1.6 Superimposing the watermark picture onto the video frame: for each frame of video, an image processing library is used for superposing watermark pictures on the video frames, and a transparency channel can be used for controlling the transparency of the watermark;
3.1.7 Outputting a video file: writing the processed video frames into a new video file to generate a video with a watermark;
3.2 Data preprocessing: processing the sensor data by using a filtering technology (the sensor data can be more accurate and reliable) and synchronizing the sensor data with the video data (realized by adopting a time stamp algorithm), so that watermark information is synchronized with the video data;
3.3 After the watermark is added, encoding and compressing the video data added with the watermark; wherein the watermark is visualized by an image processing library (Opencv); i.e. after the watermarking is completed, a rendering process is required to enable normal display in the new video file. The rendering process requires encoding and compression of the watermarked video, enabling it to reduce the size of the video file without affecting the video quality. The rendering process may also visualize the watermark through an image processing library (Opencv).
3.4 After the step 3.3), the video data is subjected to conventional optimization by using an optimized encryption algorithm, a sensor data preprocessing algorithm and a synchronization algorithm to form a video stream file;
3.5 Using a storage API in the handheld terminal (e.g., sharedreferences, etc.) to save the video stream file into a local audio-visual folder for later offline viewing.
Example 6:
the embodiment is further optimized on the basis of any one of the embodiments, and the same features as the foregoing technical solutions are not repeated herein, so as to further better implement the method for synchronously linking audio-video recording with data of the detection device and retrieving the audio-video data, and particularly adopt the following setting modes: said step 4) comprises the steps of:
4.1 Using USB to import the local video stream file stored in the handheld terminal into the computer or other storage devices and exit;
4.2 Using bluetooth or WiFi to save the video stream file stored locally in the handheld terminal in the local device memory.
Example 7:
the embodiment is further optimized on the basis of any one of the embodiments, and the same features as the foregoing technical solutions are not repeated herein, so as to further better implement the method for synchronously linking audio-video recording with data of the detection device and retrieving the audio-video data, and particularly adopt the following setting modes: said step 4.1) comprises the steps of:
4.1.1 Connected to the handheld terminal): connecting the handheld terminal to a computer through a USB; if the handheld terminal supports the USB data storage mode, directly accessing files stored in the handheld terminal or the SD card; if the handheld terminal does not start the USB data storage mode, the function needs to be started in the setting of the handheld terminal;
4.1.2 Export file): opening a file manager on a computer, finding an audio/video folder of a video stream file stored in a handheld terminal or an SD card, selecting the video stream file to be exported, and copying the video stream file to a local disk or other storage devices of the computer;
4.1.3 After the file is exported, the handheld terminal is safely pulled out.
Example 8:
the embodiment is further optimized on the basis of any one of the embodiments, and the same features as the foregoing technical solutions are not repeated herein, so as to further better implement the method for synchronously linking audio-video recording with data of the detection device and retrieving the audio-video data, and particularly adopt the following setting modes: said step 4.2) comprises the steps of:
4.2.1 Installing an application program supporting Bluetooth or WiFi file transfer on the handheld terminal, such as a common file manager application or a special Bluetooth file transfer application;
4.2.2 Starting Bluetooth or WiFi functions on the handheld terminal and pairing with a computer to be transmitted or other equipment supporting Bluetooth or WiFi transmission;
4.2.3 Opening an application program of Bluetooth or WiFi file transmission, selecting a video stream file to be exported, and adding the video stream file to a transmission list; ensuring that a connection between the handheld terminal and a computer or other bluetooth device is established.
4.2.4 Clicking the video stream file in the transmission list to start video stream file transmission; the transmission status and progress can be viewed through a transmission progress bar in the application.
4.2.5 A computer or other device supporting bluetooth or WiFi transmission receives the transmitted video stream file and stores it in device memory.
In practical applications, it should be noted that bluetooth transmission speed is slow, and the transmission distance is generally not more than 10 meters. Meanwhile, bluetooth transmission requires pairing between devices, and certain operation steps and time are required. For large file transfers, the time will be longer, so bluetooth transfers are not suitable for large and infrequent data transfers.
Example 9:
the embodiment is further optimized on the basis of any one of the embodiments, and the same features as the foregoing technical solutions are not repeated herein, so as to further better implement the method for synchronously linking audio-video recording with data of the detection device and retrieving the audio-video data, and particularly adopt the following setting modes: the method for recording and searching by adopting the text searching concretely comprises the following steps: splitting a segment of words (how to split the words needs to be considered when the word segmentation algorithm is selected) into a plurality of meaningful words based on a word segmentation algorithm mode to form segmented words; and then establishing an inverted index relation for the segmented words in the text through a full text index technology, so that efficient text query is realized. The elastiscearch is a distributed search engine developed based on the Lucene search engine library, and can realize higher performance and scalability.
Example 10:
the embodiment is further optimized on the basis of any one of the embodiments, and the same features as the foregoing technical solutions are not repeated herein, so as to further better implement the method for synchronously linking audio-video recording with data of the detection device and retrieving the audio-video data, and particularly adopt the following setting modes: the method for completing record retrieval by adopting the voice control retrieval mode comprises the following specific steps:
5.2.1 Voice recognition techniques are used to convert the sound recordings into text to form text data for subsequent processing. At present, there are several APIs for providing speech recognition services, such as hundred-degree speech recognition, ali cloud speech recognition, etc. on the market;
5.2.2 For the converted text data, natural language processing technology can be used for analysis, and related information such as time, detection items, detection values, detection places and the like in the text data can be extracted; natural language processing techniques, including lexical analysis, grammatical analysis, semantic analysis, and the like, may be implemented with associated tools and libraries.
5.2.3 The extracted related information is stored in a database, so that subsequent inquiry and retrieval are facilitated. For example, a relational database such as MySQL may be used. In the case of querying, the SQL language may be used to query records within a specified time range, or query may be performed according to conditions such as a detection item or a detection location.
5.2.4 Developing an application program to provide the functions of voice recording and retrieving related records; during recording, a user can input related information through voice, and the handheld terminal automatically recognizes and stores the related information into a database; during retrieval, a user can input query conditions through voice, and the system handheld terminal automatically retrieves related records from the database and returns the related records to the user.
It should be noted that, at present, a certain error recognition and error rate still exist in the voice recognition technology and the natural language processing technology, so that proper debugging and improvement are required, and the accuracy and the use experience of the system are improved.
Example 11:
a method for synchronously linking audio and video recording and detecting equipment data and retrieving sound control is shown in fig. 1-5, and comprises the following steps:
and (3) audio and video data acquisition:
1. and (3) signal input: inputting an audio signal into an audio-video recorder (such as a webcam) through a recording device (such as a microphone, a headset and the like); the video signal is input into an audio-video recorder through a camera or screen capturing software; the data sources of the detection equipment include but are not limited to gyroscopes, gravitational acceleration, capacitive grating displacement sensors, noise detection sensors and coulombmeters, and the various sensors and the detection equipment are transmitted by means of standard IIC, serial ports, bluetooth and the like.
The sampling rate used by the recording device when recording audio is 44.1khz or 48khz, and the sampling bit number is generally 16 bits or 24 bits; resolution typically employed in video recording includes, but is not limited to, 720p or 1080p; the audio-video recorder may employ a device supporting video capture, such as an IP camera supporting the RTSP protocol, or may use the RTSP service plug-flow of the device itself by accessing the camera into a transcoding device supporting the RTSP protocol. The detection equipment (such as a multifunctional data acquisition instrument) capable of recording various physical data (sensor data) on site supports wireless communication modes such as WI-FI or Bluetooth. Therefore, at the acquisition end, the video stream can be pushed by RTSP, and the physical detection data can be sent to the handheld terminal in a wireless communication mode such as WI-FI or Bluetooth.
When audio and video data are acquired, for example, the influence of background noise is noted when the audio data are acquired, so that the problems of excessively complex scenes and the like are avoided. The problems of acquisition angle and arrangement of the position of the audio-video recorder are also required to be noted when acquiring video data. The collected audio-video data can achieve the best effect through strict specifications and requirements.
2. Signal conversion: converting the audio signal, the video signal and the sensor signal into an audio data digital signal, a video data digital signal and a sensor data digital signal through an analog-to-digital converter; analog-to-digital converters are key devices for acquisition, which convert input signals (audio signals, video signals, sensor signals) into digital signals, the accuracy and sampling rate of which have a decisive influence on the accuracy and recording quality of the data.
3. Digital signal processing: after the step 1.2), storing the obtained digital signals in a storage device of an audio-video recorder, and performing processing and treatment such as editing, compressing, archiving and the like on the digital signals; the storage device comprises, but is not limited to, a Flash and an SD card in the handheld terminal.
Audio/video data wireless transmission:
1. the transmission mode is as follows:
WiFi transmission: using WiFi transmission requires that a WiFi network be established between the audio-visual data source (audio-visual recorder) and the handheld terminal. The audio and video data source can be a network camera and the like, and the handheld terminal can be a smart phone, a tablet computer and the like. After the WiFi network is established, the audio-video data may be transmitted from the audio-video recorder to the handheld terminal through a WiFi transmission protocol.
Bluetooth transmission: the use of bluetooth transmissions requires the establishment of a bluetooth connection between the detection device and the handheld terminal. The detection equipment is a multifunctional data acquisition instrument. The handheld terminal can be a smart phone, a tablet computer and other devices. After the bluetooth connection is established, the detection device acquisition data may be transmitted from the source device to the handheld terminal via bluetooth transmission BLE broadcast.
2. Transmission protocol:
the audio-video recorder transmits and selects the RTSP protocol for transmission. RTSP is a real-time streaming protocol, which can realize real-time transmission of audio-video data.
The detection equipment data format adopts a JSON format, and comprises equipment ID, sensor data, acquisition time stamp and other parameters. The Bluetooth BLE (Bluetooth Low energy) protocol is used for communication, a handheld terminal (including but not limited to a smart phone, a tablet computer and the like) is used as a central device, and a sensor is used as a peripheral. After the detection equipment collects data, a data packet is sent in a BLE broadcast mode, and the handheld terminal subscribes to the BLE broadcast, so that the detection data is received.
Synthesizing and storing audio and video data:
1. watermarking algorithm: processing the time information and the sensor data by utilizing a digital signal processing technology to form watermark information and encrypting and embedding the watermark information and the sensor data into video pixels of video data;
2. data preprocessing: processing the sensor data by using a filtering technology (the sensor data can be more accurate and reliable) and synchronizing the sensor data with the video data (realized by adopting a time stamp algorithm), so that watermark information is synchronized with the video data;
3. data presentation: after the watermark is added, the video data added with the watermark is encoded and compressed; wherein the watermark is visualized by an image processing library (Opencv); i.e. after the watermarking is completed, a rendering process is required to enable normal display in the new video file. The rendering process requires encoding and compression of the watermarked video, enabling it to reduce the size of the video file without affecting the video quality. The rendering process may also perform a visualization process on the watermark through an image processing library (Opencv);
4. Routine optimization: conventional optimization is also required in the implementation of the overall scheme. Such as optimizing encryption algorithms, sensor data preprocessing algorithms, synchronization algorithms, optimizing presentation processing algorithms, and the like.
5. And (3) data preservation: the video stream file is saved into a local audio-video folder for later offline viewing using a storage API in the handheld terminal (e.g., sharedreferences, etc.).
Audio-visual data export:
1. a wired mode:
1.1, connecting a handheld terminal to a computer through a USB; if the handheld terminal supports the USB data storage mode, directly accessing files stored in the handheld terminal or the SD card; if the handheld terminal does not start the USB data storage mode, the function needs to be started in the setting of the handheld terminal;
1.2, opening a file manager on a computer, finding an audio/video folder of a video stream file in a storage or SD card of a handheld terminal, selecting the video stream file to be exported, and copying the video stream file to a local disk or other storage equipment of the computer;
and 1.3, safely pulling out the handheld terminal after the file is exported.
2. Wireless mode:
2.1, an application program supporting Bluetooth file transmission, such as a common file manager application or a special Bluetooth file transmission application, can be installed on the handheld terminal;
2.2, starting a Bluetooth function on the handheld terminal, and pairing the handheld terminal with a computer to be transmitted or other equipment supporting Bluetooth transmission;
2.3, opening an application program of Bluetooth file transmission, selecting a video stream file to be exported, and adding the video stream file to a transmission list; ensuring that a connection between the handheld terminal and a computer or other Bluetooth device is established;
2.4, clicking the video stream file in the transmission list to start video stream file transmission; the transmission status and progress can be viewed through a transmission progress bar in the application.
2.5, a computer or other Bluetooth enabled device receives the transmitted video stream file and stores it in a device memory.
It should be noted that bluetooth transmission speed is slow, and the transmission distance is generally not more than 10 meters. Meanwhile, bluetooth transmission requires pairing between devices, and certain operation steps and time are required. For large file transfers, the time will be longer, so bluetooth transfers are not suitable for large and infrequent data transfers.
Audio and video data record retrieval:
1. and (5) text retrieval: the text is processed mainly by word segmentation, indexing and query establishment and the like. Word segmentation is to split a word into a plurality of meaningful words, and how to split the words needs to be considered when a word segmentation algorithm is selected. The subsequent index establishment and query require the use of full text indexing techniques to establish an inverted index relationship for the words in the text, thereby achieving efficient text query. The elastiscearch is a distributed search engine developed based on the Lucene search engine library, and can realize higher performance and scalability.
2. And (5) voice control search:
2.1, converting voice into words: voice recognition technology is used to convert the sound recording into text, forming text data for subsequent processing. At present, there are several APIs for providing speech recognition services, such as hundred-degree speech recognition, ali cloud speech recognition, etc. on the market;
2.2, word processing: for the converted text data, a natural language processing technology can be used for analysis, and related information such as time, detection items, detection values, detection places and the like in the text data can be extracted; natural language processing techniques including lexical analysis, grammatical analysis, semantic analysis, etc., may be implemented with the aid of related tools and libraries;
2.3, data storage and retrieval: and the extracted related information is stored in a database, so that subsequent inquiry and retrieval are facilitated. For example, a relational database such as MySQL may be used. When inquiring, the SQL language can be used for inquiring records in a specified time range, or inquiring is carried out according to conditions such as detection items or detection places;
2.4, application development: finally, an application program needs to be developed to provide the functions of voice recording and related record retrieval; during recording, a user can input related information through voice, and the handheld terminal automatically recognizes and stores the related information into a database; during retrieval, the user can input the query condition through voice, and the handheld terminal automatically retrieves related records from the database and returns the related records to the user.
It should be noted that, at present, a certain error recognition and error rate still exist in the voice recognition technology and the natural language processing technology, so that proper debugging and improvement are required, and the accuracy and the use experience of the system are improved.
The foregoing description is only a preferred embodiment of the present invention, and is not intended to limit the present invention in any way, and any simple modification, equivalent variation, etc. of the above embodiment according to the technical matter of the present invention are within the scope of the present invention.
Claims (8)
1. A method for synchronously linking audio-video recording and detecting equipment data and retrieving by sound control is characterized in that: comprises the following specific steps:
1) And (3) audio and video data acquisition: the method for acquiring and processing the required data by using the audio-video recorder and the detection equipment comprises the following specific steps:
1.1 Signal input: inputting an audio signal into an audio-video recorder through recording equipment; the video signal is input into an audio-video recorder through a camera or screen capturing software; collecting various sensor signals by using detection equipment;
1.2 Signal conversion: converting the audio signal, the video signal and the sensor signal into an audio data digital signal, a video data digital signal and a sensor data digital signal through an analog-to-digital converter;
1.3 After step 1.2), storing all the digital signals in a storage device of an audio-video recorder, and editing, compressing and archiving the digital signals;
2) And (3) audio-video data wireless transmission: after the step 1), the processed data is transmitted to the handheld terminal in a wireless communication mode;
3) Synthesizing and storing audio and video data: after the step 2), the data received by the handheld terminal is subjected to watermarking, data preprocessing, presentation processing, conventional optimization and storage, and the method comprises the following specific steps:
3.1 Watermarking process:
3.1.1 Reading a video file: using a video processing library to read video data in the handheld terminal and acquiring information of the frame number, the frame rate and the resolution of the video;
3.1.2 Watermark picture generation: after step 3.1.1), converting the information into a gray scale image or a color image using an image processing library;
3.1.3 Reading the watermark picture: using an image processing library to read the watermark picture and obtaining the size and transparency information of the watermark picture;
3.1.4 Converting the watermark picture into a transparency channel: converting the watermark picture into RGBA format, and storing transparency information of the watermark picture in a transparency channel;
3.1.5 Scaling the watermark picture to a suitable size: according to the resolution of the video and the size of the watermark picture, calculating the scaling required by the watermark picture, and then scaling the watermark picture to be matched with the size of the video picture by using an image processing library;
3.1.6 Superimposing the watermark picture onto the video frame: for each frame of video, an image processing library is used for superposing watermark pictures on the video frames, and a transparency channel is used for controlling the transparency of the watermark;
3.1.7 Outputting a video file: writing the processed video frames into a new video file to generate a video with a watermark;
3.2 Data preprocessing: processing the sensor data by utilizing a filtering technology and synchronizing the sensor data with the video data so as to synchronize watermark information with the video data;
3.3 After the watermark is added, encoding and compressing the video data added with the watermark; the watermark is subjected to visual processing through an image processing library;
3.4 After the step 3.3), the video data is subjected to conventional optimization by using an optimized encryption algorithm, a sensor data preprocessing algorithm and a synchronization algorithm to form a video stream file;
3.5 Storing the video stream file into a local audio-video folder by using a storage API in the handheld terminal;
4) The data is exported from the handheld terminal in a wired or wireless mode;
5) And completing record retrieval by adopting a text retrieval mode or a voice control retrieval mode.
2. The method for synchronously linking audio and video recording and detecting equipment data and retrieving sound according to claim 1, wherein the method comprises the following steps: the sensor comprises one or more of a gyroscope, an acceleration sensor, a displacement sensor, noise, temperature and humidity, laser ranging, angle, infrared thermal imaging, current and voltage, rotating speed, ultrasonic waves, magnetic leakage, vortex, acoustic emission, magnetic powder, strong magnetic, weak magnetic, ray, gas, spectrum, stress and coulometer sensors, and the sensor and the inspection equipment are transmitted through standard IIC, serial ports, bluetooth and WIFI.
3. The method for synchronously linking audio and video recording and detecting equipment data and retrieving sound according to claim 1, wherein the method comprises the following steps: the step 2) includes two ways:
mode one: establishing WiFi communication between the audio-video recorder and the handheld terminal, and transmitting the audio and video data finally processed by the audio-video recorder into the handheld terminal based on RTSP through a WiFi transmission mode;
mode two: and establishing Bluetooth communication between the detection equipment and the handheld terminal, and transmitting the sensor data finally processed by the detection equipment into the handheld terminal based on a BLE protocol through a Bluetooth transmission mode.
4. The method for synchronously linking audio and video recording and detecting equipment data and retrieving sound according to claim 1, wherein the method comprises the following steps: said step 4) comprises the steps of:
4.1 Using USB to import the local video stream file stored in the handheld terminal into the computer or other storage devices and exit;
4.2 Using bluetooth or WiFi to save the video stream file stored locally in the handheld terminal in the local device memory.
5. The method for synchronously linking audio and video recording and detecting equipment data and retrieving sound according to claim 4, wherein: said step 4.1) comprises the steps of:
4.1.1 Connected to the handheld terminal): connecting the handheld terminal to a computer through a USB; if the handheld terminal supports the USB data storage mode, directly accessing files stored in the handheld terminal or the SD card; if the handheld terminal does not start the USB data storage mode, the function needs to be started in the setting of the handheld terminal;
4.1.2 Export file): opening a file manager on a computer, finding an audio/video folder of a video stream file stored in a handheld terminal or an SD card, selecting the video stream file to be exported, and copying the video stream file to a local disk or other storage devices of the computer;
4.1.3 After the file is exported, the handheld terminal is safely pulled out.
6. The method for synchronously linking audio and video recording and detecting equipment data and retrieving sound according to claim 4, wherein: said step 4.2) comprises the steps of:
4.2.1 Installing an application program supporting Bluetooth or WiFi file transmission on the handheld terminal;
4.2.2 Starting Bluetooth or WiFi functions on the handheld terminal and pairing with a computer to be transmitted or other devices supporting Bluetooth transmission;
4.2.3 Opening an application program of Bluetooth or WiFi file transmission, selecting a video stream file to be exported, and adding the video stream file to a transmission list;
4.2.4 Clicking the video stream file in the transmission list to start video stream file transmission;
4.2.5 A computer or other device supporting bluetooth or WiFi transmission receives the transmitted video stream file and stores it in device memory.
7. The method for synchronously linking audio and video recording and detecting equipment data and retrieving sound according to claim 1, wherein the method comprises the following steps: the method for recording and searching by adopting the text searching concretely comprises the following steps: splitting a segment of speech into a plurality of meaningful words based on a word segmentation algorithm to form segmented words; and then establishing an inverted index relation for the segmented words in the text through a full text index technology.
8. The method for synchronously linking audio and video recording and detecting equipment data and retrieving sound according to claim 1, wherein the method comprises the following steps: the method for completing record retrieval by adopting the voice control retrieval mode comprises the following specific steps:
5.2.1 Voice recognition technology is used for converting the record into characters to form text data;
5.2.2 Analyzing the converted text data by using a natural language processing technology, and extracting related information including time, detection items, detection values and detection places;
5.2.3 Storing the extracted related information in a database;
5.2.4 Developing an application program to provide the functions of voice recording and retrieving related records; during recording, a user inputs related information through voice, and the handheld terminal automatically recognizes and stores the related information in a database; during retrieval, the user inputs the query condition through voice, and the handheld terminal automatically retrieves related records from the database and returns the related records to the user.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310793851.5A CN116506673B (en) | 2023-06-30 | 2023-06-30 | Audio-video recording and detecting equipment data synchronous linkage and sound control searching method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310793851.5A CN116506673B (en) | 2023-06-30 | 2023-06-30 | Audio-video recording and detecting equipment data synchronous linkage and sound control searching method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116506673A CN116506673A (en) | 2023-07-28 |
| CN116506673B true CN116506673B (en) | 2023-09-12 |
Family
ID=87318832
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310793851.5A Active CN116506673B (en) | 2023-06-30 | 2023-06-30 | Audio-video recording and detecting equipment data synchronous linkage and sound control searching method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116506673B (en) |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20110080712A (en) * | 2010-01-07 | 2011-07-13 | 주식회사 엘지유플러스 | Method and system for searching moving picture by voice recognition of mobile communication terminal and apparatus for converting text of voice in moving picture |
| CN203722704U (en) * | 2014-02-17 | 2014-07-16 | 北京尚易德科技有限公司 | Intelligentized monitoring camera capable of discriminating geographic positions and monitoring directions |
| CN105721814A (en) * | 2014-12-19 | 2016-06-29 | 意美森公司 | Systems and methods for recording haptic data for use with multi-media data |
| CN106384108A (en) * | 2016-08-31 | 2017-02-08 | 上海斐讯数据通信技术有限公司 | Text content retrieval method, word interpreting device and mobile terminal |
| CN108712624A (en) * | 2018-08-08 | 2018-10-26 | 上海启诺信息科技有限公司 | Video recording archive devices based on writing record and method |
| US10311918B1 (en) * | 2016-04-19 | 2019-06-04 | Space Projects Ltd. | System, media, and method for synchronization of independent sensors and recording devices |
| CN111432229A (en) * | 2020-03-31 | 2020-07-17 | 卡斯柯信号有限公司 | Method and device for recording, analyzing and live broadcasting driving command |
| CN112131449A (en) * | 2020-09-21 | 2020-12-25 | 西北大学 | Implementation method of cultural resource cascade query interface based on elastic search |
| CN114374813A (en) * | 2021-12-13 | 2022-04-19 | 青岛海信移动通信技术股份有限公司 | Multimedia resource management method, recorder and server |
| CN115314734A (en) * | 2022-08-01 | 2022-11-08 | 定军山数媒技术(深圳)有限公司 | Video watermark processing method and device, electronic equipment and storage medium |
| CN115962719A (en) * | 2023-03-16 | 2023-04-14 | 安格利(成都)仪器设备有限公司 | Measuring instrument |
| CN116233078A (en) * | 2023-01-17 | 2023-06-06 | 山东锐智科电检测仪器有限公司 | Real-time data transmission device and method for nondestructive testing |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10410441B2 (en) * | 2016-05-16 | 2019-09-10 | Wi-Tronix, Llc | Real-time data acquisition and recording system viewer |
-
2023
- 2023-06-30 CN CN202310793851.5A patent/CN116506673B/en active Active
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20110080712A (en) * | 2010-01-07 | 2011-07-13 | 주식회사 엘지유플러스 | Method and system for searching moving picture by voice recognition of mobile communication terminal and apparatus for converting text of voice in moving picture |
| CN203722704U (en) * | 2014-02-17 | 2014-07-16 | 北京尚易德科技有限公司 | Intelligentized monitoring camera capable of discriminating geographic positions and monitoring directions |
| CN105721814A (en) * | 2014-12-19 | 2016-06-29 | 意美森公司 | Systems and methods for recording haptic data for use with multi-media data |
| US10311918B1 (en) * | 2016-04-19 | 2019-06-04 | Space Projects Ltd. | System, media, and method for synchronization of independent sensors and recording devices |
| CN106384108A (en) * | 2016-08-31 | 2017-02-08 | 上海斐讯数据通信技术有限公司 | Text content retrieval method, word interpreting device and mobile terminal |
| CN108712624A (en) * | 2018-08-08 | 2018-10-26 | 上海启诺信息科技有限公司 | Video recording archive devices based on writing record and method |
| CN111432229A (en) * | 2020-03-31 | 2020-07-17 | 卡斯柯信号有限公司 | Method and device for recording, analyzing and live broadcasting driving command |
| CN112131449A (en) * | 2020-09-21 | 2020-12-25 | 西北大学 | Implementation method of cultural resource cascade query interface based on elastic search |
| CN114374813A (en) * | 2021-12-13 | 2022-04-19 | 青岛海信移动通信技术股份有限公司 | Multimedia resource management method, recorder and server |
| CN115314734A (en) * | 2022-08-01 | 2022-11-08 | 定军山数媒技术(深圳)有限公司 | Video watermark processing method and device, electronic equipment and storage medium |
| CN116233078A (en) * | 2023-01-17 | 2023-06-06 | 山东锐智科电检测仪器有限公司 | Real-time data transmission device and method for nondestructive testing |
| CN115962719A (en) * | 2023-03-16 | 2023-04-14 | 安格利(成都)仪器设备有限公司 | Measuring instrument |
Non-Patent Citations (1)
| Title |
|---|
| 基于S3C44B0x的数字视音频网络监控终端设计;李建坡等;吉林化工学院学报(第01期);第62-65页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116506673A (en) | 2023-07-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10559324B2 (en) | Media identifier generation for camera-captured media | |
| US9779775B2 (en) | Automatic generation of compilation videos from an original video based on metadata associated with the original video | |
| US9330431B2 (en) | System and method for synchronizing, merging, and utilizing multiple data sets for augmented reality application | |
| US8688672B2 (en) | Search apparatus and method, and program | |
| US20160099023A1 (en) | Automatic generation of compilation videos | |
| JP5359177B2 (en) | System, method, and program for linking media | |
| KR101508372B1 (en) | Content management apparatus, content management method, network system, and computer readable recording medium having computer program recorded thereon | |
| EP1876596A2 (en) | Recording and reproducing data | |
| US20130346631A1 (en) | Time-synchronizing a parallel feed of secondary content with primary media content | |
| WO2004002144A1 (en) | Metadata preparing device, preparing method therefor and retrieving device | |
| KR102228457B1 (en) | Methed and system for synchronizing usage information between device and server | |
| JP2019057891A (en) | Information processing apparatus, imaging apparatus, information processing method, and program | |
| WO2014043969A1 (en) | Information transmission method and device | |
| CN100518241C (en) | Method for obtaining two or more video synchronization frame | |
| US20160112656A1 (en) | Thermal imaging device, analyzing device, thermal image photographing method, and analyzing method | |
| CN116506673B (en) | Audio-video recording and detecting equipment data synchronous linkage and sound control searching method | |
| KR20200024541A (en) | Providing Method of video contents searching and service device thereof | |
| US11157549B2 (en) | Emotional experience metadata on recorded images | |
| JP2005295030A (en) | Map information display system | |
| CN116993289A (en) | System and method for managing interrogation record | |
| JP6179315B2 (en) | Information processing apparatus, image processing system, image processing method and program in information processing apparatus | |
| US9612519B2 (en) | Method and system for organising image recordings and sound recordings | |
| WO2014190920A1 (en) | Thermal imagery dynamic recording device, thermal imagery dynamic playback device, and thermal imagery dynamic recording and playback method | |
| WO2014190921A1 (en) | Image dynamic recording device, image playback device, and image dynamic recording and playback method | |
| WO2019000877A1 (en) | Audio data processing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |