CN107404599B - Audio and video data synchronization method, device and system - Google Patents
Audio and video data synchronization method, device and system Download PDFInfo
- Publication number
- CN107404599B CN107404599B CN201710580821.0A CN201710580821A CN107404599B CN 107404599 B CN107404599 B CN 107404599B CN 201710580821 A CN201710580821 A CN 201710580821A CN 107404599 B CN107404599 B CN 107404599B
- Authority
- CN
- China
- Prior art keywords
- audio
- video data
- target object
- distance
- audio data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/04—Synchronising
- H04N5/06—Generation of synchronising signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/76—Television signal recording
- H04N5/91—Television signal processing therefor
- H04N5/92—Transformation of the television signal for recording, e.g. modulation, frequency changing; Inverse transformation for playback
- H04N5/9201—Transformation of the television signal for recording, e.g. modulation, frequency changing; Inverse transformation for playback involving the multiplexing of an additional signal and the video signal
- H04N5/9202—Transformation of the television signal for recording, e.g. modulation, frequency changing; Inverse transformation for playback involving the multiplexing of an additional signal and the video signal the additional signal being a sound signal
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Measurement Of Velocity Or Position Using Acoustic Or Ultrasonic Waves (AREA)
- Television Signal Processing For Recording (AREA)
Abstract
The application discloses an audio and video data synchronization method, device and system, wherein the method comprises the steps of obtaining audio and video data; obtaining a distance; calculating the transmission time; updating a time stamp of the audio data; and synchronizing the audio and video data according to the updated time stamp. The device comprises an audio acquisition module and a video acquisition module; a distance acquisition module; a time calculation module; a time update module; and a data synchronization module. The system comprises a camera device for collecting video data; an audio device to collect audio data; a distance sensor on the audio device for collecting distance; and the control equipment is respectively connected with the camera equipment, the audio equipment and the distance sensor and is used for receiving data, calculating time, updating a time stamp and synchronizing data. The embodiment of the invention improves the synchronization accuracy of the audio data and the video data.
Description
Technical Field
The application belongs to the field of intelligent information processing, and particularly relates to an audio and video data synchronization method, device and system.
Background
The unmanned aerial vehicle is an unmanned aerial vehicle operated by using a wireless remote sensing device and a self-contained control program control device. In recent years, with the rapid development of the unmanned aerial vehicle miniaturization technology, the unmanned aerial vehicle technology is widely applied to the field of video recording.
At present, when an unmanned aerial vehicle is used for video recording, video data and audio data are separately collected. And then, synchronizing the audio data and the video data according to the time stamp of the audio data and the time stamp of the video data.
However, according to the audio and video data synchronization method in the prior art, that is, the recording and the video recording of the unmanned aerial vehicle are processed separately, when the audio and video data is played, the situation of audio data delay often occurs, and therefore, the audio and video data synchronization method in the prior art is not accurate.
Disclosure of Invention
In view of this, the present application provides an audio and video data synchronization method, which is mainly used to solve the technical problem in the prior art that when audio data and video data are synchronized according to corresponding timestamps, synchronization is inaccurate.
In order to solve the above technical problem, a first aspect of the present application provides an audio and video data synchronization method, including:
acquiring audio data acquired by audio equipment aiming at a target object and video data acquired by camera equipment aiming at the target object;
acquiring the distance between the audio equipment and the target object acquired by a distance sensor; wherein the distance sensor is located on the audio device;
calculating a transmission time of a sound output by the target object to the audio device based on the distance and the sound speed;
updating a timestamp of the audio data based on the transmission time;
and synchronizing the audio data and the video data according to the time stamp of the audio data after updating and the time stamp of the video data.
Preferably, the calculating of the transmission time of the sound output by the target object to the audio apparatus based on the distance and the sound speed includes: acquiring environmental factors influencing the sound velocity;
calculating to obtain the current actual sound velocity according to the environmental factor value;
calculating a transmission time of the sound output by the target object to the audio device based on the distance and the current actual sound speed.
Preferably, the updating the time stamp of the audio data based on the transmission time includes:
and subtracting the transmission time from the time stamp of the audio data to obtain the time stamp after the audio data is updated.
Preferably, the environmental factors include: temperature, and/or wind speed and wind direction, and acquiring environmental factors influencing the sound velocity, including:
acquiring the temperature acquired by a temperature sensor and/or the wind speed and the wind direction acquired by a wind direction anemometer;
the calculating to obtain the current sound speed according to the environmental factor value comprises:
and calculating to obtain the current sound velocity according to the temperature, and/or the wind speed and the wind direction.
A second aspect of the present application provides an audio-video data synchronization apparatus, the apparatus including:
the audio acquisition module is used for acquiring audio data acquired by the audio equipment aiming at the target object;
the video acquisition module is used for acquiring video data acquired by the camera equipment aiming at the target object;
the distance acquisition module is used for acquiring the distance between the audio equipment and the target object acquired by the distance sensor; wherein the distance sensor is located in the audio device;
a time calculation module for calculating a transmission time of the sound output by the target object to the audio device based on the distance and the sound speed;
a time updating module for updating the time stamp of the audio data based on the transmission time;
and the data synchronization module is used for synchronizing the audio data and the video data according to the time stamp after the audio data is updated and the time stamp of the video data.
Preferably, the time calculation module includes:
a factor acquiring unit for acquiring environmental factors affecting the sound velocity;
the sound velocity calculation unit is used for calculating and obtaining the current actual sound velocity according to the environment factor value;
a time calculation unit for calculating a transmission time of the sound output by the target object to the audio device based on the distance and the current actual sound speed.
Preferably, the time update module includes:
and the time updating unit is used for subtracting the transmission time from the time stamp of the audio data to obtain the time stamp after the audio data is updated.
Preferably, the environmental factors include: temperature, wind speed and/or wind direction, the factor acquisition unit comprising:
the acquiring subunit is used for acquiring the temperature acquired by the temperature sensor and/or the wind speed and the wind direction acquired by the anemoscope;
the sound speed calculation unit includes:
and the calculating subunit is used for calculating and obtaining the current actual sound velocity according to the temperature, and/or the wind speed and the wind direction.
A third aspect of the present invention provides an audio-visual data synchronization system, comprising: an image pickup apparatus; an audio device; a distance sensor located on the audio device; a control device connected to the image pickup device, the audio device, and the distance sensor, respectively;
the camera device is used for collecting video data aiming at a target object;
the audio device is used for collecting audio data aiming at a target object;
the distance sensor is used for acquiring the distance between the audio equipment and the target object;
the control equipment is used for acquiring the video data, the audio data and the distance; and calculating a transmission time of the sound output by the target object to the audio device based on the distance and the sound speed; updating a timestamp of the audio data based on the transmission time; and synchronizing the audio data and the video data according to the time stamp of the audio data after updating and the time stamp of the video data.
Preferably, the system further comprises a temperature sensor connected with the control device, and/or a wind vane anemometer;
the control equipment is also used for acquiring the temperature acquired by the temperature sensor and/or the wind speed and the wind direction acquired by the anemoscope; and calculating to obtain the current sound velocity according to the temperature and/or the wind speed and the wind direction.
In the embodiment of the invention, the aim of acquiring the picture and the sound aiming at the target object can be achieved by acquiring the video data and the audio data of the target object. By acquiring the distance and the sound velocity acquired by the distance sensor, the transmission time of the sound of the target object transmitted to the audio device can be calculated, the transmission time can represent the delay time of the audio data relative to the video data, and the time difference between the audio data and the video data can be eliminated through the delay time. Furthermore, the purpose of synchronizing the video data and the audio data can be realized according to the updated time stamp, and the effect that the sound and the picture are consistent in playing is achieved. The time difference between the audio data and the video data during acquisition is eliminated, and the accuracy of the synchronization mode is higher.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application. In the drawings:
fig. 1 is a flowchart of an embodiment of an audio and video data synchronization method according to an embodiment of the present application;
fig. 2 is a schematic diagram of an acquisition scenario of an audio and video data synchronization method according to an embodiment of the present application;
fig. 3 is a flowchart of another embodiment of an audio-video data synchronization method according to an embodiment of the present application;
fig. 4 is a schematic structural diagram of an embodiment of an audio-video data synchronization apparatus according to an embodiment of the present application;
fig. 5 is a schematic structural diagram of an embodiment of an audio-video data synchronization system according to an embodiment of the present application.
Detailed Description
Embodiments of the present application will be described in detail with reference to the drawings and examples, so that how to implement technical means to solve technical problems and achieve technical effects of the present application can be fully understood and implemented.
The embodiment of the invention is mainly applied to the field of audio and video recording, and mainly solves the problem that the time of video data and audio data acquired by different devices is not synchronous in the prior art.
In the prior art, video data and audio data can be collected for a target object. In general, video data may be collected using a camera device, and audio data may be collected using an audio device, which may be located at a different location from the target object. For example, the camera device may be located on a drone, and the audio device may be located at a position closer to the target object. After the video data and the audio data are collected, the video data and the audio data need to be synthesized into audio and video data so as to normally play the audio and video data with sound and pictures. When audio data and video data are synthesized, a timestamp carried by the audio data and a timestamp carried by the video data are generally corresponded, and the video data and the audio data corresponding to the timestamps are obtained. However, the inventor finds that when the audio and video are played after the video data and the audio data are synthesized into the audio and video according to the time stamps, the audio and the video are not synchronous with the picture, and the audio is delayed from the picture.
The inventor finds that the video is recorded in real time when the camera device records the video, and the acquired audio data has a certain time delay relative to the acquired video data, mainly because the transmission time is needed when the sound is transmitted from the target object to the audio device. The inventors have thus proposed the technical solutions of the present application.
In the embodiment of the invention, audio data collected by an audio device aiming at a target object and video data collected by a camera device aiming at the target object are obtained; and acquiring the distance between the audio equipment and the target object acquired by the distance sensor, and calculating the transmission time of the sound output by the target object transmitted to the audio equipment based on the distance and the sound velocity, wherein the transmission time is the delay time of the audio data relative to the video data. After the time stamp of the audio data is updated based on the transmission time, the time stamp of the audio data may correspond to the time stamp of the video data, and the audio data and the video data may be synchronized. The time delay caused by sound transmission between the audio data and the video data can be eliminated, and the synchronization accuracy of the video data and the audio data is higher.
Embodiments of the present invention will be described in detail below with reference to the accompanying drawings.
As shown in fig. 1, a flowchart of an embodiment of an audio and video data synchronization method provided in an embodiment of the present invention is mainly included in the following steps:
101: the method comprises the steps of obtaining audio data collected by audio equipment aiming at a target object and video data collected by camera equipment aiming at the target object.
The camera device may be a camera or a video camera, and is configured to capture an image of a target object to obtain video data. The audio device may be a recording pen, a microphone, or the like, and is configured to collect sound output by the target object to obtain audio data.
The camera device and the audio device are separated, and the camera device and the audio device can be positioned on different carriers or the same carrier.
In one practical application, the camera device may be located on a drone. The audio device may be located on the drone or at any position independent of the drone, and may be closer to the target object for better audio data acquisition.
As shown in fig. 2, for a scene of a target object 203 for capturing video data by a camera 201 and audio data by an audio device 202 in an embodiment of the present invention, the camera 201 is located on the drone 200, the audio device 202 is separated from the drone 200, and a distance between the audio device and the target object is a distance D.
Alternatively, the control device may acquire audio data acquired by the audio device for a target object and video data acquired by the camera device for the target object. The control device may be connected to the audio device and the camera device, respectively, to obtain audio data collected by the audio device and video data collected by the video device. The control device may be an electronic apparatus having functions of data storage, instruction transmission, data processing, and the like. When the camera device is located on the unmanned aerial vehicle, the control device may specifically be a processing device on the unmanned aerial vehicle, and of course, the control device may also be a processing device independent of the unmanned aerial vehicle.
102: and acquiring the distance between the audio equipment and the target object acquired by the distance sensor.
Wherein the distance sensor is located on the audio device.
The distance sensor may collect a distance between the audio device and the target object, and the collected distance may be transmitted to a control device.
Alternatively, the distance sensor may be a laser distance measuring sensor, an ultrasonic distance measuring sensor, or a dual-camera distance measuring device.
The laser distance measuring sensor has the advantages that the light speed is very high, so that the distance measured by the laser distance measuring sensor is wide in range and high in precision.
The ultrasonic distance measuring sensor can adopt ultrasonic sensors with small angle, small blind area and the like, and has the advantages of accurate measurement, no contact, low cost and the like.
The double-camera range finder measures and obtains the distance between the audio equipment and the target object more accurately by using the coordinate conversion principle between the pixel points collected by the cameras.
103: based on the distance and the speed of sound, a transmission time for the sound output by the target object to be transmitted to the audio apparatus is calculated.
When the sound of the target object is transmitted, a certain transmission time is required to reach the audio device. After the distance between the target object and the audio device is acquired, the transmission time of the sound output by the target object to the audio device can be calculated based on the distance and the sound velocity.
Alternatively, for convenience of calculation and simplification of the calculation step, the sound speed may be an estimated value, which may be a constant, by which a transmission time of the sound output by the target object to the audio apparatus is calculated. For example, the estimated value may be a sound velocity under a condition of 1 standard atmosphere and 15 degrees celsius, and specifically, the estimated value is 340m/s (meters/second).
Meanwhile, in order to obtain a more accurate transit time, the sound speed may be an accurate value by which the transit time of the sound output by the target object to the audio apparatus may be calculated. The precise value may be determined by the speed of transmission of sound in the environment in which the audio device is located.
The distance may be represented by D, and the sound velocity may be represented by C, and the transit time dt, i.e., the calculated transit time, is calculated in an estimation manner as D/C.
104: updating a timestamp of the audio data based on the transmission time.
The transmission time of the sound emitted by the target object during transmission is the delay time of the audio data relative to the video data. The transmission time of the audio data may be updated based on the time stamp and the transmission time of the audio data.
Optionally, the updating the time stamp of the audio data based on the transmission time may include:
and subtracting the transmission time from the time stamp of the audio data to obtain the time stamp after the audio data is updated.
Based on the transmission time, the time stamp of the audio data is updated, so that the time stamp of the audio data and the time stamp of the video data can be synchronized, the time delay of the audio data relative to the video data is eliminated, and the effect of eliminating the time delay can be achieved.
105: and synchronizing the audio data and the video data according to the time stamp of the audio data after updating and the time stamp of the video data.
The video data and the audio data acquired by the control device are separated, and the video data and the audio data need to be coded or synthesized to form audio and video data which can normally play pictures and sounds. And synchronizing the time stamp of the video data and the time stamp of the updated audio data to obtain corresponding time-synchronized video data and audio data, and playing the audio and video data synchronously with sound and pictures.
In the embodiment of the invention, the transmission time from the sound output by the target object to the audio device can be calculated by acquiring the distance between the audio device and the target object and the sound velocity acquired by the distance sensor, wherein the transmission time is the delay time of the audio data relative to the video data, so that the audio data and the audio data can be synchronized according to the transmission time, further the synchronized video data and audio data can be acquired, and the synchronization accuracy is higher.
As shown in fig. 3, a flowchart of another embodiment of the method for synchronizing audio and video data according to the embodiment of the present invention is provided, and the method mainly includes the following steps:
301: the method comprises the steps of obtaining audio data collected by audio equipment aiming at a target object and video data collected by camera equipment aiming at the target object.
302: acquiring the distance between the audio equipment and the target object acquired by a distance sensor; wherein the distance sensor is located on the audio device.
303: environmental factors affecting the speed of sound are obtained.
304: and calculating to obtain the current actual sound velocity according to the environmental factor value.
305: based on the distance and the current actual sound speed, calculating a transmission time for the sound output by the target object to be transmitted to the audio device.
The current actual speed of sound may refer to a speed of sound transmission in an actual environment in which the audio apparatus is located. The sound velocity transmission process is influenced by the environmental factors in the transmission process.
The influence of environmental factors on the sound transmission speed is considered, the more accurate sound transmission speed of the current target object can be obtained, and the synchronization accuracy of the video data and the video data can be further improved.
Optionally, the environmental factors may include: temperature, and/or wind speed and direction, and obtaining environmental factors that affect the speed of sound may include:
acquiring temperature acquired by a temperature sensor and/or wind speed and wind direction acquired by a wind direction anemometer;
the calculating to obtain the current actual sound speed according to the environmental factor value comprises:
and calculating to obtain the current actual sound velocity according to the temperature, the air pressure and/or the wind speed and the wind direction.
As a possible implementation manner, the environmental factor includes a temperature, and the higher the temperature is, the larger the current actual sound speed is.
Alternatively, the current actual sound speed may be calculated according to the following formula:
v1 SQRT (1+ T/273) formula 1
Wherein T is the temperature in celsius of the environment where the current audio is set, V1 is the current actual sound speed when the temperature is T celsius, the SQRT square root, specifically, the square root of (1+ T/273) is calculated, and 331 is the sound speed when the temperature is 0 celsius.
As another possible implementation manner, when the environmental factors include wind speed and wind direction, the current actual sound speed may be calculated as follows:
V2＝340±Vtequation 2
Wherein V2 is the current wind speed and the current actual wind speed under the wind direction, 340 is the estimated value of sound velocity, which is the sound velocity under the conditions of 1 standard atmosphere and 15 ℃, and VtRefers to the current wind speed.
When the wind direction is the same direction as the transmission direction of the target object for the audio device, it may be the estimated speed of sound plus the wind speed; the wind direction may be an estimated speed of sound minus the wind speed when the wind direction is opposite to the direction of transmission of the target object to the audio device.
As yet another possible implementation manner, when the environmental factors may include temperature, wind speed, and wind direction, the current actual sound speed may be calculated as follows:
V3＝331*SQRT(1+T/273)±Vtequation 3
Wherein, T is the temperature in centigrade of the environment where the current audio is set, V1 is the current actual sound velocity when the temperature is T centigrade, SQRT is to obtain the square root, concretely, the square root of (1+ T/273) is calculated, 331 is the sound velocity when the temperature is 0 centigrade, VtRefers to the current wind speed.
When the environmental factors can include temperature, wind speed and/or wind direction, namely the influence of the environmental influence factors on the sound speed is calculated, the accuracy of the current actual sound speed obtained through calculation is higher.
In addition, the environmental influence factor can also include atmospheric pressure, atmospheric pressure can be gathered through the ceremony meter and obtained, and under same temperature, in the same transmission medium, atmospheric pressure is bigger, and the sound velocity is bigger.
306: updating a timestamp of the audio data based on the transmission time.
307: and synchronizing the audio data and the video data according to the time stamp of the audio data after updating and the time stamp of the video data.
In the embodiment of the invention, when the transmission time of the sound output by the target object to the audio equipment is calculated, the calculation is carried out based on the distance and the current actual sound velocity, the current actual sound velocity is the sound transmission velocity of the actual environment where the audio equipment is located, and the current actual sound velocity closer to the actual sound transmission velocity is obtained under the influence of environmental factors such as temperature and/or wind speed, the transmission time obtained by calculation is more accurate, the video data and the audio data can be synchronized in a more accurate mode, the time delay between the video data and the audio data is eliminated, and the sound and the picture are synchronized when the audio and video data are played.
As shown in fig. 4, a schematic structural diagram of an embodiment of an audio and video data synchronization apparatus provided in an embodiment of the present invention is provided, where the apparatus may include the following modules:
the audio acquisition module 401: the method is used for acquiring audio data collected by an audio device aiming at a target object.
The video acquisition module 402: the method is used for acquiring the video data acquired by the camera equipment aiming at the target object.
The camera device may be a camera or a video camera, and is configured to capture an image of a target object to obtain video data. The audio device may be a recording pen, a microphone, or the like, and is configured to collect sound output by the target object to obtain audio data.
The camera device and the audio device are separated, and the camera device and the audio device can be positioned on different carriers or the same carrier.
In one practical application, the camera device may be located on a drone. The audio device may be located on the drone or at any position independent of the drone, and may be closer to the target object for better audio data acquisition.
Alternatively, the control device may acquire audio data acquired by the audio device for a target object and video data acquired by the camera device for the target object. The control device may be connected to the audio device and the camera device, respectively, to obtain audio data collected by the audio device and video data collected by the video device. The control device may be an electronic apparatus having functions of data storage, instruction transmission, data processing, and the like. When the camera device is located on the unmanned aerial vehicle, the control device may specifically be a processing device on the unmanned aerial vehicle, and of course, the control device may also be a processing device independent of the unmanned aerial vehicle.
The distance acquisition module 403: the distance sensor is used for acquiring the distance between the audio equipment and the target object acquired by the distance sensor.
Wherein the distance sensor is located on the audio device.
The distance sensor may collect a distance between the audio device and the target object, and the collected distance may be transmitted to a control device.
Alternatively, the distance sensor may be a laser distance measuring sensor, an ultrasonic distance measuring sensor, or a dual-camera distance measuring device.
The laser distance measuring sensor has the advantages that the light speed is very high, so that the distance measured by the laser distance measuring sensor is wide in range and high in precision.
The ultrasonic distance measuring sensor can adopt ultrasonic sensors with small angle, small blind area and the like, and has the advantages of accurate measurement, no contact, low cost and the like.
The double-camera range finder measures and obtains the distance between the audio equipment and the target object more accurately by using the coordinate conversion principle between the pixel points collected by the cameras.
The time calculation module 404: and the sound velocity calculating unit is used for calculating the transmission time of the sound output by the target object to the audio equipment based on the distance and the sound velocity.
When the sound of the target object is transmitted, a certain transmission time is required to reach the audio device. After the distance between the target object and the audio device is acquired, the transmission time of the sound output by the target object to the audio device can be calculated based on the distance and the sound velocity.
Alternatively, for convenience of calculation and simplification of the calculation step, the sound speed may be an estimated value, which may be a constant, by which a transmission time of the sound output by the target object to the audio apparatus is calculated. For example, the estimated value may be a sound velocity under a condition of 1 standard atmosphere and 15 degrees celsius, and specifically, the estimated value is 340m/s (meters/second).
Meanwhile, in order to obtain a more accurate transit time, the sound speed may be an accurate value by which the transit time of the sound output by the target object to the audio apparatus may be calculated. The precise value may be determined by the speed of transmission of sound in the environment in which the audio device is located.
The distance may be represented by D, and the sound velocity may be represented by C, and the transit time dt, i.e., the calculated transit time, is calculated in an estimation manner as D/C.
The time update module 405: a time stamp for updating the audio data based on the transmission time.
The transmission time of the sound emitted by the target object during transmission is the delay time of the audio data relative to the video data. The transmission time of the audio data may be updated based on the time stamp and the transmission time of the audio data.
The time update module may include:
and the time updating unit is used for subtracting the transmission time from the time stamp of the audio data to obtain the time stamp after the audio data is updated.
Based on the transmission time, the time stamp of the audio data is updated, so that the time stamp of the audio data and the time stamp of the video data can be synchronized, the time delay of the audio data relative to the video data is eliminated, and the effect of eliminating the time delay can be achieved.
The data synchronization module 406: and the synchronization module is used for synchronizing the audio data and the video data according to the time stamp of the audio data and the time stamp of the video data after the update.
The video data and the audio data acquired by the control device are separated, and the video data and the audio data need to be coded or synthesized to form audio and video data which can normally play pictures and sounds. And synchronizing the time stamp of the video data and the time stamp of the updated audio data to obtain corresponding time-synchronized video data and audio data, and playing the audio and video data synchronously with sound and pictures.
In the embodiment of the invention, the transmission time from the sound output by the target object to the audio device can be calculated by acquiring the distance between the audio device and the target object and the sound velocity acquired by the distance sensor, wherein the transmission time is the delay time of the audio data relative to the video data, so that the audio data and the audio data can be synchronized according to the transmission time, further the synchronized video data and audio data can be acquired, and the synchronization accuracy is higher.
As an embodiment, the time calculation module may include:
a factor acquiring unit for acquiring environmental factors affecting the sound velocity;
and the sound velocity calculation unit is used for calculating and obtaining the current actual sound velocity according to the environment factor value.
And the time calculation unit is used for calculating the transmission time of the sound output by the target object to the audio equipment based on the distance and the current actual sound speed.
Optionally, the environmental factors may include: temperature, and/or wind speed and direction, the factor acquiring unit may include:
the acquiring subunit is used for acquiring the temperature acquired by the temperature sensor and/or the wind speed and the wind direction acquired by the anemoscope;
the sound speed calculation unit may include:
and the calculating subunit is used for calculating and obtaining the current actual sound velocity according to the temperature, and/or the wind speed and the wind direction.
The current actual speed of sound may refer to a speed of sound transmission in an actual environment in which the audio apparatus is located. The sound velocity transmission process is influenced by the environmental factors in the transmission process.
The influence of environmental factors on the sound transmission speed is considered, the more accurate sound transmission speed of the current target object can be obtained, and the synchronization accuracy of the video data and the video data can be further improved.
Optionally, the environmental factors may include: temperature, and/or wind speed and direction, and obtaining environmental factors that affect the speed of sound may include:
acquiring temperature acquired by a temperature sensor and/or wind speed and wind direction acquired by a wind direction anemometer;
the calculating to obtain the current actual sound speed according to the environmental factor value comprises:
and calculating to obtain the current actual sound velocity according to the temperature, the air pressure and/or the wind speed and the wind direction.
As a possible implementation manner, the environmental factor includes a temperature, and the higher the temperature is, the larger the current actual sound speed is.
Alternatively, the current actual sound speed may be calculated according to the following formula:
v1 SQRT (1+ T/273) formula 1
Wherein T is the temperature in celsius of the environment where the current audio is set, V1 is the current actual sound speed when the temperature is T celsius, the SQRT square root, specifically, the square root of (1+ T/273) is calculated, and 331 is the sound speed when the temperature is 0 celsius.
As another possible implementation manner, when the environmental factors include wind speed and wind direction, the current actual sound speed may be calculated as follows:
V2＝340±Vtequation 2
Wherein V2 is the current wind speed and the current actual wind speed under the wind direction, 340 is the estimated value of sound velocity, which is the sound velocity under the conditions of 1 standard atmosphere and 15 ℃, and VtRefers to the current wind speed.
When the wind direction is the same direction as the transmission direction of the target object for the audio device, it may be the estimated speed of sound plus the wind speed; the wind direction may be an estimated speed of sound minus the wind speed when the wind direction is opposite to the direction of transmission of the target object to the audio device.
As yet another possible implementation manner, when the environmental factors may include temperature, wind speed, and wind direction, the current actual sound speed may be calculated as follows:
V3＝331*SQRT(1+T/273)±Vtequation 3
Wherein T refers to the environment where the current audio is setV1 is the current actual sound velocity at a temperature of T degrees Celsius, SQRT is the square root calculation, specifically, the square root of (1+ T/273) is calculated, 331 is the sound velocity at a temperature of 0 degrees Celsius, VtRefers to the current wind speed.
When the environmental factors can include temperature, wind speed and/or wind direction, namely the influence of the environmental influence factors on the sound speed is calculated, the accuracy of the current actual sound speed obtained through calculation is higher.
In addition, the environmental influence factor can also include atmospheric pressure, atmospheric pressure can be gathered through the ceremony meter and obtained, and under same temperature, in the same transmission medium, atmospheric pressure is bigger, and the sound velocity is bigger.
In the embodiment of the invention, when the transmission time of the sound output by the target object to the audio equipment is calculated, the calculation is carried out based on the distance and the current actual sound velocity, the current actual sound velocity is the sound transmission velocity of the actual environment where the audio equipment is located, and the current actual sound velocity closer to the actual sound transmission velocity is obtained under the influence of environmental factors such as temperature and/or wind speed, the transmission time obtained by calculation is more accurate, the video data and the audio data can be synchronized in a more accurate mode, the time delay between the video data and the audio data is eliminated, and the sound and the picture are synchronized when the audio and video data are played.
As shown in fig. 5, a schematic structural diagram of an embodiment of an audio-video data synchronization system provided in an embodiment of the present invention is shown, where the audio-video data synchronization system may include:
an image pickup apparatus 501; an audio device 502; a distance sensor 503 located on the audio device 502; a control device 504 connected to the image pickup device 501, the audio device 502, and the distance sensor 503, respectively;
the camera device 501 is configured to collect video data for a target object;
the audio device 502 is used for collecting audio data for a target object;
the distance sensor 503 is used for acquiring the distance between the audio device and the target object;
the control device 504 is configured to obtain the video data, the audio data, and the distance; and calculating a transmission time of the sound output by the target object to the audio device based on the distance and the sound speed; updating a timestamp of the audio data based on the transmission time; and synchronizing the audio data and the video data according to the time stamp of the audio data after updating and the time stamp of the video data.
Optionally, the system may further comprise a temperature sensor and/or a wind vane anemometer connected to the control device;
the control equipment is also used for acquiring the temperature acquired by the temperature sensor and/or the wind speed and the wind direction acquired by the anemoscope; and calculating to obtain the current actual sound velocity according to the temperature and/or the wind speed and the wind direction.
When the control equipment contains the temperature sensor and the wind direction anemoscope, the specific numerical value of the environment factors influencing the sound transmission speed generated by sound transmission can be obtained accurately, the accurate sound velocity can be obtained through the environment influence factors, the accurate transmission time can be obtained, and the synchronization accuracy of the video data and the audio data is further improved.
In addition, the environmental influence factor can also include atmospheric pressure, atmospheric pressure can be gathered through the ceremony meter and obtained, and under same temperature, in the same transmission medium, atmospheric pressure is bigger, and the sound velocity is bigger.
In the embodiment of the invention, an audio and video data synchronization system is provided, wherein a control device included in the system can acquire video data acquired by a camera device, audio data acquired by an audio device, and a distance between a target object acquired by a distance sensor and the audio device, and can calculate transmission time of sound output by the target object and transmitted to the audio device by acquiring the variables, the distance and the like, wherein the transmission time is delay of the audio data relative to the video data, and the video data and the audio data can be synchronized through the transmission time, so that the synchronization accuracy can be improved.
In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.
Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include non-transitory computer readable media (transient media), such as modulated data signals and carrier waves.
As used in the specification and in the claims, certain terms are used to refer to particular components. As one skilled in the art will appreciate, manufacturers may refer to a component by different names. This specification and claims do not intend to distinguish between components that differ in name but not function. In the following description and in the claims, the terms "include" and "comprise" are used in an open-ended fashion, and thus should be interpreted to mean "include, but not limited to. "substantially" means within an acceptable error range, and a person skilled in the art can solve the technical problem within a certain error range to substantially achieve the technical effect. Furthermore, the term "coupled" is intended to encompass any direct or indirect electrical coupling. Thus, if a first device couples to a second device, that connection may be through a direct electrical coupling or through an indirect electrical coupling via other devices and couplings. The description which follows is a preferred embodiment of the present application, but is made for the purpose of illustrating the general principles of the application and not for the purpose of limiting the scope of the application. The protection scope of the present application shall be subject to the definitions of the appended claims.
It is also noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a good or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such good or system. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a commodity or system that includes the element
The foregoing description shows and describes several preferred embodiments of the present application, but as aforementioned, it is to be understood that the application is not limited to the forms disclosed herein, but is not to be construed as excluding other embodiments and is capable of use in various other combinations, modifications, and environments and is capable of changes within the scope of the application as described herein, commensurate with the above teachings, or the skill or knowledge of the relevant art. And that modifications and variations may be effected by those skilled in the art without departing from the spirit and scope of the application, which is to be protected by the claims appended hereto.
Claims (8)
1. An audio and video data synchronization method is characterized by comprising the following steps:
acquiring audio data acquired by audio equipment aiming at a target object and video data acquired by camera equipment positioned on an unmanned aerial vehicle aiming at the target object;
acquiring the distance between the audio equipment and the target object acquired by a distance sensor; wherein the distance sensor is located on the audio device;
calculating a transmission time of a sound output by the target object to the audio device based on the distance and the sound speed;
updating a timestamp of the audio data based on the transmission time;
synchronizing the audio data and the video data according to the time stamp of the audio data after being updated and the time stamp of the video data;
wherein the updating the time stamp of the audio data based on the transmission time comprises:
and subtracting the transmission time from the time stamp of the audio data to obtain the updated time stamp of the audio data.
2. The method of claim 1, wherein the calculating a transit time for the sound output by the target object to transit to the audio device based on the distance and the speed of sound comprises: acquiring environmental factors influencing the sound velocity;
calculating to obtain the current actual sound velocity according to the environmental factor value;
calculating a transmission time of the sound output by the target object to the audio device based on the distance and the current actual sound speed.
3. The method of claim 2, wherein the environmental factors comprise: temperature, and/or wind speed and wind direction, and acquiring environmental factors influencing the sound velocity, including:
acquiring the temperature acquired by a temperature sensor and/or the wind speed and the wind direction acquired by a wind direction anemometer;
the calculating to obtain the current sound speed according to the environmental factor value comprises:
and calculating to obtain the current sound velocity according to the temperature, and/or the wind speed and the wind direction.
4. An audio-video data synchronization apparatus, comprising:
the audio acquisition module is used for acquiring audio data acquired by the audio equipment aiming at the target object;
the video acquisition module is used for acquiring video data acquired by camera equipment on the unmanned aerial vehicle aiming at the target object;
the distance acquisition module is used for acquiring the distance between the audio equipment and the target object acquired by the distance sensor; wherein the distance sensor is located in the audio device;
a time calculation module for calculating a transmission time of the sound output by the target object to the audio device based on the distance and the sound speed;
a time updating module for updating the time stamp of the audio data based on the transmission time;
the data synchronization module is used for synchronizing the audio data and the video data according to the time stamp after the audio data is updated and the time stamp of the video data;
wherein the time update module comprises:
and the time updating unit is used for subtracting the transmission time from the time stamp of the audio data to obtain the time stamp after the audio data is updated.
5. The apparatus of claim 4, wherein the time calculation module comprises:
a factor acquiring unit for acquiring environmental factors affecting the sound velocity;
the sound velocity calculation unit is used for calculating and obtaining the current actual sound velocity according to the environment factor value;
a time calculation unit for calculating a transmission time of the sound output by the target object to the audio device based on the distance and the current actual sound speed.
6. The apparatus of claim 5, wherein the environmental factors comprise: temperature, wind speed and/or wind direction, the factor acquisition unit comprising:
the acquiring subunit is used for acquiring the temperature acquired by the temperature sensor and/or the wind speed and the wind direction acquired by the anemoscope;
the sound speed calculation unit includes:
and the calculating subunit is used for calculating and obtaining the current actual sound velocity according to the temperature, and/or the wind speed and the wind direction.
7. An audio-video data synchronization system, comprising: a camera device located on the unmanned aerial vehicle; an audio device; a distance sensor located on the audio device; a control device connected to the image pickup device, the audio device, and the distance sensor, respectively;
the camera device is used for collecting video data aiming at a target object;
the audio device is used for collecting audio data aiming at a target object;
the distance sensor is used for acquiring the distance between the audio equipment and the target object;
the control equipment is used for acquiring the video data, the audio data and the distance; and calculating a transmission time of the sound output by the target object to the audio device based on the distance and the sound speed; updating a timestamp of the audio data based on the transmission time; synchronizing the audio data and the video data according to the time stamp of the audio data after being updated and the time stamp of the video data;
wherein, based on the transmission time, the updating, by the control device, the timestamp of the audio data specifically is:
and subtracting the transmission time from the time stamp of the audio data to obtain the time stamp after the audio data is updated.
8. The system of claim 7, further comprising a temperature sensor, and/or a anemometer connected to the control device;
the control equipment is also used for acquiring the temperature acquired by the temperature sensor and/or the wind speed and the wind direction acquired by the anemoscope; and calculating to obtain the current sound velocity according to the temperature and/or the wind speed and the wind direction.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710580821.0A CN107404599B (en) | 2017-07-17 | 2017-07-17 | Audio and video data synchronization method, device and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710580821.0A CN107404599B (en) | 2017-07-17 | 2017-07-17 | Audio and video data synchronization method, device and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107404599A CN107404599A (en) | 2017-11-28 |
| CN107404599B true CN107404599B (en) | 2020-05-19 |
Family
ID=60401034
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710580821.0A Active CN107404599B (en) | 2017-07-17 | 2017-07-17 | Audio and video data synchronization method, device and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107404599B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP4195653A4 (en) * | 2020-08-26 | 2024-01-03 | Huawei Technologies Co., Ltd. | Video processing method and electronic device |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111447396A (en) * | 2020-03-06 | 2020-07-24 | 视联动力信息技术股份有限公司 | Audio and video transmission method and device, electronic equipment and storage medium |
| CN114584648A (en) * | 2020-11-30 | 2022-06-03 | 华为技术有限公司 | Method and equipment for synchronizing audio and video |
| CN114173168A (en) * | 2021-11-17 | 2022-03-11 | 中国船舶重工集团公司第七一九研究所 | Acoustic system and method for realizing time synchronization of physical isolation information system |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103414957A (en) * | 2013-07-30 | 2013-11-27 | 广东工业大学 | Method and device for synchronization of audio data and video data |
| CN104299632A (en) * | 2014-09-23 | 2015-01-21 | 广东欧珀移动通信有限公司 | Audio synchronization method and device and corresponding systems |
| CN104618786A (en) * | 2014-12-22 | 2015-05-13 | 深圳市腾讯计算机系统有限公司 | Audio/video synchronization method and device |
| CN105392024A (en) * | 2015-10-28 | 2016-03-09 | 中国电子科技集团公司第二十八研究所 | Multi-source heterogeneous data synchronous replaying method |
| CN105828100A (en) * | 2016-03-21 | 2016-08-03 | 乐视网信息技术（北京）股份有限公司 | Audio and video files simultaneous playing method, device and system |
| CN106291522A (en) * | 2016-07-27 | 2017-01-04 | 南京信息工程大学 | A kind of supersonic range finder and distance-finding method |
| CN106353731A (en) * | 2016-09-14 | 2017-01-25 | 刘珉恺 | Audio positioning device and audio positioning method |
-
2017
- 2017-07-17 CN CN201710580821.0A patent/CN107404599B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103414957A (en) * | 2013-07-30 | 2013-11-27 | 广东工业大学 | Method and device for synchronization of audio data and video data |
| CN104299632A (en) * | 2014-09-23 | 2015-01-21 | 广东欧珀移动通信有限公司 | Audio synchronization method and device and corresponding systems |
| CN104618786A (en) * | 2014-12-22 | 2015-05-13 | 深圳市腾讯计算机系统有限公司 | Audio/video synchronization method and device |
| CN105392024A (en) * | 2015-10-28 | 2016-03-09 | 中国电子科技集团公司第二十八研究所 | Multi-source heterogeneous data synchronous replaying method |
| CN105828100A (en) * | 2016-03-21 | 2016-08-03 | 乐视网信息技术（北京）股份有限公司 | Audio and video files simultaneous playing method, device and system |
| CN106291522A (en) * | 2016-07-27 | 2017-01-04 | 南京信息工程大学 | A kind of supersonic range finder and distance-finding method |
| CN106353731A (en) * | 2016-09-14 | 2017-01-25 | 刘珉恺 | Audio positioning device and audio positioning method |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP4195653A4 (en) * | 2020-08-26 | 2024-01-03 | Huawei Technologies Co., Ltd. | Video processing method and electronic device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107404599A (en) | 2017-11-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107404599B (en) | Audio and video data synchronization method, device and system | |
| US9892760B1 (en) | Apparatus and methods for embedding metadata into video stream | |
| US9001203B2 (en) | System and program for generating integrated database of imaged map | |
| KR20100108021A (en) | Apparatus and method for calibrating image between cameras | |
| AU2015275198B2 (en) | Methods and systems for calibrating sensors using recognized objects | |
| WO2003021187A3 (en) | Digital imaging system for airborne applications | |
| CN112383675A (en) | Time synchronization method and device and terminal equipment | |
| US10880474B1 (en) | Systems and methods for mobile imaging | |
| CN107560637B (en) | Method for verifying calibration result of head-mounted display device and head-mounted display device | |
| CN116704048B (en) | Double-light registration method | |
| TW201317544A (en) | Ground target geolocation system and method | |
| US10728453B2 (en) | Motion stabilized image sensor, camera module and apparatus comprising same | |
| CN110636202A (en) | Panoramic camera control method and device and storage medium | |
| US8457484B2 (en) | Apparatus, systems and methods for acquiring information from film set devices | |
| CN114625911A (en) | Automatic remote sensing image data annotation method based on AIS data retrieval | |
| CN203982457U (en) | Temperature causes the elimination system of image virtual deformation error | |
| CN115389246B (en) | Speed precision measuring method, system and device of motion capture system | |
| US10101560B2 (en) | Systems and methods for focusing on objects to capture images thereof | |
| KR20040068341A (en) | Description generation in the form of metadata | |
| CN104142795A (en) | Information acquisition starting method, information acquisition starting device and mobile terminal | |
| CN109033164A (en) | A kind of panoramic map data acquisition system and its moving gathering termination | |
| TWI767592B (en) | Statistics-based electronic image stabilization | |
| CN216565396U (en) | Multiband multi-angle synchronous measurement nacelle and system thereof | |
| CN112911217A (en) | Multiband multi-angle synchronous measurement nacelle and system thereof | |
| JP6447615B2 (en) | Image display apparatus and method, image generation apparatus and method, and program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |