CN106057207B - Remote stereo omnibearing real-time transmission and playing method - Google Patents
Remote stereo omnibearing real-time transmission and playing method Download PDFInfo
- Publication number
- CN106057207B CN106057207B CN201610494569.7A CN201610494569A CN106057207B CN 106057207 B CN106057207 B CN 106057207B CN 201610494569 A CN201610494569 A CN 201610494569A CN 106057207 B CN106057207 B CN 106057207B
- Authority
- CN
- China
- Prior art keywords
- transmission
- time
- user
- processing unit
- sound
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
- 230000005540 biological transmission Effects 0.000 title claims abstract description 86
- 238000000034 method Methods 0.000 title claims abstract description 35
- 238000012545 processing Methods 0.000 claims abstract description 55
- 238000001514 detection method Methods 0.000 claims abstract description 48
- 230000033001 locomotion Effects 0.000 claims abstract description 43
- 238000012360 testing method Methods 0.000 claims description 14
- 210000005069 ears Anatomy 0.000 claims description 9
- 230000001133 acceleration Effects 0.000 claims description 5
- 230000003190 augmentative effect Effects 0.000 claims description 4
- 238000004364 calculation method Methods 0.000 claims description 3
- 239000011521 glass Substances 0.000 claims description 3
- 230000009286 beneficial effect Effects 0.000 abstract description 3
- 238000007654 immersion Methods 0.000 description 11
- 238000010586 diagram Methods 0.000 description 5
- 238000004088 simulation Methods 0.000 description 5
- 238000004519 manufacturing process Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 238000013459 approach Methods 0.000 description 2
- 230000000903 blocking effect Effects 0.000 description 2
- 230000007547 defect Effects 0.000 description 2
- 239000007921 spray Substances 0.000 description 2
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 2
- 230000003139 buffering effect Effects 0.000 description 1
- 238000006073 displacement reaction Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 238000001125 extrusion Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000005855 radiation Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
- G10L19/008—Multichannel audio signal coding or decoding using interchannel correlation to reduce redundancy, e.g. joint-stereo, intensity-coding or matrixing
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/60—Network streaming of media packets
- H04L65/75—Media network packet handling
- H04L65/762—Media network packet handling at the source
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Health & Medical Sciences (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Human Computer Interaction (AREA)
- Acoustics & Sound (AREA)
- Stereophonic System (AREA)
Abstract
The invention provides a remote stereo omnibearing real-time transmission and playing method which comprises a server, a transmission system and a terminal, wherein the terminal comprises a processing unit, a motion detection unit and an acoustic unit, the processing unit is respectively and electrically connected with the motion detection unit and the acoustic unit, and the motion detection unit comprises a position detection device, an attitude detection device, a speed detection unit and an angular speed detection unit. Compared with the prior art, the invention adopts the scheme of correspondingly transmitting the sound data according to the detection result of the motion detection unit, saves a lot of network bandwidth and realizes the omnibearing real-time transmission and playing of the remote stereo. The method of dividing the cube enables the sound selection to be regionalized and quantized, which is beneficial to perfectly restoring the sound and saving the data transmission quantity.
Description
Technical Field
The invention relates to the field of stereo transmission, in particular to a remote stereo omnibearing real-time transmission and playing method.
Background
In existing virtual reality and augmented reality technologies, stereophonic playback is generally location independent. The required stereo sound is produced and stored in the relevant equipment in advance and is played out when being used, so that the method is simple and convenient, but the immersion feeling of the user is seriously influenced. In the existing partial stereo omnibearing playing technology, omnibearing stereo sound related to position and angle is made and stored in related equipment, and when the equipment is used, stereo sound playing corresponding to the position and angle is called. This approach works well in the field, creating a strong sense of immersion, but is not commonly used in telestereo omni-directional real-time transmission. This is because the set of sound of each position and each direction possess very big data bulk, and the data bulk of real-time transmission can seriously occupy the bandwidth, extrudes the space of image transmission, causes the image transmission to appear blocking, and then influences virtual reality experience and immersive.
Disclosure of Invention
In order to overcome the defects that the current remote stereo cannot realize all-around transmission and the immersion is influenced, the invention provides a remote stereo all-around real-time transmission and playing method which can realize all-around transmission and has strong immersion.
The technical scheme adopted by the invention for solving the technical problems is as follows: the method for providing the remote stereo omnibearing real-time transmission and playing comprises a server, a transmission system and a terminal, wherein the terminal comprises a processing unit, a motion detection unit and an acoustic unit, the processing unit is respectively and electrically connected with the motion detection unit and the acoustic unit, the motion detection unit comprises a position detection device and a posture detection device, and the method for the remote panoramic image real-time transmission and display comprises the following steps:
s1: the motion detection unit detects the motion state of a user and transmits the detection result to the processing unit;
s2: the server divides the space where the user is located into n cubes and transmits cube information to the processing unit, and the processing unit obtains a cube region where the user is located and cube regions which can possibly reach according to the coordinate information provided by the motion detection unit to form a transmission sound region;
s3: and the server transmits the data corresponding to the transmission sound area to the terminal.
Preferably, the method for calculating the transmission sound region is:
s2.11 the processing unit records the user coordinate information (X) provided by the motion detection unit0，Y0，Z0)；
S2.12 the processing unit calculates the maximum forward offset (Delta X) of the user coordinates1，ΔY1，ΔZ1) And reverse maximum offset (Δ X)2，ΔY2，ΔZ2) Coordinate set phi { (X) that may appear to the unscrambler0-ΔX2<X<X0+ΔX1)，(Y0-ΔY2<Y<Y0+ΔY1)，(Z0-ΔZ2<Z<Z0+ΔZ1)|X，Y，Z，}；
And the cubic region occupied by the point corresponding to the set phi of S2.13 is the transmission sound region.
Preferably, the method for calculating the transmission sound region is:
s2.21 the motion detection unit records the linear velocity v of the user 20 along each direction of the x-axis, the y-axis and the z-axisx、vy、vzRecording the delay time as t10The processing unit records the maximum acceleration a of the user 20 in each direction along the x-axis, y-axis and z-axisx、ay、az；
S2.22 calculating delay time by the processing unitTime t10Set of angular coordinates phi that may occur for an internal user1＝{(((x0-(vxt10+axt10 2/2))≤x≤(x0+(vxt10+axt10 2/2)))，(((y0-(vyt10+ayt10 2/2))≤y≤(y0+(vyt10+ayt10 2/2)))，(((z0-(vzt10+azt10 2/2))≤z≤(z0+(vzt10+azt10 2/2))) x, y, z }, and phi1The cubic area occupied by the corresponding point is the transmission sound area.
Preferably, the delay time is the time taken from the time point when the user detects the motion state to the time point when the server finishes transmitting the corresponding downlink data.
Preferably, the method of calculating the transmission sound region is:
s2.31 recording in the processing unit the maximum linear velocity v of the user in each direction of the x-axis, y-axis and z-axis during the delay timex’、vy’、vz’；
S2.32 processing to calculate the aggregate phi2＝{((x0-vx’t10)≤x≤(x0+vx’t10))，((y0-vy’t10)≤y≤(y0+vy’t10))，((z0-vz’t10)≤z≤(z0+vz’t10))|x，y，z}；
S2.33 order Phi3＝(Ф1∩Ф2) Phi is3The cubic area occupied by the corresponding point is the transmission sound area.
Preferably, the server is provided with m virtual sound sources in each individual cube, the virtual sound sources can simulate sound source sounding, the processing unit requests the server for data of the m virtual sound sources, integrates the sound from all the virtual sound sources and transmits the sound to the acoustic unit through CH1 and CH2 channels.
Preferably, the processing unit determines the position and the facing direction of the user according to the coordinate information of the user provided by the motion detection unit, and simulates the acoustic information transmitted by the virtual sound source 40 to the ears of the user.
Preferably, the terminal further comprises a response testing device, and the response testing device can test the response time, wherein the response time is the time taken for the terminal to send a signal to the server until the terminal receives a corresponding return signal.
Preferably, the server determines the number and the positions of the virtual sound sources according to the response time measured by the response testing device and the performance of the terminal.
Preferably, the terminal is a virtual reality helmet or augmented reality glasses.
Compared with the prior art, the invention adopts the scheme of correspondingly transmitting the sound data according to the detection result of the motion detection unit, saves a lot of network bandwidth and realizes the omnibearing real-time transmission and playing of the remote stereo. The method of dividing the cube enables the sound selection to be regionalized and quantized, which is beneficial to perfectly restoring the sound and saving the data transmission quantity. By means of intercepting the sound production area and transmitting the sound production area, data transmission amount is reduced, and delay is avoided. The range of the sound region can be transmitted more accurately by determining the maximum offset amount of the coordinates within the user delay time, further reducing the data transmission amount. The stereo sound is perfectly restored by the setting of the virtual sound source, and the immersion feeling is further increased. The setting of the response testing device can measure the network speed condition of a user, the tone quality can be improved to the maximum extent under the condition of ensuring the normal transmission of sound by selecting the number of virtual sound sources according to the network condition and the equipment performance, and meanwhile, the delay time t can be conveniently obtained10. The sound transmitted to the two ears of the user through the virtual sound source respectively increases the reality sense of stereo.
Drawings
The invention will be further described with reference to the accompanying drawings and examples, in which:
FIG. 1 is a current remote real-time voice transmission and playback method;
FIG. 2 is a schematic diagram of stereo omni-directional playback;
FIG. 3 is a schematic diagram of the structure of the remote stereo omni-directional real-time transmission and playing method of the present invention;
FIG. 4 is a schematic diagram of the telestereo omni-directional real-time transmission and playback method of the present invention;
fig. 5 is a schematic cubic space diagram of the present invention telestereo omni-directional real-time transmission and playback method.
Detailed Description
In order to overcome the defects that the current remote stereo cannot realize all-around transmission and the immersion is influenced, the invention provides a remote stereo all-around real-time transmission and playing method which can realize all-around transmission and has strong immersion.
For a more clear understanding of the technical features, objects and effects of the present invention, embodiments of the present invention will now be described in detail with reference to the accompanying drawings.
Please refer to fig. 1. Fig. 1 shows a current remote real-time voice transmission and playing method, in which a terminal 13 includes an acoustic unit 133, data is requested from a server 11 through the terminal 13, the server 11 transmits corresponding voice data to the terminal 13 according to the data requested by the terminal 13, and the terminal 13 transmits a voice signal to the acoustic unit 133 after processing.
Please refer to fig. 2. Fig. 2 is a schematic diagram of stereo omni-directional playback. Panoramic images are often presented through virtual reality helmets, augmented reality glasses, and panoramic screens, and it is desirable to ensure that the user 20 experiences realistic images regardless of the direction in which the user is moving. The stereo sound is a necessary factor for greatly improving the immersion feeling by matching with the panoramic image. In the existing partial stereo omnibearing playing technology, omnibearing stereo sound related to position and angle is made and stored in related equipment, and when the equipment is used, stereo sound playing corresponding to the position and angle is called. This approach has encountered difficulties in the telestereo omni-directional real-time transmission. This is because in the long-range real-time transmission, the stereo sound of 360 degrees of all-round can not be made in advance, and the collection of the sound of each position and each direction possess very big data bulk, and real-time transmission these data bulk can seriously occupy the bandwidth, and the space of extrusion image transmission causes the image transmission to appear blocking, and then influences virtual reality experience and sense of immersion.
Referring to fig. 3-5, in the present invention, a server 11, a transmission system 12 and a terminal 13 are required for completing the telestereo omni-directional real-time transmission and playback, and the server 11 and the terminal 13 are connected through the transmission system 12 and communicate with each other. The terminal 13 includes a processing unit 137, an acoustic unit 133, an environment simulation unit 139 and a motion detection unit 135, wherein the processing unit 137 is electrically connected to the acoustic unit 133, the environment simulation unit 139 and the motion detection unit 135, respectively. The environment simulation unit 139 may simulate the environment scene according to the related command of the processing unit 137, and the server 11 may transmit the environment information to the terminal 13 and command the environment simulation unit 139 to express by the processing unit 137. The environment simulation unit 139 may be provided with a blower (not shown) to simulate a wind direction and a water spray device (not shown) to simulate environmental changes such as water spray, rain, and the like. The motion detection unit 135 includes a posture detection device 1353, a speed detection device 1355, and a position detection device 1357. The processing unit 137 includes a response test device 1371, and the response test device 1371 can measure a network response speed.
The server 11 divides the space where the user 20 is located into n cubes, and transmits cube data to the processing unit 137, wherein the user 20 is present in one of the cubes, which is called the sound emitting area 51. In the sound emission area 51, the server 11 sets m virtual sound sources 40. The processing unit 137 may obtain sound information and position information of the virtual sound source 40 by requesting data from the server 11. Each virtual sound source 40 may be used by the processing unit 137 to simulate sound source sounding and integrated by the processing unit 137 and then transmitted to the acoustic unit 133 via the CH1 and CH2 channels. The sound production of the virtual sound source 40 is determined according to the position coordinates and the angle coordinates of the user, and the processing unit 137 can respectively determine the distance and the direction between the virtual sound source 40 and the two ears of the user through the position coordinates and the angle coordinates, and respectively simulate the sound production to the two ears according to the related formula. The calculation formula is as follows:
Lp＝Lw-K+DIm-Ae
where Lp is the sound pressure of both human ears, and Lw is the sound pressure of a sound source, which corresponds to the loudness. The distance r is the propagation distance and is included in the parameter K.
(1) Radiation is spherical wave, divergence attenuation K:
K＝10log(10,4π)+20log(10,r)
where r is the distance between the position of the virtual sound source 40 and a single ear.
(2) Directivity factor DIm is the presence of a reflecting surface near the source (considered alone on the ground), or the source itself is not a point source, adding a reflecting surface by 3 dB.
(3) Other additional attenuations Ae, which we ignore in this equation, therefore, the equation becomes:
Lp＝Lw-10log(10,4π)+20log(10,r)+DIm
with the above formula, the simulated sound source 40 calculates the vocalization to both ears, respectively, and transmits to both ears, respectively. The processing unit 137 integrates the sounds of all the simulated sound sources 40 and transmits the final integrated result to the acoustic unit 133.
From the time the movement state and position information of the user 20 is detected, to the time the sound information is delivered to the user, a time period is required for delivery and processing. Let the starting time of this time period be T0Then the time period requires: the time when the motion state of the user 20 is detected and transmitted to the processing unit 137 is the detection time t1The processing time of the processing unit 137 is t2Time t required for the processing unit 137 to transmit data to the server 113 Server 11 processes time t4The time t at which the server 11 transmits the corresponding sound data to the processing unit 1375。(t1+t2+t3+t4+t5) This time is called the delay time, and the delay time is recorded as t10. Delay time t10The performance of the server 11 and the terminal 13 and the network transmission speed are different.It can be easily obtained that the response time measured by the response test device 1371 is (t)3+t4+t5) Due to t1And t2Is substantially constant for a particular terminal, so that the delay time t10It can pass the response time (t)3+t4+t5) And a fixed time t1、t2And (4) calculating.
Due to the motion limitation of the human, the number of cubes which the human passes through for a short time in reality is limited, and the human is called to pass through for a delay time t10The cubic area through which the inner pass motion is likely to pass is the transmission sound area 50. The transmission and playing process of the remote stereo omnibearing real-time transmission and playing method of the invention is as follows: at T0At this time, the angular coordinate information and the position coordinate information of the user 20 are detected by the motion detection unit 135, and t passes1Is transmitted to the processing unit 137, the processing unit 137 passes t2The time processing requests the server 11 to transmit the data of the sound region 50, and t passes3The time data request message is transmitted to the server 11, and the server 11 passes t4The time processing transmits the data corresponding to the transmission sound region 50 to the terminal 13 in the downstream direction, and t passes5When the time reaches the terminal and the reception is completed, it is marked as T1The time of day. At the same time, the motion detection unit 135 detects that the user 20 is at T1The angle coordinate information and the position coordinate information of the time are transmitted to the processing unit 137, and the processing unit 137 captures the corresponding T in the transmitted sound transmission region 50 after processing1The sound emission area 51 of the time angle coordinate information and the position coordinate information is transmitted to the acoustic unit 133.
In the present invention, the range of the transmission sound region 50 is important. If the transmission sound area 50 is too small, the movement of the user 20 within the buffering time may cause the sound emission area 51 to exceed the range of the transmission sound area 50, resulting in no sound being played; if the transmission sound region 50 is too large, the amount of data transmitted through the network increases, and the image bandwidth and the immersion feeling may be affected if the network is unstable.
Hair brushThe first embodiment specifies the transmission sound region 50 by calculating the maximum coordinate offset of the user 20. We establish a virtual rectangular coordinate system with the user 20 at the time having angular coordinates along the X, y and z axes as the location coordinate (X)0，Y0，Z0) Therefore, the coordinate information of the user 20 at this time is (X)0，Y0，Z0). The processing unit 137 calculates the delay time t by calculation10Maximum offset (Δ X) in the forward direction of the angular position of the inner user 201，ΔY1，ΔZ1) And reverse maximum offset (Δ X)2，ΔY2，ΔZ2) Coordinate sets that may appear to the groomer 20:
Ф＝{(X0-ΔX2<X<X0+ΔX1)，(Y0-ΔY2<Y<Y0+ΔY1)，(Z0-ΔZ2<Z<Z0+ΔZ1)|X，Y，Z}；
the set of cubes in which all coordinates within the set Φ lie is the transmission sound region 50. The corresponding coordinates may be detected by the motion detection unit 135.
There are many algorithms for maximum offset of the coordinates, one of which is calculated using the maximum angular and linear acceleration of the movement of the user 20. Here, let the maximum linear acceleration of the user 20 be a, and let the maximum linear acceleration of the user 20 be a in each direction of the x-axis, the y-axis and the z-axisx、ay、azThe linear velocity v of the user 20 moving in each of the x, y and z directionsx、vy、vz，vx、vy、vzCan be detected by the motion detection unit 135 at the delay time t10The maximum displacement of the inner user along each of the x-axis, y-axis and z-axis is (v)xt10+axt10 2/2)、(vyt10+ayt10 2/2、(vzt10+azt10 2And/2), the variation range of the coordinates is as follows:
{(((x0-(vxt10+axt10 2/2))≤x≤(x0+(vxt10+axt10 2/2)))，(((y0-(vyt10+ayt10 2/2))≤y≤(y0+(vyt10+ayt10 2/2)))，(((z0-(vzt10+azt10 2/2))≤z≤(z0+(vzt10+azt10 2/2)))}
at this time, the set of coordinates is:
Ф1＝{(((x0-(vxt10+axt10 2/2))≤x≤(x0+(vxt10+axt10 2/2)))，(((y0-(vyt10+ayt10 2/2))≤y≤(y0+(vyt10+ayt10 2/2)))，(((z0-(vzt10+azt10 2/2))≤z≤(z0+(vzt10+azt10 2/2)))|x，y，z，}。
by using the method to calculate the coordinate set, the transmission sound area 50 can be reduced to a great extent, and resources are saved better.
The second embodiment of the present invention further saves the transmission sound region 50 on the basis of the first embodiment. Because the person has the limit speed in the movement process, the person cannot continue to accelerate when the movement reaches the limit speed. Therefore, let the maximum movement speed of the user 20 be v, and let the maximum speed of the user 20 along each direction of the x-axis, the y-axis and the z-axis be vx’、vy’、vz', at a delay time t10The maximum speed of movement of the inner user 20 in each of the x, y and z directions is: v. ofx’t10、vy’t10、vz’t10The variation range of the coordinates is as follows:
{((x0-vx’t10)≤x≤(x0+vx’t10))，((y0-vy’t10)≤y≤(y0+vy’t10))，((z0-vz’t10)≤z≤(z0+vz’t10))}，
at this time, the set of coordinates is:
Ф2＝{((x0-vx’t10)≤x≤(x0+vx’t10))，((y0-vy’t10)≤y≤(y0+vy’t10))，((z0-vz’t10)≤z≤(z0+vz’t10))|x，y，z}。
the user 20 does not exceed the phi no matter how fast the head is rotated2Coordinate range of (1), let us say3＝(Ф1∩Ф2) Phi is3I.e. the set of angular coordinates that the user may present. This results in a further reduction of the transmission sound zone 50, reducing the amount of transmission data.
Theoretically, the larger the number of virtual sound sources 40, the more realistic the sound can be reproduced. But due to limitations in device performance and network bandwidth, it is not possible to increase the number of virtual sound sources 40 indefinitely. In the server 11, a plurality of kinds of virtual sound source 40 schemes are stored according to different network conditions and equipment performances, the server 11 judges the performances according to the model of the terminal 13, and determines the number and the position of the virtual sound sources 40 by integrating the response time tested by the response testing device 1371, and transmits the corresponding number of virtual sound source 40 data to the processing unit 137.
Compared with the prior art, the invention adopts the scheme of correspondingly transmitting the sound data according to the detection result of the motion detection unit 135, saves a lot of network bandwidth and realizes the omnibearing real-time transmission and playing of the remote stereo. The method of dividing the cube enables the sound selection to be regionalized and quantized, which is beneficial to perfectly restoring the sound and saving the data transmission quantity. By cuttingThe manner of taking the sound emission area 51 and transmitting the sound area 50 not only reduces the amount of data transmission but also avoids the occurrence of a feeling of delay. The range of the sound region 50 can be transmitted more accurately by determining the maximum amount of deviation of the coordinates within the delay time of the user 20, further reducing the amount of data transmission. The stereo sound is perfectly restored by the setting of the virtual sound source 40, further increasing the sense of immersion. The setting of the response test device 1371 can measure the network speed of the user 20, and the selection of the number of the virtual sound sources 40 according to the network condition and the device performance can improve the tone quality to the maximum extent under the condition of ensuring the normal transmission of the sound, and simultaneously conveniently obtain the delay time t10. The delivery of sound to the ears of the user 20 via the virtual sound source 40 further increases the realism of stereo sound.
While the present invention has been described with reference to the embodiments shown in the drawings, the present invention is not limited to the embodiments, which are illustrative and not restrictive, and it will be apparent to those skilled in the art that various changes and modifications can be made therein without departing from the spirit and scope of the invention as defined in the appended claims.
Claims (9)
1. The remote stereo omnibearing real-time transmission and playing method is characterized by comprising a server, a transmission system and a terminal, wherein the terminal comprises a processing unit, a motion detection unit and an acoustic unit, the processing unit is respectively and electrically connected with the motion detection unit and the acoustic unit, the motion detection unit comprises a position detection device and an attitude detection device, and the remote stereo omnibearing real-time transmission and playing method comprises the following steps:
s1: the motion detection unit detects the motion state of a user and transmits the detection result to the processing unit;
s2: the server divides the space where the user is located into n cubes and transmits cube information to the processing unit, and the processing unit obtains a cube region where the user is located and cube regions which can possibly reach according to the coordinate information provided by the motion detection unit to form a transmission sound region;
s3: the server transmits the data corresponding to the transmission sound area to the terminal; the calculation method of the transmission sound area comprises the following steps:
s2.11 the processing unit records the user coordinate information (X) provided by the motion detection unit0，Y0，Z0)；
S2.12 the processing unit calculates the maximum forward offset (Delta X) of the user coordinates1，ΔY1，ΔZ1) And reverse maximum offset (Δ X)2，ΔY2，ΔZ2) Coordinate set phi { (X) that may appear to the unscrambler0-ΔX2<X<X0+ΔX1)，(Y0-ΔY2<Y<Y0+ΔY1)，(Z0-ΔZ2<Z<Z0+ΔZ1)|X，Y，Z，}；
And the cubic region occupied by the point corresponding to the set phi of S2.13 is the transmission sound region.
2. The binaural cue real-time transmission and playback method according to claim 1, wherein the motion detection unit further comprises a velocity detection means, and the transmission sound zone is calculated by:
s2.21 the speed detection device detects linear speeds v of the user 20 along each direction of the x-axis, the y-axis and the z-axisx、vy、vzRecording the delay time as t10The processing unit records the maximum acceleration a of the user 20 in each direction along the x-axis, y-axis and z-axisx、ay、az；
S2.22 the processing unit calculates the delay time t10Set of angular coordinates phi that may occur for an internal user1＝{(((x0-(vxt10+axt10 2/2))≤x≤(x0+(vxt10+axt10 2/2)))，(((y0-(vyt10+ayt10 2/2))≤y≤(y0+(vyt10+ayt10 2/2)))，(((z0-(vzt10+azt10 2/2))≤z≤(z0+(vzt10+azt10 2/2))) x, y, z }, and phi1The cubic area occupied by the corresponding point is the transmission sound area.
3. The binaural cue real-time transmission and playback method according to claim 2, wherein the delay time is a time taken from a point in time when the user detected the motion state to a point in time when the server has finished transmitting the corresponding downlink data.
4. The binaural cue real-time transmission and playback method according to claim 2, wherein the transmission sound zone is calculated by:
s2.31 recording in the processing unit the maximum linear velocity v of the user in each direction of the x-axis, y-axis and z-axis during the delay timex’、vy’、vz’；
S2.32 processing to calculate the aggregate phi2＝{((x0-vx’t10)≤x≤(x0+vx’t10))，((y0-vy’t10)≤y≤(y0+vy’t10))，((z0-vz’t10)≤z≤(z0+vz’t10))|x，y，z}；
S2.33 order Phi3＝(Ф1∩Ф2) Phi is3The cubic area occupied by the corresponding point is the transmission sound area.
5. The binaural cue and playback method according to claim 1, wherein said server is provided with m virtual sound sources in each individual cube, said virtual sound sources being able to simulate sound source utterances, said processing unit requesting data of m said virtual sound sources from said server, integrating the sounds uttered by all said virtual sound sources and transmitting them to said acoustic unit through CH1 and CH2 channels.
6. The binaural cue real-time transmission and playback method according to claim 5, wherein the processing unit determines the position and the facing direction of the user based on the user coordinate information provided by the motion detection unit, and simulates the acoustic information from the virtual sound source 40 to the ears of the user.
7. The binaural cue real-time transmission and playback method according to claim 5, wherein the terminal further comprises a response test means, the response test means being adapted to test a response time, the response time being a time taken for the terminal to send a signal to the server until the terminal receives a corresponding return signal.
8. The binaural cue real-time transmission and playback method according to claim 7, wherein the server determines the number and locations of the virtual sound sources based on the response time measured by the response testing means and the performance of the terminal.
9. The binaural cue real-time transmission and playback method according to any of claims 1-8, wherein the terminal is a virtual reality headset or augmented reality glasses.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610494569.7A CN106057207B (en) | 2016-06-30 | 2016-06-30 | Remote stereo omnibearing real-time transmission and playing method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610494569.7A CN106057207B (en) | 2016-06-30 | 2016-06-30 | Remote stereo omnibearing real-time transmission and playing method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106057207A CN106057207A (en) | 2016-10-26 |
| CN106057207B true CN106057207B (en) | 2021-02-23 |
Family
ID=57166267
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610494569.7A Expired - Fee Related CN106057207B (en) | 2016-06-30 | 2016-06-30 | Remote stereo omnibearing real-time transmission and playing method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106057207B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10264380B2 (en) * | 2017-05-09 | 2019-04-16 | Microsoft Technology Licensing, Llc | Spatial audio for three-dimensional data sets |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103218198A (en) * | 2011-08-12 | 2013-07-24 | 索尼电脑娱乐公司 | Sound localization for user in motion |
| GB2508830A (en) * | 2012-12-11 | 2014-06-18 | Holition Ltd | Augmented reality system for trying out virtual clothing with associated sound |
| CN104871558A (en) * | 2012-11-28 | 2015-08-26 | 高通股份有限公司 | Image generation for collaborative sound systems |
| CN105451152A (en) * | 2015-11-02 | 2016-03-30 | 上海交通大学 | Hearer-position-tracking-based real-time sound field reconstruction system and method |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| ITMI20070009A1 (en) * | 2007-01-05 | 2008-07-06 | St Microelectronics Srl | AN INTERACTIVE ELECTRONIC ENTERTAINMENT SYSTEM |
-
2016
- 2016-06-30 CN CN201610494569.7A patent/CN106057207B/en not_active Expired - Fee Related
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103218198A (en) * | 2011-08-12 | 2013-07-24 | 索尼电脑娱乐公司 | Sound localization for user in motion |
| CN104871558A (en) * | 2012-11-28 | 2015-08-26 | 高通股份有限公司 | Image generation for collaborative sound systems |
| GB2508830A (en) * | 2012-12-11 | 2014-06-18 | Holition Ltd | Augmented reality system for trying out virtual clothing with associated sound |
| CN105451152A (en) * | 2015-11-02 | 2016-03-30 | 上海交通大学 | Hearer-position-tracking-based real-time sound field reconstruction system and method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106057207A (en) | 2016-10-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11570570B2 (en) | Spatial audio for interactive audio environments | |
| US10911882B2 (en) | Methods and systems for generating spatialized audio | |
| EP3440538B1 (en) | Spatialized audio output based on predicted position data | |
| EP3687190B1 (en) | Mapping virtual sound sources to physical speakers in extended reality applications | |
| US11109177B2 (en) | Methods and systems for simulating acoustics of an extended reality world | |
| US11917391B2 (en) | Audio signal processing method and apparatus | |
| CN111158459A (en) | Application of geometric acoustics in immersive Virtual Reality (VR) | |
| US20190306651A1 (en) | Audio Content Modification for Playback Audio | |
| JP7453248B2 (en) | Audio equipment and methods of processing it | |
| Murphy et al. | Spatial sound for computer games and virtual reality | |
| GB2536020A (en) | System and method of virtual reality feedback | |
| WO2020123856A1 (en) | Soundfield modeling for efficient encoding and/or retrieval | |
| US11221821B2 (en) | Audio scene processing | |
| US20240282289A1 (en) | Reverberation gain normalization | |
| CN106057207B (en) | Remote stereo omnibearing real-time transmission and playing method | |
| EP3777248A1 (en) | An apparatus, a method and a computer program for controlling playback of spatial audio | |
| WO2023226161A1 (en) | Sound source position determination method, device, and storage medium | |
| CN115720315A (en) | Sound production control method, head-mounted display device and computer storage medium | |
| CN111726732A (en) | Sound effect processing system and sound effect processing method of high-fidelity surround sound format | |
| US20240196154A1 (en) | Optimised real-time audio playback | |
| WO2023199815A1 (en) | Acoustic processing device, program, and acoustic processing system | |
| WO2024084949A1 (en) | Acoustic signal processing method, computer program, and acoustic signal processing device | |
| RU2797362C2 (en) | Audio device and method of its operation | |
| WO2023199778A1 (en) | Acoustic signal processing method, program, acoustic signal processing device, and acoustic signal processing system | |
| WO2024014389A1 (en) | Acoustic signal processing method, computer program, and acoustic signal processing device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20210223 |
|
| CF01 | Termination of patent right due to non-payment of annual fee |