WO2018072214A1

WO2018072214A1 - Mixed reality audio system

Info

Publication number: WO2018072214A1
Application number: PCT/CN2016/102958
Authority: WO
Inventors: 向裴
Original assignee: 向裴
Priority date: 2016-10-21
Filing date: 2016-10-21
Publication date: 2018-04-26

Abstract

Provided is a mixed reality audio system. An audio processing system (400) comprises: a sound source element parsing unit (401) for parsing a sound source signal into a sound source element requiring near-field rendering and a sound source element requiring far-field rendering; a near-field rendering unit (402) for performing near-field rendering on the sound source element requiring near-field rendering so as to realize near-field output; and a far-field rendering unit (403) for performing far-field rendering on the sound source element requiring far-field rendering so as to realize far-field output. The audio processing system (400) provides, by performing comprehensive processing on speaker signals at different distances in the system, extraordinarily realistic three-dimensional audio positioning and experience for users when used for virtual reality, augmented reality, mixed reality, and the like. In this way, the drawbacks of a three-dimensional sound rendering effect obtained by a conventional method based on far-field speaker box amplification or near-field headset amplification are overcome.

Description

Mixed reality audio system

Technical field

This invention relates to audio signal processing, and more particularly to a mixed reality audio system and its signal processing.

Background technique

Three-dimensional sound, 3D sound, immersive audio, virtual reality (VR) / augmented reality (AR) / mixed reality (MR) audio, although not called the same, but mostly point to the same field with software algorithms and supporting hardware The device realizes the auditory feeling that the listener feels immersive, has a good positioning of the sound source virtualized by the sound system, and has a good sense of presence or a virtual sense of the sound field.

Traditionally, there are several methods for achieving this goal:

I. Surrounding user's speaker array: Compared with standard, there are 5.1, 7.1, 11.1, 22.2, ATMOS, Auro 3D, DTS-X and so on. There are also non-standard customized multi-speaker placement methods. Belong to the far field or free field. This method is very easy for the user to have a good sense of immersion, because the sound itself comes from the surrounding sound source, but if you need to render a sound source close to the user, because of the limited number of speakers, it is impossible to pass optical holographic The method is done.

II. Based on the earphone, the head-related transfer function (HRTF) or the binaural room impulse response (BRIR) convolution is used to obtain the positioning of the virtual sound. This method can be added to the head tracking to increase the authenticity. The disadvantage of this method is that it is difficult to overcome the head effect, that is, due to the sound field properties of the headphones and the individual differences of the human ear, etc., after the virtual, most users will feel the sound in the head instead of outside, reducing The authenticity of the rendering.

III. Front speaker array: The speaker array in front of the user can use a signal processing method such as cross-talk cancelation, and a virtual 3D sound sense with a set of speakers only in front. The disadvantage of this method is that the relative position of the user's head is very high, and it is not suitable for multiple users to enjoy at the same time.

Summary of the invention

In view of the above problems, one of the objects of the present invention is to provide a new audio processing system. Provides extremely realistic 3D audio positioning and experience for virtual reality (VR), augmented reality (AR), mixed reality (MR) and similar situations through the integrated processing of loudspeaker signals at different distances within the system. It is hoped that the present invention overcomes the drawbacks of the conventional method based only on far-field speaker amplification or near-field earphone amplification in the three-dimensional sound rendering effect.

According to a first aspect of the present invention, there is provided an audio processing system comprising: a sound source element parsing unit that parses a sound source signal into a sound source element requiring near field rendering and a sound source element requiring far field rendering; near field rendering Unit for near-field rendering of sound source elements that require near-field rendering for near-field output; and far-field rendering units for far-field rendering of source elements that require far-field rendering for far-field output.

The audio processing system may further include a near-field far-field transition unit for balancing sound source elements that need to be near-field rendering and sound source elements that require far-field rendering from the same sound source, respectively, in the near-field rendering unit and far The output of the field rendering unit.

Preferably, the near-field far-field transition unit is further configured to cooperate with a sound source element analysis unit to adjust a sound source element that needs to be near-field rendering and a sound source that requires far-field rendering, which are parsed by the same sound source as time changes. element.

In the audio processing system, the near field rendering unit performs near field rendering using a near field headphone-like algorithm. Preferably, the near field headphone type algorithm comprises: a rendering algorithm using a head related transfer function or a binaural room impulse response convolution.

In the audio processing system, the far field rendering unit uses a far field speaker algorithm for far field rendering. Preferably, the far field speaker algorithm comprises an object based three dimensional sound rendering algorithm.

The audio processing system may further include: a near field rendering output unit for outputting the near field rendered signal; and a far field rendering output unit for outputting the far field rendered signal. Preferably, the near field rendering output unit is an open type earphone and the far field rendering output unit is a speaker array.

According to a second aspect of the present invention, an audio processing method is provided, comprising: solving a sound source signal Analysis of sound source elements that require near-field rendering and sound source elements that require far-field rendering; near-field rendering of sound source elements that require near-field rendering for near-field output; and sound source elements that require far-field rendering Perform far-field rendering for far-field output.

The audio processing method may further include: balancing the output of the sound source element that needs to be near-field rendering and the sound source element that needs far-field rendering that are parsed by the same sound source, respectively, after near-field rendering and far-field rendering.

Preferably, the balancing operation further comprises: adjusting a sound source element that needs to be near-field rendering and a sound source element that requires far-field rendering, which are parsed by the same sound source, as time changes.

In the audio processing method described, near field rendering is performed using a near field headphone class algorithm. Preferably, the near field headphone type algorithm comprises: a rendering algorithm using a head related transfer function or a binaural room impulse response convolution.

In the audio processing method described, far field rendering is performed using a far field speaker algorithm. Preferably, the far field speaker algorithm comprises an object based three dimensional sound rendering algorithm.

According to a third aspect of the present invention, a near field terminal processing system for mixed reality audio is provided, comprising: a point sound source positioning unit that judges a position of a point sound source for virtual positioning; and a near field element extraction of the scene sound source a unit that extracts an element that requires near-field rendering in a scene sound source signal; a near-field rendering unit that performs near-field rendering of the virtual localized point source signal and elements of the extracted scene source signal that require near-field rendering; and output Unit, output the signal after the near field rendering.

Preferably, the output unit is a headset, and the near field rendering is a three-dimensional rendering for the headset.

According to a fourth aspect of the present invention, a far field terminal processing system for mixed reality audio is provided, comprising: a point source source far field element extracting unit that extracts elements in a point sound source signal that require far field rendering; scene sound source processing The unit removes the near-field processing part from the scene sound source signal; the far-field rendering unit performs far-field rendering on the extracted point source signal that requires far-field rendering and the scene source signal from which the near-field processing part is removed. And an output unit that outputs the signal after the far field rendering.

Preferably, the output unit is an array of speakers, and the far field rendering is a three-dimensional rendering for a far field speaker.

The invention parses out the sound source elements that need to be near-field rendering and the sound source elements that need to be far-field rendered from the sound source signal, and performs corresponding near-field rendering and far-field rendering respectively for the two. Thereby, extremely realistic 3D audio positioning and experience is provided, especially for 3D sound signals, or VR/AR/MR and the like. Thus, the present invention overcomes the drawbacks of the conventional method based only on far-field speaker amplification or near-field earphone amplification in the three-dimensional sound rendering effect.

DRAWINGS

The invention will now be described in connection with the embodiments with reference to the accompanying drawings. In the drawing:

FIG. 1 is an example illustrating a game scene.

2 is a schematic diagram of a process flow of a near field terminal processing system in accordance with one embodiment of the present invention.

3 is a schematic diagram of a process flow of a far field terminal processing system in accordance with one embodiment of the present invention.

4 is a schematic block diagram illustrating an audio processing system in accordance with the present invention.

Figure 5 is a flow chart illustrating an audio processing method in accordance with the present invention.

detailed description

Specific embodiments of the present invention will be explained in detail below with reference to the accompanying drawings.

The present invention proposes a new software and hardware audio system design for VR/AR/MR related application scenarios, but the invention itself is not limited to such scenarios. The mixed reality audio system according to the present invention may hereinafter be referred to simply as an MRA system (Mixed Reality Audio System).

The MRA system processes the sound content of the near field through earphone-like algorithms such as HRTF/BRIR, and the speaker array processing for the far-field content. The transition between the two fields is debugged and manipulated by the MRA system.

FIG. 1 is an example illustrating a game scene.

As shown in Figure 1, a bird's-eye view of a multiplayer game is shown. The head orientations of players A and B are respectively indicated by arrows, and each virtual object in the game scene is represented by a square icon, wherein the squares of the rounded corners represent virtual objects (Beach, Boat, Wind, Airplane) far away from the player. In the example of Figure 1, Monster and Bird are very close to the player, and the headphone-related algorithms should be used to render the sound positions suitable for the relative positions of A and B on the helmets of A and B respectively. For example, Bird should be on the right side of B, and A is also on the right front and farther away; Monster should be on the left rear of B and on the right front of A. Thus, the near-field rendering systems for A and B should be operated separately, and the head rotation information of A and B needs to be processed. For objects such as Wind, Airplane, Beach, and Boat, which are far away, they can be rendered by a circle of speakers around the game space, so that both A and B hear the same set of far-field sounds, and the positions of the speakers relative to A and B are unchanged. Therefore, it is not necessary to separately process the head rotation information. For objects that transition between near and far fields, the two sets of processing modules should be combined to achieve a realistic effect.

From the perspective of signal processing, each player should have a terminal (or software virtual terminal) to control the processing related to the player information; outside the player, there is another terminal to handle the shared scene, control the far-field speakers for all players to experience, each terminal Driven by the game engine and similar software systems through server links. From the perspective of the game engine, the processing of the sound source can be summarized as the flow shown in Figures 2 and 3.

2 is a schematic diagram of a process flow of a near field terminal processing system in accordance with one embodiment of the present invention. Specifically, Figure 2 depicts the near field terminal processing flow for each player's headset. For each point source, including objects in the game and other players' sounds in the virtual world, the analysis module will find the virtual position to which each point source should be rendered based on peripherals such as gyroscopes, position capture, etc. Headphone rendering algorithms such as HRTF, BRIR convolution, modeling processing, etc. are rendered to the player's headphone output. The terminal processing of each player is independent and different. For scene sound sources, such as the entire environment, etc., most of the cases are handled by the terminal that processes the far-field sound effects of the scene, but for the sake of comprehensive description, it is not excluded to extract the parts and elements that need to be processed for near-field processing from such sound sources. For headphone rendering.

In summary, according to FIG. 2, a near field terminal processing system for mixed reality audio according to an embodiment of the present invention includes: a point sound source positioning unit that judges the position of the point sound source for virtual positioning; The source near-field element extracting unit extracts an element that needs near-field rendering in the scene sound source signal; the near-field rendering unit approximates the virtual-positioned point source signal and the extracted scene source signal that requires near-field rendering. Field rendering; and an output unit that outputs the near-field rendered signal. Wherein, the output unit is an earphone, and the near field rendering is a three-dimensional rendering for the earphone.

3 is a schematic diagram of a process flow of a far field terminal processing system in accordance with one embodiment of the present invention. Specifically, Figure 3 depicts a terminal for far field sound processing. For near-field sources, there are generally some components that require far-field processing. For example, the reverb generated by the reflection of the surrounding environment and the low-frequency portion of the headphones may require a better bass from the far-field speaker. Processing power to complete. After the partial extraction of the direction information is extracted, each point source is processed separately, and then superimposed and sent to the far field terminal for output to the speaker array. For the point source in the far field, various methods can be used, that is, all object-based three-dimensional sound rendering algorithms, such as vector based amplitude panning (VBAP), used to render the height of the point object. High-fidelity stereo image reproduction (HOA, High Order Ambisonics) or Dolby, DTS and other companies' object-based 3D sound rendering methods, etc., to render. For scene sources, most should be rendered and completed by the far field speaker array.

In summary, according to FIG. 3, a far field terminal processing system for mixed reality audio according to an embodiment of the present invention includes: a point source source far field element extracting unit that extracts elements in a point source signal that require far field rendering. The scene sound source processing unit removes the near-field processing part from the scene sound source signal; the far-field rendering unit extracts the elements of the point source signal that need to be far-field rendered and the scene source signal from which the near-field processing part is removed. Perform far-field rendering; and output unit to output the far-field rendered signal. Wherein, the output unit is a speaker array, and the far field rendering is a three-dimensional rendering for a far field speaker.

The hardware and software details of the system for implementing the present invention are discussed in more detail below.

Near-field earphone hardware: At least use open headphones, so users can hear more of the sound of the outside speakers and other players' dialogue; the best way is to not block the earphones, for example:

1. Bone conduction earphone: completely unobstructed, but in the sound positioning because the ear canal is not passed, the effect will be weakened;

2. The sports headphones with half-shield ear holes are not too much on the market, but they have appeared before;

3. Headphones that sound to the ear holes, but are completely unobstructed. There are not many such headphones, but Microsoft HoloLens and other designs have emerged. The inventor has done a similar experiment, playing the ultra-small speaker near the ear hole to play the binaural signal, and achieved a very good 3D sound virtual effect of the head-in effect. The disadvantage of this method is that the coupling between the ear and the ear hole is not good, and the low frequency part is not effective in the general method of wearing the earphone. Because we also have far-field speakers, the low-frequency part can be fully compensated by external speakers, similar to the traditional bass management system that has been widely used.

Near-field headphone algorithm: HRTF or BRIR convolution, and similar 3D rendering techniques, plus bass management system.

Far-field speaker system: 5.1, 7.1, 11.1, 22.2, DTS-X, Dolby ATMOS or a speaker system that covers the entire space customized according to the scene, and the corresponding software and hardware control software.

Far-field speaker algorithm: All object-based 3D sound rendering methods can be used, such as point source VBAP, high-order high-fidelity stereo image copy (HOA, High Order Ambisonics) for rendering point objects, or Dolby, DTS, etc. The company's various object-based 3D sound rendering methods and more. The scene source is adapted to the speaker placement using traditional multi-channel upmix/downmix, or the HOA is optimized directly for each speaker position.

Processing Engine: For existing systems, various game engines such as Unity, Unreal, MaxPlay, etc., are better practice platforms. The inventor's small-scale experiment is now carried out in Unity.

Approach far-field transition module: Each sound source is partially in the approach terminal and partly in the far-field terminal. When the sound source is converted in the near field and far field in the scene, a transitional software module is required. The simplest implementation is roughly a cross-following of the energy, so that the components of the sound source are divided into parts belonging to the player's near-field space and parts of the far-field speakers that are common to the player, but the two add up. Energy and sound source energy are equivalent. According to the specific location of each player, The volume and position of the near-field source will be different, and the far-field part must share a version, so there will be a more specific design. This patent should cover variations of various designs based on this concept. This transitional module can be understood as a change to the module in Fig. 2 and Fig. 3, but in terms of temporal changes.

In accordance with the above description, and particularly in connection with the illustrations of Figures 3 and 4, the present invention actually provides a general purpose audio processing system and audio processing method.

4 is a schematic block diagram illustrating an audio processing system in accordance with the present invention. As shown in FIG. 4, the audio processing system 400 according to the present invention includes: a sound source element parsing unit 401 that parses each sound source signal (eg, a point sound source, a scene sound source, and other possible sound source classification signals) into a required Near-field rendered sound source elements and sound source elements that require far-field rendering; near-field rendering unit 402 that performs near-field rendering of sound source elements that require near-field rendering for near-field output; and far-field rendering unit 403 It performs far-field rendering of sound source elements that require far-field rendering for far-field output.

Alternatively, as previously described, the audio processing system 400 can also include a near field far field transition unit 404. The role of the near-field far-field transition unit 404 is to balance the output of the sound source elements that need to be near-field rendering and the sound source elements that require far-field rendering that are resolved by the same sound source in the near-field rendering unit 402 and the far-field rendering unit 403, respectively. . The result of the balance is that, as mentioned above, a substantially energy-conserved mix is achieved, so that the components of the sound source are divided into parts belonging to the player's near-field space and parts of the far-field speakers that are common to the player, but two The added energy is equivalent to the energy of the sound source.

In addition, as mentioned above, the volume and position of the near-field source will be different depending on the specific location of each player, and the far-field portion can share a version. Those skilled in the art, based on the above teachings, should be able to design more specific methods and firmware/hardware to implement the above transitional functions.

On the other hand, the near-field far-field transition unit 404 can further cooperate with the sound source element parsing unit 401 to adjust the sound source elements that need to be near-field rendering and the sound source that needs far-field rendering that are resolved by the same sound source over time. element. That is to say, not only the energy of near-field rendering and far-field rendering needs to be balanced, but also the components parsed by the sound source elements need to be adjusted according to time, so as to reflect the change of the sound source's near and far position in real time (for example, the player relative When the sound source is in motion, or the sound source itself is In motion).

In FIG. 4, for the control action of the near-field far-field transition unit 404 for the sound source element analysis unit 401, the near-field rendering unit 402, and the far-field rendering unit 403, the illustration is drawn in the form of a broken line. Furthermore, the near field far field transition unit 404 itself is shown as an optional unit, indicated by the dashed box in FIG.

In audio processing system 400, near field rendering unit 402 performs near field rendering using a near field headphone-like algorithm. As mentioned previously, the near field headphone class algorithm may include a rendering algorithm using a Head Related Transfer Function (HRTF) or a binaural room impulse response (BRIR) convolution.

In audio processing system 400, far field rendering unit 403 performs far field rendering using a far field speaker algorithm. As mentioned previously, the far field speaker algorithm can include an object based three dimensional sound rendering algorithm. For example, such an algorithm may include at least one of the following: a point source VBAP (vector base amplitude phase shift), a high order high fidelity stereo image copy (HOA) rendering point object, or various types of companies such as Dolby and DTS. Object-based 3D sound rendering method.

Optionally, the audio processing system 400 according to the present invention may further include: a near field rendering output unit 405 for outputting a near field rendered signal; and a far field rendering output unit 406 for outputting the far field rendered signal. The near field rendering output unit can be an open type earphone, and the far field rendering output unit can be an array of speakers. As mentioned previously, hardware units for near field output and far field output have been discussed in detail.

Figure 5 is a flow chart illustrating an audio processing method in accordance with the present invention. As shown in FIG. 5, an audio processing method 500 in accordance with the present invention may begin at step S501, in which the sound source signal is parsed into sound source elements requiring near field rendering and sound source elements requiring far field rendering. Then, in step S503, near-field rendering is performed on the sound source elements that require near-field rendering for near-field output. In step S505, far-field rendering is performed on the sound source elements that require far-field rendering for far-field output.

Steps S503 and S505 may be performed simultaneously or sequentially. It should be noted that the rendered near-field signal and far-field signal output through the two steps need to be output to the player in synchronization.

In addition, corresponding to the near-field far-field transition unit in FIG. 4, optionally, the method 500 may further include balancing sound source elements that need to be near-field rendering and sound source elements that require far-field rendering that are resolved by the same sound source. The operation of the output after near field rendering and far field rendering, respectively.

Moreover, as described above, the balancing operation may further include: adjusting a sound source element that needs to be near-field rendering and a sound source element that requires far-field rendering, which are parsed by the same sound source, as time changes.

In the audio processing method 500, near field rendering is performed using a near field headphone-like algorithm. The near field headphone class algorithm includes a rendering algorithm using a Head Related Transfer Function (HRTF) or a binaural room impulse response (BRIR) convolution.

In the audio processing method 500, far field rendering is performed using a far field speaker algorithm. The far field speaker algorithm includes an object based three dimensional sound rendering algorithm. For example, such an algorithm may include at least one of: a point source VBAP (vector base amplitude phase shift), a high order high fidelity stereo image copy (HOA) rendering point object, or a variety of companies based on Dolby, DTS, etc. The three-dimensional sound rendering method of the object.

Application scenario

The present invention is applicable to single/multiplayer games, experience spaces, display spaces, museums, educational spaces, and the like. Thanks to this invention, users have a level of leap in the experience of three-dimensional sound.

Remarks: The “players” and “users” mentioned above can actually be called, and they all refer to the end users of such systems.

In view of the above description of the specification, those skilled in the art will appreciate that the present invention may also include the following supplementary annotation technical solutions in addition to the technical solutions listed in the claims.

Additional notes:

A computer readable recording medium on which instructions are stored which, when executed by one or more processors for audio processing, cause said one or more The processors do the following:

Parsing the sound source signal into sound source elements that require near-field rendering and sound source elements that require far-field rendering;

Near-field rendering of sound source elements that require near-field rendering for near-field output;

Far-field rendering of sound source elements that require far-field rendering for far-field output.

2. The computer readable recording medium of supplementary note 1, further comprising instructions that cause the one or more processes to perform the following operations:

Balances the sound source elements that need to be near-field rendered and the sound source elements that require far-field rendering that are parsed by the same sound source, respectively, after near-field rendering and far-field rendering.

3. The computer readable recording medium according to supplementary note 2, wherein the balancing operation further comprises: adjusting a sound source element that needs to be near-field rendering and being analyzed by the same sound source as time changes, and requiring far-field rendering Sound source element.

4. The computer readable recording medium according to supplementary note 1, wherein the near field heading algorithm is used for near field rendering.

5. The computer readable recording medium of supplementary note 4, wherein the near field headphone type algorithm comprises: a rendering algorithm using a head related transfer function or a binaural room impulse response convolution.

6. The computer readable recording medium according to supplementary note 1, wherein the far field rendering is performed using a far field speaker algorithm.

7. The computer readable recording medium of supplementary note 6, wherein the far field speaker algorithm comprises an object based three dimensional sound rendering algorithm.

8. A near field terminal processing method for mixed reality audio, comprising:

Judging the position of the point source to perform virtual positioning;

Extracting elements of the scene source signal that require near field rendering;

The near-field rendering of the virtual localized point source signal and the extracted scene source signal is required. Elements for near-field rendering;

Output the near-field rendered signal.

9. The near field terminal processing method according to supplementary note 8, wherein the near field rendered signal is output using a headphone, and the near field rendering is a three-dimensional rendering for the earphone.

10. A far field terminal processing method for mixed reality audio, comprising:

Extracting elements of the point source signal that require far field rendering;

Removing the near field processing portion from the scene sound source signal;

Far-field rendering of the extracted point source signal that requires far-field rendering and far-field rendering of the scene source signal with the near-field processing portion removed;

Output the signal after far field rendering.

11. The far field terminal processing method of supplementary note 10, wherein the far field rendered signal is output using a speaker array, the far field rendering being a three-dimensional rendering for a far field speaker.

12. A computer readable recording medium on which instructions are stored that, when executed by one or more processors performing near field terminal processing for mixed reality audio, cause said one or more The processors do the following:

Judging the position of the point source to perform virtual positioning;

The virtual localized point source signal and the extracted scene source signal require near field rendering of the near field rendering element.

13. The computer readable recording medium of supplementary note 12, wherein the near field rendering is a three-dimensional rendering for a headset.

14. A computer readable recording medium on which instructions are stored for causing said one or more when executed by one or more processors performing far field terminal processing for mixed reality audio The processors do the following:

Removing the near field processing portion from the scene sound source signal;

The far-field rendering of the extracted point source signal that requires far-field rendering and the scene source signal from which the near-field processing portion is removed is performed.

The computer readable recording medium of claim 14, wherein the far field rendering is a three-dimensional rendering for a far field speaker.

Various embodiments and implementations of the invention have been described above. However, the spirit and scope of the present invention are not limited thereto. Those skilled in the art will be able to make further applications in accordance with the teachings of the present invention, and such applications are within the scope of the present invention.

Claims

An audio processing system comprising:

The sound source element parsing unit parses the sound source signal into a sound source element requiring near field rendering and a sound source element requiring far field rendering;

Near-field rendering unit for near-field rendering of sound source elements that require near-field rendering for near-field output;

The far-field rendering unit performs far-field rendering on sound source elements that require far-field rendering for far-field output.
The audio processing system of claim 1 further comprising:

The near-field far field transition unit is configured to balance the output of the sound source element that needs to be near-field rendering and the sound source element that needs far-field rendering that are parsed by the same sound source in the near-field rendering unit and the far-field rendering unit, respectively.
The audio processing system of claim 2, wherein the near-field far-field transition unit is further configured to cooperate with the sound source element analysis unit to adjust the sound of the near-field rendering that is resolved by the same sound source over time. Source elements and sound source elements that require far-field rendering.
The audio processing system of claim 1 wherein said near field rendering unit performs near field rendering using a near field headphone class algorithm.
The audio processing system of claim 4 wherein said near field headphone type algorithm comprises a rendering algorithm using a Head Related Transfer Function (HRTF) or a binaural room impulse response (BRIR) convolution.
The audio processing system of claim 1 wherein said far field rendering unit performs far field rendering using a far field speaker algorithm.
The audio processing system of claim 6 wherein said far field speaker algorithm comprises an object based three dimensional sound rendering algorithm.
The audio processing system of claim 1 further comprising:

a near field rendering output unit for outputting a near field rendered signal; and

The far field rendering output unit is used to output the signal after far field rendering.
The audio processing system of claim 8 wherein the near field rendering output unit is an open type earphone and the far field rendering output unit is a speaker array.
An audio processing method comprising:

Parsing the sound source signal into sound source elements that require near-field rendering and sound source elements that require far-field rendering;

Near-field rendering of sound source elements that require near-field rendering for near-field output;

Far-field rendering of sound source elements that require far-field rendering for far-field output.
The audio processing method of claim 10, further comprising:

Balances the sound source elements that need to be near-field rendered and the sound source elements that require far-field rendering that are parsed by the same sound source, respectively, after near-field rendering and far-field rendering.
The audio processing method according to claim 11, wherein said balancing operation further comprises: adjusting a sound source element required for near-field rendering and a sound source element requiring far-field rendering, which are parsed by the same sound source as time changes. .
The audio processing method according to claim 10, wherein the near field rendering is performed using a near field headphone type algorithm.
The audio processing method of claim 13, wherein the near field headphone type algorithm comprises a rendering algorithm using a head related transfer function (HRTF) or a binaural room impulse response (BRIR) convolution.
The audio processing method of claim 10 wherein the far field rendering is performed using a far field speaker algorithm.
The audio processing method of claim 15 wherein said far field speaker algorithm comprises an object based three dimensional sound rendering algorithm.
A near field terminal processing system for mixed reality audio, comprising:

Point source localization unit, judging the position of the point sound source for virtual positioning;

a scene source near-field element extracting unit that extracts elements of the scene sound source signal that require near-field rendering;

a near-field rendering unit that performs near-field rendering of the virtually positioned point source signal and elements of the extracted scene source signal that require near field rendering;

The output unit outputs the signal after the near field rendering.
The near field terminal processing system of claim 17 wherein said output unit is a headset and said near field rendering is a three dimensional rendering of the headset.
A far field terminal processing system for mixed reality audio, comprising:

Point source source far field element extraction unit, extracting elements in the point source signal that require far field rendering;

a scene sound source processing unit that removes a near field processing portion from the scene sound source signal;

a far field rendering unit that performs far-field rendering of the extracted point source signal that requires far-field rendering and the scene source signal from which the near-field processing portion is removed;

The output unit outputs the signal after the far field rendering.
The far field terminal processing system of claim 19 wherein said output unit is a speaker array and said far field rendering is a three dimensional rendering of a far field speaker.