US20100197401A1

US20100197401A1 - Reliable, efficient and low cost method for games audio rendering

Info

Publication number: US20100197401A1
Application number: US12/365,298
Authority: US
Inventors: Yaniv Altshuler; Eyal Regev; Adi Ashkenazy
Original assignee: Individual
Current assignee: Individual
Priority date: 2009-02-04
Filing date: 2009-02-04
Publication date: 2010-08-05

Abstract

An efficient method for a reliable and accurate rendering of audio in a virtual reality environment, such as computer gaming. It excels other known rendering methods in its ability to achieve high quality audio rendering in real time conditions using a regular personal computer without any need for a special hardware accelerated audio card. A total of k physical properties are defined for the sound source and for the various objects that may affect the generation and distribution of the sound in the simulated environment. For each one of those physical properties some applicable values are predefined in a granularity which is a parameter of the method. This way a k dimensional grid of data points is defined. For each one of those data point the sounds that will be heard by the listener are offline calculated offline as reference samples. When the sound shall be generated in real time, for any given real data point, it is calculated as an interpolation of some near reference samples on the predefined k dimensional grid.

Description

FIELD OF THE INVENTION

The present invention relates generally to a method for rendering audio and acoustic effects in computer games.

BACKGROUND OF THE INVENTION

The level of sophistication and complexity of computer games keeps rising. In particular, the interactive 3D audio visual capabilities of games are constantly developing to provide a greater sense of realism to the gaming experience and to enhance the overall enjoyment of the players in single player and multiplayer modes. One of the significant accelerators of this process has been the stunning development of highly advanced surround audio technology for home entertainment and cinema systems.
The modern audio systems have lead to a demand for rendering of complex acoustic effects in games at a level of realism that is customary in the advanced movie industry. In order to achieve this, sound synthesis should depend on many parameters. For example, the sound of a sword that is swiftly cutting the air would depend on its motion speed as well as on its exact dimensions and shape. As another example, the sound of a ball falling on the ground depends upon the ball physical properties, the ground material and the ball speed and angle of hit.
Beyond the task of accurate sound generation, the more challenging task is simulating authentic distribution of sounds in the game virtual environment. This must take in consideration the structure of the objects that comprise this environment in terms of polygons, the acoustic properties of those objects such as reflection coefficients, the distances among the sound source, the involved objects and the listener as well as the variation in time of those distances. The above factors determine the characteristics of various acoustic effects that shall be associated with the distributed sound like echo, reverb, Doppler, flange, occlusion, obstruction etc.
The theory of calculating the above sounds and effects that may be relevant to the various scenarios occurring in games are known in the art of acoustics. In particular, there are standard software APIs and libraries that are commonly used for the complex calculations required for reliable audio rendering. The acoustic calculations can be done either offline, during the game creation phase, or in real time, when it is actually played.
The advantage of the offline mode is that it enables accurate calculations, taking in consideration the various rendering aspects, like the effects of physical properties of objects, the distances among them, their dimensions and relevant acoustic affects. However, the number of pre-generated sound samples and effects that can be pre-calculated in advance is limited owing to storage capacity limitations. Therefore, those samples would normally differ significantly from those required for the dynamically varying real time gaming conditions.
In the real time mode the sound samples and effects are generated in the player's computer or gaming console while the game is being played. The advantage is that the acoustic parameters may be accurately chosen according to the specific conditions at any given moment during the game. However, as this approach consumes significant processing power, when implemented in a regular PC (Personal Computer), this would come at the expense of the accuracy of the required calculations, hence limiting the achieved sense of realism of the gaming experience. This limitation can be overcome by using dedicated powerful hardware accelerators that are integrated in advanced audio cards or gaming consoles. However, this solution implies excess expenses for the player; furthermore the solution is rigid and cannot be readily adapted for all computer games.

SUMMARY OF THE INVENTION

The background art does not teach or suggest how to achieve reliable and accurate rendering of sounds and acoustic effects that are required in advanced games when those are played on regular computers and/or gaming consoles.
The present invention overcomes these drawbacks of the background art by providing a method for reliable and accurate rendering of sounds and acoustic effects without requiring excessive processing power. Without limitation, as an example of the many advantages of the present invention, the player may enjoy authentic acoustic experience in his or her home entertainment environment while saving the extra expense of special audio cards or gaming consoles, which normally use dedicated hardware accelerators in order to achieve this goal.
According to some embodiments of the present invention most of the processing effort that is required for reliable audio rendering is expended offline, i.e. in the game creation phase rather than in real time. This effort is mainly invested in calculating reliable sound samples for predefined discrete conditions of the game flow. The sound samples that pertain to the real game conditions are calculated at run time as interpolation of the off line calculated sound samples.
The offline rendered sound samples are calculated by taking into account the relevant parameters that characterize the sound sources, the physical properties of the objects that take part in the generation of the sounds and of the objects that affect the distribution of the sounds to the listening player, and the various acoustic effects that affect the authentic playback of sound samples to the listener.
The rendered sound samples are preferably calculated at discrete points in the multidimensional space that is created from the parameters and properties for each expected scenario in the game. These rendered sound samples are incorporated as database records by the game designer in the software package of the game that is supplied to the player.
When the player runs the game in real time on a computer or gaming console, for each real combination of values of the parameters and properties, the accurate rendered sound samples are calculated as an interpolation of the records that pertain to some points that are close to the real combination in the multidimensional space. The optionally allowed discretization error that may result due to the interpolation would affect the specific interpolation method that is chosen, as well as the number of near data points that shall be selected. It should be noted that the terms “computer” and “gaming console” are used interchangeably and without limitation throughout the present application.
In another embodiment of the invention, if the player computer has limited processing power, it may optionally send to a dedicated Game Server requests for actual interpolation calculations. The game designer too may have the option to request that dedicated Game Server for accurate interpolation calculations in order to save design resources and to allow a simpler interpolation method at the player computer.
Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. The materials, methods, and examples provided herein are illustrative only and not intended to be limiting.
Implementation of the method and system of the present invention involves performing or completing certain selected tasks or stages manually, automatically, or a combination thereof. Moreover, according to actual instrumentation and equipment of preferred embodiments of the method and system of the present invention, several selected stages could be implemented by hardware or by software on any operating system of any firmware or a combination thereof. For example, as hardware, selected stages of the invention could be implemented as a chip or a circuit. As software, selected stages of the invention could be implemented as a plurality of software instructions being executed by a computer using any suitable operating system. In any case, selected stages of the method and system of the invention could be described as being performed by a data processor, such as a computing platform for executing a plurality of instructions.
Although the present invention is described with regard to a “computer” on a “computer network”, it should be noted that optionally any device featuring a data processor and/or the ability to execute one or more instructions may be described as a computer, including but not limited to a PC (personal computer), a server, a minicomputer, a cellular telephone, a smart phone, a PDA (personal data assistant), a pager, TV decoder, game console, digital music player, ATM (machine for dispensing cash), POS credit card terminal (point of sale), electronic cash register. Any two or more of such devices in communication with each other, and/or any computer in communication with any other computer may optionally comprise a “computer network”.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention is herein described, by way of example only, with reference to the accompanying drawings. With specific reference now to the drawings in detail, it is stressed that the particulars shown are by way of example and for purposes of illustrative discussion of the preferred embodiments of the present invention only, and are presented in order to provide what is believed to be the most useful and readily understood description of the principles and conceptual aspects of the invention. In this regard, no attempt is made to show structural details of the invention in more detail than is necessary for a fundamental understanding of the invention, the description taken with the drawings making apparent to those skilled in the art how the several forms of the invention may be embodied in practice.

In the drawings:

FIG. 1 illustrates a characteristic game environment, where some game elements that would affect reliable rendering of the game sounds are particularly depicted.

FIG. 2 depicts the invented method operation flow.

FIG. 3 depicts a linear interpolation method for the specific case of a three dimensional grid of data points.

FIG. 4 illustrates a general system diagram, comprising a designer server, a game server and client players

DETAILED DESCRIPTION OF THE INVENTION

A specific embodiment of the present invention is hereby described referring to a characteristic game environment that is illustrated in FIG. 1. Some elements that would affect reliable rendering of game sounds are particularly depicted in this Figure. A falling ball hits the ground and a sound sample is consequently generated. This sound sample is determined by a set of parameters {S1} that characterize hit related primitive sound sample e.g. the ball velocity at the moment of hit and the hit angle.
The physical properties of the objects that are involved in the hit, such as material type and elasticity, affect the generated sound sample as well. The relevant physical properties of the ball are denoted by {P1}, and those of the floor are denoted by {P2}.
The generated sound is distributed in the hall space toward the listener or player as depicted in tick broken arrow. The listener hears reflections from the walls as well, a phenomenon called “reverb”. The set of wall properties that affect the reverb, like dispersion and reflection coefficients, are denoted by {P3}. The set of distances between the hit point, the walls and the listener, as well as the dimensions of the wall polygons are denoted by {P4}.
FIG. 2 exemplifies a specific embodiment of the invention for software based method of efficient rendering of reliable computer game sounds, for example by a regular computer. The acoustic environment notations in the drawing refer to FIG. 1. The upper portion of the drawing describes the process that takes place at the game development stage while its lower part refers to the game run at the player's home environment.
A typical game is composed of scenarios [201]. Such a scenario is illustrated in FIG. 1. For each one of the scenarios various objects are defined [202] such as the ball, the floor and the walls in FIG. 1. For each one of the objects various physical parameters are specified and are noted in [203] by {P1} . . . {P4}. For each one of the properties, a value or a range of values are ascribed. Such a range can refer, for example, to the distances noted in FIG. 1, which may vary depending on the ball fall location as well as the listener location.
Primitive sound samples [204] are also defined for a typical scenario. In the described scenario this primitive sound sample is the result of the ball hitting the ground, which may depend on various parameters {S1}. Those parameters are evaluated depending on possible variations of the scenario in real time game conditions. The distribution of the sound in the acoustic environment of a typical scenario may be affected by various acoustic affects such as echo, reverb, Doppler, flange, occlusion, obstruction etc {205}. An example of reverb is illustrated in FIG. 1.
Block {206} of FIG. 2 depicts the calculation of a plurality of sound samples that will eventually be heard by the player, who is the listener in typical game scenarios. The sound samples may be reproduced via any speakers array, in particularly via home entertainment/cinema surround system. Therefore a reliable calculation of the sound sample that pertains to each one of the surround speakers is required for providing the player with an authentic spatial experience of the game. The calculations are based on common knowledge in the science of acoustics, using common software libraries and APIs. The resulting sound samples are examined by the game designer who may iteratively adjust the scenario parameters for optimal sound rendering.
The final sound samples are stored in a database [207], typically in WAV format, for later use in real time conditions. Each sound event is stored as a reference group {S2*} of up to 6 sound samples to conform to conventional home multi speaker surround systems. The calculations are performed for discrete values of each one of the physical properties, or parameters, that affect the sound rendering. Assuming k such parameters, they form a k dimensional space and a corresponding k dimensional grid of discrete parameter values that we denoted “data points”. The resulting reference samples are mapped to this k dimensional grid. The actual granularity of the values that are selected for calculation on each one of the grid axes is a parameter of the system and is affected by storage limitations and by the desired accuracy of the sound rendering.
The reference sound samples {S2*} are supplied together with the game PC software package to the game user [209]. While the game is being played, the parameters attain real time values [210] in the k dimensional space, which are typically different from the k dimensional grid values. In order to get reliable sound samples {S2}, for any given real time data point, as depicted in [212], {S2} is calculated in real time, as depicted in [211], as an interpolation of some adjacent S2* samples on the grid. {S2} is then transferred from the PC to any speaker array, in particular home entertainment/cinema surround speaker system.
The interpolation method that is used in [211] can be any one of the methods that are conventional in the art of multidimensional interpolation calculation. Preferably, the method is selected with regard to one or more of grid granulation, the allowed discretization error, the desired response time and the expected PC processing power. Some well known interpolation methods that can be used are provided herein as non-limiting illustrative examples, though other methods can be used as well.
Nearest neighbors: Choosing the n (parameter, 1 by default) nearest (by Euclidian distance) neighbors and average them with the following formula:
$\overline{x} = \frac{\sum_{i = 1}^{n} w_{i} x_{i}}{\sum_{i = 1}^{n} w_{i}},$
where each Wi can be one of the following: Wi=(distance₁+1)^(−alpha), where alpha is also a parameter, 1 by default. Another option is that Wi is e^{(−alpha*distancei)}.
Linear interpolation: The principle of this method is illustrated in FIG. 3. for the specific Trilinear case, i.e. an interpolation in three dimensional grid of data points. The data points are mapped to the vertices of a normalized unit cube. Consider a unit cube with the lower-left-base vertex at the origin of the grid coordinates as shown in FIG. 3.
The values at each vertex will be denoted V000, V100, V010, . . . V111 respectively to the points (x0,y0,z0) . . . (x1,y1,z1). The value at position (x,y,z) within the cube will be denoted Vxyz and is given by
Vxyz=V000(1−x)(1−y)(1−z)+V100 x(1−y)(1−z)+V010(1−x)y(1−z)+V001(1−x)(1−y)z+V101 x(1−y)z+V011(1−x)y z+V110 x y(1−z)+V111 x y z
In general the box will not be of unit size nor will it be aligned at the origin. Simple translation and scaling (possibly of each axis independently) can be used.
Iterative interpolation: In each step choose one of the physical properties and make a polynomial interpolation only on this axis using as many points as possible with a Newton form. Given a set of k+1 data points (x₀,y₀), . . . , (x_k,y_k) where no two x_jare the same, the interpolation polynomial in the Newton form is a linear combination of Newton basis polynomials
$N (x) := \sum_{j = 0}^{k} a_{j} n_{j} (x)$
with the Newton basis polynomials defined as
$n_{j} (x) := \prod_{i = 0}^{j - 1} (x - x_{i})$
while the coefficients are defined by a_j:=[y₀, . . . ,y_j] where [y₀, . . . ,y_j] is the notation for divided differences. Thus the Newton polynomial can be written as N(x):=[y₀]+[y₀, y₁](x−x₀)+ . . . +[y₀, . . . , y_k](x−x₀)(x−x₁) . . . (x−x_k−1).
FIG. 4 exemplifies some variations of the embodiment that is illustrated in FIG. 2. The game creation stages are represented here by “Game Design Server” [401]. However, the game software and grids of reference sound samples at relative low granulation data points are transferred to a dedicated “Game Server” [402]. Game Server can receive requests for accurate interpolation calculations from the game designer in order to save him the burden of accurate calculations and to allow, as well, simpler interpolation calculations at the client player's computer.
Client computer [403] depicts a computer available to a player that is a regular computer, as in FIG. 2. Thin client computer [404] depicts a thin client, which may for example be a mobile phone or PDA, or other computational device with limited processing power, that is available to the player. Thin client computer [404] downloads the game software from the Game Server 402, not including the grids. Whenever actual audio samples are required at real time the thin client computer [404] sends Game Server 402 the real game conditions, Game Server 402 calculates the actual samples by interpolating adjacent reference samples and provides the thin client computer [404] with the results. FIG. 4 also shows that optionally a regular client computer [403] may request interpolation calculations from Game Server [402] if client computer [403] has limited processing power.
While the invention has been described with respect to a limited number of embodiments, it will be appreciated that many variations, modifications and other applications of the invention may be made.

Claims

1. A method for sound rendering to be used in a computer based virtual reality system comprising:

generating a sound sample emanating from a sound source;

associating one or more acoustic effects to said sound sample for affecting the rendering of said sound sample;

predetermining one or more values for one or more physical parameters for affecting rendering of said sound sample according to said one or more acoustic effects, wherein said physical parameters relate to an interaction of a plurality of objects in the virtual reality system; and

interpolating said predetermined one or more values of said one or more physical parameters to determine an effect on said sound sample.

2. The method of claim 1, wherein said predetermining said one or more values comprises associating a total of k physical properties to said sound sample and to any object that may affect perception of said sound sample after rendering, where k may be any positive number, and defining a k dimensional space thereof;

selecting one or more pre-defined values for each one of said k physical properties to generate a k dimensional grid of data points that is mapped on said k dimensional space;

calculating the sound sample or samples for any data point in said k dimensional grid; and

selecting an interpolation method to be applied to said k dimensional grid for calculating the sound sample or samples that would be heard by said listener for any desired combination of k real values of said k properties.

3. The method of claim 2 where the acoustic effects that may affect the rendering of said sound sample in said acoustic environment may be echo, reverb, Doppler, flange, occlusion, obstruction etc.

4. The method of claim 2 wherein said physical parameters comprise one or more of object elasticity, object reflection coefficient, polygon dimensions, or relative distances among said sound source, said objects and said listener etc.

5. The method of claim 2 wherein a plurality of sound samples are calculated to simulate a plurality of corresponding sounds for playback by a multi track audio surround system.

6. The method of claim 2 wherein said selecting said interpolation method to be applied is at least partially determined according to an allowed discretization error from said interpolation.

7. The method of claim 6, wherein said selecting said interpolation method to be applied is further at least partially determined according to the number of data points for each one of the k physical properties in said k dimensional grid.

8. The method of claim 2 wherein at least a portion of the sound samples that are associated with said grid data points are calculated by interpolating other sound samples on said grid that were accurately calculated beforehand.

9. The method of claim 2 wherein the virtual reality system comprises a computer game, wherein the plurality of objects are provided through said computer game and wherein the listener comprises at least one game player.

10. The method of claim 8 wherein said calculating the sound sample or samples for any data point in said k dimensional grid is performed before game play such that a plurality of precalculated multidimensional grids are provided with said computer game, and wherein said calculating said interpolation at the game real time conditions is done by the player computer for any actual combination of values of physical properties that pertain to any real situation during the game.

11. The method of claim 9 further comprising: following any event of new sound samples generation, forecasting the most probable conditions for the next event of sound generation, and reading one or more new predefined sound samples that are not yet cached in the dynamic memory of the computer.

12. The method of claim 8 wherein said calculating the sound sample or samples for any data point in said k dimensional grid is performed before game play such that a plurality of precalculated multidimensional grids are provided with said computer game, and wherein said calculating said interpolation at the game real time conditions, for any actual combination of values of physical properties that pertain to any real situation during the game, is done by a dedicated game server to which the player computer connects and requests the required sound samples when actually required.

13. The method of claim 11 wherein the player computer is a thin client such as a mobile phone or a PDA (personal digital assistant).