EP4681195A1

EP4681195A1 - Grain scheduling for granular synthesis

Info

Publication number: EP4681195A1
Application number: EP24712192.4A
Authority: EP
Inventors: Tommy Falk; Raksha Ramakrishna
Original assignee: Telefonaktiebolaget LM Ericsson AB
Current assignee: Telefonaktiebolaget LM Ericsson AB
Priority date: 2023-03-16
Filing date: 2024-03-13
Publication date: 2026-01-21
Also published as: WO2024189108A1; CN120826737A

Abstract

A method for rendering audio corresponding to an audio recording, wherein the audio recording is divided into a plurality of grains. In one embodiment, the method includes obtaining a first target descriptor coordinate, wherein the first target descriptor coordinate identifies a first location in an N-dimensional descriptor space, where N > 0. The method also includes defining a first set of candidate grains based on the first target descriptor coordinate, wherein the first set of candidate grains comprises k1 of the plurality of grains, where k1 > 1. The method also includes assigning a final weight to each grain in the first set of candidate grains. The method also includes randomly selecting a grain from the first set of candidate grains based on the assigned final weights such that the probability that a given grain in the first set of candidate grains is selected is a function of the final weight assigned to the given grain. The method further includes rendering the selected grain.

Description

GRAIN SCHEDULING FOR GRANULAR SYNTHESIS

TECHNICAL FIELD

[001] Disclosed are embodiments related to granular synthesis.

BACKGROUND

[002] Audio rendering is a process used for presenting audio, such as audio within an extended reality (XR) scene (e.g., a virtual reality (VR), augmented reality (AR), or mixed reality (MR) scene) in order to give a listener the impression that sound is coming from physical sources within the scene at a certain position. The presentation can be made through headphone speakers or other speakers. If the presentation is made via headphone speakers, the processing used is called binaural rendering and uses spatial cues of human spatial hearing that make it possible to determine from which direction sounds are coming. The cues involve inter-aural time delay (ITD), inter-aural level difference (ILD), and/or spectral difference.

[003] Procedural audio refers to the creation of sound in real-time as a response to live input. As an example, consider the sound of a car engine in a virtual space where the sound changes based on the speed or acceleration or the car. This mechanism is commonly used in video games for better user experience. It is believed that for the use case of XR (e.g., AR or VR), there are many sounds that would benefit from being dynamically generated so that they can react to changes in the scene in real-time. For example, the sound generated when a user touches a surface or operates an engine. In reference [1], sounds of a virtual object, for example a sword, axe, or wand, are simulated based on their position and orientation. There is a base tone and an overtone. Both are modulated to change pitch, timbre, amplitude to convey speed of movement of the virtual object. The live input may come from a user via sensors, such as hand controllers or a headset, it could be control data generated in real-time by some software process such as a physics simulation or pre-defined automation data. Regardless of how the input data was generated, the audio Tenderer needs to handle incoming data and generate sound in response to this data in real-time.

[004] There exist many different methods for procedural audio (see, e.g., reference

[2]), including synthetic sound synthesis using audio processing modules, machine learning methods trained on real recordings, and concatenative synthesis methods that make use of original recordings and rearrange segments of these recordings to generate variations.

Because the class of concatenative synthesis methods makes direct use of real recordings, the generated audio sounds very natural thereby enhancing user experience.

[005] Granular synthesis is a type of concatenative synthesis where a sound recording is divided into small fragments called “grains.” (See, e.g., reference [3]). By a careful selection of the fragments (grains) at rendering time, a plausible dynamically changing sound can be generated.

[006] A granular synthesis process includes two main steps: (1) grain extraction and (2) grain synthesis. Grain extraction refers to extraction of pertinent grains from the original longer recording. The extraction method depends on the type of sound source and the desired features to be extracted. Grain synthesis refers to the technique of selecting the appropriate order of grains; this selection of the ordering could also be based on the user input in real-time.

[007] Many sound design tools support grain extraction and synthesis, such as, for example Soundseed grain for Audiokinectic Wwise, Alchemy for Logic Pro or AudioMotors for FMOD. The tools allow for manual or semi -automated extraction of grains by the sound designer and other simple manipulations. The designer can choose the grain length, the amplitude envelope or shape of each grain among other controls. In the case of AudioMotors, an automated grain extraction tool specialized for motor sounds is provided.

[008] Grain extraction can be done manually by the sound designer or in a data- driven manner by identifying the relevant features of the audio for segmentation purposes. Relevant features include, for example, pitch period in the case of pitched sounds, spectral energy at a given frequency, mel-frequency cepstrum coefficients (MFCC), and local maxima of the amplitude envelope.

[009] There are also many methods for granular synthesis. A common method is to select grains at random and perform overlap and add (OLA) operation. This method is not amenable to all types of sound sources and does not capture temporal correlation between adjacent grains.

[0010] Corpus-based concatenative synthesis (CBCS) methods are based on selecting grains from a corpus of sound segments that are sampled from a database of heterogeneous sound sources. They utilize descriptors that are associated to sound segments to organize the corpus and perform searches within the descriptor space to pick the next grain. Note that the concept of a descriptor is not limited to features of audio signal (see, e.g., reference [4]). A user can annotate grains with perceptual descriptors when a direct mapping between desired effect and feature in the audio signal is not possible.

[0011] The descriptor space is multi-dimensional with the number of dimensions being equal to the number of descriptors. Search for the appropriate grain is performed in a computationally efficient manner by utilizing weighted Euclidean distance between a target descriptor location (e.g., point or area) in the descriptor space (hereafter referred to as “target descriptor coordinate”) and grain locations in the descriptor space. Reference [5] proposes warping functions for the distance measure to better select the set of grains and also to avoid repetitions of previously rendered grains.

[0012] For efficient search in the descriptor space, kD-tree search is used. Either k- nearest neighbors of the target descriptor coordinate or grains that are within a radius ‘r’ from the target descriptor coordinate are chosen. In reference [6], the corpus is organized as zones so that grains from different zones are not picked consequently when k-nearest neighbor search is used.

[0013] The software CATERPILLAR (see Reference [7]) performs concatenative synthesis in an offline setup where a sequence of target descriptors is given. The program uses Viterbi algorithm to identify the sequence of grains to match the target descriptors. The cost function is a combination of distance from target descriptor coordinate and concatenation cost which is based on similarity of consecutive grains.

[0014] CataRT on the other hand is a real-time system and so it chooses the subsequent grain at random from a set of grains that are the k-nearest neighbors or a radius with the target descriptor coordinate being the center (see, e.g., reference [8]).

[0015] For smoother transitions in granularly synthesized sound, reference [9] uses feature descriptors like pitch, loudness, spectral centroid, fundamental frequency, periodicity, and autocorrelation coefficient at lag 1. Feature descriptors are computed for every grain and correlation among these feature descriptors is captured using a Gaussian Mixture Model (GMM) from which grains are sampled for synthesis. Reference [10] discusses granular synthesis where the next grain is picked based on feature descriptors of the current grain for a continuity in timbre. A kD-tree search is performed to select the candidate grains closest in Euclidean distance to the current grain in the feature descriptor space.

[0016] Granular synthesis is used in a different method but in the context of rendering acoustic textures of extended sources in virtual scenes (see reference [11]). The described method numerically solves wave/sound propagation equation for the entire virtual scene with sound source pulses distributed over the acoustic source, for example, a river, to compute event loudness density (ELD). ELD is obtained by deconvolving the sound pressure at location x over many time steps using the source pulse to compute the aggregate impulse response at a given location. Then, the extracted time-varying impulse response is used to accumulate the ELD statistics, which is the histogram of loudness of peaks in the impulse response. The desired grain density (e.g., grains per second) at a given listener position is obtained by scaling the total number of events at that position. Then the grain amplitude is drawn from the ELD as a distribution. This is done for all sound sources in each scene for a given listener position.

[0017] Granular synthesis is also discussed in relation with haptics and gestures. A change in the user’s gesture or touch is mapped to a change in the feature of audio thereby leading to interactive audio rendering. For example, in reference [12], the amplitude of sound is manipulated based on user input. The haptic input-force and energy of audio signal are related quadratically. In reference [13], audio data is used as an input for haptic effects.

SUMMARY

[0018] Certain challenges presently exist. For instance, in existing systems, the method of choosing the k closest neighbors of a target descriptor coordinate in the descriptor space is fixed and not adaptive, and this can lead to spurious outputs when the neighborhood is not compact and the granular database consists of clusters that do not have similar sizes. While in some systems (e.g., CataRT) a user can adaptively choose the neighborhood size, it is still limited to 3-D beyond which visualization is not possible and therefore the user cannot adjust it. Also, the grains in the neighborhood of the target descriptor coordinate are selected uniformly at random without making a distinction between most and least likely grains to be played. Accordingly, the conventional systems may not produce natural sounding output, especially when a user input does not stay long enough at a target descriptor coordinate. In such a case, it is desirable to render the most representative grain closest to the target descriptor coordinate than choose at random. Additionally, while some systems (e.g., CATERPILLAR) use a Viterbi algorithm to find the next grain that minimizes a cost function describing the discontinuity introduced by the concatenation of the next grain, this optimization serves to minimize the discontinuity but does not avoid unnatural sounding results due to repeating the same grain, or a pattern of grains.

[0019] Accordingly, in one aspect there is provided a method for rendering audio corresponding to an audio recording, wherein the audio recording is divided into a plurality of grains. In one embodiments, the method includes obtaining a first target descriptor coordinate, wherein the first target descriptor coordinate identifies a first location in an N- dimensional descriptor space, where N > 0. The method also includes defining a first set of candidate grains based on the first target descriptor coordinate, wherein the first set of candidate grains comprises kl of the plurality of grains, where kl > 1. The method also includes assigning a final weight to each grain in the first set of candidate grains. The method also includes randomly selecting a grain from the first set of candidate grains based on the assigned final weights such that the probability that a given grain in the first set of candidate grains is selected is a function of the final weight assigned to the given grain. The method further includes rendering the selected grain.

[0020] In another embodiment, the method includes obtaining a first target descriptor coordinate, wherein the first target descriptor coordinate identifies a first location in an N- dimensional descriptor space, where N > 0 and defining a first set of candidate grains based on the first target descriptor coordinate, wherein the first set of candidate grains comprises kl of the plurality of grains, where kl > 1, and the first set of candidate grains comprises a first grain and a second grain. The method also includes assigning a final weight to each grain in the first set of candidate grains, wherein assigning a final weight to each grain in the first set of candidate grains comprises assigning a first final weight to the first grain and assigning a second final weight to the second grain. The method also includes selecting a grain from the first set of candidate grains based on the assigned final weights and rendering the selected grain, wherein the first final weight assigned to the first grain is: a function of the distance between the first target descriptor coordinate and the first grain, a function of the amount of time that has elapsed since the first grain was last rendered, a function of a trajectory associated with the first grain and a target trajectory, and/or a function of a measure of a closeness between the first grain and the most recently rendered grain (e.g., a time difference indicating a difference between a timestamp for the first grain and a timestamp for the most recently rendered grain assuming both grains were extracted from the same recording or the same segment).

[0021] In another aspect there is provided a computer program comprising instructions which when executed by processing circuitry of an apparatus causes the apparatus to perform any of the methods disclosed herein. In one embodiment, there is provided a carrier containing the computer program wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer readable storage medium. In another aspect there is provided an apparatus that is configured to perform the methods disclosed herein. The apparatus may include memory and processing circuitry coupled to the memory.

[0022] An advantage of the embodiments disclosed herein is that they facilitate granular synthesis that generates natural progressions of grains given real-time target descriptor coordinates as input. By using a weighted selection (i.e., a selection process that uses weights assigned to grains to make the grain selection), a combination of exact control and natural variation of the sound can be achieved. The complexity of the solution is low, which enables the rendering of many granular sound sources also on devices with low computational power. Low real-time complexity also enables the solution to produce reactive and responsive sounds without delays.

BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate various embodiments.

[0024] FIG. 1 illustrates a system according to an embodiment.

[0025] FIG. 2A illustrates an example two-dimensional descriptor space.

[0026] FIG. 2B illustrates an example two-dimensional descriptor space.

[0027] FIG. 3 A illustrates an example two-dimensional descriptor space.

[0028] FIG. 3B illustrates a process for determining an optimal k value for a given grain according to an embodiment.

[0029] FIG. 4 A illustrates an example audio signal.

[0030] FIG. 4B illustrates an example schedule of grains.

[0031] FIG. 4C illustrates an example schedule of grains according to an embodiment. [0032] FIG. 5 is a flowchart illustrating a process according to an embodiment.

[0033] FIG. 6 is a flowchart illustrating a process according to an embodiment.

[0034] FIGS. 7A and 7B show a system according to some embodiments.

[0035] FIG. 8 is a block diagram of an apparatus according to some embodiments.

DETAILED DESCRIPTION

[0036] FIG. 1 illustrates a system 100, according to some embodiments, for performing granular synthesis. System 100 includes a grain extraction unit 102 which extracts grains from an original audio recording 111. That is, grain extraction unit divides the original audio recording into small fragments, called “grains.” The extracted grains are stored in a grain database 104 that is accessed at rendering time by a grain scheduling unit 106 (a.k.a., grain selection unit), which is a component of a granular rendering unit 108. In some embodiments, there is one grain database per procedural audio source, each of which is available to the rendering unit 108. Each grain stored in grain database is associated with one or more vectors of one or more descriptor values, each vector corresponding to a particular descriptor.

[0037] When creating a grain database 104, an audio designer decides what aspects should be used as descriptors. In some cases, it might be features of the sound itself, such as pitch or loudness, but it could also be other aspects that relate to how the sound was generated, such as the speed of movement that generates a contact sound between two objects sliding against each other or the opening angle of a door that generates a screeching sound when opened and closed. The descriptors should be chosen so that the sound can be re-generated dynamically by the Tenderer given a target descriptor coordinate or trajectory.

[0038] As noted above, each grain stored in grain database is associated with one or more descriptor values. Accordingly, the grains of an original recording need to be annotated with the descriptor values. In the case that a descriptor is an audio feature, the descriptor value may be possible to measure directly from the audio signal itself. In other cases, the descriptor values need to be provided somehow as extra metadata of the recordings. This may be, for example, done by logging data from some sensors during the recording and providing this data in companion files. In some cases, the annotation can be done manually by creating a log of data that describes how a descriptor changes during the recording or it can be done manually for each extracted grain. [0039] When extracting grains from the original recording(s), the descriptor values are stored as metadata for each grain. Using the descriptor values, each grain can be positioned in a multi-dimensioned descriptor space where the value of each descriptor describes a position along one axis within this space. If only one descriptor is used, the descriptor space is one-dimensional (ID), but if more descriptors are used the dimensionality of the descriptor space increases. An example two-dimensional (2D) descriptor space is shown in FIG. 2A, where each circle represents a grain. As shown in FIG. 2A, each grain has a location (e.g., a point or area) within the 2D descriptor space, this location is referred to as the grain coordinate.

[0040] In one embodiment, the descriptor metadata for a sequence of grains extracted from one recording describes a trajectory within the descriptor space, which corresponds to how the descriptors evolved during the original recording.

[0041] At rendering time, the scheduling of the grains (i.e., the selection of grains to render) is based on a target descriptor coordinate in the descriptor space. The target descriptor coordinate specifies what descriptor values the generated sound output should have, which means that grains close to that coordinate in the descriptor space are to be used most prominently. A target descriptor coordinate may come from many types of sources, such as a physics engine simulating the interaction of virtual bodies, live input parameters from hand controllers or other sensors, pre-defined automation parameters.

[0042] An important aspect of granular synthesis rendering is that repetition of the same grain often sounds very unrealistic and artificial. If the grains are short, less than 50ms, repeating the same grain will result in a very metallic and static sound. If the grains are longer, the repetition will be heard like a repeating pattern, which often results in a sound that is not plausible.

[0043] The scheduling of grains needs to avoid repetition of the same grain but at the same time select grains that are close to the target descriptor coordinate in the descriptor space. This disclosure, therefore, uses a weighted selection (e.g., a weighted random selection) procedure that will generate ever evolving sequences of grains (i.e., an ordered set of grains) that closely follow the target descriptor coordinates. An example of a sequence of grains is: [grain-7, grain-6, grain-7, grain-9, grain-11, grain-10],

[0044] Each grain may be assigned a predefined weight as well as a set of dynamic weights that may change over time. The predefined weight can be useful in cases where a grain is an outlier that should not be used too often but can add a realistic variation to the generated sound if used every now and then. Another use case is to use predefined weights to control the frequency of grains that represent e.g., bird chirps compared to grains that represent the background sound of a forest.

[0045] An input (e.g., a signal generated by a user interaction) that controls a procedural audio source is mapped to a target descriptor coordinate. Based on the target descriptor coordinate, one or more grains from a set of candidate grains are selected for rendering using a weighted selection (e.g., a weighted random selection or a selection where the grain with highest weight is selected). The selected grains are rendered using standard granular synthesis methods where metadata regarding overlap percent and crossfade window are specified by the sound designer beforehand.

[0046] The input that controls the procedural audio source can change in real-time; consequently, the target descriptor coordinate can change over time as the input changes (the target descriptor coordinate can also change over time even if the input does not change).

[0047] In one embodiment, the grain scheduling algorithm using weighted selection has the following steps.

[0048] Step 1 : Obtain a target descriptor coordinate (e.g., map an input, such as a user input or other input, to a target descriptor coordinate in a descriptor space).

[0049] Step 2: Determine the size of a neighborhood adaptively, e.g., calculate a k value depending on the target descriptor coordinate or calculate a radius value (r) depending on the target descriptor coordinate. Alternatively, obtain a pre-calculated value of k or radius from metadata of the grain database.

[0050] Step 3 : Select a set of candidate grains from the database using the k value or radius value. For example, select the grains from the database that are the k-nearest neighbors of the target descriptor coordinate. This search can be performed using off-the- shelf computationally efficient algorithms like the kD-tree search. As another example, include in the set of candidate grains each grain having a grain coordinate that is within a distance of r from the target descriptor coordinate.

[0051] Step 4: Assign a final weight (a.k.a., probability value) to each one of the grains in the set of candidate grains. The final weight assigned to a given grain may be based on: [0052] i) the distance of the position of the grain in the descriptor space (i.e., the grain coordinate) from the target descriptor coordinate in the descriptor space,

[0053] ii) the difference in the trend of descriptors of a grain, compared to the target descriptor trajectory,

[0054] iii) the temporal history of previously used grains,

[0055] iv) the difference in time instant in the original recording between the grain and the previously used grain, if they are from the same recording, and/or

[0056] v) predefined probability weight of the grain in the database.

[0057] Step 5: Perform a weighted selection of grains from the set of candidate grains using the final weights assigned in step 4. For example, perform a weighted random selection, or, as another example, select the grain with the highest final weight or lowest final weight. In this manner, grains are selected based on the target descriptor coordinate and further based on the final weights assigned to the grains in the set of candidate grains.

[0058] If the target descriptor coordinate changes, the steps are repeated. Otherwise, the weighted selection of grains continues (step 4 onwards) with changes made to the weights on the basis of temporal history of previous grains.

[0059] Step 2 - Determination of value of k.

[0060] Conventionally, the value of k is a user-defined constant. It is not desirable, however, to keep the value of k fixed at all times because doing so could lead to choosing too few or too many grains which in turn could lead to under-utilization of the grains or scheduling grains that are dissimilar to the target descriptor value, respectively.

[0061] Accordingly, this disclosure provides, in one embodiment, an adaptive choice of k based on the density of grains available in an area surrounding the target descriptor coordinate.

[0062] An example of why one should use different values of k for different target descriptor coordinates is illustrated in FIG. 2 A and 2B. In FIG. 2 A, the target descriptor coordinate is close to a cluster of 3 grains, whereas in FIG. 2B, the target descriptor coordinate is in the vicinity of more grains. In both these scenarios, it is not optimal to use the same value of k. For the case of FIG. 2A, k=3 is appropriate. If k > 3, then this will lead to choosing grains that are not in the cluster and therefore lead to a discontinuity in the texture of rendered sound. In FIG. 2B, if k = 3, then too few grains are selected. A larger value here will lead to richer textures with less repetition since a variety of grains can be selected.

[0063] The value of k should be chosen adaptively based on the target descriptor coordinate in the descriptor space. In one embodiment, each grain in the database is assigned an optimum k value. Then, for a target descriptor coordinate, the value of k is set equal to the optimal k value assigned to the grain that is closest to the target descriptor coordinate.

[0064] The assignment of optimal k per grain in the database can be performed offline or during the construction of the grain database. Either all distances from grain i to other grains in the database are recorded or there can be a threshold on the maximum number of neighbors to stop the distance computation.

[0065] Then the distances are sorted from lowest to highest. The difference between distances for consecutive values of k will have sudden jump at a value where the distance increases drastically. This is treated as a cut-off value for k and k + 1 is assigned to the grain. The addition of 1 is to include the grain itself in the value of k.

[0066] A criterion to determine the cut-off value would be to either set an absolute threshold on the difference in sorted distances between consecutive neighbors or to use normalized percentage increases in consecutive sorted distances.

[0067] Alternatively, one could also use the concept of adaptive radius to choose the set of candidate grains. The current method in literature involves using a fixed radius with target descriptor coordinate as the center of a circle and choose all the grains within this circle of fixed radius to be included in the set of candidate grains. By the same reasoning as above, it might be beneficial to change the radius adaptively based on the position of the target descriptor coordinate. There, instead of choosing different value of k, one would use different values of radius.

[0068] FIG. 3 A illustrates the computation of k for a certain grain 301 represented by the black circle. In FIG. 3A, grain 301 and its 6 corresponding neighbors are shown. In FIG. 3B, the sorted distances are shown for grain 301. Because the jump in distance values is observable for k=4, an optimum k value of 5 assigned to grain 301.

[0069] Step 3 - Weight Assignments

[0070] Each of the factors that influence the final weight assigned to a grain are described below. The target descriptor is denoted as u and weight (a.k.a., probability) associated with grain i as p_t. The descriptor index is denoted by j = 1,2, ... , D where D is the number of descriptors or dimensions of the descriptor space. The descriptor value of grain i at dimension j is given and that of the target as Uj.

[0071] The criteria below that influence the probability of choosing a grain are expressed as proportional relationships since the final value of the weight is obtained after normalizing i.e., ensuring that the weights corresponding to all k grains add to 1.

[0072] i) Distance from target descriptor coordinate

[0073] A distance metric is used to define the proximity between target descriptor coordinate and other descriptor coordinates corresponding to grains. An example of the distance metric is a weighted Euclidean distance where the difference in coordinates in each dimension is weighted by the inverse of standard deviation of the corresponding descriptor values,

[0074] The closer a grain’s descriptor coordinate to the target descriptor coordinate, the higher the weight associated. Let be the distance from grain i to the target descriptor coordinate. For example, the probability that grain i is chosen can be inversely proportional to the distance. P; oc l/d₍. Accordingly, one can set p₍₁ = ' ldi.

[0075] In some cases, the different descriptors should not have the same amount of influence on the grain selection. For example, if one descriptor is the pitch of the sound and another is a descriptor that has less strong effect on the perceptual character of the sound, the distance in the dimension corresponding to the pitch may be given a higher weight than the distance in the dimension that corresponds to the other descriptor. This can be achieved by adding an extra variable weight, m_;-,to each dimension when calculating the distance:

[0076] ii) Difference in trend of target and original descriptor trajectories

[0077] When performing grain extraction, the descriptor coordinates are used as the main selection criterion. But the trend of the original descriptor trajectory also gives important information about the grain. For example, if an engine sound is modelled with a granular database with one descriptor that denotes the RPM (revolutions per minute) of the engine, the trend of the descriptor corresponds to the acceleration or deceleration of the engine at the time instant in the recording that the grain was extracted from. A grain that was extracted from a portion of the recording when the engine was accelerating will have a pitch that is slightly lower at the start than at the end and will therefor fit best when the desired output is the sound of an accelerating engine.

[0078] In the more general case, where a multi-dimensional descriptor space is used, the descriptor trend is a vector that corresponds to the direction of the trajectory that describes how the descriptors were changing at the time of the original recording. Similarly, the trend of the target descriptor trajectory describes the direction that the target descriptor coordinate is moving in the descriptor space.

[0079] The trend describes both the direction and the rate of change. Referring again to the example of the engine, if the granular database includes grains that correspond to the same RPM but with different acceleration, the grains that correspond to a similar acceleration as that of the target descriptor trajectory should be preferred.

[0080] FIG. 4 A shows a descriptor trajectory of an original recording of an engine sound where the descriptor is the RPM of the engine. The recording is divided into 15 grains. FIG. 4B shows an example of the how grains may be selected to match a target descriptor trajectory where the descriptor trend of the grains is not considered during grain selection. As can be seen, grains with both increasing and decreasing RPM are used in combination. The resulting descriptor trajectory shows an irregular behavior which may result in a degradation in perceived quality, especially if the descriptor represents the pitch of the sound. In FIG. 4C, the grain selection also considers the descriptor trend so that only grains with decreasing RPM are used, which would result in a smoother sound.

[0081] Considering the descriptor trend during grain scheduling is extra important for sound sources where the character is different for different trends. For example, an engine may sound different when accelerating compared to when decelerating. Making sure to match the descriptor trend avoids problems where grains with different character are used together.

[0082] The descriptor trend of a grain can be calculated as the difference in descriptor value at the end of the grain as compared to the start of the grain divided by the duration of the grain. In the case of a multi-dimensional descriptor space the trend is a vector that describes the mean rate of change in descriptor coordinates during the grain in the original recording, for example with three descriptors the trend, to, of a grain would be a three- dimensional vector:

[0083] When scheduling grains at rendering time, the trend of the target descriptor trajectory can be calculated similarly as the difference in descriptor coordinates since they were last updated divided by the time elapsed since they were last updated, Tu.

[0084] The difference in trend can then calculated as

[0085] The weight assigned to a grain can then be calculated as a function of the norm of ti>, for example: p_i2 = max (1.0, ■ — ■), where mi is a variable that controls the how the probability decreases with increased difference in descriptor trend.

[0086] iii) Temporal history of previously used grains

[0087] A history of previously rendered grains is kept for a certain time window so as to not repeat them and decrease the probability of choosing the grain if it was already rendered. The decrease in probability is directly proportional to a function of the difference between the current time instant t and the last time instant at which the grain i was rendered t^h

[0088] If grain i has not been selected in the past or in a certain time window, the value of last time instant is set to zero, t-¹ = 0. The function f can be linear, quadratic, logarithmic or any monotonically increasing function of the argument t — t-¹ with nonnegative output.

[0089] iv) Difference in time instant in original recording

[0090] For some sound sources, the way that the sound evolves may not be completely described by the changes in descriptors. Sometimes sound evolves in a way that depends on what happened earlier. For example, the screeching sound of an old door may have a slightly different screeching sound every time it is opened, even if the opening of it is done at the same speed etc. In these cases, the sound of grains with similar descriptor values and descriptor trend may sound very different and combining them may result in unnatural discontinuities that were not there in the original recording. A way to avoid these discontinuities is to assign a higher probability to grains that come from the same part of the original recording as the grain that was used previously. Grains that came from the same part of the original recording are expected to be closely related and resemble each other in character and are therefore good candidates when selecting the next grain.

[0091] In order to measure how close one grain from a particular recording is to another grain from the same recording, a time difference can be calculated which corresponds to the difference in time instant in the recording that the two grains were extracted from. If the grains were close, this time difference is small. To calculate the time difference between two grains, metadata that tells from which original recording each grain was extracted and at what time instant can be used. This metadata, therefore, provides a measure of closeness between the two grains. This metadata can be specified in a compact way as two values: a recording identifier assigned to the recording of origin and a timestamp that identifies the time instant in that recording at which the grain can be found.

[0092] A weight for a grain can then be calculated based on the difference timestamps in a way that a smaller difference gives a higher probability to choose the evaluated grain.

For example, the weight, /?,, of grain z could be calculated as

[0093] Where R, is the recording identifier assigned to the recording from which grain z was extracted, Ro is the recording identifier assigned to the recording from which the previously rendered gain was extracted, h and to are the respective timestamps for the two grains, and b is a design constant that sets the probability to use a grain that comes from another recording. The function f() takes the difference in time instant as input and calculates a probability for the grain. In one embodiment function f decreases linearly with an increase in difference in time instant with a slope specified with a variable a. t 1.0 — b

> 11.0 — alG

Pi4 — j

I

[0094] This has the effect that the probability reduces from 1.0 to b as the difference between the respective timestamps increases but never goes below b. [0095] In one embodiment the recording identifier A, can also be set to refer to segments of a recording, i.e., one recording can be divided into segments where each segment has its own index. This can be useful when one recording contain segments that are not to be seen as related by the Tenderer.

[0096] Final Weight

[0097] The final weight assigned to grain z (i.e., pi) is calculated by accumulating the weights assigned to grain z from each stage, i.e., p_L = PuPi2Pi3Pi4Pis, where p,s is the predetermined weight mentioned above. In some embodiments, not all five weights are needed and can then be skipped by setting the corresponding weight to 1.0 or by completely exclude it from the calculation.

[0098] In one embodiment, the different weights are given different levels of influence on the final weight by modifying the individual weights with a fractional exponent, such as e.g.,

Pi = Pu^1/2Pi2Pi3Pi4^1/4Pi ,

[0099] where the weight p_ti from the first stage is made less influential by using a fractional exponent of Yi and the weight p,4 is made even less influential by using a fractional exponent of 1/4.

[00100] After the weights p_L are computed for all the grains in the set of candidate grains from the steps above, they have to be normalized, i.e., each of them divided by the sum.

Pt <- ¹ where k is the size of the set of candidate grains.

[00101] Then grains are selected based on their final weight (e.g., by sampling from the distribution). It is noted that the temporal history of a grain and the descriptor trend influence the final weight at every time instant a grain is rendered when the target descriptor coordinate does not change. Therefore, as long as the target descriptor remains constant, if at time instant t grain z is chosen, at time t+h, the final weight of that grain is influenced by temporal history and descriptor trend. The final weight of grain z becomes

[00102] To avoid the repetition of the grain i at t + 1, we can define (/i) as

[00103] However, due to normalization operation, the probabilities of the other k — 1 grains change as well.

[00104] FIG. 5 is a flow chart illustrating a process 500, according to an embodiment, for rendering audio corresponding to an audio recording, wherein the audio recording is divided into a plurality of grains. Process 500 may begin in step s502.

[00105] Step s502 comprises obtaining a first target descriptor coordinate, wherein the first target descriptor coordinate identifies a first location in an N-dimensional descriptor space, where N > 0.

[00106] Step s504 comprises defining a first set of candidate grains based on the first target descriptor coordinate, wherein the first set of candidate grains comprises kl of the plurality of grains, where kl > 1.

[00107] Step s506 comprises assigning a final weight to each grain in the first set of candidate grains.

[00108] Step s508 comprises randomly selecting a grain from the first set of candidate grains based on the assigned final weights such that the probability that a given grain in the first set of candidate grains is selected is a function of the final weight assigned to the given grain.

[00109] Step s510 comprises rendering the selected grain.

[00110] FIG. 6 is a flow chart illustrating a process 600, according to an embodiment, for rendering audio corresponding to an audio recording, wherein the audio recording is divided into a plurality of grains. Process 600 may begin in step s602.

[00111] Step s602 comprises obtaining a first target descriptor coordinate, wherein the first target descriptor coordinate identifies a first location in an N-dimensional descriptor space, where N > 0.

[00112] Step s604 comprises defining a first set of candidate grains based on the first target descriptor coordinate, wherein the first set of candidate grains comprises kl of the plurality of grains, where kl > 1, and the first set of candidate grains comprises a first grain and a second grain. [00113] Step s606 comprises assigning a final weight to each grain in the first set of candidate grains, wherein assigning a final weight to each grain in the first set of candidate grains comprises assigning a first final weight to the first grain and assigning a second final weight to the second grain. The first final weight assigned to the first grain is: a function of the distance between the first target descriptor coordinate and the first grain, a function of the amount of time that has elapsed since the first grain was last rendered, a function of a trajectory associated with the first grain and a target trajectory, and/or a function of a measure of a closeness between the first grain and the most recently rendered grain.

[00114] Step s608 comprises selecting a grain from the first set of candidate grains based on the assigned final weights.

[00115] Step s610 comprises rendering the selected grain.

[00116] Example Use Case

[00117] FIG. 7A illustrates a XR system 700, according to one embodiment, in which the embodiments disclosed herein may be applied. As shown in FIG. 7A, XR system 700 comprises an XR headset 720 (e.g., XR goggles, XR glasses, XR head mounted display (HMD), etc.) that is configured to be worn by a user and that is operable to display to the user an XR scene (e.g., an XR scene in which the user is virtually immersed), speakers 734 and 735 for producing sound for the user, and an input device 750 for receiving input from the user (in this example the input device 750 is in the form of a joystick).

[00118] FIG. 7B illustrates that XR headset 720 may comprise an orientation sensing unit 721, a position sensing unit 722, and an XR rendering device 724. In this embodiment, XR rendering device 724 includes an audio Tenderer that includes grain database 104 and granular rendering unit 108.

[00119] Orientation sensing unit 721 is configured to detect a change in the orientation of the user and provides information regarding the detected change to XR rendering device 724. In some embodiments, XR rendering device 724 determines the absolute orientation (in relation to some coordinate system) given the detected change in orientation detected by orientation sensing unit 721. In some embodiments, orientation sensing unit 721 may comprise one or more accelerometers and/or one or more gyroscopes.

[00120] In addition to receiving input from sensing units 721 and 722, XR rendering device 724 may also receive input from input device 750 and may also obtain XR scene configuration information (e.g., the grain metadata). Based on these inputs and the XR scene configuration, XR rendering device 724 renders an XR scene in real-time for the user. That is, in real-time, XR rendering device produces XR content, including, for example, video data that is provided to a display driver 726 so that display driver 726 will display on a display screen 727 images included in the XR scene and audio data that is provided to speaker driver 728 so that speaker driver 728 will play audio for the using speakers 734 and 735 (the audio data or portion thereof may be generated by granular rendering unit 108). While XR rendering device 724 is shown as being within XR headset 720 in this embodiment, in other embodiments XR rendering device 724 (or one or more components thereof, such as grain database 104 and granular rendering unit 108) are located remotely from XR headset 720, in which case XR headset 720 and XR rendering device 724 have communication means (transmitter, receiver) for enabling XR rendering device 724 to transmit XR content to XR headset 720 (e.g., XR rendering device or components thereof may be implemented in the “cloud”).

[00121] FIG. 8 is a block diagram of an XR rendering device 724, according to some embodiments, for performing the methods disclosed herein. As shown in FIG. 8, XR rendering device 724 may comprise: processing circuitry (PC) 802, which may include one or more processors (P) 855 (e.g., one or more general purpose microprocessors and/or one or more other processors, such as an application specific integrated circuit (ASIC), field- programmable gate arrays (FPGAs), and the like), which processors may be co-located in a single housing or in a single data center or may be geographically distributed (i.e., XR rendering device 724 may be a distributed computing apparatus); at least one network interface 848 (e.g., a physical interface or air interface) comprising a transmitter (Tx) 845 and a receiver (Rx) 847 for enabling XR rendering device 724 to transmit data to and receive data from other nodes connected to a network 110 (e.g., an Internet Protocol (IP) network) to which network interface 848 is connected (physically or wirelessly) (e.g., network interface 848 may be coupled to an antenna arrangement comprising one or more antennas for enabling XR rendering device 724 to wirelessly transmit/receive data); and a storage unit (a.k.a., “data storage system”) 808, which may include one or more non-volatile storage devices and/or one or more volatile storage devices (as illustrated in FIG. 8, grain database (DB) may be stored in storage unit 808). In embodiments where PC 802 includes a programmable processor, a computer readable storage medium (CRSM) 842 may be provided. CRSM 842 may store a computer program (CP) 843 comprising computer readable instructions (CRI) 844. CRSM 842 may be a non-transitory computer readable medium, such as, magnetic media (e.g., a hard disk), optical media, memory devices (e.g., random access memory, flash memory), and the like. In some embodiments, the CRI 844 of computer program 843 is configured such that when executed by PC 802, the CRI causes XR rendering device 724 to perform steps described herein (e.g., steps described herein with reference to the flow charts). In other embodiments, XR rendering device 724 may be configured to perform steps described herein without the need for code. That is, for example, PC 802 may consist merely of one or more ASICs. Hence, the features of the embodiments described herein may be implemented in hardware and/or software.

[00122] Summary of Various Embodiments

[00123] Al. A method for rendering audio corresponding to an audio recording, wherein the audio recording is divided into a plurality of grains, the method comprising: obtaining a first target descriptor coordinate, wherein the first target descriptor coordinate identifies a first location in an N-dimensional descriptor space, where N > 0; defining a first set of candidate grains based on the first target descriptor coordinate, wherein the first set of candidate grains comprises kl of the plurality of grains, where kl > 1; assigning a final weight to each grain in the first set of candidate grains; randomly selecting a grain from the first set of candidate grains based on the assigned final weights such that the probability that a given grain in the first set of candidate grains is selected is a function of the final weight assigned to the given grain; and rendering the selected grain.

[00124] A2. The method of embodiment Al, wherein each one of the plurality of grains is associated with a grain coordinate (e.g., a set of one or more descriptor values) that identifies a location of the grain in the N-dimensional descriptor space, and the first set of candidate grains is defined based on the first target descriptor coordinate and the grains’ coordinates.

[00125] A3. The method of embodiment A2, wherein defining the first set of candidate grains based on the first target descriptor coordinate and the grains’ coordinates comprises determining a nearest-neighbor set of grains consisting of kl of the plurality of grains, wherein none of the plurality of grains that are not included in the nearest-neighbor set of grains is closer to the first target descriptor coordinate than any one of the grains included in the nearest-neighbor set of grains, and the candidate set of candidate grains consists of the grains included in the nearest-neighbor set of grains. [00126] A4. The method of embodiment A3, further comprising determining kl based on the number of grains in the plurality of grains that have a grain coordinate that is within a threshold distance of the first target descriptor coordinate.

[00127] A5. The method of embodiment A3, wherein each grain included in the plurality of grains is assigned an optimal k value, and the method comprises setting kl equal to the optimal k value assigned to the grain within the plurality of grains having a grain coordinate that is closest to the target descriptor coordinate.

[00128] A6. The method of embodiment A2, wherein defining the first set of candidate grains based on the first target descriptor coordinate and the grains’ coordinates comprises: determining a first radius value, rl; and including in the first set of candidate grains each one of the plurality of grains that has a grain coordinate that is within a distance of rl from the first target descriptor coordinate.

[00129] A7. The method of embodiment A6, further comprising determining rl based on the number of grains in the plurality of grains that have a grain coordinate that is within a threshold distance of the first target descriptor coordinate.

[00130] A8. The method of any one of embodiments A1-A7, wherein assigning a final weight to each grain in the first set of candidate grains comprises: assigning a first weight to a first grain included in the set of candidate grains; determining a first final weight based on the first weight; and assigning the first final weight to the first grain.

[00131] A9. The method of embodiment A8, wherein first weight is: a function of the distance between the first target descriptor coordinate and the first grain, a function of the amount of time that has elapsed since the first grain was last rendered, a function of a trajectory associated with the first grain and a target trajectory, or a function of a measure of a closeness between the first grain and the most recently rendered grain (e.g., a time difference indicating a difference between a timestamp for the first grain and a timestamp for the most recently rendered grain assuming both grains were extracted from the same recording or the same segment).

[00132] A10. The method of any one of embodiments A1-A9, further comprising: after randomly selecting a grain from the first set of candidate grains based on the assigned final weights, assigning a new final weight to the rendered grain or removing the rendered grain from the first set of candidate grains; after assigning a new final weight to the rendered grain or removing the rendered grain from the first set of candidate grains, randomly selecting another grain from the first set of candidate grains based on the currently assigned final weights; and rendering the selected another grain.

[00133] Al l. The method of any one of embodiments A1-A9, further comprising: after randomly selecting a grain from the first set of candidate grains, obtaining a second target descriptor coordinate; defining a second set of candidate grains based on the second target descriptor coordinate, wherein the second set of candidate grains comprises k2 of the plurality of grains, where k2 > 1; assigning a final weight to each grain in the second set of candidate grains; randomly selecting a grain from the second set of candidate grains based on the assigned final weights such that the probability that a given grain in the second set of candidate grains is selected is a function of the final weight assigned to the given grain; and rendering the grain randomly selected from the second set of candidate grains.

[00134] Bl. A method for rendering audio corresponding to an audio recording, wherein the audio recording is divided into a plurality of grains, the method comprising: obtaining a first target descriptor coordinate, wherein the first target descriptor coordinate identifies a first location in an N-dimensional descriptor space, where N > 0; defining a first set of candidate grains based on the first target descriptor coordinate, wherein the first set of candidate grains comprises kl of the plurality of grains, where kl > 1, and the first set of candidate grains comprises a first grain and a second grain; assigning a final weight to each grain in the first set of candidate grains, wherein assigning a final weight to each grain in the first set of candidate grains comprises assigning a first final weight to the first grain and assigning a second final weight to the second grain; selecting a grain from the first set of candidate grains based on the assigned final weights; and rendering the selected grain, wherein the first final weight assigned to the first grain is: a function of the distance between the first target descriptor coordinate and the first grain, a function of the amount of time that has elapsed since the first grain was last rendered, a function of a trajectory associated with the first grain and a target trajectory, and/or a function of a measure of a closeness between the first grain and the most recently rendered grain.

[00135] B2. The method of embodiment Bl, wherein each one of the plurality of grains is associated with a grain coordinate (e.g., a set of one or more descriptor values) that identifies a location of the grain in the N-dimensional descriptor space, and the first set of candidate grains is defined based on the first target descriptor coordinate and the grains’ coordinates. [00136] B3. The method of embodiment B2, wherein defining the first set of candidate grains based on the first target descriptor coordinate and the grains’ coordinates comprises determining a nearest-neighbor set of grains consisting of kl of the plurality of grains, wherein none of the plurality of grains that are not include in the nearest-neighbor set of grains is closer to the first target descriptor coordinate than any one of the grains included in the nearest-neighbor set of grains, and the candidate set of candidate grains consists of the grains included in the nearest-neighbor set of grains.

[00137] B4. The method of embodiment B3, further comprising determining kl based on the number of grains in the plurality of grains that have a grain coordinate that is within a threshold distance of the first target descriptor coordinate.

[00138] B5. The method of embodiment B3, wherein each grain included in the plurality of grains is assigned an optimal k value, and the method comprises setting kl equal to the optimal k value assigned to the grain within the plurality of grains having a grain coordinate that is closest to the target descriptor coordinate.

[00139] B6. The method of embodiment B2, wherein defining the first set of candidate grains based on the first target descriptor coordinate and the grains’ coordinates comprises: determining a first radius value, rl; and including in the first set of candidate grains each one of the plurality of grains that has a grain coordinate that is within a distance of rl from the first target descriptor coordinate.

[00140] B7. The method of embodiment B6, further comprising determining rl based on the number of grains in the plurality of grains that have a grain coordinate that is within a threshold distance of the first target descriptor coordinate.

[00141] B8. The method of any one of embodiments B1-B7, further comprising: after selecting a grain from the first set of candidate grains based on the assigned final weights, assigning a new final weight to the rendered grain or removing the rendered grain from the first set of candidate grains; after assigning a new final weight to the rendered grain or removing the rendered grain from the first set of candidate grains, selecting another grain from the first set of candidate grains based on the currently assigned final weights; and rendering the selected another grain.

[00142] B9. The method of any one of embodiments B1-B8, further comprising: after selecting a grain from the first set of candidate grains, obtaining a second target descriptor coordinate; defining a second set of candidate grains based on the second target descriptor coordinate, wherein the second set of candidate grains comprises k2 of the plurality of grains, where k2 > 1; assigning a final weight to each grain in the second set of candidate grains; selecting a grain from the second set of candidate grains based on the assigned final weights; and rendering the grain randomly selected from the second set of candidate grains.

[00143] Cl. A computer program comprising instructions which when executed by processing circuitry of an XR rendering device causes the XR rendering device to perform the method of any one of claims Al -Al 1 or Bl -BIO.

[00144] C2. A carrier containing the computer program of embodiment Cl, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer readable storage medium.

[00145] DI. An XR rendering device for rendering audio corresponding to an audio recording, wherein the audio recording is divided into a plurality of grains, the XR rendering device comprising: memory; and processing circuitry coupled to the memory, wherein the the apparatus is configured to perform a method that comprises: obtaining a first target descriptor coordinate, wherein the first target descriptor coordinate identifies a first location in an N- dimensional descriptor space, where N > 0; defining a first set of candidate grains based on the first target descriptor coordinate, wherein the first set of candidate grains comprises kl of the plurality of grains, where kl > 1; assigning a final weight to each grain in the first set of candidate grains; randomly selecting a grain from the first set of candidate grains based on the assigned final weights such that the probability that a given grain in the first set of candidate grains is selected is a function of the final weight assigned to the given grain; and rendering the selected grain.

[00146] D2. The XR rendering device of embodiment DI, wherein the XR rendering device is further configured to perform the method of any one of claims A2-A11.

[00147] El. An XR rendering device for rendering audio corresponding to an audio recording, wherein the audio recording is divided into a plurality of grains, the XR rendering device comprising: memory; and processing circuitry coupled to the memory, wherein the the apparatus is configured to perform a method that comprises: obtaining a first target descriptor coordinate, wherein the first target descriptor coordinate identifies a first location in an N- dimensional descriptor space, where N > 0; defining a first set of candidate grains based on the first target descriptor coordinate, wherein the first set of candidate grains comprises kl of the plurality of grains, where kl > 1, and the first set of candidate grains comprises a first grain and a second grain; assigning a final weight to each grain in the first set of candidate grains, wherein assigning a final weight to each grain in the first set of candidate grains comprises assigning a first final weight to the first grain and assigning a second final weight to the second grain; selecting a grain from the first set of candidate grains based on the assigned final weights; and rendering the selected grain, wherein the first final weight assigned to the first grain is: a function of the distance between the first target descriptor coordinate and the first grain, a function of the amount of time that has elapsed since the first grain was last rendered, a function of a trajectory associated with the first grain and a target trajectory, and/or a function of a measure of a closeness between the first grain and the most recently rendered grain.

[00148] E2. The XR rendering device of embodiment El, wherein the XR rendering device is further configured to perform the method of any one of claims B2-B10.

[00149] While various embodiments are described herein, it should be understood that they have been presented by way of example only, and not limitation. Thus, the breadth and scope of this disclosure should not be limited by any of the above-described exemplary embodiments. Moreover, any combination of the above-described elements in all possible variations thereof is encompassed by the disclosure unless otherwise indicated herein or otherwise clearly contradicted by context.

[00150] Additionally, while the processes described above and illustrated in the drawings are shown as a sequence of steps, this was done solely for the sake of illustration. Accordingly, it is contemplated that some steps may be added, some steps may be omitted, the order of the steps may be re-arranged, and some steps may be performed in parallel. Further, as used herein “a” means “at least one” or “one or more.”

[00151] References

[1] US20180068487A1 : Systems and methods for simulating sounds of a virtual object using procedural audio (Disney Enterprises Inc.).

[2] Farnell, Andy, "An introduction to procedural audio and its application in computer games,” Audio mostly conference Vol. 23. 2007.

[3] D. Gabor, “Theory of communication. Part 1 : The analysis of information,” Journal of the Institution of Electrical Engineers-Part III: Radio and Communication Engineering, vol. 93, no. 26, pp. 429-441, 1946. [4] D. Schwarz, "Corpus-Based Concatenative Synthesis," in IEEE Signal Processing Magazine, vol. 24, no. 2, pp. 92-104, March 2007, doi: 10.1109/MSP.2007.323274.

[5] D. Schwarz "Distance mapping for corpus-based concatenative synthesis." Sound and Music Computing (SMC) 2011.

[6] Aaron Einbond and Diemo Schwarz. " Spatializing timbre with corpus-based concatenative synthesis." International Computer Music Conference Proceedings. Vol. 2010. International Computer Music Association, 2010.

[7] D. Schwarz. "A system for data-driven concatenative sound synthesis." Digital Audio Effects (DAFx). 2000.

[8] D. Schwarz et al. "Real-time corpus-based concatenative synthesis with catart." 9th International Conference on Digital Audio Effects (DAFx). 2006.

[9] Diemo Schwarz and Norbert Schnell. "Descriptor-based sound texture sampling." Sound and music computing (SMC) 2010.

[10] Diemo Schwarz and Sean O'Leary. "Smooth granular sound texture synthesis by control of timbral similarity." Sound and Music Computing (SMC) 2015.

[11] Zechen Zhang, Nikunj Raghuvanshi, John Snyder, and Steve Marschner. 2019. Acoustic texture rendering for extended sources in complex scenes. ACM Trans. Graph. 38, 6, Article 222 (December 2019), 9 pages, https://doi.org/10.1145/3355089.3356566.

[12] Barrass, Stephen, and Matt Adcock. "Interactive granular synthesis of haptic contact sounds." Audio Engineering Society conference: 22nd international conference: virtual, synthetic, and entertainment audio. Audio Engineering Society, 2002.

[13] US20190094975A1 : Haptic Effect Conversion System Using Granular Synthesis: Immersion Corp.

Claims

1. A method (500) for rendering audio corresponding to an audio recording, wherein the audio recording is divided into a plurality of grains, the method comprising: obtaining (s502) a first target descriptor coordinate, wherein the first target descriptor coordinate identifies a first location in an N-dimensional descriptor space, where N > 0; defining (s504) a first set of candidate grains based on the first target descriptor coordinate, wherein the first set of candidate grains comprises kl of the plurality of grains, where kl > 1; assigning (s506) a final weight to each grain in the first set of candidate grains; randomly selecting (s508) a grain from the first set of candidate grains based on the assigned final weights such that the probability that a given grain in the first set of candidate grains is selected is a function of the final weight assigned to the given grain; and rendering (s510) the selected grain.

2. The method of claim 1, wherein each one of the plurality of grains is associated with a grain coordinate that identifies a location of the grain in the N-dimensional descriptor space, and the first set of candidate grains is defined based on the first target descriptor coordinate and the grains’ coordinates.

3. The method of claim 2, wherein defining the first set of candidate grains based on the first target descriptor coordinate and the grains’ coordinates comprises determining a nearest-neighbor set of grains consisting of kl of the plurality of grains, wherein none of the plurality of grains that are not included in the nearest-neighbor set of grains is closer to the first target descriptor coordinate than any one of the grains included in the nearest-neighbor set of grains, and the candidate set of candidate grains consists of the grains included in the nearest- neighbor set of grains.

4. The method of claim 3, further comprising determining kl based on the number of grains in the plurality of grains that have a grain coordinate that is within a threshold distance of the first target descriptor coordinate.

5. The method of claim 3, wherein each grain included in the plurality of grains is assigned an optimal k value, and the method comprises setting kl equal to the optimal k value assigned to the grain within the plurality of grains having a grain coordinate that is closest to the first target descriptor coordinate.

6. The method of claim 2, wherein defining the first set of candidate grains based on the first target descriptor coordinate and the grains’ coordinates comprises: determining a first radius value, rl; and including in the first set of candidate grains each one of the plurality of grains that has a grain coordinate that is within a distance of rl from the first target descriptor coordinate.

7. The method of claim 6, further comprising determining rl based on the number of grains in the plurality of grains that have a grain coordinate that is within a threshold distance of the first target descriptor coordinate.

8. The method of any one of claims 1-7, wherein assigning a final weight to each grain in the first set of candidate grains comprises: assigning a first weight to a first grain included in the set of candidate grains; determining a first final weight based on the first weight; and assigning the first final weight to the first grain.

9. The method of claim 8, wherein the first weight is: a function of the distance between the first target descriptor coordinate and the first grain, a function of the amount of time that has elapsed since the first grain was last rendered, a function of a trajectory associated with the first grain and a target trajectory, or a function of a measure of a closeness between the first grain and the most recently rendered grain.

10. The method of any one of claims 1-9, further comprising: after randomly selecting a grain from the first set of candidate grains based on the assigned final weights, assigning a new final weight to the rendered grain or removing the rendered grain from the first set of candidate grains; after assigning a new final weight to the rendered grain or removing the rendered grain from the first set of candidate grains, randomly selecting another grain from the first set of candidate grains based on the currently assigned final weights; and rendering the selected another grain.

11. The method of any one of claims 1-9, further comprising: after randomly selecting a grain from the first set of candidate grains, obtaining a second target descriptor coordinate; defining a second set of candidate grains based on the second target descriptor coordinate, wherein the second set of candidate grains comprises k2 of the plurality of grains, where k2 > 1 ; assigning a final weight to each grain in the second set of candidate grains; randomly selecting a grain from the second set of candidate grains based on the assigned final weights such that the probability that a given grain in the second set of candidate grains is selected is a function of the final weight assigned to the given grain; and rendering the grain randomly selected from the second set of candidate grains.

12. A method (600) for rendering audio corresponding to an audio recording, wherein the audio recording is divided into a plurality of grains, the method comprising: obtaining (s602) a first target descriptor coordinate, wherein the first target descriptor coordinate identifies a first location in an N-dimensional descriptor space, where N > 0; defining (s604) a first set of candidate grains based on the first target descriptor coordinate, wherein the first set of candidate grains comprises kl of the plurality of grains, where kl > 1, and the first set of candidate grains comprises a first grain and a second grain; assigning (s606) a final weight to each grain in the first set of candidate grains, wherein assigning a final weight to each grain in the first set of candidate grains comprises assigning a first final weight to the first grain and assigning a second final weight to the second grain; selecting (s608) a grain from the first set of candidate grains based on the assigned final weights; and rendering (s610) the selected grain, wherein the first final weight assigned to the first grain is: a function of the distance between the first target descriptor coordinate and the first grain, a function of the amount of time that has elapsed since the first grain was last rendered, a function of a trajectory associated with the first grain and a target trajectory, and/or a function of a measure of a closeness between the first grain and the most recently rendered grain.

13. The method of claim 12, wherein each one of the plurality of grains is associated with a grain coordinate that identifies a location of the grain in the N-dimensional descriptor space, and the first set of candidate grains is defined based on the first target descriptor coordinate and the grains’ coordinates.

14. The method of claim 13, wherein defining the first set of candidate grains based on the first target descriptor coordinate and the grains’ coordinates comprises determining a nearest-neighbor set of grains consisting of kl of the plurality of grains, wherein none of the plurality of grains that are not include in the nearest-neighbor set of grains is closer to the first target descriptor coordinate than any one of the grains included in the nearest-neighbor set of grains, and the candidate set of candidate grains consists of the grains included in the nearest- neighbor set of grains.

15. The method of claim 14, further comprising determining kl based on the number of grains in the plurality of grains that have a grain coordinate that is within a threshold distance of the first target descriptor coordinate.

16. The method of claim 14, wherein each grain included in the plurality of grains is assigned an optimal k value, and the method comprises setting kl equal to the optimal k value assigned to the grain within the plurality of grains having a grain coordinate that is closest to the first target descriptor coordinate.

17. The method of claim 13, wherein defining the first set of candidate grains based on the first target descriptor coordinate and the grains’ coordinates comprises: determining a first radius value, rl; and including in the first set of candidate grains each one of the plurality of grains that has a grain coordinate that is within a distance of rl from the first target descriptor coordinate.

18. The method of claim 17, further comprising determining rl based on the number of grains in the plurality of grains that have a grain coordinate that is within a threshold distance of the first target descriptor coordinate.

19. The method of any one of claims 12-18, further comprising: after selecting a grain from the first set of candidate grains based on the assigned final weights, assigning a new final weight to the rendered grain or removing the rendered grain from the first set of candidate grains; after assigning a new final weight to the rendered grain or removing the rendered grain from the first set of candidate grains, selecting another grain from the first set of candidate grains based on the currently assigned final weights; and rendering the selected another grain.

20. The method of any one of claims 12-19, further comprising: after selecting a grain from the first set of candidate grains, obtaining a second target descriptor coordinate; defining a second set of candidate grains based on the second target descriptor coordinate, wherein the second set of candidate grains comprises k2 of the plurality of grains, where k2 > 1 ; assigning a final weight to each grain in the second set of candidate grains; selecting a grain from the second set of candidate grains based on the assigned final weights; and rendering the grain randomly selected from the second set of candidate grains.

21. A method (600), wherein an audio recording is divided into a plurality of grains, the plurality of grains comprising a first grain, the method comprising: obtaining (s602) a first target descriptor coordinate, wherein the first target descriptor coordinate identifies a first location in an N-dimensional descriptor space, where N > 0; and selecting (s608) a grain from the plurality of grains, wherein the selection is based on: i) a function of the distance between the first target descriptor coordinate and the first grain and/or a function of the amount of time that has elapsed since the first grain was last rendered, and ii) a function of a trajectory associated with the first grain and a target trajectory and/or a function of a measure of a closeness between the first grain and a recently rendered grain.

22. The method of claim 21, wherein the method further comprises defining (s604) a first set of candidate grains based on the first target descriptor coordinate, wherein the first set of candidate grains comprises kl of the plurality of grains, where kl > 1, and the first set of candidate grains comprises the first grain and a second grain; and assigning (s606) a final weight to each grain in the first set of candidate grains, wherein assigning a final weight to each grain in the first set of candidate grains comprises assigning a first final weight to the first grain and assigning a second final weight to the second grain, wherein selecting (s608) a grain from the plurality of grains comprises selecting a grain from the first set of candidate grains based on the assigned final weights, wherein the first final weight assigned to the first grain is: i) a function of the distance between the first target descriptor coordinate and the first grain and/or a function of the amount of time that has elapsed since the first grain was last rendered, and ii) a function of the trajectory associated with the first grain and the target trajectory and/or a function of the measure of the closeness between the first grain and a recently rendered grain.

23. A computer program (843) comprising instructions (844) which when executed by processing circuitry (802) of an extended reality, XR, rendering device (724) causes the XR rendering device (724) to perform the method of any one of claims 1-22.

24. A carrier containing the computer program of claim 23, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer readable storage medium (842).

25. An extended reality, XR, rendering device (724) for rendering audio corresponding to an audio recording, wherein the audio recording is divided into a plurality of grains, the XR rendering device comprising: memory (842); and processing circuitry (802) coupled to the memory, wherein the apparatus is configured to perform a method (500) that comprises: obtaining (s502) a first target descriptor coordinate, wherein the first target descriptor coordinate identifies a first location in an N-dimensional descriptor space, where N > 0; defining (s505) a first set of candidate grains based on the first target descriptor coordinate, wherein the first set of candidate grains comprises kl of the plurality of grains, where kl > 1; assigning (s506) a final weight to each grain in the first set of candidate grains; randomly selecting (s508) a grain from the first set of candidate grains based on the assigned final weights such that the probability that a given grain in the first set of candidate grains is selected is a function of the final weight assigned to the given grain; and rendering (s510) the selected grain.

26. The XR rendering device of claim 25, wherein the XR rendering device is further configured to perform the method of any one of claims 2-11.

27. An extended reality, XR, rendering device (724) for rendering audio corresponding to an audio recording, wherein the audio recording is divided into a plurality of grains, the XR rendering device comprising: memory (842); and processing circuitry (802) coupled to the memory, wherein the apparatus is configured to perform a method (600) that comprises: obtaining (s602) a first target descriptor coordinate, wherein the first target descriptor coordinate identifies a first location in an N-dimensional descriptor space, where N > 0; defining (s604) a first set of candidate grains based on the first target descriptor coordinate, wherein the first set of candidate grains comprises kl of the plurality of grains, where kl > 1, and the first set of candidate grains comprises a first grain and a second grain; assigning (s606) a final weight to each grain in the first set of candidate grains, wherein assigning a final weight to each grain in the first set of candidate grains comprises assigning a first final weight to the first grain and assigning a second final weight to the second grain; selecting (s608) a grain from the first set of candidate grains based on the assigned final weights; and rendering (s610) the selected grain, wherein the first final weight assigned to the first grain is: a function of the distance between the first target descriptor coordinate and the first grain, a function of the amount of time that has elapsed since the first grain was last rendered, a function of a trajectory associated with the first grain and a target trajectory, and/or a function of a measure of a closeness between the first grain and the most recently rendered grain.

28. The XR rendering device of claim 27, wherein the XR rendering device is further configured to perform the method of any one of claims 13-20.

29. An extended reality, XR, rendering device (724) for rendering audio corresponding to an audio recording, wherein the audio recording is divided into a plurality of grains, the XR rendering device comprising: memory (842); and processing circuitry (802) coupled to the memory, wherein the apparatus is configured to perform a method (600) that comprises: obtaining (s602) a first target descriptor coordinate, wherein the first target descriptor coordinate identifies a first location in an N-dimensional descriptor space, where N > 0; and selecting (s608) a grain from the plurality of grains, wherein the selection is based on: i) a function of the distance between the first target descriptor coordinate and the first grain and/or a function of the amount of time that has elapsed since the first grain was last rendered, and ii) a function of a trajectory associated with the first grain and a target trajectory and/or a function of a measure of a closeness between the first grain and a recently rendered grain.

30. The XR rendering device of claim 29, wherein the method further comprises defining (s604) a first set of candidate grains based on the first target descriptor coordinate, wherein the first set of candidate grains comprises kl of the plurality of grains, where kl > 1, and the first set of candidate grains comprises the first grain and a second grain; and assigning (s606) a final weight to each grain in the first set of candidate grains, wherein assigning a final weight to each grain in the first set of candidate grains comprises assigning a first final weight to the first grain and assigning a second final weight to the second grain, wherein selecting (s608) a grain from the plurality of grains comprises selecting a grain from the first set of candidate grains based on the assigned final weights, wherein the first final weight assigned to the first grain is: i) a function of the distance between the first target descriptor coordinate and the first grain and/or a function of the amount of time that has elapsed since the first grain was last rendered, and ii) a function of the trajectory associated with the first grain and the target trajectory and/or a function of the measure of the closeness between the first grain and a recently rendered grain.

31. An extended reality, XR, rendering device (724) configured to perform the method according to at least one of the claims 1 to 22.