JP2007501553A - Apparatus and method for generating, storing and editing audio representations in an audio scene - Google Patents

Abstract

The audio signal processing circuit (12) has a large number of input channels (16) carrying separate signals (Ek1-Ekm). The circuit may be a WFS (Wave Field Synthesis) rendering unit. It has a large number of output channels (14) carrying loudspeaker signals (LS1- LSn). A display device (18) provides the inputs for the audio signal processing circuit. Temporally non- overlapping audio objects are allocated to the same input channel. The input for the display device comes from the device generating the audio scene (10) with separate audio objects.

Description

  The present invention is in the field of wavefront synthesis, and more particularly to an apparatus and method for generating, storing and editing audio representations in an audio scene.

  There is a growing need for new technologies and innovative products in the field of entertainment electronics. Here, there are important premises for a new multimedia system to succeed in providing optimal functions or performances, respectively. This is achieved by using digital technology and in particular computer technology. Accordingly, as an example, the present invention is applied to improve the realistic sensation of the impression of an audio image. Prior art audio systems have significant weaknesses in the quality of spatial audio playback in real and virtual environments.

  Multi-channel speaker reproduction methods for audio signals are well known and have been standardized for many years. All conventional techniques have the disadvantage that both the placement of the speakers and the position of the listener are already fixed in the transmission format. If the speakers are misplaced with respect to the listener, the audio quality is greatly degraded. The optimum sound is possible only in a very narrow range of the reproduction space, the so-called sweet spot.

  A more robust speaker box during audio playback with an improved natural space impression can be achieved with the support of new technologies. The principle of this technology, so-called wavefront synthesis (WFS), was studied at TU Delft and was first published in the late 1980s (Berkhout, AJ Doffries ( de Vries), D. Vogel, P .: Acoustic control by wave-field synthesis (JASA 93, 993).

  Wavefront synthesis has never been used in practice so far because of the huge requirements of this method of computer capability and transmission speed. However, advances in microprocessor technology and the audio coding field allow this technology to be used today for specific applications. The first product in the specialized field is expected next year. Over the years, the first wavefront synthesis application in the consumer sector will be sold.

  The basic idea of WFS is based on the application of the Huygens principle of wave theory.

  Each point obtained by the wave is a starting point when the element wave propagates in a spherical or circular path.

  When applied to acoustics, any number of incoming wavefronts can be reproduced by a very large number of speakers (so-called speaker arrays) arranged side by side. In the simplest case, when playing a single point sound source and linearly arranging speakers, the audio signal of each speaker is time-delayed and amplitude-magnified so that the sound fields output from the individual speakers are appropriately superimposed. It is necessary to supply. By using several sound sources, the contribution rate for each speaker is calculated separately for each sound source, and the obtained signals are added. If a sound source to be reproduced exists in a room having a reflective wall surface, reflection can be reproduced as an additional sound source through the speaker array. Therefore, the calculation effort is very dependent on the sound source, the reflection characteristics of the recording room, and the number of speakers.

  In particular, a unique advantage of this technology is that natural spatial audio impressions are possible over a wide range of playback space. In contrast to known techniques, the direction and distance from the sound source are reproduced very accurately. To a limited extent, a virtual sound source can be placed between the actual speaker array and the listener.

  Wavefront synthesis works well for environments with known characteristics, but irregularities occur when the characteristics change or wavefront synthesis is performed based on environmental characteristics that do not match the actual characteristics of the environment. To do.

  However, wavefront synthesis techniques can be used effectively to add corresponding spatial audio perception to virtual perception. Conventionally, during the production in the virtual studio, the true virtual impression was transmitted in the virtual scene in the foreground. Sound impression that matches the image is usually incorporated into the audio signal later by a manual operation process called so-called post-production, but it is considered to be very costly and time consuming to realize, Ignored. This generally causes a contradiction between individual sensory impressions, and the designed space, that is, the designed scene, seems to be inferior to the real thing.

  Generally speaking, audio material, such as a movie, consists of a number of audio objects. An audio object is a sound source in a movie setting. Considering the scene of a movie, for example, when two people are standing facing each other and talking at the same time, for example, a person on a horse and a train approaching, at a certain time, There will be four sound sources in this scene. That is, two people, a person riding on an approaching horse, and a running train. Assuming that two people who are talking stop talking at the same time, at some time, at least two audio objects need to be at least active. If two people are silent at this time, that is, a person on a horse and a train. However, if one person is talking at a different time, three audio objects are active. That is, a person on a horse, a train, and one person. If two people are actually talking at the same time, four audio objects are active at this time. That is, a person on a horse, a train, a first person, and a second person.

  Generally speaking, an audio object represents itself, such that an audio object represents a sound source in a movie setting that is active or “alive” at a particular time. It also means that the audio object is characterized by a start time and an end time. In the former example, the person on the horse and the train are active during all settings, for example. If both are approaching, the listener will recognize this by the louder sound of the person on the horse and the train. In the optimum wavefront synthesis setting, if the positions of these sound sources can also be adapted, they change accordingly. On the other hand, the two speakers speaking are always creating a new audio object. This is because the current audio object ends whenever one speaker stops speaking, and a new audio object starts when the other speaker starts speaking. This ends when the other speaker stops speaking. When the first speaker starts speaking again, it also starts a new audio object.

  There are existing wavefront synthesis rendering means that can generate a certain amount of speaker signal from a certain amount of input channels, ie knowing the position of each of the speakers of the wavefront synthesis speaker array.

  The wavefront synthesis rendering device is, in a sense, the “heart” of the wavefront synthesis system. This calculates the loudspeaker signal of multiple loudspeakers in the loudspeaker array so that the amplitude and phase are accurate, thus providing the user with an optimal acoustic impression as well as an optimal visual impression.

  In the late 1960s, multi-channel audio was introduced into movies, and it has always been the goal of acoustic engineers to give the listener the impression that they are actually in the scene. Adding surround channels to the playback system is another milestone. In the 90s, new digital systems were introduced and the number of audio channels increased. Currently, the 5.1 or 7.1 system is the standard system for movie playback.

  In many cases, these systems will have great potential to creatively support the impression of a movie and need to give great potential to sound effects, ambient sounds, or surround mixing music. On the other hand, wavefront synthesis techniques are very flexible and provide the greatest freedom in this regard.

  However, with the 5.1 or 7.1 system being used, a “standardized” approach has been taken to handle the mixing of movie soundtracks.

  The playback system usually has a fixed speaker position. In the case of 5.1, the left channel (“left”), center channel (“center”), right channel (“right”), surround left channel (“surround left”), and surround right channel (“surround right”) ). Because of these fixed (few) positions, the ideal sound image that the acoustic engineer seeks is limited to a small number of seats, so-called sweet spots. If a virtual sound source is used during the arrangement of 5.1 above, an improvement may be seen, but not necessarily a satisfactory result.

  Movie sounds usually consist of lines, sound effects, environmental sounds, and music. These elements are mixed together, taking into account the constraints of the 5.1 and 7.1 systems. In general, the lines are mixed in the central channel (and in the 7.1 system, the half left and half right positions). This means that if the actor crosses the screen, the sound will not follow. Since the sound effect of the moving sound object can be realized only when moving quickly, the listener cannot recognize that the sound is transmitted from one speaker to the other.

  Since the audible gap between the front speaker and the surround speaker is large, the sound source in the horizontal direction cannot be arranged, so that the object cannot move slowly from the rear to the front or vice versa.

  Also, because surround speakers cannot be placed in the speaker's diffusion array, a sound image representing a certain envelope for the listener is generated. Therefore, in order to avoid the annoying interference sound field associated with the sound source arranged accurately in this way, the sound source arranged accurately behind the listener is omitted.

  Wavefront synthesis, a completely new way to build a sound field that is perceived by the listener, overcomes these fundamental drawbacks. The importance of application to a movie theater is that an accurate sound image can be achieved without being restricted by the two-dimensional arrangement of objects. This offers a wide variety of possibilities when designing for cinema and sound mixing. With complete sound image reproduction achieved by wavefront synthesis technology, the sound source can be freely arranged. In addition to the sound source in the listener's space, the sound source can be placed with a focus on the sound source outside the listener's space.

  Furthermore, a stable sound source direction and a stable sound source position can also be generated using point-shaped radiating sound sources or plane waves. Finally, the sound source can be freely moved through the listener's space or through the listener's space.

  This provides great potential for providing creative reality and the possibility of accurately placing the sound source, eg, for all lines, according to the image on the screen. At the same time, the listener can actually be inserted into the movie both visually and acoustically.

  Due to historical circumstances, the sound design, ie the behavior of the sound recording clerk, is based on the channel or track paradigm. It means that the playback setup is determined by the encoding format or the number of speakers, ie 5.1 system or 7.1 system. In particular, certain sound systems still require specific encoding formats. As a result, the master file cannot be changed at all unless all mixing is performed again. For example, it is not possible to selectively change the clift rack in the final master file. That is, it cannot be changed unless all other sounds in this scene are changed.

  On the other hand, the viewer / listener is indifferent to the channel. It is indifferent to which sound system the sound is generated, whether the original sound content is expressed in an object-oriented or channel-oriented manner, etc. The listener is also not interested in how and how the audio settings were mixed. The only thing that takes into account the listener is sound impression. That is, whether you like the sound settings for the movie, or whether the sound settings should be in the movie or not.

  On the other hand, it is essential that the new concept is accepted by people working on the new concept. The sound recording staff is in charge of sound mixing. The sound recording clerk is “tuned” to work in a channel orientation because of the channel orientation paradigm. For them, for example, in a 5.1 sound system cinema, the goal is to actually mix six channels. This is not about audio objects, but about channel orientation. In this case, the audio object generally has neither a start time nor an end time. Instead, the speaker signal is active from the first second of the movie to the last second of the movie. This is due to the fact that a sound is generated through one of (several) speakers of a typical cinema sound system. This is because there is always a need for a sound source through a specific speaker even for background music alone.

  For this reason, when an audio signal is input to an input channel with associated information, it has a certain amount of input channels that generate speaker signals for individual speakers or speaker groups of the wavefront synthesized speaker array. In addition, an existing wavefront synthesis rendering apparatus is used so as to operate in a channel orientation.

  On the other hand, in principle, as long as there are a large number of unrestricted audio objects that can be observed in a movie, that is, to be observed in an audio scene, the wavefront synthesis technique makes the audio scene fundamental. Is more “transparent”. If the amount of audio objects in the audio scene generally exceeds the default maximum amount of input channels of the audio processing means in general, channel-oriented wavefront synthesis rendering means may be problematic. In addition, for a user, i.e. a sound recorder, for example, a number of audios that exist at one particular time but do not exist at another time, i.e. a defined start and a defined end time. Generating an audio representation of an audio scene, such as an object, is confusing and therefore creates a psychological threshold between the sound recorder and wavefront synthesis. In reality, however, it is expected that sound recording staff will build great creative possibilities.

Berkhout, A .; J. et al. , De, Vries, D .; And Vogel, P .; Author "Acoustic control by Wave-field Synthesis", JASA 93, 1993

  It is an object of the present invention to provide a concept for generating, storing and editing audio representations in an audio scene that is widely accepted by users seeking corresponding tools.

  The object is to generate, store and edit an audio representation in an audio scene according to claim 1, and to generate, store and edit an audio representation in an audio scene according to claim 15. Or a computer program according to claim 16.

  The present invention is based solely on the finding that object-oriented descriptions can be processed in a clear and efficient manner, as occurs for audio objects in common movie settings. An object-oriented description of an audio scene having an audio signal and having objects associated with a defined start time and a defined end time corresponds to an actual general situation. It never happens that the sound exists for the whole time. Instead, for example, in a dialogue, it is common that the dialogue partner starts and stops talking, or the sound usually has a beginning and an end. As far as that is concerned, the object-oriented audio scene description that maps each actual sound source to its own object is optimal in terms of transparency, clarity, efficiency, and comprehension because it applies to natural situations.

  On the other hand, for example, a sound recording clerk who wants to generate an audio representation from an audio scene, that is, wants to “synchronize” the audio representation in an audio scene in a cinema, slipping their creative possibilities Sound recording staff may also take into account special sound effects. For the channel paradigm, it is generally used in conjunction with either a hardware-implemented mixing desk or a software-implemented mixing desk, which results in a conversion to a channel-oriented working method. In a hardware-implemented mixing desk or software-implemented mixing desk, each channel has a regulator, a button, and the like, thereby operating an audio signal in the channel, that is, performing “mixing”.

  According to the present invention, a balance between an object-oriented audio expression that brings life to life and a channel-oriented expression that can demonstrate the value of a sound recording person can be obtained. This is achieved by the mapping means used for mapping to a plurality of input channels of the audio processing means. According to the invention, the mapping means assigns a first audio object to the input channel, assigns a second audio object whose start time is after the end time of the first audio object to the same input channel, A third audio object whose start time is after the start time of the first audio object and before the end time of the first audio object is configured to be assigned to another one of the plurality of input channels. .

  We found that this temporal assignment of assigning simultaneously occurring audio objects to different input channels of a wavefront rendering renderer but assigning consecutively generated audio objects to the same input channel is very channel efficient. . Meaning that, on average, a relatively small number of input channels of a wavefront rendering renderer is occupied makes it clear, and on the other hand, it is convenient for the computational efficiency of the computationally intensive wavefront rendering renderer. ing. Since the number of simultaneously occupied channels is a relatively small number on average, the user, i.e. the sound recorder, for example, has the problem of which objects are active at this moment and which are not active at this moment. Quickly get an overview of the complexity of an audio scene at a particular time without having to search through multiple input channels. On the other hand, the user can easily operate the audio object as an object-oriented expression by the channel regulator used by the user.

  The expectation that the inventive concept is widely accepted is to provide the user with the inventive concept in a familiar working environment. However, this includes even more innovative possibilities. Thus, an inventive concept based on mapping an object-oriented audio approach to a channel-oriented rendering approach will demonstrate the value of all requirements. On the other hand, as already mentioned, the object-oriented description of the audio scene is naturally optimally applied, so it becomes efficient and clear. On the other hand, it takes into account the user's habits and needs in terms of adapting the technology to the user and vice versa.

Preferred embodiments of the present invention will be described with reference to the accompanying drawings.
FIG. 1 is a block circuit diagram of an inventive apparatus for generating an audio representation.
FIG. 2 is a schematic illustration of a user interface for the concept shown in FIG.
FIG. 3a is a schematic illustration of the user interface of FIG. 2, according to one embodiment of the present invention.
FIG. 3b is a schematic illustration of the user interface of FIG. 2 according to another embodiment of the present invention.
FIG. 4 is a block circuit diagram of an inventive device according to a preferred embodiment.
FIG. 5 is a time diagram of an audio scene having various audio objects.
FIG. 6 is a comparison of 1: 1 conversion between objects and channels and object channel assignment according to the present invention for the audio scene shown in FIG.

  FIG. 1 shows a block circuit diagram of an inventive device for generating an audio representation in an audio scene. The inventive device includes means 10 for providing an object-oriented description of the audio scene. The object-oriented description of the audio scene includes a plurality of audio objects, and the audio objects are associated with at least an audio signal, a start time, and an end time. The inventive device further includes audio processing means 12 for generating a plurality of speaker signals LSi14. This is channel-oriented and generates a plurality of speaker signals 14 from a plurality of input channels EKi. Mapping means for mapping an object-oriented description of an audio scene to a plurality of input channels 16 of the channel-oriented audio signal processing means 12 between the providing means 10 and channel-oriented audio signal processing means formed as, for example, a WFS rendering device There are 18. The mapping means 18 assigns the first audio object to the input channel such as EK1, and assigns the second audio object whose start time is after the end time of the first audio object to the same input channel such as the input channel EK1. Assign a third audio object whose start time is after the start time of the first audio object and before the end time of the first audio object to another input of a plurality of input channels such as input channel EK2. It is configured to be assigned to a channel. The mapping means 18 is thus configured to assign audio objects that do not overlap in time to the same input channel and to assign audio objects that overlap in time to different parallel input channels.

  In a preferred embodiment, the channel-oriented audio signal processing means 12 includes a wavefront synthesis rendering device. The audio object is also specified so as to be associated with the virtual position. This virtual position of the object can also be changed while the object is alive. This corresponds to, for example, a case where a person riding on a horse approaches the center of the scene and the gallop of the person riding on the horse gradually increases, and in particular, gradually approaches the listener space. In this case, the audio object is not only the audio signal associated with this audio object and the start time and end time, but also the position of the virtual sound source that changes over time, and if applicable, It can also include audio object characteristics such as whether it is necessary to have point source characteristics or whether it is necessary to emit a plane wave corresponding to a virtual position at an infinite distance to the viewer. Technically, it also knows the characteristics of the sound source, that is, the characteristics of the audio object. This may be taken into account by the channel-oriented audio signal processing means 12 facility of FIG.

  According to the present invention, the structure of the apparatus is constructed hierarchically. The channel-oriented audio signal processing means for receiving the audio object is connected via the mapping means without being directly connected to the providing means. As a result, information about all audio scenes is obtained and stored only in the providing means, and there is less information on all audio settings that the mapping means and the channel-oriented audio signal processing means must save. Instead, both the mapping means 18 and the audio signal processing means 12 operate according to the instructions of the audio scene supplied from the providing means 10.

  In the preferred embodiment of the present invention, the apparatus shown in FIG. 1 further comprises a user interface shown as 20 in FIG. The user interface 20 is configured to have one user interface channel for each input channel and preferably have an operating device for each user interface channel. Since the usage rate of EK1 for the input channel EKm is displayed on the user interface 20, the user interface 20 is connected to the mapping means 18 via the user interface input 22 in order to obtain allocation information from the mapping means. On the output side, the user interface 20 is connected to the providing means 10 when each user interface channel has an operation device function. In particular, the user interface 20 is configured to provide the providing means 10 with an audio object operated on the original version. Thus, the modified audio scene is acquired and then provided to the mapping means 18 for distribution to the input channels accordingly and distribution to the channel-oriented audio signal processing means 12.

  Depending on the embodiment, the user interface 20 is configured as a user interface as shown in FIG. 3a. That is, the user interface always represents only the current object. Alternatively, the user interface 20 is constructed as shown in FIG. That is, always represent all objects in the input channel. In both FIGS. 3a and 3b, the timeline 30 is shown to include objects A, B and C in the order of occurrence. Object A includes a start time 31a and an end time 31b. Randomly, in FIG. 3a, the end time 31b of the first object A occurs simultaneously with the start time of the second object B. This has an end time 32b and occurs randomly and at the same time as the start time of the third object C. Again, this has an end time 33b. Start times 32a and 33b correspond to end times 31b and 32b, but are not shown in FIGS. 3a and 3b for simplicity.

  In the mode shown in FIG. 3a, only the current object is displayed as a user interface channel. The mixing desk channel symbol 34 is shown on the right side of FIG. 3a. This includes a slider 35 and a style button 36. The characteristic or virtual position of the audio signal of the object B is changed. As soon as the time mark of FIG. 3 a indicated by 37 reaches the end time 32 b of the object B, the style channel illustration 34 displays the object C, not the object B. For example, if object D occurs simultaneously with object B, the user interface of FIG. 3a further indicates a channel, such as input channel i + 1. The explanation shown in FIG. 3a provides an easy-to-understand summary of the number of parallel audio objects at a certain time to the sound recorder. That is, the actual number of active channels is displayed. Input channels that are not active are not displayed at all in the embodiment of the user interface 20 of FIG. 2 shown in FIG. 3a.

  In the embodiment shown in FIG. 3b, all objects in the input channel are displayed adjacently. The display of unused input channels is also not performed. However, the input channel i to which the channels assigned in the order of occurrence in time belong is represented by three times. That is, what was the object channel A at a certain time becomes the object channel B at another time, and becomes the object channel C at another time. In accordance with the present invention, an object is currently on channel i so that the sound recorder can predict further manipulation of the object's audio signal via this channel regulator or channel switch through corresponding software or hardware regulators. Provided for object B, for example in color or brightness, to provide a clear overview to the sound recorder, for example, that multiple objects will be executed on this channel sooner or later. Channel i can be suitably emphasized (reference number 38 in FIG. 3b), such as channel i. Accordingly, the user interface 20 of FIG. 2, and in particular its embodiment of FIGS. 3a and 3b, determines the “occupancy” of the input channel of the channel-oriented audio signal processing means generated by the mapping means 18 if desired. Configured to provide visual illustrations.

  Next, referring to FIG. 5, a simple example of the function of the mapping means 18 of FIG. 1 is shown. FIG. 5 shows an audio scene with various audio objects A, B, C, D, E, F, and G. It can be seen that objects A, B, C, and D overlap in time. In other words, these objects A, B, C, and D are all active at a particular time 50. On the other hand, the object E does not overlap with the objects A and B. As can be seen from time 52, object E overlaps only with objects D and C. For example, as can be seen from time 54, object F and object D overlap. The same applies to objects F and G. For example, although it overlaps at time 56, object G does not overlap with objects A, B, C, D, and E.

  A simple channel association, which is often inconvenient, is to assign each audio object to an input channel, as shown in the example of FIG. The 1: 1 conversion shown on the left side of the table of FIG. 6 will be obtained. The disadvantage of this concept is that if you have a large number of input channels or very fast in the case of a movie, there are a lot of audio objects, depending on the number of input channels of the wavefront rendering device, the actual movie settings The number of virtual sound sources that can be processed is limited. Of course, this is not desirable because technical limitations must not hinder creative possibilities. On the other hand, this 1: 1 conversion is uncertain in the following points. In general, each input channel may get an audio object, but considering a specific audio scene, a relatively small number of input channels are generally active, but the overview always provides all audio channels. However, the user cannot easily exercise this.

  Furthermore, with this concept of assigning 1: 1 audio objects to the input channels of the audio processing means, the number of input channels can be reduced in order to minimize the number of audio objects and to limit the number of audio objects. Will lead to the fact that there is a need to provide very many audio processing means. This immediately increases the amount of calculation and requires the calculation capability and storage capacity of the audio processing means to calculate individual speaker signals. This directly increases the price of such a system.

  As achieved by the mapping means 18 according to the invention, the inventive assignment object channel of the example shown in FIG. 5 is shown in the right part of the table of FIG. Accordingly, parallel audio objects A, B, C, and D are assigned sequentially to input channels EK1, EK2, EK3, and EK4, respectively. Although it is not necessary to assign the object E to the input channel EK5 as shown in the left half of FIG. 6, it can be freely assigned to the input channel EK1 or the input channel EK2 as shown by parentheses. The same applies to object F. In principle, this may be assigned to all channels except the input channel EK4. The same applies to object G. This may be assigned to all channels except for the channel to which the object F has been previously assigned (in this example, the input channel EK1).

  In a preferred embodiment of the invention, the mapping means 18 is configured to always occupy the channel whose original number is as small as possible. Whenever possible, the adjacent input channels EKi and EKi + 1 are occupied. This ensures that there are no holes. On the other hand, this “neighboring function” is not essential. This is because, for example, the first of the input channels of the audio processing means is as long as this channel can be manipulated with an inventive user interface by means of the regulator 35 or the button 36 of the mixing desk channel illustration 34, which is just the current channel. This is because it is meaningless to the user of the audio author system according to the present invention whether it is the th, the seventh, or what number is currently being operated. Therefore, the user interface channel i does not necessarily correspond to the input channel i, but the user interface channel i corresponds to the input channel EKm, for example, and the user interface channel i + 1 corresponds to the input channel k and the like. Channel assignment can also be performed.

  At the same time, the presence of a channel hole is avoided by performing user interface channel remapping. That is, the sound recording staff always knows the current user interface channels shown immediately and clearly adjacent to each other.

  Of course, the inventive concept of user interface can also be transferred to an existing hardware mixing console. This includes the actual hardware regulator and hardware buttons, and operates to perform optimal audio mixing with manual input of the sound recording. An advantage of the present invention is such a hardware mixing console, which is generally very familiar and important to sound recording personnel and can always be used with the very current channel. The current channel is clearly displayed to the sound recording clerk, for example by an indicator such as an LED typically present on a mixing console.

  The present invention is more flexible in that it eliminates the generating wavefront synthesis speaker setup from, for example, a movie theater playback setup. Thus, according to the present invention, audio content is encoded in a format that can be rendered by various systems. This format is an audio scene. That is, object-oriented audio representation, not speaker signal representation. As far as it is concerned, the display method is understood as an application to a content playback system. According to the present invention, not only several master channels but also descriptions of all object-oriented scenes are processed by the wavefront synthesis reproduction process. The scene is rendered for each playback. This is typically processed in real time to apply to the current state. In general, this application takes into account the number of speakers and their positions, the characteristics of the reproduction system, such as frequency characteristics, sound differential pressure levels, indoor acoustic conditions, or even video reproduction conditions.

  Compared to the channel-based approach of current systems, the major difference in wavefront synthesis mixing is that sound objects can be placed freely. In a normal reproduction system based on the stereophonic principle, the position of a sound source is assumed and encoded. This is important for a mixing concept belonging to visual contents such as movies, for example, because the arrangement of sound sources is approximated by referring to the video by an accurate system setup.

  Wavefront synthesis systems, however, require an absolute position relative to the sound object. This is provided as additional information for the audio signal of the audio object along with the audio object, in addition to the start time and end time of the audio object.

  In conventional channel-oriented approaches, the basic idea was to reduce the number of tracks in some premixing operations. These premixing operations are organized in categories such as speech, music, sound, and sound effects. During the mixing process, all necessary audio signals are fed to the mixing console and mixed simultaneously by different acoustic engineers. Each premixing reduces the number of tracks until there is one track per playback speaker. These final tracks constitute the final master file (final master).

  All relevant mixing tasks, such as equalization, dynamic, position, etc., are performed using a mixing desk or even a dedicated device.

  The purpose of post-engineering reengineering is to minimize user training and to integrate new and inventive systems into the user's existing knowledge. In the wavefront synthesis application of the present invention, all of the objects to be rendered in different tracks or locations have a master file / distribution format as opposed to a conventional production facility, reducing the number of tracks during the generation process. Optimize so that. On the other hand, for practical reasons, it is necessary to provide the re-recording engineer with the possibility to perform wavefront synthesis using an existing mixing console.

  Therefore, according to the present invention, the current mixing console is used for conventional mixing operations. These mixing console outputs are then introduced into an inventive system to perform spatial mixing to produce an audio representation in the audio scene. The wavefront synthesis author tool according to the present invention is implemented as a workstation, which means that it has the ability to record the final mixed audio signal and convert it into a distribution format in a separate step. For this reason, according to the invention, two aspects are taken into account. The first is that all audio objects or tracks are still in the final master. The second aspect is that no arrangement is made on the mixing console. So-called authoring, that is, post-processing of the sound recording staff, means that it is one of the final steps of the production chain. According to the present invention, the inventive wavefront synthesis of the system according to the present invention, ie the inventive device for generating the audio representation, is implemented as a stand-alone workstation. By supplying the audio output from the mixing desk to the system, it may be incorporated into a different production environment. As far as it is concerned, a mixing desk represents a user interface connected to a device for generating an audio representation in an audio scene.

  An inventive system according to a preferred embodiment of the present invention is shown in FIG. The same reference numerals as in FIG. 1 or 2 indicate the same elements. The basic system design aims at modularity and is based on the ability to incorporate an existing mixing console as a user interface into an inventive wavefront synthesis author system.

  For this reason, a central control device 120 that communicates with other modules is configured inside the audio processing means 12. This makes it possible to use certain module options as long as all use the same communication protocol. Considering the system shown in FIG. 4 as a black box, the user interface 20 can generally be observed with multiple inputs (from the providing means 10) and multiple outputs (speaker signal 14). Next to the user interface, the actual WFS rendering device 122 is incorporated in the black box. This performs the actual wavefront synthesis operation of the speaker signal using various input information. A room simulation module 124 is also provided. This is configured to perform certain room simulations that are used to generate room characteristics of the recording room or to manipulate room characteristics of the recording room.

  In addition to the audio recording means 126, a recording / reproducing means (also 126) is provided. The means 126 preferably comprises an external input. In this case, the entire audio signal is provided and supplied either originally object-oriented or stationary channel-oriented. At that time, the audio signal does not come from the scene protocol and then follows the control task. The supplied audio data is then converted from means 126 to an object-based representation, if necessary, and then internally supplied to the mapping means 18 and then object / channel mapping is performed.

  All audio connections between modules can be switched by the matrix module 128. At the request of the central controller 120, the corresponding channel is connected to the corresponding channel. In the preferred embodiment, the user has the function of supplying the audio processing means 12 with the virtual sound source signal through 64 input channels, so in this embodiment there are 64 input channels EK1 to EKm. At the same time, pre-mixing of the virtual sound source signal may be performed using an existing console as a user interface. Next, spatial mixing is performed by the wavefront synthesis author system and the WFS rendering device 122, particularly the heart.

  The complete scene description is stored in the providing means 10. This is also called a scene protocol. However, main communication or necessary data traffic is performed at the central controller 120. The scene description can be changed by, for example, the user interface 20, particularly the hardware mixing console 200 or the software GUI, that is, the software graphic user interface 202, and the changed scene can be changed via the user interface control device 204. The change is supplied to the providing means 10 as a protocol. By providing a modified scene protocol, it uniquely represents the entire logical structure of the scene.

  In order to realize the object-oriented solution approach, each sound object is associated with a display channel (input channel) by the mapping means 18. An object exists for a certain time. Typically, as shown in accordance with FIGS. 3a, 3b, and 6, a large number of objects exist on a particular channel in the order of occurrence. An inventive author system supports this object orientation, but the wavefront synthesis rendering device itself does not know about the object. Simply receive the signals in the audio channels and a description of how to render these channels. The providing means that knows the scene protocol, that is, the object and the associated channel, may convert the object-related metadata (eg, sound source position) into channel-related metadata and send it to the WFS rendering device 122. As schematically shown in block form functional protocol 129 in FIG. 4, communication between other modules is performed with a dedicated protocol so that the other modules contain only the information needed.

  The inventive control module also supports hard disk storage of scene descriptions. Preferably, two file formats are distinguished. One file format is an author format, and audio data is stored as compressed PCM data. Further, using session-related information such as audio object, that is, sound source grouping, layer information, and the like, it is stored in a dedicated file format based on XML.

  The other type is a distribution file format. In this format, audio data can be compressed and stored, and there is no need to further store session related data. It should be noted that audio objects still exist in this format and may be distributed using the MPEG-4 standard. According to the present invention, it is preferable to always perform wavefront synthesis rendering in real time. This eliminates the need to store pre-rendered audio information, i.e., already finished speaker signals, in any file format. This is a great advantage as far as speaker signals occupy a very large part of the data. This is not due to the large number of speakers used at least in the wavefront synthesis environment.

  One or more wavefront synthesis rendering device modules 122 are typically supplied with virtual sound source signals and channel-oriented scene descriptions. The wavefront synthesis rendering apparatus calculates a drive signal according to the wavefront synthesis theory of each speaker, that is, the speaker signal of the speaker signal 14 in FIG. The wavefront synthesis rendering apparatus further calculates a subwoofer speaker signal. This is also necessary to support wavefront synthesis systems at low frequencies. The room simulation signal from the room simulation module 124 is rendered using a large number (usually 8 to 12) of stationary plane waves. Based on this concept, different solution approaches can be integrated to perform room simulation. When the room simulation module 124 is not used, the wavefront synthesis system has already generated an acceptable sound image that stably recognizes the sound source direction in the listening range. However, there are certain flaws with respect to sound source depth recognition, since they usually do not add any initial spatial reflection or reverberation to the sound source signal. According to the present invention, it is preferable to use a room simulation module. This reproduces the wall reflection. For example, an initial reflection is generated using a mirror sound source model and modeled. These mirror sound sources may be processed again as scene protocol audio objects, or may actually be added by the audio processing means itself. The recording / playback device 126 represents a useful supplement. In order to perform only spatial mixing, a sound object for which mixing has been completed in the conventional manner during pre-mixing may be supplied from a conventional mixing desk to an audio object playback device. Further, it is preferable to include an audio recording module that records the output channel of the mixing desk by time code control and stores the audio data in the reproduction module. The playback module receives the start time code, i.e. plays a particular audio object associated with the individual output channel supplied from the display means 18 to the playback device 126. The recording / playback apparatus can also start and stop the playback of individual audio objects separately according to the description of the start time and stop time associated with the audio object. As soon as the mixing procedure is finished, the audio content is extracted from the playback device module and exported to the distribution file format. Thus, the delivery file format includes the finished scene protocol for a scene that is ready for mixing. The purpose of the inventive user interface concept is to implement a hierarchical structure. This applies to cinema mixing processing operations. Here, the audio object is recorded as a sound source that exists as an expression of an individual audio object at an arbitrary time. The start time and stop / end time are typical for sound sources, ie audio objects. A sound source or audio object requires system resources during the time that the object or sound source is “alive”.

  Preferably, apart from the start time and stop time, each sound source also includes metadata. These metadata are “type” (plane wave or point sound source at a specific time), “direction”, “volume”, “muting” and “flag” of direction-dependent volume and direction-dependent delay. All of these metadata may be used automatically.

  Also, regardless of the object-oriented solution approach, for example, objects that are “living” throughout the movie, or generally throughout the scene, generally acquire an inventive author system in that they also acquire their own channels. It is preferable to use it for the channel concept. These objects in principle are meant to represent simple channels with the 1: 1 conversion described with reference to FIG.

  In a preferred embodiment of the invention, at least two objects may be grouped. Each group can be selected with which parameter to group, and how to calculate using the master of the group can be selected. The sound source group exists for an arbitrary time determined by the start time and end time of the member.

  An example of a group utility consists of using for virtual standard surround setup. These can also be used for virtual fade out of the scene or virtual zoom in of the scene. Alternatively, grouping is used to incorporate surround reverberation and to record WFS mixing.

  Furthermore, it is preferable to construct a logical essence, that is, a layer. In order to build a mixing or scene, the preferred embodiment of the present invention arranges groups and sound sources in different layers. By using layers, predubbing may be simulated at an audio workstation. It is also possible to change display attributes during author processing such as using layers to display or hide different parts of the current mixing target.

  A scene consists of the aforementioned components for an arbitrary duration. This duration can also be a film spool or just a certain duration of a part of the movie, for example the whole movie or for example 5 minutes. A scene is again composed of a number of layers, groups, and sound sources belonging to the scene.

  Preferably, the complete user interface 20 includes both a graphic software part and a hardware part to allow haptic control. This is preferred, however, due to cost, the user interface can also be implemented entirely as a software module.

  A graphic system design concept based on so-called “space” is used. There are a few different spaces in the user interface. Each space represents a project from a different approach that is a dedicated editing environment and can take advantage of all the tools needed for the space. Therefore, it is no longer necessary to pay attention to the various windows. All the tools necessary for the environment are in the corresponding space.

  In order to provide the acoustic engineer with an overview of all audio signals at any time, the adaptive mixing space already described with reference to FIGS. 3a and 3b is used. It can be compared with a conventional mixing desk that only displays active channels. In the adaptive mixing space, audio object information is presented instead of simple channel information. As already indicated, these objects are associated with the input channels of the WFS rendering apparatus by the mapping means 18 of FIG. Apart from the adaptive mixing space, there is also a so-called timeline space. This provides an overview of all input channels. Each channel is represented as having its corresponding object. Although automated channel association is preferred for simplification, the user can use object-to-channel association.

  Another space is a placement and editing space. This shows the scene in a 3D view. This space allows the user to record and edit the movement of the sound source object. The movement may be generated using a joystick or other input / display device known as a graphic user interface, for example.

  Finally, there is an indoor space. This supports the room simulation module 124 of FIG. 4 to provide room editing functions. Each room is described by a specific set of parameters stored in a room default library. Depending on the indoor model, various graphic user interfaces may be used together with various types of parameter sets.

  Depending on the situation, the inventive method for generating the audio representation may be implemented in hardware or software. Embodiments can also be performed on digital storage media, particularly floppy disks or CDs with electronically readable control signals. Thus, it may be associated with a programmable computer system that performs the inventive method. Thus, in general, when the computer program product is executed on a computer, the present invention also comprises a computer program product having the program code stored on a machine readable carrier that performs the inventive method. In other words, when the computer program is executed on a computer, the present invention can therefore be implemented as a computer program comprising program code for executing the method.

FIG. 2 is a block circuit diagram of an inventive device for generating an audio representation. FIG. 2 is a schematic explanatory diagram of a user interface for the concept shown in FIG. 1. FIG. 3a is a schematic illustration of the user interface of FIG. 2, according to one embodiment of the present invention. FIG. 3b is a schematic illustration of the user interface of FIG. 2 according to another embodiment of the present invention. FIG. 2 is a block circuit diagram of an inventive device according to a preferred embodiment. FIG. 4 is a time diagram of an audio scene having various audio objects. FIG. 6 is a 1: 1 conversion comparison between object and channel and object channel assignment for the audio scene shown in FIG.

Claims (16)

A device for generating, storing and editing audio representations in an audio scene,
Audio processing means (12) for generating a plurality of speaker signals from a plurality of input channels (EK1, EK2, ..., Ekm) (16);
Means (10) for providing an object-oriented description of an audio scene, wherein the object-oriented description of the audio scene includes a plurality of audio objects, the audio object being associated with an audio signal, a start time and an end time; ,
A first audio object is assigned to the input channel, a second audio object whose start time is after the end time of the first audio object is assigned to the same input channel, and the start time is assigned to the first audio object. An object-oriented description of an audio scene configured to assign a third audio object after the start time and before the end time of the first audio object to another of the plurality of input channels is provided in the audio processing apparatus. A device comprising mapping means (18) for mapping to a plurality of input channels.   The apparatus of claim 1, wherein the audio processing means (12) comprises wavefront synthesis means (122) configured to calculate a plurality of speaker signals of the speakers, wherein the positions of the plurality of speakers are known.   The audio object is further associated with a virtual position, and the audio processing means (12) is configured to take into account the virtual position of the audio object in generating a plurality of speaker signals. 2. The apparatus according to 2.   The apparatus according to claim 1, wherein the audio processing apparatus is connected only to the providing apparatus via the mapping apparatus and receives audio object data to be processed.   The number of input channels of the audio processing means is preset to be less than the allowable number of audio objects in the audio scene, and at least two audio objects are presented so as not to overlap in time. 4. The apparatus according to any one of 4.   A user interface (20) is further provided, the user interface comprising a number of individual user interface channels, wherein the user interface channel is associated with an input channel of the audio processing device, and the user interface (20) comprises a mapping means (80 6. An apparatus according to any one of the preceding claims, which identifies an audio object that is just assigned to a user interface channel at a certain time.   7. The apparatus according to claim 6, wherein the user interface (20) is configured to identify a user interface channel associated with an input channel of the audio processing means that is currently assigned one audio object. .   The user interface is configured as a hardware mixing console having hardware operation means for each user interface channel, and each hardware operation means is associated with an indicator that identifies the currently active user interface channel. The apparatus according to claim 7.   The user interface comprises a graphical user interface configured to display only user interface channels associated with the input channels of the audio processing means currently assigned one audio object on the electronic display device. Item 8. The device according to Item 7.   The user interface (20) further comprises means for manipulating the user interface channel, which is configured to manipulate an audio object assigned to an input channel of the audio processing means (12) corresponding to the user interface channel, The interface connects to the providing means (10), replaces the audio object with its manipulated version, and the mapping means (18) replaces the manipulated version with the manipulated version instead of the audio object as input to the audio processing means (12). 10. Apparatus according to any of claims 6 to 9, configured to be assigned to a channel.   The apparatus according to claim 10, wherein the operating means is arranged to change the position, type or audio signal of the audio object.   The user interface is configured to indicate a temporal occupancy of the user interface channel, where the temporal occupancy represents a temporal sequence of audio objects assigned to the user interface channel, and the user interface is further 10. An apparatus according to any of claims 6 to 9, wherein the apparatus is configured to specify a current time (37) in a typical occupancy.   The user interface (20) is configured to show temporal occupancy as a time axis, which comprises an indicator (37) that moves over time, along with assigned audio objects that are proportional to their length. The apparatus according to claim 12. The providing means (10) is configured to be able to group audio objects so that the grouped audio objects are clearly indicated by group information regarding group attribution,
The mapping means (18) sets the group information so that the manipulation of the group characteristics affects all members of the group apart from the fact that which input channel of the audio processing means is associated with the audio object of the group. 14. An apparatus according to any of claims 1 to 13, configured to store. A method for generating, storing and editing audio representations in an audio scene,
Generating (12) a plurality of speaker signals from a plurality of input channels (EK1, EK2,..., Ekm) (16);
Providing an object-oriented description of the audio scene, wherein the object-oriented description of the audio scene includes a plurality of audio objects, the audio object being associated with an audio signal, a start time, and an end time;
A first audio object is assigned to the input channel, a second audio object whose start time is after the end time of the first audio object is assigned to the same input channel, and the start time is assigned to the first audio object. By assigning a third audio object after the start time and before the end time of the first audio object to another of the plurality of input channels, the object-oriented description of the audio scene is Mapping (18) to a plurality of input channels.   A computer program having program code for executing the method of claim 15 when the program is executed on a computer.

Images