WO2016071697A1 - Interface graphique sphérique interactive pour manipulaton et la mise en place d'objets audio avec rendu ambiophoniques. - Google Patents

Interface graphique sphérique interactive pour manipulaton et la mise en place d'objets audio avec rendu ambiophoniques. Download PDF

Info

Publication number
WO2016071697A1
WO2016071697A1 PCT/GB2015/053355 GB2015053355W WO2016071697A1 WO 2016071697 A1 WO2016071697 A1 WO 2016071697A1 GB 2015053355 W GB2015053355 W GB 2015053355W WO 2016071697 A1 WO2016071697 A1 WO 2016071697A1
Authority
WO
WIPO (PCT)
Prior art keywords
data element
sound
environment
shows
data
Prior art date
Application number
PCT/GB2015/053355
Other languages
English (en)
Inventor
Aaron Lee SMILES
Original Assignee
Sinetic Av Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sinetic Av Ltd filed Critical Sinetic Av Ltd
Publication of WO2016071697A1 publication Critical patent/WO2016071697A1/fr

Links

Classifications

    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H1/00—Details of electrophonic musical instruments
    • G10H1/0008—Associated control or indicating means
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/16—Sound input; Sound output
    • G06F3/165—Management of the audio stream, e.g. setting of volume, audio stream path
    • G—PHYSICS
    • G11—INFORMATION STORAGE
    • G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
    • G11B27/00—Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
    • G11B27/10—Indexing; Addressing; Timing or synchronising; Measuring tape travel
    • G11B27/34—Indicating arrangements 
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04S—STEREOPHONIC SYSTEMS 
    • H04S7/00—Indicating arrangements; Control arrangements, e.g. balance control
    • H04S7/30—Control circuits for electronic adaptation of the sound field
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04S—STEREOPHONIC SYSTEMS 
    • H04S7/00—Indicating arrangements; Control arrangements, e.g. balance control
    • H04S7/40—Visual indication of stereophonic sound image
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2210/00—Aspects or methods of musical processing having intrinsic musical character, i.e. involving musical theory or musical parameters or relying on musical knowledge, as applied in electrophonic musical tools or instruments
    • G10H2210/155—Musical effects
    • G10H2210/265—Acoustic effect simulation, i.e. volume, spatial, resonance or reverberation effects added to a musical sound, usually by appropriate filtering or delays
    • G10H2210/281—Reverberation or echo
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2210/00—Aspects or methods of musical processing having intrinsic musical character, i.e. involving musical theory or musical parameters or relying on musical knowledge, as applied in electrophonic musical tools or instruments
    • G10H2210/155—Musical effects
    • G10H2210/265—Acoustic effect simulation, i.e. volume, spatial, resonance or reverberation effects added to a musical sound, usually by appropriate filtering or delays
    • G10H2210/295—Spatial effects, musical uses of multiple audio channels, e.g. stereo
    • G10H2210/301—Soundscape or sound field simulation, reproduction or control for musical purposes, e.g. surround or 3D sound; Granular synthesis
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2210/00—Aspects or methods of musical processing having intrinsic musical character, i.e. involving musical theory or musical parameters or relying on musical knowledge, as applied in electrophonic musical tools or instruments
    • G10H2210/155—Musical effects
    • G10H2210/265—Acoustic effect simulation, i.e. volume, spatial, resonance or reverberation effects added to a musical sound, usually by appropriate filtering or delays
    • G10H2210/295—Spatial effects, musical uses of multiple audio channels, e.g. stereo
    • G10H2210/305—Source positioning in a soundscape, e.g. instrument positioning on a virtual soundstage, stereo panning or related delay or reverberation changes; Changing the stereo width of a musical source
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2220/00—Input/output interfacing specifically adapted for electrophonic musical tools or instruments
    • G10H2220/091—Graphical user interface [GUI] specifically adapted for electrophonic musical instruments, e.g. interactive musical displays, musical instrument icons or menus; Details of user interactions therewith
    • G10H2220/101—Graphical user interface [GUI] specifically adapted for electrophonic musical instruments, e.g. interactive musical displays, musical instrument icons or menus; Details of user interactions therewith for graphical creation, edition or control of musical data or parameters
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2220/00—Input/output interfacing specifically adapted for electrophonic musical tools or instruments
    • G10H2220/091—Graphical user interface [GUI] specifically adapted for electrophonic musical instruments, e.g. interactive musical displays, musical instrument icons or menus; Details of user interactions therewith
    • G10H2220/101—Graphical user interface [GUI] specifically adapted for electrophonic musical instruments, e.g. interactive musical displays, musical instrument icons or menus; Details of user interactions therewith for graphical creation, edition or control of musical data or parameters
    • G10H2220/131—Graphical user interface [GUI] specifically adapted for electrophonic musical instruments, e.g. interactive musical displays, musical instrument icons or menus; Details of user interactions therewith for graphical creation, edition or control of musical data or parameters for abstract geometric visualisation of music, e.g. for interactive editing of musical parameters linked to abstract geometric figures
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2220/00—Input/output interfacing specifically adapted for electrophonic musical tools or instruments
    • G10H2220/155—User input interfaces for electrophonic musical instruments
    • G10H2220/401—3D sensing, i.e. three-dimensional (x, y, z) position or movement sensing.
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2250/00—Aspects of algorithms or signal processing methods without intrinsic musical character, yet specifically adapted for or used in electrophonic musical processing
    • G10H2250/471—General musical sound synthesis principles, i.e. sound category-independent synthesis methods
    • G10H2250/511—Physical modelling or real-time simulation of the acoustomechanical behaviour of acoustic musical instruments using, e.g. waveguides or looped delay lines
    • G10H2250/531—Room models, i.e. acoustic physical modelling of a room, e.g. concert hall
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04H—BROADCAST COMMUNICATION
    • H04H60/00—Arrangements for broadcast applications with a direct linking to broadcast information or broadcast space-time; Broadcast-related systems
    • H04H60/02—Arrangements for generating broadcast information; Arrangements for generating broadcast-related information with a direct linking to broadcast information or to broadcast space-time; Arrangements for simultaneous generation of broadcast information and broadcast-related information
    • H04H60/04—Studio equipment; Interconnection of studios
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04S—STEREOPHONIC SYSTEMS 
    • H04S2400/00—Details of stereophonic systems covered by H04S but not provided for in its groups
    • H04S2400/11—Positioning of individual sound objects, e.g. moving airplane, within a sound field
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04S—STEREOPHONIC SYSTEMS 
    • H04S2400/00—Details of stereophonic systems covered by H04S but not provided for in its groups
    • H04S2400/13—Aspects of volume control, not necessarily automatic, in stereophonic sound systems

Definitions

  • the present invention relates to a data processing and comprehension method comprising a method of inputting data in a first format and outputting the data aurally.
  • the invention relates to a method and practice of electronic sound/physical modelling synthesis; whereby the sound is synthesised by interlinked visual and physical variables, which may or may not be in 3 -dimensions (3-D), and may or may not be in direct relationship with 3D sound modelling via a 3D graphical user interface (GUI).
  • GUI graphical user interface
  • DJ disk jockey
  • a disk jockey relies on many tools to create a musical performance. For example, vinyl records may be manipulated on multiple turntables, or multiple compact disks (CDs) may be manipulated, to create a variety of sound effects.
  • the disk jockey may vary the speed, pitch, direction, and sequence of playback and may apply a variety of effects to the played back music.
  • EDM artist Electronic Dance Music artist
  • the sheet music contains musical information known as a composition, which is written by a composer.
  • the performance of a composition is led and directed by, what is commonly called, a conductor.
  • a DJ, EDM artist, or conductor makes decisions on the sequence of pieces of music over the course of the show.
  • the artistic value is more relative to curation, than creation.
  • a composer transcribes musical parts, which are scored to represent where/when they will be played in the overall musical piece. Each piece (or section) is given a tempo, time signature, and a musical key.
  • each part is written in sync with the overall tempo, time signature, and key of the piece, so that each individual part fits with the rhythmic, harmonic, and melodic structure of the overall piece.
  • the transcript of the compositional parts would be distributed to each of the separate instrumentalists, and a conductor would instruct each instrumentalist where to sit (soundstaging), when and how to play, and when to stop.
  • Each conductor adds his own personal artistry to each composition. In the practices of live popular music, the role of the conductor is omitted.
  • the composition and parts are rehearsed and usually performed, or improvised, without formal direction or use of sheet music (transcripts).
  • the separate instrumental parts are arranged on a timeline sequence displayed via a graphics engine, usually on a 2 -dimensional graphical user interface.
  • This sequence is similar to classical sheet music.
  • the sequence is first given a set tempo and time signature, and then parts are added to the timeline, usually in individual rows, to compose a piece of music.
  • This piece is usually exported as a digital media file, which is then presented, usually by a DJ, as a piece of recorded music.
  • parts are arranged in 2- dimensional rows and columns. However, an overall piece of music is not pre- composed. Individual parts are composed, but not pre-sequenced to complete an entire piece of music.
  • the parts are stored as individual loops, which are automatically analysed and processed to match any tempo and time signature set by the composer. These loops are sequenced in real-time during a presentation of music. This is similar to a conductor of music, without the physical placement/positioning of instrumental parts on stage (known as soundstaging).
  • Soundstaging is the act of creating, or creating the perception, of a soundstage in the presentation of a piece, or pieces, of music.
  • soundstaging is the arrangement of instrumentalists on stage, in front of a conductor, and/or audience, and how the audience perceives the arrangement. This perception is usually created by the physical placement of the instruments i.e. left or right, rows back or forth, and sometimes height or depth (if they are situated in the orchestra pit).
  • Modern technology allows more control over the delivery of the soundstage via systems comprising of microphones, mixing desks and loudspeakers. Using individual volume, equalisation, and panning controls, as well as, special sound effects (such as Reverb), these systems allow for live control of the overall mix of the sound, as perceived by the audience.
  • a similar principle is applied to recorded music and electronic music production, where tracks are mixed down to project the desired soundstage to the listener. For example, for a pop track, the vocalist will be at the front/centre of the soundstage, whereas, for a house/techno track, the kick drum will be at the front/centre of the soundstage.
  • the creation of sound can be organic or synthetic.
  • Organic sound production involves traditional instruments made from organic material i.e. wood and metal.
  • Synthetic sound production involves electronic instruments, or synthesisers, made from electronic components and computer processors. Sounds produced by traditional musical instruments are embodied by the shape, function, and material of the instrumental object, and are recognised equally by their visual form.
  • This method is limited to predefined scenarios and known sounds, and lacks the provision for synthesising original complex frequencies/sounds.
  • current technologies are limited in the type of data they can use as an input for mixing and manipulating sound outputs, and rely heavily on using sound inputs.
  • One example of using visual representations to manipulate pre-recorded sound is US6,898,291.
  • Various parameters of sound (frequency, volume for example) are mapped to parameters of a visual image displayed in a three- dimensional environment. Changing the parameters of the visual image (size, location, texture, for example) change the corresponding parameter of the prerecorded sound.
  • Other types of inputs are just not envisaged, and therefore data outputs are limited to being aural, and merely being variations of the sounds input to such a system.
  • the present invention aims to address these problems by providing a data comprehension method, comprising of inputting at least a first data element; displaying the first data element in a visual format; Manipulating the first data element in the visual format using pre-defined gestures to create a modified data element; and outputting the modified data element aurally.
  • One advantage of the present invention is to be able to be virtually present within the audio mix visually, and navigate around the mix in real-time, at a real-world scale, as if the user is a conductor and the graphical user interface is a soundstage.
  • Another advantage is that rather than using pre-recorded sound, sounds are synthesized from the visual data element, creating the possibility of using a wide variety of inputs.
  • outputting the modified data element aurally comprises outputting as an output sound and presenting the output sound ambisonically.
  • the method further comprises the steps of assigning a first object to the first data element; displaying the first data element visually in the form of the first object, such that the pre-defined gestures manipulate the object; and assigning an aural component to the first object.
  • assigning a first object to the first data element displaying the first data element visually in the form of the first object, such that the pre-defined gestures manipulate the object; and assigning an aural component to the first object.
  • assigning an object to each data element and assigning an aural component to each object.
  • the aural component varies in accordance with the manipulation.
  • the first data element is modified within an environment data element and the method further comprises the steps of: displaying the environment data element in a visual format; and manipulating the environment data element in the visual format using pre-defined gestures to create a modified first data element.
  • the environment data element is selected from the group of: gravitational, atmospheric, meteorological, ambient, terrestrial, celestial, astrological, cosmological, or technological variables.
  • the method further comprises the steps of: assigning a first area to the environment in which the first data element exists; displaying the environment visually in the form of the first area, such that the first area is the active area, or soundstage; and when more than a first area is input, assigning an inactive area to further areas to create global environment having a navigable grid such that manipulating a position within the grid modifies the active area.
  • the first data element represents a physical, optical, tactile, odorous or aural variable.
  • the modified data element is output aurally via at least one sound emitting device.
  • the first data element represents a sound
  • the modified data element or environment forms part of a soundstage.
  • the visual format may be in 3-D, in which case the gestures are enacted in 3-D.
  • the visual format is in 2-D.
  • the visual format is displayed on a touch-sensitive screen device.
  • the visual format is displayed on a screen, and the first data element is manipulated via a remote touch sensitive device.
  • the object is rendered in the form of voxels, and the gesture influences the voxel nodes.
  • the present invention also provides a computer program product adapted to perform the above method.
  • the present invention also provides a computer-readable storage medium or data carrier comprising the computer program.
  • Figure 1 is a flow chart illustrating the method of the present invention
  • Figure 2 illustrates how the object position along the Z-axis determines the intensity (loudness) of the sound output
  • Figure 3 illustrates how the object position along the X-axis determines the stereo width (panning) left and right of the sound output
  • Figure 4 shows the outer side partitions of the 3D environment used to isolate object(s) within a partitioned area to a specific sound output
  • Figure 5 illustrates grouped together soundstages, with objects set within each environment and the current active soundstage in the centre;
  • Figure 6 illustrates map navigation and how the current active soundstage changes dependent on location data
  • Figure 7 illustrates an active soundstage can be anywhere on the map
  • Figure 8a shows waveforms A andB in phase
  • Figure 8b shows waveform B half a cycle forward of waveform A
  • Figure 8c shows waveform B half a cycle behind waveform A
  • Figure 8d shows waveforms A and B completely out of phase
  • Figure 9a shows a sine wave waveform cycle
  • Figure 9b shows a square wave waveform cycle
  • Figure 9c shows a triangle wave waveform cycle
  • Figure 9d shows a sawtooth wave waveform cycle
  • Figure 10a shows a 3D spherical object
  • Figure 10b shows a 3D cubic object
  • Figure 10c shows a 3D conal object
  • Figure lOd shows a 3D pyramidal object
  • Figure 11a shows a cube that has almost been transformed into a sphere
  • Figure lib shows a cube that has been rounded at the edges
  • Figure 11c shows a cube that has been indented on each of the sides from the centre
  • Figure lid shows a cube that has been fully rounded on all vertical sides to form a cylinder
  • Figure He shows a cube that has been tilted to one side to form a parallelogram shape
  • Figure llf shows a cube that has had one side tilted to form a trapezium shape prism
  • Figure llg shows a cube that has been formed into a hexagonal prism
  • Figure llh shows a cube that has been formed into a multi-sided diamond-like shape
  • Figure Hi shows a cube that has been rounded at the edges and corrugated on all sides
  • Figure llj shows a cube that has been formed into a curved pyramid
  • Figure Ilk shows a cube that has been formed into an indented cone
  • Figure 12a shows an equally combined sine waveform and square waveform
  • Figure 12 shows a square waveform with rounded corners
  • Figure 12c shows an indented square wave
  • Figure 12d shows a square wave with a curved top
  • Figure 12e shows a square wave that is ramped at the start
  • Figure 12f shows a square wave that is sloped to end
  • Figure 12 g shows a square wave that is ramped at start and end
  • Figure 12h shows a square wave that has been ramped at the start and sloped to end
  • Figure 12i shows a square wave with ripples at the peak, which may produce a damped oscillation
  • Figure 12 j shows a square wave that has a ramped curve at the start to peak and a sloped end
  • Figure 12k shows a square wave that has a ramped curve from the start to the end;
  • Figure 13 shows two different complex objects;
  • Figure 14 shows an object and how an object's [3D] dimensions relate to the Cartesian coordinate system
  • Figure 15 shows different combinations of cross-section slices of different object surface layer textures
  • Figure 16a shows a monophonic texture pattern traditionally notated for sheet music
  • Figure 16b shows a heterophonic texture pattern traditionally notated for sheet music
  • Figure 16c shows a homophonic texture pattern traditionally notated for sheet music
  • Figure 16d shows a homorhythmic texture pattern traditionally notated for sheet music
  • Figure 16e shows a polyphonic texture pattern traditionally notated for sheet music
  • Figure 17 shows a cross-section slice of an object, which displays each of the layers within the object and their relative distances between each layer;
  • Figure 18 shows a standard mobile media device, including built in camera, microphone, and light sensor;
  • Figure 19 shows a diagram of the magnetic interaction relationship between objects.
  • Figure 20 shows a diagram of an example speaker array completely surrounding a dance floor from all angles/directions.
  • the present invention provides a data processing and comprehension method, where different types of variables can be converted into sounds, enabling the user to manipulate or process the data to comprehend the data more easily.
  • the present invention utilises the input of at least a first data element, for example, a colour or a sound, and displays this first data element in a visual format, for example, as an image on the screen of a smart phone.
  • This visual display allows the manipulation of the first data element, using pre-defined gestures, for example, by pinching or swiping the image, to create a modified data element. For example, a sound may be displayed visually as a coloured square.
  • the square may be manipulated by changing its size, orientation, or the brightness or contrast of its colour.
  • the modified data element is output aurally. This may be instantaneous or sequentially. This may be through a loudspeaker, a pair of headphones or other sound device.
  • the step of displaying the sound as a coloured square involves assigning a first object to the first data element, so that the first data element is displayed visually in the form of the first object. This first object has a specific aural component assigned to it. For example, a yellow square will always output as the same tone.
  • a pre-assigned visual element such as a shape, resulting in the output of an aural element.
  • an object is assigned to each data element, and an aural component is assigned to each object.
  • many data elements may be assigned to any one object, if desired.
  • the aural component assigned to that object varies in accordance with the manipulation. For example, increasing the size of a yellow square increases the volume of the assigned tone.
  • the modified data element is output aurally via at least one sound outputting device. The outputting of the data as sound enables the user to comprehend the data by means of an ambisonic environment.
  • the first data element may represent any desirable quantity, for example, a physical, optical, tactile, odorous or aural variable.
  • the first data element may represent a sound, and the modified data element forms part of or is used within the soundstage.
  • the visual format may be in 3-D, such that the gestures are enacted in 3-D. This greatly enhances the manipulation possibilities, and creates a composition environment in 3-D.
  • the visual format is in 2-D, for example, displayed on a touch-sensitive screen if the visual format is displayed on a screen, and the first data element could be manipulated via a remote touch sensitive device.
  • the generated and output sound may also be presented in 3D, commonly referred to as ambisonic.
  • ambisonic is used to mean all 3D Sound/Periphonic Sound Spatialisation techniques, and includes, but is not limited to, technologies such as Distance-Based Amplitude Panning (DbAP), Vector-Based Amplitude Panning (VbAP), Higher Order Ambisonics, Holophony, Wave Field Synthesis, Virtual Microphone Control (ViMic), and 3 -Dimensional Highly Directive Virtual Microphone (3D-VMS).
  • DbAP Distance-Based Amplitude Panning
  • VbAP Vector-Based Amplitude Panning
  • Higher Order Ambisonics Holophony
  • Wave Field Synthesis Virtual Microphone Control
  • ViMic Virtual Microphone Control
  • 3D-VMS 3 -Dimensional Highly Directive Virtual Microphone
  • Typical gestures used in the control of mobile smart devices can be assigned to data, and used to manipulate data to produce a variety of sounds.
  • data is rendered in the form of voxels, and the gesture influences the voxel nodes.
  • Rendering sounds (and other data) as voxels creates objects that can be manipulated.
  • the data and application flow required to allow the manipulation of the voxel nodes is shown in Figure 1.
  • the term voxels is used to mean all values and units of 3D graphical space, including, but not limited to texels, and any combination such values and units, which may or may not include 2D values and units, such as, but not limited to pixels.
  • FIG. 1 is a flow chart illustrating the method of the present invention.
  • solid arrows represent the flow within the application, and hollow arrows represent the flow of data.
  • a user decides to run the application 1001, and accesses a Start menu 1002.
  • the user chooses between a Start File Menu 1004 and a Start Settings Menu 1006, depending upon whether it is desired to start an AV (audio visual) project or create settings for a project.
  • the Start File Menu is chosen, the user can chose to either construct a new project 1008 or load an existing project 1010.
  • an AV project is assigned at 1012, checking current AV project memory data at 1014 and loading the main menu to initialise the current project at 1016.
  • step 1018 is loaded at step 1018 from a search of filenames in the hard disk 1020, the data deserialised at 1022 and fed back to step 1012 where the AV project is assigned, and steps 1014 and 1016 repeated.
  • the Main Menu is loaded at step 1024 this can be used to select tools relating to the data display environment at step 1026, sound objects at step 1028 as well as the ingame menu at step 1030 and the settings menu at step 1032.
  • the ingame menu 1030 enables the user to reach the file menu, to save or create new AV projects once sounds have been manipulated.
  • the environment tools accessed at step 1026 can be used, for example, to focus gestures and therefore movements into a brush area on the surface or in the region where gestures will be made.
  • Object tools can be used to create new tools or vary the effect a gesture has on a particular display, which can be saved relative to individual projects (at step 1040) or in general (at step 1042).
  • a certain amount of reverb is applied (step 1044), or other effects.
  • Figure 2 shows how object positioning along the Z-axis determines the intensity (loudness) of the sound output. This is a key manner in which the output of the data aurally is managed. Intensity may or may not also be presented as volume, gain, loudness, magnitude, sound pressure level, velocity, and any other relative sound level variable.
  • Figure 3 shows object positioning along the X-axis, which determines the stereo width (panning) left and right of the sound output. Again, this can be varied to give a wider range of aural data output formats. Stereo height (vertical panning) is applied to object placement along the Y-axis.
  • the Y-axis could or could not also be applied to sound equalisation (EQ) - position coordinates above the centre line apply higher EQ values, whilst position coordinates below the centre line apply lower EQ values.
  • the object size determines how much of an EQ bandwidth is applied i.e. a larger object would apply a larger EQ bandwidth.
  • the frequency bandwidth an object doesn't occupy will either be reduced or removed.
  • a duplicate object would need to be created, and both objects separately positioned in the respective spaces to apply the desired EQs.
  • Figure 4 shows the outer side partitions of the 3D environment used to isolate object(s) within a partitioned area to a specific sound output (see Figure 20 below). Further partitioning could be made to include the separate EQ bands if desired (or in another/EQ mode). The size and amount of partitions may be changed. The output assignment may be changed. The central un-isolated space uses all outputs, similar to surround sound stereo, which will present the output sound relative to the placement of the object within the un-isolated environment. Objects can be automated to follow defined paths to generate synchronous ambisonic panning effects.
  • Figure 5 shows grouped together soundstages, with objects set within each environment and the current active soundstage in the centre. This grouping forms a navigable map (as shown in Figure 6).
  • Figure 6 shows map navigation and the current active soundstage change dependent on location data.
  • Figure 7 shows that the active soundstage can be anywhere on the map, irrespective of territorial boundaries. All objects outside of the active soundstage area are stopped. All objects within the active soundstage area are played.
  • Figure 8 shows two waveforms, A and B. In Figure 8a the first waveform relationship shows both A and B in perfect phase, Figure 8b shows B forward half a cycle, Figure 8c shows B behind half a cycle, and Figure 8d shows both A and B completely out of phase.
  • Figure 9a shows a sine wave waveform cycle
  • Figure 9b shows a square wave waveform cycle
  • Figure 9c shows a triangle wave waveform cycle
  • Figure 9d shows a sawtooth wave waveform cycle
  • Figure 10a shows a 3D spherical object
  • Figure 10b shows a 3D cubic object
  • Figure 10c shows a 3D conal object
  • Figure lOd shows a 3D pyramidal object.
  • the waveform will be sinusoidal; B) If an object is cubic, the waveform will be a square wave; C) If an object is equilateral or isosceles prismatic and/or has rounded sides, then the waveform will be a triangle wave; D) If the shape is right-angle prismatic and/or has flat sides, then the waveform will be sawtooth.
  • Figure 11 shows a group of objects that originally started out as cubes and have been changed in various ways.
  • a cube is used as a base shape to serve as an example only.
  • Figure 11a shows a cube that has almost been transformed into a sphere.
  • Figure lib shows a cube that has been rounded at the edges.
  • Figure 11c shows a cube that has been indented on each of the sides from the centre.
  • Figure lid shows a cube that has been fully rounded on all vertical sides to form a cylinder.
  • Figure He shows a cube that has been tilted to one side to form a parallelogram shape.
  • Figure llf shows a cube that has had one side tilted to form a trapezium shape prism.
  • Figure llg shows a cube that has been formed into a hexagonal prism.
  • Figure llh shows a cube that has been formed into a multi-sided diamond-like shape.
  • Figure Hi shows a cube that has been rounded at the edges and corrugated on all sides.
  • Figure llj shows a cube that has been formed into a curved pyramid.
  • Figure Ilk shows a cube that has been formed into an indented cone.
  • Figure 12 shows the waveform shapes created by such shape manipulation, as described in Figure 11 - as in Figure 11 a cube is used as an example to explain shape manipulation, hereby a square wave is used as the base waveform shape to demonstrate the relationship between the two.
  • Figure 12a shows an equally combined sine waveform and square waveform, which may produce a large amount of high frequency loss.
  • Figure 12b shows a square waveform with rounded corners, which may produce a small amount of high frequency loss.
  • Figure 12c shows an indented square wave, which may produce an amount of low frequency loss.
  • Figure 13d shows a square wave with a curved top, which may produce a low frequency boost (accentuated fundamental).
  • Figure 12e shows a square wave that is ramped at the start, which may produce total high frequency loss.
  • Figure 12f shows a square wave that is sloped to end, which may produce a low frequency phase shift.
  • Figure 12g shows a square wave that is ramped at start and end, which may produce both a high and a low frequency loss.
  • Figure 12h shows a square wave that has been ramped at the start and sloped to end, which may produce a high frequency loss and low frequency phase shift.
  • Figure 12i shows a square wave with ripples at the peak, which may produce a damped oscillation.
  • Figure 12j shows a square wave that has a ramped curve at the start to peak and a sloped end, which may produce high frequency loss and low frequency phase shift.
  • Figure 12k shows a square wave that has a ramped curve from the start to the end, which may produce low frequency loss and low frequency phase shift.
  • Figure 13 shows two different complex objects.
  • Such complex objects may be included in a scene, included in a soundstage, or imported to the system via any of the aforementioned hardware input devices, including virtual reality headset spatial awareness to include objects present in the user's real environment, computer vision analysis of 2D or 3D images.
  • Figure 13a shows an animated motor vehicle.
  • the output sound production for this image may be relative to one (layer) atonal sine wave with added white noise and reverb.
  • Other visual effect analysis may be included (that cannot be represented in the diagram), such as, brightness, which would also apply relative effect to sound output.
  • Figure 13b shows an animated (pale green) house.
  • the output sound production for this image may be relative to a complex sound made up of two (layers) square and triangle wave frequencies at different musical tones - ##C1, A#3##.
  • Other visual effect analysis may be included (that cannot be represented in the diagram), such as, brightness, which would also apply relative effect to sound output.
  • Figure 14 shows an object and how an object's [3D] dimensions relate to the Cartesian coordinate system.
  • X-axis relates to object Width;
  • Y-axis relates to object Height;
  • Z-axis relates to object Depth.
  • the centre point of the Cartesian coordinate marker remains the centre of the object - plus (+) and minus (-) lengths of each axis can be changed independently from this centre point.
  • Figure 15 shows different combinations of cross-section slices of different object surface layer textures. These diagrams describe the relation of object surface layer texturing combinations in a defined proximity of one another and the possible resulting compositional texturing, a: Monophony: If there is only one object and the surface is set to default (no texture) then the sound/music will be monophonic i.e.
  • Heterophony 1 If there is only one object layer and the surface is set to have texture then the sound/music will be heterophonic.
  • Heterophony 2 If there is more than one object and the surface of one object is set to have no texture, but the other objects are set to have different textures, then the sound/music may become heterophonic.
  • Homophony If there is more than one object and all surface layers are set to have no texture, then the sound/music may become homophonic.
  • Homorhythm If there is more than one object and all surface layers are set to have texture, but all the textures are the same, then the sound/music may become homorhythmic.
  • Polyphony If there is more than one object and all surface layers are set to have texture, but all the textures are different, then the sound/music may become polyphonic.
  • Figure 16a shows a monophonic texture pattern traditionally notated for sheet music.
  • Monophony consists of melody without accompanying harmony.
  • Figure 16b shows a heterophonic texture pattern traditionally notated for sheet music.
  • Heterophony is characterized by the simultaneous variation of a single melodic line.
  • Such a texture can be regarded as a kind of complex monophony in which there is only one basic melody, but realized at the same time in multiple voices, each of which plays the melody differently, either in a different rhythm or tempo, or with various embellishments and elaborations.
  • this is created by, either, (1) one textured master layer, or, (2) an untextured master layer being accompanied by layers of varying textures.
  • Figure 16c shows a homophonic texture pattern traditionally notated for sheet music. In homophony two or more parts move together in harmony, the relationship between them creating chords. Hence, this is created by multiple untextured (monophonic) layers.
  • Figure 16d shows a homorhythmic texture pattern traditionally notated for sheet music. Homorhythm is similar to homophony, however, is only rhythmically similar, not harmonically. Hence, this is created by multiple textured layers, but all textures being similar.
  • Figure 16e shows a polyphonic texture pattern traditionally notated for sheet music. Polyphony consists of two or more simultaneous lines, all of independent melody. Hence, this is created by multiple textured layers, with all textures being different.
  • Figure 17 shows a cross-section slice of an object, which displays each of the layers within the object and their relative distances between each layer.
  • the centre distance is the size of the core.
  • the distance between each layer may be changed.
  • the distance between each layer determines the audible level of each layer in relation to each layer present in the object.
  • the surface layer is always the master level. If any two layers are close together, their levels will be equal, but the further apart from the surface layer, the lower the level will be in relation.
  • the order of the layers may or may not be changed accordingly for mixing purposes.
  • Figure 18 shows a standard mobile media device, including built in camera, microphone, and light sensor. Many mobile devices also feature GPS location services, Bluetooth, Wi-Fi, and accelerometers. Such I/Os and I/O data maybe used as control data in the generation of sound in such a method and processes, as described within this system. For example, a live feed of facial tracking to recognise a busy environment, location data to determine urban or rural landscapes, light sensitivity to control various effects, etc.
  • sensors may be used as inputs to control the data element(s). This may include, but is not limited to, computer vision, kinetic/motion, gesture, and biofeedback - namely such sensors may include, but are not limited to, Leap Motion, which uses infrared LEDs for precise 3D hand and finger recognition and tracking; Google Soli, which uses radar technology that can track sub -millimetre hand and finger motions at high speed and accuracy; Microsoft Kinect/Softkinetic DepthSense, which uses Time-of-Flight CMOS sensors for 3D full-body recognition and tracking; Myo, which uses mechanomyogram (MMG) sensors that observe muscle activity in the forearm(s) for hand and finger movement and gesture recognition and tracking.
  • Leap Motion which uses infrared LEDs for precise 3D hand and finger recognition and tracking
  • Google Soli which uses radar technology that can track sub -millimetre hand and finger motions at high speed and accuracy
  • Microsoft Kinect/Softkinetic DepthSense which uses Time-of-Flight CMOS sensors for 3D
  • Figure 19 shows a diagram of the magnetic interaction relationship between objects.
  • Objects may be given magnetic properties of varying strengths, either with a plus (+) or a minus (-) polarization. When objects are in proximity of such fields, this determines whether they are attracted to or repelled away from one another. The size of such proximal area is determined by the strength of the magnetic field applied to any object.
  • Figure 20 shows a diagram of an example speaker array completely surrounding a dance floor from all angles/directions. The polygons represent speakers that may each be full speakers systems (providing all frequency ranges, including high, mid, low, sub), or may be separated by EQ ranges i.e. high ranges, mid ranges, low ranges, sub ranges.
  • the present invention provides a system for sound mixing and sequencing in 3 -dimensions (3D), in real-time, whereby sound sources are visually displayed via a graphics engine in 3D, within a 3D environment.
  • the processes described herein refers to 3 -dimensions, but may or may not be represented in 2 -dimensions, digitally displayed and modified digitally via a sound and graphics engine, or calculated linearly through transcription.
  • the 3 -dimensional environment herein is described using a Y-up, left-handed (Cartesian) coordinate system, but could be applied to any similar method.
  • Sensor data may or may not be used to either control data, or as input data. This may or may not be camera data, microphone data, location data, biometric data, or uploaded multimedia.
  • Non-musical data from such sensors and/or media such as, motion, gesture, position, orientation, location, ambient noise, face tracking, heart rate, colour analysis, contrast, etc. may or may not be used as input data for the invention to generate further manipulation.
  • the entire environment is a virtual soundstage.
  • Each object's position within an active soundstage can be moved around, which is output in real-time via a sound system.
  • Manipulation of object position can control sound output panning and intensity.
  • Positioning of objects in the environment can control sound frequency equalisation.
  • the outer sides of the 3D environment can be partitioned to isolate object(s) within a partitioned area to a specific sound output or EQ.
  • An example output arrangement may represent a dome of tiered speaker arrays/rings surrounding a dance floor or concert hall. This allows the user to mix objects within isolated sources and points of the soundstage.
  • Each virtual soundstage represents real-world environmental conditions, which can be controlled. These conditions affect variables of the sound, as they would in reality. Such variables include, but are not limited to atmospheric pressure, temperature, and humidity.
  • each soundstage space can be controlled.
  • the size of the space controls a reverb effect that is applied to each object within the space.
  • a larger space will apply a wide reverb, whereas a smaller space will apply a narrow reverb.
  • the overall perceived effect on the sound will be dependent on each object position.
  • Each space can also be set as open or closed, which sets the reverb to open or closed reverb.
  • the type of reverb applied is representative of natural reverb that would occur in a similar real-world environment.
  • Completed soundstages can be grouped together to form an extended environment map, or grid, which can be navigated in an upward, downward, sideways, or diagonal direction, like a virtual open world. As the user navigates around the map, the current active soundstage changes dependent on the user's current location. Through this navigation, the user can sequence entire presentations of music.
  • Example 2 Visual and Physical Modelling Synthesis
  • the invention may be used as a process, method, and/or system for visual-physical modelling synthesis of complex sounds using frequencies of visual light and properties of physical matter, and natural or unnatural user and environmental conditions in 2 -dimensions and/or 3 -dimensions, whereby the visual, matter, and shape data embodies the sound to be produced (similar to traditional instruments).
  • Visual and physical data such as hue, brightness, weight, and shape, are analysed and used by the system's algorithm to process and synthesise sound, using non- instrumental/non-musical values to create objects for physical modelling synthesis.
  • the data processing described herein refers to 3 -dimensions, but may or may not be represented in 2 -dimensions, digitally displayed and modified digitally via a sound and graphics engine, or calculated linearly through transcription.
  • Sensor or media data may or may not be used to either control data or as input data. This may include, but not limited to, motion and gesture data, camera data, microphone data, location data, biometric data, electroencephalography (EEG) data, mechanomyogram (MMG) data, uploaded multimedia, and/or any combination of the aforementioned.
  • Non-musical data received by such sensors and/or media such as, motion, gesture, position, orientation, location, ambient noise, face tracking, heart rate, colour analysis, contrast, etc.
  • the invention may or may not be used as input data for the invention to generate audio-visual synthesis output and/or for further manipulation. This could, for example, range from controlling the system via a motion sensor by hand within a 3D virtual reality (VR) world, or analysing a photographic image or live camera feed to generate sound.
  • VR virtual reality
  • Sound is synthesised from variables of light frequencies and visual effects. These are broken down into individual parameters of visual light frequency, which translate and control variables that synthesise audible sound frequencies. More than one frequency can be used; each individual frequency is hereby called a layer(s). Each layer adds a signal/tone that can be individually edited and controlled as described throughout this document. This is similar to additive synthesis. These layers are interconnected and serve as part of a whole object.
  • Colour/Hue sets the actual frequency of the base tone. This may or may not be more or less the range shown below, or may or may not be more or less a full 256 RGB or CMYK colour range. Ranges of colours larger, or smaller, than what is shown below will be spread, or condensed, between the colour bands shown below, however, such bands remain within their colour of the spectrum i.e. in a colour range of 256 colours, all sound frequency between the musical note range F3 to A3 would still remain within the yellow spectrum, but in this instance Yellow would be divided into different tones of yellow.
  • White contains all possible frequencies, which represents white noise.
  • Black has no frequency and represents atonal sounds, such as, but not limited to drum hits.
  • Infrared contains very low-end frequencies down to frequencies below human hearing.
  • Ultraviolet contains extreme high-end frequencies, up to those above human hearing.
  • the volume of a colour determines the amplitude of a layer (between a predefined range 0-maximum).
  • the visual Brightness/Luma of a colour determines the amount of additional white noise (between a predefined range 0-maximum) added to a layer, or object.
  • the visual tint effect of a colour determines an amount of High Pass Filter (HPF) applied (between a predefined range 0-maximum) to a layer, or object. The maximum amount of effect applied will fully kill the audible sound so that nothing can be heard.
  • the visual Shade effect of a colour determines the amount of Low Pass Filter (LPF) applied (between a predefined range 0-maximum) to a layer, or object. The maximum amount of effect applied will fully kill the audible sound so that nothing can be heard.
  • the visual saturation effect of a colour determines the amount of Band Pass Filter (BPF) applied (between a predefined range 0-maximum) to a layer, or object. The maximum amount of effect applied will fully kill the audible sound so that nothing can be heard.
  • the visual contrast effect parameter (of a predefined range +100% and -100% of 0) controls the phase relationship between two or more layers of an object, by moving their cycle forwards, or backwards. If more than one layer is used, this can be used to move each layer in and out of phase.
  • the visual sharpness/acutance effect parameter (of a predefined range +100% and -100% of 0) determines the accuracy of a layer in relation to the musical time (tempo).
  • the layer When set at 0, the layer will be in sync with the tempo; when set at +100%, the layer will be a full musical beat ahead of the tempo; when set at -100%, the layer will be a full musical beat behind the tempo - the difference between the percentage range respectively represents such time difference offsets i.e. +50% would set the layer ahead of the tempo by half a beat, whereas, -25% would set the layer behind a quarter of a beat. This could be viewed as syncopation (delay and anticipation).
  • the visual invert effect inverts all layers of an object. This flips each layer to a polar opposite frequency between 20HZ and 20,000HZ - the centre point for such mirroring is 9990HZ. This is similar to frequency inversion.
  • Visual definition controls the fidelity (bit rate) of a layer, or object. Higher pixel rate and better aspect ratios provide higher audio bit rates, and vice versa. Lowering the definition to blur or distort the visual applies degrees of distortion to a layer, or object.
  • the distance between layers may or may not be controlled, which controls the audio level (not to be confused with volume/amplitude) relationship between each of the layers present in any one object.
  • 3D Waveshape Modelling The shape of an object directly controls the audio waveform shape.
  • the basic/fundamental shapes and waveform shapes are:
  • shapes are the basis to how shape data of objects translate to the output sound/waveform. This is not limited to basic shapes, and may or may not include 3D shapes. Other considerations may include polyhedrons, superquadrics, superellipsoids, supertoroids, and fractaloids. Any complex shape is generally a combination of basic shapes. Similarly, more complex waveform shapes are formed from combinations of the basic waveform shapes. If such shapes are used, then respective complex waveform shapes will be generated.
  • the present invention provides fluid modelling of 3D animated shapes, which are user- defined and can be infinitely graphically complex, and include motion/animation (change shape over time), which are instantaneously calculated, set/modulated, and output as a corresponding variable complex audio waveform shape.
  • the system provides a method where physical properties of matter determine the timbre aspects of sound. Physical properties can be assigned to objects that translate as properties of sound.
  • the physical weight data of an object determines the speed of attack of the amplitude envelope (velocity) and amount of amplitude modulation (AM)/tremolo applied (between 0-maximum).
  • the physical state data of an object determines the amount of frequency modulation (FM)/vibrato applied (between 0- maximum). If an object(s) is used in a live environment/active soundstage, such as that described in Ambisonic Soundstage above, where a temperature control is applied, this can change the state of an object i.e. heating above a set melting/boiling point may melt a solid object into a liquid state, or boil a liquid object into a gas state, or cooling below a set freezing point may freeze a liquid into a solid state.
  • FM frequency modulation
  • the physical density data of an object determines (between a predefined range 0- maximum) the amount of oscillation/intensity applied to AM and FM (between 0 - maximum).
  • the physical width data of an object determines the stereo width of the sound (between 0-maximum). This is measured both left and right of the centre point of an object. Distance measurements left and right may or may not be adjusted independently.
  • the physical height data of an object determines the stereo height of the sound, and the length of decay and release of the amplitude envelope (between 0-maximum). This is measured both up and down of the centre point of an object.
  • the upward measurement determines the length of decay, while the downward measurement determines the length of release.
  • Distance measurements both up and down may or may not be adjusted independently, although only by the overall height measurement of an object affects stereo height.
  • the physical depth (or length) data of an object determines the length of sustain of the amplitude envelope (between 0- maximum). This is measured both back and fourth of the centre point of an object. Distance measurements both backward and forward may or may not be adjusted independently, although only the overall distance measurement affects the length of sustain.
  • Other first data elements may include, but is not limited to, viscosity, half-life, buoyancy, or absorbency.
  • iv) Physical Texture Composition The physical/visual surface texture of layer(s)/object(s) determines the added (musical) harmonic, rhythmic, and melodic texture of objects in close proximity to one another. Such textures may include, but not limited to, flat/smooth (no texture), or grainy, course, or rough.
  • the number and/or combination of object/layer surface textures within specific vicinity determine such added musical texture. This proximity can be user defined and is also directly affected by magnetism and other naturally occurring relationships between objects and conditions of the environment.
  • the amount/level of visual texturing applied to layer(s)/object(s) may or may not be increased or decreased, and such levelling may or may not have an affect on the level of complexity and texturing added to the sound/music. This may or may not also include contrapuntal texturing, such as, but not limited to, canon and fugue.
  • Flanger, phaser, and chorus are all applied through layering and controlling layers in different ways. If a delayed layer is added to an original (master) layer with a continuously variable delay (usually smaller than 10ms), then one layer is slowed, so that it becomes out-of-phase with its partner, and produces a phasing effect, this creates a Flanger effect.
  • Moving layers in and out of phase with the master makes the phasing effect appear to slide up the frequency spectrum. This phasing up-and-down the register can be performed rhythmically. If a layer (s) is split, and a portion is filtered with an all-pass filter to produce a phase-shift, and then the unfiltered and filtered signals are mixed, this produces a Phaser effect.
  • a delayed layer is added to a master layer with a constant delay, and the delay is short, in order not to be perceived as echo, but above 5ms to be audible - If the delay is too short, it will destructively interfere with the un-delayed layer, and create a flanging effect - the delayed signals will be slightly pitch shifted to more realistically convey the effect of multiple voices, this produces a Chorus effect.
  • the reflection of a surface of a layer determines the amount, and type, of reverberation applied to a layer, or an object.
  • Such surface type effects may or may not include, but not limited to, diffusion, emission, specular, and opacity.
  • Such reverb sound effect parameters may or may not include, but not limited to, size, outdoor/indoor, and delay.
  • a strobe/pulsate light effect applied to an audio transform will affect the sound at different intervals depending on the rate of strobe. This could be applied to a beat ratio of 1:1, 1:2, 1:4, 1:18, or 1:16.
  • the Hollowness of the core an object determines an amount of audio DSP effect, echo, which is applied - more echo is applied to objects that are hollower.
  • a magnetism value can be applied to objects. This determines how objects may interact in proximity to other objects, if used in a live environment/active soundstage (such as described in Ambisonic Soundstage). Polarisation would mean objects would either attract or repel. The strength of Magnetism applied would determine the force and distance such an effect would have on neighbouring objects.
  • the general environment represents a [virtual] physical environment, to which real- world laws, physics, variables, and/or conditions called environment data elements can be set, which manipulate first data elements/objects within the environment.
  • An environment data element represents either an environmental condition that defines the physical environment a first data element/object may be positioned in, or an environmental force/condition that acts upon a first data element/object and/or the physical environment simultaneously, for example, this may be a terrestrial environment data element, such as the shape or state of the landscape, the first data element(s) is in, or a meteorological environment data element, such as a rain shower, that acts upon the first data element within the environment.
  • the environmental data element may be displayed in a visual format, such that manipulating the environment data element in the visual format using pre-defined gestures creates a modified first data element/object.
  • This is analogous with the method outlined above in Example 1, where the data elements are modified using pre-determined gestures whilst the data element is in a visual format.
  • Environment data elements may be manipulated to affect objects within the environment, which, for example, may be selected from the group of; gravitational, atmospheric, meteorological, ambient, terrestrial, celestial, astronomical, astrological, cosmological, or technological variables.
  • the environment can also be used to create a navigable map or grid, either by dividing the global environment into separate soundstage areas, or by generating new soundstage areas, which can be connected to the existing environment, enabling the user to move around the environment/areas, in an upward, downward, sideward, or diagonal direction, such that the global environment is a 3D open world.
  • the method of the present invention involves assigning a first area to the environment. Next is displaying the first area visually. This causes the first area to become the active soundstage, or the active area. When more than a first area is created or input, it is possible to assign inactive areas to the other areas. Using both the connected active area and inactive areas creates a navigable map, or grid, and is collectively known as the global environment.
  • only one soundstage environment area is active at any time, which is determined by the user's location within the global environment.
  • the user perspective is generally at front-centre of the main/selected/active soundstage - a similar perspective to that of the position of a conductor, or mix engineer, of an orchestra, or stage, in a live performance situation.
  • the active soundstage area changes in response.
  • the active area can be modified either by changing position within the navigable map or grid, or by increasing or decreasing the size of the scope of the active soundstage/area; Increasing the size of scope activates more of the global environment, and creates less inactivity - Decreasing the size of the scope deactivates more of the global environment, and creates more inactivity.
  • active and inactive soundstages analogous to the concept of the active and inactive areas, is described above with reference to Figures 5 and 6.
  • the connected areas represented and described are in 2D and relative to a real- world 2D map or atlas of a 3D planet, such that any navigation off the represented 2D map in any direction would place the location point at a polar opposite position on the map.
  • a gravitational environment data element value may be applied to the global environment, soundstage areas, or specified sections of a soundstage. The gravity value would affect objects within the selected area(s) as it would naturally in a real- world environment, dependant on the properties of the object(s).
  • this may be, but not limited to, gravity becoming lighter and object(s) floating, or gravity becoming heavier and object(s) dropping - objects may or may not float out of the audible area of the environment, or, may rest or bounce off the floor depending on the gravity value applied and the material value given to the object and the floor of the environment.
  • Other environmental values may affect such behaviours, for example, but not limited to, the state and material of the environment itself.
  • An ambient temperature environment data element value may be applied to the global environment, soundstage areas, or specified sections of a soundstage.
  • the temperature value would affect objects within the selected area(s) as it would naturally in a real-world environment, dependent on the properties of the object(s). For example, this may be, but not limited to, temperature decreasing and object(s) condensing or solidifying, or temperature increasing and object(s) melting or evaporating.
  • Other environmental values may affect such behaviours, for example, but not limited to, the state and material of the environment itself.
  • Other atmospheric environment data elements may include, but not limited to; pressure, humidity, and atmospheric electricity, or meteorological environment data elements, including weather conditions that may modify the ambient and atmospheric environment data elements. Whereby the object(s) and output sound(s) will be naturally affected relative to such first data elements adopting such environment data element changes as they would in a real-world environment.
  • Further environment data elements may or may not also include natural and unnatural phenomenon, such as, but not limited to; technological, such as, destructive weapons (i.e. electromagnetic pulse, bombs) and mechanical devices; terrestrial, such as, natural disasters; celestial, astrological, or astronomical, such as, supernova explosions/implosions, lunar changes, planetary alignments, ultraviolet radiation, solar flares.
  • a planet, sun, or moon may be selected as a environment data element, so that the atmosphere closely resembles that observed through scientific study; cosmological, such as, molecular scientific experiments, which may include colliding atoms and mimicking real-world results by colliding objects and sounds.
  • Different areas of the global environment or soundstages may be partitioned or user-defined and assigned different environment data elements.
  • Odours may be manipulated, for example, by assigning visual elements to scents, aromas or chemical representations of smells.
  • Software mechanics like artificial intelligence (AI) characteristics and behaviours may be applied to objects. Such characteristics may include proximal defence and attack response mechanisms - AV outputs would be defined by the combination of present audio and visual variables and the interaction between two or more objects being in proximity/contact. This may also include applying predefined paths for objects to follow, as described in Fig. 4.
  • Other such AI may be relative to environmental changes, such as temperature and humidity, which may change or fluctuate over a predefined time or event.
  • Animation, motion graphics, and or visual effects may be applied to Objects, which may include changes to any of the aforementioned definable variables of the Object, over a pre-defined time, subsequently applying automation to the aural output.
  • the described environment refers to standard conditions, such as, but not limited to, room temperature and pressure, and Earth's gravitational force and oxygen-rich gaseous environment.
  • the environment atmosphere can be fully modified, for example, but not limited to, any of the aforementioned properties/conditions, as well as, state and matter - solid (s), liquid (s), gas(es) - which may include, but not limited to, natural or unnatural/real or imaginary elements/materials (i.e. red mercury).
  • objects residing in the environment are affected by such changes as they would be affected in nature.
  • a computer program product adapted to perform the above method, or a computer- readable storage medium or data carrier comprising the program may be provided.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Acoustics & Sound (AREA)
  • Signal Processing (AREA)
  • Multimedia (AREA)
  • Theoretical Computer Science (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • General Health & Medical Sciences (AREA)
  • Human Computer Interaction (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • Processing Or Creating Images (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

La présente invention concerne un traitement de données audio et une interface utilisateur graphique 3D virtuelle consistant : en la représentation spatiale de sources sonores en tant qu'objets, en la manipulation des objets dans l'espace virtuel résultant en une modification de l'emplacement et des caractéristiques des sons réel pour produire un plateau son actif, grâce à une mise en correspondance des caractéristiques de sons réels et des propriétés d'objet virtuel. L'environnement de plateau son actif virtuel est sélectionné sans coupure par navigation dans une grille de multiples environnements de plateaux sons virtuels. Le procédé peut être appliqué à d'autres variables que son. Le procédé consiste à entrer un élément de données et à afficher l'élément de données dans un format visuel ; l'élément de données peut ensuite être manipulé dans le format visuel à l'aide de gestes prédéfinis pour créer un élément de données modifié, émis par voie acoustique sous forme d'un son spatial 3D (par exemple un son ambisonique (TM) ou ambiophonique sphérique).
PCT/GB2015/053355 2014-11-05 2015-11-05 Interface graphique sphérique interactive pour manipulaton et la mise en place d'objets audio avec rendu ambiophoniques. WO2016071697A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
GB1419740.4A GB2532034A (en) 2014-11-05 2014-11-05 A 3D visual-audio data comprehension method
GB1419740.4 2014-11-05

Publications (1)

Publication Number Publication Date
WO2016071697A1 true WO2016071697A1 (fr) 2016-05-12

Family

ID=52118772

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/GB2015/053355 WO2016071697A1 (fr) 2014-11-05 2015-11-05 Interface graphique sphérique interactive pour manipulaton et la mise en place d'objets audio avec rendu ambiophoniques.

Country Status (2)

Country Link
GB (1) GB2532034A (fr)
WO (1) WO2016071697A1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10466960B2 (en) * 2018-04-02 2019-11-05 Avid Technology, Inc Augmented reality audio mixing

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10292001B2 (en) 2017-02-08 2019-05-14 Ford Global Technologies, Llc In-vehicle, multi-dimensional, audio-rendering system and method
GB2575840A (en) * 2018-07-25 2020-01-29 Nokia Technologies Oy An apparatus, method and computer program for representing a sound space

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6898291B2 (en) * 1992-04-27 2005-05-24 David A. Gibson Method and apparatus for using visual images to mix sound
EP1919259A2 (fr) * 2006-10-23 2008-05-07 Sony Corporation Système, appareil, procédé et programme permettant de commander une sortie
US20080229200A1 (en) * 2007-03-16 2008-09-18 Fein Gene S Graphical Digital Audio Data Processing System
US20110271186A1 (en) * 2010-04-30 2011-11-03 John Colin Owens Visual audio mixing system and method thereof
EP2450880A1 (fr) * 2010-11-05 2012-05-09 Thomson Licensing Structure de données pour données audio d'ambiophonie d'ordre supérieur
US20140119581A1 (en) * 2011-07-01 2014-05-01 Dolby Laboratories Licensing Corporation System and Tools for Enhanced 3D Audio Authoring and Rendering

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6898291B2 (en) * 1992-04-27 2005-05-24 David A. Gibson Method and apparatus for using visual images to mix sound
EP1919259A2 (fr) * 2006-10-23 2008-05-07 Sony Corporation Système, appareil, procédé et programme permettant de commander une sortie
CN101175337A (zh) * 2006-10-23 2008-05-07 索尼株式会社 输出控制系统和方法、输出控制装置和方法、程序
US20080229200A1 (en) * 2007-03-16 2008-09-18 Fein Gene S Graphical Digital Audio Data Processing System
US20110271186A1 (en) * 2010-04-30 2011-11-03 John Colin Owens Visual audio mixing system and method thereof
EP2450880A1 (fr) * 2010-11-05 2012-05-09 Thomson Licensing Structure de données pour données audio d'ambiophonie d'ordre supérieur
US20140119581A1 (en) * 2011-07-01 2014-05-01 Dolby Laboratories Licensing Corporation System and Tools for Enhanced 3D Audio Authoring and Rendering

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
JEAN-MARC JOT: "Spatialisateur, User Manual - "Spat~"", ESPACES NOUVEAUX, 22 November 2012 (2012-11-22), Paris, pages 1 - 29, XP055246817, Retrieved from the Internet <URL:http://forumnet.ircam.fr/wp-content/uploads/2012/10/Spat4-UserManual1.pdf> [retrieved on 20160202] *

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10466960B2 (en) * 2018-04-02 2019-11-05 Avid Technology, Inc Augmented reality audio mixing

Also Published As

Publication number Publication date
GB2532034A (en) 2016-05-11
GB201419740D0 (en) 2014-12-17

Similar Documents

Publication Publication Date Title
US7589727B2 (en) Method and apparatus for generating visual images based on musical compositions
Momeni et al. Dynamic independent mapping layers for concurrent control of audio and video synthesis
Chowning Turenas: the realization of a dream
Pressing Some perspectives on performed sound and music in virtual environments
Barrett Interactive spatial sonification of multidimensional data for composition and auditory display
WO2016071697A1 (fr) Interface graphique sphérique interactive pour manipulaton et la mise en place d&#39;objets audio avec rendu ambiophoniques.
Bardelli et al. A sonification of the zCOSMOS galaxy dataset
McGee et al. Sound element spatializer
Bain Real time music visualization: A study in the visual extension of music
Momeni Composing instruments: Inventing and performing with generative computer-based instruments
Lossius Sound space body: Reflections on artistic practice
McGee Scanning Spaces: Paradigms for Spatial Sonification and Synthesis
Stockmann et al. A musical instrument based on 3d data and volume sonification techniques
Pottier et al. Interpretation and space
Bernardo et al. The Smart Stage: Designing 3D interaction metaphors for immersive and ubiquitous music systems
Sánchez Carranco Vibrating instruments in virtual reality: A cohesive approach to the design of virtual reality musical instruments
Opie Granular synthesis: Experiments in live performance
Moyers Jr Pragmatism, eclecticism and accessibility in composing and improvising electroacoustic and audiovisual work
Costabile Composing for the Performance Space: A Practice-Based Investigation on the Design of Interfaces for Spatial Sound and Lighting
Chagas Composition in circular sound space: Migration 12-channel electronic music (1995–97)
Kalidas " 12" Music Visualisation: Exploring Generative and Non-Generative techniques within a rhythmic framework in a performative spatial setting.
Rice Stretchable music: A graphically rich, interactive composition system
Weinel et al. Melting in the Mind’s Eye: Real-Time Performances
Jaroszewicz Compositional strategies in spectral spatialization
Ox Collaboration, the Color Organ

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 15800888

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 15800888

Country of ref document: EP

Kind code of ref document: A1