US20220059063A1 - Music Generator Generation of Continuous Personalized Music - Google Patents

Music Generator Generation of Continuous Personalized Music Download PDF

Info

Publication number
US20220059063A1
US20220059063A1 US17/408,105 US202117408105A US2022059063A1 US 20220059063 A1 US20220059063 A1 US 20220059063A1 US 202117408105 A US202117408105 A US 202117408105A US 2022059063 A1 US2022059063 A1 US 2022059063A1
Authority
US
United States
Prior art keywords
user
music
control element
composition
user input
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
US17/408,105
Other languages
English (en)
Inventor
Edward Balassanian
Patrick E. Hutchings
Toby Gifford
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Aimi Inc
Original Assignee
Aimi Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Aimi Inc filed Critical Aimi Inc
Priority to US17/408,105 priority Critical patent/US20220059063A1/en
Assigned to AIMI INC. reassignment AIMI INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: BALASSANIAN, EDWARD, HUTCHINGS, PATRICK E.
Assigned to AIMI INC. reassignment AIMI INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: GIFFORD, TOBY
Publication of US20220059063A1 publication Critical patent/US20220059063A1/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/60—Information retrieval; Database structures therefor; File system structures therefor of audio data
    • G06F16/63—Querying
    • G06F16/638—Presentation of query results
    • G06F16/639—Presentation of query results using playlists
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/60—Information retrieval; Database structures therefor; File system structures therefor of audio data
    • G06F16/65—Clustering; Classification
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/004—Artificial life, i.e. computing arrangements simulating life
    • G06N3/006—Artificial life, i.e. computing arrangements simulating life based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/042—Knowledge-based neural networks; Logical representations of neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/088—Non-supervised learning, e.g. competitive learning
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H1/00—Details of electrophonic musical instruments
    • G10H1/0008—Associated control or indicating means
    • G10H1/0025—Automatic or semi-automatic music composition, e.g. producing random music, applying rules from music theory or modifying a musical piece
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H1/00—Details of electrophonic musical instruments
    • G10H1/02—Means for controlling the tone frequencies, e.g. attack or decay; Means for producing special musical effects, e.g. vibratos or glissandos
    • G10H1/04—Means for controlling the tone frequencies, e.g. attack or decay; Means for producing special musical effects, e.g. vibratos or glissandos by additional modulation
    • G10H1/053—Means for controlling the tone frequencies, e.g. attack or decay; Means for producing special musical effects, e.g. vibratos or glissandos by additional modulation during execution only
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2210/00—Aspects or methods of musical processing having intrinsic musical character, i.e. involving musical theory or musical parameters or relying on musical knowledge, as applied in electrophonic musical tools or instruments
    • G10H2210/031—Musical analysis, i.e. isolation, extraction or identification of musical elements or musical parameters from a raw acoustic signal or from an encoded audio signal
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2210/00—Aspects or methods of musical processing having intrinsic musical character, i.e. involving musical theory or musical parameters or relying on musical knowledge, as applied in electrophonic musical tools or instruments
    • G10H2210/101—Music Composition or musical creation; Tools or processes therefor
    • G10H2210/105—Composing aid, e.g. for supporting creation, edition or modification of a piece of music
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2210/00—Aspects or methods of musical processing having intrinsic musical character, i.e. involving musical theory or musical parameters or relying on musical knowledge, as applied in electrophonic musical tools or instruments
    • G10H2210/101—Music Composition or musical creation; Tools or processes therefor
    • G10H2210/111—Automatic composing, i.e. using predefined musical rules
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2220/00—Input/output interfacing specifically adapted for electrophonic musical tools or instruments
    • G10H2220/091—Graphical user interface [GUI] specifically adapted for electrophonic musical instruments, e.g. interactive musical displays, musical instrument icons or menus; Details of user interactions therewith
    • G10H2220/101—Graphical user interface [GUI] specifically adapted for electrophonic musical instruments, e.g. interactive musical displays, musical instrument icons or menus; Details of user interactions therewith for graphical creation, edition or control of musical data or parameters
    • G10H2220/116—Graphical user interface [GUI] specifically adapted for electrophonic musical instruments, e.g. interactive musical displays, musical instrument icons or menus; Details of user interactions therewith for graphical creation, edition or control of musical data or parameters for graphical editing of sound parameters or waveforms, e.g. by graphical interactive control of timbre, partials or envelope
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2220/00—Input/output interfacing specifically adapted for electrophonic musical tools or instruments
    • G10H2220/155—User input interfaces for electrophonic musical instruments
    • G10H2220/351—Environmental parameters, e.g. temperature, ambient light, atmospheric pressure, humidity, used as input for musical purposes
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2240/00—Data organisation or data communication aspects, specifically adapted for electrophonic musical tools or instruments
    • G10H2240/075—Musical metadata derived from musical analysis or for use in electrophonic musical instruments
    • G10H2240/081—Genre classification, i.e. descriptive metadata for classification or selection of musical pieces according to style
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2240/00—Data organisation or data communication aspects, specifically adapted for electrophonic musical tools or instruments
    • G10H2240/075—Musical metadata derived from musical analysis or for use in electrophonic musical instruments
    • G10H2240/085—Mood, i.e. generation, detection or selection of a particular emotional content or atmosphere in a musical piece
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2240/00—Data organisation or data communication aspects, specifically adapted for electrophonic musical tools or instruments
    • G10H2240/121—Musical libraries, i.e. musical databases indexed by musical parameters, wavetables, indexing schemes using musical parameters, musical rule bases or knowledge bases, e.g. for automatic composing methods
    • G10H2240/125—Library distribution, i.e. distributing musical pieces from a central or master library
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2240/00—Data organisation or data communication aspects, specifically adapted for electrophonic musical tools or instruments
    • G10H2240/121—Musical libraries, i.e. musical databases indexed by musical parameters, wavetables, indexing schemes using musical parameters, musical rule bases or knowledge bases, e.g. for automatic composing methods
    • G10H2240/131—Library retrieval, i.e. searching a database or selecting a specific musical piece, segment, pattern, rule or parameter set
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2240/00—Data organisation or data communication aspects, specifically adapted for electrophonic musical tools or instruments
    • G10H2240/121—Musical libraries, i.e. musical databases indexed by musical parameters, wavetables, indexing schemes using musical parameters, musical rule bases or knowledge bases, e.g. for automatic composing methods
    • G10H2240/131—Library retrieval, i.e. searching a database or selecting a specific musical piece, segment, pattern, rule or parameter set
    • G10H2240/141—Library retrieval matching, i.e. any of the steps of matching an inputted segment or phrase with musical database contents, e.g. query by humming, singing or playing; the steps may include, e.g. musical analysis of the input, musical feature extraction, query formulation, or details of the retrieval process
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2240/00—Data organisation or data communication aspects, specifically adapted for electrophonic musical tools or instruments
    • G10H2240/325—Synchronizing two or more audio tracks or files according to musical features or musical timings
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2250/00—Aspects of algorithms or signal processing methods without intrinsic musical character, yet specifically adapted for or used in electrophonic musical processing
    • G10H2250/005—Algorithms for electrophonic musical instruments or musical processing, e.g. for automatic composition or resource allocation
    • G10H2250/015—Markov chains, e.g. hidden Markov models [HMM], for musical processing, e.g. musical analysis or musical composition
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2250/00—Aspects of algorithms or signal processing methods without intrinsic musical character, yet specifically adapted for or used in electrophonic musical processing
    • G10H2250/311—Neural networks for electrophonic musical instruments or musical processing, e.g. for musical recognition or control, automatic composition or improvisation
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2250/00—Aspects of algorithms or signal processing methods without intrinsic musical character, yet specifically adapted for or used in electrophonic musical processing
    • G10H2250/541—Details of musical waveform synthesis, i.e. audio waveshape processing from individual wavetable samples, independently of their origin or of the sound they represent
    • G10H2250/641—Waveform sampler, i.e. music samplers; Sampled music loop processing, wherein a loop is a sample of a performance that has been edited to repeat seamlessly without clicks or artifacts

Definitions

  • This disclosure relates to audio engineering and more particularly to generating music content.
  • Streaming music services typically provide songs to users via the Internet. Users may subscribe to these services and stream music through a web browser or application. Examples of such services include PANDORA, SPOTIFY, GROOVESHARK, etc. Often, a user can select a genre of music or specific artists to stream. Users can typically rate songs (e.g., using a star rating or a like/dislike system), and some music services may tailor which songs are streamed to a user based on previous ratings. The cost of running a streaming service (which may include paying royalties for each streamed song) is typically covered by user subscription costs and/or advertisements played between songs.
  • Song selection may be limited by licensing agreements and the number of songs written for a particular genre. Users may become tired of hearing the same songs in a particular genre. Further, these services may not tune music to users' tastes, environment, behavior, etc.
  • FIG. 1 is a diagram illustrating an exemplary music generator.
  • FIG. 2 is a block diagram illustrating an exemplary overview of a system for generating output music content based on inputs from multiple different sources, according to some embodiments.
  • FIG. 3 is a block diagram illustrating an example music generator with a composition subsystem and a performance subsystem, according to some embodiments.
  • FIG. 4 is a block diagram illustrating more a detailed embodiment of the composition and performance subsystems, according to some embodiments.
  • FIGS. 5A-5B are block diagrams illustrating graphical user interfaces, according to some embodiments.
  • FIG. 6 is a block diagram illustrating an example music generator system that includes analysis and composition modules, according to some embodiments.
  • FIG. 7 is a diagram illustrating an example buildup section of music content, according to some embodiments.
  • FIG. 8 is a diagram illustrating example techniques for arranging sections of music content, according to some embodiments.
  • FIGS. 9 and 10 are flow diagrams illustrating example methods, according to some embodiments.
  • audio file refers to sound information for music content.
  • sound information may include data that describes music content in as raw audio in a format such as way, aiff, or FLAC. Properties of the music content may be included in the sound information. Properties may include, for example, quantifiable musical properties such as instrument classification, pitch transcription, beat timings, tempo, file length, and audio amplitude in multiple frequency bins.
  • an audio file includes sound information over a particular time interval.
  • audio files include loops.
  • the term “loop” refers to sound information for a single instrument over a particular time interval. Various techniques discussed with reference to audio files may also be performed using loops that include a single instrument. Audio files or loops may be played in a repeated manner (e.g., a 30 second audio file may be played four times in a row to generate 2 minutes of music content), but audio files may also be played once, e.g., without being repeated.
  • FIGS. 1 and 2 This disclosure initially describes, with reference to FIGS. 1 and 2 , an example music generator module and an overall system organization with multiple applications. Techniques for music generation are discussed with reference to FIGS. 3-4 .
  • FIGS. 5A-5B show exemplary application interfaces.
  • the disclosed music generator includes audio files, metadata (e.g., information describing the audio files), and techniques for combining audio files based on the metadata.
  • the generator may create music experiences using rules to identify the audio files based on metadata and target characteristics of the music experience. It may be configured to expand the set of experiences it can create by adding or modifying rules, audio files, and/or metadata.
  • the adjustments may be performed manually (e.g., artists adding new metadata) or the music generator may augment the rules/audio files/metadata as it monitors the music experience within the given environment and goals/characteristics desired. For example, listener-defined controls may be implemented for gaining user feedback on music goals or characteristics.
  • FIG. 1 is a diagram illustrating an exemplary music generator, according to some embodiments.
  • music generator module 160 receives various information from multiple different sources and generates output music content 140 .
  • module 160 accesses stored audio file(s) and corresponding attribute(s) 110 for the stored audio file(s) and combines the audio files to generate output music content 140 .
  • music generator module 160 selects audio files based on their attributes and combines audio files based on target music attributes 130 .
  • audio files may be selected based on environment information 150 in combination with target music attributes 130 .
  • environment information 150 is used indirectly to determine target music attributes 130 .
  • target music attributes 130 are explicitly specified by a user, e.g., by specifying a desired energy level, mood, multiple parameters, etc. For instance, listener-defined controls, described herein, may be implemented to specify listener preferences used as target music attributes.
  • target music attributes 130 include energy, complexity, and variety, although more specific attributes (e.g., corresponding to the attributes of the stored tracks) may also be specified.
  • higher-level target music attributes are specified, lower-level specific music attributes may be determined by the system before generating output music content.
  • Complexity may refer to the number of audio files, loops, and/or instruments that are included in a composition.
  • Energy may be related to the other attributes or may be orthogonal to the other attributes. For example, changing keys or tempo may affect energy. However, for a given tempo and key, energy may be changed by adjusting instrument types (e.g., by adding high hats or white noise), complexity, volume, etc.
  • Variety may refer to an amount of change in generated music over time. Variety may be generated for a static set of other musical attributes (e.g., by selecting different tracks for a given tempo and key) or may be generated by changing musical attributes over time (e.g., by changing tempos and keys more often when greater variety is desired).
  • the target music attributes may be thought of as existing in a multi-dimensional space and music generator module 160 may slowly move through that space, e.g., with course corrections, if needed, based on environmental changes and/or user input.
  • the attributes stored with the audio files contain information about one or more audio files including: tempo, volume, energy, variety, spectrum, envelope, modulation, periodicity, rise and decay time, noise, artist, instrument, theme, etc.
  • audio files are partitioned such that a set of one or more audio files is specific to a particular audio file type (e.g., one instrument or one type of instrument).
  • module 160 accesses stored rule set(s) 120 .
  • Stored rule set(s) 120 specify rules for how many audio files to overlay such that they are played at the same time (which may correspond to the complexity of the output music), which major/minor key progressions to use when transitioning between audio files or musical phrases, which instruments to be used together (e.g., instruments with an affinity for one another), etc. to achieve the target music attributes.
  • the music generator module 160 uses stored rule set(s) 120 to achieve one or more declarative goals defined by the target music attributes (and/or target environment information).
  • music generator module 160 includes one or more pseudo-random number generators configured to introduce pseudo-randomness to avoid repetitive output music.
  • Environment information 150 includes one or more of: lighting information, ambient noise, user information (facial expressions, body posture, activity level, movement, skin temperature, performance of certain activities, clothing types, etc.), temperature information, purchase activity in an area, time of day, day of the week, time of year, number of people present, weather status, etc.
  • music generator module 160 does not receive/process environment information.
  • environment information 150 is received by another module that determines target music attributes 130 based on the environment information.
  • Target music attributes 130 may also be derived based on other types of content, e.g., video data.
  • environment information is used to adjust one or more stored rule set(s) 120 , e.g., to achieve one or more environment goals.
  • the music generator may use environment information to adjust stored attributes for one or more audio files, e.g., to indicate target musical attributes or target audience characteristics for which those audio files are particularly relevant.
  • module refers to circuitry configured to perform specified operations or to physical non-transitory computer readable media that store information (e.g., program instructions) that instructs other circuitry (e.g., a processor) to perform specified operations.
  • Modules may be implemented in multiple ways, including as a hardwired circuit or as a memory having program instructions stored therein that are executable by one or more processors to perform the operations.
  • a hardware circuit may include, for example, custom very-large-scale integration (VLSI) circuits or gate arrays, off-the-shelf semiconductors such as logic chips, transistors, or other discrete components.
  • VLSI very-large-scale integration
  • a module may also be implemented in programmable hardware devices such as field programmable gate arrays, programmable array logic, programmable logic devices, or the like.
  • a module may also be any suitable form of non-transitory computer readable media storing program instructions executable to perform specified operations.
  • music content refers both to music itself (the audible representation of music), as well as to information usable to play music.
  • a song recorded as a file on a storage medium is an example of music content; the sounds produced by outputting this recorded file or other electronic representation (e.g., through speakers) is also an example of music content.
  • music includes its well-understood meaning, including sounds generated by musical instruments as well as vocal sounds.
  • music includes, for example, instrumental performances or recordings, a cappella performances or recordings, and performances or recordings that include both instruments and voice.
  • musical attributes such as rhythm or rhyme—for example, speeches, newscasts, and audiobooks—are not music.
  • One piece of music “content” can be distinguished from another piece of music content in any suitable fashion.
  • a digital file corresponding to a first song may represent a first piece of music content
  • a digital file corresponding to a second song may represent a second piece of music content.
  • music content can also be used to distinguish particular intervals within a given musical work, such that different portions of the same song can be considered different pieces of musical content.
  • different tracks e.g., piano track, guitar track
  • the phrase “music content” can be used to refer to some portion of the stream (e.g., a few measures or a few minutes).
  • Music content generated by embodiments of the present disclosure may be “new music content”—combinations of musical elements that have never been previously generated.
  • the concept of a “controlling entity” relative to an instance of music content generation is described.
  • the phrase “new music content” does not refer to the concept of a controlling entity. Accordingly, new music content refers to music content that has never before been generated by any entity or computer system.
  • the present disclosure refers to some “entity” as controlling a particular instance of computer-generated music content.
  • entity owns any legal rights (e.g., copyright) that might correspond to the computer-generated content (to the extent that any such rights may actually exist).
  • an individual that creates e.g., codes various software routines
  • a computer-implemented music generator or operates (e.g., supplies inputs to) a particular instance of computer-implemented music generation will be the controlling entity.
  • a computer-implemented music generator may be created by a legal entity (e.g., a corporation or other business organization), such as in the form of a software product, computer system, or computing device.
  • such a computer-implemented music generator may be deployed to many clients.
  • the controlling entity may be the creator, the distributor, or the clients in various instances. If there are no such explicit legal agreements, the controlling entity for a computer-implemented music generator is the entity facilitating (e.g., supplying inputs to and thereby operating) a particular instance of computer generation of music content.
  • computer generation of “original music content” by a controlling entity refers to 1) a combination of musical elements that has never been generated before, either by the controlling entity or anyone else, and 2) a combination of musical elements that has been generated before, but was generated in the first instance by the controlling entity.
  • Content type 1) is referred to herein as “novel music content,” and is similar to the definition of “new music content,” except that the definition of “novel music content” refers to the concept of a “controlling entity,” while the definition of “new music content” does not.
  • Non-original music content is music content that is not “original music content” for that controlling entity.
  • Some pieces of music content may include musical components from one or more other pieces of music content. Creating music content in this manner is referred to as “sampling” music content, and is common in certain musical works, and particularly in certain musical genres. Such music content is referred to herein as “music content with sampled components,” “derivative music content,” or using other similar terms. In contrast, music content that does not include sampled components is referred to herein as “music content without sampled components,” “non-derivative music content,” or using other similar terms.
  • derivative is intended to have a broader meaning within the present disclosure than the term “derivative work” that is used in U.S. copyright law.
  • derivative music content may or may not be a derivative work under U.S. copyright law.
  • derivative in the present disclosure is not intended to convey a negative connotation; it is merely used to connote whether a particular piece of music content “borrows” portions of content from another work.
  • the phrases “new music content,” “novel music content,” and “original music content” are not intended to encompass music content that is only trivially different from a pre-existing combination of musical elements. For example, merely changing a few notes of a pre-existing musical work does not result in new, novel, or original music content, as those phrases are used in the present disclosure. Similarly, merely changing a key or tempo or adjusting a relative strength of frequencies (e.g., using an equalizer interface) of a pre-existing musical work does not produce new, novel, or original music content.
  • the phrases, new, novel, and original music content are not intended to cover those pieces of music content that are borderline cases between original and non-original content; instead, these terms are intended to cover pieces of music content that are unquestionably and demonstrably original, including music content that would be eligible for copyright protection to the controlling entity (referred to herein as “protectable” music content).
  • the term “available” music content refers to music content that does not violate copyrights of any entities other than the controlling entity. New and/or original music content is often protectable and available. This may be advantageous in preventing copying of music content and/or paying royalties for music content.
  • rule-based engines various other types of computer-implemented algorithms may be used for any of the computer learning and/or music generation techniques discussed herein. Rule-based approaches may be particularly effective in the music context, however.
  • a music generator module may interact with multiple different applications, modules, storage elements, etc. to generate music content. For example, end users may install one of multiple types of applications for different types of computing devices (e.g., mobile devices, desktop computers, DJ equipment, etc.). Similarly, another type of application may be provided to enterprise users. Interacting with applications while generating music content may allow the music generator to receive external information that it may use to determine target music attributes and/or update one or more rule sets used to generate music content. In addition to interacting with one or more applications, a music generator module may interact with other modules to receive rule sets, update rule sets, etc. Finally, a music generator module may access one or more rule sets, audio files, and/or generated music content stored in one or more storage elements. In addition, a music generator module may store any of the items listed above in one or more storage elements, which may be local or accessed via a network (e.g., cloud-based).
  • a network e.g., cloud-based
  • FIG. 2 is a block diagram illustrating an exemplary overview of a system for generating output music content based on inputs from multiple different sources, according to some embodiments.
  • system 200 includes rule module 210 , user application 220 , web application 230 , enterprise application 240 , artist application 250 , artist rule generator module 260 , storage of generated music 270 , and external inputs 280 .
  • User application 220 , web application 230 , and enterprise application 240 receive external inputs 280 .
  • external inputs 280 include: environment inputs, target music attributes, user input, sensor input, etc.
  • user application 220 is installed on a user's mobile device and includes a graphical user interface (GUI) that allows the user to interact/communicate with rule module 210 .
  • GUI graphical user interface
  • web application 230 is not installed on a user device, but is configured to run within a browser of a user device and may be accessed through a website.
  • enterprise application 240 is an application used by a larger-scale entity to interact with a music generator.
  • application 240 is used in combination with user application 220 and/or web application 230 .
  • application 240 communicates with one or more external hardware devices and/or sensors to collect information concerning the surrounding environment.
  • Rule module 210 in the illustrated embodiment, communicates with user application 220 , web application 230 , and enterprise application 240 to produce output music content.
  • music generator 160 is included in rule module 210 .
  • rule module 210 may be included in one of applications 220 , 230 , and 240 or may be installed on a server and accessed via a network.
  • applications 220 , 230 , and 240 receive generated output music content from rule module 210 and cause the content to be played.
  • rule module 210 requests input from applications 220 , 230 , and 240 regarding target music attributes and environment information, for example, and may use this data to generate music content.
  • Stored rule set(s) 120 are accessed by rule module 210 .
  • rule module 210 modifies and/or updates stored rule set(s) 120 based on communicating with applications 220 , 230 , and 240 .
  • rule module 210 accesses stored rule set(s) 120 to generate output music content.
  • stored rule set(s) 120 may include rules from artist rule generator module 260 , discussed in further detail below.
  • Artist application 250 in the illustrated embodiment, communicates with artist rule generator module 260 (which may be part of the same application or may be cloud-based, for example).
  • artist application 250 allows artists to create rule sets for their specific sound, e.g., based on previous compositions. This functionality is further discussed U.S. Pat. No. 10,679,596.
  • artist rule generator module 260 is configured to store generated artist rule sets for use by rule module 210 . Users may purchase rule sets from particular artists before using them to generate output music via their particular application. The rule set for a particular artist may be referred to as a signature pack.
  • Stored audio file(s) and corresponding attribute(s) 110 are accessed by module 210 when applying rules to select and combine tracks to generate output music content.
  • rule module 210 stores generated output music content 270 in storage element.
  • one or more of the elements of FIG. 2 are implemented on a server and accessed via a network, which may be referred to as a cloud-based implementation.
  • stored rule set(s) 120 , audio file(s)/attribute(s) 110 , and generated music 270 may all be stored on the cloud and accessed by module 210 .
  • module 210 and/or module 260 may also be implemented in the cloud.
  • generated music 270 is stored in the cloud and digitally watermarked. This may allow detection of copying generated music, for example, as well as generating a large amount of custom music content.
  • one or more of the disclosed modules are configured to generate other types of content in addition to music content.
  • the system may be configured to generate visual content based on target music attributes, determined environmental conditions, currently-used rule sets, etc.
  • the system may search a database or the Internet based on current attributes of the music being generated and display a collage of images that dynamically changes as the music changes and matches the attributes of the music.
  • music generator module 160 may implement a variety of artificial intelligence (AI) techniques (e.g., machine learning techniques) to generate output music content 140 .
  • AI techniques implemented include a combination of deep neural networks (DNN) with more traditional machine learning techniques and knowledge-based systems. This combination may align the respective strengths and weaknesses of these techniques with challenges inherent in music composition and personalization systems.
  • Music content has structure at multiple levels. For instance, a song has sections, phrases, melodies, notes and textures.
  • Some machine learning techniques may be effective at analyzing and generating high level and low-level details of music content.
  • DNNs may be good at classifying the texture of a sound as belonging to a clarinet or an electric guitar at a low level or detecting verses and choruses at a high level.
  • the middle levels of music content details, such as the construction of melodies, orchestration, etc. may be more difficult.
  • DNNs are typically good at capturing a wide range of styles in a single model and thus, DNNs may be implemented as generative tools that have a lot of expressive range.
  • music generator module 160 utilizes expert knowledge by having human-composed audio files (e.g., loops) as the fundamental unit of music content used by the music generator module.
  • human-composed audio files e.g., loops
  • social context of expert knowledge may be embedded through the choice of rhythms, melodies and textures to record heuristics in multiple levels of structure.
  • expert knowledge may be applied in any areas where musicality can be increased without placing too strong of limitations on the trainability of music generator module 160 .
  • music generator module 160 uses DNNs to find patterns of how layers of audio are combined vertically, by layering sounds on top of each other, and horizontally, by combining audio files or loops into sequences.
  • music generator module 160 may implement an LSTM (long short-term memory) recurrent neural network, trained on MFCC (Mel-frequency cepstral coefficient) audio features of loops used in multitrack audio recordings.
  • a network is trained to predict and select audio features of loops for upcoming beats of music based on knowledge of the audio features of previous beats. For example, the network may be trained to predict the audio features of loops for the next 8 beats based on knowledge of the audio features of the last 128 beats. Thus, the network is trained to utilize a low-dimension feature representation to predict upcoming beats.
  • music generator module 160 uses known machine learning algorithms for assembling sequences of multitrack audio into musical structures with dynamics of intensity and complexity.
  • music generator module 160 may implement Hierarchical Hidden Markov Models, which may behave like state machines that make state transitions with probabilities determined by multiple levels of hierarchical structure. As an example, a specific kind of drop may be more likely to happen after a buildup section but less likely if the end of that buildup does not have drums.
  • the probabilities may be trained transparently, which is in contrast to the DNN training where what is being learned is more opaque.
  • a Markov Model may deal with larger temporal structures and thus may not easily be trained by presenting example tracks as the examples may be too long.
  • a feedback control element (such as a thumbs up/down on the user interface) may be used to give feedback on the music at any time. Correlations between the music structure and the feedback may then be used to update structural models used for composition, such as transition tables or Markov models. This feedback may also be collected directly from measurements of heart-rate, sales, or any other metric where the system is able to determine a clear classification.
  • Expert knowledge heuristics, described above, are also designed to be probabilistic where possible and trained in the same way as the Markov model.
  • training may be performed by composers or DJs. Such training may be separate from listener training. For example, training done by listeners (such as typical users) may be limited to identifying correct or incorrect classification based on positive and negative model feedback, respectively.
  • training may include hundreds of timesteps and include details on layers used and volume control to give more explicit detail into what is driving changes in music content.
  • training performed by composers and DJs may include sequence prediction training similar to global training of DNNs, described above.
  • a DNN is trained to predict interactions a DJ might have with their audio interface at any moment in time given a sequence of the most recently played music. In some embodiments, these interactions may be recorded and used to develop new heuristics that are more transparent.
  • the DNN receives a number of previous measures of music as input and utilizes a low-dimension feature representation, as described above, with additional features that describe modifications to a track that a DJ or composer has applied. For example, the DNN may receive the last 32 measures of music as input and utilize the low-dimension feature representation along with additional features to describe modifications to the track that a DJ or composer has applied. These modifications may include adjustments to gain of a particular track, filters applied, delay, etc.
  • a DJ may use the same drum loop repeated for five minutes during a performance but may gradually increase the gain and delay on the track over time. Therefore, the DNN may be trained to predict such gain and delay changes in addition to loop selection.
  • the feature set may be all zeros for that instrument, which may allow the DNN to learn that predicting all zeros may be a successful strategy, which may lead to selective layering.
  • DJs or composers record live performances using mixers and devices such as TRAKTOR (Native Instruments GmbH). These recordings are typically captured in high resolution (e.g., 4 track recording or MIDI).
  • the system disassembles the recording into its constituent loops yielding information about the combination of loops in a composition as well as the sonic qualities of each individual loop. Training the DNN (or other machine learning) with this information provides the DNN with the ability to correlate both composition (e.g., sequencing, layering, timing of loops, etc.) and sonic qualities of loops to inform music generator module 160 how to create music experiences that are similar to the artists performance without using the actual loops the artist used in their performance.
  • composition e.g., sequencing, layering, timing of loops, etc.
  • sonic qualities of loops to inform music generator module 160 how to create music experiences that are similar to the artists performance without using the actual loops the artist used in their performance.
  • composition subsystem may select and arrange audio files for composing music content while a performance subsystem may filter, add effects, mix, master, etc. to the selected audio files to generate output music content to actually be played.
  • both the composition subsystem and the performance subsystem operate based on user input to music control elements (which may be shown via one or more user interfaces).
  • an artist may provide user input to the composition subsystem (which may be implemented on a server system, for example) and an end user may provide user input to the performance subsystem (which may be implemented on a user device such as a mobile phone, for example).
  • the same user may provide input to both modules.
  • the system allows users to define their own music controls and implements machine learning or other algorithms to adjust music parameters based on adjustments to the user-defined controls, which may be more abstract. For example, a user may define a “harmony,” “happiness,” or “moody” control and the composition subsystem may determine the type of lower-level parameters to adjust and the amount of adjustment to achieve the artist's vision for this control element.
  • all or a subset of the user-defined music controls are also available on the performance subsystem, e.g., so that users can implement artist-defined controls.
  • the two subsystems may communicate in order to provide proper performance parameters based on user-defined music controls.
  • Disclosed techniques may advantageously allow artists to provide customized music control elements to end users. Further, disclosed techniques may allow centralized control of certain aspects of automatic music composition while still allowing end-users to customize real-time performance of generative music.
  • the subsets of music controls available to a composition subsystem and a performance system may or may not overlap. Further, users of the performance module may specify their own abstract music controls, which may be used in conjunction with abstract music controls defined by the composition module, in some embodiments.
  • disclosed techniques may facilitate the generation of more human-like music composition by a music generator system.
  • the present inventors have recognized that providing abstract control over music generation allows the music generator system to generate music content according to stylistic, structural, or artistic preferences of the user.
  • FIG. 3 is a block diagram illustrating an example music generator system 300 that includes a composition subsystem 310 and a performance subsystem 320 , according to some embodiments.
  • Composition subsystem 310 in the illustrated embodiment, generates composition control output information based on user input.
  • the composition control output information identifies selected audio files to be combined and indicates their relative timing for inclusion in output music content.
  • Composition subsystem 310 may determine how to arrange sections of music content and how and when to transition between sections, as discussed in further detail below.
  • composition subsystem 310 also generates control element configuration signals for performance subsystem 320 .
  • these signals configure performance subsystem 320 to implement user input to abstract music controls in an intended manner.
  • the configuration may specify how to adjust various performance operations (such as filtering, adding effects, mixing, mastering, etc.) based on adjustments to user-defined music controls.
  • composition subsystem 310 receives user input.
  • This user input may specify custom music control elements, for example.
  • the user input may also provide label information for previous compositions by a particular artist. For example, for the abstract control element “harmony,” an artist may label previous compositions on a scale of 0 to N based on the artist's perception of their harmoniousness, to allow training of a machine learning model to adjust music composition parameters to reflect different levels of the harmony control element.
  • music control elements may have various encodings, including accepting user input that specifies specific values, increases/decreases, binary inputs such as good/bad, etc.
  • Performance subsystem 320 in the illustrated embodiment, is configured to generate output music content based on the composition control and control element configuration information from composition subsystem 310 . As shown, performance sub-system 320 also receives user input, e.g., potentially for all of a subset of the music control elements implemented by composition subsystem 310 (performance subsystem 320 may also implement additional music control elements, in some embodiments). The set of control elements available for user input on the performance subsystem 320 is controlled by user input to composition subsystem 310 , in some embodiments.
  • composition control at multiple levels may advantageously provide musical outputs while allowing user customization for various music styles.
  • various functionality performed by the composition subsystem 310 in disclosed embodiments may be moved to performance subsystem 320 and vice versa, in other embodiments.
  • composition subsystem 310 and performance subsystem 320 are implemented on different computer systems and communicate via a network. As discussed in detail below, communications between interfaces may use a custom string manipulation and storage technique to allow a real-time monitoring and control API (e.g., using the open sound control (OSC) protocol). In other embodiments, the subsystems are implemented on the same computer system.
  • OSC open sound control
  • the performance subsystem 320 may also send control element configuration information to the composition subsystem 310 (not explicitly shown), e.g., in order for end-user adjustments to affect composition operations such as selecting audio files to be combined.
  • control element configuration information may be sent to the composition subsystem 310 (not explicitly shown), e.g., in order for end-user adjustments to affect composition operations such as selecting audio files to be combined.
  • Generally disclosed techniques may allow a user-defined control element defined on one device/interface to be available to users on other devices/interfaces and potentially control parameter adjustments on other subsystems or devices.
  • FIG. 4 is a block diagram illustrating a more detailed example music generator system, according to some embodiments.
  • composition subsystem 310 includes a module for customized music controls 410 A, composition module(s) 420 , and machine learning module 430 .
  • performance subsystem 320 includes a module for customized music controls 410 B and performance module(s) 450 . Note that more detailed examples of composition module(s) 420 and performance module(s) 450 included in different subsystems are discussed below with reference to FIG. 6 .
  • Module 410 A may communicate with machine learning module 430 based on user input to determine how to implement custom music controls using lower-level parameters.
  • machine learning module 430 may access previous compositions 440 (which may be labeled as a whole or with labels for different portions, by the user, according to one or more custom music controls) to determine how to implement abstract controls using lower-level composition parameters (and these parameters may include parameters used by both subsystem 310 and subsystem 320 ).
  • the training may use various appropriate machine learning models, e.g., deep neural networks, to determine the purpose of a user-specified control element.
  • module 410 A communicates control element configuration information to module 410 B, which communicates with performance module(s) 450 to adjust performance parameters based on user input.
  • composition module(s) 420 adjust composition parameters based on signals from module 410 A.
  • artists may explicitly specify relationships between custom music controls and lower-level parameters. For example, for a “harmony” control, an artist may specify different parameters for creating tension (e.g., adding layers, riser effects, cutting a melody, etc.) for compositions with stronger harmonic movement than for compositions with lesser harmony.
  • the explicit controls may be used alone or in combination with machine learning analysis to generate lower-level parameters, in some embodiments.
  • the following discussion sets out various parameters that may be indirectly controlled via custom user interface elements, e.g., based on machine learning or user indication of relationships between controls and parameters.
  • the music generator system may expose both abstract and lower-level controls, in some embodiments.
  • the generation of continuous music is achieved through the combination of algorithmically defined musical sections and transitions.
  • Sections may be defined by their musical function, e.g., building, sustaining and reducing tension.
  • sections of tension are created slowly by repetition of musical content, having a louder or more layered mix of sounds or playing contrasting musical rhythms or pitches over each other.
  • tension may be established quickly by defying listeners expectations. For example, ending a repeated phrase halfway through a repetition cycle, suddenly changing key, introducing a new texture that has not appeared previously in the music, sudden increases in volume, etc. may quickly establish tension. If the same technique for creating tension is repeated frequently, the listener may come to expect them and their effectiveness at creating tension is greatly diminished.
  • the creation of effective continuous music may result from sequencing the buildup and release of tension in diverse ways.
  • composition subsystem 310 exposes certain controls with default or optional values. Controls may include gain mix, audio effects chain, musical arrangement over time, and concurrent layering of musical phrases, for example. Unlike a standard composition tools where only compositional processes that are added by the user have musical effect, in cases where an option value is not utilized, a machine learning algorithm may be used to affect compositional processes.
  • Examples of parameters controlled by composition subsystem 310 may include maximum repetition of a single loopable piece of audio, use of synthesized noise, and default EQ levels, for example.
  • the maximum repetition may be set as a hard limit or left at infinity, where there is no limit, but other composition processes are likely to drive departure from repetition without it.
  • a “Synthesized noise” control could be used as a binary switch for enabling or disabling synthesized noise in some points of the composition, without controlling when it will be applied.
  • the timing may be influenced by other controls and machine learning algorithms.
  • the default EQ levels may allow a reference level for frequency, gain and bandwidth for 8 bands, which may be adjusted by other composition processes including those led by machine learning algorithms.
  • the composition subsystem 310 may define which types of audio files can be used in different temporal structures of a composition without defining exactly when the temporal structures will take place in the composition.
  • a group may be enabled or disabled within different musical arrangement structures.
  • a group being enabled means that it may be utilized by the subsystem but is not guaranteed to be.
  • groups include:
  • the following sections may be utilized: buildup: slow increase of tension; sustain: maintain the level of tension; drop: a climax point followed by low tension; breakdown: slow decrease of tension.
  • the following transitions may be utilized: Add layer: sudden tension increase; remove layer: sudden tension decrease; riser effects: fast tension increase; cut melody midway: fast tension increase.
  • the composition subsystem 310 may learn how to apply various sections and transitions based on the composition preference of a particular artist of group of artists, for example.
  • probability-based rules guide the sequence of transitions to encourage variations to the slow build and release of tension and transitions are used to create unexpected diversions in these slower trends. Excessive use of transitions may reduce the overall tension movement of the section they occupy, so maximum and minimum frequency of transitions may be scaled to section lengths.
  • the following table provides an example of transition probabilities for different sections.
  • a user of the composition subsystem 310 modifies such a table to change the stepwise musical structure without predetermined absolute timing of sections:
  • transitioning from a breakdown section moves to a buildup section 40% of the time, to a drop section 30% of the time, and a sustain section 30% of the time.
  • composition subsystem may further break sections down further to create a hierarchical structure of sections and subsections.
  • Example subsections include:
  • subsection rules As each subsection produces transitions at their beginning, end and in some cases middle, certain subsections only occur in specific sections or at specific points of sections. As examples of subsection rules:
  • Example layer combination include:
  • brief musical deviations may be used to create variation and micro-tension that is particularly effective in loop-based, repeated music.
  • derivation techniques are used in some embodiments:
  • Performance subsystem 320 is configured to operate on the composition control information from the composition subsystem (e.g., a composition script) to generate output music content.
  • the composition subsystem e.g., a composition script
  • composition subsystem 310 makes determinations by correlating feedback data mediated by performance subsystem 320 with music controls in the script.
  • the act of playing the performance-control subsystem passively suggests a certain degree of preference. Retroactive analysis of compositions the user has consumed and feedback provided will help drive the machine learning systems to adjust both the represented loops on a user device and the composition techniques.
  • the performance-control subsystem may use feedback analytics to find audio content of similar styles and attributes from its core set.
  • the performance-control subsystem may allow end users to accomplish the same with the set of loops that the performance subsystem provides.
  • the performance subsystem may provide a large (and growing) set of loops for playback.
  • the user may have inputs including thumbs up and down that would help the performance-control subsystem learn what types of loops and mixing techniques to use when creating soundscapes for them.
  • UX controls may be provided to the user that will allow them more fine-grained control over the way the performance-control subsystem mixes content. This could include mixing levels, section length, complexity of music, use of mixing techniques such as builds and drops etc.
  • these qualities may be captured by the performance subsystem and then associated with positive or negative feedback. Over time, this data will help guide the performance-control subsystem in training of machine learning module (not explicitly shown in FIG. 4 ) with positive and negative reinforcement, and may allow the performance-control subsystem to adjust parameters for music playback (e.g., parameters for the rules engine).
  • the model used to adjust these parameters need not be relegated to a single user.
  • machine learning may be used to apply changes to composition and playback based on aggregate feedback on a music app. In other words, if a given user hits thumbs down and the pattern identified at this time matches a pattern many other users hit thumbs down on, then the system may adjust the music accordingly in the same way it did for the other users. This allows the performance-control subsystem to make personalized changes for users but nevertheless base them on aggregate data collected across many users.
  • Correlation of feedback with music properties may be mediated with measurements of the environment. In this way more nuanced models of user preference are constructed. For example, instead of “this listener likes these types of beats,” the performance-control subsystem may learn “this listener likes these types of beats at this time of day, or when it is raining.”
  • the success of personalization for end users may ultimately be measured by positive events associated with playback. These include events such as “thumbs up,” playtime, etc. In addition to these events other parameters could be associated with successful personalization. For example, a restaurant playing the performance-control subsystem could associated growth in sales per hour as a metric for success. The performance-control subsystem may monitor the POS and associated positive changes in sales per hour with a positive reflection of the composition. In this manner the performance-control subsystem may implicitly train itself without needing explicit input. This model of training could be applied to any underlying set of loops since the training is as much about the composition of music as it is the exact loops being used.
  • communications between subsystems implement real-time monitoring and control, e.g., using an API over OSC.
  • Disclosed techniques discussed in detail below may facilitate real-time performance in such a context. Note that in distributed embodiments, the amount of data being sent over a network may become significant when monitoring all parameters and RMS levels—although still small compared to the full audio stream for example. Because these messages may be individually parsed, however, the string manipulations involved can become quite expensive. Creating such strings to be sent from a real-time thread may be performed with care, to reduce memory allocations and blocking calls.
  • OSC uses plain-text forward-slash delimited strings (similar to URLs) to address endpoints, for example “/master/compressor/threshold” or “/performance_module/effect_module/parameter_C.” These strings may be used to route various information from composition module to the proper endpoint within performance_module, for example, and in the other direction, e.g., for monitoring purposes.
  • the manipulation of delimited strings is thus a common operation in some embodiments—particularly functions analogous to ‘split’ and ‘join’ which convert between delimited strings and arrays of strings, as well as prepending and appending to a given delimited string.
  • the DelimitedString class may also be designed to allow efficient ‘trimming’ at either end.
  • the receiver of an OSC message e.g., the performance subsystem 320
  • the DelimitedString may manage this efficiently by simply moving the “HEAD” pointer that references an offset into the fullstring backing store. It may also manage a state stack, to store a state, iteratively lop off parts of the address, then restore the state. This may allow for the same DelimitedString to be used over again.
  • FIGS. 5A-5B are block diagrams illustrating graphical user interfaces, according to some embodiments.
  • FIG. 5A contains a GUI displayed by user application 510 and FIG. 5B contains a GUI displayed by enterprise application 530 .
  • the GUIs displayed in FIGS. 5A and 5B are generated by a website rather than by an application.
  • any of various appropriate elements may be displayed, including one or more of the following elements: dials (e.g., to control volume, energy, etc.), buttons, knobs, display boxes (e.g., to provide the user with updated information), etc.
  • user application 510 displays a GUI that contains section 512 for selecting one or more artist packs.
  • packs 514 may alternatively or additionally include theme packs or packs for a specific occasion (e.g., a wedding, birthday party, graduation ceremony, etc.).
  • the number of packs shown in section 512 is greater than the number that can be displayed in section 512 at one time. Therefore, in some embodiments, the user scrolls up and/or down in section 512 to view one or more packs 514 .
  • the user can select an artist pack 514 based on which he/she would like to hear output music content.
  • artist packs may be purchased and/or downloaded, for example.
  • Selection element 516 allows the user to adjust one or more music attributes (e.g., energy level). In some embodiments, selection element 516 allows the user to add/delete/modify one or more target music attributes. In various embodiments, selection element 516 may render one or more UI control elements (e.g., music controls 500 ).
  • music attributes e.g., energy level
  • selection element 516 allows the user to add/delete/modify one or more target music attributes.
  • selection element 516 may render one or more UI control elements (e.g., music controls 500 ).
  • Selection element 520 allows the user to let the device (e.g., mobile device) listen to the environment to determine target musical attributes.
  • the device collects information about the environment using one or more sensors (e.g., cameras, microphones, thermometers, etc.) after the user selects selection element 520 .
  • application 510 also selects or suggests one or more artist packs based on the environment information collected by the application when the user selected element 520 .
  • Selection element 522 allows the user to combine multiple artist packs to generate a new rule set.
  • the new rule set is based on the user selecting one or more packs for the same artist.
  • the new rule set is based on the user selecting one or more packs for different artists.
  • the user may indicate weights for different rule sets, e.g., such that a highly-weighted rule set has more effect on generated music than a lower-weighted rule set.
  • the music generator may combine rule sets in multiple different ways, e.g., by switching between rules from different rule sets, averaging values for rules from multiple different rule sets, etc.
  • selection element 524 allows the user to adjust rule(s) in one or more rule sets manually. For example, in some embodiments, the user would like to adjust the music content being generated at a more granular level, by adjusting one or more rules in the rule set used to generate the music content. In some embodiments, this allows the user of application 510 to be their own disk jockey (DJ), by using the controls displayed in the GUI in FIG. 5A to adjust a rule set used by a music generator to generate output music content. These embodiments may also allow more fine-grained control of target music attributes.
  • DJ disk jockey
  • enterprise application 530 displays a GUI that also contains an artist pack selection section 512 with artist packs 514 .
  • the enterprise GUI displayed by application 530 also contains element 516 to adjust/add/delete one or more music attributes.
  • the GUI displayed in FIG. 5B is used in a business or storefront to generate a certain environment (e.g., for optimizing sales) by generating music content.
  • an employee uses application 530 to select one or more artist packs that have been previously shown to increase sales (for example, metadata for a given rule set may indicate actual experimental results using the rule set in real-world contexts).
  • Input hardware 540 sends information to the application or website that is displaying enterprise application 530 .
  • input hardware 540 is one of the following: a cash register, heat sensors, light sensors, a clock, noise sensors, etc.
  • the information sent from one or more of the hardware devices listed above is used to adjust target music attributes and/or a rule set for generating output music content for a specific environment.
  • selection element 538 allows the user of application 530 to select one or more hardware devices from which to receive environment input.
  • Display 534 in the illustrated embodiment, displays environment data to the user of application 530 based on information from input hardware 540 .
  • display 532 shows changes to a rule set based on environment data. Display 532 , in some embodiments, allows the user of application 530 to see the changes made based on the environment data.
  • the elements shown in FIGS. 5A and 5B are for theme packs and/or occasion packs. That is, in some embodiments, the user or business using the GUIs displayed by applications 510 and 530 may select/adjust/modify rule sets to generate music content for one or more occasions and/or themes.
  • Audio in constructed tracks may be analyzed (e.g., in real-time) and filtered to mix and master the output stream.
  • Various feedback may be sent to the server, including explicit feedback such as from user interaction with sliders or buttons and implicit feedback, e.g., generated by sensors, based on volume changes, based on listening lengths, environment information, etc.
  • control inputs have known effects (e.g., to specify target music attributes directly or indirectly) and are used by the composition module.
  • a loop library is a master library of loops, which may be stored by a server. Each loop may include audio data and metadata that describes the audio data.
  • a loop package is a subset of the loop library.
  • a loop package may be a pack for a particular artist, for a particular mood, for a particular type of event, etc.
  • Client devices may download loop packs for offline listening or download parts of loop packs on demand, e.g., for online listening.
  • a generated stream in some embodiments, is data that specifies the music content that the user hears when they use the music generator system. Note that the actual output audio signals may vary slightly for a given generated stream, e.g., based on capabilities of audio output equipment.
  • a composition module constructs compositions from loops available in a loop package.
  • the composition module may receive loops, loop metadata, and user input as parameters and may be executed by a client device.
  • the composition module outputs a performance script that is sent to a performance module and one or more machine learning engines.
  • the performance script in some embodiments, outlines which loops will be played on each track of the generated stream and what effects will be applied to the stream.
  • the performance script may utilize beat-relative timing to represent when events occur.
  • the performance script may also encode effect parameters (e.g., for effects such as reverb, delay, compression, equalization, etc.).
  • a performance module receives a performance script as input and renders it into a generated stream.
  • the performance module may produce a number of tracks specified by the performance script and mix the tracks into a stream (e.g., a stereo stream, although the stream may have various encodings including surround encodings, object-based audio encodings, multi-channel stereo, etc. in various embodiments).
  • a stream e.g., a stereo stream, although the stream may have various encodings including surround encodings, object-based audio encodings, multi-channel stereo, etc. in various embodiments.
  • the performance module when provided with a particular performance script, the performance module will always produce the same output.
  • An analytics module in some embodiments, is a server-implemented module that receives feedback information and configures the composition module (e.g., in real-time, periodically, based on administrator commands, etc.).
  • the analytics module uses a combination of machine learning techniques to correlate user feedback with performance scripts and loop library metadata.
  • FIG. 6 is a block diagram illustrating an example music generator system that includes analysis and composition modules, according to some embodiments.
  • the system of FIG. 6 is configured to generate a potentially-infinite stream of music with direct user control over the mood and style of music.
  • the system includes analysis module 610 , composition module 310 , performance module 320 , and audio output device 640 .
  • analysis module 610 is implemented by a server and composition module 310 and performance module 320 are implemented by one or more client devices.
  • modules 610 , 310 , and 320 may all be implemented on a client device or may all be implemented server-side.
  • Analysis module 610 stores one or more artist packs 612 and implements a feature extraction module 614 , a client simulator module 616 , and a deep neural network 618 .
  • feature extraction module 614 adds loops to a loop library after analyzing loop audio (although note that some loops may be received with metadata already generated and may not require analysis). For example, raw audio in a format such as way, aiff, or FLAC may be analyzed for quantifiable musical properties such as instrument classification, pitch transcription, beat timings, tempo, file length, and audio amplitude in multiple frequency bins. Analysis module 610 may also store more abstract musical properties or mood descriptions for loops, e.g., based on manual tagging by artists or machine listening. For example, moods may be quantified using multiple discrete categories, with ranges of values for each category for a given loop.
  • the first beat begins 6 milliseconds into the file
  • the tempo is 122 bpm
  • the file is 6483 milliseconds long
  • the loop has normalized amplitude values of 0.3, 0.5, 0.7, 0.3, and 0.2 across five frequency bins.
  • the artist may label the loop as “funk genre” with the following mood values:
  • Analysis module 610 may store this information in a database and clients may download subsections of the information, e.g., as loop packages. Although artists packs 612 are shown for purposes of illustration, analysis module 610 may provide various types of loop packages to composition module 310 .
  • analysis module 610 generates composition parameters for composition module 310 to improve correlation between desired feedback and use of certain parameters. For example, actual user feedback may be used to adjust composition parameters, e.g., to attempt to reduce negative feedback.
  • module 610 uses a technique such as backpropagation to determine that adjusting probability parameters used to add more tracks reduces the frequency of this issue. For example, module 610 may predict that reducing a probability parameter by 50% will reduce negative feedback by 8% and may determine to perform the reduction and push updated parameters to the composition module (note that probability parameters are discussed in detail below, but any of various parameters for statistical models may similarly be adjusted).
  • negative feedback e.g., explicit low rankings, low volume listening, short listening times, etc.
  • module 610 uses a technique such as backpropagation to determine that adjusting probability parameters used to add more tracks reduces the frequency of this issue. For example, module 610 may predict that reducing a probability parameter by 50% will reduce negative feedback by 8% and may determine to perform the reduction and push updated parameters to the composition module (note that probability parameters are discussed in detail below, but any of various parameters for statistical models may similarly be adjusted).
  • module 610 may increase a parameter such that the probability of selecting loops with high tension tags is increased when users ask for high tension music.
  • the machine learning may be based on various information, including composition outputs, feedback information, user control inputs, etc.
  • Composition module 310 in the illustrated embodiment, includes a section sequencer 622 , section arranger 624 , technique implementation module 626 , and loop selection module 628 .
  • composition module 310 organizes and constructs sections of the composition based on loop metadata and user control input (e.g., mood control).
  • Section sequencer 622 sequences different types of sections.
  • section sequencer 622 implements a finite state machine to continuously output the next type of section during operation.
  • composition module 310 may be configured to use different types of sections such as an intro, buildup, drop, breakdown, and bridge, as discussed in further detail below with reference to FIG. 10 .
  • each section may include multiple subsections that define how the music changes throughout a section, e.g., including a transition-in subsection, a main content subsection, and a transition-out subsection.
  • Section arranger 624 constructs subsections according to arranging rules. For example, one rule may specify to transition-in by gradually adding tracks. Another rule may specify to transition-in by gradually increasing gain on a set of tracks. Another rule may specify to chop a vocal loop to create a melody.
  • the probability of a loop in the loop library being appended to a track is a function of the current position in a section or subsection, loops that overlap in time on another track, and user input parameters such as a mood variable (which may be used to determine target attributes for generated music content). The function may be adjusted, e.g., by adjusting coefficients based on machine learning.
  • Technique implementation module 310 is configured to facilitate section arrangement by adding rules, e.g., as specified by an artist or determined by analyzing compositions of a particular artist.
  • a “technique” may describe how a particular artist implements arrangement rules at a technical level. For example, for an arrangement rule that specifies to transition-in by gradually adding tracks, one technique may indicate to add tracks in order of drums, bass, pads, then vocals while another technique may indicate to add tracks in order of bass, pads, vocals, then drums. Similarly, for an arrangement rule that specifies to chop a vocal loop to create a melody a technique may indicate to chop vocals on every second beat and repeat a chopped section of loop twice before moving to the next chopped section.
  • Loop selection module 628 selects loops according to the arrangement rules and techniques, for inclusion in a section-by-section arranger 624 . Once sections are complete, corresponding performance scripts may be generated and sent to performance module 320 . Performance module 320 may receive performance script portions at various granularities. This may include, for example, an entire performance script for a performance of a certain length, a performance script for each section, a performance script for each sub-section, etc. In some embodiments, arrangement rules, techniques, or loop selection are implemented statistically, e.g., with different approaches used different percentages of the time.
  • Performance module 320 includes filter module 631 , effect module 632 , mix module 633 , master module 634 , and perform module 635 .
  • these modules process the performance script and generate music data in a format supported by audio output device 640 .
  • the performance script may specify the loops to be played, when they should be played, what effects should be applied by module 632 (e.g., on a per-track or per-subsection basis), what filters should be applied by module 631 , etc.
  • the performance script may specify to apply a low pass filter ramping from 1000 to 20000 Hz from 0 to 5000 milliseconds on a particular track.
  • the performance script may specify to apply reverb with a 0.2 wet setting from 5000 to 15000 milliseconds on a particular track.
  • Mix module 633 in some embodiments, is configured to perform automated level control for the tracks being combined. In some embodiments, mix module 633 uses frequency domain analysis of the combined tracks to measure frequencies with too much or too little energy and applies gain to tracks in different frequency bands to even the mix. Master module 634 , in some embodiments, is configured to perform multi-band compression, equalization (EQ), or limiting procedures to generate data for final formatting by perform module 635 . The embodiment of FIG. 6 may automatically generate various output music content according to user input or other feedback information, while the machine learning techniques may allow for improved user experience over time.
  • EQ multi-band compression, equalization
  • FIG. 7 is a diagram illustrating an example buildup section of music content, according to some embodiments.
  • the system of FIG. 6 may compose such a section by applying arranging rules and techniques.
  • the buildup section includes three subsections and separate tracks for vocals, pad, drum, bass, and white noise.
  • the transition in subsection in the illustrated example, includes a drum loop A, which is also repeated for the main content subsection.
  • the transition in subsection also includes a bass loop A.
  • the gain for the section begins low and increases linearly throughout the section (although non-linear increases or decreases are contemplated).
  • the main content and transition-out subsection, in the illustrated example include various vocal, pad, drum, and bass loops.
  • disclosed techniques for automatically sequencing sections, arranging sections, and implementing techniques may generate near-infinite streams of output music content based on various user-adjustable parameters.
  • a computer system displays an interface similar to FIG. 7 and allows artists to specify techniques used to compose sections. For example, artists may create structures such as shown in FIG. 7 which may be parsed into code for the composition module.
  • FIG. 8 is a diagram illustrating example techniques for arranging sections of music content, according to some embodiments.
  • a generated stream 810 includes multiple sections 820 that each include a start subsection 822 , development subsection 824 , and transition subsection 826 .
  • multiple types of each section/subsection are show in tables connected via dotted lines.
  • the circular elements, in the illustrated embodiment are examples of arranging tools, which may further be implemented using specific techniques as discussed below. As shown, various composition decisions may be performed pseudo-randomly according to statistical percentages. For example, the types of subsections, the arranging tools for a particular type or subsection, or the techniques used to implement an arranging tool may be statistically determined.
  • a given section 820 is one of five types: intro, buildup, drop, breakdown, and bridge, each with different functions that control intensity over the section.
  • the state sub-section in this example, is one of three types: slow build, sudden shift, or minimal, each with different behavior.
  • the development sub-section, in this example is one of three types, reduce, transform, or augment.
  • the transition sub-section in this example, is one of three types: collapse, ramp, or hint.
  • the different types of sections and subsections may be selected based on rules or may be pseudo-randomly selected, for example.
  • the behaviors for different subsection types are implemented using one or more arranging tools.
  • a slow build in this example, 40% of the time a low pass filter is applied and 80% of the time layers are added.
  • 25% of the time loops are chopped.
  • additional arranging tools are shown, including one-shot, dropout beat, apply reverb, add pads, add theme, remove layers, and white noise.
  • one or more arranging tools may be implemented using specific techniques (which may be artist specified or determined based on analysis of an artist's content). For example, one-shot may be implemented using sound-effects or vocals, loop chopping may be implemented using stutter or chop-in-half techniques, removing layers may be implemented by removing synth or removing vocals, white noise may be implemented using a ramp or pulse function, etc.
  • the specific technique selected for a given arranging tool may be selected according to a statistical function (e.g., 30% of the time removing layers may remove synths and 70% of the time it may remove vocals for a given artist).
  • arranging rules or techniques may be determined automatically by analyzing existing compositions, e.g., using machine learning.
  • FIG. 9 is a flow diagram method for controlling a user-specific music control element, according to some embodiments.
  • the method shown in FIG. 9 may be used in conjunction with any of the computer circuitry, systems, devices, elements, or components disclosed herein, among others.
  • some of the method elements shown may be performed concurrently, in a different order than shown, or may be omitted. Additional method elements may also be performed as desired.
  • a computing system receives user input specifying a user-defined music control element.
  • a computing system receives user input specifying labels for the user-defined music control element for one or more musical compositions.
  • a computing system trains, based on the labels, one or more machine learning models to: adjust, based on user input to the user-defined music control element, one or more composition parameters for selecting audio tracks to be combined to generate output music content and adjust, based on user input to the user-defined music control element, one or more performance parameters for generating output music content based on selected audio tracks.
  • a computing system causes output music content to be generated, according to the one or more composition parameters and one or more performance parameters, by combining multiple audio tracks.
  • the computing system transmits configuration information based on the one or more machine learning models to a user device, where the configuration information indicates how to adjust performance parameters based on user input to the user-defined music control element. In some embodiments, the computing system adjusts, according to the one or more machine learning models, one or more composition parameters based on user input to the user-defined music control element. In some embodiments, a user device of the computing system adjusts, according to the configuration information, one or more composition parameters based on user input to the user-defined music control element.
  • the transmission uses delimited strings to specify a data target, wherein the method further comprising forming strings based on a data structure with pointers to offsets in a string, wherein the string formation includes prepending or appending to an existing string.
  • the training is further based on user input that explicitly specifies relationships between composition parameters and user input to the user-defined music control element.
  • the one or more composition parameters includes parameters for: building, sustaining, and reducing tension between musical sections, enabling or disabling audio file categories, adjusting one or more of the following music aspects for one or more audio file categories: volume, reverb amount, delay punch probability, and delay wetness, etc.
  • the one or more performance parameters include at least one of the following types of parameters: filter parameters, effects parameters, and mix parameters.
  • FIG. 10 is a flow diagram method for generating music content based on a user-specific music control element, according to some embodiments.
  • the method shown in FIG. 10 may be used in conjunction with any of the computer circuitry, systems, devices, elements, or components disclosed herein, among others.
  • some of the method elements shown may be performed concurrently, in a different order than shown, or may be omitted. Additional method elements may also be performed as desired.
  • a computing system receives configuration information, where the configuration information specifies how to adjust one or more performance parameters based on user adjustments to a user-defined music control element and where the one or more performance parameters are for generating output music content based on selected audio tracks.
  • a computing system receives user input via a first user interface, where the user input specifies an adjustment to the user-defined music control element.
  • a computing system adjusts one of more of the one or more performance parameters based on the user input and the configuration information.
  • a computing system generates output music content by combining multiple audio files according to the one or more adjusted performance parameters.
  • the computing system receives composition control information (e.g., a composition script) that indicates the multiple audio files to be combined and indicates transitions between musical sections.
  • composition control information e.g., a composition script
  • the composition control information is generated based on user input (e.g., by an artist via an artist user interface or an end-user via the first user interface) specifying an adjustment to the user-defined music control element.
  • the user-defined music control element is defined via a second user interface generated by a server system that transmitted the configuration information. In some embodiments, the user-defined music control element is defined by a different user than a user that provides the user input via the first user interface.
  • the computing system parses the configuration information using pointers to locations within a delimited string that indicates a target of the configuration information.
  • This disclosure may discuss potential advantages that may arise from the disclosed embodiments. Not all implementations of these embodiments will necessarily manifest any or all of the potential advantages. Whether an advantage is realized for a particular implementation depends on many factors, some of which are outside the scope of this disclosure. In fact, there are a number of reasons why an implementation that falls within the scope of the claims might not exhibit some or all of any disclosed advantages. For example, a particular implementation might include other circuitry outside the scope of the disclosure that, in conjunction with one of the disclosed embodiments, negates or diminishes one or more the disclosed advantages. Furthermore, suboptimal design execution of a particular implementation (e.g., implementation techniques or tools) could also negate or diminish disclosed advantages.
  • embodiments are non-limiting. That is, the disclosed embodiments are not intended to limit the scope of claims that are drafted based on this disclosure, even where only a single example is described with respect to a particular feature.
  • the disclosed embodiments are intended to be illustrative rather than restrictive, absent any statements in the disclosure to the contrary. The application is thus intended to permit claims covering disclosed embodiments, as well as such alternatives, modifications, and equivalents that would be apparent to a person skilled in the art having the benefit of this disclosure.
  • references to a singular form of an item i.e., a noun or noun phrase preceded by “a,” “an,” or “the” are, unless context clearly dictates otherwise, intended to mean “one or more.” Reference to “an item” in a claim thus does not, without accompanying context, preclude additional instances of the item.
  • a “plurality” of items refers to a set of two or more of the items.
  • a recitation of “w, x, y, or z, or any combination thereof” or “at least one of . . . w, x, y, and z” is intended to cover all possibilities involving a single element up to the total number of elements in the set. For example, given the set [w, x, y, z], these phrasings cover any single element of the set (e.g., w but not x, y, or z), any two elements (e.g., w and x, but not y or z), any three elements (e.g., w, x, and y, but not z), and all four elements.
  • w, x, y, and z thus refers to at least one element of the set [w, x, y, z], thereby covering all possible combinations in this list of elements. This phrase is not to be interpreted to require that there is at least one instance of w, at least one instance of x, at least one instance of y, and at least one instance of z.
  • labels may precede nouns or noun phrases in this disclosure.
  • different labels used for a feature e.g., “first circuit,” “second circuit,” “particular circuit,” “given circuit,” etc.
  • labels “first,” “second,” and “third” when applied to a feature do not imply any type of ordering (e.g., spatial, temporal, logical, etc.), unless stated otherwise.
  • a determination may be solely based on specified factors or based on the specified factors as well as other, unspecified factors.
  • an entity described or recited as being “configured to” perform some task refers to something physical, such as a device, circuit, a system having a processor unit and a memory storing program instructions executable to implement the task, etc. This phrase is not used herein to refer to something intangible.
  • circuits may be described in this disclosure. These circuits or “circuitry” constitute hardware that includes various types of circuit elements, such as combinatorial logic, clocked storage devices (e.g., flip-flops, registers, latches, etc.), finite state machines, memory (e.g., random-access memory, embedded dynamic random-access memory), programmable logic arrays, and so on. Circuitry may be custom designed, or taken from standard libraries. In various implementations, circuitry can, as appropriate, include digital components, analog components, or a combination of both. Certain types of circuits may be commonly referred to as “units” (e.g., a decode unit, an arithmetic logic unit (ALU), functional unit, memory management unit (MMU), etc.). Such units also refer to circuits or circuitry.
  • ALU arithmetic logic unit
  • MMU memory management unit
  • circuits/units/components and other elements illustrated in the drawings and described herein thus include hardware elements such as those described in the preceding paragraph.
  • the internal arrangement of hardware elements within a particular circuit may be specified by describing the function of that circuit.
  • a particular “decode unit” may be described as performing the function of “processing an opcode of an instruction and routing that instruction to one or more of a plurality of functional units,” which means that the decode unit is “configured to” perform this function.
  • This specification of function is sufficient, to those skilled in the computer arts, to connote a set of possible structures for the circuit.
  • circuits, units, and other elements may be defined by the functions or operations that they are configured to implement.
  • the arrangement and such circuits/units/components with respect to each other and the manner in which they interact form a microarchitectural definition of the hardware that is ultimately manufactured in an integrated circuit or programmed into an FPGA to form a physical implementation of the microarchitectural definition.
  • the microarchitectural definition is recognized by those of skill in the art as structure from which many physical implementations may be derived, all of which fall into the broader structure described by the microarchitectural definition.
  • HDL hardware description language
  • Such an HDL description may take the form of behavioral code (which is typically not synthesizable), register transfer language (RTL) code (which, in contrast to behavioral code, is typically synthesizable), or structural code (e.g., a netlist specifying logic gates and their connectivity).
  • the HDL description may subsequently be synthesized against a library of cells designed for a given integrated circuit fabrication technology, and may be modified for timing, power, and other reasons to result in a final design database that is transmitted to a foundry to generate masks and ultimately produce the integrated circuit.
  • Some hardware circuits or portions thereof may also be custom-designed in a schematic editor and captured into the integrated circuit design along with synthesized circuitry.
  • the integrated circuits may include transistors and other circuit elements (e.g., passive elements such as capacitors, resistors, inductors, etc.) and interconnect between the transistors and circuit elements. Some embodiments may implement multiple integrated circuits coupled together to implement the hardware circuits, and/or discrete elements may be used in some embodiments. Alternatively, the HDL design may be synthesized to a programmable logic array such as a field programmable gate array (FPGA) and may be implemented in the FPGA.
  • FPGA field programmable gate array

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Software Systems (AREA)
  • Computing Systems (AREA)
  • Artificial Intelligence (AREA)
  • Mathematical Physics (AREA)
  • Evolutionary Computation (AREA)
  • Biophysics (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Computational Linguistics (AREA)
  • Biomedical Technology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Molecular Biology (AREA)
  • Acoustics & Sound (AREA)
  • Databases & Information Systems (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Medical Informatics (AREA)
  • Auxiliary Devices For Music (AREA)
  • Electrophonic Musical Instruments (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
US17/408,105 2020-08-21 2021-08-20 Music Generator Generation of Continuous Personalized Music Pending US20220059063A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US17/408,105 US20220059063A1 (en) 2020-08-21 2021-08-20 Music Generator Generation of Continuous Personalized Music

Applications Claiming Priority (8)

Application Number Priority Date Filing Date Title
US202063068415P 2020-08-21 2020-08-21
US202063068411P 2020-08-21 2020-08-21
US202063068431P 2020-08-21 2020-08-21
US202063068436P 2020-08-21 2020-08-21
US202063068421P 2020-08-21 2020-08-21
US202063068433P 2020-08-21 2020-08-21
US202063068426P 2020-08-21 2020-08-21
US17/408,105 US20220059063A1 (en) 2020-08-21 2021-08-20 Music Generator Generation of Continuous Personalized Music

Publications (1)

Publication Number Publication Date
US20220059063A1 true US20220059063A1 (en) 2022-02-24

Family

ID=77739139

Family Applications (2)

Application Number Title Priority Date Filing Date
US17/408,076 Pending US20220059062A1 (en) 2020-08-21 2021-08-20 Comparison Training for Music Generator
US17/408,105 Pending US20220059063A1 (en) 2020-08-21 2021-08-20 Music Generator Generation of Continuous Personalized Music

Family Applications Before (1)

Application Number Title Priority Date Filing Date
US17/408,076 Pending US20220059062A1 (en) 2020-08-21 2021-08-20 Comparison Training for Music Generator

Country Status (6)

Country Link
US (2) US20220059062A1 (ja)
EP (1) EP4200719A1 (ja)
JP (1) JP7547621B2 (ja)
KR (1) KR20230052298A (ja)
CN (1) CN115885276A (ja)
WO (1) WO2022040410A1 (ja)

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20210312898A1 (en) * 2018-08-13 2021-10-07 Viscount International S.P.A. Generation system of synthesized sound in music instruments
US20220059062A1 (en) * 2020-08-21 2022-02-24 Aimi Inc. Comparison Training for Music Generator
US20220262328A1 (en) * 2021-02-16 2022-08-18 Wonder Inventions, Llc Musical composition file generation and management system
US11551652B1 (en) * 2019-11-27 2023-01-10 Amazon Technologies, Inc. Hands-on artificial intelligence education service
WO2024081836A1 (en) * 2022-10-14 2024-04-18 Aimi Inc. Dynamic control of generative music composition
WO2024178038A1 (en) * 2023-02-24 2024-08-29 Aimi Inc. Machine learning model trained based on music and decisions generated by expert system

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP7440651B2 (ja) * 2020-02-11 2024-02-28 エーアイエムアイ インコーポレイテッド 音楽コンテンツの生成

Citations (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080092721A1 (en) * 2006-10-23 2008-04-24 Soenke Schnepel Methods and apparatus for rendering audio data
US20090132075A1 (en) * 2005-12-19 2009-05-21 James Anthony Barry interactive multimedia apparatus
US20140052282A1 (en) * 2012-08-17 2014-02-20 Be Labs, Llc Music generator
US20150013528A1 (en) * 2013-07-13 2015-01-15 Apple Inc. System and method for modifying musical data
US20180219638A1 (en) * 2015-09-30 2018-08-02 Yamaha Corporation Audio processing device and audio processing method
US20190362696A1 (en) * 2018-05-24 2019-11-28 Aimi Inc. Music generator
US20200135237A1 (en) * 2017-06-29 2020-04-30 Virtual Voices Pty Ltd Systems, Methods and Applications For Modulating Audible Performances
US20200168186A1 (en) * 2017-08-02 2020-05-28 Yamaha Corporation Learning model generation method, learning model generation device, and automatic musical performance robot
US20210248213A1 (en) * 2020-02-11 2021-08-12 Aimi Inc. Block-Chain Ledger Based Tracking of Generated Music Content
US20210343262A1 (en) * 2020-04-29 2021-11-04 At&T Intellectual Property I, L.P. System for ledger and parametric marketplace for algorithmic content creations
US20210390938A1 (en) * 2020-03-06 2021-12-16 Algoriddim Gmbh Method, device and software for applying an audio effect
US20220059062A1 (en) * 2020-08-21 2022-02-24 Aimi Inc. Comparison Training for Music Generator

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6051770A (en) * 1998-02-19 2000-04-18 Postmusic, Llc Method and apparatus for composing original musical works
JP4626376B2 (ja) * 2005-04-25 2011-02-09 ソニー株式会社 音楽コンテンツの再生装置および音楽コンテンツ再生方法
US9880805B1 (en) * 2016-12-22 2018-01-30 Brian Howard Guralnick Workout music playback machine

Patent Citations (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090132075A1 (en) * 2005-12-19 2009-05-21 James Anthony Barry interactive multimedia apparatus
US20080092721A1 (en) * 2006-10-23 2008-04-24 Soenke Schnepel Methods and apparatus for rendering audio data
US20140052282A1 (en) * 2012-08-17 2014-02-20 Be Labs, Llc Music generator
US20150013528A1 (en) * 2013-07-13 2015-01-15 Apple Inc. System and method for modifying musical data
US20180219638A1 (en) * 2015-09-30 2018-08-02 Yamaha Corporation Audio processing device and audio processing method
US20200135237A1 (en) * 2017-06-29 2020-04-30 Virtual Voices Pty Ltd Systems, Methods and Applications For Modulating Audible Performances
US20200168186A1 (en) * 2017-08-02 2020-05-28 Yamaha Corporation Learning model generation method, learning model generation device, and automatic musical performance robot
US20210027754A1 (en) * 2018-05-24 2021-01-28 Aimi Inc. Music generator
US20190362696A1 (en) * 2018-05-24 2019-11-28 Aimi Inc. Music generator
US20230020181A1 (en) * 2018-05-24 2023-01-19 Aimi Inc. Music generator
US20210248213A1 (en) * 2020-02-11 2021-08-12 Aimi Inc. Block-Chain Ledger Based Tracking of Generated Music Content
US20210248983A1 (en) * 2020-02-11 2021-08-12 Aimi Inc. Music Content Generation Using Image Representations of Audio Files
US20210247955A1 (en) * 2020-02-11 2021-08-12 Aimi Inc. Listener-Defined Controls for Music Content Generation
US20210390938A1 (en) * 2020-03-06 2021-12-16 Algoriddim Gmbh Method, device and software for applying an audio effect
US20210343262A1 (en) * 2020-04-29 2021-11-04 At&T Intellectual Property I, L.P. System for ledger and parametric marketplace for algorithmic content creations
US20220059062A1 (en) * 2020-08-21 2022-02-24 Aimi Inc. Comparison Training for Music Generator

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20210312898A1 (en) * 2018-08-13 2021-10-07 Viscount International S.P.A. Generation system of synthesized sound in music instruments
US11615774B2 (en) * 2018-08-13 2023-03-28 Viscount International S.P.A. Generation system of synthesized sound in music instruments
US11551652B1 (en) * 2019-11-27 2023-01-10 Amazon Technologies, Inc. Hands-on artificial intelligence education service
US20220059062A1 (en) * 2020-08-21 2022-02-24 Aimi Inc. Comparison Training for Music Generator
US20220262328A1 (en) * 2021-02-16 2022-08-18 Wonder Inventions, Llc Musical composition file generation and management system
US11869468B2 (en) * 2021-02-16 2024-01-09 Roam Hq, Inc. Musical composition file generation and management system
WO2024081836A1 (en) * 2022-10-14 2024-04-18 Aimi Inc. Dynamic control of generative music composition
WO2024178038A1 (en) * 2023-02-24 2024-08-29 Aimi Inc. Machine learning model trained based on music and decisions generated by expert system

Also Published As

Publication number Publication date
EP4200719A1 (en) 2023-06-28
JP2023538411A (ja) 2023-09-07
JP7547621B2 (ja) 2024-09-09
KR20230052298A (ko) 2023-04-19
US20220059062A1 (en) 2022-02-24
WO2022040410A1 (en) 2022-02-24
CN115885276A (zh) 2023-03-31

Similar Documents

Publication Publication Date Title
US11450301B2 (en) Music generator
US20220059063A1 (en) Music Generator Generation of Continuous Personalized Music
US11625217B2 (en) Music generator
US11635936B2 (en) Audio techniques for music content generation

Legal Events

Date Code Title Description
AS Assignment

Owner name: AIMI INC., COLORADO

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:BALASSANIAN, EDWARD;HUTCHINGS, PATRICK E.;REEL/FRAME:057244/0745

Effective date: 20210818

AS Assignment

Owner name: AIMI INC., COLORADO

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:GIFFORD, TOBY;REEL/FRAME:057254/0682

Effective date: 20210823

STPP Information on status: patent application and granting procedure in general

Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS