EP3086315A1 - Vorrichtung und verfahren zur bearbeitung von audioinformationen, die mit einer visuellen darstellung synchronisiert sind - Google Patents

Vorrichtung und verfahren zur bearbeitung von audioinformationen, die mit einer visuellen darstellung synchronisiert sind Download PDF

Info

Publication number
EP3086315A1
EP3086315A1 EP15305610.6A EP15305610A EP3086315A1 EP 3086315 A1 EP3086315 A1 EP 3086315A1 EP 15305610 A EP15305610 A EP 15305610A EP 3086315 A1 EP3086315 A1 EP 3086315A1
Authority
EP
European Patent Office
Prior art keywords
audio information
modification
visual representation
musical
audio
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
EP15305610.6A
Other languages
English (en)
French (fr)
Inventor
Marie Guegan
Anne Lambert
Alexey Ozerov
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Thomson Licensing SAS
Original Assignee
Thomson Licensing SAS
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Thomson Licensing SAS filed Critical Thomson Licensing SAS
Priority to EP15305610.6A priority Critical patent/EP3086315A1/de
Publication of EP3086315A1 publication Critical patent/EP3086315A1/de
Withdrawn legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H1/00—Details of electrophonic musical instruments
    • G10H1/0008—Associated control or indicating means
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2220/00—Input/output interfacing specifically adapted for electrophonic musical tools or instruments
    • G10H2220/091—Graphical user interface [GUI] specifically adapted for electrophonic musical instruments, e.g. interactive musical displays, musical instrument icons or menus; Details of user interactions therewith
    • G10H2220/101—Graphical user interface [GUI] specifically adapted for electrophonic musical instruments, e.g. interactive musical displays, musical instrument icons or menus; Details of user interactions therewith for graphical creation, edition or control of musical data or parameters
    • G10H2220/121—Graphical user interface [GUI] specifically adapted for electrophonic musical instruments, e.g. interactive musical displays, musical instrument icons or menus; Details of user interactions therewith for graphical creation, edition or control of musical data or parameters for graphical editing of a musical score, staff or tablature

Definitions

  • the present principles relate to an apparatus and a method for editing audio information, and more particularly, to an apparatus and a method for editing audio information using a visual representation synchronized with the audio information.
  • a musical score is a form of musical notation that describes how music should be played. The more complex the musical piece, the more elaborate the notation required. A musician typically learns to read this type of musical notation in order to play music that they may not be familiar with. The reverse can also be true in that a musician may want to create a musical score in order that others can be exposed to their compositions or style of playing a piece of music. Thus, musicians often play and record versions of a musical score for various reasons. If problems or errors in the musicians performance are found, this will often be edited by professionals who directly edit an audio file that was recorded as the musician played. However, amateur musicians may not have the technical training to edit the audio files nor the means to hire a professional to perform this type of task.
  • Musical score edits are automatically transcribed into instructions that are utilized to change a corresponding audio file that has been previously aligned to the original musical score.
  • An alignment item can be represented as a pair of a musical symbol from the score and a set of corresponding timestamps in the audio file.
  • the user edits the musical score by opening a digital form of the music score and enters a modification to the score.
  • the edit is then applied to the aligned audio file and a new modified audio file is created that reflects the modified musical score.
  • an apparatus for editing audio information can comprise a first editor that receives at least one modification to a visual representation of audio information and a second editor that transforms the at least one modification to the visual representation into at least one modification to audio information synchronized with the visual representation and applies the transformed modification to the audio information to output modified audio information.
  • the apparatus can comprise a processor configured to receive audio information that has been synchronized with a visual representation of the audio information, receive at least one modification of the visual representation of the audio information, transform the at least one visual presentation modification into at least one audio information modification and apply the audio information modification to the audio information to output modified audio information.
  • the apparatus can include means for receiving at least one modification to a visual representation of the audio information and means for transforming the at least one modification to the visual representation into at least one modification to audio information synchronized with the visual representation and applying the transformed modification to the audio information to output modified audio information.
  • the apparatus can further comprise means for receiving the audio information and synchronizing the audio information with the visual representation of the audio information.
  • aspects of the apparatus can include audio information that is audio signal data aligned with the visual representation, the visual representation of the audio information is a musical score aligned with the audio information; the musical score has musical notation synchronized with the audio information; at least one modification that comprises at least one of changing a duration of a musical note, changing a tonality of a musical note, changing a volume of a musical note, adding a new musical note and changing an occurrence of a musical note; a transformed modification that is applied to the audio information as a relative change; and/or the transformed modification to the audio information is applied as an absolute change.
  • a method for editing audio information can comprise receiving at least one modification of a visual representation of the audio information, transforming the at least one modification of the visual presentation into at least one audio information modification, and applying the audio information modification to the audio information to create modified audio information.
  • Another aspect of the method can include further comprising receiving the audio information and synchronizing the audio information with the visual representation of the audio information.
  • Yet other aspects of the method can include audio information that is audio signal data aligned with the visual representation; a visual representation of the audio information that is a musical score aligned with the audio information; a musical score that has musical notation synchronized with the audio information; a modification that comprises at least one of changing a duration of a musical note, changing a tonality of a musical note, changing a volume of a musical note, adding a new musical note and changing an occurrence of a musical note; and/or a visual presentation modification that is applied to the audio information by at least one of a relative change and an absolute change.
  • the present principles relate to a non-transitory program storage device, readable by a computer.
  • said non-transitory computer readable program product tangibly embodies a program of instructions executable by a computer to perform the method described above, in any of its embodiments.
  • the present principles relate to a computer readable storage medium carrying a software program comprising program code instructions for performing the method of the present principles, in any of its embodiments/aspects, when said non transitory software program is executed by a computer.
  • a computer readable storage medium can take the form of a computer readable program product embodied in one or more computer readable medium(s) and having computer readable program code embodied thereon that is executable by a computer.
  • a computer readable storage medium as used herein is considered a non-transitory storage medium given the inherent capability to store the information therein as well as the inherent capability to provide retrieval of the information therefrom.
  • a computer readable storage medium can be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing.
  • a user is given the ability to edit an audio file containing audio tracks using only a corresponding musical score. Contrary to traditional approaches, it requires no prior knowledge from the user about audio file processing.
  • the techniques also enable the user to automate the processing for a given sequence of notes.
  • One embodiment of these techniques consists of the user importing the score and the audio file, editing the score, and exporting the new audio file.
  • edits to musical scores are automatically transcribed into elementary instructions that can be utilized to change a corresponding synchronized audio file.
  • an audio recording and a musical score that was played in the audio recording are aligned or synchronized together.
  • the output is an alignment between the audio file and the musical score, consisting of an ordered sequence of alignment items.
  • An alignment item can be represented as a pair of a musical symbol from the score and a set of corresponding timestamps in the audio file.
  • the user then opens the digital sheet music and enters a modification to a sheet.
  • the edit is then applied to the aligned audio file and a new modified audio file is created.
  • Some of the advantages of utilizing these techniques include, but are not limited to, providing an alternative way to existing methods for editing an audio file of a musical recording, providing a way of editing a recording to users who do not have any knowledge in audio signal processing, providing a faster way to edit musical recordings, providing a way to apply the exact same processing to several recordings of the same score - by interacting with the score only and providing a way to apply the exact same processing to a sequence appearing several times in the music piece.
  • the techniques described herein can be applied to music postprocessing such as, but not limited to, editing the mistakes of a player when the music was recorded (e.g., wrong notes in the original score or wrong notes played), changing the interpretation of the player (volume, rhythm, speed), repeating a few bars by adding begin/end repeat bar-lines or by copy-pasting selected bar sequence, composing new music by ordering the bars differently and/or changing the music key or the tonality of the full music recording or of a part of it. It can also be applied to multitrack composition such as, but not limited to, mixing together different tracks that were recorded separately (thus, possibly not synchronized), by just aligning them with the corresponding scores and combining the scores together. It can also be applied to movie postproduction such as, but not limited to, adapting the recorded music track to a video, for example by adapting the original soundtrack to a specific scene in a movie so as to better synchronize highlights in the music and in the scene.
  • FIG. 1 An example of a system 100 for this process is illustrated in FIG. 1 .
  • An audio recording 102 of a musical score 104 and the musical score 104 are provided for alignment or synchronization 106.
  • an 'audio recording' is typically a capture of 'audio tracks' which can include a broad spectrum of audio frequencies. These audio tracks can be digitized into an 'audio file.' These terms can be used interchangeably to represent the audio recording 102.
  • the audio tracks can be analog as well. In general terms, whether it is digital or analog, the audio recording 102 can be referred to as 'audio information.'
  • the alignment/synchronization is accomplished by representing a segment of the audio recording 102 with at least one musical note from the musical score 104.
  • the segment of the audio recording can be a point in time and/or a range of time. For example, when a note of the musical score is initiated (e.g., an attack for a piano note), a link to the time in the audio recording 102 can be made. In another example, when the note is first initiated, a link to the time in the audio recording 102 can be made along with an ending time and/or a duration of time.
  • an A flat note can start at 1:02 (one minute, two seconds) and end at 1:04 (one minute, four seconds) or start at 1:02 (one minute, two seconds) and last (duration) for 2 seconds.
  • One skilled in the art will appreciate that the alignment/synchronization between the audio recording 102 and the musical score 104 can be accomplished in many different ways.
  • a user 110 can interact with the musical score 104 using a musical score editor 112.
  • the musical score editor 112 can present a musical score to the user 110 such that the user 110 can edit individual notes and/or a series or sequence of notes of the musical score 104.
  • the musical score editor 112 accepts edits or modifications from the user 110 and processes the edits into at least one modification instruction 114 for the musical score 104.
  • a score-based audio editor 116 implements the modification instruction(s) on the aligned audio recording and creates a modified audio recording 118 that represents the modified musical score that includes the user's edits to the original musical score 104.
  • a user is a flute player. She records herself playing a particular piece of sheet music.
  • the sheet music is of a digital form and saved on an electronic device in a musical format such as MusicXML (Music Extensible Markup Language).
  • the user is able to read the music using a system that can represent the file graphically.
  • the obtained recording is a single audio track where she plays the whole musical piece 202 as illustrated in the example 200 of FIG. 2 .
  • the audio recording is then aligned/synchronized to the musical piece 202.
  • the alignment can be obtained by various techniques such as, for example, saving the alignment while a user reads and plays the music, including by following the notes that are being played and/or by tracking the eye movement on the sheet music, etc.
  • the alignment can also be obtained by carrying out postprocessing of the audio file and the sheet music.
  • the output of the alignment can be in the simplified form of musical note and time alignment notations such as the following example:
  • the user listens to the recording and may not be satisfied with it. For example, she wants to modify it but she does not know anything about signal processing of an associated audio file. As she can read the sheet music, she opens the musical piece in a musical score editor that enables her to modify it. She makes several modifications to the score of the musical piece 302.
  • the modified musical piece 302 is illustrated in the example 300 of FIG. 3 .
  • the first note was an eighth note and is now a quarter note, it is now twice as long as before 304, there is a "piano" symbol indicating that the first, second and last bars should be played piano 306, the third note is now a D instead of a C 308, and the repeat bar was deleted and thus the first bar should not be repeated anymore 310.
  • the first way - absolute change apply the modifications as they are written - the first note is prolonged to the duration of 4 th notes according to the tempo followed by the player, the volume of the last 3 bars is set to piano, the third note is set to a D and the second occurrence of the first bar is deleted.
  • the corresponding modified audio file, aligned with the sheet music, can then follow this pattern:
  • the first note was prolonged from 1 to 2 seconds, starting from second 2 instead of 3.
  • the repeat line was deleted.
  • the four following notes were kept from seconds 4 to 12 but since the repeat line was deleted, the second occurrences of these notes were also deleted.
  • the volume was set to 1 ("piano") for all other notes.
  • the third note was changed from C to D. The last note was brought earlier, starting second 16 instead of second 28.
  • the second way of interpreting these changes can be a relative change - apply the modifications relatively to what was recorded: the first note is prolonged to twice its original duration, the volume of the last 3 bars is reduced, the third note is set one tone higher and the second occurrence of the first bar is deleted.
  • These types of relative changes can be easier to process.
  • the following elementary operations can be included in an implementation of a "score-based audio editor" - changing the duration of one note, changing the tonality of one note, changing the volume of one note, adding a new note and/or changing the absolute position of one note.
  • other modifications are combinations of these five elementary ones.
  • audio source separation methods can be used.
  • the goal of audio source separation is to separate (disassociate) individual sources (e.g., vocal track, piano track, etc...., in the case of musical recordings).
  • the audio source separation techniques can be applied so as to extract individual musical notes within the corresponding musical sources.
  • the performance of audio source (or notes) separation can be greatly improved when the musical score is available along with the corresponding audio recording, which is the case here.
  • the third operation (changing the volume of one note) can be performed simply by multiplying the note waveform by the corresponding positive gain.
  • the first, second and third operations can be alternatively implemented not as modifications of the notes themselves, but as replacement of these notes by the notes with desired properties from other parts of the performance, if such notes are available. Such an alternative could lead to better results in some situations. Indeed, for example changing the volume of one note (the third operation), meaning making this note so as it was played stronger, does not consist exactly in multiplying the waveform by a gain. Thus, copying a note with desired amplitude from another place can lead to a better result.
  • the note with the corresponding characteristics can be found in some other place of the same recording and copied to the desired place.
  • the desired note (with exactly the characteristics as specified) does not exist in the recording, it can be obtained from some other close note (e.g., all characteristics are OK except the duration) by modifying it using the first, second and/or third operations.
  • FIG. 4 an example 400 of an apparatus that transforms changes to a visual representation of audio information 404 (e.g., a musical score, etc.) into changes applied to audio information 402 (e.g., audio track, audio recording, audio file, etc.) in accordance with an embodiment of the present principles is shown.
  • the audio information 402 can be, but is not limited to, audio signal data and the like associated with the visual representation of the audio information 404.
  • the audio information 402 can be stored data such as file data and/or it can be streamed data and the like.
  • the audio information is typically, but not limited to, audio frequency information related to musical scores and the like.
  • the original form of the audio information 402 can be digital or analog.
  • Analog audio information is typically digitized before use, but one skilled in the art can appreciate that modifications can be made to analog recordings as well.
  • the visual representation of the audio information 404 can be, but is not limited to, musical notations and the like such as those found in musical scores. One skilled in the art can appreciate that other types of notations and symbology can be used to make a visual representation of audio information.
  • a synchronizer 406 is used to align or synchronize the visual representation of the audio information 404 with the audio information 402.
  • This alignment information is received by an audio information editor 412.
  • the visual representation of the audio information 404 is received by a visual representation editor 408.
  • the visual representation editor 408 provides an interactive interface with a user and/or automated process and the like to receive changes or modifications to the received visual representation of the audio information 404. Once changes are received, the visual representation editor 408 provides the changes/modifications to the audio information editor 412.
  • the audio information editor 412 alters the changes from the realm of the visual representation to changes in the realm of the audio information. This is facilitated by the synchronization of the audio information with the visual representation of the audio information.
  • a musical note on a score (visual representation) is changed to a lower octave.
  • the audio information editor 412 converts this change in notation to a change in frequency (audio information). It 412 then correlates the position of the notation in the musical score to the timing location in the audio information.
  • the audio information editor 412 then applies the change to the audio information creating modified audio information 414 that now correlates (aligns with) to the modified visual representation of the audio information.
  • FIG. 5 is a flow diagram of a method 500 of transforming changes from a visual representation of audio information into changes applied to audio information in accordance with an embodiment of the present principles.
  • Audio information that has been synchronized with a visual representation of the audio information is received 502. As discussed above, synchronization facilitates in applying modifications between the two forms of information.
  • At least one modification of the visual representation of the audio information is also received 504. The received modification can be via a user and/or some type of automated process and the like.
  • the at least one visual representation modification is transformed into at least one audio information modification 506. Means to transform the modification from the visual representation to the audio information realms have been discussed above as well.
  • the audio information modification is then applied to the audio information to create modified audio information 508. After the modification is applied to the audio information, it will then be synchronized with the modified visual representation of the audio information.
  • modified audio information comprising a modified audio file is generated by the process.

Landscapes

  • Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Acoustics & Sound (AREA)
  • Multimedia (AREA)
  • Electrophonic Musical Instruments (AREA)
  • Management Or Editing Of Information On Record Carriers (AREA)
EP15305610.6A 2015-04-22 2015-04-22 Vorrichtung und verfahren zur bearbeitung von audioinformationen, die mit einer visuellen darstellung synchronisiert sind Withdrawn EP3086315A1 (de)

Priority Applications (1)

Application Number Priority Date Filing Date Title
EP15305610.6A EP3086315A1 (de) 2015-04-22 2015-04-22 Vorrichtung und verfahren zur bearbeitung von audioinformationen, die mit einer visuellen darstellung synchronisiert sind

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
EP15305610.6A EP3086315A1 (de) 2015-04-22 2015-04-22 Vorrichtung und verfahren zur bearbeitung von audioinformationen, die mit einer visuellen darstellung synchronisiert sind

Publications (1)

Publication Number Publication Date
EP3086315A1 true EP3086315A1 (de) 2016-10-26

Family

ID=53052777

Family Applications (1)

Application Number Title Priority Date Filing Date
EP15305610.6A Withdrawn EP3086315A1 (de) 2015-04-22 2015-04-22 Vorrichtung und verfahren zur bearbeitung von audioinformationen, die mit einer visuellen darstellung synchronisiert sind

Country Status (1)

Country Link
EP (1) EP3086315A1 (de)

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6355869B1 (en) * 1999-08-19 2002-03-12 Duane Mitton Method and system for creating musical scores from musical recordings
WO2006066075A1 (en) * 2004-12-15 2006-06-22 Museami, Inc System and method for music score capture and synthesized audio performance with synchronized presentation

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6355869B1 (en) * 1999-08-19 2002-03-12 Duane Mitton Method and system for creating musical scores from musical recordings
WO2006066075A1 (en) * 2004-12-15 2006-06-22 Museami, Inc System and method for music score capture and synthesized audio performance with synchronized presentation

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
"Pro Tools 11 tutorial: Exploring the Score Editor | lynda.com", 18 October 2013 (2013-10-18), XP054976101, Retrieved from the Internet <URL:https://www.youtube.com/watch?v=CY1hFUmZ59s> [retrieved on 20150929] *

Similar Documents

Publication Publication Date Title
US7626112B2 (en) Music editing apparatus and method and program
US7714222B2 (en) Collaborative music creation
US7825321B2 (en) Methods and apparatus for use in sound modification comparing time alignment data from sampled audio signals
US20080196575A1 (en) Process for creating and viewing digital sheet music on a media device
JP5143569B2 (ja) 音響的特徴の同期化された修正のための方法及び装置
US20150310850A1 (en) System and method for singing synthesis
Rosenzweig et al. Erkomaishvili Dataset: A Curated Corpus of Traditional Georgian Vocal Music for Computational Musicology.
US9601029B2 (en) Method of presenting a piece of music to a user of an electronic device
US20060112812A1 (en) Method and apparatus for adapting original musical tracks for karaoke use
US11087727B2 (en) Auto-generated accompaniment from singing a melody
GB2422755A (en) Audio signal processing
Arzt et al. Artificial intelligence in the concertgebouw
JP2003208170A (ja) 演奏制御装置及び演奏制御用プログラム、並びに記録媒体
Scherbaum et al. Tuning systems of traditional georgian singing determined from a new corpus of field recordings
WO2018207936A1 (ja) 譜面自動検出方法および装置
WO2011125204A1 (ja) 情報処理装置及び方法、並びにコンピュータプログラム
US7470856B2 (en) Method and apparatus for reproducing MIDI music based on synchronization information
Schwarz et al. Methods and datasets for DJ-mix reverse engineering
Nakano et al. VocaRefiner: An interactive singing recording system with integration of multiple singing recordings
EP3086315A1 (de) Vorrichtung und verfahren zur bearbeitung von audioinformationen, die mit einer visuellen darstellung synchronisiert sind
JP2013024967A (ja) 表示装置、表示装置の制御方法およびプログラム
JP5338312B2 (ja) 自動演奏同期装置、自動演奏鍵盤楽器およびプログラム
JP2005202425A (ja) 楽曲の伴奏音と歌詞字幕映像を同期出力する装置
Rosenzweig Interactive Signal Processing Tools for Analyzing Multitrack Singing Voice Recordings
WO2011060504A1 (en) Music tuition or practice method and apparatus

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION IS DEEMED TO BE WITHDRAWN

18D Application deemed to be withdrawn

Effective date: 20170427