Classifications

• • G—PHYSICS
  • G10—MUSICAL INSTRUMENTS; ACOUSTICS
  • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
  • G10H7/00—Instruments in which the tones are synthesised from a data store, e.g. computer organs
• • G—PHYSICS
  • G10—MUSICAL INSTRUMENTS; ACOUSTICS
  • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
  • G10H1/00—Details of electrophonic musical instruments
  • G10H1/0008—Associated control or indicating means
• • G—PHYSICS
  • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
  • G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
  • G09B15/00—Teaching music
  • G09B15/02—Boards or like means for providing an indication of notes
  • G09B15/023—Electrically operated
• • G—PHYSICS
  • G10—MUSICAL INSTRUMENTS; ACOUSTICS
  • G10K—SOUND-PRODUCING DEVICES; METHODS OR DEVICES FOR PROTECTING AGAINST, OR FOR DAMPING, NOISE OR OTHER ACOUSTIC WAVES IN GENERAL; ACOUSTICS NOT OTHERWISE PROVIDED FOR
  • G10K15/00—Acoustics not otherwise provided for
• • G—PHYSICS
  • G11—INFORMATION STORAGE
  • G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
  • G11B20/00—Signal processing not specific to the method of recording or reproducing; Circuits therefor
  • G11B20/10—Digital recording or reproducing
• • G—PHYSICS
  • G10—MUSICAL INSTRUMENTS; ACOUSTICS
  • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
  • G10H2220/00—Input/output interfacing specifically adapted for electrophonic musical tools or instruments
  • G10H2220/005—Non-interactive screen display of musical or status data
  • G10H2220/015—Musical staff, tablature or score displays, e.g. for score reading during a performance.
• • G—PHYSICS
  • G10—MUSICAL INSTRUMENTS; ACOUSTICS
  • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
  • G10H2220/00—Input/output interfacing specifically adapted for electrophonic musical tools or instruments
  • G10H2220/155—User input interfaces for electrophonic musical instruments
  • G10H2220/441—Image sensing, i.e. capturing images or optical patterns for musical purposes or musical control purposes
  • G10H2220/451—Scanner input, e.g. scanning a paper document such as a musical score for automated conversion into a musical file format

Definitions

• a variety of devices can help performers improve their musical instrument playback performance.
• electronic metronomes help a performer maintain a steady count.
• Systems have been developed for computer display of a music score (sheet music), making a wide variety of music conveniently available for practice by the performer. See, for example, U.S. Patent Application 2004/0040433 to M. Errico.
• Other systems assist in optical recognition of music scores for storage as digital data and subsequent computer display. See, for example, U.S. Patent No. 5,825,905 to T. Kikuchi.
• a performer could utilize a synthesized performance (audio rendition) of a music score and could listen at any time to difficult passages in a music score (or in fact, listen to the entire score) played correctly, With the correct pitches (in tune) and the correct rhythms. The performer could then practice by duplicating or reproducing the correct ways of playing. It also would be helpful if a performer could view a music score on a dynamic display that is synchronized with the synthesized audio rendition, and practice playing a musical instrument or singing according to the displayed musical score. It would also be helpful if a performer could record his or her performance and then play back the performance at any time, for assessment of the performance and for comparison with the correct (synthesized) rendition.
• the present invention provides capture and subsequent interpretation of a passage of music score (or an entire piece of music or a song) for solo instrument, multiple instruments, voice or multiple voices, or any combination thereof, processing of the data so as to produce a synthesized audio presentation and synchronized concomitant display of a visual presentation of the music score corresponding to the audio presentation, and supports recording of a performer's musical performance of that music score for later playback of the performer's musical performance.
• the means for providing these features can comprise application software on a host digital computer. Alternatively, these features can be provided by a handheld device that is self-contained.
• Both embodiments, the host computer and handheld device include means for receiving a digital representation of the music score, a display that shows a visual presentation of the music score, and a facility for a synchronized synthesized audio rendition of the score.
• the digital representation of the music score can be received from a digital image capture device or over a network connection from a data source.
• the embodiments also can provide for recording of a user performance and playback of the user's performance.
• music score data can be received from an external source such that the computing device can produce an audio presentation of the music score data and can produce a synchronized visual presentation of music notes corresponding to the audio presentation.
• the handheld device can be adapted to receive external memory cards that can store entire musical works, volumes of works, method books, and the like in digital data format.
• Internet and/or telecom interfaces can allow for downloads in digital data format.
• the device can download music scores in digital data format. Such downloads can be stored in external memory cards or similar media.
• Image capture of input music score data can be supported through digital photography or optical scanning of music scores.
• the application software implementation can include performance evaluation features and playback assistance features.
• a "music minus one" feature can be provided to enable the user to digitally capture a music score that is for more than one instrument or for more than one vocal line (or any combination thereof); opt to have the synthesized audio presentation leave out a specified instrumental or vocal line of the music score ("minus one") so that the user may play and/or sing along with the synthesized audio presentation.
• the synchronized visual presentation of the music score can include any or all of the instrumental and/or vocal parts of the original data.
• the user may opt to leave out more than one part of the synthesized audio presentation, such as additional instruments or vocal lines, resulting in "music minus two" or “music minus three", and so forth, depending on the number of elements left out.
• Figure 1 is an illustration of a hand-held device constructed in accordance with an embodiment of the invention.
• Figure 2 is an illustration of the processing that is performed by a device constructed in accordance with an "eMusel" embodiment of the invention.
• Figure 3 is an illustration of the processing that is performed by a device constructed in accordance with an "eMuse2" embodiment of the invention.
• Figure 4 is an illustration of the processing that is performed by a device constructed in accordance with an "eMuseX" embodiment of the invention.
• Figure 5 is an illustration of a display screen produced in accordance with the invention for playback of a synthesized audio rendition in accordance with the invention.
• Figure 6 is an illustration of the processing that is performed by a device constructed in accordance with the invention to provide a "music dictation" feature.
• FIG. 7 is a block diagram of an embodiment of a music score capture and interpretation device with audio-visual presentation, constructed in accordance with the invention.
• the features of the invention are implemented in software, comprising an application that can be installed on a digital computer.
• the software implementation preferably provides input and output interfaces for the performer. That is, the host computer in which the software is installed typically includes a display for producing a visual presentation of a music score that the performer can read, to sing along or play the performer's musical instrument.
• the computer also typically includes an input interface, such as a microphone, for recording the performer's session, and includes an output interface, such as speakers, to enable the performer to listen to the recorded performance.
• the computer implementation can include image capture, wherein a music score comprising notes on a staff can be digitized via an optical input means and then entered into the computer.
• the digitized music score can be interpreted via OCR techniques, with the resulting interpreted data being processed so as to produce a synthesized audio rendition of the music score, including when appropriate a synthesized vocal rendition matching words with appropriate pitch, such that the audio rendition is synchronized with a visual presentation of the score.
• the computer software implementation is referred to as a "Level X” implementation or is referred to as the "eMuse X” product (the name “eMuse” referring to a product embodiment from Princeton Music Labs LLC of Princeton, New Jersey, USA, the assignee of all rights in the invention).
• the features of the invention are embodied in a handheld device that can include a display, an input interface, audio and visual output interfaces, and OCR image interpretation interfaces.
• the handheld device implementation includes a variety of convenient user control knobs and mechanisms for convenient navigation of the device functions.
• the display supports a visual presentation of menu options for selection of functions by the user.
• a computing device interprets and processes music score data by receiving the music score data from an external source and subsequently producing a synthesized audio rendition of the music score data and a synchronized visual presentation of music score.
• the external source can consist of a network data source that provides the music score data to the computing device over a network connection.
• the network connection can consist of communication between the computing device and the network over a wireless connection.
• the music score data can be read from a recorded medium by accepting the recorded medium into a reader of the computing device that then obtains the music score data from the recorded medium.
• the recorded medium contains sufficient data for synthesized audio rendition in accordance with a musical instrument digital interface (MIDI) specification for synthesized music production. That is, the computing device can receive data that specifies a music score and can generate or synthesize corresponding musical tones in a selected tempo, timbre, clef, key signature, time signature, and the like.
• the recorded medium can comprise a flash memory device.
• the computing device can be provided with ability for recording a user performance of a music score and providing playback of the recorded user performance. The user performance playback can occur independently of the synthesized music score rendition, or can occur simultaneously.
• the user performance playback can be provided along with a visual representation of the musical notes corresponding to the recorded user performance.
• a "music dictation" feature is provided.
• the music score data used by the device to generate both the synthesized audio rendition and the synchronized visual presentation of the music score can be obtained by the device optically capturing a digital image of a music score, then interpreting and processing the digital information to produce a collection, of data representing appropriate music notes, thus generating data that corresponds to the musical score.
• musical contextual information can be provided that determines characteristics of the synthesized audio rendition of the music score data, all of which may be adjusted by the user.
• Such musical contextual information can include multiple key signatures, time signatures timbre, tempo and expressive terms such as legato, crescendo, ritard, etc.
• producing an audio playback of the music score data and a visual presentation of musical notes is effected through communication with a network data source.
• the network data source provides the music score data to the computing device.
• the network data source can provide to the computing device the musical contextual information that determines musical characteristics of the synthesized audio rendition of the music score data.
• the network data source can provide the musical context information over a wireless connection.
• producing a synthesized audio rendition of the music score data and a synchronized visual presentation of music score is effected by inserting a recorded medium into a reader of the computing device. If desired, the computing device obtains the music score data from the recorded medium, and the recorded medium can also provide the musical contextual information to the computing device for determining musical characteristics of the synthesized audio rendition of the music score data.
• One optional feature is to provide recording of the user's instrumental and/or vocal performance of the music score. Another alternative is to produce a synthesized audio rendition in accordance with a musical instrument digital interface (MIDI) specification. In addition, producing the visual presentation can consist of displaying the music score synchronized with the corresponding synthesized audio rendition. Another option is to provide simultaneous synchronized playback, playback of both the visual presentation and audio rendition of the music score data and both the audio component of the recorded user performance and a synchronized corresponding visual display of the music score generated by the user performance.
• MIDI musical instrument digital interface
• a computing device can optically digitally capture a music score and interpret the digital image, generating music score data for the computing device that corresponds to the digitally captured music score, and produce a synthesized audio rendition of the music score data and a synchronized visual presentation of music score.
• the computing device can receive musical contextual information that is used by the computing device to determine musical characteristics of the synthesized audio rendition of the music score data.
• the musical contextual information can include multiple key signatures, time signatures timbre, tempo and expressive terms such as legato, crescendo, ritard, etc. that can be selected by the user to determine the musical characteristics of the synthesized audio rendition of the music score data.
• the computing device identifies the musical contextual information from the optically digitally captured music score, and optionally can obtain the musical contextual information from a network data source. If desired, the network data source provides the musical contextual information over a wireless connection with the computing device.
• the computing device can be provided with its own loudspeakers for audio playback of synthesized renditions and/or performances recorded by the user. Additionally, the device can include an output jack for connection to headphones or external loudspeakers or the like, and can also be provided with wireless transmission capability that allows the device to transmit an audio performance to a wireless sound playback system (such as a home stereo system that has been enabled with wireless components).
• the device has sufficient computing memory to enable it to store musical passages of predetermined length.
• Figure 1 is an illustration of a handheld device that provides the functionality and features described herein.
• Figure 1 shows the eMuse product 100 as a hand-held, battery powered
• the "point and shoot” consumer electronic device that captures an image of a selected musical passage-either a few measures or even an entire page-from sheet music, formats and encodes the passage digitally, then plays the selected passage, all virtually instantaneously in real time.
• the playback sound is instrument-specific, playing in a timbre (i.e., the sound of a piano, violin, flute, etc.) selected by the user. Multiple timbres can be played simultaneously.
• the device 100 is designed to recognize musical variables, such as key signature, meter, volume, and tempo, and account for musical prose terms, such as forte, piano, cress., legato, and the like.
• An LCD screen 102 displays the musical passage 104 as the device is playing it back, with each played note highlighted in a unique color so as to distinguish it from the rest of the music score.
• the Figure 1 device 100 incorporates a digital "music card” feature, which supports insertion of a data card (not illustrated) that is encoded with one or more musical pieces, prerecorded or downloaded from the Company's website, and allows for quick reference to specific measures.
• the data card can comprise recordable media such as typically used by cellular telephones and digital cameras.
• the data card is received into a card slot 1O6 of the devi ⁇ e 100.
• the device 100 also works with digital data representing encoded musical pieces.
• the encoded musical pieces also can be downloaded from the Company's website. Downloaded files can also be saved within memory of the device.
• the encoded musical pieces provide an enriched listening experience, allowing for quick reference to specific measures of the music score.
• a record/playback feature of the device 100 allows the user to immediately evaluate a recorded performance with reference to the music score. That is, the device 100 can record a user's performance of the musical piece and play back the user's performance, along with (or simultaneous with) playback of the received musical piece. The user performance playback can be presented with a corresponding visual presentation, providing the "music dictation" feature described further in this document. Both a metronome and a musical tone tuner capability are also incorporated into the device, and the device can be adjusted for "music minus one.” In a multi-staff or multi-part piece of music, the "music minus one" feature allows the user to determine which part(s) of the piece will be played back by the MIDI interface. This allows the user to play/sing a specific part along with the device.
• Figure 1 shows the device 100 as a hand-held device with a flip-top 108 having the LCD display screen 102 incorporated in the underside of the lid or top 108.
• the device also incorporates a speaker 110 for music playback (a stereo pair are illustrated in Figure 1) and a microphone 112 for recording a user performance.
• Image capture capability can be implemented as shown in Figure 1 , wherein the flip-top 108 includes a camera system comprising an optical lens mounted in the outside surface of the flip-top 108 and triggered by a lens shutter button 114 for purposes of image capture.
• Various user interface controls are provided, shown as thumbwheels for adjusting volume 116, playback tempo 118, menu navigation 120, and position 122.
• the device 100 can be provided with a USB port 124, for more easily connecting with a network or other devices.
• An on/off switch 125 turns the device 100 on and off.
• Control buttons are also provided for controlling, as illustrated in Figure 1 , functions involving renditions of the music score (synthesized performance) and playback of the user's performance.
• Figure 1 shows exemplary control buttons for the functions of play/pause 126, stop 128, next/reset 130, and record 132.
• Figure 1 also shows a metronome readout display 134 that shows a timing indicator and is correlated with the tempo wheel 118. Selection of the metronome function, and the other features of the device 100, is accomplished with menu navigation of the display 102 in conjunction with operation of the menu wheel 120 and/or the position wheel 122.
• a keyboard or keypad 135 can be used for input via dedicated function keys of the keypad or alphanumeric input from the keypad.
• On the display screen 102 graphical images of music notes from the music score 104 are provided, moving across the screen, with individual notes appearing on-screen or being highlighted in the display as the notes are played.
• the display preferably provides a music staff image that is extended during playback so a predetermined number of notes or measures of music are displayed as the music score is played.
• a hand-held, portable "point and shoot” device that "reads” a musical score and plays it back.
• the playback sound is instrument- specific; the device has adjustable timbre (i.e., the sound of a piano, violin, flute, etc.). It is able to recognize and incorporate additional musical variables, such as key signature, meter, volume, and tempo.
• a record/playback feature allows the user immediately to evaluate a performance with reference to the musical score. Both a metronome and a tuner are also included.
• a "music-card” feature supports insertion of a card encoded with an entire musical piece allows for quick reference to specific measures.
• the software embedded in the eMuse device differs from other currently available products in that eMuse receives digitized image data for a music score and interprets that data into corresponding musical notes, and does so with an accuracy rate approaching 100%.
• Other conventional software products are designed for use by music editors and/or composers using desktop PCs, and expect the user to manually correct scanning and OCR errors in identification of musical notes.
• eMuse as a hand-held, untethered, portable device is designed for use by all music performers, and in such use it is anticipated that there will be no errors to correct.
• a digital camera system 114 captures an image of a passage (a single note, several measures, or even an entire page) within a musical score.
• the digital camera can be built into the device 100 and can comprise a lens and image transducer combination that will be familiar to those skilled in the art.
• the LCD display 102 allows the user to determine exactly which measures are captured.
• the device can read a single stave musical line, duets, trios, quartets, or even a full conductor's score.
• the device 100 offers multiple simultaneous timbres.
• the OCR module receives the "photograph" of the musical excerpt, comprising digitzed image data.
• Important additional musical contextual information such as key signature and meter, is also sent to the OCR module, via a music score digital image or via a "cheat sheet” (downloaded from the PML website, then transmitted wirelessly or via the USB port to the device-see below) that lists all available key signatures and time signatures.
• the "cheat sheet” also includes a section from which the user can select the desired timbre(s), or the user can manually specify (input) the desired timbre(s).
• the OCR module sends the sound information to the MIDI module that produces synthesized sound.
• This offers adjustable timbre; the user specifies the type of instrument (piano, violin, flute, etc.) for the particular musical passage or piece.
• the module also offers adjustable tempo so that the user can hear the passage slower (or faster) than the metronomic (if any) indicated in the score without any alteration of pitch.
• the device plays back through its own small loudspeaker, and also has a headphone jack 134 and wireless capability for headphones and/or external speakers.
• the LCD monitor display 102 helps the user make sure that the measures being captured (photographed) are the measures that are intended to be heard.
• the LCD monitor display complete with a cursor 136, displays the music score 104 as the passage is played back, either from a passage that was photographed by the user or from a music-card with stored data.
• the cursor indicates the exact musical position in the score of the current note(s) being played as the passage is played in real time, regardless of the specified tempo.
• the display 102 can instead indicate the note being played by highlighting the note (e.g., making it brighter) or by giving it a different display color from the other notes as it is played.
• the LCD display shows the names of the notes (both in English and in solfege) 138, particularly for a single-line passage. If the passage is comprised of multiple simultaneous musical lines, the user can specify the line for which the names of notes are displayed.
• the display 102 also shows an indicator of the music score passage selected for play. The indicator is referred to as the passage marker 139.
• the passage marker 139 is shown as numeral "1" enclosed in a circle. The numerical "1" indicates the first measure of the music score is currently being displayed, and the circle indicates that playback was initiated at the first measure. If playback continues to the next measure, a "2" would be displayed in place of the "1", but the "2" would not be circled. Recording Sensor
• the microphone 112 is provided so that the user can record him/herself playing (and/or singing) the musical passage in question and immediately play back the recording to compare the user's performance with that of the device 100 (that is, of a previously recorded or synthesized rendition). This feature is helpful for students to make adjustments in notes, tuning, rhythm, and dynamics. As noted above, a user performance can be recorded via the microphone to provide the "music dictation" feature.
• the device 100 is preferably provided in wireless versions to permit wireless communications with networks and other wireless-enabled device, and to permit downloads of encoded music files with contextual information.
• the features despribed herein can be provided by eMuse software installed to a wireless platform, such as a PDA or smartphone, for portable music interaction.
• wireless eMuse devices can use computing and memory (and playback audio) of the home PC and/or stereo system. Power
• Power is via rechargeable batteries; DC input (9/12 volts) is also available through an external connection 140.
• Music-card feature A card (information storage device) digitally encoded with an entire musical piece (or a simple method book) can be inserted into the OCR module at the card slot 106. This allows the user quick reference (auditory and visual-see Visual Display above) to specific measures.
• Music-cards These will be available for retail purchase and can comprise conventional media, such as Secure Digital (SD) cards, or CompactFlash cards, or XD cards, or “Memory Stick” devices such as available from Sony Corporation.
• PML will offer a substantial library of music (computer file representations of scores, both visual and aural), ranging from method books to more complex standard Classical repertory to jazz and pop "hits", available for password encrypted downloading for eMuse users. These files will be downloadable to the user's home PC, with the user then either burning a "music-card” or transmitting the file to the wireless eMuse.
• the eMuse devices can communicate over telecom networks to download encoded music files from music retailers (such as Tower, HMV, etc.) and ring tone providers. "Music Dictation"
• a user's performance can be recorded by the device and the user's performance can be subjected to a music note interpretation processing to generate data from which is produced a display of the music notes corresponding to the user's recorded performance.
• the device can take "musical dictation" and can convert a live audio performance by the user into a visual display of the music score corresponding to the performance.
• the music interpretation features of the device can process both music score data received by optical or electronic network communication, and can process music score data produced by a user's live performance, captured by a microphone.
• eMusel is a flip-top handheld unit with intuitive controls designed for ease of use. It incoporates an integrated camera and
• the proprietary eMuse software interprets the captured image, including the musical contextual information, and produces a synthesized audio rendition of the score. As the music plays, a synchronized visual representation of the score appears on the display screen, and the notes being played at any time are highlighted in color. Controls on the device enable the user to adjust tempo, key signature, etc., or to delete one or more parts (for "music minus one" use).
• the device has the capability to connect to external sound systems for improved audio quality.
• eMusel also incorporates recording and playback capability (including the music dictation feature) so that the users can record their performance of the work and compare this performance with the original.
• eMuse2 incorporates all of the features and functionality of eMusel, with the addition of an integrated music card reader.
• Music cards will be available in two forms, pre-encoded with music score data, including musical contextual information, or in blank form. Using a blank music card, the user can download music score data from Internet web sites (either PML's own web site or those of third parties that have licensed the eMuse software) and "burn" the data onto the music card.
• An optional external music card reader and appropriate interface can be provided with other embodiments (such as eMusel and eMuseX) for use of music cards.
• eMuseX which is is designed for home or studio use, is a software product that enables a personal computer to perform the same functions as the handheld eMuse device 100.
• a scanner or high-resolution webcam attached to the computer captures a digitial image of a music score, and this image is projected onto the user's computer display screen. The user then selects a portion of the score and the synthesized audio rendition of the selected portion is played through speakers attached to the computer.
• the user will also be able to download music score data to the hard drive of the computer, or purchase pre-encoded music cards, which are read by a media reader connected to the computer's USB port.
• the software described herein can be used in a variety of platforms.
• aspects of eMuse could be embedded in a high-end cell phone in which the cell-phone camera photographs a specific passage in a musical score.
• the captured image is then compressed and sent to a remote server, which performs OCR operations on the image data to interpret the image into corresponding musical note information.
• the server then sends back both a midi file and a graphic file, enabling this version of eMuse to play the music that was photographed and display the notes on the LCD as they are played.
• eMuse software can be installed in a user's platform of choice-such as a camera-equipped telephone or similar PDA, in addition to the devices preloaded with eMuse software.
• the software that interprets the captured music score image into a corresponding set of notes utilizes machine learning techniques and will be trained to achieve an accuracy rate approaching 100%, while interpreting substantially in real time.
• the conventionally available musical notation OCR software for converting musical note images into corresponding notes are generally intended for offline editing, at a time reomved from the actual image capture, and cannot achieve the near-100% accuracies.
• Currently available music score conversion software can be procured from companies such as
• eMuse encoded contextual files can be received over a telecommunications link, either wired or wireless, such as WiFi, Bluetooth and/or other telecom connections.
• a telecommunications link either wired or wireless, such as WiFi, Bluetooth and/or other telecom connections.
• FIG. 2 is a process flow diagram that illustrates operation of the eMusel embodiment.
• a digital representation of a music score is provided to the eMusel device.
• the digital representation can be received by a visual presentation 202 to the eMusel device, such as a printed page, which is digitally captured using a digital image capture device 204, such as a digital camera that operates with the eMusel device.
• the digital data derived from optical input 206 is then provided to a note data interpretation process 2 €8.
• the digital representation of the music score can be provided electronically 210, such as by wireless transmission of digital data corresponding to the music score or wired transmission of the data over a network, or input through a storage media such as a memory card or other media.
• the electronically received version of the music score 210 is then provided to the note data interpretation process 208.
• the note data interpretation process 208 receives the digital data corresponding to the music score and processes it to produce a set of musical notes and concomitant information sufficient to specify the musical score and enable its reproduction by suitable hardware.
• the process 208 comprises a processor trained with machine learning techniques to recognize the music score digital data 206, 210 and produce appropriate transformed data.
• the process 208 can be trained, for example, using neural network software engineering techniques to increase the accuracy of the interpretation process up to substantially 100% accuracy.
• the incoming music score data must be produced for audio and visual presentation to the user in real time, and therefore interpretation of the incoming music score data must be in real time and must approach 100% accuracy of interpretation (transformation).
• the process 208 utilizes optical character recognition (OCR) techniques, but is adapted for music note recognition and interpretation of digital data (electronic or optical scan derived) to an appropriate representation.
• OCR optical character recognition
• the interpretation process output 212 comprises a visual presentation of the music score, which is provided to a display screen 214 of the device, and also a synthesized audio rendition of the music score, which is provided to appropriate device systems and hardware 216 for audio presentation through loudspeakers of the device, or the like.
• eMuse2 Figure 3 is a process flow diagram that illustrates operation of the eMuse2 embodiment. The operations depicted in Figure 2 are similar to the operations depicted in Figure 3, and like operations are given like reference numerals. Thus, in Figure 3, a digital representation of a music score is provided to the eMusel device either by visual presentation 202 and image capture 204 followed by presentation 206 to the note data interpretation process 208, or electronic receipt and presentation 210 to the process 208.
• the eMuse2 embodiment adds another alternative means of music score presentation, that of using a music card interface comprising a memory store 304 that mates with a reader or slot 306.
• the memory store 304 can comprise, for example, digital media such as Secure Digital (SD) cards, CompactFlash cards, MemoryStick media, and the like, while the reader 306 will comprise a suitable card reader to accept the data stored on the card 304.
• the music score data is then presented to the display 214 and audio reproduction 216 devices for playback.
• FIG. 4 is a process flow diagram that illustrates operation of the eMuseX embodiment.
• the eMuseX embodiment comprises a software package that can be installed in a host computing device, such as a desktop or laptop computer, a PDA device, or a "smart phone".
• a host computer with the eMuseX software installed is activated to begin executing the eMuseX software.
• the host computer is also installed with software that implements the note data interpretation process 208 described previously and also includes a suitable image capture combination or a suitable electronic data receiving combination.
• the image capture combination can comprise a digital camera with an interface to the host computer, and the electronic data receiving combination can comprise wireless communication interface, a hard wired network communication interface, or a recorded data interface such as a memory card reader or a magnetic disk or optical disc (e.g., CD or DVD) drive.
• the music score digital representation is provided to the note data interpretation process of the host computer, either by operation of the image capture combination or by operation of the electronic data receiving combination.
• the interpreted musical score data is provided to the host computer for processing and presentation 406, such that a display presentation 408 and an audio reproduction presentation 410 are generated by systems of the host computer for presentation to the user.
• the audio presentation 41O and display presentation 408 will generally correspond to the respective audio presentation 216 and display presentation 214 of the dedicated devices ( Figure 2 and Figure 3), but might differ in details because of the different resources available to the host computer as compared to the dedicated devices.
• Figure 5 illustrates a display screen that is produced by a device that incorporates the processing described herein, such as a handheld device or a host computer in which the application software is installed, to provide playback of a synthesized audio rendition.
• the device first receives music score data, such as by receiving a programmed music card into a music card reader of the device, or by receiving the information from a network interface, through either wireless or cable connection.
• the music score data can include, for example, the music score, and also musical contextual information that determines musical characteristics of the synthesized audio rendition of the music score.
• the operating software of the device reads the music score data
• the device displays a menu screen to provide the user with information about the music score and to request instruction from the user as to playback.
• Figure 5 shows a music score menu screen 502 as displayed on the display 504 of a handheld device ( Figure 1) or in a program window on the display of a host computer with the eMuse application software installed ( Figure 4).
• the menu screen shows the contents of the music score, in the Figure 5 example, the music score is shown as "Bach-Goldberg Variations”.
• the menu 502 also shows the number of sections (e.g. tracks) in the music score, indicated in the Figure 5 example as being thirty-two.
• the menu also shows the timbre, in Figure 5 this is shown as being "piano".
• the menu asks the user for the section (or track) number at which playback will start.
• a default value (such as section 1) may appear in a display box 506 on the initial menu screen 502.
• the user can indicate that playback should start at the indicated position, Section 1 , such as by selecting a "yes” or “play” command on a handheld device or by selecting “enter” or “yes” on the menu screen.
• the device will begin playback of the synthesized audio rendition and will display the music score (music notes) on the display 504 in synchronization with the audio rendition.
• the user can enter a different section number in the input box 506 and then select "yes” or “play” or “enter”, and then the device will start to play the music score at the indicated section.
• FIG. 6 is a process flow diagram that illustrates operation to provide a "music dictation" feature.
• This feature can be provided with handheld embodiments and host computer embodiments of the invention.
• a microphone records a user's performance of live music and a corresponding set of live note data is produced.
• the microphone output is recorded in digital format to produce the live note data, or is processed after recording to produce the live note data, comprising digital data that corresponds to the notes in the audio track of the user's performance. That is, the live note data is analogous to the music score data described elsewhere in this document.
• the live note data derived from the microphone 602 is then provided to a note data interpretation process 604.
• the interpretation process of the device processes the live note data to produce a visual display of the corresponding music score 606, and the device also produces an audio playback of the user's performance 608.
• the user's audio performance and the visual music notes that correspond to the user's audio performance are provided in a synchronized playback.
• the playback of the user's performance can be coordinated and compared with a synthesized audio/visual rendition of a music score of the recorded work. Additional Functionality • Multiple simultaneous timbres (instruments) can be read and played, leading to possible usage as an enabler of "music minus one"
• a tuner is incorporated into the device
• the device is able to transpose musical passages into any key.
• the device displays the names of the notes of a musical passage (both English and solfege) on the LCD as that passage is being played •
• the OCR module retains the memory of a musical passage until the next passage is "photographed” so that a particular passage can be heard repeatedly without “photographing” it again.
• Shutter botton 114 for initiating image capture and "photographing" a musical passage
• Play button 126 - for playing the "photographed” measures, a selection from a "music card,” and/or playback of a user's recorded performance.
• Metronome 142 - could use numeric keypad to select tempo of a playback passage, as well as the conventional use of a metronome.
• Tuner 144- could use numeric keypad to specify frequency for tuning - e.g., A440 or A442. If desired, one or more of these buttons and dials can be combined in a simple +/- toggle. Construction
• FIG. 7 is a block diagram of an embodiment of a music score capture and interpretation device 700 with audio-visual presentation, constructed in accordance with the invention.
• a control processor 702 such as a computer central processor unit (CPU), controls operations within the device 700.
• the CPU 702 communicates with other components, including data and commands, over a device bus or systems communication bus 704. Commands and data are stored in memory 706, which can include program memory or ROM 708 and data memory or RAM 710.
• the memory 706 can be a mixture of volatile and non-volatile memory.
• the CPU executes commands and program instructions stored in program memory 708 to provide the features described herein. Operational data, such as music score data and the like, are stored in the data memory 710.
• the device 700 includes a network interface 716 for communication with other computers over a network.
• the network interface can provide for communication via a wireless link (such as WiFi or Bluetooth connections) or a wired (cable) connection (such as network cable or USB connections).
• the device 700 also includes a keyboard 718, for receiving user inputs and commands, and includes a display 720, for presentation of data to the user.
• the display can comprise a display screen of a handheld device constructed in accordance with the invention, or can comprise a display of a host computer in which an application software embodiment of the invention is installed.
• the device also includes audio output 722, such as loudspeakers that -can produce the audio rendition of a music score.
• the audio output facility 722 can also include headphone connections for private listening or other line out connections.
• the device 700 also includes an image capture facility 724, such as an integrated digital camera system having a lens and shutter control button.
• the image capture facility can be an externally connected system, such as where a digital camera might be connected to a host computer via a network connection such as a USB port or wireless Bluetooth link.
• the device also includes an audio capture facility 726, such as a microphone connected to the device. Examples Of Usage Following are scenarios of device usage, which illustrate how embodiments of the invention can be put to use.
• Alex (age 13) has played clarinet in the Junior High School band a couple of years. His band director is rehearsing the group in a Sousa March for the Memorial Day Parade, and since the clarinet section frequently rushes the tempo, he bought 4 music cards of this March for those students. When it was time to practice after dinner, Alex realized that he'd forgotten his music card at school, so he went to the PML site and downloaded another one onto his home PC. He transmitted this one (wireless) to his eMuse, plugged in headphones, "photographed" mm.14-16 at the end of the second phrase (one of the hard parts for him), and selected "clarinet" timbre.
• the user inserts a music card into the product's music card slot, selects a music piece, selects timbres, selects a tempo, selects a location in the piece (the "passage marker”), and pushes the play button.
• the stop button stops the music.
• the play button starts play of the passage, at the beginning or taking it up again where it left off in a previous session (the "play location marker"), unless the user presses the back arrow button, in which case the play location marker moves back to the passage marker.
• the menu wheel 120 changes roles and always controls either tempo or the location of the passage marker. If it controls tempo, then volume and the passage marker have separate wheels; if it controls the passage marker, then volume and tempo have separate wheels. There are two reasons for having three separate wheels:
• the volume control may be an analog potentiometer. Details
• the unit powers up when the user depresses the on/off button. It may also power up when a flash drive is inserted. It shuts off when the flash drive is removed or after a selected period of inactivity, or when the user depresses the on/off button again.
• the unit Upon application of power, the unit enters the menu sequence. This is a series of choices that the user must make before play can begin. It appears as a sequence of lists (usually just two). The user can scroll through each list with the menu wheel. The next button registers the highlighted choice and moves to the . next step.
• sequence of lists is dynamic, but may contain the following:
• These instructions may take the form of labels for the wheel and button.
• the unit advances to play mode.
• the screen When the unit is in play mode, the screen displays music notation (with one or more location markers) in the middle, and the current metronome marking, probably near its adjustment wheel.
• Music notation is presented on the display screen in accordance with the physical size of the screen; generally sufficient size is available to show a single staff or system of staves.
• the product keeps track of two places in the music: the point selected by the user as the beginning of a passage to play (the "passage marker"), and the point that is currently being played (the "play location marker”).
• the passage marker is noted by the device to indicate, for example, a measure that is being played.
• the play location marker is described herein as the cursor.
• the user can choose to show the play location marker (cursor) only, since the back arrow button moves the play location marker to the passage marker location, and since the position wheel moves both markers at once.
• the play location marker will not change appearance during playback. Notes on the display will change color or brightness when they are sounding during playback.
• Controls affecting positioning are the passage marker wheel, the play button, and the back arrow button.
• the two buttons move the play location marker without affecting the passage marker; the wheel moves both markers.
• the default tempo appears on the screen near its wheel in the form of a metronome marking 142.
• the tempo can be changed at any time.
• the display shows the list of pieces on the music card drive.
• Next button 130 Selects the highlighted item from the list controlled by the menu wheel and moves to the next step. Once the menu sequence completes, the next button becomes ineffective. Until the reset button is pushed or power is cycled, the menu wheel continues to control tempo, and any changes are reflected immediately. Menu wheel
• the tempo wheel 118 controls the playback tempo of the music passage. Passage marker wheel
• Control of the passage marker is through the menu wheel.
• Preferred operation of the menu wheel in Passage Marker mode is as follows: If the unit is playing when the wheel is moved, the unit stops.
• Control of the playback volume is through the volume wheel.
• Play button 126 Commences play at the play location marker and moves the marker.
• the size and shape of the screen can be set at runtime by resizing the window on which it and the emulated product controls appear. Meta controls may be added to set limits to resolution, brightness, and contrast.
• the device operates on standard MusicXML files, and also can process condensed or compressed forms of those files.
• the flash drive interface 106 accepts memory cards in various formats. In other words, a meta control is used to select a directory containing MusicXML files. The device then displays those files at the beginning of each menu sequence. If the directory representing the flash drive contains subdirectories, they are not displayed. If a real flash drive is inserted into the machine, it can be selected.

Applications Claiming Priority (2)

Publications (1)

Family

ID=36309014

Family Applications (1)

Country Status (6)

Families Citing this family (43)

Family Cites Families (12)

• 2005
  • 2005-12-15 EP EP05854326A patent/EP1831859A1/de not_active Withdrawn
  • 2005-12-15 KR KR1020077016095A patent/KR20070095332A/ko not_active Application Discontinuation
  • 2005-12-15 WO PCT/US2005/045580 patent/WO2006066075A1/en active Application Filing
  • 2005-12-15 US US11/303,812 patent/US20060150803A1/en not_active Abandoned
  • 2005-12-15 JP JP2007546929A patent/JP2008524656A/ja active Pending
  • 2005-12-15 CN CNA2005800467022A patent/CN101103386A/zh active Pending

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events