WO1997046991A1

WO1997046991A1 - Method and system for providing visual representation of music

Info

Publication number: WO1997046991A1
Application number: PCT/US1997/009402
Authority: WO
Inventors: Raymond H. Stoll; Garin L. Heibert
Original assignee: Seedy Software, Inc.
Priority date: 1996-06-07
Filing date: 1997-06-06
Publication date: 1997-12-11
Also published as: AU3293797A; EP0958568A4; EP0958568A1

Abstract

A system for displaying data which describes the constituent elemnts of a song, and facilitates a musicians ability to learn to play the song on an instrument. The basic constituent elements of the song comprise its chords, or notes, and any accompanying lyrics. Each of these elements of the song is identified and recorded in a file (20), along with the corresponding time that the elements occur in the performance of the song. During playback of the song, the elements in the song file (20) are correlated with the time of performance, and displayed to the musician in synchronism on the display (18, 24). Additional features permit the musician to call up graphic displays of appropriate fingering charts or tablature for each chord, as well as to transpose the song from one key to another.

Description

METHOD AND SYSTEM FOR PROVIDING VISUAL REPRESENTATION OF MUSIC

Background of the Invention

Millions of people, young and old, are continually learning to play guitar, piano and other musical instruments. Further, those musicians are constantly learning more and more songs.

Typically, to learn to play a song on a guitar, the musician might first spend a certain amount of money at a store to purchase an audio CD of an album. The next step is to go to a different store specializing in musical instruments and sheet music to find a song book or sheet music. This is a print publication providing lyrics, chord progressions, music notation, and sometimes tablature. Tablature is a graphic representation of the actual notes as they would be played on a guitar. It is not uncommon to search many stores in hopes of finding the specific book required, as the song book may not have been published depending on the popularity of the album, as measured by the audio CD sales. Further, if it is published, it frequently requires a special order from the publisher due to the increasing costs of inventory, shipping, returns, etc. If available, however, the musician spends more money for the song book. For many years musicians and students have relied on music books published and printed on paper to learn certain songs. Typically this is known as "sheet music". Sheet music can be purchased in music stores, and some record stores. While there are thousands of titles available, they are all pre-printed in multi-sheet or book form.

Since sheet music and books are printed on paper, they are expensive, environmentally unfriendly, cumbersome, and in most cases, require the ability to "read" music to learn the song. The paper-based book is also limited to the single dimension of "reading, " and as such is not interactive. There is no direct connection to the song, and in many cases the sheet music is not scored in the same musical key as the song, requiring complicated key transposing, if the learning musician wants to play along with the song.

Very few novice musicians understand how to transpose keys, and are frequently disappointed to learn that after purchasing the song book, they still can't play the song. This is not a small problem to the learning musician who wishes to learn an instrument, more than he desires to learn to read music.

Obviously, a more learned student can perform the key transposition, but that almost certainly defeats the purpose of the song book.

The paper song book concept is as old as the music publishing industry itself. With no alternative to the paper book, musicians and people trying to learn music have simply purchased what was available to them. That has changed. The cost of paper, printing, inventory, shelf space and shipping has caused the music retailers, publishers and musicians to consider alternatives to the paper songbook. The state of the art direction of the industry today is such that there are numerous approaches to the task of antiquating the paper book. Most, if not all, approaches employ an approach known as MIDI (Music Instrument Digital

Interface), which is a technology requiring expensive specialized hardware and software. Although MIDI technology is here to stay, it is limited to the serious musician who is proficient in the discipline, and has the financial resources for the equipment and software, not to mention the time and training, and experience to work with it.

Summary

The present invention is directed to a software program which includes a novel software engine to generate a music book in a data format. This software allows the user to "see" the song played on a personal computer or CD ROM player, while it synchronizes and plays the original artist's composition from an audio CD. The software program, which can be a data module such as a small diskette, provides the user with the lyrics, guitar/piano chord changes, and guitar/piano chord fingering charts of specifically programmed songs. These sequences are presented on the screen and through the sound card of any personal computer or CD ROM player.

Further, the program provides the ability to play along in synchronization with the artist's original audio CD. The user can actually see and hear the song as it develops. The user simply inserts the audio CD in the player, and the software plays the song and scrolls a visual program on the screen. The user can fast search forward or backward, stop, mute, show chord fingering, lyrics, intros, endings, or any combination of the above.

The software program is a module developed to provide the above listed capabilities on a song-by-song basis. As a result, a learning musician can purchase an album (in the normal audio CD format), and actually have the software included on the CD. This virtually eliminates the need to go to a different store for the learning tools, and is in fact cheaper than the existing alternative. Brief Description of the Figures

Figure 1 is a general block diagram of a system in accordance with the present invention for displaying information about a song;

Figure 2A and 2B are illustrations of a song file data format;

Figure 3 is a signal graph showing the onset of noise at the beginning of a song to detect a synchronization start time;

Figure 4 is an illustration of a window in which song elements are displayed;

Figure 5 is an illustration of an option window from which a user can select a graphic display of a chord;

Figure 6A and 6B are illustrations of a finger chart and tablature chart for a chord; and Figure 7 is a display of the Settings menu for transposing the key of a song.

DETAILED DESCRIPTION

The present invention comprises, in general, a system via which information is displayed that instructs, or otherwise assists, a musician to play a particular song on an instrument while listening to a pre-recorded version of that song. In the embodiment which is described hereinafter, specific reference is made to an example in which a person is learning to play a guitar while listening to a compact disc (CD). It will be appreciated, however, that the practical applications of the invention are not limited to this particular instrument or mode of playing a song.

Figure 1 illustrates the basic elements of a system which incorporates the principles of the present invention. A song which is recorded on a CD 10 is reproduced through an audio CD playback component of a personal computer 12. The audio playback component can be of any suitable conventional type, and generally includes a read head 14 which reads digital data recorded on the CD, and an audio synthesizer 16 which converts the digital data into an audio signal. The audio signal is fed to one or more speakers 18 connected to the computer, to reproduce the song. Associated with the song being reproduced is a song file 20. The song file contains preprogrammed data which describes musical components of the song. The song file can be distributed to the consumer in a number of different ways. For example, it can be recorded on the CD 10, along with the song itself. Alternatively, it can be separately provided, for example on a diskette, or downloaded from a network and stored on a hard disk drive, or other mass storage device, of the computer.

Each song has its own unique song file 20 which describes the notes, chords and lyrics (if any) which make up the song. In a preferred embodiment of the invention, if the CD 10 relates to an album which contains a number of songs, the consumer is provided with a plurality of song files, each of which is associated with one of the songs on the album.

The data stored in the song file is read by a program 22 which is executing on the computer 12, and causes the chords and lyrics to be displayed on a monitor 24 associated with the computer, by means of its display controller 26. The display of the chords and lyrics, in accordance with the data in the song file 20, is synchronized with the playback of the song from the CD 10. For this purpose, a synchronization signal is obtained from the audio signal that is read from the CD 10, and used to control the display of the information.

A song can be represented on a computer by dividing it into two parts - a single "song header" followed by one or more "song elements" as illustrated in Figure 2A. A song header contains the following general information about the song which does not change through time:

Song Header

Descriptor Description

SongVersion A version number allowing for changes in the format of the song file

SongName The song's name expressed as an ASCII string. Strings are represented by a two-byte integer containing the number of characters in the string, and then that number of bytes - containing the string information itself.

SongKey The song's key expressed as an ASCII string.

CDID An ID code for the audio CD which this song can be played from and synchronized with. Track The track number on the CD to be synchronized with.

NumElem The number of elements (part/chord/lyric combinations) which make up the song. The song also has one or more "song elements, " made up of the following information:

Song Element Descriptor Description

Part A string containing a part reference in a song.

Chord A string containing a chord reference in a song.

Lyric A string containing a lyric reference in a song.

Sec A real number representing when the element takes place during a performance. NumTabElements Number of Tablature elements attached to this song element

X X location in Tablature grid

Y Y location in Tablature grid

TabElem String to be displayed on Tablature grid

The song file exists in the memory of a computer - RAM, magnetic, or optical. The file is a binary concatenation of a single song header and one or more song elements, as illustrated in Figure 2B. The number of elements is stored in the header under the descriptor NumElem.

The data in a song file is manually determined, and entered by a person as he or she listens to the song of interest. More specifically, as the programmer listens to the song, he or she records each of the words, or lyrics, in the song.

Each new lyric forms a separate element. Each time a chord change occurs in the instrumental portion of the song, the new chord is recorded in association with an associated lyric, if the lyric is spoken at the time of the chord change.

Otherwise, the new chord forms its own element. For each element, an associated time value is recorded. In a preferred embodiment of the invention, the time value indicates the length of time which has elapsed from a specific point at the beginning of the song. By associating each element with a specific time in the performance of the song, the subsequent display of the elements can be synchronized with the playback of the song.

The determination of the instant in time which forms the synchronization point for all display of elements is made with reference to noise levels in the audio signal. Referring to Figure 3, at the beginning of a song track on a CD, before the song actually begins, the magnitude of the audio signal 28 is below a threshold noise level. At this time, there is no audible sound to be heard. At the very beginning of the song, the signal level begins to rise to a point where it crosses the threshold. This point, at which the signal level crosses the threshold, is designated as time to, and forms the synchronization point that is used to determine when each element is to be emphasized. Preferably, the threshold level is set so that the level of the signal is of sufficient magnitude to be reliably detected, but still below the level at which audible sound can be heard by the human ear. Once this level is reached, a timer is started, to count elapsed time. During the programming of a song file, this elapsed time is recorded in the

"Sec" field of each song element. During playback, the elapsed time determines which element is to be emphasized at any given point.

In operation, a song is displayed by "painting" the screen of the monitor 24, based on the current time within the performance of a song. The songware program 22 queries the elapsed time counter to determine the current position in the performance of the song. The program then determines which line of elements is to be displayed first. This is done by scanning the song data for a time which matches or exceeds the current performance time. While scanning, the computer keeps track of the position of two "beginning-of-line" markers. Referring to Figure 4, once the scan is complete, the computer can start painting the screen by displaying the Song Name and Song Key at the top (centered), and then the song elements starting with the first beginning-of-line marker. In the example of Figure 4, the first display line contains six elements. The number of elements in any line can be determined at the time the song file is created. An arrow is placed to the left of the second "beginning-of-line" marker to show the current line position. In this example, the current line contains the next four song elements. The computer continues displaying lines of song elements until there is no longer room for additional lines on the screen.

During the performance of a song - a performance being the playback of a song with or without accompaniment with an audio CD - the computer display shows the current song element, as well as some older and some newer elements. How many newer and older elements are displayed is dependent on how much information can fit on the display device. A song file can be displayed by having the computer cycle through the following steps: 1) evaluate the current time relative to when the performance was begun

2) scan through the song file elements to find the element with the Sec field closest to the current time

3) record the last two "beginning of line" markers before the current element

4) if there has not been a change in the marker positions since the last evaluation, go back to step #1

5) in the upper-left corner of the display area, display the element pointed to by the oldest marker 6) continue displaying new elements from left to right across the display, until a "beginning of line" marker is reached

7) go back to the left edge of the display area and move below the last line of elements

8) display new lines of elements until the newest line is being drawn off the bottom of the display area

9) display an arrow to the left of the second line of elements

10) go to step #\

By cycling through the above steps, the computer display always shows the current position in the performance, marked with an arrow. Within a given line, the particular element that is most closely related to the current time of performance is emphasized. For example, all of the elements might normally by displayed on the screen using a certain color, such as black. When the time value associated with a given element matches the performance time, i.e. the elapsed time from the synchronization point, the data displayed for that element is changed to a different color, such as red. The display of that element is maintained in the other color, until the time of performance matches the next successive element in the line. Thus, for example, if the four elements displayed in the current line in the example of Figure 4 have respective time values of 0.35, 0.50, 0.75 and 1.00 second, at time t = 0.35 sec, the data for the first element is displayed in red, and all other elements are displayed in black. At time t = 0.50 sec, the display for the first element returns to black, and the second element is displayed in red. This procedure continues for each of the successive elements, as the performance time matches their associated recorded time values.

For a given element, the lyric string is comprised of the particular word spoken by the singer at the associated time. The chord string is normally displayed as a letter, or series of letters, which correspond to the known chords on the instrument. For example, if the first chord in a song is D-minor, the first element would be displayed as "Dd". If the second chord is F-major, the element at which this chord change occurs would display "F", along with any corresponding lyric that is sung at the same time.

A more experienced musician is likely to recognize each of these chords by their alphabetical description, and immediately know how to play them on the instrument. However, a beginning student may not know where to place his fingers on the neck of a guitar, and which strings to pick to play the chord. In accordance with a further feature of the invention, the information necessary to instruct the user is displayed upon entry of a suitable command. In a preferred embodiment, the user calls up this information by positioning a cursor over the display of the chord of interest, and clicking a mouse button. For example, if the first chord in the song is displayed as "Dd", and the user is unsure how to play that chord, by clicking the mouse button while the cursor is positioned over the chord value, a first window is displayed, as illustrated in Figure 5. This first window provides the user with the option of obtaining information in one of two formats, labeled "Chords" and "Tablature. " If the user selects the "Chord" option, a further window is displayed which illustrates the correct fingering diagram for playing that chord on the guitar, as depicted in Figure 6A.

Alternatively, if the user selects the "Tablature" option, information pertaining to the chord is displayed in a format similar to that which is employed for sheet music. Specifically, a tablature diagram consist of a number of horizonal lines which correspond to the strings of the guitar, e.g. six in the illustrated example. For each string that is to be fingered by the player, a number is displayed which indicates the particular fret at which the player's finger should be positioned for that string. For this purpose, the song element information that is stored in the song file includes four fields that relate to the tablature information. The first field, "NumTabElements" , indicates the number of tablature elements for the given song element, i.e. the number of strings that need to be fingered. For each tablature element, there is an associated X coordinate, Y coordinate, and "TabElem" string, comprising the number of the fret that is to be fingered. In the example of Figure 6B, there are three tablature elements, which respectively contain the "TabElem" values two, three and two. Another operation which is often difficult for beginning musicians is the transposition of a song from one key to another. For example, a song may be written in the key of D, but the user may desire to play it in the key of C, to more easily sing along with it. The songware window includes a menu command labeled "Settings", which allows the user to change the key for a song. For example, if the user desires to change the key from D to C, such a change is implemented by moving down two steps on the musical scale, i.e. from D to D- flat, and from D-flat to C. All other notes and chords in the song are decremented by the same amount. Thus, if the original progression of notes in a song is "DAGE", for example, by switching to the key of C, that progression changes to "CGFD". In other words, each note has been decremented by two steps.

When the user selects the "Settings" command from the menu bar, he is presented with a number of options relating to the number of steps by which the key for a song can be incremented or decremented, as illustrated in Figure 7. By the selecting the value "-2" for example, a song which is in the original key of D is then transposed to the key of C. As a result of this transposition, every chord and note is decremented by a value of two. In addition, if the user mouse-clicks on the chord of any element, the fingering diagram that is displayed for the chord corresponds to the transposed value.

From the foregoing, it can be seen that the present invention comprises a system by which various types of data relating to the constituent elements of a song are displayed to a user, in synchronism with the playback of the song itself, to thereby enhance the user's ability to play the song on a musical instrument. Within the capabilities described above, various options can be provided to the user, to further facilitate the learning process. For example, the user can employ fast-forward and reverse functions, to enable him to repeat and search for particularly difficult parts. In addition, the user can choose to mute the actual playback from the CD, for example to more clearly hear the notes as they are being played on the instrument. In a similar manner, the user can choose to have only certain components of each element displayed, e.g. chords only or lyrics only.

The preceding description of one embodiment of the invention has been made in the context of a song which is prerecorded on a compact disk. It will be appreciated, of course, that this is not the only medium via which a user can obtain a song. Another medium which is gaining increasing popularity is the downloading of audio files over networks, e.g. the internet. The invention is equally applicable to songs which are provided to the user in this manner. In such a case, the song header might include a different field in place of the "CDID" field described above. For example, the field might comprise a variable link string that is the internet location for an audio stream, expressed as a uniform resource locator (URL). In all other respects, however, the implementation of the invention can be the same as described above. It will be appreciated by those of ordinary skill in the art that the present invention can be embodied in other specific forms without departing from the spirit or essential characteristics thereof. The presently disclosed embodiments are considered in all respects to be illustrative and not restrictive. The scope of the invention is indicated by the appended claims, rather then the foregoing description, and all changes that come within the meaning and range of equivalence thereof are intended to be embraced therein.

Claims

IN THE CLAIMS:

1. A method for displaying data simultaneously with the reproduction of a song to assist a viewer in the playing of a musical instrument to accompany the song, comprising the steps of: identifying constituent elements of the song during an initial reproduction of the song; recording information describing each of said constituent elements in a data file that pertains to the song, said file including for each of said recorded constituent elements, respective associated time values which indicate the time at which each constituent element occurred in the play of the song; reproducing the song subsequent to said recording step; counting elapsed time from a predetermined point at the beginning of the song during said subsequent reproduction; reading time values recorded in said data file; and displaying the recorded information that describes a constituent element when its associated time value corresponds to the elapsed time counted from the beginning of the song.

2. The method of claim 1 wherein said constituent elements comprise at least musical chords in the song.

3. The method of claim 2 wherein said constituent elements further include lyrics of the song.

4. The method of claim 1 wherein the information for plural successive constituent elements in the song is simultaneously displayed, and the displayed information for each constituent element is emphasized at a time when its associated time value corresponds to the counted elapsed time. 5. The method of claim 2, further including the step of displaying graphical information which informs the user how to play a chord on the instrument.

6. The method of claim 5, wherein said graphical information comprises a fingering chart for a guitar.

7. The method of claim 5, wherein said graphical information comprises a tablature chart.

8. A system for displaying data simultaneously with the reproduction of a song to assist a viewer in the playing of a musical instrument to accompany the song, comprising: a memory storing a data file containing information describing constituent elements of the song, said file including for each of said constituent elements, respective associated time values which indicate the time at which each constituent element occurred in the play of the song; means for reproducing the song; a timer for counting elapsed time from a predetermined point at the beginning of the song during a reproduction; a display device for displaying the stored information that describes a constituent element when its associated time value corresponds to the elapsed time counted from the beginning of the song. AMENDED CLAIMS

[received by the International Bureau on 7 November 1997 (07.11.97); original claims 1 and 8 amended; new claims 9-20 added; remaining claims unchanged (4 pages)]

1. A method for displaying data simultaneously with the reproduction of a song to assist a viewer in the playing of a musical instrument to accompany the song, said song comprising an audio signal, comprising the steps of: identifying constituent elements of the song during an initial reproduction of the song; recording information describing each of said constituent elements in a data file that pertains to the song, said file including for each of said recorded constituent elements, respective associated time values which indicate the time at which each constituent element occurred in the play of the song; reproducing the song subsequent to said recording step; counting elapsed time from a point at which the magnitude of said audio signal at the beginning of the song crosses a threshold during said subsequent reproduction; reading time values recorded in said data file; and displaying the recorded information that describes a constituent element when its associated time value corresponds to the elapsed time counted.

4. The method of claim 1 wherein the information for plural successive constituent elements in the song is simultaneously displayed, and the displayed information for each constiment element is emphasized at a time when its associated time value corresponds to the counted elapsed time.

5. The method of claim 2, further including the step of displaying graphical information which informs the user how to play a chord on the instrument.

8. A system for displaying data simultaneously with the reproduction of a song, which comprises an audio signal, to assist a viewer in the playing of a musical instrument to accompany the song, comprising: a memory storing a data file containing information describing constituent elements of the song, said file including for each of said constituent elements, respective associated time values which indicate the time at which each constiment element occurred in the play of the song; means for reproducing the song; a timer for counting elapsed time from a point at which the magnitude of said audio signal crosses a threshold at the beginning of the song during a reproduction; a display device for displaying the stored information that describes the constituent elements when its associated time value corresponds to the elapsed time counted.

9. A method for displaying data simultaneously with the reproduction of a song to assist a viewer in the playing of a musical instrument to accompany the song, said song comprising an audio signal, comprising the steps of: identifying a plurality of constiment elements of the song during an initial reproduction of the song, said constiment elements comprising a plurality of different types of song elements; recording information describing each of said constiment elements in a data file that pertains to the song, said file including for each of said recorded constiment elements, respective associated time values which indicate the time at which each constituent element occurred in the play of the song; reproducing the song subsequent to said recording step; counting elapsed time from a predetermined point at the beginning of the song during said subsequent reproduction; reading time values recorded in said data file; and displaying the recorded information that describes the constiment elements when their respective associated time values correspond to the elapsed time counted at the beginning of the song, in a manner such that the different types of song elements are arranged relative to one another in accordance with their respective associated time values.

10. The method of claim 9 wherein one type of song element comprises musical chords in the song.

11. The method of claim 10 another type of said song element is the lyrics of the song.

12. The method of claim 11 wherein said constiment elements are displayed in a manner such that time is represented along the axis, wherein a musical chord and a lyric which have the same associated time value are displayed at the same position along said axis.

13. The method of claim 9 wherein the information for plural successive constiment elements in the song is simultaneously displayed, and the displayed information for each constiment element is emphasized at a time when its associated time value corresponds to the counted elapsed time.

14. The method of claim 10, further including the step of displaying graphical information which informs the user how to play a chord on the instrument.

15. The method of claim 13, wherein said graphical information comprises a fingering chart for a guitar.

16. The method of claim 13, wherein said graphical information comprises a tablature chart.

17. A system for displaying data simultaneously with the reproduction of a song, which comprises an audio signal, to assist a viewer in the playing of a musical instrument to accompany the song, comprising: a memory storing a data file containing information describing constiment elements of the song, said constiment elements comprise a plurality of song elements, said file including for each of said constiment elements, respective associated time values which indicate the time at which each constiment element occurred in the play of the song; means for reproducing the song; a timer for counting elapsed time from a predetermined point at the beginning of the song during a reproduction; a display device for displaying the stored information that describes the constiment elements when their respective associated time values correspond to the elapsed time counted, in a manner such that the different types of song elements are arranged relative to one another in accordance with their respective associated time values.

18. The method of claim 16 wherein one type of song element comprises musical chords in the song.

19. The method of claim 17 wherein another type of song element is the lyrics of the song.

20. A method for storing data representing a song, comprising the steps of: identifying constiment elements of the song during an initial reproduction of the song, said constiment elements comprising musical chords and lyrics in the song; recording information describing each of said constiment elements in a data file that pertains to the song on a recording medium, said file including for each of said musical chords and lyrics, respective associated time values which indicate the time at which each such constiment element occurred in the play of the song.