JP4127088B2 - Music playback and video display control apparatus and program thereof - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a music playback and video display control device that displays a display object as a video on a display screen in association with music playback.
[0002]
[Prior art]
2. Description of the Related Art Conventionally, a sequencer application program is known that plays music data in which performance data is described together with information on the playback position of the performance data as time progresses and displays a musical score corresponding to the performance data.
Specifically, along with the original sequencer module that records and plays back MIDI (Musical Instrument Digital Interface) data and audio waveform data, a music score display module that displays music based on MIDI data, and a piano roll display module that displays piano roll Is included.
Further, the playback position of the performance data can be controlled by clicking the playback button, stop button, playback position counter, etc. on the setting display screen with the mouse. Further, the playback tempo can be controlled by operating the tempo setting button on the setting display screen.
[0003]
However, the above-described musical score display and piano roll screen display are all directly connected to the contents of the music, and the setting display screen virtually displays a mechanical operation panel. Therefore, even though the displayed image is convenient for editing performance data, it is dry and cannot be seen and enjoyed.
On the other hand, while reproducing music data, displaying an image of a musical instrument corresponding to a timbre and changing the size of the image according to the volume of the performance sound of each timbre is known (Patent Document 1). reference).
[0004]
In addition, a player icon for playing an instrument corresponding to the tone of the performance part is displayed with the performance stage in the background, and the position of the player icon is determined according to the pan (location) information of each performance part while playing the music data. The player icon image is moved according to the tempo and the volume of each performance part, and the pan information setting of each performance part can be changed according to the display position of the player icon. Those are known (see Patent Document 2).
Therefore, the user can watch and enjoy different moving images for each piece of music data by watching the movement linked to the performance of the displayed player icon, and can control the performance using the moving image as a user interface.
However, since it is a moving image that expresses performance, there is an emphasis on performance status display and performance operability, and various displays cannot be expected. Therefore, there was little fun.
[0005]
[Patent Document 1]
JP-A 64-91173
[Patent Document 2]
Japanese Patent Application Laid-Open No. 11-352962
[0006]
[Problems to be solved by the invention]
The present invention has been made to solve the above-described problems, and displays a moving image that is expressed in accordance with the music to be played, and actively changes the moving image display screen by the user. It is an object of the present invention to provide a music playback and moving image display control device capable of controlling music playback, and a program thereof.
[0007]
[Means for Solving the Problems]
  According to the present invention, in the first aspect of the present invention, the music reproduction that uses the music data and the animation data and displays the moving image of the display object on the display screen in association with the music reproduction that reproduces the music data as time progresses A video display control device, wherein the music data is a performanceWith eventsThe performanceAn eventPlayback positionThe time information indicating the time of the performance event indicatingDescribedWith a performance sequenceThe animation data isBy providing a display sequence in which a display position of a display event for displaying the display object and time information indicating the time of the display event are described in pairs,The display position of the display object is described so as to move along a predetermined route that is set in advance. In the music playback, note length information, note pitch information, note volume information, each note A setting means for associating at least one control information of the musical sound attribute information that does not change with the display form of the display object in the moving image display, and setting the other side according to the setting on one side, and time progression Display position progress control means for causing the display position of the display object set by the setting means to advance in accordance with the description of the animation data in correspondence with the current playback position of the music playback that changes according to A display corresponding to a playback position where the display position of the display object is different from the current playback position of the music playback by an operation. When it is moved to the location, and has a playback position progression control means to continue the time-marching from said different reproducing position.
  Accordingly, since the display object having a display mode corresponding to the reproduced musical sound advances in accordance with the reproduction position of music reproduction, the enjoyment increases. Further, since there is an interaction that the user can change the music playback by operating the moving image display screen, the music playback can be actively enjoyed.
  A user who does not have a high level of comprehension experiences a musical sense by experiencing the interaction between music playback and video display.
  It is more effective if the display object is an image of a different type from an image related to music reproduction.
  Since the performance data is used, it is possible to easily set at least one piece of control information of the musical sound and to detect and control the time progression of music playback.
[0008]
A plurality of types of display object display modes and tone control information are provided, and the setting means described above includes at least one control information of a tone and display object information from among the provided types. It is possible to determine the pair with the display mode and perform music reproduction and moving image display.
In this case, since video display and music playback can be performed in conjunction with each other according to the user's operation, the user's enjoyment, more interest, and understanding of the user's music or music Can help.
Further, the display position progress control means described above can move the display position of the display object along a predetermined path in accordance with the playback position of music playback that changes with time. The predetermined route can be set using a correspondence table of time and display position or using a predetermined function.
In this case, since the correspondence relationship between the current reproduction position of music reproduction and the display position of the display object is clearly defined, time progress of the display position and the reproduction position is facilitated.
Furthermore, when the display position of the display object is controlled to deviate from the predetermined path by the display position progress control means, the playback position progress control means stops the time progress in music playback or repeats only accompaniment. If it is played back, the user can be notified that the predetermined route has been missed.
[0009]
  According to a second aspect of the present invention, in the music playback and moving image display control device according to the first aspect,The display mode of the display object associated with the musical sound attribute information by the setting means is the type of image representing the character..
According to a third aspect of the present invention, in the music playback and moving image display control device according to the first aspect, the display mode of the display object that the setting means associates with the note length information is a cycle of the repeated action of the character. Yes, the display mode of the display object associated with the note pitch information is the color of the character, and the display mode of the display object associated with the note volume information is the amplitude of the character's repetitive motion or the size of the character It is.
[0010]
  Claim4In the invention described in claim 1,1In the music playback and moving image display control device according to claim 1, before the performance data is set by the setting meansShortageThere is provided musical score display data creating means for creating musical score display data for displaying a musical score on the display screen in accordance with performance data reflecting at least one control information and information on the reproduction position of the performance data. .
  Therefore, it is possible to easily know the interrelationship between music playback and moving image display on the music score display screen, and to view the score while having fun.
[0011]
  Claim5In the invention described in the above, the automatic accompaniment apparatus described in claim 1 described above is realized in the form of a program for causing a computer to execute. Similarly, claim 2To 4The invention described in (1) can also be realized in the form of a program executed by a computer.
[0012]
DETAILED DESCRIPTION OF THE INVENTION
FIG. 1 is a system diagram illustrating the overall configuration of an embodiment of the present invention.
In the figure, 1 is an electronic music apparatus such as an electronic musical instrument, 2 is a personal computer, 3 is a communication network such as the Internet, and 4 is a web server.
The web server 4 is a download site that provides electronic music picture book content. An electronic music picture book is multimedia content having music data and animation data. The music data of the electronic music picture book is reproduced as time progresses, and a moving image is displayed on the display screen in association with the music reproduction, for example, by moving the character image in front of the background screen.
Although the case where the music data is music performance data such as MIDI will be described, it may be a digital audio waveform.
When the personal computer 2 has a display and a music playback function, the electronic music picture book described above can be used on the personal computer 2 side.
On the other hand, when the electronic music apparatus 1 has the above-described function, an electronic music picture book can be acquired using the personal computer 2 as an interface and used on the electronic music apparatus 1 side.
[0013]
FIG. 2 is an explanatory diagram showing an example of the file structure of the electronic music picture book used in the embodiment shown in FIG.
The electronic music picture book includes an animation data file shown in FIG. 2A and a music data file shown in FIG. Both files may be one sequence data file, but by configuring both files independently, the user can combine multiple types of animation content and multiple types of music content, or as a music data file A general music data file for automatic performance can be used.
[0014]
The music data file in FIG. 2B includes a header part and a performance sequence part. It is essentially the same as a conventional music data file.
In the header portion, a song number, a song title, a correspondence table of channel (number) and tone color (number), information on tempo, time signature, tonality, etc. are described.
The performance sequence section is described according to time progress, for example, with a performance event and time information indicating the time of the performance event as a pair. The time information indicates, for example, a reproduction position. Therefore, the absolute time scales with the tempo.
In this embodiment, it is described by the time (measure: beat: clock) from the start of display, but may be described by the elapsed time (duration) from the previous performance event. Also in this case, the subsequent processing becomes easier if the time is rewritten to the time from the start of display and stored in the memory.
[0015]
On the other hand, the animation data file in FIG. 2A has a header part and a display sequence part.
In the header portion, an animation number, an animation name, a correspondence table indicating a correspondence relationship between a plurality of characters (numbers) and images (numbers), character image data, and background image data are described. A plurality of types of character image data are prepared.
In the display sequence part, the display position of the display event (character) and the time information indicating the time of the display event are paired. The moving path of the character is defined by the description of the display sequence.
[0016]
The time information is a time for controlling the display position of the character, has a corresponding relationship with the time information for music reproduction, and the same information (music information reproduction position) may be used as the time information.
In this embodiment, the time is described from the display start time, but may be described in duration. Also in this case, it is more convenient to rewrite the time from the start of display and save it in the memory.
When the character progresses at equal time intervals, only the character display position needs to be described.
Further, when there is a predetermined function relationship between the time information and the display position, this function need only be described.
[0017]
FIG. 3 is an explanatory diagram showing an example of an electronic music picture book utilization system executed on the electronic music apparatus 1 and personal computer 2 side of FIG.
In FIG. 3, a case where an application program installed in the personal computer 2 performs animation display and music data reproduction will be described with reference to FIGS.
FIG. 4 is an explanatory diagram of a correspondence relationship between musical tone control information and display objects in the embodiment shown in FIG.
FIG. 5 is an explanatory diagram of a correspondence relationship between the moving route of the character display position and the music playback position in the embodiment shown in FIG.
In FIG. 3, the application program has functions of an animation display module 11, a conversion module 12, a sequencer module 13, a score display module 13a, a sound module 14, an I / O module 15, and the like.
Each of the modules 11 to 14 can be implemented as WWW browser plug-in software having a function of accessing a web page on the Internet.
The mouse 16 and the display 17 are connected as input / output devices.
[0018]
The animation display module 11 uses the animation data file shown in FIG. 2A, for example, an animation (moving image) having at least one type of character (display object) that moves as a foreground image on the background image, for example. It has a function to display.
Characters, movement paths, and the like have default values defined, and the user can change settings using the mouse 16 and other operators such as a keyboard of a personal computer or an electronic music device.
[0019]
On the other hand, the sequencer module 13 manages the time progression of music reproduction based on the music reproduction position.
The music data file shown in FIG. 2 (b) is stored in the memory, and the performance data included in the music data is generated from a predetermined playback start position at a specified tempo according to the playback position information of the performance data. 14 for output.
As will be described later, there is a case where at least one control information of the musical tone is set via the conversion module 12 by a user operation on the animation display module 11 side. In this case, the control information is reflected in the performance data and then output to the sound source according to the time series of the playback position of the performance data.
The sequencer module 13 is realized by the function of an existing performance sequencer program. The sound source module 14 is not necessary when a hardware sound source is used.
[0020]
The illustrated sequencer module 13 includes a score (score) display module as plug-in software.
The performance data included in the music data is visually displayed on the display, and the display becomes an operation screen, and the start and stop of the playback, the playback position, the tempo, and the tone are changed according to the operation of the mouse 16 and other performance operators. In addition to specifying, you can also edit music data. Normally, an animation screen is displayed, and a score display module is displayed at the same time or switched as necessary. The musical score display can be replaced with a piano roll display.
[0021]
Next, the conversion module 12 controls the overall content playback by controlling the interaction between music playback (which may include score display) and video display.
As a functional means to control the interaction,
First, there is a setting function for associating at least one control information of a musical sound in music reproduction with a display mode of a character (display object) in moving image display and setting the other side according to the setting on one side.
Secondly, there is a display position progress control function for advancing the display position of the character (display object) corresponding to the playback position of music playback that changes with time.
Third, when the display position of the character (display object) is moved to a display position corresponding to a playback position different from the current playback position of music playback by a user operation, the time progress is started from this different playback position. There is a playback position progress control function to continue.
[0022]
The setting function described above performs mutual setting by associating at least one musical tone control information with at least one display mode of a character using a conversion table or the like. Music performance is reproduced by reflecting at least one control information of the set musical sound in performance data and outputting the performance data to a sound source according to the time series of the reproduction position of the performance data.
FIG. 4 shows an example of a correspondence relationship between musical tone control information and display objects.
As musical tone control information, musical tone attribute information that does not change for each note (for example, tone color, performance part, etc.) is suitable. However, the note length, pitch, volume, etc. which change for every note may be sufficient.
On the other hand, the display mode of a character in moving image display includes the type, size, color, and repetitive action of an image (symbol) representing the character. In addition, although a character means a character or an anthropomorphic display mark, it is not necessarily limited to these and is generally called a display object.
The note length corresponds to, for example, the cycle of the character's repetitive motion (eg, swinging left and right), the pitch corresponds to the character's color, and the sound intensity corresponds to the character's repetitive motion amplitude or character's Make the size correspond.
[0023]
Therefore, the conversion module modifies the character image corresponding to the timbre set for music playback, or conversely, modifies the timbre by setting the character image on the video display screen. can do. As a result, an even more interesting video expression can be achieved.
The mutual setting function described above can also be applied when displaying a character that does not move with time.
Music reproduction can also be performed using music data based on digital music waveforms. In this case, it is easy to control the sound volume. When it is desired to control the tone color of a musical tone, if a plurality of digital music waveforms having different tone colors are previously placed in an electronic music picture book, the digital music waveform may be selected based on tone color control information. As for the playback position of the digital music waveform, it is only necessary to create a music data file by adding time information.
[0024]
Next, with the display position progression control function described above, the image on which the character is displayed can be advanced according to the time progression of music reproduction.
For example, as shown in FIG. 5, the animation data file and the music data file of FIG. 2 are created so that the start point and end point of the character display position coincide with the start point and end point of the music playback position. ing.
For example, the character display position is moved in units of one beat. The movement of the character can be designed at a constant speed, but the value changes according to the change in the music playback tempo.
As will be described later with reference to FIG. 7, the display position of the character does not necessarily travel on a straight path, but moves on a free curved path as time progresses.
[0025]
The playback position progress control function described above changes the music playback position using the moving image display screen as a user interface.
In FIG. 5, when the character is moving to the point A, and the character display position is forcibly moved from the point A to the point B by a user operation (for example, a drag and drop operation using the mouse 16). In addition, time progress is continued from the music playback position corresponding to the display position after movement. Therefore, the music playback position is skipped.
In this embodiment, the time information is supplied from the sequencer module 13 for music playback to the animation display module 11 via the conversion module 12. Alternatively, the conversion module 12 may have a sequence management function to supply time information to the sequencer module 13 and the animation display module 11.
[0026]
Returning to FIG. 3 again, the I / O module 15 will be described.
The I / O module 15 outputs operation information from the mouse 16 to other modules and outputs image display data to the display 17. It also serves as a connection interface with the electronic music apparatus 1 and inputs keyboard operation information (note on / note off, note number, velocity) as user operation information, or automatic performance data as model music data to the electronic music apparatus 1 It may be transferred and a guide lamp for performance guidance may be turned on. In this case, the performance evaluation result of the user can be reflected in the character display mode on the animation display screen 51.
[0027]
In the above description, the animation display is exemplified, but a moving image display in general may be used.
Here, the moving image display includes not only a background image, a front image, a character image, etc., in which at least the character image moves (including those that sometimes move in a predetermined manner), but also an integrated display screen without distinction. It may be moving.
You may use the image data of the frame image (a character is included) actually image | photographed with the camera. In this case, the animation display sequence portion shown in FIG. 2A describes image data at a time specified by each time information, or a number for specifying the image data. In the latter case, the image data is described collectively in the header part.
In this case, the moving image display is performed by switching the frame image for each reproduction position specified by each time information and moving the frame image. Even in such a case, in order to designate a character with the mouse 16, data on the display position of the character is necessary.
[0028]
FIG. 6 is a hardware configuration diagram for realizing one embodiment of the present invention.
In the figure, 31 is a bus, 32 is a CPU (Central Processing Unit), 33 is a ROM (Read Only Memory), and 34 is a RAM (Random Access Memory). The RAM 34 is provided with a working area for the CPU 32, and the ROM 33 stores a basic input / output system program for operating the CPU 32. The operating system program, the application program, and the electronic music picture book are stored in the hard magnetic disk of the external storage device 37, loaded into the RAM 34, and animation display, staff display, and automatic playback of music data are performed. .
[0029]
An input operation unit 35 is a keyboard or a mouse.
Reference numeral 36 denotes a display which displays an animation screen, a staff, a sequence operation screen, and the like. Each screen can be an individual window screen, and a plurality of window screens can be displayed simultaneously.
Reference numeral 38 is a sound source circuit, and 39 is a sound system. The sound source circuit 38 receives automatic performance data and sound source control parameters corresponding to the performance data transferred from the electronic musical instrument 41, synthesizes the performance sound, supplies it to the sound system 39, and outputs it from the speaker. The tone generator circuit 38 is configured by dedicated hardware, a DSP (Digital Signal Processor) and a microprogram. Alternatively, a sound source function is realized by a software sound source program that causes the CPU 32 to perform waveform synthesis.
[0030]
37 is a drive for recording media such as HDD (Hard Disk Drive), CD-ROM (Compact Disk Read Only Memory), MO (Magneto Optical Disk), DVD (Digital Versatile Disk), etc., as well as FDD (Flexible Disk). Drive) and an external storage device such as a flash memory (semiconductor memory card).
A network interface 40 connects an external device such as an electronic musical instrument 41 to the bus 31. Not only a dedicated MIDI interface but also a general-purpose interface or direct connection or LAN (Local Area Network) connection, or connection to a server or the like via the Internet 42.
[0031]
FIG. 7 is an explanatory diagram showing an example of the animation display screen 51.
In the figure, 52 is a character hangar, 53 is a car character, 54 is a train character, 55 is a ship character, and 56 is an airplane character. Each character corresponds to a different timbre, for example, organ, guitar, flute, and piano.
When the piano part is selected on the sequencer module 13 side, the airplane character 56 automatically moves to the start point. When the music reproduction is started, the airplane character 56 moves on the flight route 57 (movement route) as time progresses. You may design the display sequence part of the animation data file shown to Fig.2 (a) so that it may become constant speed. The illustrated flight route 57 is a curve having the right end of the display screen as a start point and the left end as an end point.
The flight route 57 may extend over a plurality of pages. In this case, the page is automatically switched when the end of the display page is reached.
Alternatively, when the end point is reached, the same route may be repeated by returning to the start point.
[0032]
The user moves the pointer of the mouse 16 to the airplane character 56 at the position of the current position A, left-clicks the button of the mouse 16, and drags and drops 58 to the position of the movement position B, thereby warping the airplane character 56 ( Instantaneous movement). If the movement position B is also on the flight route 57, the airplane character 56 starts moving from time to time, and music playback is started from the playback position corresponding to the movement position B. To do.
Further, when the car character 53 is dragged and dropped 59 from the hangar 52 to the current position A as shown by the movement route 59, the tone is automatically changed to the organ on the music playback side, and is moved to the current position A. Playback starts from the corresponding playback position.
When either the speed-up button 60 or the speed-down button 61 is clicked with the mouse 16, the movement speed is increased / decreased, for example, by 15% per second during the click-on. In conjunction with this operation, the music playback side increases / decreases the tempo.
Reference numeral 62 denotes an icon for displaying a screen for selecting one of a plurality of types of animation data files (animation numbers). The music data is selected on a music score display screen described later.
[0033]
In the above description, in order to move the character display position on the display screen, the display position of the pointer of the mouse 16 is overlapped with the display position of the airplane character 56, and then a drag and drop operation is performed.
As another operation, just by dragging and dropping the mouse 16 at an appropriate position on the animation display screen 51, the airplane character 56 is moved on the flight route 57 according to the value of the x coordinate or the y coordinate. You may make it move with.
The flight route 57 is displayed visually on the animation display screen 51, but if it is hidden, it will be more interesting.
[0034]
If the airplane character 56 is not on the flight route 57 when the mouse 16 is dropped, the display position of the airplane character 56 may be left as it is. It may be moved. For example, it is conceivable to instantaneously move to a display position on the flight route 57 that takes the same coordinates as the display position on the nearest flight route 57 or the x-coordinate or y-coordinate of the display position when dropped.
As described above, when the airplane character 56 deviates from the flight route 57, various methods can be considered as to how the music reproduction proceeds.
Typically, music playback is stopped, and when the display position is set again on the flight route 57, music playback is started from the playback position. For the performance of the melody part corresponding to the airplane character 56, the music reproduction is stopped, but if there is an accompaniment part, the accompaniment may be reproduced repeatedly.
[0035]
FIG. 8 is an explanatory diagram showing an example of the score display screen 71.
In the figure, reference numeral 72 denotes a score display section. The upper row is the staff of the melody part, and the lower row is the staff of the accompaniment part. Each one-stage display switches to the staff at the next playback position as the playback position progresses.
73 to 75 are grid lines that delimit reproduction positions, 73 corresponds to a bar line, and 75 corresponds to a beat line. A cursor 76 displays the current playback position. Move to the right as time progresses.
77 is an icon for displaying a screen for selecting one of a plurality of types of music data (song number) in order to change the music data. Initially, default music data is selected. Not only the music data attached to the music picture book but also general music data can be used.
Music data can be edited with this score display module. Reference numeral 78 denotes an icon for storing the changed music data.
79 is an icon for displaying a selection screen for setting the tone color of the melody part. The tone color of the melody part corresponds to the characters 53 to 56 shown in FIG. A channel assigned to the melody part may be designated. A plurality of parts may be selected, and each may be associated with a plurality of characters and displayed as an animation.
[0036]
80 and 81 are icons for changing the tempo. 82 to 86 are sequence operation icons for controlling the reproduction position. Reference numeral 82 denotes a reproduction icon, which is used to start reading performance data from the memory storing the music data, or to resume the progress of the time paused by the next pause icon 83. Reference numeral 85 denotes a reproduction end icon. Reference numeral 86 denotes an icon for returning or advancing the current reproduction position in units of measures.
The sequence of music playback is controlled by the above-described icon groups 82 to 86 for sequence operation, and the display position of the character shown in FIG. 7 corresponding to the icon indicates a continuous movement or instantaneously moves in units of measures. Or
[0037]
9 and 10 are first and second flowcharts for explaining the operation of the embodiment of the present invention.
The overall application shown in FIG. 9A is a flowchart for starting and ending the processing of an electronic music picture book.
For example, it can be performed by the conversion module 12 of FIG.
In this case, the conversion module 12 also performs the process shown in FIG. That is, each time a processing request signal is received from the sequencer module 13 or the animation display module 11 in S101, the mutual conversion processing in S102 is executed.
[0038]
FIG. 10A is a flowchart for explaining processing of the animation display module 11.
In S113, the operation is controlled by the overall control application shown in FIG.
A user operation is detected in S114, the animation display screen is changed in S115, and the conversion module 12 is accessed to transfer the changed operation and / or operation amount to the sequencer module 13 side.
In S117, the animation display screen 51 is changed so that the information transferred from the sequencer module 13 side via the conversion module 12 is reflected.
FIG. 10B is a flowchart for explaining the processing of the sequencer module 13.
In S125 and S127, processing in the direction opposite to that shown in FIG.
That is, in S125, the music playback data and the score display are changed according to the operation on the staff display screen, and according to the received control information in S127. At that time, it is also possible to change the music data stored in the memory by regarding S124 and S126 as a kind of editing process.
[0039]
In the above description, the animation display module 11, the conversion module 12, the sequencer module 13, and the sound source module 14 shown in FIG. 3 are all executed on the personal computer 2 side. However, since the processing of the conversion module 12, the animation display module 11, and the sequencer module 13 shown in FIGS. 9 and 10 can be executed by independent programs, they can be executed individually by separate hardware devices, as a whole. It is possible to realize the same function.
For example, the animation display module 11 and the conversion module 12 can be placed on the personal computer 2 side, and the sequencer module 13 and the sound source module can be placed on the electronic music apparatus 1 side.
The electronic music picture book is recorded on a recording medium such as a CD-ROM or a semiconductor memory, and can be loaded into the personal computer 2 or the electronic music apparatus 1 for use.
[0040]
All the modules shown in FIG. 3 (the score display module 13a is not essential) can be provided to a communication terminal such as a game machine or a mobile phone terminal, a PDA (Personal Digital Assistant), a home appliance, or the like. This can be realized by causing the CPU built into the device to execute the control program.
In addition to simply downloading the contents of the electronic music picture book, it is also possible to execute a program that realizes the function of each module as a Java (TM) applet from the web server 4 so as to be executed on the web browser.
[0041]
【The invention's effect】
As is apparent from the above description, the present invention has an effect that the enjoyment is increased and the music reproduction can be actively and actively controlled.
There is an effect that a user who does not have a high level of comprehension experiences the interaction between music reproduction and moving image display, thereby cultivating a musical sense.
[Brief description of the drawings]
FIG. 1 is a system diagram illustrating an overall configuration of an embodiment of the present invention.
FIG. 2 is an explanatory diagram showing a file configuration example of an electronic music picture book used in the embodiment shown in FIG. 1;
3 is an explanatory diagram showing an example of an electronic music picture book use system executed on the electronic music apparatus and personal computer side of FIG. 1; FIG.
4 is an explanatory diagram of a correspondence relationship between musical tone control information and display objects in the embodiment shown in FIG. 3; FIG.
5 is an explanatory diagram of a correspondence relationship between a moving route of a character display position and a music playback position in the embodiment shown in FIG. 3; FIG.
FIG. 6 is a hardware configuration diagram for realizing an embodiment of the present invention.
FIG. 7 is an explanatory diagram illustrating an example of an animation display screen according to the embodiment of the present invention.
FIG. 8 is an explanatory diagram showing an example of a score display screen according to an embodiment of the present invention.
FIG. 9 is a first flowchart for explaining the operation of the exemplary embodiment of the present invention.
FIG. 10 is a second flowchart for explaining the operation of the exemplary embodiment of the present invention.
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 1 ... Electronic music apparatus, 2 ... Personal computer, 3 ... Communication network, 4 ... Web server, 11 ... Animation display module, 12 ... Conversion module, 13 ... Sequencer module, 13a ... Music score display module, 14 ... Sound source module, 51 ... Animation display screen, 71 ... Music score display screen

Claims (5)

A music playback and video display control device that displays music on a display screen in association with music playback that uses music data and animation data and plays back the music data as time progresses,
The music data are those provided with a performance sequence which time information indicating the time of a performance event that indicates the playback position of the performance event and the performance event is described as a pair,
The animation data includes a display sequence in which the display position of the display event for displaying the display object and the time information indicating the time of the display event are described as a pair, so that the display position of the display object is set in advance. Written to move along a given route,
In the music playback, at least one control information among note length information, note pitch information, note volume information, and musical tone attribute information that does not change for each note, and a display mode of the display object in the moving image display Correspondingly, setting means for setting the other side according to the setting on one side,
Display position progression control means for advancing the display position of the display object whose display mode is set by the setting means in accordance with the description of the animation data, corresponding to the current reproduction position of the music reproduction that changes with time progress; ,
When the display position of the display object is moved to a display position corresponding to a playback position different from the current playback position of the music playback by a user operation, a playback position that continues the time progress from the different playback position Progress control means,
A device for controlling music playback and moving image display, comprising: The display mode of the display object that the setting means associates with the musical sound attribute information is the type of image representing the character.
The music playback and moving image display control device according to claim 1. In the setting means, the display mode of the display object associated with the note length information is a cycle of the repeated action of the character, the display mode of the display object associated with the note pitch information is the color of the character, and the note volume information The display mode of the display object associated with is the amplitude of the repetitive motion of the character or the size of the character.
The music playback and moving image display control device according to claim 1. Creation of musical score display data for displaying a musical score on the display screen in accordance with performance data in which the at least one control information set by the setting means is reflected in the performance data and information on a reproduction position of the performance data Musical score display data creation means,
The music playback and moving image display control apparatus according to claim 1, wherein A music playback and video display control program that displays music on a display screen in association with music playback that uses music data and animation data and plays back the music data as time progresses.
The music data are those provided with a performance sequence which time information indicating the time of a performance event that indicates the playback position of the performance event and the performance event is described as a pair,
The animation data includes a display sequence in which the display position of the display event for displaying the display object and the time information indicating the time of the display event are described as a pair, so that the display position of the display object is set in advance. Is set in advance to move along a predetermined route,
In the music playback, at least one control information among note length information, note pitch information, note volume information, and musical tone attribute information that does not change for each note, and a display mode of the display object in the moving image display Correspondingly, a setting step for setting the other side according to the setting on one side,
A display position progress control step of causing the display position of the display object, whose display mode is set by the setting means, to advance according to the description of the animation data, corresponding to the current playback position of the music playback that changes with time progress; ,
When the display position of the display object is moved to a display position corresponding to a playback position different from the current playback position of the music playback by a user operation, a playback position that continues the time progress from the different playback position Progress control step,
A program for playing music and displaying moving images, characterized by causing a computer to execute.

Images