JP2013224999A - Program for implementing musical image display method and musical image display device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To implement a musical image display method which is capable of meeting user's request to enjoy variation of a plurality of visual effects during playback or performance of a musical piece and maker's and creator's requests to give users pleasure of the variation.SOLUTION: When a user gives an instruction to start visual effect display and editing processing, a display layer of a first part and a display layer of a second part are mixed in accordance with current settings, and the result is displayed on a touch panel of a musical image display device. Next, when a switch for designating, for example, a section A on an electronic instrument connected to the musical image display device is depressed, a message assigned to this switch is generated and is transmitted to the musical image display device. The musical image display device changes a pointer to visual effects to a point associated with the section A, generates image data on the basis of a single piece of visual effect data indicated by the pointer and stores the image data in the display layer of the first part.

Description

  The present invention realizes a music image display method for storing a plurality of video effects and performing screen display according to an input performance instruction based on any one of the video effects selected from the plurality of video effects. And a music image display apparatus.

  2. Description of the Related Art Music image display devices that store a plurality of video effects and perform screen display according to an input performance instruction based on one of the video effects selected from the plurality of video effects are conventionally known. Yes.

As such a music image display device, there is one configured as follows (for example, see Patent Document 1). That is,
(A) storing a drawing pattern database for melody, a drawing pattern database for harmony, and a drawing pattern database for rhythm for storing a plurality of drawing patterns respectively corresponding to each musical element (melody, harmony and rhythm) of the music;
(B) Select each drawing pattern for each musical element with the style selection switch,
(C) Enter a music event (performance information),
(D) Analyzing the input music event,
(E) If the music event is a melody event, refer to the melody drawing pattern database to generate image data of a drawing pattern corresponding to the music event;
(F) If the music event is a harmony event, refer to the harmony drawing pattern database to generate image data of a drawing pattern corresponding to the music event;
(G) If the music event is a rhythm event, the image data of the drawing pattern corresponding to the music event is generated with reference to the rhythm drawing pattern database;
(H) When two or more of the melody display switch, the harmony display switch, and the rhythm display switch are turned on, the display mode is set by the tile display switch, the overlay switch, or the time division display switch. Among the image data generated in the above (E) to (G), the image data of the musical element in the on state is displayed.

JP 2000-276137 A

  However, in the above-described conventional music image display device, the style selected by the style selection switch is the style of the music to be played or played back. Therefore, while the music is being played or played, the style is selected for each musical element. Image data of the same drawing pattern is generated and displayed, and it is not possible to switch to another drawing pattern during performance or playback, and to generate and display image data of the drawing pattern after the switching. As a result, while playing or playing music, we cannot respond to the user's request to enjoy changes in multiple video effects, or the manufacturer or creator's request to make the changes enjoyable for the user. It was.

  The present invention has been made paying attention to this point, and it is an object of the present invention to provide a program and a music image display device for realizing a music image display method capable of meeting the above-mentioned demands of users, manufacturers, and creators. And

  In order to achieve the above object, a program according to a first aspect of the present invention relates to a receiving procedure for receiving a performance instruction and a video effect set comprising a plurality of video effects stored in a storage means when a predetermined condition is satisfied. A selection procedure for selecting the one video effect, and a display procedure for causing the display means to display the video effect selected by the selection procedure by changing the display mode of the video effect according to the performance instruction received by the reception procedure. It is for making a computer perform.

  Here, “performance instruction” is not limited to “performance information”, but is a broad concept including “touch operation in response to a sound generation instruction” in step S7 of FIG. 5 described later.

  The program according to claim 2 further causes the computer to execute an acquisition procedure for acquiring designation information for designating one video effect to be selected from the video effect group in the program according to claim 1. In the selection procedure, a video effect designated by the designation information acquired by the acquisition unit is selected from the video effect group.

  The program according to claim 3 is characterized in that, in the program according to claim 1, in the selection procedure, when any one video effect is selected from the video effect group, the program is selected according to a predetermined order.

  In order to achieve the above object, the music image display device according to claim 4 can be realized by the same technical idea as that of claim 1.

  According to the invention described in claim 1 or 4, when a predetermined condition is satisfied, any one video effect is selected and selected from the video effect group composed of a plurality of video effects stored in the storage means. Since the display mode of the video effect is changed according to the received performance instruction and displayed on the display means, that is, the predetermined condition is established independently of the reproduction or performance of the music. During playback or performance, it is possible to respond to a user's request to enjoy a plurality of changes in the video effect and a manufacturer or creator's request to have the user enjoy the change. In addition, since a video effect set consisting of a plurality of video effects is prepared in advance and each video effect is switched in the set, the intention when the user, manufacturer and creator prepare the set, for example, Collect video effects with an image that matches the music or video effects with a sense of unity into a set, and switch each video effect within the set to accurately reflect the intention to enjoy or enjoy the change in the video effect. be able to.

It is a figure which shows an example of the system configuration | structure of a music system containing the music image display apparatus which concerns on one embodiment of this invention. It is a block diagram which shows schematic structure of the slate device in FIG. It is a figure for demonstrating the connection between the various data memorize | stored in the memory in FIG. It is a figure which shows an example of the video effect displayed on the touch panel in FIG. 3 is a flowchart showing a procedure of video effect display / editing processing executed by the slate device of FIG. 2, particularly a CPU. It is a flowchart which shows the detailed procedure of the control process according to the sound generation instruction | indication in FIG. It is a flowchart which shows the detailed procedure of the control process according to the performance information in FIG. It is a flowchart which shows the procedure of a timer interruption process.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a diagram showing an example of a system configuration of a music system including a music image display device according to an embodiment of the present invention.

  As shown in the figure, in the music image display device of the present embodiment, a general-purpose slate device 1 (CPU 10) having a touch panel 11 is used by a predetermined control program and application program (FIG. 5). This is realized by executing a video effect display / editing program described later. The slate device 1 is directly connected to an electronic keyboard instrument 2 and an external LCD (liquid crystal display) 3 via an adapter 4.

  The adapter 4 outputs the performance information input from the slate device 1 to the electronic keyboard instrument 2 and outputs the performance information input from the electronic keyboard instrument 2 to the slate device 1. The adapter 4 also outputs the audio signal input from the electronic keyboard instrument 2 to the slate device 1. Further, the adapter 4 outputs the video signal input from the slate device 1 to the external LCD 3. In the present embodiment, MIDI (musical instrument digital interface) data is used as the performance information format for convenience of explanation. Of course, other formats that do not adopt the MIDI format, for example, performance information such as OSC (Open Sound Control) may be used.

  In the illustrated example, the electronic keyboard instrument 2 includes a keyboard 2a composed of 24 keys, a plurality (four) of section specifying switches 2b for switching and specifying various sections ("A" to "D") of automatic accompaniment, and various settings. Are provided with a plurality of (six) switches 2c, a small LCD 2d for displaying various information, and a speaker 2e for generating musical sounds. The electronic keyboard instrument 2 has an automatic accompaniment function, and by using the automatic accompaniment function, for example, accompaniment sound is generated from the speaker 2e by reproducing automatic accompaniment data stored in a built-in memory (not shown). When the performer (user) presses the keyboard 2a according to the above, the key pressing sound is also generated from the speaker 2e. When the electronic keyboard instrument 2 generates the accompaniment sound and the key depression sound, the MIDI data corresponding to the accompaniment sound and the key depression sound (mainly a MIDI channel message, but a SysEx message (system exclusive message)). And audio signals are generated and output. The MIDI data and audio signal output from the electronic keyboard instrument 2 are transmitted to the slate device 1 via the adapter 4 as described above. When the user presses one of the section specifying switches 2b, the electronic keyboard instrument 2 displays MIDI data (this is one of “A” to “D”) assigned to the pressed switch (this data). In this case, a SysEx message is generated and output. Various parameters related to musical tone, such as tone, tempo, pitch bend, modulation, aftertouch, foot pedal (sustain pedal, soft tenuto pedal, soft pedal, etc.) operation status, filter coefficient (cutoff resonance, etc.), etc. Even when the value is changed, the electronic keyboard instrument 2 generates and outputs MIDI data including the type and value of the parameter in order to notify the changed value.

  The slate device 1 displays one type or a plurality of types (two types in the present embodiment) of video effects on the touch panel 11, and displays the accompaniment sound and key depression sound from the electronic keyboard instrument 2. The display is changed according to the contents of the corresponding MIDI data. At this time, the slate device 1 accepts a user's touch operation on the touch panel 11, generates a musical sound signal corresponding to the touch operation and generates a sound, and changes a display mode of the video effect on the touch panel 11 according to the touch operation. It is also configured to be displayed in a variable manner. Further, the slate device 1 generates and outputs a video signal corresponding to the video effect so that the video effect displayed on the touch panel 11 can also be displayed on the external LCD 3. The external LCD 3 inputs this video signal via the adapter 4 and displays it. As a result, a video effect similar to the video effect displayed on the touch panel 11 of the slate device 1 is displayed on the external LCD 3.

  In the present embodiment, the slate device 1, the electronic keyboard instrument 2, and the external LCD 3 are connected by wire via the adapter 4, but the present invention is not limited to this, and may be connected wirelessly. Moreover, both may be connected via a network 5 such as a local area network (LAN), the Internet, or a telephone line, instead of being directly connected one-to-one via the adapter 4.

  In the present embodiment, the electronic keyboard instrument 2 is externally connected to the slate device 1, and the generation of the accompaniment sound and the generation of the key-pressing sound (performance sound) are left to the electronic keyboard instrument 2. However, the present invention is not limited to this. The function of the electronic keyboard instrument 2 may be realized in the slate device 1 by software, and the function performed by the slate device 1 in cooperation with the electronic keyboard instrument 2 may be performed by the slate device 1 alone.

  FIG. 2 is a block diagram showing a schematic configuration of the slate device 1.

  As shown in the figure, the slate device 1 includes a CPU (central processing unit) 10 that controls the entire device. A touch panel 11, an operator 12, various sensors 13, a memory 14, a timer 15, an adapter interface (I / F) 16, a communication interface (I / F) 17, and a sound system 18 are connected to the CPU 10.

  The operation element 12 is not a software button displayed on the touch panel 11 but a hardware switch. Typical examples thereof include a power switch 12a (see FIG. 1) and a reset switch (not shown).

  The various sensors 13 include a microphone, a geomagnetic sensor, a GPS (Global Positioning System), an acceleration sensor that can detect gravitational acceleration, an ultrasonic sensor, a camera, an optical sensor that detects infrared light or visible light, and the like.

  The memory 14 is a ROM (read only memory) that stores a control program executed by the CPU 10 and various application programs, various data including image data used for screen display by each program and sound data used for sound generation, and the like. And RAM (random access memory) for temporarily storing various data including the image data and audio data, various input information, calculation results, and the like. In the present embodiment, the above control program, various application programs, various data, and the like are appropriately rewritable and non-volatile as ROM in order to be able to be added, deleted or updated as appropriate. Adopted.

  The timer 15 measures an interrupt time and various times in a timer interrupt process to be described later with reference to FIG.

  The adapter I / F 16 connects the adapter 4 to the slate device 1 and transmits / receives various data to / from an external device connected to the adapter 4.

  The communication I / F 17 connects the slate device 1 to the network 5 (see FIG. 1) and transmits / receives various data to / from other devices connected to the network 5. When a server computer (not shown) is connected as another device, addition, deletion, or updating of the above-described program or the like may be performed via the server computer. In this case, the communication I / F 17 is used to download various programs and various data from the server computer. The slate device 1 serving as a client transmits a command requesting download of a program or data to the server computer via the communication I / F 17 and the network 5. Upon receiving this command, the server computer distributes the requested program and data to the slate device 1 via the network 5, and the slate device 1 receives the program and data via the communication I / F 17. Downloading is completed by accumulating in the flash memory.

  The sound system 18 includes, for example, a DAC (digital-to-analog converter), an amplifier, a speaker, and the like. The sound system 18 converts a musical sound signal generated when the CPU 10 is executing the control program and various application programs into sound. In the present embodiment, the musical tone signal is generated by a so-called software sound source that is generated by the CPU 10 executing a sound source program. Of course, the present invention is not limited to a mode in which a musical sound signal is generated using a software sound source, and a hardware sound source may be used, or a sound source in which a part is configured by hardware and the rest is configured by software may be used.

  In this embodiment, the slate device 1 is adopted as the music image display device. However, the present invention is not limited to this, and a normal PC (personal computer) including a pointing device (such as a mouse) and a display device may be adopted. In addition, the present invention is not limited to a general-purpose device, and the present invention may be constructed on a dedicated device that includes only minimum elements that can implement the present invention.

  FIG. 3 is a diagram for explaining the connection between various data stored in the memory 14.

  As shown in the figure, the flash memory of the memory 14 stores a large number of single image effect data SEm (m = 1, 2, 3,...) That is the basis of the image effect displayed on the touch panel 11. Yes. Each piece of the video effect data SEm is composed of, for example, an execution code executed by the CPU 10 and data representing an image component used when the execution code is executed. This single video effect data SEm is generally downloaded from the server computer and stored in the flash memory, but a part of the video effect data SEm is stored in the flash memory in advance when the slate device 1 is shipped from the factory. Otherwise, the user may download and add from the server computer as necessary.

  FIG. 4 is a diagram illustrating an example of a video effect displayed on the touch panel 11 based on the single video effect data SEm. FIGS. 9A to 9D correspond to the video effect data SE1 to SE4 in FIG.

  Returning to FIG. 3, the flash memory also stores a large number of video effect groups #n (n = 1, 2, 3,...) Each having a plurality of video effects (four in this embodiment). Yes. Each video effect group #n is composed of four pointers associated with sections A to D, respectively. Each pointer points to a video effect, but some pointers point to a set of pointers instead of the single video effect data SEm. This set of pointers is a mixed video effect MEk (k = 1, 2, 3,...).

  The video effect group #n and the video effect MEk may be downloaded from the server computer and stored in the flash memory in the same manner as the single video effect data SEm, or some of them are stored in advance at the time of shipment from the factory. If necessary, it may be downloaded from a server computer and added. Also, the video effect group #n and the video effect MEk can be created or changed according to the user's editing operation and stored in the flash memory when an editing mode described later with reference to FIG. 5 is selected.

  When the user selects any one of the video effect groups #n stored in the flash memory, a pointer indicating the selected video effect group, that is, the current video effect group (hereinafter, “current video effect group”). Is read from the flash memory and stored in the work area of the RAM of the memory 14. In the illustrated example, the video effect group # 1 is selected, and a pointer to the video effect group # 1 is stored in the RAM as a pointer to the current video effect group.

  When any one video effect group #n is selected and a pointer to the current video effect group is stored in the RAM, any one of the four video effects (pointer to) belonging to the current video effect group is stored. One is always selected. When the pointer to the current video effect group is stored from a state that is not stored in the RAM, or when another video effect group #n is selected and the pointer to the current video effect group is changed, the current video effect group is changed. Which one of the four video effects belonging to the set (pointer to) is selected, may be determined by default, or the designated one is selected after waiting for the user's instruction. You may do it. The pointer indicating the selected video effect is hereinafter referred to as “pointer to the current video effect”. In the illustrated example, as the pointer to the current video effect, the pointer corresponding to section B is selected from the four video effect pointers belonging to the pointer to the current video effect group.

  Thus, in the illustrated example, the video effect group # 1 is selected from the video effect groups #n stored in the flash memory, and the video effect associated with the section B is further selected from the video effect group # 1. Yes. By this selection, a pointer to the current video effect and a pointer to the current video effect group are stored in the RAM.

  When the user instructs display of the video effect on the touch panel 11 in this storage state in the memory 14, the pointer to the current video effect indicates the video effect ME1 to be mixed. The single video effect data SE2 and SE4 indicated by the pointers to the video effects respectively associated with the first part (for example, melody part) and the second part (for example, accompaniment part) are stored in the flash memory. Read out. The read video effect data SE2 and SE4 are stored in the work area of the RAM. Then, the execution code included in each of the video effect data SE2 and SE4 is executed by the CPU 10, and each image data for displaying the video effects of the first part and the second part is generated. The generated image data of the first part is stored in an area (hereinafter referred to as “display layer of the first part”) provided in the work area of the RAM, and the generated image data of the second part is stored in the work area. Is stored in an area (hereinafter referred to as “second part display layer”).

  Note that the pointer to the current video effect may point to one single video effect data SEk as described above. In the illustrated example, the video effect associated with the section A of the video effect group # 1 corresponds to this case. In this case, when the user instructs display of the video effect on the touch panel 11, the single video effect data SE2 is read from the flash memory and stored in the work area of the RAM. Then, the execution code included in the video effect data SE2 is executed by the CPU 10, and the image data of the first part is generated and stored in the display layer of the first part.

  The single video effect data SE2 is indicated by pointers to two video effects respectively associated with the sections A and B of the video effect group # 1. In other words, depending on the single video effect data SEk, there are cases where a plurality of pointers indicate the same.

  The control process executed by the slate device 1 configured as described above will be described first with reference to FIGS. 3 and 4 and then in detail with reference to FIGS.

  When the user gives an instruction to start a video effect display / editing process (to be described later with reference to FIG. 5) in the storage state in the memory 14 of FIG. 3, the display layer of the first part and the display layer of the second part are displayed. The calculation is performed according to the current mixing setting, and the mixing result is displayed on the touch panel 11. Specifically, the video effect in FIG. 4B and the video effect in FIG. 4D are superimposed and displayed on the touch panel 11. However, since the video effect example in FIG. 4 shows a video effect whose display mode is changing, the video effect displayed on the touch panel 11 immediately after the video effect display / editing process is started is slightly different. Is different.

  When MIDI data is supplied from the electronic keyboard instrument 2 in the display state of the touch panel 11 or when the user performs a touch operation for sound generation on the touch panel 11, display of the video effect on the touch panel 11 is performed accordingly. The aspect is changed and displayed.

  Next, when the user presses, for example, a switch that designates section A among the section designation switches 2b, a SysEx message including the name of the section assigned to the switch, that is, “section A” is generated, and the electronic keyboard instrument 2 is generated. To the slate device 1. The slate device 1 changes the pointer to the current video effect to the pointer associated with the section A in the pointer to the current video effect group, and, as described above, the single video effect data SE2 pointed to by the pointer is changed. Read from the flash memory and store in the RAM work area. The slate device 1 executes the execution code included in the video effect data SE2, generates image data for displaying the video effect of the first part, and stores it in the display layer of the first part. Thereafter, the video effect on the touch panel 11 is displayed based only on the image data stored in the display layer of the first part, so that the video effect corresponding to FIG. 4D is not displayed.

  Further, when the user presses another section designation switch, the corresponding one or two single image effect data SEk is read out via the pointer associated with the section assigned to the section designation switch. Based on this, the video effect is switched and displayed on the touch panel 11.

  Note that the selection method for switching the current video effect within the current video effect group is not limited to the method in which the user switches to a specific one by pressing any one of the section designation switches 2b as described above. There are a method of switching selection in a predetermined order every time elapses of time, a method of switching selection according to the contents of MIDI data transmitted from the electronic keyboard instrument 2, etc. Since it will be described in detail later, it will not be further described here.

  As described above, in the present embodiment, during playback or performance of a song, one of the video effects is selected and switched from among a plurality of video effects as a trigger, It is possible to meet a user's request to enjoy a plurality of changes in the video effect and a manufacturer or creator's request to make the user enjoy the change. In addition, since a video effect set consisting of a plurality of video effects is prepared in advance and each video effect is switched in the set, the intention when the user, manufacturer and creator prepare the set, for example, Collect video effects with an image that matches the music or video effects with a sense of unity into a set, and switch each video effect within the set to accurately reflect the intention to enjoy or enjoy the change in the video effect. be able to.

  Next, this control process will be described in detail.

  FIG. 5 is a flowchart showing the procedure of the video effect display / editing process executed by the slate device 1, particularly the CPU 10.

  This video effect display / editing process is one of the application programs stored in the flash memory, and the user can, for example, in a menu image displayed on the touch panel 11 (see step S44 in FIG. 8 described later). It is activated by touching the corresponding menu.

When the video effect display / editing process is activated, the CPU 10 first performs an initial setting (step S1). Initial settings are mainly (1) clearing the RAM work area;
(2) storing a pointer to the current video effect group and a pointer to the current video effect in the RAM;
(3) One or two single video effect data SEm indicated by the pointer to the current video effect stored by the processing of (2) above is read from the flash memory, stored in the RAM, and the stored video effect data Generating image data based on SEm and storing the generated image data in the display layer;
(4) Mixing settings, association of performance information and parts, and update conditions are stored in RAM;
Each process is performed.

  As will be described later, the end process (step S16) executed when the present video effect display / editing process ends, the pointer to the current video effect group and the pointer to the current video effect are placed in a predetermined area of the flash memory. In the process (2), the CPU 10 reads the pointer to the current video effect group and the pointer to the current video effect stored in the predetermined area and stores them in the RAM. However, if this video effect display / editing process is started for the first time from a state where it has never been started, or the pointer to the current video effect group and the pointer to the current video effect are not saved in the above end processing Since there is no pointer to the current video effect group to be read out and no pointer to the current video effect in the first place, in that case, to the video effect associated with the default video effect group and the default section in the group. May be read out and stored in the RAM as a pointer to the current video effect group and a pointer to the current video effect.

  In the present embodiment, there are two types of “display layers” in the process (3) as described above: the first part display layer and the second part display layer. As described above, there are two types of pointers to the video effect for each section constituting the video effect group #n, indicating the single video effect data SEm and indicating the video effect MEk to be mixed. is there. Since the former pointer indicates only one piece of video effect data SEm, the “display layer” in this case is either the display layer of the first part or the display layer of the second part. Then, it means the display layer of the first part. Therefore, the image data generated based on the single video effect data SEm is stored in the display layer of the first part. On the other hand, the latter pointer points to two single image effect data SEm via the mixed video effect MEk. In this case, the “display layer” is the first part display layer and the second part display layer. Means both. Accordingly, the image data generated based on one single video effect data SEm indicated by the pointer associated with the first part among the two pointers constituting the video effect MEk to be mixed is displayed in the first part. Image data generated based on a single piece of video effect data SEm stored in the layer and pointed to by the pointer associated with the second part is stored in the display layer of the second part.

  As described above, the “mixing setting” in the process (4) is used for calculation when the display layer of the first part and the display layer of the second part are overlapped. Specific examples of the mixing setting include how to overlap (whether the display layer of the first part or the display layer of the second part is on top), the transmittance of the display layer to be overlaid, and the calculation method (AND, OR). , XOR, etc.).

  The “association of performance information and part” in the process (4) is the table data (hereinafter referred to as “performance information / part correspondence”) referred to in the “assignment from MIDI channel to part” process in step S31 of FIG. Table ") and MIDI channels are associated with each of the first part and the second part without duplication. However, all the MIDI channels do not always need to be associated with either the first part or the second part, and there may be a MIDI channel that is not associated with any part.

  The “update condition” in the process (4) is a condition for causing the CPU 10 to execute a process of changing the current video effect to another video effect in the current video effect group (see step S6 described later). The contents of the update condition will be described later and will not be described here any further.

  Since the “mixing setting”, the “performance information / part correspondence table”, and the “update condition” are all stored in a predetermined area of the flash memory by the end process, like the two pointers in the process (2). In the process (4), the CPU 10 reads out “mixing settings”, “performance information / part correspondence table” and “update conditions” stored in the predetermined area and stores them in the RAM. When “mixing setting”, “performance information / part correspondence table” and “update condition” are not saved, the default “mixing setting”, “performance information / part correspondence table” and “update condition” are stored in the RAM. Just remember.

  Next, when the user performs an operation of associating the performance information with the part, that is, changing the content of the “performance information / part correspondence table”, the CPU 10 performs the performance information / part correspondence in accordance with the operation. The table is changed (step S2). A typical example of this change operation is an operation for specifying a MIDI channel and adding or deleting a part to be changed. However, there are several performance information / part correspondence table candidates that can be changed, If any one of them is selected and specified, the selected and specified table may be replaced with the previous one.

  When the user performs an operation for changing the content of the mixing setting, the CPU 10 changes the mixing setting to the content corresponding to the operation (step S3). This change operation is different from the change operation in step S2 described above except that the change target is different, and the operation method may be the same, and will not be described further.

  Further, when the user performs an operation for changing the update condition, the CPU 10 changes the update condition to a condition corresponding to the operation (step S4). A typical example of this change operation is an operation in which several candidate update conditions that can be changed (in this embodiment, two types described later) are prepared, and the user selects and specifies one of them. If the update condition can be changed, the operation method is not limited to this.

Next, the CPU 10 determines whether or not the update condition is met (step S5). When the update condition is met, the current video effect is changed to another video effect in the current video effect group (step S6). Proceed to step S7. Here, in the present embodiment, the update condition and the method for changing the video effect are as follows:
(C1) Reception of section change instruction: Change to the video effect indicated by the pointer associated with the section instructed to change;
(C2) elapse of a predetermined time: change to the next video effect in a predetermined order;
There are two types, and the user can select either one (see step S4).

  The “section change instruction” in the update condition (C1) is performed when the electronic keyboard instrument 2 transmits to the slate device 1 the one in which any of the sections A to D is embedded in the SysEx message as described above. Is called. Therefore, when the update condition (C1) is selected, the CPU 10 determines whether the input MIDI data is a SysEx message, and if it is a SysEx message, any of the sections A to D is included therein. By determining whether it is embedded or not, it is determined whether the update condition is met. As a result of this determination, if the update condition is met, that is, if any of the sections A to D is embedded in the SysEx message, the CPU 10 sets a pointer to the current video effect to the current video effect group. The pointer is changed to a pointer to the video effect associated with the embedded section. Then, the CPU 10 reads out one or two single video effect data SEm indicated by the pointer to the current video effect after the change from the flash memory and stores it in the RAM, and the image data based on the stored video effect data SEm. And the generated image data is stored in the corresponding display layer.

The “predetermined time” in the update condition (C2) is counted by the timer 15. Therefore, when the update condition (C2) is selected, the CPU 10 causes the timer 15 to measure “predetermined time”. Assuming that 15 seconds is set as the “predetermined time”, for example, the CPU 10 determines whether or not the update condition is met by determining whether or not the timer 15 has counted 15 seconds. As a result of this determination, if the update condition is met, that is, if the timer 15 times 15 seconds, the CPU 10 sets the pointer to the current video effect to the next video effect in the pointer to the current video effect group. Change to a pointer. Here, “next” is “next” in “predetermined order”. Examples of “predetermined order” include
(O1) Section A → B → C → D → A;
(O2) Section D → C → B → A → D;
(O3) Section A → B → C → D → C → B → A;
(O4) Random;
And so on. However, in the above example, the “predetermined order” is only described by “section”, and the “section” in the above order is not actually selected, but the video effect associated with the “section” is selected. Is selected. Then, the CPU 10 reads out one or two single video effect data SEm indicated by the pointer to the current video effect after the change from the flash memory and stores it in the RAM, and the image data based on the stored video effect data SEm. And the generated image data is stored in the corresponding display layer. When the update condition (C2) is selected, the “predetermined time” and the “predetermined order” may be set freely by the user. Of course, the present invention is not limited to this, and the default “predetermined time” and “predetermined order” may be automatically set.

  On the other hand, if it is determined in step S5 that the update condition is not met, step S6 is skipped and the process proceeds to step S7.

  In step S <b> 7, the CPU 10 determines whether or not there has been a touch operation for sound generation instruction. If the operation mode has not shifted to the edit mode during the execution of the video effect display / editing process, the current video effect is displayed on the touch panel 11. When the user performs a touch operation on the current image effect on the touch panel 11, the CPU 10 detects the touch operation as a sound generation instruction. The determination in step S7 determines whether or not there has been a touch operation as a sound generation instruction.

  As a result of the determination in step S7, when there is a touch operation for a sound generation instruction, the CPU 10 executes a control process corresponding to the sound generation instruction (step S8), and then proceeds to step S9, but there is no sound operation instruction touch operation. Sometimes, the CPU 10 skips step S8 and proceeds to step S9.

  FIG. 6 is a flowchart showing a detailed procedure of the control process according to the sound generation instruction.

  In the figure, the CPU 10 determines the pitch of the musical sound to be generated from the touch position on the touch panel 11, determines the velocity of the musical sound from the touch intensity, and determines the duration of the musical sound from the duration of the touch. The sound source unit is instructed to generate the tone, velocity, and duration of the musical tone (step S21). Here, the “touch strength” that is the basis for determining the velocity is, for example, an acceleration sensor corresponding to an output from an acceleration sensor included in the various sensors 13, that is, vibration generated when the user touches the touch panel 11. Is calculated based on the signal output from In addition, since the sound source unit is configured by a software sound source in the present embodiment, as described above, the CPU 10 delivers the determined pitch, velocity, and duration to the sound source unit. In response to this, the tone generator unit generates a tone signal instructed to be generated and outputs the generated tone signal to the sound system 18.

  Then, the CPU 10 determines the video mode of the current video effect to be changed in accordance with the sound generation instruction, reflects the determined video mode on the display layer of the first part (step S22), and then ends this control process. . The “determine the video mode of the current video effect” in step S22 is executed by the CPU 10 using the execution code included in the single video effect data SEm of the first part stored in the RAM as a user touch operation on the touch panel 11. It is performed by executing based on the corresponding touch information (touch position, touch intensity, touch duration, etc.). As the current video effect, for example, it is assumed that the video effect in FIG. 4C is displayed on the touch panel 11. However, since the video effect in FIG. 4C is in a state in which the video mode is changed, the current video effect is that in which the video mode is not changed, that is, three pillars do not protrude. X Only 5 holes are open. When the user touches the hole at the position of the column in a state where the current video effect is displayed on the touch panel 11, a column having a length corresponding to the touch strength protrudes from the hole. In the image effect example of FIG. 4C, these three pillars are changed image modes.

  Returning to FIG. 5, next, the CPU 10 determines whether or not performance information has been received (step S9). In the present embodiment, this performance information is MIDI data transmitted by the electronic keyboard instrument 2, particularly a MIDI channel message. That is, even if the MIDI data is the same, if the SysEx message is received, the CPU 10 does not determine that the performance information has been received.

  As a result of the determination in step S9, when a MIDI channel message is received, the CPU 10 executes a control process corresponding to the performance information (step S10), and then proceeds to step S11. On the other hand, when a MIDI channel message is not received, The CPU 10 skips step S10 and proceeds to step S11.

  FIG. 7 is a flowchart showing a detailed procedure of the control process according to the performance information.

  In the figure, the CPU 10 refers to the performance information / part correspondence table, and the MIDI channel of the received MIDI channel message is associated with the first part, the second part, or any one of them. It is determined whether it is also associated with a part, and the process is distributed according to the determination result (step S31). Specifically, if the MIDI channel message is associated with the first part, the CPU 10 proceeds to step S32, and if the MIDI channel message is associated with the second part. The CPU 10 advances the process to step S33, and if the MIDI channel message is not associated with any part, the CPU 10 ends the control process.

  In step S32, the CPU 10 extracts the pitch, velocity, and duration from the MIDI channel message, and determines the video mode of the current video effect of the first part to be changed according to the extracted pitch, velocity, and duration. Then, the determined video mode is reflected on the display layer of the first part.

  In step S33, the CPU 10 extracts the pitch, velocity, and duration from the MIDI channel message, and determines the video mode of the current video effect of the second part to be changed according to the extracted pitch, velocity, and duration. Then, the determined video mode is reflected on the display layer of the second part.

  Since each process of the said step S32 and S33 is substantially the same as the process of step S22 of the said FIG. 6, it does not explain any more.

  Returning to FIG. 5, next, the CPU 10 determines whether or not another video effect group #n is selected and instructed (step S11). Since the user can instruct selection of another video effect group #n that is not the current video effect group by touching the corresponding menu in the menu image displayed on the touch panel 11, the determination in step S11 is performed. Then, the CPU 10 determines whether or not there is a touch operation on the menu.

  As a result of the determination in step S11, when another video effect group #n is instructed, the CPU 10 reads out the selected video effect group #n from the flash memory and stores it in the RAM as the current video effect group. Then, another video effect group #n is selected (step S12). Further, the CPU 10 stores, in the RAM, the pointer associated with the default section among the pointers to the four video effects constituting the current video effect group as the current video effect. Further, the CPU 10 generates image data based on one or two single video effect data SEm by executing the same process as the process (3) based on the current video effect group and the current video effect. And stored in the corresponding display layer.

  On the other hand, if the result of determination in step S11 is that no other video effect group has been selected, the CPU 10 skips step S12 and proceeds to step S13.

  In step S13, the CPU 10 determines whether an edit mode has been instructed. The editing mode can be instructed by the user touching the corresponding menu in the menu image displayed on the touch panel 11. Therefore, in the determination of step S <b> 13, the CPU 10 determines whether the menu has been touched. Determine if.

  As a result of the determination in step S13, when the editing mode is instructed, the CPU 10 creates or changes a video effect group and saves it according to the user operation (step S14). Note that when creating or changing a video effect group, a video effect to be mixed may be specified, and therefore, in the edit mode, a video effect to be mixed can also be created or changed and saved.

  On the other hand, if the result of determination in step S13 is that editing mode has not been instructed, the CPU 10 skips step S14 and proceeds to step S15.

  In step S15, the CPU 10 determines whether termination has been instructed. Since the end instruction can be performed by the user touching the corresponding menu in the menu image displayed on the touch panel 11, the CPU 10 determines in step S15 whether the touch operation has been performed on the menu. Determine.

  If the end is instructed as a result of the determination in step S15, the CPU 10 executes an end process (step S16), and then ends the video effect display / edit process. In the end processing, the CPU 10 stores a pointer to the current video effect group, a pointer to the current video effect, mixing settings, performance information / part correspondence table, update conditions, and the like in a predetermined area of the flash memory.

  On the other hand, if the end is not instructed as a result of the determination in step S15, the CPU 10 returns the process to step S2 and repeats the processes in steps S2 to S15.

  FIG. 8 is a flowchart showing the procedure of timer interrupt processing. This timer interruption process is started every time the timer 15 measures 1/30 seconds, for example.

  In FIG. 8, first, the CPU 10 processes the contents of the display layer of the first part or the display layer of the second part according to the current video effect (step S41). “The display layer of the first part or the display layer of the second part” means that when the content to be processed is the first part, the content of the display layer of the first part is processed, and the content to be processed is the first When targeting two parts, it means that the content of the display layer of the second part is processed, and by chance both parts may not be processed or only one part may be processed. In addition, both parts may be processed. Further, since the process of step S41 is for generating a video effect in which the display mode changes with the passage of time, this step is performed when the display mode of the current video effect does not vary with time. The process of S41 is not executed. “According to the current picture effect” means this. FIG. 4A shows an example of a video effect displayed on the touch panel 11 when a MIDI channel message associated with the first part is transmitted from the electronic keyboard instrument 2. In the image effect of the illustrated example, for example, when a MIDI channel message having a note name “E” arrives, a circle with a note name “E” at the center of a radius corresponding to the velocity is displayed. Go down as time goes by. The process of step S41 generates the “drop” display mode by processing the contents of the display layer of the first part. Another example is the “ripple” shown in FIG. Of course, the display example is not limited to the above example as long as the display mode varies with the passage of time.

  Next, the CPU 10 calculates the display layer of the first part and the display layer of the second part according to the mixing setting (step S42). That is, between the display layer of the first part and the display layer of the second part prepared by the processing up to step S41, the overlapping method, transmittance, and calculation method (AND, OR, XOR, etc.) are performed.

  Then, the CPU 10 outputs the calculation result for display on the external LCD 3 (step S43), further superimposes the menu image on the calculation result, and outputs it for display on the touch panel 11 (step S44).

  In this embodiment, the distribution to the first part or the second part is performed based on the MIDI channel. However, the present invention is not limited to this, and based on a combination of ports of a plurality of MIDI ports or ports and channels. May be performed.

  Further, in the present embodiment, the “performance information” in step S9 of FIG. 5 is limited to “MIDI channel message”, but this is merely for the sake of simplicity, and is included in the “SysEx message”. May also be considered as “performance information”. For example, in a SysEx message, channel information is provided in a unique format so that it can be handled in the same way as a MIDI channel message, or a common message or a real-time message (included in the SysEx message) is also communicated according to an operation during performance. Thus, it may be handled in the same manner as the “performance information”. Furthermore, common messages and real-time messages may be handled as selection instruction information.

  In this embodiment, the number of pointers in the video effect group #n, the number of pointers in the video effect MEk to be mixed, and the number of parts of the video effect displayed simultaneously are “4”, “2”, and “ Needless to say, the number is not limited to this as long as it is 2 ″.

  In the present embodiment, the electronic keyboard instrument 2 has a keyboard 2a with 24 keys. However, the number of keys on the keyboard is not limited to this, and 49, 61, 76, 88, etc. The number of keys may be selected, or an electronic keyboard instrument having various numbers of keys may be arbitrarily selected and connected. In the latter case, for example, when a 49-key electronic keyboard instrument is connected, even if the video effect is properly displayed on the entire screen of the touch panel 11 while changing the display mode, the connection is not possible. For example, when the electronic keyboard instrument is changed to 88 keys, a part of the display mode in which the video effect is changed may protrude from the screen of the touch panel 11 and may not be displayed. In order to cope with this problem, the video effect may be adjusted (such as enlargement or reduction of the display range) according to the number of keys of the connected electronic keyboard instrument. At this time, the slate device 1 needs to acquire the number of keys of the connected electronic keyboard instrument. As the acquisition method, a method for allowing the user to input the number of keys or the slate device 1 electronically transmits MIDI data or the like. There is a method of inquiring the model of the keyboard instrument and determining the number of keys from the answered model.

  Furthermore, in the present embodiment, the display mode of the video effect is that the MIDI channel message (including pitch, velocity, and duration) transmitted from the electronic keyboard instrument 2 and the touch of the user's pronunciation instruction on the touch panel 11 are displayed. In addition to this, other MIDI channel messages, specifically various parameters (tone, tempo, pitch bend, modulation, aftertouch, foot pedal (sustain pedal, soft tenute pedal) And a soft channel pedal (such as a soft pedal), a MIDI channel message including a filter coefficient (cutoff resonance, etc.), and various sensor outputs from the various sensors 13.

In the present embodiment, (C1) and (C2) have been described as the update condition and the method for changing the video effect.
(C11) Various sensor outputs: determine a section and change to a video effect indicated by a pointer associated with the determined section;
(C12) Receive predetermined MIDI data: determine a section and change to a video effect indicated by a pointer associated with the determined section;
(C21) Various sensor outputs: Change to the next video effect in a predetermined order;
(C22) Receive predetermined MIDI data: Change to the next video effect in a predetermined order;
May be added.

As a specific example of the above (C11),
Example 1: A threshold value and a section are determined for each sensor, and when a sensor exceeds the threshold value, the video effect is changed to a pointer indicated by the section.
Example 2: An image from a camera is imaged, and when a person performs a predetermined pose or motion, a predetermined section is determined and changed to a video effect indicated by a pointer associated with this section;
Can be mentioned.

As a specific example of the above (C12),
Example 1: When MIDI data for timbre switching or accompaniment switching (not section switching) is received, the category to which the “timbre” or “accompaniment” belongs after switching is determined, the section corresponding to the category is determined, and this section Change to the video effect pointed to by the associated pointer;
Can be mentioned.

As a specific example of the above (C21),
Example 1: A threshold value is determined for each sensor, and when a certain sensor exceeds the threshold value, it is changed to the next video effect in a predetermined order;
Example 2: Recognizing an image from a camera and changing to the next video effect in a predetermined order when a person performs a predetermined pose or motion;
Can be mentioned.

As a specific example of the above (C22),
Example 1: Each time MIDI data for timbre switching or accompaniment switching (not section switching) is received, it is changed to the next video effect in a predetermined order;
Can be mentioned.

  Further, the update condition may be determined by a combination of various sensor outputs and MIDI data, or the update condition may be determined by the audio signal output from the electronic keyboard instrument 2. When this update condition is satisfied, the section is determined and the video effect indicated by the pointer associated with the determined section is changed, or the video effect is changed to the next video effect in a predetermined order. . When a section is determined by an audio signal, information for designating the section may be embedded in the audio signal or watermark. The audio signal input to the slate device 1 is recorded as a moving image together with a video effect displayed on the touch panel 11. Further, the characteristics of the audio signal may be used to change the display mode of the video effect.

  A program in which a storage medium storing software program codes for realizing the functions of the above-described embodiments is supplied to a system or apparatus, and a computer (or CPU or MPU) of the system or apparatus is stored in the storage medium. It goes without saying that the object of the present invention can also be achieved by reading and executing the code.

  In this case, the program code itself read from the storage medium realizes the novel function of the present invention, and the program code and the storage medium storing the program code constitute the present invention.

  As a storage medium for supplying the program code, for example, a flexible disk, hard disk, magneto-optical disk, CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD + RW, magnetic A tape, a non-volatile memory card, a ROM, or the like can be used. Further, the program code may be supplied from the server computer via the network 5.

  Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also the OS running on the computer based on the instruction of the program code performs the actual processing. It goes without saying that a case where the functions of the above-described embodiment are realized by performing part or all of the above and the processing thereof is included.

  Further, after the program code read from the storage medium is written into a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer, the function expansion is performed based on the instruction of the program code. It goes without saying that the CPU or the like provided in the board or the function expansion unit performs part or all of the actual processing and the functions of the above-described embodiments are realized by the processing.

DESCRIPTION OF SYMBOLS 10 ... CPU (receiving means, selection means, control means), 11 ... Touch panel (receiving means, display means), 14 ... Memory (storage means), 16 ... Adapter I / F (receiving means)

Claims (4)

A receiving procedure for receiving performance instructions;
A selection procedure for selecting any one video effect from a video effect group consisting of a plurality of video effects stored in the storage means when a predetermined condition is satisfied;
A program for causing a computer to execute a display procedure for causing a display unit to display a display mode of a video effect selected by the selection procedure in accordance with a performance instruction received by the reception procedure. A program for causing a computer to further execute an acquisition procedure for acquiring designation information for designating one video effect to be selected from the video effect group,
The program according to claim 1, wherein in the selection procedure, a video effect designated by the designation information obtained by the obtaining unit is selected from the video effect group.   2. The program according to claim 1, wherein, in the selection procedure, when selecting any one of the video effects from the video effect group, the selection is performed according to a predetermined order. A receiving means for receiving performance instructions;
Storage means for storing a video effect group comprising a plurality of video effects;
Selecting means for selecting any one of the video effects from the video effect set stored in the storage means when a predetermined condition is satisfied;
A music image comprising: control means for controlling the display mode of the video effect selected by the selection means to be changed in accordance with a performance instruction received by the receiving means and to be displayed on the display means. Display device.

Images