JP2010008893A - Electronic music device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an electronic music device capable of recording only a performance scene. <P>SOLUTION: An image taking device 20 is installed as described in (a). A power switch of the image taking device 20 is turned on, and when a user presses a recording mode selection switch in a state that a moving image data are always supplied from the image taking device 20 to an electronic keyboard instrument, the electronic keyboard instrument changes its operation mode to a recording mode to bring a record stand-by state. Image recognition is performed whether or not the typical conductor action of 4 beats as in (c) (gesture for record start indication) performed by user's own hands is included in the moving image data. When the gesture for record start indication is recognized, it is determined that current action is positioned in which position from "1" to "4" like (c). When it is determined that it is positioned at the position "1" at a second time, a record start command is generated and outputted. According this, the image recording of the moving image data supplied from the image taking device 20 is started. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an electronic music apparatus capable of recording a performance scene of a user.

  2. Description of the Related Art Conventionally, an electronic music apparatus capable of recording a user's performance scenery has been known.

As such an electronic music apparatus, there is one that records a performance scene of a user, various performance sounds, and the like on a video tape attached to the video camera in accordance with an instruction input via an operation unit of the video camera. (For example, refer to Patent Document 1).
JP 2003-208164 A

  However, in the above-described conventional electronic music apparatus, the instruction to start or stop the recording is made via the operation unit of the video camera, that is, the operation element (instruction for instructing the start / stop provided in the operation unit ( For example, the button is operated by the user, so not only the performance scenery but also unnecessary user actions before and after that, specifically, the recording start button is operated and then the performance is actually started. The user's actions until the start or the user's actions from the end of the performance to the operation of the recording stop button were also recorded. Of course, unnecessary recording parts can be cut by editing after recording, but the editing is troublesome.

  The present invention has been made paying attention to this point, and an object thereof is to provide an electronic music apparatus capable of recording only a performance scene.

  In order to achieve the above object, an electronic music apparatus according to claim 1 is provided with supply means for supplying moving image data obtained by photographing a scene including a user, a performance operator for generating performance information, and a user. Generating means for generating a musical sound signal based on performance information generated by operating the performance operator, recognition means for recognizing a user's predetermined gesture from moving image data supplied by the supply means, Based on the recording / recording means for recording the supplied moving image data and recording the musical sound signal generated by the generating means and / or the generated performance information, and the gesture recognized by the recognition means. And control means for controlling the recording / recording state of the recording / recording means.

  The electronic music apparatus according to claim 2 is the electronic music apparatus according to claim 1, wherein automatic performance means for performing automatic performance based on automatic performance data and the tempo of the automatic performance based on the recognized gesture are provided. Setting means for setting, wherein the generation means generates a musical tone signal even based on an automatic performance by the automatic performance means.

  According to the first aspect of the present invention, a user's predetermined gesture is recognized from the supplied moving image data, and the recording / recording state of the recording / recording means is controlled based on the recognized gesture. Therefore, it is possible to prevent a bad-looking video unrelated to the scene to be recorded such as an operation of operating a button to start or stop recording of the scene including the user from being recorded.

  According to the second aspect of the present invention, since the tempo can be set based on the recognized gesture, the tempo can be set at the start of recording / recording without setting the tempo in advance. .

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a block diagram showing a schematic configuration of an electronic keyboard instrument to which an electronic music apparatus according to an embodiment of the present invention is applied.

  As shown in the figure, the electronic keyboard instrument of the present embodiment includes a performance operator 1 including a keyboard for inputting performance information including pitch information, and a plurality of switches and dials for inputting various information. , A setting operator 2 including a joystick, a detection circuit 3 for detecting an operation state of the performance operator 1, a detection circuit 4 for detecting an operation state of the setting operator 2, a CPU 5 for controlling the entire apparatus, A ROM 6 for storing a control program executed by the CPU 5 and various table data, a RAM 7 for temporarily storing performance information, various input information, calculation results, and the like, and displays various information such as a liquid crystal display (LCD) and Display device 8 having a light emitting diode (LED) and the like, various application programs including the control program, various music data, various data, etc. A storage device 9 is connected to an imaging device 20 that captures a performance scene of a user, and a communication interface (I / F) 10 that transmits / receives data to / from the imaging device 20 and the performance operator 1 are input. A sound source / effect circuit for converting performance information obtained by reproducing any piece of music data stored in the storage device 9 into a musical sound signal, and applying various effects to the musical sound signal 11 and a sound system 12 such as a DAC (Digital-to-Analog Converter), an amplifier, and a speaker for converting a musical sound signal from the sound source / effect circuit 11 into sound.

  The components 3 to 11 are connected to each other via the bus 13, the imaging device 20 is connected to the communication I / F 10, and the sound system 12 is connected to the sound source / effect circuit 11.

  The storage device 9 is, for example, a storage medium such as a flexible disk (FD), a hard disk (HD), a CD-ROM, a DVD (digital multipurpose disk), a magneto-optical disk (MO), and a semiconductor memory, and its driving device. The storage medium may be detachable from the drive device, or the storage device 9 itself may be detachable from the electronic keyboard instrument. Alternatively, neither the storage medium nor the storage device 9 may be detachable. Note that the storage device 9 (the storage medium) can also store the control program executed by the CPU 5 as described above, and if the control program is not stored in the ROM 6, the control program is stored in the storage device 9. By reading it into the RAM 7, it is possible to cause the CPU 5 to perform the same operation as when the control program is stored in the ROM 6. In this way, control programs can be easily added and upgraded.

  As the communication I / F 10, for example, a music dedicated wired I / F that exclusively transmits and receives a music signal such as a MIDI signal, a general-purpose short-distance wired I such as USB (Universal Serial Bus) and IEEE 1394 (I Triple 1394). / F, general-purpose network I / F such as Ethernet (registered trademark), and general-purpose short-range wireless I / F such as wireless local area network (LAN) and Bluetooth (registered trademark). In this embodiment, the USB is adopted as the communication I / F 10, but other types of I / Fs may be used instead, and other types of I / Fs are added thereto. You may do it.

  The imaging device 20 is a camera such as a Web camera, a digital video camera, or a digital still camera that supports, for example, a USB video class. Motion image data obtained by shooting is motion-JPEG, DV ( Digital Video), converted into compressed data such as MPEG-2 or MPEG-4, or output as uncompressed data. The communication I / F 10 is provided with a USB host terminal and a USB video class driver. The communication I / F 10 transmits the above-mentioned various types of moving image data output from the imaging device 20 via USB communication. Receive according to protocol. Note that the communication I / F 10 is not limited to one that can receive moving image data in all of the above-described various formats, and may be any device that can receive moving image data in at least one of the formats. In addition, the communication protocol for moving image data is not limited to that specified in the USB video class. When a communication protocol other than the communication protocol defined in the USB video class is adopted, the moving image data communicated according to this communication protocol is not limited to the above-described various formats. When the electronic keyboard instrument side is configured to control the shooting direction, zoom magnification, and the like of the imaging device 20, the communication I / F 10 transmits the control information to the imaging device 20 according to the USB communication protocol. In the present embodiment, an externally connected device via the communication I / F 10 is employed as the imaging device 20, but the present invention is not limited to this, and a device incorporated in an electronic keyboard instrument may be employed.

  In this embodiment, the electronic music apparatus is constructed on the electronic keyboard instrument. However, the present invention is not limited to this, and the electronic music apparatus may be constructed on a general-purpose PC externally connected to the keyboard. Further, other musical instrument forms such as a string instrument type, a wind instrument type, and a percussion instrument type may be employed instead of the keyboard instrument form.

  FIG. 2 is a block diagram showing a functional configuration of the electronic keyboard instrument of the present embodiment.

  In the figure, an imaging unit 101 mainly corresponds to the imaging device 20, the communication I / F 10 and the CPU 5, is shot using the imaging device 20, and is supplied into the electronic keyboard instrument via the communication I / F 10. The moving image data is supplied to the gesture recognition unit 102 and the moving image recording unit 103 via the CPU 5.

  The gesture recognition unit 102 mainly corresponds to the CPU 5 and the RAM 7 and is realized by the CPU 5 executing a predetermined control program (software module). The gesture recognition unit 102 temporarily stores the moving image data supplied from the imaging unit 101 and performs predetermined image processing on the moving image data to recognize a predetermined gesture performed by the user. Here, the predetermined gesture refers to a gesture for starting a recording operation in the moving picture recording unit 103 (hereinafter referred to as “recording start instruction gesture”), and a gesture for stopping the started recording operation (hereinafter referred to as “ Recording stop instruction gesture ”) and a gesture for performing tempo setting (hereinafter referred to as“ tempo setting instruction gesture ”). When the gesture recognition unit 102 recognizes that the moving image data from the imaging unit 101 includes a recording start instruction gesture (strictly speaking, as described later, “recording start instruction timing”). Until a gesture corresponding to is recognized), a command to start recording (hereinafter referred to as “recording start command”) is output, and when it is recognized that a recording stop instruction gesture is included, A command for instructing tempo setting (hereinafter referred to as “tempo setting command”) when a command for instructing stop (hereinafter referred to as “recording stop command”) is output and it is recognized that a tempo setting instruction gesture is included. Is output. Of the three types of commands output from the gesture recognition unit 102, a recording start command and a recording stop command are supplied to the moving image recording unit 103 and the audio recording unit 108, and a tempo setting command is supplied to the automatic performance unit 105. Is done. Note that the feature of the present invention is how to use the gesture recognized from the moving image data, and not how to recognize the gesture from the moving image data. A known recognition process may be employed.

  The moving image recording unit 103 mainly corresponds to the CPU 5, the RAM 7, and the storage device 9, and is realized by the CPU 5 executing a predetermined control program. When the recording start command is supplied from the gesture recognition unit 102, the moving image recording unit 103 records the moving image data supplied from the imaging unit 101 as time passes. Since any data format may be employed when recording moving image data, it may be recorded in the same format as the moving image data format supplied from the moving image recording unit 103. After conversion into the format, recording may be performed in the converted format. In this embodiment, the moving image data recording start / stop in the moving image recording unit 103 is performed in response to the recording start / stop command supplied from the gesture recognition unit 102. In addition, It may be made in response to a user operating a recording start / stop switch (both not shown) included in the setting operator 2.

  The automatic performance data storage unit 104 mainly corresponds to a partial area secured at a predetermined position in the RAM 7 and stores automatic performance data automatically played by the automatic performance unit 105. A plurality of automatic performance data is stored in the electronic keyboard instrument (the storage device 9) of the present embodiment, and the automatic performance data storage unit 104 selects one of them (described later). A plurality of possible cases) is stored. Here, the automatic performance data is for allowing the user to perform a back performance when playing a certain piece of music using the performance operator 1, and specifically, music data (one piece of music data ( Song data), accompaniment style data, chord progression data, and the like. It should be noted that any format may be adopted as the automatic performance data format, but in this embodiment, the SMF (Standard MIDI File) format often used in ordinary electronic musical instruments is adopted. To. The accompaniment style data is composed of, for example, style data of a predetermined length (for example, one bar length) corresponding to each of a plurality of sections (“Intro”, “Main”, “Ending”, “Fill-in”, etc.) Each style data is composed of sequence data composed of a plurality of sets each including a key event and its reproduction timing. The pitch of each key event is expressed with reference to, for example, “C major chord”. When the user designates a favorite chord from the keyboard, for example, the automatic performance unit 105 selects the pitch of each key event. Is converted to a pitch suitable for the chord and output to the musical tone generation unit 107 at the subsequent stage. As described above, the accompaniment style data is constituted by instruction information (sequence) indicating the actual data (storage location) for each section, instead of the actual data (style data) for each section, as described above. When automatic performance is performed, actual data indicated by performance position instruction information may be acquired. Further, the chord progression data is composed of sequence data composed of a plurality of chord data (= chord event + its reproduction timing), for example. When the accompaniment is generated by reproducing the chord progression data, the chord data constituting the chord progression data may be reproduced as it is to generate the accompaniment, or the chord data may be generated based on the chord data designated by the user. The accompaniment may be generated by converting the progress data into other chord progress data.

  The automatic performance unit 105 mainly corresponds to the CPU 5 and the RAM 7 and is realized by the CPU 5 executing a predetermined control program. The automatic performance unit 105 reads automatic performance data at a position indicated by a read pointer (not shown) provided in the RAM 7 from the automatic performance data storage unit 104. When the read automatic performance data is reproduction timing data, the automatic performance data is reproduced. After waiting until the timing, the read pointer is advanced by “1”, the key event data at that position is read and supplied to the tone generator 107, and then the read pointer is advanced by “1” while the read automatic performance data is the key. If it is event data, it is supplied to the tone generator 107 as it is, and then the read pointer is advanced by “1”. Here, the reproduction timing data represents the time difference (delta time) between adjacent key event data as an integer value with a predetermined note length (for example, 96th note length) as a unit. That is, “waiting until the reproduction timing” means decrementing the value of the read reproduction timing data by “1” every predetermined note length and waiting until it becomes “0”. Therefore, the automatic performance unit 105 needs to measure the time corresponding to the predetermined note length, but the time corresponding to the note length is a time corresponding to the tempo. In other words, the time corresponding to the note length is the tempo. Therefore, the automatic performance unit 105 determines a time corresponding to a predetermined note length based on the tempo according to the tempo setting command from the gesture recognition unit 102. And the automatic performance part 105 measures the determined time using the timer (not shown) built in CPU5, for example.

  If the key event data is read when the accompaniment style data is stored as the automatic performance data, the automatic performance unit 105 is designated with the pitch of the key event data as described above. It is converted to a pitch that matches the chord data and output. As described above, the chord data designation method is exemplified by the method in which the user designates using the keyboard as described above. However, the chord data designation method is not limited to this. May be adopted. In this case, chord progression data to be reproduced is stored in an area different from the area where accompaniment style data is stored in the automatic performance data storage section 104, and the automatic performance section 105 stores the accompaniment style data. The chord progression data may be read in parallel with the reading.

  The performance operator 106 mainly corresponds to the performance operator 1 and the detection circuit 3, and generates performance information (in the present embodiment, MIDI message) according to the performance operation by the user. Output to the unit 109.

  The musical tone generation unit 107 mainly corresponds to the sound source / effect circuit 11, and in the musical tone generation environment (tone color, volume, effect, etc.) set by the user, the key event data and the performance operator supplied from the automatic performance unit 105. A musical tone signal is generated based on the performance information supplied from 106, and an effect is appropriately given, and then output to the audio recording unit 108.

  The audio recording unit 108 mainly corresponds to the CPU 5 and the RAM 7 and is realized by the CPU 5 executing a predetermined control program. When the recording start command is supplied from the gesture recognition unit 102, the audio recording unit 108 records the musical tone signal (audio data) supplied from the musical tone generation unit 107 as time passes, and when the recording stop command is supplied. Stop recording the music signal. Here, any recording format of audio data may be adopted, which may be uncompressed or compressed. In the present embodiment, the recording start / stop instruction in the audio recording unit 108 is also performed by the recording start / stop command in the same manner as the recording start / stop instruction in the moving image recording unit 103. Since both start / stop instructions are given at the same timing, different start / stop commands (for example, a recording start / stop command for the moving picture recording unit 103 and a recording start / stop command for the audio recording unit 108) are generated. This is because there is no need to supply them. Of course, even in this case, different start commands may be generated and supplied to the moving image recording unit 103 and the audio recording unit 108, respectively.

  The MIDI recording unit 109 mainly corresponds to the CPU 5 and the RAM 7 and is realized by the CPU 5 executing a predetermined control program. The MIDI recording unit 109 records the MIDI message supplied from the performance operator 106 in the SMF format. Note that the recording start / stop instruction in the MIDI recording unit 109 may also be performed by a recording start / stop command.

  The file creation unit 110 mainly corresponds to the CPU 5 and the RAM 7 and is realized by the CPU 5 executing a predetermined control program. The file creation unit 110 stores the moving image data recorded by the moving image recording unit 103, the audio data recorded by the audio recording unit 108, and the SMF format MIDI data recorded by the MIDI recording unit 109 in one file. A multimedia file is created and output to the file storage unit 111.

  The file storage unit 111 mainly corresponds to a partial area secured at a predetermined position in the RAM 7 and stores a multimedia file supplied from the file creation unit 110. The stored multimedia file can be supplied to an external device via the communication I / F 10 or a removable storage medium, and can be uploaded to, for example, a video posting site server or a blog site server.

  In the present embodiment, the multimedia file is generated with a plurality of types (three types) of data stored in one file. However, the present invention is not limited to this, and each of the plurality of types of data is generated. A file may be created individually, and the individual files may be associated with each other so that they can be played back simultaneously.

  Further, although the electronic keyboard instrument of the present embodiment is provided with both the audio recording unit 108 and the MIDI recording unit 109, only one of them may be provided. Or, as a function, both the audio recording unit 108 and the MIDI recording unit 109 are provided, and either one of the two is valid, only the audio recording unit 108 is valid, or only the MIDI recording unit 109 is valid according to a user's selection operation. You may comprise so that selection is possible.

  A control process executed by the electronic keyboard instrument configured as described above will be described first with reference to FIG. 3 and then in detail with reference to FIG.

  FIG. 3 is a diagram for explaining a method for photographing a performance scene using the electronic keyboard instrument according to the present embodiment. FIG. 3A shows the imaging apparatus 20 for photographing a user who plays the electronic keyboard instrument. An example of installation is shown, and FIG. 5B shows a captured image (moving image) displayed on the LCD of the display device 8 when a performance scene is photographed with the imaging device 20 installed in FIG. ) 8a shows an example, FIG. 8C shows an example of the recording start instruction gesture, and FIG. 8D shows an example of the recording stop instruction gesture.

  As shown in FIG. 3A, the imaging device 20 is installed, a power switch (not shown) of the imaging device 20 is turned on, and moving image data is always transferred from the imaging device 20 to the electronic keyboard instrument of the present embodiment. When the user presses a recording mode selection switch (not shown) included in the setting operator 2, for example, the electronic keyboard instrument according to the present embodiment changes its operation mode to the recording mode. To enter the recording standby state. The gesture recognition unit 102 performs image recognition on the moving image data supplied. The gesture for image recognition is either the recording start instruction gesture or the recording stop instruction gesture. In the present embodiment, the recording start instruction gesture and the recording stop instruction gesture are both directed. Adopting a command gesture). More specifically, when the automatic performance data to be automatically played is, for example, a music of 4 beats, the recording start instruction gesture is typically performed by the user with his / her hand as shown in FIG. This is a typical four-beat operation. On the other hand, the recording stop instruction gesture is a “stop” command operation (command end operation) performed by the user with his / her hand as shown in FIG.

  When the gesture recognizing unit 102 recognizes that the recording start instruction gesture is included in the moving image data, the current operation is in any position from “1” to “4” in FIG. When it is determined that the current position is “1” for the second time, the recording start command is generated and output to the moving image recording unit 103, the audio recording unit 108, and the MIDI recording unit 109. In response to this, the moving image recording unit 103 starts recording the moving image data supplied from the imaging unit 101, the audio recording unit 108 starts recording the musical sound signal supplied from the musical sound generating unit 107, The MIDI recording unit 109 starts recording the MIDI message supplied from the performance operator 106.

When the gesture recognition unit 102 recognizes the recording start instruction gesture and recognizes the tempo setting instruction gesture, which is a partial gesture in the recording start instruction gesture, the gesture recognition unit 102 calculates a tempo value from the gesture. Then, a tempo setting command for setting the calculated tempo value is output to the automatic performance unit 105. Here, the tempo is defined as the number of beats per minute (BPM), the 4-beat command operation of FIG. 3C is adopted as the recording start instruction gesture, and the tempo setting instruction gesture is set to one beat. If the equivalent commanding action, that is, the action of shaking the hand down from the position ("4" position) that the user holds to start commanding to the "1" position, is adopted, the tempo value is as follows: Is calculated. That is,
(A) Measure the time (seconds) that elapsed when the user performed a command action equivalent to one beat (tempo setting instruction gesture). (B) Calculate 60 seconds by dividing by the measurement time of (a) above. The Of course, the method of calculating the tempo value is not limited to this. For example, as a tempo setting instruction gesture, a command action corresponding to 4 beats, that is, a position (“4”) prepared for the user to start commanding. ) From the position (1) to the original position (position "4") through each position 1-2-3,
(A ′) The time (seconds) elapsed when the user performs a command action corresponding to 4 beats (b ′) 60 seconds is a quarter of the measurement time of (a ′) (= 1 beat) (Elapsed time per hit) may be calculated by division.

  When the automatic performance unit 105 receives the tempo setting command from the gesture recognition unit 102, the automatic performance unit 105 sets the tempo value used when performing the automatic performance to the tempo value specified by the tempo setting command.

  On the other hand, in a state where the moving image recording unit 103 is recording moving image data (in this case, of course, the audio recording unit 108 is also recording a musical sound signal and the MIDI recording unit 109 is also recording a MIDI message), the gesture is performed. When the recognizing unit 102 recognizes that the recording stop instruction gesture is included in the moving image data, the gesture recognizing unit 102 notifies the moving image recording unit 103, the audio recording unit 108, and the MIDI recording unit 109. Output a recording stop command. In response to this, the moving picture recording unit 103 stops the recording of the moving image data, the audio recording unit 108 stops the recording of the musical tone signal, and the MIDI recording unit 109 stops the recording of the MIDI message.

  As described above, in the electronic keyboard instrument of the present embodiment, the recording of the performance scenery can be started or stopped only by the user performing a predetermined gesture toward the imaging device 20, so that the recording of the performance scenery can be performed. It is possible to prevent an unfavorable video unrelated to the performance scene, such as an operation of operating a button to start or stop, from being recorded.

  Further, since the tempo can be set based on the gesture, the tempo can be set at the start of recording without setting the tempo in advance.

  Next, this control process will be described in detail.

  FIG. 4 is a flowchart showing the procedure of the recording process executed by the electronic keyboard instrument of the present embodiment, particularly the CPU 5, and this recording process is started when the user presses the recording mode selection switch, for example. The

This video recording process mainly includes (1) video recording standby state setting process (step S1).
(2) Automatic performance data selection process (step S2)
(3) Recording start command generation process (step S4)
(4) Tempo setting command generation processing (step S6)
(5) Automatic performance start command generation process (step S8)
(6) Performance information generation process (step S10)
(7) Automatic performance stop command generation process (step S12)
(8) Recording stop command generation process (step S14)
(9) Recording end processing (step S15)
It is constituted by.

  When the main recording process is started, the CPU 5 first advances the process to the recording standby state setting process (1), and when the moving picture recording unit 103, the audio recording unit 108, and the MIDI recording unit 109 receive a recording start command. Immediately, the moving image recording unit 103 can record moving image data, the audio recording unit 108 can record a musical sound signal, and the MIDI recording unit 109 is set to a state in which a MIDI message can be recorded (recording standby state). When this (1) recording standby state setting process is executed, the imaging device 20 is already connected to the electronic keyboard instrument of the present embodiment via the communication I / F 10, and the power source of the imaging device 20 is It is assumed that it is on. In other words, the “recording standby state” here refers to the electronic keyboard instrument (particularly, the moving picture recording unit 103, the audio recording unit 108, and the MIDI recording unit 109) of the present embodiment. It is not a thing. However, when the imaging device 20 is configured to be controllable to a recording standby state (for example, a state in which the power is turned on and moving image data is output) from the electronic keyboard instrument side of the present embodiment, the imaging device The process of setting 20 to the recording standby state may also be performed within this (1) recording standby state setting process. In the present embodiment, the recording in the moving picture recording unit 103 and the recordings in the audio recording unit 108 and the MIDI recording unit 109 are started at the same time and ended at the same time. In addition, since a MIDI message is also recorded in the present embodiment, hereinafter, “recording” includes not only recording of moving image data in the moving image recording unit 103 but also recording of musical sound signals in the audio recording unit 108. In addition, recording of a MIDI message in the MIDI recording unit 109 is also included.

  Next, the CPU 5 advances the process to the automatic performance data selection process (2). As described above, a plurality of pieces of automatic performance data that can be automatically played by the automatic performance unit 105 are stored in the storage device 9, and the automatic performance data storage unit 104 stores one automatic performance data selected by the user. Performance data is stored. In this (2) automatic performance data selection processing, the CPU 5 displays a list of a plurality of automatic performance data (names) stored in the storage device 9 on the display device 8, and is selected by the user from the list. The read automatic performance data is read out and stored in the automatic performance data storage unit 104.

  Next, the CPU 5 performs image recognition as to whether or not the recording start instruction gesture is included in the moving image data supplied from the imaging unit 101, and when the recording start instruction gesture is included, It is determined whether or not the operation is at the recording start timing (step S3). This process is a part of the process executed by the gesture recognition unit 102. In the present embodiment, the recording start instruction gesture is a command operation (command gesture) by the user's hand, and the automatic performance data selected in the (2) automatic performance data selection process is, for example, a music of four beats. In some cases, a typical command action for a four-beat tune as shown in FIG. 3 (c) (from the rest position of the “stand” through the 1-2-3 via the position “4” of the “stand” Is an operation that leads to “ Then, the CPU 5 determines whether or not the current operation corresponds to “1” for the second time after starting the command, and when it is determined to correspond to “1” for the second time, that is, the recording start instruction timing. In some cases, the CPU 5 advances the process to the recording start command generation process (3). In this (3) recording start command generation process, the CPU 5 generates a recording start command and supplies it to the moving picture recording unit 103, the audio recording unit 108 and the MIDI recording unit 109.

  As described above, in this embodiment, the command gesture is recognized as the recording start instruction gesture, and the recording start instruction is performed at the predetermined timing. However, the recording start instruction gesture is not limited to this, but the face Other gestures such as a facial expression (also considered as a gesture in the present embodiment) and a swing motion may be adopted. As facial expressions, blinking (recording start instruction at a predetermined number of blinks), gaze on the imaging device 20 (recording start instruction when gazing continuously for a predetermined time), smile (when smiling) Recording start instruction) and the like, and the swing operation can include horizontal or vertical swing (recording start instruction with a predetermined number of swings).

  Next, the CPU 5 performs image recognition as to whether or not the tempo setting instruction gesture is included in the recording start instruction gesture recognized in step S3 (step S5). This process is also a part of the process executed by the gesture recognition unit 102. When the recording start instruction gesture recognized in step S5 is a command gesture, the tempo setting instruction gesture is a partial gesture constituting the command gesture. If the command gesture is, for example, the gesture shown in FIG. 3C, the tempo setting instruction gesture is a gesture in which the user swings his / her hand down from the “stand” stationary position to the “1” position as described above. It is. When recognizing the tempo setting instruction gesture, the CPU 5 advances the process to the tempo setting command generation process (4). In this (4) tempo setting command generation process, the CPU 5 calculates a tempo value based on the recognized tempo setting instruction gesture, generates a tempo setting command including the tempo value, and supplies the tempo setting command to the automatic performance unit 105. To do. When the recording start instruction gesture is the swing operation, the tempo setting instruction gesture is the same as the recording start instruction gesture (in this case, the recording start instruction gesture is recognized). At the same time, the CPU 5 recognizes the tempo setting instruction gesture), and the CPU 5 may detect the tempo value from the swing cycle and generate a tempo setting command.

  Next, the CPU 5 checks whether or not an automatic performance start instruction has been issued (step S7). If an automatic performance start instruction has been issued, the process proceeds to the automatic performance start command generation process (5). In this (5) automatic performance start command generation process, the CPU 5 generates an automatic performance start command and supplies it to the automatic performance unit 105. In response to this, the automatic performance section 105 starts automatic performance of the automatic performance data stored in the automatic performance data storage section 104. Here, it is assumed that the start instruction of the automatic performance is made by the user pressing an automatic performance start switch (not shown) included in the setting operator 2, for example. However, in the flowchart of this recording process, when the automatic performance start switch is pressed, the moving image recording by the moving image recording unit 103 has already been started, so that the operation of the user pressing the automatic performance start switch is also recorded. In order to prevent this operation from being included in the performance scene, the automatic performance is automatically performed after a predetermined time has elapsed since the start of recording to the moving image recording unit 103 was instructed by the recording start command generation process of (3). A start command may be generated.

  When the user performs a performance operation using the performance operator 1, the CPU 5 advances the process to the performance information generation process (6) (step S9 → S10). In this (6) performance information generation process, the CPU 5 generates performance information corresponding to the performance operation (in this embodiment, as described above, the MIDI message), and the musical tone generation unit 107 and the MIDI recording unit 109. To supply.

  Next, the CPU 5 checks whether or not an automatic performance stop instruction has been given, or whether or not the automatic performance has progressed and reached the end of the automatic performance data (step S7), and whether or not the automatic performance stop instruction has been given or automatically When the end of the performance data is reached, the process proceeds to the automatic performance stop command generation process (7). In this (7) automatic performance stop command generation process, the CPU 5 generates an automatic performance stop command and supplies it to the automatic performance unit 105. In response to this, the automatic performance unit 105 ends the automatic performance. Here, it is assumed that the automatic performance stop instruction is given by, for example, the user pressing an automatic performance stop switch (not shown) included in the setting operator 2. However, since the recording is not finished at this time, the operation of pressing the automatic performance stop switch is also recorded. In order to prevent this operation from being included in the performance scenery, when the recording stop command generation processing in (8) described later is instructed to end the recording to the moving picture recording unit 103 (and the audio recording unit 108), An automatic performance stop command may be automatically generated and output to the automatic performance unit 105 prior to the instruction, and the end of the recording may be instructed after a predetermined time has elapsed. If the automatic performance is automatically terminated on the moving image recording unit 103 side when the end of the automatic performance data is reached, the CPU 5 does not need to generate an automatic performance stop command.

  Next, the CPU 5 performs image recognition as to whether or not the recording stop instruction gesture is included in the moving image data supplied from the imaging unit 101 (step S13). This process is a part of the process executed by the gesture recognition unit 102. In this embodiment, the recording stop instruction gesture is a command termination operation by the user's hand, and is a “stop” operation as shown in FIG. Note that “stop” includes various techniques such as “stop”, “rebound stop”, and “stop”, but it goes without saying that any of them may be adopted as a recording stop instruction gesture. When the recording stop instruction gesture is recognized by the image recognition in step S13, the CPU 5 advances the process to the recording stop command generation process (8). In this (8) recording stop command generation process, the CPU 5 generates a recording stop command and supplies it to the moving image recording unit 103, the audio recording unit 108, and the MIDI recording unit 109. In the determination processing in step S13, it is further determined whether or not the user has instructed to stop recording by pressing a recording stop switch (not shown) included in the setting operator 2, for example. This is because the end instruction operation (operation of pressing the recording stop switch) may be recorded when the recording is ended, or it is troublesome to give an instruction by the gesture until the end of the recording. Is.

  Subsequently, the CPU 5 advances the process to the recording end process (9). In this (9) recording end process, the CPU 5 ends the recording mode and instructs the file creation unit 110 to create the multimedia file and store it in the file storage unit 111.

  In this embodiment, the gesture for recognizing an image is a single gesture predetermined for each command to be generated. However, the present invention is not limited to this, and a plurality of gestures can be recognized for each command and adopted for each command. One gesture may be configured to be selectable by the user, or a plurality of gestures may be recognized regardless of a command, and a user may be configured to be able to select which gesture is associated with which command. Furthermore, you may make it learn a user's gesture habit.

  In this embodiment, the tempo setting instruction gesture is a part or all of the recording start instruction gesture. However, the present invention is not limited to this, and a gesture independent of the recording start instruction gesture may be employed. .

  In addition, the configuration for detecting the tempo from the gesture is not essential. This is because tempo data is generally set in the automatic performance data, and there is no problem even if the automatic performance is simply performed according to the tempo of the tempo data. If the tempo is detected from the gesture, an automatic performance is performed using the detected tempo, or an automatic performance is performed according to the tempo of the tempo data already set in the automatic performance data. It is preferable for the user to be able to select in advance whether to do this.

  Further, in the present embodiment, the recognized gesture is used to generate a command for controlling the moving image recording unit 103, the audio recording unit 108, the MIDI recording unit 109, and the automatic performance unit 105. However, the present invention is not limited to this. It may be used to generate a command for controlling the imaging device 20. For example, when the user designates up, down, left, and right with hands or eyes (line of sight), a control command that directs the photographing direction of the imaging device 20 in the designated direction is generated, or according to the swing angle. It is possible to generate a control command that changes the zoom magnification.

  Further, in the present embodiment, whenever a user performs using the performance operator 1, automatic performance is performed using automatic performance data. However, some users do not need back performance for their own performance. It is preferable to provide a function that allows the user to select whether or not to perform automatic performance. This selection function is particularly useful when an automatic performance start instruction is automatically issued.

  A program in which a storage medium storing software program codes for realizing the functions of the above-described embodiments is supplied to a system or apparatus, and a computer (or CPU or MPU) of the system or apparatus is stored in the storage medium. It goes without saying that the object of the present invention can also be achieved by reading and executing the code.

  In this case, the program code itself read from the storage medium realizes the novel function of the present invention, and the program code and the storage medium storing the program code constitute the present invention.

  As a storage medium for supplying the program code, for example, a flexible disk, hard disk, magneto-optical disk, CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD + RW, magnetic A tape, a non-volatile memory card, a ROM, or the like can be used. Further, the program code may be supplied from a server computer via a communication network.

  Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also the OS running on the computer based on the instruction of the program code performs the actual processing. It goes without saying that a case where the functions of the above-described embodiment are realized by performing part or all of the above and the processing thereof is included.

  Further, after the program code read from the storage medium is written into a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer, the function expansion is performed based on the instruction of the program code. It goes without saying that the CPU or the like provided in the board or the function expansion unit performs part or all of the actual processing and the functions of the above-described embodiments are realized by the processing.

It is a block diagram which shows schematic structure of the electronic keyboard instrument to which the electronic music apparatus which concerns on one embodiment of this invention is applied. It is a block diagram which shows the function structure of the electronic keyboard instrument of FIG. It is a figure for demonstrating the imaging | photography method of the performance scenery with the electronic keyboard instrument of FIG. It is a flowchart which shows the procedure of the video recording process which the electronic keyboard instrument of FIG. 1, especially CPU performs.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Performance operator, 5 ... CPU (supply means, production | generation means, recognition means, recording / recording means, control means, automatic performance means, setting means), 7 ... RAM (generation means, recognition means, recording / recording means, Control means, automatic performance means, setting means), 10 ... communication I / F (supply means), 11 ... sound source / effect circuit (generation means)

Claims (2)

Supply means for supplying moving image data obtained by photographing a scene including a user;
A performance controller for generating performance information;
Generating means for generating a musical sound signal based on performance information generated by a user operating the performance operator;
Recognition means for recognizing a user's predetermined gesture from the moving image data supplied by the supply means;
Recording / recording means for recording the supplied moving image data and recording the musical sound signal generated by the generating means and / or the generated performance information;
An electronic music apparatus comprising: control means for controlling a recording / recording state of the recording / recording means based on a gesture recognized by the recognition means. Automatic performance means for performing automatic performance based on automatic performance data;
Setting means for setting a tempo of the automatic performance based on the recognized gesture;
The electronic music apparatus according to claim 1, wherein the generation unit generates a musical sound signal even based on an automatic performance by the automatic performance unit.

Images