JP6098083B2 - Performance device, performance method and program - Google Patents

Description

  The present invention relates to a performance device that generates a musical sound in response to a user's performance operation, and particularly to a performance device that generates a musical sound in response to a user's performance operation for a virtual instrument that does not exist (referred to as an air musical instrument) The present invention relates to a performance method and a program for an air musical instrument.

  2. Description of the Related Art Conventionally, an apparatus for playing an “air drum” is known as a performance apparatus that generates musical sounds in response to a user's performance operation on an air musical instrument. In this method, an acceleration sensor is provided on a stick member, and a user who is a performer detects an operation of swinging down the stick member, and generates a musical sound corresponding to an instrument struck by the stick member.

  In this air drum, the player's performance is shot with a camera, the shot image is combined with the virtual image of the drum set and displayed on the monitor screen, and the positional relationship between the swinging action of the stick member and each instrument of the drum set There is known a technique that makes it possible to visually grasp the above (for example, see Patent Document 1).

Japanese Patent No. 3599115

  In the air drum described above, the position where the stick member is struck can be obtained by recognizing a marker (light emitting body) provided at the tip of the stick member on the photographed image. However, when the player swings the stick member, the marker may be temporarily hidden by equipment such as an arm or a stand and may not be recognized on the captured image. This is called “Lost Marker”. In the state where the marker is lost, even if the movement of the stick member being swung down by the acceleration sensor is known, it is not known where the air drum is struck, so the instrument to be played cannot be identified and no musical sound can be generated.

  The present invention has been made in view of the above points, and provides a performance device, a performance method, and a program that can generate a musical tone by determining the musical instrument struck by the performer even if the marker is lost due to the performance operation. The purpose is to provide.

The performance device according to the present invention includes a performance member having a marker, photographing means for photographing a player who performs an air musical instrument using the performance member, and the same performance operation during the first performance and the first performance. Lost detection means for detecting that the marker has been lost from an image photographed by the photographing means during the second performance, and during the lost of the marker detected by the lost detection means during the first performance Position estimation means for estimating the position of the marker when the performer strikes the air musical instrument with the performance member, and after the position estimation means estimates the position of the marker during the first performance. and above when the marker when the second play is lost, the corresponding position of the second said markers lost during playing, the first of said position estimating means during the performance Accordingly, characterized by comprising a tone generation control means for generating a predetermined tone based on the position of the inferred said marker.

  According to the present invention, even if a marker is lost due to a performance operation, a musical tone can be generated by determining the musical instrument struck by the performer.

FIG. 1 is a diagram for explaining the outline of a performance apparatus according to the first embodiment of the present invention. FIG. 1 (a) is a diagram showing a system configuration of the performance apparatus, and FIG. It is a figure which shows the positional relationship of operation | movement and a virtual drum set. FIG. 2 is a block diagram showing a hardware configuration of the stick unit of the performance device according to the embodiment. FIG. 3 is a perspective view showing an external configuration of a stick portion of the performance device according to the embodiment. FIG. 4 is a block diagram showing a hardware configuration of the camera unit unit of the performance device according to the embodiment. FIG. 5 is a block diagram showing a hardware configuration of the center unit portion of the performance device according to the embodiment. FIG. 6 is a flowchart showing processing of the stick unit of the performance device according to the embodiment. FIG. 7 is a diagram illustrating a change in output related to the acceleration in the vertical direction of the motion sensor unit when a performance operation is performed using the stick unit of the performance device according to the embodiment. FIG. 8 is a flowchart showing processing of the camera unit unit of the performance device according to the embodiment. FIG. 9 is a flowchart showing processing of the center unit unit of the performance device according to the embodiment. FIG. 10 is a view for explaining the positional relationship between the camera unit of the performance device and the performer in the embodiment. FIG. 11 is a diagram for explaining the cause of the marker being lost in the performance device according to the embodiment. FIG. 12 is a diagram for explaining a method of estimating the marker position when the marker of the performance device in the embodiment is lost. FIG. 13 is a flowchart showing the performance processing of the center unit portion of the performance device in the embodiment. FIG. 14 is a flowchart showing the performance processing of the center unit section of the performance device according to the embodiment. FIG. 15 is a diagram showing an example of a first lost management table of the performance device in the embodiment. FIG. 16 is a diagram showing the relationship between the positions before and after the lost marker and the position of the shot operation in the same embodiment. FIG. 16A shows the case where the shot operation is performed at an intermediate position between A and B. (B) is an example when a shot operation is performed at a position 2/3 between AB. FIG. 17 is a flowchart showing the performance processing of the center unit section of the performance device according to the second embodiment of the present invention. FIG. 18 is a flowchart showing a performance process of the center unit portion of the performance device according to the embodiment. FIG. 19 is a diagram showing an example of a second lost management table of the performance device in the embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(First embodiment)
[Outline of the performance device 1]
FIG. 1 is a diagram for explaining an outline of a performance device 1 according to the first embodiment of the present invention. FIG. 1 (a) is a diagram showing a system configuration of the performance device 1, and FIG. FIG. 6 is a diagram showing a positional relationship between the performance operation of the virtual drum set and the virtual drum set D.

  The performance device 1 in this embodiment is a device for playing an “air drum” as an air musical instrument. The performance device 1 includes stick units 10A and 10B, a camera unit unit 20, a center unit unit 30, and a monitor unit 40.

  In the present embodiment, since a virtual drum performance using two sticks is assumed, two stick portions 10A and 10B are provided. However, the number of stick parts is not limited to this, and may be one or three or more. Hereinafter, when the stick parts 10A and 10B are not distinguished, they are collectively referred to as “stick part 10”.

  The stick part 10 is made of a stick-like member extending in the longitudinal direction. The stick unit 10 is used as a performance member for issuing a performance instruction for an air musical instrument in accordance with a player's operation. In this embodiment, a user who is a performer has one end (base side) of the stick unit 10 and performs an air drum performance by swinging up and down the stick unit 10 around the wrist or the like.

  In order to detect a player's performance using the stick unit 10, various sensors such as an acceleration sensor are provided on the other end (tip side) of the stick unit 10 (a motion sensor unit 14 described later). ). The stick unit 10 transmits a note-on event to the center unit unit 30 based on the performance performance of the performer detected by these sensors.

  Further, a marker portion 15 (see FIG. 2) is provided on the tip side of the stick portion 10. The marker unit 15 is made of a light emitter such as an LED (light emitting diode) and is used to detect the movement of the stick unit 10.

  The camera unit unit 20 includes an optical camera (imaging device), and photographs a player who plays the air drum using the stick unit 10 at a predetermined frame rate. More specifically, the camera unit unit 20 captures an imaging space where the player has the stick unit 10. The camera unit unit 20 specifies the position coordinates of the marker unit 15 provided at the tip of the stick unit 10 from the photographed image, and transmits it to the center unit unit 30.

  The center unit 30 corresponds to the control main body of the performance device 1. When the center unit unit 30 receives a note-on event from the stick unit 10, the center unit unit 30 generates a predetermined musical sound according to the position coordinate data of the marker unit 15 at the time of reception. Specifically, the center unit unit 30 stores the position coordinate data of the virtual drum set D shown in FIG. 1B in association with the imaging space of the camera unit unit 20. Based on the position coordinate data and the position coordinate data of the marker unit 15 at the time of receiving the note-on event, the instrument struck by the stick unit 10 is specified, and a musical sound corresponding to the instrument is generated.

  In addition, on the screen of the monitor unit 40, the user being played and the virtual drum set D photographed by the camera unit unit 20 are combined and displayed. The user can perform by hitting the position corresponding to each instrument constituting the virtual drum set D with the stick unit 10 while viewing the screen of the monitor unit 40.

  The “instrument” here refers to a percussion instrument (a drum set cymbal, a tom, etc.) that produces a sound when hit with a stick. The monitor unit 40 shown in FIG. 1 is arbitrarily connected to the center unit unit 30. If it is not necessary to confirm the position of each musical instrument of the virtual drum set D during performance, the monitor unit 40 is unnecessary.

  Hereinafter, the configuration of the performance device 1 will be specifically described.

[Configuration of Stick Unit 10]
FIG. 2 is a block diagram showing a hardware configuration of the stick unit 10 of the performance apparatus 1, and FIG. 3 is a perspective view showing an external configuration of the stick unit 10.

  The stick unit 10 includes a CPU 11, a ROM 12, a RAM 13, a motion sensor unit 14, a marker unit 15, and a data communication unit 16.

  The CPU 11 performs posture detection, shot detection, action detection, and the like, which will be described later, as control related to the stick unit 10. Further, the CPU 11 performs communication control with the center unit unit 30 via the data communication unit 16.

  The ROM 12 stores a program 12a corresponding to various processes executed by the CPU 11 and marker feature information 12b. Here, in this embodiment, since the two stick portions 10A and 10B are used for playing the air drum, the marker portion 15 (first marker) of the stick portion 10A and the marker portion 15 (second marker) of the stick portion 10B. Need to be distinguished.

  The marker feature information 12b is information for distinguishing between the first marker and the second marker. For example, the shape, size, hue, saturation, luminance, and point reduction speed at the time of light emission are defined. ing. In the stick unit 10 </ b> A and the stick unit 10 </ b> B, the CPU 11 reads different marker feature information 12 b from the ROM 12 and performs light emission control of the marker unit 15.

  The RAM 13 stores data acquired or generated in the processing of the CPU 11 such as various sensor values output by the motion sensor unit 14.

  The motion sensor unit 14 includes various sensors for detecting the state of the stick unit 10 and outputs a predetermined sensor value. Here, as a sensor for realizing the motion sensor unit 14, for example, an acceleration sensor, an angular velocity sensor, a magnetic sensor, or the like can be used.

(Acceleration sensor)
As the acceleration sensor, a triaxial sensor that outputs acceleration generated in each of the three axial directions of the X axis, the Y axis, and the Z axis can be used. As shown in FIG. 3, an axis that coincides with the longitudinal axis of the stick portion 10 is defined as a Y axis, an axis that is parallel to a substrate (not shown) on which an acceleration sensor is disposed and is orthogonal to the Y axis is represented by X. An axis that is orthogonal to the X axis and the Y axis can be a Z axis. At this time, the acceleration sensor may acquire the acceleration of each component of the X axis, the Y axis, and the Z axis, and may calculate a sensor composite value obtained by combining the respective accelerations.

  Here, the performer holds one end (the base side) of the stick unit 10 and performs a swing-up / down operation centering on the wrist or arm. When the stick unit 10 is stationary, the acceleration sensor calculates a value corresponding to the gravitational acceleration 1G as a sensor composite value. When the stick unit 10 is moving, the acceleration sensor calculates a sensor combined value of the acceleration of the movement and the gravitational acceleration 1G.

  The sensor composite value is obtained by calculating the square root of the sum of the squares of the accelerations of the X-axis, Y-axis, and Z-axis components, for example.

(Angular velocity sensor)
As the angular velocity sensor, for example, a sensor provided with a gyroscope can be used. Here, referring to FIG. 3, the angular velocity sensor outputs the rotation angle 301 of the stick unit 10 in the Y-axis direction and the rotation angle 311 of the stick unit 10 in the X-axis direction.

  A rotation angle 301 in the Y-axis direction is a rotation angle of the front-rear axis viewed from the player when the player holds the stick unit 10 and can be referred to as a roll angle. The roll angle corresponds to an angle 302 indicating how much the XY plane is tilted with respect to the X axis, and the player holds the stick unit 10 in his hand and rotates it left and right around the wrist. Caused by.

  The rotation angle 311 in the X-axis direction is a rotation angle on the left and right axes viewed from the player when the player holds the stick unit 10 and can be referred to as a pitch angle. The pitch angle corresponds to an angle 312 indicating how much the XY plane is tilted with respect to the Y axis, and is generated when the player holds the stick unit 10 in his / her hand and swings his / her wrist up and down.

  Although not shown, the angular velocity sensor may also output the rotation angle in the Z-axis direction. At this time, the rotation angle in the Z-axis direction is basically the same as the rotation angle 311 in the X-axis direction, and is generated when the player holds the stick unit 10 in his / her hand and swings his / her wrist in the left / right direction. The pitch angle.

(Magnetic sensor)
As the magnetic sensor, a sensor capable of outputting magnetic sensor values in the three-axis directions of the X axis, the Y axis, and the Z axis shown in FIG. 3 can be used. From such a magnetic sensor, a vector indicating north (magnetic north) by a magnet is output for each of the X-axis direction, the Y-axis direction, and the Z-axis direction. The component in each axial direction that is output differs depending on the posture (orientation) of the stick unit 10. Therefore, from these components, the CPU 11 can calculate the roll angle of the stick unit 10 and the rotation angles in the X-axis direction and the Z-axis direction.

  The motion sensor unit 14 (specifically, the CPU 11 that has received the sensor value from the motion sensor unit 14) moves the stick unit 10 based on the above-described sensor value (in other words, the performance operation of the player). To detect.

  The acceleration sensor will be described as an example. The CPU 11 performs a virtual instrument hitting timing (shot) by the stick unit 10 based on the acceleration (or the X-axis, Y-axis, and Z-axis sensor combined values) output from the acceleration sensor. Timing). Further, as will be described later, the CPU 11 detects a swing-down operation or a swing-up operation of the stick unit 10 based on a sensor value from the motion sensor unit 14.

  Returning to FIG. 2, the marker unit 15 is made of a light emitter such as an LED provided on the tip side of the stick unit 10, and emits / lights out under the control of the CPU 11. Specifically, the marker unit 15 emits light based on the marker feature information 12 b read from the ROM 12 by the CPU 11. In this case, since the marker feature information 12b of the stick unit 10A is different from the marker feature information 12b of the stick unit 10B, the camera unit unit 20 uses the position coordinates of the marker unit (first marker) of the stick unit 10A and the stick unit. The position coordinates of the 10B marker portion (second marker) can be distinguished and acquired.

  The data communication unit 16 performs predetermined wireless communication with at least the center unit unit 30. The wireless communication may be performed by any method, and in the present embodiment, wireless communication is performed with the center unit unit 30 by infrared communication.

  The data communication unit 16 may perform wireless communication with the camera unit 20, and perform wireless communication with the stick unit 10 </ b> A and the stick unit 10 </ b> B, or with the monitor unit 40. It is good as well. Further, as the wireless communication, for example, short-range wireless communication such as Bluetooth (registered trademark) may be used.

[Configuration of Camera Unit 20]
FIG. 4 is a block diagram showing a hardware configuration of the camera unit 20 of the performance device 1.

  The camera unit unit 20 includes a CPU 21, a ROM 22, a RAM 23, a marker detection unit 24, and a data communication unit 25.

  As control related to the camera unit unit 20, the CPU 21 mainly performs control to calculate the position coordinate data of the marker units 15 (first marker and second marker) of the stick units 10A and 10B from the captured images. Further, the CPU 21 transmits the calculated position coordinate data and the like to the center unit 30 via the data communication unit 25.

  The ROM 22 stores programs corresponding to various processes executed by the CPU 21. The RAM 23 stores values acquired or generated in the process, such as the position coordinate data of the marker unit 15 detected by the marker detection unit 24. Further, the RAM 23 stores the marker feature information 12b of each of the stick units 10A and 10B given from the center unit unit 30.

  The marker detection unit 24 is used as an imaging unit, and captures a moving image of a performer who performs a performance operation with the stick unit 10 at a predetermined frame rate. Further, the marker detection unit 24 outputs image data for each frame to the CPU 21.

  The camera unit unit 20 has a function of specifying the position coordinates of the marker unit 15 of the stick unit 10 in the imaging space. The specification of the position coordinates of the marker unit 15 may be performed by the marker detection unit 24, or may be performed by the CPU 21. Similarly, the marker characteristic information 12b of the marker unit 15 may be specified by the marker detection unit 24 or may be specified by the CPU 21.

  The data communication unit 25 performs predetermined wireless communication (for example, infrared communication) with at least the center unit unit 30. The data communication unit 25 may perform wireless communication with the stick unit 10 and the monitor unit 40.

[Configuration of Center Unit 30]
FIG. 5 is a block diagram showing a hardware configuration of the center unit 30 of the performance device 1.

  The center unit 30 is used as a control device for the performance device 1. The center unit 30 includes a CPU 31, a ROM 32, a RAM 33, a switch operation detection circuit 34, a display circuit 35, a sound source device 36, and a data communication unit 37.

  The CPU 31 performs control for realizing each function provided in the center unit unit 30. In particular, as functions related to the present invention, the CPU 31 includes a lost detection function 31a, a position estimation function 31b, and a tone generation control function 31c.

  The lost detection function 31a detects that the marker unit 15 of the stick unit 10 has been lost from an image captured by the camera unit unit 20 during the first performance (rehearsal performance).

  The position estimation function 31b estimates the position of the marker unit 15 when the performer strikes the pneumatic instrument with the stick unit 10 during the lost of the marker unit 15 detected by the lost detection function 31a.

  The musical sound generation control function 31c is used to detect the position of the marker unit 15 estimated by the position estimation function 31b when the marker unit 15 is lost during the second performance (the actual performance) performed following the first performance (the rehearsal performance). A predetermined musical tone is generated based on the position.

  Further, the CPU 31 performs communication control between the stick unit 10 and the camera unit unit 20 via the data communication unit 37.

  The ROM 32 stores a program 32a corresponding to various processes executed by the CPU 31. The ROM 32 is provided with a waveform table 32b. In the waveform table 32b, waveform data of various tones, for example, wind instruments such as flute, saxophone and trumpet, keyboard instruments such as piano, stringed instruments such as guitar, bass drums, hi-hats, snares, cymbals, and percussion instruments such as toms. Data is stored in association with position coordinates and the like. The CPU 31 reads out the corresponding waveform data from the waveform table 32b of the ROM 32 on the basis of the position coordinates of the marker unit 15 at the time of shot detection (when a note-on event is received), thereby generating a tone corresponding to the performer's performance operation as a sound source. Generated through device 36. The waveform data may be read from a waveform ROM provided in the sound source device 36.

  The RAM 33 stores values acquired and generated in the process, such as the state of the stick unit 10 (shot detection, etc.) received from the stick unit 10 and the position coordinates of the marker unit 15 received from the camera unit unit 20.

  The switch operation detection circuit 34 is connected to the switch 341 and receives input information via the switch 341. The input information includes, for example, a change in tone volume and tone color, and switching of display on the display device 351.

  The display circuit 35 is connected to the display device 351 and performs display control of the display device 351. In addition to various menu screens, the display device 351 displays a composite image of an image captured by the camera unit 20 and a virtual image of an air musical instrument. It is also possible to send the data displayed on the display device 351 to the monitor unit 40 by wireless communication, and confirm the enlarged display on the screen of the monitor unit 40.

  The tone generator 36 reads waveform data from the waveform table 32b of the ROM 32 in accordance with an instruction from the CPU 31, generates musical tone data from the waveform data, converts the musical tone data into an analog signal, and outputs it from a speaker (not shown).

  The data communication unit 37 performs predetermined wireless communication (for example, infrared communication) with the stick unit 10, the camera unit unit 20, and the monitor unit 40.

  Here, before explaining the processing relating to the present invention, the basic operation of the performance device 1 will be described in order to facilitate understanding.

[Processing of the stick unit 10]
FIG. 6 is a flowchart showing the processing of the stick unit 10 of the performance device 1. Note that the processing shown in this flowchart is executed when the CPU 11 of the stick unit 10 reads the program 12a stored in the ROM 12.

  First, the CPU 11 of the stick unit 10 reads the marker feature information 12b stored in the ROM 12 (step S11). In this case, the CPU 11 of the stick units 10A and 10B reads different marker feature information 12b. The reading of the different marker feature information 12b can be performed by an arbitrary method. For example, it may be performed by the stick units 10 </ b> A and 10 </ b> B communicating directly or via the center unit unit 30. Alternatively, one marker feature information 12b may be associated with each of the stick portions 10A and 10B in advance, and the CPU 11 of the stick portions 10A and 10B may read the unique marker feature information 12b associated with each.

  The CPU 11 temporarily stores the marker feature information 12b read from the ROM 12 in the RAM 13, and then transmits it to the center unit 30 via the data communication unit 16 (step S12). At this time, the CPU 11 transmits the marker feature information 12b to the center unit 30 in association with identification information (stick identification information) that can distinguish the stick portions 10A and 10B.

  Next, the CPU 11 reads out motion sensor information from the motion sensor unit 14, that is, sensor values output by various sensors, and stores them in the RAM 13 (step S13). Thereafter, the CPU 11 performs a posture detection process of the stick unit 10 based on the read motion sensor information (step S14). In the posture detection process, the CPU 11 detects the posture of the stick unit 10, for example, the tilt of the stick unit 10, the displacement of the roll angle and the pitch angle, based on the motion sensor information.

  Subsequently, the CPU 11 performs shot detection processing based on the motion sensor information (step S15). Here, when the performer performs using the stick unit 10, generally, the same operation as the operation of playing an actual musical instrument is performed.

  That is, the performer first swings up the stick unit 10 and then swings it down toward the virtual instrument. Then, just before hitting the stick unit 10 against a virtual musical instrument, a force is applied to stop the operation of the stick unit 10. At this time, since it is assumed that the performer generates a musical sound at the moment when the stick unit 10 is struck on a virtual musical instrument, it is desirable that the performer can generate a musical sound at a timing assumed by the performer. Therefore, in the present embodiment, the musical sound is generated at the moment when the player strikes the stick on the surface of the virtual musical instrument or slightly before that.

  Here, with reference to FIG. 7, an example of the tone generation timing of the musical sound using the stick unit 10 will be described.

  FIG. 7 is a diagram showing a change in output related to the acceleration in the vertical direction of the motion sensor unit 14 when a performance operation is performed using the stick unit 10. In addition, the acceleration in the vertical direction means an acceleration in the vertical direction with respect to the horizontal plane. This may be calculated by decomposing from the acceleration of the Y-axis component. Alternatively, it may be calculated by decomposing from the acceleration in the Z-axis direction (acceleration in the X-axis direction depending on the roll angle). In FIG. 7, a positive acceleration indicates a downward acceleration applied to the stick portion 10, and a negative acceleration indicates an upward acceleration applied to the stick portion 10.

  Even when the stick unit 10 is stationary (portion represented by a), gravitational acceleration is applied to the stick unit 10. For this reason, the motion sensor unit 14 of the stick unit 10 that is stationary detects a constant acceleration in the vertical upward direction, that is, in the negative direction against the gravitational acceleration. It should be noted that the acceleration applied to the stick unit 10 is zero when the stick unit 10 is in a free fall state.

  When the performer lifts the stick unit 10 with the swing-up operation in a stationary state, the player moves in a direction opposite to the gravitational acceleration. For this reason, the acceleration applied to the stick part 10 increases in the minus direction. After that, when the lifting speed is decreased in order to make it stand still, the upward acceleration decreases, and the acceleration in the minus direction of the stick unit 10 detected by the motion sensor unit 14 decreases (part represented by b). Then, the acceleration at the time when the swing-up operation reaches the highest point is only the gravitational acceleration (portion represented near the boundary between b and c).

  When the stick unit 10 reaches the top by the swing-up operation, the performer performs the swing-down operation of the stick unit 10. In the swing-down operation, the stick unit 10 moves in the downward direction. Therefore, the acceleration applied to the stick unit 10 increases in the positive direction more than the negative acceleration detected against the gravitational acceleration. Thereafter, the player decreases the acceleration in the downward direction toward the shot, so that the acceleration applied to the stick unit 10 increases in the negative direction. During this time, only the gravitational acceleration is again applied to the stick portion 10 through the timing at which the swing-down operation reaches the highest speed (portion represented by c).

  Thereafter, when the player applies further acceleration in the swing-up direction to the stick unit 10 toward the shot, the applied acceleration increases in the minus direction. When the shot is finished, the stick unit 10 comes to rest again, and returns to a state in which the acceleration in the negative direction against the gravitational acceleration is detected (portion represented by d).

  In the present embodiment, as the moment when the performer strikes the stick unit 10 on the surface of the virtual musical instrument, the moment when the acceleration in the swing-up direction is applied after the swing-down operation is performed is detected. That is, in the portion represented by d in FIG. 7, the point A in which the acceleration applied from the swinging down state, in other words, the applied acceleration is increased further by a predetermined value in the minus direction, is used as the shot detection timing.

  The shot detection timing is set as the sound generation timing, and when it is determined that the sound generation timing as described above has arrived, the CPU 11 of the stick unit 10 generates a note-on event and transmits it to the center unit unit 30. As a result, the center unit 30 executes a sound generation process to generate a musical sound.

  Returning to FIG. 6, in the shot detection process shown in step S <b> 15, as described above, a note-on event is generated based on the motion sensor information (for example, the X-axis, Y-axis, and Z-axis sensor composite values of the acceleration sensor). . At this time, the generated note-on event may include various kinds of information such as the tone volume, tone color, velocity, envelope, etc. according to the strength of the shot, the magnitude of acceleration, and the like. Note that the tone volume can be obtained from the maximum value of the sensor composite value, for example.

  Further, the CPU 11 performs a process of detecting information (hereinafter referred to as action information) indicating a predetermined operation (action) of the performer based on the motion sensor information, that is, an action detection process (step S16). Here, the action action refers to an action other than a shot performed by the user using a stick. For example, there is a case where the user moves the stick marker to draw a predetermined shape in the imaging space. Detects when double-clicking is performed at the position. For example, when the user slides the stick sideways, the drum set can be changed so as to turn the page.

  The CPU 11 transmits information detected in the processes of steps S14 to S16, that is, posture information, shot information, and action information to the center unit unit 30 via the data communication unit 16 (step S17), and the process of step S13. Return to. At this time, the CPU 11 transmits posture information, shot information, and action information to the center unit unit 30 in association with stick identification information for distinguishing the stick units 10A and 10B.

[Processing of Camera Unit 20]
FIG. 8 is a flowchart showing processing of the camera unit 20 of the performance device 1. Note that the processing shown in this flowchart is executed by the CPU 21 of the camera unit unit 20 reading the program 22a stored in the ROM 22.

  The CPU 21 of the camera unit 20 performs marker detection condition acquisition processing (step S21). In this process, the CPU 21 acquires marker detection condition information transmitted from the center unit 30 and stores it in the RAM 23.

  The marker detection condition information is a condition for detecting each of the marker portions 15 of the stick portions 10A and 10B, and is generated from the marker feature information 12b (see steps S31 and S32 in FIG. 9 described later). Here, as described above, for example, the shape, size, hue, saturation, or luminance of the marker can be used as the marker feature information 12b.

  Next, the CPU 21 performs marker detection condition setting processing (step S22). In this process, the CPU 21 performs various settings for detecting the marker portion 15 of the stick portions 10A and 10B based on the marker detection condition information.

  Next, the CPU 21 performs a first marker detection process and a second marker detection process (steps S23 and S24). In these processes, the CPU 21 detects the position coordinates, size, angle, etc. of the marker unit 15 (first marker) of the stick unit 10A and the marker unit 15 (second marker) of the stick unit 10B detected by the marker detection unit 24. Marker detection information is acquired and stored in the RAM 23. At this time, the marker detection unit 24 detects marker detection information for the marker unit 15 that is emitting light.

  CPU21 transmits the marker detection information acquired by step S23 and step S24 to the center unit part 30 via the data communication part 25 (step S25), and returns to the process of step S23.

[Processing of Center Unit 30]
FIG. 9 is a flowchart showing the processing of the center unit 30 of the performance device 1. Note that the processing shown in this flowchart is executed by the CPU 31 of the center unit 30 reading the program 32a stored in the ROM 32.

  The CPU 31 of the center unit unit 30 receives the marker feature information 12b from the stick unit 10 and stores it in the RAM 33 (step S31). The CPU 31 generates marker detection condition information from the marker feature information and the detection condition set via the switch 341, and transmits the marker detection condition information to the camera unit 20 via the data communication unit 37 (step S32).

  Next, the CPU 31 receives marker detection information for each of the first marker and the second marker from the camera unit 20 and stores them in the RAM 33 (step S33). Further, the CPU 31 receives posture information, shot information, and action information associated with stick identification information from each of the stick units 10A and 10B, and stores them in the RAM 33 (step S34).

  Here, the CPU 31 determines whether or not there is a shot (step S35). In this process, the CPU 31 determines whether or not there is a shot depending on whether or not a note-on event has been received from the stick unit 10. At this time, if it is determined that there is a shot, the CPU 31 performs a shot process (step S36). In the shot process, the CPU 31 reads waveform data corresponding to the position coordinates, size, angle, and the like included in the marker detection information from the waveform table 32b of the ROM 32, and outputs the waveform data to the sound source device 36 together with the volume data included in the note-on event. As a result, the tone generator 36 generates a corresponding musical sound based on the received waveform data.

  On the other hand, after step S36 or when determining NO in step S35, the CPU 31 determines whether or not there is an action based on the action information received from the stick unit 10 (step S35). At this time, if it is determined that there is an action, the CPU 31 performs an action process based on the received action information (step S38), and returns to the process of step S33. When determining that there is no action, the CPU 31 returns to the process of step S33.

The above is the basic operation of the performance device 1.
Although not described in the above flowcharts, the center unit 30 sends an image of the performer photographed by the camera unit 20 during the performance to the center unit 30. It is also possible to synthesize the image of the virtual drum set D shown in FIG.

  The performer can perform the performance while confirming the position of each instrument of the virtual drum set D with the composite image displayed on the monitor unit 40. However, since the performance while watching the monitor unit 40 is game-like, it is preferable to perform without looking at the monitor unit 40 when the reality is required. That is, it is preferable to play with the stick unit 10 as if there is a drum set in front of you.

  Here, as shown in FIG. 10, the position at which the stick unit 10 is hitting is obtained by recognizing the marker unit 15 (light emitting body) provided at the tip of the stick unit 10 on the captured image. However, the marker unit 15 may be hidden by equipment such as the performer's body or a stand during performance, and may not be recognized on the captured image.

  For example, as shown in FIG. 11, when the player crosses his arms and swings the two stick parts 10A and 10B, the tip of the stick part 10A held in one hand is the other hand. May hide. At this time, since the marker unit 15A cannot be recognized on the photographed image 20b, even if a shot is detected by the motion sensor unit 14 of the stick unit 10 while the marker is lost in this way, the instrument struck by the stick unit 10A. Cannot be specified.

  FIG. 12 is a diagram for explaining a method for estimating the position of such a marker in the lost state. In this example, when the stick portion 10A (the marker portion 15) held in one hand of the performer moves from point A to point B, the tip portion of the stick portion 10A is hidden by the other hand, and the marker portion 15A. Represents a lost state.

  Assume that the marker portion 15A is lost between points A and B, and that a shot operation is performed at the point C in the lost state (the player swings down the stick portion 10A and strikes the instrument).

Due to rehearsal performance, the time t ₁ when the marker portion 15A is lost and the position of the point A immediately before the loss (x ₁ , y ₁ ), the time t ₃ when the marker portion 15A is re-recognized, and the position of the point B ( x ₃ , y ₃ ) are obtained in advance. Here, if the stick portion 10A is linearly moved between A and B at a constant speed, the relationship between the position between A and B, times t ₁ and t ₃ , and the shot occurrence time t ₂ during lost C The position (x ₂ , y ₂ ) of the point can be estimated.

This will be described in detail below.
Now, as shown in FIG. 10, it is assumed that the performer performs a drum performance using the stick unit 10 at a position away from the camera unit 20 by a predetermined distance T. At that time, the shooting range 20b of the camera unit unit 20 is assumed. It is assumed that an image of a player photographed in the center is sent to the center unit 30 in real time.

  FIGS. 13 and 14 are flowcharts showing the performance processing of the center unit 30 of the performance device 1. Note that the processing shown in this flowchart is executed by the CPU 31 of the center unit 30 reading the program 12a stored in the ROM 12.

  When the performer sets the performance mode by operating the switch 341 provided in the center unit 30, the rehearsal performance is first started as the first performance mode (YES in step S41). In the rehearsal performance, it is assumed that the performer holds the stick unit 10 and performs the same performance operation as the actual performance. Specifically, as described with reference to FIG. 1, the performer has two stick portions 10A and 10B, and performs an operation (shot operation) of swinging down the stick portions 10A and 10B toward each instrument of the virtual drum set D. .

  When the rehearsal performance is started, the CPU 31 of the center unit 30 starts a timer (not shown) and measures the performance time (step S42). Further, the CPU 31 detects the marker portion 15 provided at the tip portion of the stick portion 10 from the image photographed by the camera unit portion 20 (step S43).

  While the marker unit 15 is detected on the photographed image (NO in step S44), the CPU 31 specifies a musical instrument from the position coordinates of the marker unit 15 (step S45), and is provided in the stick unit 10 at that time. The shot motion is detected by the obtained motion sensor unit 14 (step S46). Specifically, as described with reference to FIG. 7, for example, if an acceleration sensor is used as the motion sensor unit 14, the player swings up the stick unit 10 from the change in signal of the acceleration sensor and then swings it down toward the instrument. Detecting a movement.

  When a shot motion is detected by the motion sensor unit 14 (YES in step S46), the CPU 31 generates a sound of a musical instrument corresponding to the marker position via the sound source device 36 (step S47). The identification of the musical instrument in step S45 may be performed later after the shot is detected, but if it is performed in advance, the processing time after the shot detection can be shortened, and the delay time until the sound generation after the shot detection is reduced. It can be shortened.

  On the other hand, if the marker unit 15 is lost during performance and cannot be recognized on the captured image (YES in step S44), the CPU 31 obtains the time when the marker unit 15 is lost from the timer and captures the captured image. The position of the marker immediately before the lost is acquired, and the time and position data are associated and recorded in the first lost management table 33a of the RAM 33 (step S39).

  Here, when a shot motion is detected by the motion sensor unit 14 during the lost (YES in step S50), the CPU 31 acquires the time at that time from the timer and stores it in the first lost management table 33a of the RAM 33. Record (step S51). In this case, no musical sound is generated because the musical instrument cannot be specified by the marker position. Further, when a plurality of shot operations are detected during one lost, the time for each shot is recorded in the first lost management table 33a.

  When the marker unit 15 is lost and appears again on the captured image and shifts to a re-recognizable state (YES in step S52), the CPU 31 obtains the time from the timer and re-recognizes the captured image. The marker position at the time of completion is acquired, and the time and position data are associated and recorded in the first lost management table 33a of the RAM 33 (step S53).

  Such processing is repeatedly performed during the rehearsal performance. In the meantime, when the marker unit 15 is lost, the lost time and marker position, the re-recognition time and marker position, and, if there is a shot operation, the data in the RAM 33 including the data at that time are included. 1 in the lost management table 33a.

FIG. 15 shows an example of the first lost management table 33a.
The lost time and marker position, the time and marker position when re-recognized after the lost, and the time when the shot occurred are recorded in the first lost management table 33a as a series of lost data. At this time, the marker position at the time of occurrence of the shot has not been recorded yet.

  When the rehearsal performance ends (YES in step S48), the CPU 31 stops the timer and ends the performance time measurement (step S54). Then, the CPU 31 executes data processing for the lost occurrence period as follows (step S55).

  That is, the CPU 31 first checks the presence / absence of a shot operation during the occurrence of lost with reference to the first lost management table 33a. When there is a shot operation during the occurrence of lost, the CPU 31 reads the lost time and marker position, the time and marker position at the time of re-recognition, and the shot occurrence time recorded in the first lost management table 33a. Based on these data, the marker position at the shot occurrence time is estimated.

Referring to the example of FIG. 15, by rehearsal performance, the time t ₁ when the marker portion 15A of one stick portion 10A is lost and the position (x ₁ , y ₁ ) of the point A immediately before the lost can be re-recognized after the loss. It is assumed that the time t _{3 at} that time, the position of the point B (x ₃ , y ₃ ), and the time t ₂ at the time of the shot occurrence are recorded in the first lost management table 33a.

Here, if the stick portion 10A moves linearly between A and B at a constant speed (see FIG. 12), the time interval between the point C and the point A where the shot operation occurred during the movement (t ₂ −t _1). ) And the time interval (t ₃ -t ₂ ) between point B and point C, the position (x ₂ , y ₂ ) of point C can be obtained.

In this case, as shown in FIG. 16A, for example, the time interval between point C and point A (t ₂ -t ₁ ) is 1 second, and the time interval between point B and point C (t ₃ -t _{If 2} ) is also 1 second, the ratio between the two is 1: 1. Therefore, since the position (x ₂ , y ₂ ) of the point C is a point that internally divides the intermediate position between A and B, that is, between A and B into 1: 1, x ₂ = (x ₁ + x ₃ ) / 2, y ₂ = (y ₁ + y ₃ ) / 2.

Further, as shown in FIG. 16B, for example, the time interval between point C and A (t ₂ -t ₁ ) is 2 seconds, and the time interval between point B and C (t ₃ -t ₂ ). Is 1 second, the ratio of the two is 2: 1. Accordingly, the position of the point C _(x 2, _{y 2)} is between A-B 2: from a point which internally divides _{1, x 2 = x 1/} 3 + 2x 3/3 = (x 1 + 2x 3) / _3, is obtained by _{y 2 = y 1/3 +} 2y 3/3 = (y 1 + 2y 3) / 3.

This way, the location of point C at the time of shot generation _(x 2, _{y 2)} is calculated, CPU 31 records the first lost management table 33a in association with the position data at time _{t 2.} Then, the same processing is performed for all the lost occurrence period data recorded in the first lost management table 33a. When processing is performed for all the data in the lost occurrence period, the processing here is finished.

  Following the rehearsal performance, the actual performance is started as the second performance mode. The start of the actual performance may be instructed by the operation of the switch 341 provided in the center unit unit 30 in the same manner as the start of the rehearsal performance in step S41. Note that both the rehearsal performance in step S41 and the start of the actual performance include various operations such as shaking the stick unit 10 in a predetermined direction, drawing a predetermined shape, and operating a switch of the stick unit 10. It is possible to give a start instruction by the method.

  When the actual performance is started (YES in step S56), the CPU 31 of the center unit 30 starts an internal timer (not shown) and measures the performance time as in the rehearsal performance (step S57). Sometimes, the marker unit 15 provided at the tip of the stick unit 10 is detected from the image photographed by the camera unit unit 20 (step S58).

  While the marker unit 15 is detected on the photographed image (NO in step S59), the CPU 31 specifies a musical instrument from the position coordinates of the marker unit 15 (step S60), and is provided in the stick unit 10 at that time. The shot motion is detected by the obtained motion sensor unit 14 (step S61). When a shot motion is detected by the motion sensor unit 14 (YES in step S61), the CPU 31 generates a musical instrument sound corresponding to the marker position via the sound source device 36 (step S62). The identification of the instrument in step S60 may be performed later after the shot is detected, but if it is performed in advance, the processing time after the shot detection can be shortened, and the delay time until the sound generation after the shot detection is reduced. It can be shortened.

  If the marker unit 15 is lost during performance and cannot be recognized on the captured image (YES in step S59), the CPU 31 executes the following lost handling process.

  That is, the CPU 31 acquires the time when the marker unit 15 is lost from the timer, acquires the marker position immediately before the lost from the photographed image, and approximates the rehearsal performance with reference to the first lost management table 33a of the RAM 33. It is searched whether or not lost has occurred at the time / marker position (step S63).

  If there is a lost corresponding to the rehearsal performance, the CPU 31 determines whether or not a shot motion is detected by the motion sensor unit 14 during the lost (step S64). When the shot operation is detected (YES in step S64), the CPU 31 confirms whether or not the shot occurrence time is close to the shot occurrence time during the rehearsal performance recorded in the first lost management table 33a.

  If it is close to the shot occurrence time at the time of rehearsal performance, the CPU 31 determines that the shot operation is in the lost time at the timing close to the rehearsal performance, reads the marker position at the time from the first lost management table 33a, and The sound of the musical instrument corresponding to the marker position is generated via 36 (step S65).

For example, it is assumed that lost occurs at the position of point A (x ₁ , y ₁ ) around time t ₁ from the start of performance during the actual performance. Assuming that the actual performance is performed in the same manner as the rehearsal performance, it is predicted that the lost occurred at the same timing even during the rehearsal performance. Therefore, if there is a shot operation time t ₂ before and after in its lost, the position of the C points obtained during rehearsal playback (x _{1, y 1)} can be inferred to be a marker position at the time of the shot, corresponding to the marker position Can generate the sound of the instrument.

  In this way, by confirming whether or not the loss occurred at the timing close to the rehearsal performance, and using the lost information and shot information at the time of the rehearsal performance, even if a marker loss occurs during the actual performance, the musical sound can be reproduced. Can be generated.

  Further, by searching and using the lost information at the time / marker position that approximates the lost during the actual performance from the lost information during the rehearsal performance recorded in the first lost management table 33a, another timing can be obtained. It is possible to prevent a malfunction that misuses the lost data generated in the process and generates another musical sound. Furthermore, even when a shot motion occurs a plurality of times during one lost, the shot information at the time approximate to the shot motion detected during the actual performance is retrieved from the first lost management table 33a and used. Thus, it is possible to prevent a malfunction in which another musical sound is generated by erroneous use of shot operation data generated at another timing.

  In step S63, if the lost time and marker position that occurred during the actual performance does not exist during the rehearsal performance, the marker position in the lost cannot be estimated and no sound is produced. And

  After the lost, when the marker unit 15 is re-recognized (YES in step S66), the process returns to the normal performance process. When a predetermined performance time has elapsed, the series of processes here ends (step S67).

  In this way, even if the performer performs a performance operation in which the marker unit 15 of the stick unit 10 is easily lost, such as by crossing arms, the position where the marker unit 15 is lost by rehearsal performance is obtained in advance. In the actual performance, it is possible to continue the performance by generating a predetermined musical sound in accordance with the movement of the stick unit 10.

(Second Embodiment)
Next, a second embodiment of the present invention will be described.
In the first embodiment, under the assumption that the stick moves linearly at a constant speed during the loss, the player hits the air instrument during the loss based on the time before and after the loss and the marker position. The position (shot position) was assumed. However, when the player strikes the stick unit 10 with a relatively fast movement or moves in various directions, the shot position can be accurately estimated only from the time and positional relationship before and after the lost. Is difficult. Therefore, in the second embodiment, the shot position can be accurately estimated even if the stick portion 10 is repeatedly hit with a fast movement or the like.

  Since the basic apparatus configuration is the same as that of the first embodiment, a method for dealing with marker lost will be described here.

  17 and 18 are flowcharts showing the performance processing of the center unit 30 of the performance device 1. Note that the processing shown in this flowchart is executed by the CPU 31 of the center unit 30 reading the program 12a stored in the ROM 12.

  Similar to the first embodiment, first, the rehearsal performance is started as the first performance mode (YES in step S71). The CPU 31 of the center unit 30 measures the performance time by starting a timer (not shown) (step S72), and a marker provided at the tip of the stick unit 10 from the image taken by the camera unit 20 at this time. The unit 15 is detected (step S73).

  While the marker unit 15 is detected on the photographed image (NO in step S74), the CPU 31 specifies a musical instrument from the position coordinates of the marker unit 15 (step S75), and is provided in the stick unit 10 at that time. The shot motion is detected by the detected motion sensor unit 14 (step S76). When a shot motion is detected by the motion sensor unit 14 (YES in step S76), the CPU 31 generates a sound of a musical instrument corresponding to the marker position via the sound source device 36 (step S77). The identification of the instrument in step S75 may be performed later after the shot is detected, but if it is performed in advance, the processing time after the shot detection can be shortened, and the delay time until the sound generation after the shot detection is reduced. The points that can be shortened are the same as those described in the first embodiment.

  On the other hand, when the marker unit 15 is lost during performance and cannot be recognized on the captured image (YES in step S74), the following processing is performed in the second embodiment.

  That is, the CPU 31 acquires the time when the marker unit 15 is lost from the timer, acquires the marker position immediately before the lost from the captured image, associates the time and position data, and the second lost management table 33b of the RAM 33. (Step S79). Further, the CPU 31 executes the process of step S80 for each unit time while looping the process of step S80 to step S83 during the lost, and the measurement data obtained from the motion sensor unit 14 each time is obtained from the second time. Record in the lost management table 33b (step S80).

  Here, when a shot motion is detected by the motion sensor unit 14 during the lost (YES in step S81), the CPU 31 obtains the time at that time from the timer and stores it in the second lost management table 33b of the RAM 33. Record it (step S82). In this case, no musical sound is generated because the musical instrument cannot be specified by the marker position. Further, when a plurality of shot operations are detected during one lost, the time for each shot is recorded in the second lost management table 33b.

  After the marker unit 15 is lost, the above process is repeated until the marker unit 15 appears on the captured image again and shifts to a re-recognizable state, and the measurement data of the motion sensor unit 14 obtained in the meantime is stored in the second lost management table 33b. It will be recorded continuously.

FIG. 19 shows an example of the second lost management table 33b.
The lost time and marker position, the time when a shot occurs, and the sensor measurement data are recorded in the second lost management table 33b as a series of lost data. In this case, if, for example, a three-axis acceleration sensor is used as the motion sensor unit 14, the X-axis, Y-axis, and Z-axis direction accelerations of the stick unit 10 being lost are detected, and each of these axis directions is detected. The detected acceleration measurement data is continuously recorded. At this time, the marker position at the time of occurrence of the shot has not been recorded yet.

  When the rehearsal performance ends (YES in step S78), the CPU 31 stops the timer and ends the measurement of the performance time (step S84). Then, the CPU 31 executes data processing for the lost occurrence period as follows (step S85).

  That is, the CPU 31 first checks the presence / absence of a shot operation during the occurrence of lost with reference to the second lost management table 33b. If there is a shot operation during the occurrence of lost, the CPU 31 displays the lost time and marker position, the measurement data of the motion sensor unit 14 being lost, and the shot occurrence time recorded in the second lost management table 33b. Reading and estimating the marker position at the shot occurrence time based on these data.

Referring to the example of FIG. 19, the time t ₁ when the marker portion 15A of one stick portion 10A is lost by rehearsal performance, the position of the point A (x ₁ , y ₁ ) immediately before the loss, and the motion sensor in the lost state measurement data parts 14 and the time t ₂ when the shot occurs is recorded in the second lost management table 33b.

Here, if the stick unit 10A moves between A and B (see FIG. 12), it can be determined from the measurement data of the motion sensor unit 14 how the stick unit 10 moved during that time. Therefore, if we track the movement of the marker portion 15A based on the measurement data by the position of the time t ₁ and point A at Lost (x _{1, y 1)} to the starting point, shot operation is a time t ₂ which is detected The position (x ₂ , y ₂ ) of the point C can be estimated as the marker position at the time point.

More specifically, for example, when the motion sensor unit 14 is composed of an acceleration sensor, the acceleration is recorded continuously from the time t ₁ at the time of lost and the position of the point A (x ₁ , y ₁ ). the velocity data in each unit time obtained by integrating the measured data of the sensor that accumulates (integrates) each unit time, calculates the movement of the stick 10 until time t ₂ when the shot operation is detected, the shot The marker position C point (x ₂ , y ₂ ) at the time t ₂ when the motion is detected can be estimated.

This way, the location of point C at the time of shot generation _(x 2, _{y 2)} is calculated, CPU 31 records the second lost management table 33b in association with the position data at time _{t 2.} Then, the same processing is performed on the data of all lost occurrence periods recorded in the second lost management table 33b. When processing is performed for all the data in the lost occurrence period, the processing here is finished.

  Following the rehearsal performance, the actual performance is started as the second performance mode. As in the first embodiment, the actual performance and the rehearsal performance may be started by operating the switch 341 provided in the center unit 30, or the stick unit 10 may be shaken in a predetermined direction. It is possible to give a start instruction by various methods such as drawing a predetermined shape or operating a switch of the stick unit 10.

  When the actual performance is started (YES in step S86), as in the rehearsal performance, the CPU 31 of the center unit 30 starts an internal timer (not shown) and measures the performance time (step S87). Sometimes the marker unit 15 provided at the tip of the stick unit 10 is detected from the image taken by the camera unit 20 (step S88).

  While the marker unit 15 is detected on the photographed image (NO in step S89), the CPU 31 specifies a musical instrument from the position coordinates of the marker unit 15 (step S90), and is provided in the stick unit 10 at that time. The shot motion is detected by the obtained motion sensor unit 14 (step S91). If a shot motion is detected by the motion sensor unit 14 (YES in step S91), the CPU 31 generates a musical instrument sound corresponding to the marker position via the sound source device 36 (step S92). Note that the specification of the musical instrument in step S90 may be performed after the shot is detected or may be performed in advance, as described in the first embodiment.

  If the marker unit 15 is lost during performance and cannot be recognized on the captured image (YES in step S89), the CPU 31 executes the following lost handling process.

  That is, the CPU 31 acquires the time when the marker unit 15 is lost from the timer, acquires the marker position immediately before the lost from the photographed image, and approximates the rehearsal performance with reference to the first lost management table 33a of the RAM 33. It is searched whether or not lost has occurred at the time / marker position (step S93).

  If there is a lost corresponding to the rehearsal performance, the CPU 31 determines whether or not a shot motion is detected by the motion sensor unit 14 during the lost (step S94). When the shot operation is detected (YES in step S94), the CPU 31 confirms whether or not the shot occurrence time is close to the shot occurrence time during the rehearsal performance recorded in the second lost management table 33b.

  If it is close to the shot occurrence time during the rehearsal performance, the CPU 31 determines that the shot operation is a lost shot that occurs at a timing close to the rehearsal performance, reads the marker position at that time from the second lost management table 33b, and The sound of the musical instrument corresponding to the marker position is generated via 36 (step S95).

For example, it is assumed that lost occurs at the position of point A (x ₁ , y ₁ ) around time t ₁ from the start of performance during the actual performance. Assuming that the actual performance is performed in the same manner as the rehearsal performance, it is predicted that the lost occurred at the same timing even during the rehearsal performance. Therefore, if there is a shot operation time t ₂ before and after in its lost, the position of the C points obtained during rehearsal playback (x _{1, y 1)} can be inferred to be a marker position at the time of the shot, corresponding to the marker position Can generate the sound of the instrument.

  In this way, by confirming whether or not the loss occurred at the timing close to the rehearsal performance, and using the lost information and shot information at the time of the rehearsal performance, even if a marker loss occurs during the actual performance, the musical sound can be reproduced. Can be generated.

  Further, by searching and using the lost information at the time / marker position that approximates the lost during the actual performance from the lost information during the rehearsal performance recorded in the second lost management table 33b, another timing can be obtained. It is possible to prevent a malfunction that misuses the lost data generated in the process and generates another musical sound. Furthermore, even when a shot motion occurs a plurality of times during one lost, the shot information at the time approximate to the shot motion detected during the actual performance is retrieved from the second lost management table 33b and used. Thus, it is possible to prevent a malfunction in which another musical sound is generated by erroneous use of shot operation data generated at another timing.

  In step S93, if the lost time and marker position that occurred during the actual performance does not exist during the rehearsal, the marker position in the lost cannot be estimated, so no sound is produced. And

  After the lost, when the marker unit 15 is re-recognized (YES in step S66), the process returns to the normal performance process. When a predetermined performance time has elapsed, the series of processes here ends (step S67).

  In this way, even if the performer performs a performance operation in which the marker unit 15 of the stick unit 10 is easily lost, such as by crossing arms, the position where the marker unit 15 is lost by rehearsal performance is obtained in advance. In the actual performance, it is possible to continue the performance by generating a predetermined musical sound in accordance with the movement of the stick unit 10.

  Furthermore, in the second embodiment, since the measurement data of the motion sensor unit 14 is used to estimate the lost position, the lost position can be estimated even if the player moves the stick unit 10 quickly and repeatedly hits it. There are advantages such as.

In the second embodiment, the lost time and marker recorded in the second lost management table 33b when a shot has been detected during the rehearsal performance in the lost period data processing in step S85. The marker position C point (x ₂ , y ₂ ) at the time of occurrence of the shot is estimated based on the position, the measurement data of the motion sensor unit 14 being lost, and the shot occurrence time, but is not limited thereto.

  That is, for example, regardless of the occurrence of a shot, the marker position at that time is estimated and calculated for each unit time for all periods in which the marker is lost, and all of them are recorded in the second lost management table 33b. It is also possible to keep it.

  With this configuration, even when a shot is performed at a timing that was not detected during rehearsal performance during marker lost during the actual performance, the estimated position of the marker at the time of the shot occurrence is stored in the second lost management table. It is possible to read out from 33b and generate the sound of the musical instrument corresponding to the position.

  As described above, according to the present embodiment, by determining the position where the marker is lost in advance during rehearsal performance, even if the marker is lost during the actual performance, it is possible to determine the instrument that the player has struck. Musical sound can be generated.

  In the above-described embodiment, the acceleration sensor is described as an example of the direction detection sensor provided in the stick unit 10. However, the same applies to other sensors such as an angular velocity sensor. The effect is obtained.

  In the above embodiment, an air drum is described as an example of a virtual percussion instrument. However, the present invention is not limited to this, and any instrument that generates a musical sound by swinging / raising the stick unit 10 can be used. It can be applied to other musical instruments such as xylophone.

  In addition, any processing among the processing to be performed by the stick unit 10, the camera unit unit 20, and the center unit unit 30 in the above embodiment is performed by other units (stick unit 10, camera unit unit 20, and center unit unit 30. ). For example, the shot detection process (FIG. 6) to be performed by the CPU 11 of the stick unit 10 may be performed by the CPU 31 of the center unit unit 30.

  In short, several embodiments of the present invention have been described, but these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

  In addition, the methods described in the above-described embodiments are, for example, magnetic disks (flexible disks, hard disks, etc.), optical disks (CD-ROM, DVD, etc.), recording media such as semiconductor memories, as programs that can be executed by a computer. Can be applied to various devices, or transmitted by a communication medium and applied to various devices. A computer that implements this apparatus reads the program recorded on the recording medium, and executes the above-described processing by controlling the operation by this program.

Hereinafter, the invention described in the scope of claims of the present application will be appended.
[1]
A performance member having a marker, in which a performance instruction of an air musical instrument is issued in accordance with a player's operation;
Photographing means for photographing an imaging space in which the performance member exists;
Lost detection means for detecting that the marker has been lost from an image photographed by the photographing means during the first performance;
Position estimation means for estimating the position of the marker when the player strikes the air musical instrument with the performance member during the lost of the marker detected by the lost detection means;
A musical tone generation control means for generating a predetermined musical tone based on the position of the marker estimated by the position estimation means when the marker is lost during a second performance.

[2]
Recording means for recording the time and position when the marker is lost, the time when the performance member hits the air instrument during the lost time, the time and position when the marker can be re-recognized after lost,
The position estimating means is
The performance device according to [1], wherein the position of the marker at the shot occurrence time is estimated from the relationship between the time and position before and after the lost of the marker recorded in the recording means.

[3]
A sensor for detecting the movement of the performance member;
The time and position when the marker is lost, the time when the performance member hits the air musical instrument during the lost time, and the movement of the performance member detected by the sensor during the lost time are recorded. A recording means for continuously recording measurement data;
The position estimating means is
Estimating the position of the marker at the shot occurrence time by tracking the movement of the marker based on the measurement data of the sensor from the time and position at the time of the loss of the marker recorded in the recording means. The performance device according to [1], which is characterized.

[4]
The musical sound generation control means is
When it is determined whether or not the marker has been lost at a timing close to the first performance during the second performance, and the marker has been lost at a timing close to the first performance The performance device according to any one of [1] to [3], wherein a predetermined musical tone is generated based on the position of the marker estimated by the position estimation means.

[5]
An air musical instrument performance method executed by a computer,
An imaging step in which an imaging space in which a performance member having a marker is present, in which an instruction to perform an air musical instrument is issued in response to a player's operation, is captured by an imaging unit;
A lost detection step of detecting that the marker has been lost from an image photographed by the photographing means during the first performance;
A position estimation step for estimating the position of the marker when the player strikes the air musical instrument with the performance member during the lost of the marker detected by the lost detection step;
A musical tone generation control step for generating a predetermined musical tone based on the position of the marker estimated by the position estimation step when the marker is lost during a second performance.

[6]
A program for performance control of an air musical instrument executed by a computer,
The computer
An imaging means for imaging an imaging space where a performance member having a marker is present, in which a performance instruction of an air musical instrument is issued in response to a player's operation,
Lost detection means for detecting that the marker has been lost from an image photographed by the photographing means during the first performance;
Position estimation means for estimating the position of the marker when the player strikes the air musical instrument with the performance member during the lost of the marker detected by the lost detection means;
A computer-readable program for functioning as a tone generation control means for generating a predetermined tone based on the position of the marker estimated by the position estimation means when the marker is lost during a second performance.

  DESCRIPTION OF SYMBOLS 1 ... Performance apparatus, 10 ... Stick part, 11 ... CPU, 12 ... ROM, 12a ... Program, 12b ... Marker characteristic information, 13 ... RAM, 14 ... Motion sensor part, 15 ... Marker part, 16 ... Data communication part, 17 Operation buttons, 18 Vibrator function, 20 Camera unit, 20a Shooting lens, 20b Shooting range, 21 CPU, 22 ROM, 22a Program, 23 RAM, 24 Marker detection unit, 25 Data Communication unit 26, 27 ... lamp, 30 ... center unit, 31 ... CPU, 31a ... lost detection function, 31b ... position estimation function, 31c ... musical sound generation control function, 32 ... ROM, 32a ... program, 32b ... waveform table, 33 ... RAM, 33a ... first lost management table, 33b ... second lost management table, 34 Switch operation detecting circuit, 341 ... switch, 35 ... display circuit, 351 ... display, 36 ... sound source device, 37 ... communication unit, 40 ... monitor.

Claims (6)

A performance member having a marker, in which a performance instruction of an air musical instrument is issued in accordance with a player's operation;
Photographing means for photographing an imaging space in which the performance member exists;
Lost detection means for detecting that the marker has been lost from images photographed by the photographing means during a first performance and a second performance having the same performance as the first performance ;
Position estimation means for estimating the position of the marker when the player strikes the air musical instrument with the performance member during the lost of the marker detected by the lost detection means during the first performance ;
If you lost the said marker and when the second performance even after estimation of the position of the marker by the position estimating means during the first play, the location of the markers lost during the second performance And a corresponding musical tone generation control means for generating a predetermined musical tone based on the position of the marker estimated by the position estimating means during the first performance. Recording means for recording the time and position when the marker is lost, the time when the performance member hits the air instrument during the lost time, the time and position when the marker can be re-recognized after lost,
The position estimating means is
2. The performance apparatus according to claim 1, wherein the position of the marker at the shot occurrence time is estimated from the relationship between the time before and after the lost of the marker recorded in the recording means and the position. A sensor for detecting the movement of the performance member;
Measurement data indicating the time and position when the marker is lost, the time when the performance member hits the air instrument during the loss, and the movement of the performance member detected by the sensor during the loss Recording means for continuously recording
The position estimating means is
Estimating the position of the marker at the shot occurrence time by tracking the movement of the marker based on the measurement data of the sensor from the time and position at the time of the loss of the marker recorded in the recording means. The performance device according to claim 1, wherein: The musical sound generation control means is
When it is determined whether or not the marker has been lost at a timing close to the first performance during the second performance, and the marker has been lost at a timing close to the first performance 4. The performance apparatus according to claim 1, wherein a predetermined musical tone is generated based on the position of the marker estimated by the position estimating means. An air musical instrument performance method executed by a computer,
An imaging step in which an imaging space in which a performance member having a marker is present, in which an instruction to perform an air musical instrument is issued in response to a player's operation, is captured by an imaging unit;
A lost detection step of detecting that the marker has been lost from images photographed by the photographing means during the first performance and during the second performance of the same performance as the first performance ;
A position estimation step of estimating the position of the marker when the player strikes the air musical instrument with the performance member during the lost of the marker detected by the lost detection step during the first performance ;
If the marker is lost during the second performance after the position is estimated during the first performance and the marker is lost during the second performance, the position of the marker lost during the second performance And a corresponding musical tone generation control step for generating a predetermined musical tone based on the position of the marker estimated by the position estimation step during the first performance. A program for performance control of an air musical instrument executed by a computer,
The computer
An imaging means for imaging an imaging space where a performance member having a marker is present, in which a performance instruction of an air musical instrument is issued in response to a player's operation,
Lost detection means for detecting that the marker has been lost from images photographed by the photographing means during the first performance and during the second performance of the same performance as the first performance ;
Position estimation means for estimating the position of the marker when the player strikes the air musical instrument with the performance member during the lost of the marker detected by the lost detection means during the first performance ;
If you lost the said marker and when the second performance even after estimation of the position of the marker by the position estimating means during the first play, the location of the markers lost during the second performance A computer-readable program for functioning as a tone generation control unit that generates a predetermined tone based on the position of the marker estimated by the position estimation unit during the first performance .

Images