JP2013195626A - Musical sound generating device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a musical sound generating device, even when it is hard to detect a performance operation element due to performance environment, capable of accommodating thereto without changing a performance place as far as possible.SOLUTION: A musical sound generating device 1 includes: a stick part 10 that is provided with a marker part 15 and is operated by a player; an imaging part that detects a position of the marker part 15 in the performance operation, as an operation position of the stick part 10; a sound production control part that, when the operation position in the performance operation of the stick part 10 is within a prescribed performance area, gives a sound production instruction to a sound production part for producing a prescribed musical sound; a maker detection/non-detection determination part for determining whether or not the imaging part detects the position of the marker part 15; and a stick control part that, when the imaging part does not detect the position of the marker part 15, changes a light-emitting state of the marker part 15.

Description

  The present invention relates to a musical sound generator.

2. Description of the Related Art Conventionally, a musical sound generating device (musical instrument game device) has been proposed that generates an electronic sound (musical sound) corresponding to a performance operation when a performance operation of a performer (user) is detected (see, for example, Patent Document 1). Also, in such a device that emits an electronic sound, when a player performs a motion of holding a member (controller) corresponding to a stick of a percussion instrument in his / her hand, a device (virtual drum device) that produces a percussion instrument sound accordingly. In such an apparatus, a sensor is provided in the controller, and when the performer holds and shakes this member by hand, the sensor detects the operation, and the detection result The percussion instrument sound corresponding to the sound is generated.
For example, an actual musical instrument such as a percussion instrument is heavy and requires a large space for installation, so that it is difficult to easily perform and practice at home.
In this regard, according to the musical sound generator that performs the virtual musical instrument performance by detecting the musical performance of the performer and generating a musical sound according to this, the musical sound of the musical instrument is generated without the need for an actual musical instrument. Therefore, it is possible to easily enjoy the performance without being restricted by the performance place and the performance space.

Japanese Patent No. 3599115 Japanese Patent No. 4586525

However, in such a virtual instrument playing device, when a camera or the like is used as a means for detecting the movement of a performance operator such as a stick unit, the performance operation is performed depending on the color of the wall of the performance place, the lighting conditions, etc. It may be difficult to detect children.
Conventionally, when the performance operator cannot be detected, it is necessary to change the performance place, which is inconvenient for the performer.

  The present invention has been made in view of the circumstances as described above, and provides a musical sound generating apparatus that can cope with the performance operator without changing the performance place as much as possible even when it is difficult to detect the performance operator depending on the performance environment. It is the purpose.

In order to solve the above-described problem, the musical sound generating device of the present invention includes:
A performance operator equipped with a light emitting section;
Position detecting means for detecting a position of the light emitting unit on a predetermined plane during a performance operation;
Detectability determination means for determining whether or not the position of the light emitting unit on the predetermined plane can be detected by the position detection means;
A light emission state changing unit that changes a light emission state of the light emitting unit when it is determined by the detectability determining unit that a position on the predetermined plane of the light emitting unit cannot be detected;
When a position on the predetermined plane of the light emitting unit is detected by the detection possibility determination unit, and the detected position is included in a performance area provided on the predetermined plane, a musical tone generating unit that generates a musical tone Pronunciation control means for giving pronunciation instructions to,
It is characterized by having.

  According to the present invention, even when it is difficult to detect a performance operator depending on the performance environment, there is an effect that it is possible to respond without changing the performance place as much as possible.

(A) is a figure which shows the outline | summary of one Embodiment of the musical sound generator based on this invention, (B) is the figure which showed the virtual drum set in this embodiment. It is a block diagram which shows the hardware constitutions of the stick part shown in FIG. It is a perspective view of the stick part shown in FIG. It is a block diagram which shows the hardware constitutions of the camera unit part shown in FIG. It is a block diagram which shows the hardware constitutions of the center unit part shown in FIG. It is a figure showing the change of the acceleration of the vertical direction of a motion sensor part. It is explanatory drawing which shows the mode of the marker part at the time of shaking a stick part on the wall surface of a performance place. (A) shows an example in which the light emission intensity of the marker part shown in FIG. 7 is changed, (B) shows an example in which the light emission color of the marker part shown in FIG. 7 is changed, and (C) shows in FIG. It is a figure which shows the example which changed the luminescent color and luminescent intensity of the marker part to show. It is a flowchart which shows the flow of a process of a stick part. It is a flowchart which shows the flow of a process of a camera unit part. It is a flowchart which shows the flow of a process of a center unit part.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. The embodiments described below are given various technically preferable limitations for carrying out the present invention, but the scope of the present invention is not limited to the following embodiments and illustrated examples.

[Outline of musical tone generator]
First, with reference to FIG. 1 (A) and FIG. 1 (B), the outline | summary of one Embodiment of the musical sound generator based on this invention is demonstrated.
FIG. 1A is a diagram showing an outline of a device configuration of a musical sound generating device according to the present embodiment.
As shown in FIG. 1A, the musical sound generator 1 of the present embodiment is configured to include stick units 10A and 10B, a camera unit unit 20, and a center unit unit 30. In addition, in order to implement | achieve the virtual drum performance using two sticks, the musical tone generator 1 of this embodiment shall be provided with two stick parts 10A and 10B, However, the number of stick parts is this. The number is not limited, and may be one or three or more. Hereinafter, when the stick portions 10A and 10B are not distinguished from each other, they are collectively referred to as “stick portion 10”.

The stick unit 10 is a rod-like member extending in the longitudinal direction, and functions as a performance operator operated by the performer. That is, the performer performs a performance operation by holding one end (base side) of the stick unit 10 and performing a swing-down operation centering on the wrist or the like.
In this embodiment, in order to detect such a player's performance, various sensors such as an acceleration sensor (motion sensor unit 14 described later, see FIG. 2) are provided at the other end (tip side) of the stick unit 10. It has been.
Detection results detected by various sensors (for example, raw data acquired by various sensors) are transmitted to the center unit 30.
In addition, a marker unit 15 is provided as a light emitting unit on the distal end side of the stick unit 10 so that the camera unit unit 20 can easily determine the distal end of the stick unit 10 during imaging.

The camera unit 20 is an imaging device including an imaging unit 24 (for example, an optical camera, see FIG. 4) that captures a moving image of a performer who performs a performance operation while holding the stick unit 10 at a predetermined frame rate. In this embodiment, the imaging unit 24 functions as a position detection unit that detects the position of the marker unit 15 during the performance operation as the operation position of the stick unit 10.
In the present embodiment, the camera unit unit 20 detects the position of the marker unit 15 of the stick unit 10 from the image acquired by the imaging unit 24 serving as a position detection unit, so that the stick unit 10 based on the performance performance of the performer. As the state of (performance operator), the operation position during the performance operation of the stick unit 10 is detected.

When the center unit 30 receives the detection results detected by the various sensors (the motion sensor unit 14 described later) from the stick unit 10, the center unit 30 responds to the detection results and the position coordinate data of the marker unit 15 when the detection results are received. Then, a predetermined musical sound is generated from a sound output unit 362 such as a speaker of the sound generation unit 36 which is a sound generation means.
Specifically, the center unit unit 30 associates with the imaging space of the camera unit unit 20 in a ROM (to be described later) or the like as its storage means, and a virtual drum set D as shown in FIG. Is stored. Based on the position coordinate data of the virtual drum set D and the position coordinate data of the marker unit 15 when the detection result is received by the motion sensor unit 14, the center unit unit 30 specifies the instrument hit by the stick unit 10, A predetermined musical sound corresponding to the musical instrument is generated.
Hereinafter, the tone generator 1 according to the present embodiment will be specifically described.

[Configuration of Musical Sound Generation Device 1]
First, with reference to FIGS. 2 to 5, the configuration of the stick unit 10, the camera unit unit 20, and the center unit unit 30 constituting the musical sound generating device 1 in the present embodiment will be described in detail.
2 is a block diagram illustrating a hardware configuration of the stick unit 10, FIG. 3 is a perspective view of the stick unit 10, and FIG. 4 is a block diagram illustrating a hardware configuration of the camera unit unit 20. FIG. 5 is a block diagram illustrating a hardware configuration of the center unit 30.

[Configuration of Stick Unit 10]
As shown in FIG. 2, the stick unit 10 includes a stick control unit 11 including a CPU (Central Processing Unit), a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, and a motion sensor unit. 14, a marker unit 15, and a data communication unit 16. In addition, the stick unit 10 includes a power supply unit (not shown) that supplies power to each unit.

The ROM 12 stores processing programs for various processes executed by the stick control unit 11.
The ROM 12 stores marker feature information used for light emission control of the marker unit 15.
Here, the marker feature information refers to each marker unit 15 (in this embodiment, the first marker of the stick unit 10A and the second of the stick unit 10B when there are a plurality of marker units 15 in the same imaging space. For example, in addition to the shape, size, hue, saturation, or luminance during light emission, the blinking speed during light emission can be used. When there are a plurality of stick portions 10, each stick portion 10 has different marker feature information, and light emission control is performed according to each marker feature information.

  The RAM 13 stores values acquired or generated in the process, such as various sensor values output by the motion sensor unit 14.

The stick control unit 11 executes control of the entire stick unit 10.
For example, when a sensor value (hereinafter referred to as “motion sensor data”) as a detection result is output from the motion sensor unit 14, the motion sensor data is transmitted to the center unit unit 30 via the data communication unit 16. Communication control with the center unit 30 is performed.
The stick control unit 11 reads the marker feature information from the ROM 12 and performs light emission control of the marker unit 15 according to the marker feature information. As described above, when there are a plurality of stick portions 10A and 10B, each stick portion 10A and 10B performs light emission control according to different marker characteristic information. This can be detected by distinguishing the marker unit 15 (first marker) of the stick unit 10A and the marker unit 15 (second marker) of the stick unit 10B from the image captured by the imaging unit 24 of the camera unit unit 20. It is for doing so. That is, when there are a plurality of stick parts 10 and the marker part 15 emits light in the same manner, there is a possibility that it cannot be determined which stick part 10 each marker part 15 belongs to when imaged by the camera unit part 20. is there. For this reason, the stick control unit 11 of the stick unit 10A and the stick control unit 11 of the stick unit 10B read different marker feature information, and perform light emission control of the marker unit 15 according to the read marker feature information, thereby causing each marker unit 15 to emit light. It is easy to distinguish between the two.

Further, in the present embodiment, the stick control unit 11 is inferior in detecting the position of the marker unit 15 by the imaging unit 24 serving as the position detection unit by the marker detection availability determination unit 312 (see FIG. 5) serving as the detection unit. When it is determined that there is, it functions as a light emission state changing means for changing the light emission state of the marker unit 15.
In the present embodiment, the stick control unit 11 receives an instruction signal for requesting a change in the light emission state of the marker unit 15 from the marker detection determination unit 312 of the center unit unit 30 that is a detection determination unit. When receiving this signal, the stick controller 11 changes the light emission state of the marker unit 15 in accordance with the instruction.

Here, the light emission state of the marker unit 15 is the light emission intensity, the light emission color, or a combination thereof of the marker unit 15. The light emission intensity is changed by adjusting the luminance of light emitted from the marker unit 15, for example. The emission color is changed by changing, for example, an LED to be lit to an LED that develops another color.
Specifically, the stick control unit 11 can adjust the light emission luminance of the marker unit 15 to a plurality of stages such as “weak”, “medium”, and “strong”, for example, to detect the marker. When the possibility determining unit 312 instructs to increase the light emission intensity of the marker unit 15, the stick control unit 11 changes the luminance of the marker unit 15 from “low” to “medium”. We cope by raising.
Further, in the present embodiment, the marker unit 15 is configured by, for example, LEDs of a plurality of colors such as three colors of “red”, “blue”, and “yellow”, and the emission color of the marker unit 15 from the marker detection availability determination unit 312. Is instructed to change the color, for example, the stick control unit 11 has a color (for example, “blue”) different from the currently lit LED (for example, “red”) among the LEDs constituting the marker unit 15, for example. ) By turning on the LED.

  Note that the order in which the stick control unit 11 increases the light emission intensity of the marker unit 15 and the color LED to be lit is determined in advance. The instruction signal from the marker detection availability determination unit 312 may include a change content instruction (for example, change the emission color from “red” to “blue”), and the stick control unit 11 However, the light emission state of the marker unit 15 may be changed accordingly. The stick unit 10 is provided with means for detecting the hue of the background BG (see FIG. 7 and the like) such as the wall of the performance place where the musical sound generator 1 is played, the illuminance of the performance place, etc. It is also possible to determine the optimum light emission intensity and light emission color based on the detection result of and to change the light emission state of the marker unit 15.

Note that when there are a plurality of stick units 10 as in the present embodiment, the marker detection availability determination unit 312 that is a detection availability determination unit determines whether or not the position of the marker unit 15 has been detected. This is performed for each of the ten marker sections 15, and an instruction signal is output for each of them as necessary.
When there are a plurality of stick portions 10, it is necessary to distinguish between the marker portion 15 (first marker) of each stick portion 10A and the marker portion 15 (second marker) of the stick portion 10B as described above. The stick control unit 11 of the stick unit 10A and the stick control unit 11 of the stick unit 10B change the light emission state, so that both the marker unit 15 (first marker) and the marker unit 15 (second marker) have the same light emission state. The light emission state is changed based on different marker feature information so that the same color, the same luminance, and the like are not obtained.

  The control performed by the stick control unit 11 is not limited to this. For example, the marker unit is based on an instruction from the center unit unit 30 or a sensor value (motion sensor data) that is a detection result by the motion sensor unit 14. You may make it perform light emission control, such as 15 lighting / extinguishing.

The motion sensor unit 14 is provided in the stick unit 10, for example, and is a variety of sensors for detecting the state of the stick unit 10 (for example, the swinging position, swinging speed, swinging angle, etc.). 14 outputs a predetermined sensor value (motion sensor data) as a detection result. A detection result (motion sensor data) detected by the motion sensor unit 14 is transmitted to the center unit unit 30 via the data communication unit 16.
Here, as a sensor which comprises the motion sensor part 14, an acceleration sensor, an angular velocity sensor, a magnetic sensor, etc. can be used, for example.

As the acceleration sensor, a three-axis sensor that outputs acceleration generated in each of the three axial directions of the X axis, the Y axis, and the Z axis can be used. For the X axis, Y axis, and Z axis, as shown in FIG. 3, the axis that coincides with the longitudinal axis of the stick 10 is defined as the Y axis, and a substrate (not shown) on which an acceleration sensor is arranged. An axis that is parallel and orthogonal to the Y-axis can be an X-axis, and an axis that is orthogonal to the X-axis and the Y-axis can be a Z-axis. The acceleration sensor may acquire the acceleration of each component of the X axis, the Y axis, and the Z axis, and may calculate a sensor composite value obtained by combining the respective accelerations.
In the present embodiment, when performing using the musical tone generator 1, the performer holds one end (the base side) of the stick unit 10 and the other end (the tip side) of the stick unit 10 around the wrist or the like. By performing a swing-up and swing-down operation that moves up and down, the stick unit 10 is caused to rotate. Here, when the stick unit 10 is stationary, the acceleration sensor calculates a value corresponding to the gravitational acceleration 1G as a sensor composite value, and when the stick unit 10 is rotating, the acceleration sensor. Calculates a value larger than the gravitational acceleration 1G as a sensor composite value. The sensor composite value is obtained, for example, by calculating the square root of the sum of the squares of the accelerations of the X-axis, Y-axis, and Z-axis components.

Moreover, as an angular velocity sensor, the sensor provided with the gyroscope, for example can be used. In the present embodiment, as shown in FIG. 3, the angular velocity sensor outputs a rotation angle 501 of the stick unit 10 in the Y-axis direction and a rotation angle 511 of the stick unit 10 in the X-axis direction.
Here, the rotation angle 501 in the Y-axis direction can be referred to as a roll angle because it is the rotation angle of the front-rear axis viewed from the player when the player holds the stick unit 10. The roll angle corresponds to an angle 502 indicating how much the XY plane is tilted with respect to the X axis, and the player holds the stick unit 10 in his hand and rotates it left and right around the wrist. Caused by.
Further, the rotation angle 511 in the X-axis direction can be referred to as a pitch angle since it is the left-right rotation angle viewed from the player when the player holds the stick unit 10. The pitch angle corresponds to an angle 512 indicating how much the XY plane is tilted with respect to the Y axis, and is generated when the player holds the stick unit 10 in his / her hand and swings his / her wrist up and down.
Although not shown, the angular velocity sensor may also output the rotation angle in the Z-axis direction. At this time, the rotation angle in the Z-axis direction has basically the same property as the rotation angle 511 in the X-axis direction, and is generated when the player holds the stick unit 10 in his hand and swings his wrist in the left-right direction. The pitch angle.

  As the magnetic sensor, a sensor capable of outputting magnetic sensor values in the three-axis directions of the X axis, the Y axis, and the Z axis shown in FIG. 3 can be used. From such a magnetic sensor, a vector indicating north (magnetic north) by a magnet is output in each of the X-axis direction, the Y-axis direction, and the Z-axis direction. Since the components in each axial direction that are output differ depending on the posture (orientation) of the stick unit 10, the stick control unit 11 determines the roll angle of the stick unit 10 and the rotation angles in the X-axis direction and the Z-axis direction from these components. Can be calculated.

Here, the detection result (motion sensor data) detected by the motion sensor unit 14 will be described with reference to FIG. Here, a detection result by an acceleration sensor among the various sensors is shown.
When the performer performs using the stick unit 10, generally, the same operation as the operation of hitting an actual musical instrument (for example, drum) is performed. In such a (performance) operation, the performer first raises the stick unit 10 and then swings it down toward the virtual musical instrument's strike surface (performance surface). Since the hitting surface does not actually exist, the performer exerts a force to stop the operation of the stick unit 10 immediately before hitting the stick unit 10 against a virtual musical instrument.
FIG. 6 is a diagram showing a change in acceleration in the vertical direction of the motion sensor unit 14 when a performance operation is performed using such a stick unit 10, and is transmitted from the stick unit 10 to the center unit unit 30. It shows an example of motion sensor data.
The vertical acceleration means the acceleration in the vertical direction with respect to the horizontal plane, and may be calculated by decomposing from the acceleration of the Y-axis component. The acceleration in the Z-axis direction (acceleration in the X-axis direction depending on the roll angle) It is good also as decomposing and calculating from. In FIG. 6, a negative acceleration indicates a downward acceleration applied to the stick unit 10, and a positive acceleration indicates an upward acceleration applied to the stick unit 10.

Even when the stick unit 10 is stationary (the portion indicated by a in FIG. 6), since the gravitational acceleration is applied to the stick unit 10, the motion of the stick unit 10 that is stationary against the gravitational acceleration. The sensor unit 14 detects a constant acceleration in the vertically downward direction, that is, in the minus direction. It should be noted that the acceleration applied to the stick unit 10 is zero when the stick unit 10 is in a free fall state.
Next, when the player lifts the stick unit 10 with the swing-up operation in a state where the stick unit 10 is stationary as in the section indicated by b in FIG. 6, the player moves in a direction more counter to gravitational acceleration. As a result, the acceleration applied to the stick portion 10 increases in the negative direction. Then, when the lifting speed is decreased to make it stand still, the acceleration changes from the minus direction to the plus direction, and the acceleration at the time when the swing-up operation reaches the highest speed (see p1 in FIG. 6) is only the gravitational acceleration ( That is, the gravitational acceleration is 1G).

Next, as in the section indicated by c in FIG. 6, when the stick unit 10 reaches the top by the swing-up operation, the performer performs the swing-down operation of the stick unit 10. In the swing-down operation, the stick unit 10 moves in a direction according to the gravitational acceleration, so that the acceleration applied to the stick unit 10 increases in a plus direction with respect to the gravitational acceleration. Thereafter, when the swing-down operation reaches the highest speed, only the gravitational acceleration (that is, the gravitational acceleration 1G) is again applied to the stick portion 10 (see p2 in FIG. 6).
Thereafter, when the swing-up operation is performed again on the stick unit 10 as in the section indicated by d in FIG. 6, the applied acceleration increases in the negative direction, and when the swing-up operation is to be stopped, the applied acceleration is in the negative direction. Turn to the plus direction.
While the performance operation continues, the change in acceleration as shown in FIG. 6 is repeated according to the player's swinging up and down operation of the stick unit 10, and this change in acceleration is detected by the motion sensor unit 14.

Returning to FIG. 2, the marker unit 15 is a light emitter such as an LED provided on the tip side of the stick unit 10, and emits light and extinguishes according to control from the stick control unit 11.
Specifically, the marker unit 15 emits light based on the marker feature information read from the ROM 12 by the stick control unit 11. As described above, the marker feature information of the stick portion 10A and the marker feature information of the stick portion 10B are set to be different. For this reason, as will be described later, the camera unit 20 can distinguish between the marker portion (first marker) of the stick portion 10A and the marker portion (second marker) of the stick portion 10B, and the respective position coordinates are determined. It is possible to distinguish and obtain which stick unit 10 is.
Further, as described above, the marker unit 15 can be adjusted in luminance in a plurality of stages and is configured by a plurality of types of LEDs having different colors, and the light emission intensity and light emission are controlled by the control of the stick control unit 11. The color can be switched as appropriate.

  The data communication unit 16 performs predetermined wireless communication with at least the center unit unit 30. The predetermined wireless communication may be performed by any method, and in the present embodiment, wireless communication with the center unit unit 30 is performed by infrared communication. Note that the data communication unit 16 may perform wireless communication with the camera unit unit 20, or may perform wireless communication with the stick unit 10A and the stick unit 10B. Further, the communication method by the data communication unit 16 is not limited to wireless communication. For example, the stick unit 10 and the center unit unit 30 may be connected by a cable, and the data communication unit 16 may perform wired communication.

[Configuration of Camera Unit 20]
Next, the configuration of the camera unit 20 will be described with reference to FIG.
The camera unit unit 20 includes a camera control unit 21 including a CPU, a ROM 22, a RAM 23, an imaging unit 24, and a data communication unit 25. The camera unit unit 20 includes a power supply unit (not shown) that supplies power to each unit.

The ROM 22 stores processing programs for various processes executed by the camera control unit 21.
The RAM 23 is data indicating the position of the marker unit 15 obtained by the imaging unit 24 (such as an image of a performer performing a performance operation), and the marker unit 15 (first number) of the stick units 10A and 10B calculated from the data indicating the position. The values acquired or generated in the processing, such as the data of the position coordinates of each of the first marker and the second marker, are stored.
The RAM 23 stores marker detection condition information transmitted from the center unit unit 30. The marker detection condition information is a condition for distinguishing and detecting each of the marker portions 15 of the stick portions 10A and 10B, and is generated from the marker feature information by the main body control portion 31 of the center unit portion 30.

The camera control unit 21 performs control of the entire camera unit unit 20.
In the present embodiment, the camera control unit 21 includes, for example, at least a marker unit such as data indicating the position of the marker unit 15 obtained by the imaging unit 24 (that is, an image of a player performing a performance operation with the stick unit 10). 15 is a marker that calculates the position coordinates of the marker portions 15 (first marker and second marker) of the stick portions 10A and 10B based on the marker detection condition information). Perform position detection processing.
That is, the camera control unit 21 detects the marker unit 15 of the stick unit 10 from the image obtained by the imaging unit 24 that is a state detection unit, and the position coordinates on the space (the marker of the stick unit 10 in the imaging space). Position coordinates) of the unit 15 is specified. When the marker portions 15 of the plurality of stick portions 10A and 10B are detected, the camera control portion 21 reads the marker detection condition information from the RAM 23, and each marker portion 15 reads the marker portion 15 ( It is determined whether it is the first marker) or the marker portion 15 (first marker and second marker) of the stick portion 10B.

  In addition, the camera control unit 21 calculates the position coordinate data (that is, the determination result of the first marker and the second marker) of the calculated marker unit 15 (first marker and second marker) of the stick units 10A and 10B. The associated position coordinate data of each marker unit 15 (hereinafter referred to as “marker detection position information”) is transmitted to the center unit unit 30 via the data communication unit 25.

The image pickup unit 24 is, for example, an optical camera, and picks up an image of the periphery of the stick unit 10 including at least the marker unit 15 such as a moving image of a player who performs a performance operation with the stick unit 10 at a predetermined frame rate.
In the present embodiment, the imaging unit 24 functions as a position detection unit that detects the position of the marker unit 15 (light emitting unit) during the performance operation as the operation position of the stick unit 10 (performance operator).
Here, the operation position of the stick unit 10 (performance operator) at the time of the performance operation refers to the performance surface of the virtual musical instrument at the time of the performance operation (for example, a striking surface to be hit by the stick unit 10 in the case of a drum). This is the position where the stick unit 10 is swung down, and is the position of the marker unit 15 at the tip of the stick unit 10 at the shot timing described later.

  Imaging data for each frame captured by the imaging unit 24 is output to the camera control unit 21. The camera unit unit 20 specifies the position coordinates of the marker unit 15 of the stick unit 10 in the imaging space, but specifies the position coordinates of the marker unit 15 (that is, marker position detection processing, see FIG. 15). May be performed by the imaging unit 24, or may be performed by the camera control unit 21. Similarly, the imaging unit 24 may determine whether the marker unit 15 in the captured image is the marker unit 15 (first marker and second marker) of which stick unit 10A or 10B. It may be performed in the camera control unit 21.

The data communication unit 25 performs predetermined wireless communication (for example, infrared communication) with at least the center unit unit 30. In the present embodiment, the data communication unit 25 receives marker detection condition information from the center unit unit 30. Further, the data communication unit 25 transmits marker detection position information to the center unit unit 30.
Note that the data communication unit 16 may perform wireless communication with the stick unit 10. Further, the communication method by the data communication unit 25 is not limited to wireless communication. For example, the camera unit unit 20 and the center unit unit 30 may be connected by a cable and the data communication unit 25 may perform wired communication.

[Configuration of Center Unit 30]
Next, the configuration of the center unit 30 will be described with reference to FIG.
The center unit unit 30 includes a main body control unit 31 including a CPU, a ROM 32, a RAM 33, a switch operation detection circuit 34, a display circuit 35, a sound generation unit 36, and a data communication unit 37. Composed. The center unit 30 includes a power supply unit (not shown) that supplies power to each unit.

The ROM 32 stores processing programs for various processes executed by the main body control unit 31.
Further, the ROM 32 stores waveform data of various timbres (namely, sound source data of each musical instrument), for example, a virtual drum set D (FIG. 1B) assumed in the present embodiment, such as bass drum, hi-hat, snare, cymbal, and tom. )) Is stored in association with the position coordinates of the performance area 5 (see FIG. 2, etc.) in the space of each instrument. The timbre waveform data stored in the ROM 32 is not limited to these percussion instruments. For example, the timbre waveform data of wind instruments such as flutes, saxophones, and trumpets, keyboard instruments such as pianos, and stringed instruments such as guitars are stored in the ROM 32. May be.

The RAM 33 receives information related to the state of the stick unit 10 received from the stick unit 10, that is, a detection result (motion sensor data) related to the movement of the stick unit 10 (performance operator) detected by the motion sensor unit 14 of the stick unit 10. Or, the position coordinate data (first marker detection position information and second marker) of the marker unit 15 (first marker and second marker) of the stick units 10A and 10B received from the camera unit unit 20 as the state detection means. Detection position information) and the like are stored.
Further, the RAM 33 stores the marker feature information of the marker unit 15 (first marker) of the stick unit 10A and the marker unit 15 (first marker and second marker) of the stick unit 10B transmitted from the stick units 10A and 10B. Storing.

The main body control unit 31 performs control of the center unit unit 30 as a whole.
In the present embodiment, the main body control unit 31 includes a sound generation control unit 311, a marker detection availability determination unit 312, and the like when viewed functionally. The functions as the sound generation control unit 311, the marker detection availability determination unit 312 and the like are realized by the cooperation of the CPU and a program stored in the ROM 32 or the like.

The sound generation control unit 311 generates a sound to the sound generation unit 36 that is a sound generation unit that generates a predetermined musical sound when the operation position of the stick unit 10 that is a performance operator is within a predetermined performance area 5. It is a pronunciation control means for giving an instruction.
That is, the sound generation control unit 311 corresponds to the detection result (motion sensor data) related to the movement of the stick unit 10 (performance operator) detected by the motion sensor unit 14 and the marker detection position information sent from the camera unit unit 20. Then, a sound generation instruction signal (note-on event) is given to the sound generation unit 36, which is a sound generation means, so as to generate a predetermined musical sound. Here, the note-on event (sound generation instruction signal) includes information such as shot timing (sound generation timing), volume, tone color (that is, instrument type), and the like.

  Specifically, first, when a detection result (motion sensor data) relating to the state of the stick unit 10 (performance operator) detected by the motion sensor unit 14 is transmitted from each stick unit 10 (performance operator), the sound is generated. Based on this, the control unit 311 detects the state of the stick unit 10 held by the performer, such as the direction and angle (in other words, the performance state of the performer). Further, the sound generation control unit 311 detects the timing of hitting the virtual musical instrument by the stick unit 10 (shot timing) based on the acceleration (or sensor combined value) output from the acceleration sensor.

Here, detection of shot timing by the sound generation control unit 311 will be described with reference to FIG.
As described above, FIG. 6 is a diagram illustrating a change in acceleration in the vertical direction of the motion sensor unit 14 when a performance operation is performed using the stick unit 10.
Since the performer assumes that the musical sound is generated at the moment when the stick unit 10 is struck on the virtual musical instrument, it is desirable that the musical sound generating device 1 can generate the musical sound at the timing assumed by the performer. . Therefore, in the present embodiment, the musical sound is generated at the moment when the player strikes the stick portion on the virtual musical instrument hitting surface or slightly in front of it.

That is, in this embodiment, the sound generation control unit 311 of the center unit unit 30 performs the swing-up operation after the swing-down operation is performed as the moment when the performer strikes the stick unit 10 on the virtual musical instrument strike surface. Detect the moment when it starts. That is, the sound generation control unit 311 further increases by a predetermined value in the minus direction from the stationary state, in other words, from the time when the applied acceleration is only the gravitational acceleration (see p2 in FIG. 6) in the section indicated by d in FIG. The detected point A is detected as shot timing (sound generation timing).
Then, the sound generation control unit 311 generates a note-on event (a sound generation instruction signal) so that sound is generated when this shot timing is detected.

Further, the sound generation control unit 311 determines the speed of the swing-down operation of the stick unit 10 by the player based on the detection result (motion sensor data) regarding the movement of the stick unit 10 (performance operator) detected by the motion sensor unit 14. The sheath strength, the position and angle where the stick unit 10 is swung down are detected. In this embodiment, the sound generation control unit 311 uses only the detection result (sensor value) regarding the movement of the stick unit 10 (performance operator) detected by the motion sensor unit 14 when performing sound generation control on the sound generation unit 36. In addition, the marker detection position information of each of the marker portions 15 (first marker and second marker) of the stick portions 10A and 10B received from the camera unit portion 20 is also taken into consideration.
Then, the sound generation control unit 311 generates a note-on event corresponding to the performance action of the performer. That is, based on the motion sensor data and the marker detection position information, it is determined which instrument is struck at what speed, strength, and timing, and the instrument type (timbre) that matches this determination result. Is read out from the ROM 33, and a sound generation instruction signal (note-on event) is generated as a sound generation unit so that the predetermined musical sound is sounded at a predetermined shot timing with a volume corresponding to the hit speed and strength. 36.

  The marker detection availability determination unit 312 determines whether or not the position of the marker unit 15 has been detected by the imaging unit 24 that is a position detection unit that detects the position of the marker unit 15 during the performance operation as the operation position of the stick unit 10. It is a detection feasibility determination means.

When the performance is performed by the stick unit 10 in a place where strong illumination is applied or in front of a background BG having a hue similar to the emission color of the marker unit 15, the marker unit in the image obtained by the imaging unit 24 is used. 15 and the background BG may be difficult to distinguish. In this case, it is difficult to specify the position of the marker unit 15 from the image, and although the performance operation is actually performed and the shot (the swinging operation of the stick unit 10) is performed, the camera control unit 21 is performed. In this case, there is a detection failure (lost) in which the detection of the marker unit 15 and the specification of the position coordinates cannot be appropriately performed at this shot timing.
The marker detection availability determination unit 312 is a functional unit that determines whether or not a detection failure (lost) of the position detection unit has occurred.
In the present embodiment, for example, before the performance operation is started, as shown in FIG. 7, the player swings the stick unit 10 as the performance operator in the same manner as the performance operation in front of the background BG such as the wall of the performance place. The motion is detected by the motion sensor unit 14 of the stick unit 10 and the imaging unit 24 of the camera unit unit 20. Both of these detection results, that is, the motion sensor data acquired by the motion sensor unit 14 and the marker detection position information detected by the camera unit unit 20 are output to the center unit unit 30.
The marker detection availability determination unit determines whether or not the imaging unit 24 of the camera unit unit 20 can appropriately detect the marker unit 15 of the stick unit 10 based on these data sent to the center unit unit 30. .

Specifically, for example, when it is detected from the motion sensor data acquired by the motion sensor unit 14 that the stick unit 10 has been swung, the marker detection position information sent from the camera unit is included in this. If not, the marker detection availability determination unit 312 determines that the imaging unit 24 has not been able to detect the position of the marker unit 15 (ie, detection failure).
It should be noted that the number of swinging motions of the stick unit 10 from which the marker detection availability determination unit 312 determines the detection failure of the position detection means is not particularly limited. If there is at least one swinging motion, this is a detection failure. The presence or absence can be determined.
However, the discrepancy between the motion sensor data and the marker detection position information indicates that a person has passed some error (for example, between the marker unit 15 and the imaging unit 24) other than the detection failure (lost) of the imaging unit 24 as the position detection means. If the number of times that the marker detection position information corresponding to the shot detection based on the motion sensor data is not obtained is greater than a predetermined value, the imaging unit It is preferable to judge 24 detection failures (lost). The ratio of the number of times that the marker detection position information is not obtained (error rate) is determined to be a detection failure (lost) of the imaging unit 24, and can be set as appropriate.

  When the marker detection possibility determination unit 312 determines that the imaging unit 24 has not been able to detect the position of the marker unit 15 (that is, detection failure (lost) of the imaging unit 24 has occurred), first, the emission intensity of the marker unit 15 is detected. An instruction signal for instructing the stick unit 10 to change is output. Thereby, the light emission intensity of the marker unit 15 is changed by the stick control unit 11 (see, for example, FIG. 8A). Further, when the same detection is performed for all the emission intensities, if it is determined that the marker unit 15 is still in a state where it cannot be detected, the marker detection availability determination unit 312 changes the emission color of the marker unit 15. An instruction signal for instructing the stick unit 10 is output. Thus, the light emission color of the marker unit 15 is changed by the stick control unit 11 (see, for example, FIG. 8B). Further, when the same detection is performed for all the emission colors, if it is determined that the marker unit 15 is still not detected, the marker detection availability determination unit 312 determines both the emission intensity and the emission color of the marker unit 15. An instruction signal for instructing the stick unit 10 to change is output. Thereby, the light intensity and color of the marker unit 15 are changed by the stick control unit 11 (see, for example, FIG. 8C).

In addition, the main body control unit 31 performs communication control between the stick unit 10 and the camera unit unit 20 via the data communication unit 37.
In the present embodiment, the main body control unit 31 receives the motion sensor data from the stick unit 10 via the data communication unit 37 and also receives the first marker and marker detection position information of the first marker from the camera unit unit 20. To do.
Further, the main body control unit 31 generates marker detection condition information based on the marker feature information, and transmits the marker detection condition information to the camera unit unit 20 via the data communication unit 37.
In addition, when an instruction signal for changing the light emission state (light emission intensity, light emission color) of the marker unit 15 is output from the marker detection availability determination unit 312, the main body control unit 31 transmits this via the data communication unit 37. Output to the stick unit 10.

The switch operation detection circuit 34 is connected to the switch 341 and receives input information via the switch 341. The input information includes, for example, a change in the tone volume of the tone to be generated, a change in tone color of the tone to be generated, and switching of the display on the display device 351.
The data communication unit 37 performs predetermined wireless communication (for example, infrared communication) between the stick unit 10 and the camera unit unit 20.

The display circuit 35 is connected to the display device 351 and performs display control of the display device 351.
The display device 351 is a display unit configured with, for example, a liquid crystal display (LCD).
In the present embodiment, the display device 351 is a position detection unit even if the light emission state (light emission intensity / light emission color) of the marker unit 15 (light emission unit) is changed by the stick control unit 11 which is a light emission state change unit. When the imaging unit 24 cannot detect the position of the marker unit 15, a message screen or the like indicating that the position of the marker unit 15 cannot be detected is displayed, and the display device 351 displays the marker unit to the player. It functions as an informing means for informing 15 position detection inability.
Note that the content of the message displayed on the display device 351 is not particularly limited, and for example, a message that prompts the player to change the performance place may be displayed.

The sound generating unit 36 is a sound generating unit that generates a predetermined musical sound, and includes a musical sound data generating unit 361 and an audio output unit 362.
The musical sound data generation unit 361 is a functional unit that reads waveform data from the ROM 32 in accordance with a sound generation instruction from the sound generation control unit 311, generates musical sound data, and converts the generated musical sound data into an analog signal.
The audio output unit 362 is, for example, a speaker, and generates a musical sound based on the musical sound data generated by the musical sound data generating unit 361. Note that the audio output unit 362 is not limited to a speaker, and may be an output terminal that outputs sound to headphones or the like.

[Processing of the musical tone generator 1]
Next, processing of the musical sound generating device 1 will be described with reference to FIGS.
FIG. 9 is a flowchart showing the process of the stick unit 10, FIG. 10 is a flowchart showing the process of the camera unit unit 20, and FIG. 11 is a flowchart showing the process of the center unit unit 30.

[Processing of the stick unit 10]
As shown in FIG. 9, the stick control unit 11 of the stick unit 10 reads the marker feature information stored in the ROM 12 (step S1). In this process, the stick control units 11 of the stick units 10A and 10B read different marker feature information. Different marker characteristic information can be read out by an arbitrary method. For example, the stick units 10A and 10B may be communicated directly or through the center unit unit 30. One marker feature information may be associated in advance for every 10 and the stick control units 11 of the stick units 10A and 10B may read the unique marker feature information associated with each.

When the stick control unit 11 reads the marker feature information, the stick control unit 11 stores the marker feature information in the RAM 13 and transmits it to the center unit unit 30 via the data communication unit 16 (step S2). At this time, the stick control unit 11 transmits the marker feature information to the center unit unit 30 in association with identification information (stick identification information) that can distinguish the stick units 10A and 10B.
Moreover, the stick control part 11 lights the marker part 15 according to marker feature information (step S3).

Before the performance operation is started, the performer swings the stick unit 10 on which the marker unit 15 is lit one or several times in front of the background BG such as the wall surface of the performance place, so that the marker unit 15 is properly detected. Please check the operation. When the stick unit 10 is shaken by the player, the motion sensor unit 14 detects the operation.
The stick control unit 11 reads motion sensor data from the motion sensor unit 14, that is, sensor values (detection results) output from various sensors, and stores them in the RAM 13 (step S4). Thereafter, the stick control unit 11 associates the read motion sensor data with the stick identification information, and transmits it to the center unit unit 30 via the data communication unit 16 (step S5).
The stick controller 11 determines whether or not a marker detection signal indicating that the marker unit 15 has been properly detected from the center unit 30 by the camera unit 20 has been received (step S6). When it is determined that the marker detection signal has not been received (step S6; NO), the stick control unit 11 determines whether a light emission intensity change instruction has been received from the center unit unit 30 (step S7). When it is determined that the instruction to change the light emission intensity has been received, the light intensity is changed to be increased by adjusting the brightness of the marker unit 15 (step S8).
On the other hand, if it is determined that the emission intensity change instruction has not been received (step S7; NO), the stick control unit 11 further determines whether or not the emission color change instruction has been received from the center unit unit 30. (Step S9) When it is determined that the light emission color change instruction has been received (Step S9; YES), the light emission color of the marker unit 15 is changed (Step S10).
If it is determined that the light emission color change instruction has not been received (step S9; NO), the stick control unit 11 further determines whether or not a light emission color / light emission intensity change instruction has been received from the center unit 30. If it is determined that an instruction for changing the emission color / emission intensity is received (step S11; YES), both the emission color and emission intensity of the marker unit 15 are changed (step S12). At this time, the emission color may be a color once attempted to be changed or a color that has not yet been applied.

When the emission intensity of the marker unit 15 is changed (step S8), when the emission color of the marker unit 15 is changed (step S10), when the emission color and emission intensity of the marker unit 15 are changed (step S12), and emission If none of the intensity change instruction, the emission color change instruction, and the emission color / emission intensity change instruction has been received (step S11; NO), the stick control unit 11 returns to step S4 and repeats the process.
On the other hand, if it is determined that a marker detection signal has been received (step S6; YES), the stick control unit 11 receives motion sensor data from the motion sensor unit 14, that is, a sensor value (detection) corresponding to the performance performance of the performer. The result is read and stored in the RAM 13 (step S13). Thereafter, the stick control unit 11 associates the read motion sensor data with the stick identification information and transmits the data to the center unit unit 30 via the data communication unit 16 (step S14).
The stick control unit 11 determines whether or not the performance operation has been completed (step S15). If it is determined that the performance operation has been completed (step S15; YES), the marker unit 15 is turned off (step S16). . Note that the stick control unit 11 determines that the performance operation has ended, for example, whether there is a certain change in the sensor value output by the motion sensor unit 14, or whether the switch (not shown) of the stick unit is OFF. The case where the performance end signal is transmitted from the center unit 30 or the like. On the other hand, if the stick control unit 11 determines that the performance operation has not ended (NO in step S615), the process returns to step S13 and the process is repeated.

[Processing of Camera Unit 20]
As shown in FIG. 10, the camera control unit 21 of the camera unit unit 20 acquires marker detection condition information transmitted from the center unit unit 30 and stores it in the RAM 23. (Step S21). The marker detection condition information is a condition for detecting each of the marker portions 15 of the stick portions 10A and 10B, and is generated from the marker feature information (see step S31 and step S32 in FIG. 11). As the marker feature information, for example, as described above, the shape, size, hue, saturation, or luminance of the marker can be used.
Subsequently, the camera control unit 21 performs various settings of the imaging unit 24 based on the marker detection condition information (step S22).

  Subsequently, the camera control unit 21 causes the camera of the imaging unit 24 to capture an image including the marker unit 15 and the like of the stick unit 10, and performs first marker position detection processing (step S23) and second marker position detection processing (step). S24) is performed. In these processes, the camera control unit 21 determines the position coordinates of the marker unit 15 (first marker) of the stick unit 10A and the marker unit 15 (second marker) of the stick unit 10B from the image captured by the imaging unit 24. Marker detection position information such as size and angle (information on the operation position during the performance operation of the performance operator in the present embodiment) is acquired and stored in the RAM 23.

Subsequently, the camera control unit 21 transmits the marker detection position information acquired in step S23 and step S24 to the center unit unit 30 via the data communication unit 25 (step S25).
The camera control unit 21 determines whether or not the performance operation has been completed (step S26). When determining that the performance operation has been completed (step S26; YES), the camera control unit 21 ends the process and ends the performance operation. If it is determined that it is not present (step S26; NO), the process returns to step S23 and is repeated.

[Processing of Center Unit 30]
As shown in FIG. 11, first, the main body control unit 31 of the center unit unit 30 receives the marker feature information from the stick unit 10 and stores it in the RAM 33 (step S31). Subsequently, the main body control unit 31 generates marker detection condition information from the marker characteristic information and the detection condition set via the switch 341, and transmits the marker detection condition information to the camera unit unit 20 via the data communication unit 37 (step). S32).

  Subsequently, the main body control unit 31 receives the marker detection position information of the first marker and the second marker from the camera unit unit 20 and stores them in the RAM 33 (step S33). The main body control unit 31 receives motion sensor data such as posture information, shot information, and action information associated with the stick identification information from each of the stick units 10A and 10B, and stores the motion sensor data in the RAM 33 (step S34). .

  The marker detection enable / disable determining unit 312 collates the marker detection position information of each of the first marker and the second marker received from the camera unit unit 20 with the motion sensor data received from each of the stick units 10A and 10B. It is determined whether or not the position of the marker unit 15 is properly detected in the unit 20 (step S35). Specifically, if the marker detection position information of the marker unit 15 is detected at the timing when the motion sensor data is generated, the marker detection availability determination unit 312 appropriately recognizes the marker unit 15 in the camera unit unit 20. , It is determined that it has been detected. In addition, when motion sensor data is generated but the marker detection position information of the marker unit 15 is not detected or an error is detected, the marker detection availability determination unit 312 determines the wall surface of the performance place, etc. It is determined that the combination of the background BG and the marker unit 15 is poor and the camera unit 20 cannot recognize the marker unit 15.

  When it is determined that the position of the marker unit 15 is not properly detected in the camera unit 20 (step S35; NO), the marker detection availability determination unit 312 determines the emission intensity of the marker unit 15 in the current emission color. It is determined whether or not all the variations have been inspected (step S36). If it is determined that there is an emission intensity that has not been inspected (step S36; NO), an instruction to change the emission intensity is given to the stick unit 10. Send. For example, when the current emission color of the marker unit 15 is “red” and there are three levels of “weak”, “medium”, and “strong” as variations of emission intensity that can be changed in the stick unit 10, the current emission intensity is If it is “weak”, the marker detectability determination unit 312 transmits a light emission intensity change instruction to the stick unit 10. If the current light emission intensity of the marker unit 15 is already at the highest level (for example, “strong” if there are three levels of “weak”, “medium”, and “strong”), “weak” and “medium” are temporarily assumed. Even when the inspection is not actually performed, it may be determined that all the variations of the emission intensity have been inspected.

  On the other hand, if it is determined that all the emission intensities have been inspected (step S36; YES), the marker detection availability determination unit 312 next determines whether all the emission colors of the marker unit 15 have been inspected. Is determined (step S38), and if it is determined that there is a light emission color that has not been inspected (step S38; NO), a light emission color change instruction is transmitted to the stick unit 10 (step S39). For example, if the current emission color of the marker unit 15 is “red” and there are three variations of emission color that can be changed in the stick unit 10, “red”, “blue”, and “yellow”, “red” and “red” If “yellow” has been inspected but “blue” has not yet been inspected, the marker detectability determination unit 312 transmits an emission color change instruction to the stick unit 10. Note that the marker detectability determination unit 312 may simply send a light emission color change instruction so as to change to a different color that has not been tried yet, or specify a light emission color to change to “blue”. The emission color change instruction may be transmitted.

On the other hand, when it is determined that all the emission colors have been inspected (step S38; YES), the marker detection availability determination unit 312 further inspects all combinations of the emission color and the emission intensity of the marker unit 15. If it is determined whether or not there is a combination that has not yet been inspected (step S40; NO), a light emission color / light emission intensity change instruction is transmitted to the stick unit 10. (Step S41). If the combination of the strongest emission intensities (for example, “strong” in the three stages of “weak”, “medium”, and “strong”) has already been inspected for the same emission color, the emission is weaker than this. Even when the combination with the intensity is not actually inspected, it may be determined that all the combinations of the emission color and the emission intensity have been inspected.
When the marker detection possibility determination unit 312 determines that all the combinations of the emission color and the emission intensity have been inspected (step S40; YES), the marker unit is not changed in the light emission state of the marker unit 15 of the stick unit 10. This is a case where 15 detection failures cannot be dealt with. For this reason, in this case, a notification screen (warning screen) indicating that the marker unit 15 cannot be detected is displayed on the display device 351 as the notification means (step S42), and the process is terminated. At this time, a message for prompting the change of the performance place may be displayed on the display device 351. At this time, instead of the display on the display device 351 or together with the display on the display device 351, an alarm, a message, or the like may be output to the performer as a sound.

When a light emission intensity change instruction is transmitted (step S37), a light emission color change instruction is transmitted (step S39), or a light emission color / light intensity change instruction is transmitted (step S41), the process returns to step S33 and processed. repeat.
On the other hand, when it is determined that the position of the marker unit 15 has been properly detected in the camera unit 20 (step S35; YES), the marker detection availability determination unit 312 has detected that the marker unit 15 has been detected by the stick unit 10. The marker detection signal is transmitted (step S43).
Then, the main body control unit 31 receives the marker detection position information of the first marker and the second marker from the camera unit unit 20 and stores them in the RAM 33 (step S44). Further, the main body control unit 31 receives motion sensor data such as posture information, shot information, and action information associated with the stick identification information from each of the stick units 10A and 10B, and stores it in the RAM 33 (step S45). .

Subsequently, the sound generation control unit 311 determines whether or not there is a shot based on the received motion sensor data (step S46). That is, for example, in the case of motion sensor data as shown in FIG. 6, when there is an operation of swinging down the point A shown in FIG. After being performed, the instant at which the swing-up operation is started is detected as a shot timing (sound generation timing). When it is determined that there is no shot (step S46; NO), the main body control unit 31 returns to step S44 and repeats the process.
On the other hand, if it is determined that there is a shot (step S46; YES), the sound generation control unit 311 determines the sound generation timing, the volume of the musical sound to be generated based on the motion sensor data (for example, the sensor combined value of the acceleration sensor), Generates a note-on event including a tone (musical instrument type) and the like. The volume of the musical sound can be obtained from the maximum value of the sensor composite value, for example. As a result, at the predetermined sound generation timing in the normal performance mode, the musical sound based on the sound source of the predetermined musical instrument is generated from the sound generator 36 at the predetermined volume (step S47).
In this case, the main body control unit 31 always determines whether or not the performance has ended (step S48), and if it determines that the performance has ended (step S48; YES), the musical tone generation process ends. . If it is determined that the performance has not ended (step S48; NO), the process returns to step S44 and the process is repeated.

As described above, according to the musical sound generating device 1 of the present embodiment, when the position of the marker unit 15 during the performance operation cannot be detected, the emission intensity, the emission color, and both of the marker unit 15 are stepped. Thus, the marker unit 15 can be distinguished from the background BG such as the wall surface of the performance place. Thereby, even if it does not change a performance place etc., it can respond to the detection defect of the marker part 15 as much as possible by changing the light emission state of the marker part 15, and can suppress a burden on a player to the minimum. it can.
Moreover, in this embodiment, the stick control part 11 which is a light emission state change means changes the light emission intensity of the marker part 15 which is a light emission part, when the position of the marker part 15 cannot be detected. Thereby, even when the performance place is bright or the light of the illuminating lamp prevents the imaging unit 24 from detecting the position of the marker unit 15, the detection failure of the marker unit 15 is eliminated as much as possible without moving the performance place. can do.
Moreover, in this embodiment, the stick control part 11 changes the luminescent color of the marker part 15 which is a light emission part, when the position of the marker part 15 cannot be detected. Thereby, even when the hue of the background BG such as the wall of the performance place is close to the hue of the emission color of the marker section 15 and prevents the imaging section 24 from detecting the position of the marker section 15, the movement place is moved as much as possible. Therefore, the detection failure of the marker unit 15 can be eliminated.
Moreover, in this embodiment, when the position of the marker unit 15 cannot be detected, the stick control unit 11 can change both the emission color and the emission intensity of the marker unit 15 that is a light emitting unit. Thereby, in many cases, it becomes possible to eliminate the detection failure of the marker unit 15 by changing the light emission state of the marker unit 15.
In addition, even if the stick control unit 11 changes the light emission state of the marker unit 15, if the position of the marker unit 15 cannot be detected by the imaging unit 24, the display device 351 displays that the position of the marker unit 15 cannot be detected. To the person. For this reason, the performer can know that the musical sound generator 1 cannot be performed smoothly under the above conditions and the cause thereof, and the musical sound generator 1 such as changing the performance place or adjusting the lighting lamp. It is possible to take necessary measures to perform the performance.

  Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and it goes without saying that various modifications can be made without departing from the scope of the invention.

For example, arbitrary processing among the processing to be performed by the stick unit 10, the camera unit unit 20, and the center unit unit 30 in the above embodiment is performed by other units (stick unit 10, camera unit unit 20, and center unit unit 30. ).
That is, for example, the stick control unit 11 of the stick unit 10 (performance operator) may function as a sound generation control unit, and the stick control unit 11 may analyze detection results of various sensors. In this case, not the detection results (raw data of various sensors, etc.) of the motion sensor unit 14 as the state detection means, but the note-on event (sound generation instruction signal) based on the detection results of the various sensors is the center unit unit 30. Sent to.
In this embodiment, the marker detection availability determination unit 312 of the center unit 30 functions as a detection availability determination unit, and the stick control unit 11 obtains a change in the light emission state of the marker unit 15 from the marker detection availability determination unit 312. When the instruction signal is received, the light emission state of the marker unit 15 is changed according to the instruction. However, the method of changing the light emission state of the marker unit 15 is not limited to this.
For example, the camera control unit 21 of the camera unit unit 20 functions as a detection possibility determination unit, detects that the imaging unit 24 cannot detect the marker unit 15, and instructs to change the light emission state of the marker unit 15. A signal may be directly output to the stick unit 10.
As described above, when the stick control unit 11 functions as a sound generation control unit and the stick control unit 11 analyzes the detection results of various sensors, the stick control unit 11 further includes an imaging unit. 24 may function as a detection possibility determination unit that determines whether or not the marker unit 15 can be detected. In this case, the marker detection position information is transmitted from the camera unit 20 to the stick unit 10, and the motion sensor data acquired by the motion sensor unit 14 in the stick control unit 11 is collated with the marker detection position information. Thus, it is determined whether or not the marker unit 15 is properly detected.
Further, for example, in the main body control unit 31 of the center unit unit 30, data indicating the position of the marker unit 15 detected by the imaging unit 24 of the camera unit unit 20 (that is, data of an image obtained by imaging a performer or the like) and marker feature information Based on the above, a process of calculating the position coordinates of the marker portions 15 (first marker and second marker) of the stick portions 10A and 10B may be performed. In this case, the camera unit unit 20 only needs to capture an image and transmit it to the center unit unit 30, so that the marker unit 15 of each of the stick units 10 </ b> A and 10 </ b> B is identified in the RAM 23 of the camera unit unit 20. It is not necessary to store the marker detection condition information.
Moreover, the marker 15 part may not emit light itself like an LED as long as the camera unit part 20 can detect the state of the marker part 15. For example, the structure which arrange | positions a reflecting plate in the marker part 15 and utilizes the reflected light of external light is also considered.

  In the present embodiment, the case where the display device 351 functions as a notification unit that notifies the performer that the position of the marker unit 15 cannot be detected has been described as an example. However, the notification unit is not limited to the display device 351. For example, instead of being displayed on the display device 351 or together with the display on the display device 351, an alarm or a message for notifying that the position of the marker unit 15 cannot be detected is output from the audio output unit 362 such as a speaker of the sound generation unit 36. Also good.

In the present embodiment, only the acceleration data measured by the acceleration sensor is exemplified as the motion sensor data acquired by the motion sensor unit 14 as the state detection unit, but the content of the motion sensor data is limited to this. Instead, for example, the angular acceleration may be measured by a gyro and the measured value may be used.
In the present embodiment, the rotation around the axis parallel to the stick portion 10 is not described. However, these rotations may be measured by an angular acceleration sensor or the like.

  In the present embodiment, the musical sound generator 1 is a detecting means for detecting the movement of the stick unit 10 (performance operator) (for example, the swing-down position, swing-down speed, swing-down angle, etc.) based on the performer's performance. The motion sensor unit 14 of the stick unit 10 and the imaging unit 24 of the camera unit unit 20 are provided. However, the detection means is not limited to those exemplified here. For example, the motion sensor unit 14 may be a pressure sensor. Or a detection means using various sensors capable of measuring distances and angles.

  In the present embodiment, the virtual drum set D (see FIG. 1B) is described as an example of the virtual percussion instrument. However, the present invention is not limited to this, and the present invention can swing the stick unit 10 down. It can be applied to other musical instruments such as xylophones that generate musical sounds by movement.

  Further, in the present embodiment, the case where the sound generation unit 36 which is a sound generation unit is provided in the center unit unit 30 has been described as an example, but the sound generation unit may be configured separately from the center unit unit 30. Good. In this case, the sounding means and the center unit 30 are connected by wire or wirelessly, and the sounding means performs a predetermined sound according to an instruction signal from the center unit 30.

In the present embodiment, the performance operator is a stick-shaped stick unit 10 as an example, but the performance operator is not limited to this. For example, a portable terminal device such as a mobile phone may be used as a performance operator.
The musical sound generating device 1 generates a sound of a predetermined instrument by performing a performance operation of hitting a space with a performance operator, and does not actually hit the performance operator on the strike surface of the instrument. Even when a precision electronic device such as a device is used as a performance operator, there is no possibility that the performance operator will be damaged.

  In the present embodiment, the stick unit 10, the camera unit unit 20, and the center unit unit 30 are all provided with a power supply unit. However, the stick unit 10, the camera unit unit 20, and the center unit unit 30 are provided. May be connected to an external power source to supply power from the external power source.

Although several embodiments of the present invention have been described above, the scope of the present invention is not limited to the above-described embodiments, but includes the scope of the invention described in the claims and equivalents thereof. .
The invention described in the scope of claims attached to the application of this application will be added below. The item numbers of the claims described in the appendix are as set forth in the claims attached to the application of this application.
[Appendix]
[Claim 1]
A performance operator equipped with a light emitting section;
Position detecting means for detecting a position of the light emitting unit on a predetermined plane during a performance operation;
Detectability determination means for determining whether or not the position of the light emitting unit on the predetermined plane can be detected by the position detection means;
A light emission state changing unit that changes a light emission state of the light emitting unit when it is determined by the detectability determining unit that a position on the predetermined plane of the light emitting unit cannot be detected;
When a position on the predetermined plane of the light emitting unit is detected by the detection possibility determination unit, and the detected position is included in a performance area provided on the predetermined plane, a musical tone generating unit that generates a musical tone Pronunciation control means for giving pronunciation instructions to,
A musical sound generating device comprising:
[Claim 2]
The musical sound generating device according to claim 1, wherein the light emission state changing unit changes a light emission intensity of the light emitting unit.
[Claim 3]
The musical sound generating device according to claim 1 or 2, wherein the light emission state changing means changes a light emission color of the light emitting unit.
[Claim 4]
In addition, when the light emitting state of the light emitting unit is changed by the light emitting state changing unit, when the position of the light emitting unit cannot be detected by the position detecting unit, the information detecting unit further notifies that the position of the light emitting unit cannot be detected. The musical tone generator according to any one of claims 1 to 3, wherein the musical tone generator is provided.

DESCRIPTION OF SYMBOLS 1 Musical sound generator 10 Stick part 11 Stick control part 14 Motion sensor part 15 Marker part 20 Camera unit part 24 Imaging part 30 Center unit part 31 Main body control part 36 Sound generation part 311 Sound generation control part 312 Marker detection availability determination part 351 Display apparatus D drum set

Claims (4)

A performance operator equipped with a light emitting section;
Position detecting means for detecting a position of the light emitting unit on a predetermined plane during a performance operation;
Detectability determination means for determining whether or not the position of the light emitting unit on the predetermined plane can be detected by the position detection means;
A light emission state changing unit that changes a light emission state of the light emitting unit when it is determined by the detectability determining unit that a position on the predetermined plane of the light emitting unit cannot be detected;
When a position on the predetermined plane of the light emitting unit is detected by the detection possibility determination unit, and the detected position is included in a performance area provided on the predetermined plane, a musical tone generating unit that generates a musical tone Pronunciation control means for giving pronunciation instructions to,
A musical sound generating device comprising:   The musical sound generating device according to claim 1, wherein the light emission state changing unit changes a light emission intensity of the light emitting unit.   The musical sound generating device according to claim 1 or 2, wherein the light emission state changing means changes a light emission color of the light emitting unit. In addition, when the light emitting state of the light emitting unit is changed by the light emitting state changing unit, when the position of the light emitting unit cannot be detected by the position detecting unit, the information detecting unit further notifies that the position of the light emitting unit cannot be detected. The musical tone generator according to any one of claims 1 to 3, wherein the musical tone generator is provided.

Images