JP2000276138A - Music sound controller - Google Patents

Abstract

PROBLEM TO BE SOLVED: To control the issuing of music sound by detecting the position of a mobile operation object. SOLUTION: A position detecting part 21 inputs image data of a subject on an image memory and detects the position of a specified part in the subject on an image display surface obtained by photographing the subject. A sound issuing area setting part 23 sets a plurality of sound issuing areas on the image display surface and also music sound to be issued are assigned to each sound issuing area. A sound issuing indicating part 22 inputs the position detecting output of the part 21 and the setting output of the part 23, and executes judgement concerning the movement condition of the specified part of the object.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a musical tone control device for controlling the generation of musical tones by detecting the position of a moving operation object.

[0002]

2. Description of the Related Art Exercise therapy is used as rehabilitation for persons with dysfunctions such as paralysis of limbs, joint disorders and muscular weakness. However, since it is monotonous, there is a problem that the motivation for rehabilitation is not understood.
Conventionally, in exercise therapy, there has been no known support apparatus for recovering physical ability while having fun by linking music to body movements. On the other hand, as a technique related to operation input of an electronic musical instrument, there is an electronic musical instrument that produces a musical sound in response to a gesture, as is known in Japanese Patent Application Laid-Open No. 9-68974. However, this is the musical instrument used by a healthy person and can be used for restoring physical ability, but it is not made especially with a view to supporting the restoration of physical ability.

[0003] Considering conventional operation input devices for electronic musical instruments, there are many devices such as a keyboard that require considerable skill in operation. On the other hand, an electronic musical instrument that produces musical sounds in response to the above-described gestures cannot be played unless a special input device is attached to the body, and furthermore, inputs only relying on musical sounds that are produced, There has been a problem that it is not possible to input a musical tone while visually confirming it. Therefore, the operator can freely move his or her limbs in the space without being restrained by the operating device,
There has been a demand for a musical instrument capable of generating a musical sound by visually recognizing an entire space as an operation unit such as a keyboard of the musical instrument and capable of visually confirming the operation. With such a musical instrument, not only the improvement as an operation input device of an electronic musical instrument but also the desire to perform well by the operator is born, and from this point, it can be expected as a support device for rehabilitation for moving limbs and the like. .

[0004]

SUMMARY OF THE INVENTION The present invention has been made to solve the above-mentioned problems, and it is intended to detect the position of a moving operation object without attaching a special operation input device to the body. It is an object of the present invention to provide a new tone control device capable of easily controlling the tone generation. As an input device for an electronic musical instrument, it can be easily played, and can also be used as a support device for rehabilitation in recovering physical ability or a support device for improving physical functions. .

[0005]

According to the first aspect of the present invention, there is provided a musical tone control apparatus for detecting a position of a moving operation object and controlling the generation of a musical sound. Position detection means for detecting a position, input of the output of the position detection means, at least the movement time from any reference position of the operation object to the current position of the operation object and the current position of the operation object at least both elements And determining whether or not the musical tone is to be sounded in response thereto, and, when the musical tone is to be sounded, comprising sounding instructing means for instructing the sounding of the musical sound. Therefore, a musical tone is generated when a predetermined moving condition relating to a moving time from an arbitrary reference position of the operation object to the current position and the current position of the operation object is satisfied.

As a result, a new gesture-type musical sound control device that generates a musical sound only when an operating object such as a card, a stick, or the like held by the operator's hand, head, or hand holds a specific moving condition. Can also be used. Also, if the operation object is not operated in consideration of the determined moving conditions,
Since the desired musical sound is not generated, the operator of the operation object is motivated to play well, and the support device leads to rehabilitation. More specifically, for example, the first condition is that the current position of the operation object is in one of one or more specific sounding regions.
For example, the second condition is that the time required to move from the reference position to the current position is equal to or longer than a predetermined time. The predetermined time can be arbitrarily set and changed by considering the moving speed of the operation object and the like. The reference position is
For example, it may be the first or immediately preceding position where the presence of the operation object is detected, or a predetermined registration position of the operation object. The pitches of the plurality of sounding regions are assigned according to the positions of the sounding regions on the absolute coordinates. Alternatively, the reference position is assigned as a predetermined pitch in accordance with a relative position (including a relative distance) with respect to the reference position.

According to a second aspect of the present invention, in the musical tone control device, position detecting means for detecting a position of a specific portion of the subject on an image display surface on which the subject is photographed, and at least one position on the image display surface. A sounding area setting means for assigning musical tones to be sounded to the sounding area, and an output of the position detecting means, wherein the specific portion is on the image display surface.
After a lapse of a predetermined time since it is determined that the one area does not exist, it is determined that the one area is present in another area on the image display surface different from the one area, and the other area is generated by the sound generation. When it is determined that the tone belongs to the area, the apparatus further comprises a sounding instruction unit that instructs the sounding of the musical tone assigned to the sounding area where the specific portion currently exists.

[0008] Therefore, it is possible to know the moving state of a specific part of the subject by the tone generation. Further, the position of the operation object can be easily recognized on an image display surface obtained by photographing the operation object as a specific portion of the subject. Only when the specific part of the subject satisfies the predetermined moving condition,
Since a musical tone can be generated, a desired musical tone is not generated unless the operation object is arbitrarily moved. As a result, the operator of the operation object is motivated to play well, and thus is effective as a rehabilitation support device that moves the body, arms, and legs of the operator. Further, it is possible to make the operator confirm that the specific part of the subject has truly moved to a different sounding area with a purpose.

[0009] One or a plurality of sound emission areas can be assigned on the absolute coordinates of the image display surface, or can be assigned based on a relative position (relative distance) from a reference position. The reference position may be a position where the specific part of the subject is initially stopped, a position where the image is first registered for image recognition, or the like. More specifically, the above-described sounding instruction means starts the determination as the initial state when the specific portion is substantially in the stopped state first, and when the specific portion is substantially again stopped in the initial state. Further, if the determination is started again with the current substantially stopped state as the initial state, the tone control operation can be continuously performed.

According to a third aspect of the present invention, in the musical tone control device according to the second aspect, the sound generation instruction means may be configured such that the position of the specific portion is the same as the certain area after the predetermined time has elapsed. It is also instructed to sound a tone assigned to the sounding area where the specific portion is located, even when it is determined that the sound is in the sounding area and when the certain area is judged to be in the sounding area. is there. Therefore, on the image display surface, the specific portion of the subject is
If the operation is performed in such a state that it cannot be detected that the tone is present in a certain sound emitting area, it is possible to instruct the sound generation assigned to the sound emitting area again. As a more specific operation, the specific portion of the subject may be intentionally moved forward or backward in the depth direction of the screen, or the shape of the specific portion of the subject may be temporarily changed. As a result, the movement of the specific part of the subject in the depth direction and the movement of changing the shape can be confirmed by the musical sound. Also, when viewed as a gesture-type tone control device, the same tone can be repeatedly instructed to sound.

According to a fourth aspect of the present invention, in the musical tone control device according to the second or third aspect, the sounding area setting means sets a plurality of sounding areas in a predetermined direction on the image display surface. And assigning musical tones of pitches according to the order of the sounding regions along the predetermined direction. Therefore, the scale of the scale is constituted by the arrangement of the plurality of sounding regions, and the moving direction of the specific portion of the subject can be known by the musical sound. At this time, in the direction orthogonal to the directions of the plurality of sounding regions, the range of the sounding region is limited (a rectangular sounding region) or is not restricted (a band-like sounding region). )be able to. When the range of the tone generation area is limited, a plurality of rows of tone generation areas of different musical tones such as different pitch octaves can be provided.

According to a fifth aspect of the present invention, in the tone control device of the fourth aspect, the sounding area setting means sets the predetermined direction by an external input signal. Therefore, the arrangement direction of a plurality of sound emission regions can be set in accordance with the movement of the specific portion of the subject. As a result, when used as a support device for rehabilitation, the operator adjusts the direction of the sounding area to the direction in which the body, arms, feet, etc. can be moved, so that the operator can determine how far the sound can be moved. You can check. Even in the case of a gesture-type musical sound control device, the arrangement direction of the sounding regions can be adjusted to a direction in which the body, arms, legs, and the like in the current posture are easy to move.

According to a sixth aspect of the present invention, in the musical tone control device, position detecting means for detecting positions of first and second specific portions of the subject on an image display surface on which the subject is photographed, and At least one on the display surface
Sounding area setting means for setting one sounding area and assigning a musical tone to be sounded to the sounding area, and inputting the output of the position detecting means, wherein the first specific portion is located on the image display surface. After a lapse of a predetermined time since it is determined that the one area does not exist, it is determined that the one area is present in another area on the image display surface different from the one area, and the other area is the sound generation area. When it is determined that the second specific part belongs to the first timbre, it is instructed that the tone assigned to the tone generation area in which the first specific part currently exists is to be produced by the first timbre. After a predetermined period of time has elapsed after it has been determined that there is no longer any one area on the image display surface, it is determined that there is another area on the image display surface different from the one area. And when it is determined that the other area belongs to the sounding area, the tone of the musical tone assigned to the sounding area where the second specific portion is present is different from the first tone color. And sounding instructing means for instructing to perform the second tone. Therefore, it is possible to instruct the tone generation of two different tone colors. As a result, when used as a rehabilitation support device, the operator can simultaneously check related movements of the body, arms, legs, and the like based on differences in timbre.

According to a seventh aspect of the present invention, in the musical tone control device, position detecting means for detecting positions of first and second specific portions of the subject on an image display surface on which the subject is photographed, and At least one on the display surface
A sound generation area setting means for setting one sound generation area and assigning a musical tone to be sounded to the sound generation area; and an output of the position detection means, wherein the first specific portion is located on the image display surface. After a lapse of a predetermined time after it is determined that the region no longer exists in one region, it is determined that the region is present in another region on the image display surface different from the certain region, and the other region is the sound generation region. When it is determined that the second specific part belongs to the first performance part, it is instructed that the tone assigned to the sound generation area where the first specific part exists is to be performed in the first performance part. After a lapse of a predetermined time after it is determined that the area does not exist in one area on the image display surface, it is determined that the area exists in another area on the image display surface different from the one area. In addition, when it is determined that the other area belongs to the sounding area, the sound of the musical tone assigned to the sounding area where the second specific portion is present is changed to a second sound different from the first performance part. Sounding instruction means for instructing the performance part to perform. Therefore, it is possible to instruct the pronunciation of two different performance parts, for example, a melody part and an accompaniment part. As a result, when used as a support device for rehabilitation, the operator can simultaneously check the related movements of the body, arms, legs, etc., depending on the difference in the performance parts.

According to the present invention, in the musical tone control device, position detecting means for detecting positions of first and second specific portions of the subject on an image display surface on which the subject is photographed, and Sounding area setting means for setting a plurality of sounding areas in a predetermined direction on a display surface and allocating musical tones having a pitch according to the order of the sounding areas along the predetermined direction, and an output of the position detecting means To detect the direction of change in the position of the second specific portion, and a predetermined time has elapsed since it was determined that the first specific portion did not exist in a certain area on the image display surface. Later, when it is determined that there is another area on the image display surface different from the certain one area, and it is determined that the other area belongs to the sounding area, the first Identifying the pitch assigned to the sounding area where the specific portion is present, shifting the identified pitch by an octave according to the direction of change in the position of the second specific portion, and Sounding instructing means for instructing sound. Therefore, it is possible to easily raise or lower the octave of the musical sound to be generated in accordance with the moving direction of the second specific portion of the subject, for example, the head (bending of the neck) or the left hand.

According to a ninth aspect of the present invention, in the musical tone control device according to any one of the second to eighth aspects, the musical tone control device further comprises a storage unit, wherein the storage instruction unit is configured to: Each time the pronunciation of the sound is instructed, the time lapse information from the previous time to the current time of the sounding instruction is stored together with the information of the musical tone to be sounded. Therefore, in addition to the information of the musical tone for which pronunciation is instructed,
Time lapse information until the next musical tone is instructed to be sounded can be stored as the note length of the previous musical tone. The stored musical tone information and time lapse information can be used as a stock phrase for composing by appropriately reading out a specified performance period or storing the specified performance period again.

In a tenth aspect of the present invention, in the recording medium storing a tone control program, the tone control program detects a position of a specific portion of the subject on an image display surface on which the subject is photographed. Detecting means, at least one sounding area on the image display surface, sounding area setting means for assigning a musical tone to be sounded to the sounding area, and an output of the position detecting means, Is determined, after a lapse of a predetermined time since it is determined that it does not exist in a certain area on the image display surface, is determined to exist in another area on the image display surface different from the certain one area, And when it is determined that the other area belongs to the sounding area, the tone assigned to the sounding area where the specific portion currently exists. It is for causing a computer to function as a sounding instruction means for instructing the sound. Therefore, it is possible to provide a recording medium that causes a computer to execute the same functions as those of the tone control device according to the second aspect.

[0018]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS FIG. 1 is a first explanatory diagram of an embodiment of a tone control device according to the present invention. FIG. 1A is a use state diagram, and FIG. 1B is an explanatory diagram showing a part of an image display surface at the time of registration. In FIG. 1A, 1 is an operator, 1a is a hand, and 2 is a computer main body. A keyboard 3 and an image display device (C
RT display) 4, a speaker 5, and a video camera 6 are connected. In this embodiment, the general-purpose computer 2 includes a card for processing a video signal from the video camera 6 and a sound card for generating a musical sound signal. The operator 1 moves the hand 1 a toward the image display device 4. The image of the operator 1 is photographed by the video camera 6, image-processed in the computer main body 2, and simultaneously displayed on the image display device 4. Operator 1
Move the hand 1a or take a pose while looking at the image display surface.

On the image display surface at the time of registration shown in FIG. 1B, 11 is a display image of the operator, 11a is a display image of the hand, 12 is a recognition window, and 13 is a sound generation area. A case where the hand 1a is a specific part (part) of the subject will be exemplified. At the time of registration, in a predetermined area of the image display surface 4a,
A recognition window 12 for recognizing the hand display image 11a is defined, and a display image in the recognition window 12 is registered. When the hand 1a is moved during the subsequent performance, the display image 11a of the hand moves on the image display surface 4a, and the recognition window 12 is moved following the movement. At a plurality of positions on the image display surface 4a,
3 is set.

In the example shown in FIG.
2, a sound generation area 13 is provided. During the performance, the hand display image 1
When it is determined that the center of 1a is located, a tone signal having a pitch corresponding to the tone generation area 13 is emitted for a predetermined period. The display of the subject on the image display surface 4a is preferably such that the operator 1 moves the hand 1a while looking at the image display surface 4a by displaying a mirror image of the subject.

FIG. 2 is a second explanatory diagram of one embodiment of the tone control device of the present invention. FIG. 2A is a functional block diagram, and FIG. 2B is an explanatory diagram for explaining a determination made by the sounding instruction unit in a flowchart expression. In FIG. 2A, 21
Is a position detection unit, 22 is a sounding instruction unit, 23 is a sounding region setting unit, 21a is an image storage unit of a subject specific part, and 21b is an image recognition unit. Object image data is written from an image input unit such as the video camera 6 to an image memory or the like. The position detection unit 21 inputs the subject image data in the image memory and detects the position of the specific part of the subject on the image display surface where the subject is photographed.

The position detecting section 21 has, as an example, an image storing section 21b and an image recognizing section 21b for a specific portion of a subject. At the time of the registration mode, an image of a specific portion of the subject, for example, a display image of a part of the subject is obtained from the subject image data written in the image memory or the like.
It is registered in the image storage unit 21b of the specific part of the subject. In addition,
The entire display image of the subject can be registered as a specific portion of the subject. In the performance mode, the position of the specific portion of the subject moves on the image display surface. The image recognizing unit 21b includes an image sequentially input from the image input unit via the image memory and the image storage unit 21 of the subject specifying portion.
The position of the specific portion of the subject is detected by comparing with the image registered in b.

The sounding area setting section 23 sets a plurality of sounding areas on the image display surface and assigns musical tones to be sounded to each sounding area. The sounding instructing unit 22 receives the position detection output of the position detecting unit 21 and the setting output of the sounding region setting unit 23, and determines a moving condition of a specific portion of the subject. FIG. 2B shows the moving conditions.

In S24, the initial position of the specific part of the subject is initialized. The initial setting position may be an arbitrary reference position, but is, for example, a stop position when registering an image of a specific portion of the subject. At the time of the initial setting, a special sounding instruction can be given. In S25, it is determined whether or not the position of the specific portion of the subject output from the position detection unit 21 is at the immediately preceding position. When it does not exist at the immediately preceding position, the process proceeds to S26, and when it exists, the determination at S25 is performed again. Note that the position referred to here is not a pinpoint position, but is assumed to be at the same position when it is within a predetermined range even if it slightly moves from the position at which the presence was detected immediately before. That is, in such a case, it is assumed that the vehicle is substantially stopped. The first case where it is determined that the object does not exist at the immediately preceding position is a case where it is detected that the specific portion of the subject has moved to another position. In some cases, the location of the part cannot be detected.

In the determination of S26, it is determined whether or not a predetermined time has elapsed since it was determined in S25 that the vehicle does not exist at the immediately preceding position.
Returning to the determination at 26, when the predetermined time has elapsed, S
27 is determined. In the determination of S27, it is determined whether or not the presence of the specific portion of the subject is detected by the position detection unit 21. When the presence of the specific portion of the subject is not detected, the process returns to the determination at S27 again, but when it is detected, the determination at S28 is performed. In the determination of S28, it is determined whether or not the position of the specific portion of the subject output from the position detection unit 21 belongs to one or a plurality of sounding regions set by the sounding region setting unit 23. If not, the process returns to the determination of S25, and if it belongs to this sound generation region, the process of S29 is performed. In S29, an instruction is made to sound the musical tone assigned to this sounding area, and the process returns to S25. Alternatively, in any case, the process may be ended without returning to the determination at S25.

Returning to S25, if the determinations in S25 to S28 are to be repeated, the stop position is set to the initial position in S24 on condition that the current position of the specific portion of the subject is stopped. The determination may be started again as the immediately preceding position similar to. Furthermore, every time, S2
In step 9, the time at which the sound is instructed is detected, the time interval from the previous sound instructed time to the current sound instructed time is calculated, and this time can be recorded as note information for the previously sounded tone.

Therefore, even if the user moves to a different area, if the time required for the movement is short, no sound is instructed even if the player moves to the sounding area, and even after a predetermined time has elapsed, the presence of the sound in the different area is detected. The pronunciation is instructed only when Therefore, even if the sound passes through a certain sound generation area until the predetermined time elapses, no sound is generated.
Also, while the position detection detects that the specific part of the subject does not move from the immediately preceding position and is substantially stopped, if this area is a sounding area and sounding is instructed once, However, the pronunciation instruction is not repeated.

In the determination of S27, an additional condition is that an area in which the presence of the specific portion of the subject is detected is different from an area in which the presence of the specific part is detected immediately before.
May be determined. In this case, in S25, the presence of the specific portion of the subject cannot be detected at one time, and if it is determined in S27 that the subject is present at the same position as immediately before, the process proceeds to S26 described above. Even if the determination condition is satisfied, the determination in S28 can be prevented from being performed. Therefore, when the position where the specific portion of the subject exists is the sounding area, and the presence cannot be detected only temporarily, the second sounding may not be instructed. Conversely, when the above-described additional condition is not set, the sound of the same sounding region is repeated twice or more arbitrarily so that the presence of the specific portion of the subject cannot be detected temporarily. You can do so.

In the above description, in the determination in S28, the condition that the region where the specific portion of the subject was detected immediately before was one of one or a plurality of sounding regions was not used. The condition may be an additional condition. In this case, it is possible to set a moving condition excluding a case where a specific portion of the subject is stopped and a position is detected in an area other than one or a plurality of sound emission areas. As the moving condition, an appropriate condition is adopted in consideration of a situation in which a hand or the like is moved in rehabilitation or a method of operating as a gesture-type musical sound control device.

FIG. 3 is a schematic diagram of the hardware configuration of an embodiment of the musical sound control device of the present invention. In the figure, the same parts as those in FIG. A general-purpose computer containing a video signal processing card and a sound card is a CPU (Central Processing Unit).
31, image processing unit 32, external storage device 33, ROM (Re
ad Only Memory) 34, RAM (Random Access Memor)
y) 35, a tone generator 36 and the like. The image of the subject captured by the video camera 6 is stored in an image processing unit 32.
Is input to the CPU 31 after being A / D converted. Instead of the video camera 6, a digital camera capable of continuously photographing a still image of a subject may be used. CPU3
1 is a basic input / output system (B
It operates by executing an IOS program, an operating system (OS) program loaded from an external storage device 33 such as a hard disk into the RAM 35, and an application program.

The CPU 31 receives an image signal, implements the functions shown in FIG. 2, and displays an image on the display unit 4. The CPU 31 instructs the generation of a musical tone and outputs musical tone parameters to the musical tone generating unit 36. The musical tone generating unit 36 generates a musical tone waveform and converts the musical tone waveform data into D / D.
A conversion and amplification are performed, and a tone is output from the speaker 5.
It should be noted that the CPU 31 can be executed up to the generation of the musical tone waveform. The music for automatic performance reproduction is temporarily stored in the RAM 35, but can be finally stored in the external storage device 33 and can be read back to the RAM 35.

The image processing section 32 may have only a function of image frame processing or A / D conversion. Preprocessing may be performed. In the above description, a configuration in which a general-purpose personal computer is used to realize a tone control function by a program has been described, but a dedicated tone control device may be used.

FIG. 4 to FIG. 6 are explanatory diagrams of a tone generation area according to an embodiment of the tone control apparatus of the present invention. The sound generation area can be set by the operator himself so as to match the direction in which the operator moves his hand or the like. FIG. 4 is an image display surface in a mode in which a plurality of sound emission areas are set in correspondence with the absolute positions of the image display surface. A plurality of sounding area rows 41 to 43 are set. At the time of registration, a recognition window 12 having its center at the leftmost sounding area 13 in the sounding area row 41 is displayed. The display image is stored. At the time of registration, sound generation area 13, for example, to correspond to the pitch name C _3, as the pitch in the right direction indicated by an arrow 43 is increased,
Pitch is assigned. In the illustrated example, sound generation area column 41 is assigned a one octave from the pitch name C ₃ to B _4, the sound generation area column 42 located in this upper stage, the pitch name C
₄ to B ₅ are assigned to the sound generation area column 43 located in the lower part of sound generation area column 41, it is allocated from the note name C ₂ to B _3.

In the performance mode, when the center point of the display image 11a of the registered hand is located in one of the plurality of sounding areas 13, the pitch assigned to this sounding area is
Musical sounds such as piano sounds and percussion sounds are produced for a predetermined duration. However, as described later with reference to FIG.
The center point of the hand display image 11a is one of the plurality of sounding regions 13
It does not try to produce musical tones only on the condition that it is located at one position.

[0035] FIG. 5 is a sound generation area 13 corresponding to the pitch C ₃
Is the image display surface in a mode in which the reference position corresponds to the relative position of the image display surface and the rising direction of the pitch is set to the right. During registration, the recognition window 12 around the sound generation area 13 corresponding to the pitch C ₃ is displayed. The position of the sound generation area 13 can be freely determined by a user's input at the time of registration. Pronunciation area columns 41-4
3 is arranged obliquely, and the direction in which the pitch rises from the sound generation area 13 corresponding to the pitch C ₃ is set to rise to the right as shown by an arrow 44.

[0036] FIG. 6 is a sound generation area 13 corresponding to the pitch C ₃
Is the image display surface in a mode in which the reference position corresponds to the relative position of the image display surface and the rising direction of the pitch is set to the lower left. During registration, the recognition window 12 around the sound generation area 13 corresponding to the pitch C ₃ is displayed. The position of the sound generation area 13 can be freely determined by a user's input at the time of registration. The tone generation area rows 41 to 43 are arranged obliquely, and the direction in which the pitch rises is set to be lower left as indicated by an arrow 44.

[0037] Incidentally, the pitch of the assignments shown in FIGS. 4 to 6, is an example, to assign the note name C ₃ to one pronunciation region arrays to C _4, only the tone pitch corresponding to the white keys A sounding area sequence for assigning or assigning a pitch corresponding to a black key may be provided in another stage. In addition, it is possible not to provide a tone generation area row for a pitch one octave higher and a pitch one octave lower. In this case, the shape of each sounding area 13 may be a band in a direction orthogonal to the sounding area row 41 instead of a square as shown in the figure. Also, in order to respond to movements such as bending the neck, do not set the direction to a linear direction,
The direction may be along an arbitrary curve.

The plurality of sounding area rows 41 to 43, the display of the sound names in each sounding area, and the arrow 44 are described for explanation, and need not be displayed on the actual image display surface. . During the performance operation, the recognition window 12 may be displayed. However, since the confirmation mark 51 is displayed as shown in FIG. 7, the recognition window 12 may not be displayed during the performance operation.

FIG. 7 is an explanatory diagram of the tone control function of one embodiment of the tone control device of the present invention. In the figure, 11a-1
Numerals 11a-9 indicate the position of the display image 11a of the hand when the hand 1a is moving. 51 is the hand 1a
This is a confirmation mark when recognizing. There are various patterns of hand movements. Here, four typical movements will be described. (1) When the hand 1a is moved slowly in the pitch arrangement direction. In the figure, it is a period of the display images 11a-1 to 11a-5 of the hand. (2) When the hand 1a is moved quickly in the pitch arrangement direction. In the figure, it is a period of display images 11a-5 to 11a-7 of the hand. (3) A case where the hand 1a is temporarily not image-recognized (presence cannot be detected) without moving the hand 1a in the pitch arrangement direction. The state in which the image cannot be temporarily recognized is, for example, when the hand 1a is moved away from the video camera 6, returned to the original position, or moved closer to the video camera 6, and then returned to the original position. In the figure, a hand display image 11a-7
11a-9. Display image 11a-7 of the hand
11a-9 overlaps on the screen display unit in a typical example, but is shown in a shifted state. (4) Further, the hand 1a is set to a state in which image recognition cannot be performed,
When the hand 1a is moved in the pitch arrangement direction. Illustration in this case is omitted.

(1) When the hand 1a is slowly moved in the pitch arrangement direction. When the center of the hand display image 11a enters the sounding area, the musical sound is sounded only once for a predetermined duration. In the illustrated example, when the display image 11a-1 of the hand, the center enters the sound area, the tone of the pitch name C ₃ is pronounced. next,
In the case of the hand display image 11a-2, the center thereof moves out of the sounding area, and in the case of the hand display image 11a-3, the center thereof enters the sounding area, and the musical tone of the note name C ^# ₃ becomes 1 It is pronounced only for a predetermined duration. Then, when the display image 11a-4 of the hand, move the center to the outside of sound generation area, when the display image 11a-5 of the hand, the center enters the sound area, the tone of the pitch names D ₃ Is sounded only once for a predetermined duration. In this way, it is possible to generate a passing sound during the movement of the hand 1a while moving the hand 1a in the pitch arrangement direction.

(2) When the hand 1a is moved quickly in the pitch arrangement direction. There center of the display image 11a-5 of the hand in the sound region of a tone D _3, tone D ₃ is pronounced by a predetermined duration once. Thereafter, the hand 1a is quickly moved in the pitch arrangement direction. The center of the display image 11a-5 of the hand rests 100msec from no longer exists in the sound region of D _3, the center of the display image 11a-7 of the hand is to come to the sound area of G _3. At this time, the tone of the pitch name G ₃ is pronounced once a predetermined duration. However, not pronounced when the center of the display image 11a-6 of the hand has passed the position of the ^{_{D # 3, E 3, F}} 3, F # 3. That is, it is possible to prevent the elapsing sound from being emitted.

(3) A case in which the hand 1a is temporarily moved to a state where the image cannot be recognized without moving the hand 1a in the pitch arrangement direction. First, once a tone pitch name G ₃ the center of the display image 11a-7 of the hand is in the tone generation region of the pitch name G ₃ to sound a predetermined duration. When the hand 1a was moved away from the video camera, it became out of focus or blurred,
Alternatively, an image 11a-8 of both affected hands is input. Then, as a result of the image recognition, the hand display image 1
1a-8 becomes unrecognizable. After 100 msec,
Again, when the hand 1a is brought closer to the video camera 6 and returned to the original position, the display image 11a-9 of the hand can be recognized. As a result, after 100 msec has elapsed since the display image 11a-7 of the hand cannot be recognized, the display image 11a-7
To the center of a-9 is in the tone generation region of the pitch name G _3,
Tone pitch name G ₃ is pronounced by a predetermined duration once again. In this way, it is possible to cause a musical tone having the same note name to be generated a plurality of times for a predetermined duration.

(4) The case where the hand 1a is moved in the pitch arrangement direction with the hand 1a in a state where the image cannot be recognized. The center of the display image 11a-5 of the hand tone D ₃ In the sound generation area of the tone D ₃ is pronounced by a predetermined duration once. Thereafter, by moving the hand 1a away from the video camera 6, as a result of the image recognition, the display image 11a-5 of the hand cannot be recognized. With the hand 1a kept away, the hand 1a is moved in the pitch arrangement direction. After 100 msec elapses after the hand display image 11a-5 cannot be recognized, the hand display image 11a-7
Center of, come to pronounce region of G _3, again, close the hand 1a to the video camera 6, and back to the position of the original distance, musical pitch name G ₃ is pronounced once a predetermined duration You. In this way, the center of the hand display image 11a-6 is D ^# ₃ , E regardless of the speed at which the hand 1a is moved in the pitch arrangement direction.
_3, when the F _3, passes through the position of the F ^# ₃ can be prevented to pronounce passing notes.

The musical tone control device of the present invention is used together with a musical tone generating unit or is used as a musical tone generating device integrated with a musical tone generating unit. In other words, unless the user moves his / her hand while paying attention to the determined moving conditions, the musical instrument does not produce a desired musical tone. If you have no intention to play and move your hand or the like, you will only hear uncomfortable musical sounds. Therefore, the operator is motivated to perform well, and this leads to rehabilitation in which the limbs are moved.

By the above-described musical tone control, the operator can
The position at which a has been moved can be recognized by the pitch of the musical tone to be generated. If a musical tone is generated simply by passing through the tone generation area, a plurality of musical tones will be generated simply. However, move your hand 1a fast,
If the sound is passed through the sounding area, no middle tone (elapsed sound) is sounded, so that the final movement position can be clearly confirmed. In this case, needless to say, the elapsed sound may be set to be generated, or the rehabilitation may be performed while listening to the elapsed sound. Even if an elapsed sound is produced, a musical tone is generated only for a predetermined duration when the hand display image 11a is present in one of the discretely provided sounding regions. .

As described above, it is possible to arbitrarily arbitrarily move the hand 1a away once, as described above, to continuously generate musical tones having the same note name. On the other hand, by holding the hand 1a in the same sounding region, the sound having the same note name may be set to be continuously sounded. It is relatively preferable to make a tone having a predetermined duration length repeated.

Next, the image recognition described with reference to FIGS. 8 and 9 is an example of the principle of image recognition used in an embodiment of the musical tone control device of the present invention. FIG. 8 is a first explanatory diagram of image recognition in one embodiment of the tone control device of the present invention. FIG. 8A is a diagram showing an image in which a display image of a specific portion of a subject in a recognition window is captured and registered. FIGS. 8B to 8D are diagrams illustrating pixels to be compared in the case where the image matching level is changed to perform image identification based on the difference in the number of pixels to be compared.
FIG. 8C shows a case where the matching level of the image is lowered.
FIG. 8D shows a case where the image matching level is set to a medium level.
(E) is a figure which shows the case where the matching level of an image is made high.

In FIG. 8A, image recognition is performed by, for example, extracting features of the hue and brightness (shading) of the subject. In image recognition, usually, preprocessing such as image enhancement processing and image quality improvement processing is performed, and pattern matching is performed after performing feature extraction processing. However, for the sake of simplicity, the description will be made on the assumption that image recognition is performed simply by matching or mismatching image data for each pixel. The pixel here is not necessarily the minimum unit for displaying an image,
It means the minimum unit of image comparison. When registering the hand display image 11a, as shown in FIG.
The background, the color and brightness of the subject are fetched into the memory as a registered image 61, and are stored in pixel units. The portion denoted by reference numeral 62 indicates a pixel region corresponding to the display image of the hand.

The image recognition is performed when the position of the registered image 61 is detected on the image display surface. At this time, the matching level of the image recognition (the threshold level of the matching degree for determining the matching) is lowered or increased according to the situation of the position detection. That is, the video camera 6
Display image 1 of the hand in the image of the subject sequentially input from
During the period in which the position of 1a cannot be detected, starting from a state where the matching level of image recognition is relatively low for a wide area on the image display surface, the image of the registered image 61 is compared, and the hand display image 11a is displayed. Is detected, the image is compared with the image of the registered image 61 in a state where the matching level of the image is relatively high with respect to the region near the display image 11a of the hand on the image display surface. In this manner, in image recognition when searching for the position of the display image 11a of the hand, by setting the matching level of image recognition to be relatively low, the processing time for position detection is reduced, The followability of the recognition window 12 to the display image 11a is increased.

As a specific method of changing the matching level of the image recognition described above, for example, the matching level of the image recognition is lowered by using only the hue as a comparison target.
By using the density change in addition to the hue as a comparison target, the matching level of image recognition can be increased. Alternatively, as shown in FIGS. 8B to 8D, the matching level of image recognition can be changed by changing the number of target pixels 63 to be compared in the recognition window 12. . For example, in FIG. 8C, five pixels including the center point and four diagonal points in the recognition window 12 are sampled as target pixels 63 to be compared, and the corresponding pixels of the registered image 61 are sampled. By looking at the match,
Reduce the matching level of image recognition. Of course, the matching level of the image recognition can be changed depending on the combination of whether or not to perform shading identification as described above.

FIG. 9 is a second explanatory diagram of image recognition in an embodiment of the musical sound control device of the present invention. FIG.
(A) shows a plurality of specific points [X,
Y] and a plurality of position search areas centered on the specific point [X, Y]. FIG. 9B is an explanatory diagram around one specific point [X, Y] when the movement of the specific portion of the subject is small, and FIG. 9C is one diagram when the motion of the specific portion of the subject is large. FIG. 4 is an explanatory diagram around a specific point [X, Y]. In the figure, 71 is a specific point [X, Y], 72 is a position search area, 12-1, 12
-2 is a recognition window before and after movement, 73-1, 7
3-2 is the center point of the recognition window before and after the movement, 7
Reference numeral 4 denotes a moving area on the outer periphery of the recognition window.

As shown in FIG. 9A, the image display surface 4a
A plurality of specific points [X, Y] are arranged on the upper side, and the image display surface 4a is divided by a plurality of position search areas 72 centered on each specific point [X, Y]. The center point 73 of the recognition window 12 is one specific point [X, Y] 7
It is possible to move within the first position search area 72, and if it exceeds this range, a new different specific point [X, Y]
Is designated, and a new specific point [X, Y] 71 can be moved in the position search area 72.

A case where the movement of a specific portion of the subject is small as shown in FIG. 9B will be described. First, at the center point 73-1 of the recognition window, the recognition window 12-
It is assumed that the display image in 1 has feature data of the registered image. Since the center point 73 of the recognition window is scanned in the position search area 72, when the specific part of the subject moves, the recognition window 12-2 is moved to the position 73-2.
It is detected that the display image in there is feature data of the registered image. At this time, the movement vector from the center point 73-1 to 73-2 of the recognition window is small, and the current center point 73
Considering the position of -2, it is assumed that the center of the specific portion of the subject will not be out of the range of the position search area 72 in the next image recognition. If this is the case,
The center point 73 of the recognition window is scanned in the position search area 72 to determine whether or not the display image in the recognition window 12 includes the feature data of the registered image.

A case where the movement of a specific portion of the subject is large as shown in FIG. 9C will be described. First, at the center point 73-1 of the recognition window, the recognition window 12-
It is assumed that the display image in 1 has feature data of the registered image. Since the center point 73 of the recognition window is scanned in the position search area 72, when the specific part of the subject moves, the recognition window 12-2 is moved to the position 73-2.
It is detected that the display image in there is feature data of the registered image. At this time, the movement vector from the center point 73-1 to 73-2 of the recognition window is large, and the current center point 73-3
Considering the position of −2, it is estimated that the specific portion of the subject will be out of the position search area 72 at the next image recognition.

In such a case, image recognition is performed by moving the center 73 of the recognition window to a position search area 72 centered on a different specific point [X, Y] 71. That is, based on the current position of the center 73 of the recognition window and the movement vector (which is also the movement vector of the specific part of the subject), the destination of the specific part of the subject at the next image recognition is estimated, By specifying the specific point [X, Y] 71, the center point 7 of the recognition window
3 is moved to this specific point [X, Y] 71, the center point 73 of the recognition window is scanned in the new position search area 72, and the display image in the recognition window 12 has the registered image feature data. Determine whether

However, when scanning is performed within the new position search area 72 and it is determined whether or not the display image in the recognition window 12 includes the characteristic data of the registered image, if the characteristic data is not detected, FIG. (A),
The center point 73 of the recognition window extends over all the specific points [X, Y] 71 and each specific point [X, Y] 7
1 and sequentially determine whether or not the display image in the recognition window 12 includes the characteristic data of the registered image. At this time, the matching level of image recognition described with reference to FIG. 8 is set low. As described above, while the feature data of the registered image is detected in the display image, the range of scanning the center point 71 of the recognition window, in other words, the range of the display image to be compared is changed to the position search area 72.
By limiting it to a part of the entire screen,
The time required for image recognition is reduced. In the description with reference to FIG. 9, the position of the specific point [X, Y] 71 is set independently of the sound generation area. However, a specific point [X, Y] 71 is provided at each center of the plurality of sound generation areas 13.
May be set.

FIGS. 10 to 17 are flow charts for explaining the operation of the embodiment of the tone control apparatus of the present invention. FIG. 10 is a flowchart of the main routine. FIG. 11 is a flowchart of the parameter and mode setting process. FIG. 12 is a flowchart of the image recognition registration process. FIG. 13 is a first flowchart of the performance process. FIG. 14 is a second flowchart of the performance process. FIG. 15 is a flowchart of the timer process 1. FIG. 16 is a flowchart of the timer process s. FIG. 17 is a flowchart of the automatic performance reproduction process. FIG. 18 is an explanatory diagram showing a time change of the ts parameter, the t1 parameter, and the t2 parameter, which are time parameters.

In the main routine in FIG.
In S81, the values of various parameters are initialized,
The process proceeds to S82. In S82, timbre setting and the like are performed as parameter setting processing, and the process proceeds to S83. In S83, operation modes for registration, performance, and reproduction are set, and the process proceeds to S84. In S84, when in the registration mode, a display image of a specific portion of the subject is fetched and image registration processing is performed.
Processing proceeds to In S85, in the performance mode, a real-time performance process is performed, and the process proceeds to S86. In S86, in the playback mode, the automatic performance playback process is performed, and the process returns to S82.
Separately from the main routine of S81 to S86, the timer 1 process S87 and the timer s that operate by the timer interrupt
There is a process S88.

The parameter setting process and the mode setting process will be described with reference to FIG. In S91, it is determined whether or not any of the tone color setting switches has an ON event. If there is, the process proceeds to S92; otherwise, the process proceeds to S93. In S92,
The timbre parameter corresponding to the timbre setting switch having the ON event is sent to the tone signal generator, and the process proceeds to S93. The timbre is not limited to a musical instrument timbre, but may be a human voice or an animal cry. In S93, it is determined whether or not any other parameter input switch operation has been performed, and if so, the process proceeds to S94, and if none of the other parameter input switches has been operated, S95 is performed. Processing proceeds to Here, the other parameter input switch is a switch for inputting various parameters other than the tone color setting switch determined in S91.

In step S94, various parameters (n, n1, n2, etc.) are changed in response to the switch operation.
The process proceeds to 5. In S95, it is determined whether or not the registration switch has an ON event.
The process proceeds to 96, and if not, the process proceeds to S97. In S96, the W / R (flag) is set to 1 to set the registration mode, and the process proceeds to S97. In S97, it is determined whether or not the reproduction switch has an ON event. If there is, the process proceeds to S98. If not, the process returns to the main routine. S98
, The sequence data Q for automatic performance reproduction
The read pointer of (p) is reset, and in S99, W / R is set to 2 to return to the main routine in the sequence data reproduction mode.

The image registration processing mode will be described with reference to FIG. In S101, it is determined whether or not W / R is 1, and if it is 1, the process proceeds to S102,
If it is not 1, the process returns to the main routine. In S102, the center of the recognition window of the specific part of the subject is
The image is defined in the reference sounding area at the time of registration, a recognition window is displayed on the image display surface as a frame, and the process proceeds to S103. Here, the reference sound generation area is the sound generation area 13 (note name C ₃ ) at the time of registration shown in FIG. Although omitted in this flowchart, it is preferable that the position of the sound generation area 13 at the time of registration can be freely set by an operator using a keyboard or a mouse pointer.

In S103, it is determined whether or not there is an ON event in the RgSW (registration execution switch), and if there is, the process proceeds to S104.
02 is returned. In S104, the image data input to the video camera 6 is registered in a cache memory (image data memory), and the process proceeds to S105.
Although the image data itself may be stored in pixel units,
It may be stored after the feature extraction. In S105, a direction in which the pitch is changed is input together with the movement of the specific part (part) of the subject, and the process proceeds to S106. Here, the direction in which the pitch is changed is a rightward direction indicated by an arrow 43 in FIG. In FIG. 5, it is the upper right direction indicated by the arrow 44. In FIG.
It is the lower left direction shown by the arrow 44. The user usually sets a direction in which the pitch is changed in a direction in which the specific part is to be moved or in a direction opposite to the direction. S
In step 106, setting of a sounding area and determination of a tone name of a musical tone to be assigned to each sounding area are performed in accordance with the direction in which the pitch input in step S105 is changed.
Processing proceeds to However, a mode in which the sound emission area is set in correspondence with the absolute position (absolute coordinates) of the image display surface as shown in FIG. 4 and a sound emission area 13 at the time of image registration as shown in FIGS. There is a mode in which a sound generation area is set at a relative position.

In S107, W / R is set to 0, a value n is set to a ts parameter (time parameter), a value n1 is set to a t1 parameter (time parameter), and F1 related to a sound generation condition described later is set. The flag is set to 0, the pointer for automatic performance (p) is reset, and the process returns to the main routine. W / R = 0 means a performance mode in which the operator moves the hand 1a to generate a musical tone. FIG. 18 (a)
As shown in (1), the ts parameter is a time parameter for the timer function that counts the time of 10 msec and then becomes a constant value. It is set to n in S107, set to 0 in S121 of FIG. 13 described later, and counted up by an interrupt process of FIG. 16 described later, becomes n after 10 msec has elapsed, and does not increase beyond n. As shown in FIG. 18B, the t1 parameter is 1
This is a time parameter for a timer function that counts the time of 00 msec and then becomes a constant value n1, and detects the elapsed time from when the specific part of the subject no longer exists at a position immediately before. 12 is set to n1 in S107 of FIG. 12, set to 0 in S119 of FIG. 13 described later, set to 1 in S116 or S125 of FIG. n1 and does not increase beyond n1.

The performance processing mode will be described with reference to FIGS. In S111, it is determined whether or not W / R is 0. If it is 0, the process proceeds to S112, and if it is not 0, the process returns to the main routine.
In S112, it is determined whether or not the timer time (ts = n) has been reached. When ts = n for the first time,
The process proceeds to S113C, and when ts = n (the value of n is continuously maintained for a predetermined long time), the process proceeds to S113C.
The process proceeds to A. Otherwise, ts is 0 to (n-1)
In this case, the CPU exits the performance processing routine and performs other processing in the main routine. In other words, when the performance process is started immediately after the registration process, since ts = n is still set, S11
The process proceeds to 3C.

The process of extracting whether or not registered image data exists in all image data captured by the video camera 6 is as follows.
This is performed by S113A to S113C. Immediately after registration, the process is started from S113C. However, since the description from S113A is easier to understand, the description will be given here.
The processing of S113A is mainly processing at an arbitrary time after a while after the registration processing. At this time, the characteristic data of the registered image is often in a state in which it is difficult to catch immediately. The process is started by lowering it. Since the time is always ts = n,
The recognition process from S113A starts.

In S113A, the center point 73 of the recognition window 12 is sequentially scanned at a plurality of specific points [X, Y] 71 on the image display surface (see FIG. 9) input by the video camera 6, and the specific points [ X, Y]
It is determined whether or not there is feature data of the registered image in the recognition window 12 having the center at 71. If there is feature data, the process proceeds to S113B.
The process proceeds to 114. In the image recognition in step S113A, since the image recognition is performed at a low matching level, even if the center of the registered image of the specific part of the subject is in the vicinity slightly shifted from a specific point [X, Y], In the recognition window having the center at the specific point [X, Y], the specific part of the subject can be detected.

In S113B, the periphery of the specific point [X, Y] when the feature of the registered image is detected in S113A is sequentially scanned, and the display image in the recognition window centered on this periphery is added to the registered image. It is determined whether or not there is a feature. If there is a feature, the process proceeds to S118; otherwise, the process proceeds to S114. In the image recognition in this step S113B, the matching level is increased and the image recognition is performed relatively strictly. To perform image recognition by sequentially scanning the periphery of the specific point [X, Y] means to sequentially scan the center 73 of the recognition window in the position search area 72 in FIG. When “yes” is obtained in S113B, for the first time, the display image 1 of the hand, which is the specific portion of the subject, is displayed.
It is recognized that the center of 1a is within the position search area 72 centered on the specific point [X, Y].

The process proceeds to S113C from S112 or S123, S128 and S129 of FIG. 14 described later. That is, when the registered image is image-recognized and the position is detected, including immediately after the registration mode, the process proceeds to S113C.
The process proceeds to. Around the current specific point [X, Y],
That is, the position search area 72 in which the center 73 of the recognition window is located is sequentially scanned, and it is determined whether or not the display image in the recognition window 12 centered on the periphery has the characteristics of the registered image. If there is, the process proceeds to S118, and if not, the process proceeds to S114. The image recognition in step S113C is performed by the image recognition of the medium matching level described with reference to FIG.

In S114, flag F1 is set to 0, and the process proceeds to S115. This flag F1 is
In the position detection state, once in S119, t1
= 0. While the position detection state is maintained, it is used so that the second pronunciation instruction is not issued. In S115 to S116, when the specific portion of the subject is detected in the immediately preceding state and the state is substantially stopped (F2 = 1), the current S11
When it is determined that the registered image cannot be recognized by the determination in 3A to S113C, the t1 parameter is set to 1 and a count for elapse of a predetermined time is started. t1
Is not set to 0 because the value of t1 = 0 is used to indicate that 100 msec has elapsed. t1
There is not much difference between counting up from = 0 and counting up from t1 = 1.

In S115, it is determined whether or not the flag F2 is 1, and if it is 1, the process proceeds to S116. If not, the process proceeds to S117. In S116, the t1 parameter is set to 1 and the flag F2 is set to 0. Thereafter, the count value is not reset until a substantial stop state (F2 = 1) is reached in S123. In S117, it is determined whether or not there is an ON event in the tone color setting switch or the registration switch during the performance mode. If there is, the process returns to the main routine. If not, the process returns to S112.

In S118 to S119, when a predetermined time n1 has elapsed from the start of counting of t1, t1 is set to 0, and S141 of interrupt processing to be described later is executed.
Satisfies one condition of the pronunciation instruction in. In S118, it is determined whether or not F1 = 0 and t1 = n1. If yes, the process proceeds to S119, and if no, the process proceeds to S119. In S119, t2
The current value of the parameter (time parameter) is transferred to the t2 ′ parameter, the t2 parameter is set to 0, the t1 parameter is set to 0, and the process proceeds to S120. As shown in FIG. 18C, the t2 parameter is 4 sec.
Is a time parameter for detecting a sound generation interval that maintains a constant value after counting the time of the sound. It is set to 0 in S119 of FIG. 13, and is counted up to n2 in S151 of an interrupt process described later with reference to FIG.

In S120, it is determined whether or not ts has become n. If it is, the process proceeds to S121, and if not, the process returns to S113C to repeat image recognition. That is, after elapse of 10 msec after the image recognition of the specific portion of the subject in the position search area 72 has been continued, the process of determining the moving state of the specific portion of the subject and the next process of recognizing the specific portion of the subject as shown in FIG. I do. In S122 of FIG. 14, it is determined whether or not the difference between the position of the center point 73 of the recognition window and the position of the center point 73 of the previous recognition window (10 msec before) is large. When small, S123
If it is larger, the process proceeds to S124. However, immediately after registration, since there is no previous value, it is determined that the position difference is 0, and the position data of the center point 73 of the previous recognition window is transferred.

The process proceeds to S123 when the specific portion of the subject is substantially stopped. F2 = 1, the position data of the center point 73 of the current recognition window is stored as the position data of the center point 73 of the previous recognition window, the process returns to S113C in FIG. Restart and S113
The image recognition in C is repeated. On the other hand, the process proceeds to S124 when the specific portion of the subject is moving from the position 10 msec earlier. In S124, F
It is determined whether or not 2 = 1 (substantial stop state).
In the case of, the process proceeds to S125.
The process proceeds to 6. In S125, the t1 parameter is set to 1 and the flag F2 is set to 0. Since then
In S123, the count value is not set to 1 until a substantial stop state (F2 = 1) is reached.

In S126, assuming that the specific portion of the subject has moved from the immediately preceding position, the flag F1 is set to 0, and the movement vector (direction and speed) is calculated from the position of the center point 73 of the previous and current recognition windows. Calculate,
The process proceeds to S127. In S127, based on the current position of the center point 73 of the recognition window and the movement vector, whether the position of the center point 73 of the recognition window deviates from the periphery of the current specific point [Y, Y] (the position search area 72). If you guess or not,
The process proceeds to S129, and if it is estimated that it does not deviate, the process proceeds to S128.

At S128, taking into account the value of the movement vector, the scan of the center point 73 of the recognition window next to the specific point [X, Y] (the position search area 72) and next is performed. Determine the starting position.
Then, the position data of the center point 73 of the current recognition window is moved to the position data of the center point 73 of the previous recognition window, the process returns to S113C, and the scanning of the recognition window is restarted from the determined start position to perform image recognition. I do.

On the other hand, in step S129, based on the position of the center point 73 of the current recognition window and the value of the movement vector, it is determined that the next specific point [X, Y], that is, the movement destination of the specific part of the subject. Estimated position search area 7
Determine 2. Then, a specific point [X,
Y] is stored as the previous value of the position data of the center point 73 of the recognition window, the process returns to S113C in FIG. 13, and the image is displayed in the position search area 72 of the newly designated specific point [X, Y]. Perform recognition.

Here, through FIGS. 12 to 14, how the recognition window 12 is moved and the specific part of the object is tracked from the time of registration of the specific part of the object to the time of real-time performance. I will explain again. When the W / R and time parameters are set in S107 of the registration processing in FIG. 12, the performance processing mode shown in FIGS. 13 and 14 is set. Immediately after the registration, the process proceeds to S111, S112, and S113C in FIG. In S113C, the center point 73 of the recognition window is moved in the position search area 72 by the image recognition of the medium matching level, and the display image in the recognition window 12 is compared with the registered image.

As a result of the comparison, if there is feature data of the registered image, the process proceeds to S114. Immediately after registration, S118 to S119, S120, S121, S122, S123,
The process proceeds to S113C. In S113C, the position of the center point 73 of the recognition window when the characteristic data of the registered image is detected is also the center position of the specific portion of the subject. Thereafter, steps S120 to S113C are repeated,
Every time 0 msec elapses, “yes” is determined in S120, and after S121, the subsequent processing shown in FIG. 14 is performed.

While the loop is being repeated, the display image of the specific portion of the subject moves. Therefore, the last time (1
(Before 0 msec) from the center point 73 of the recognition window when the feature data is detected in the process of S113C.
The position difference up to the center point 73 of the recognition window when the feature data is detected in the current process of S113C increases to some extent. Then, S122 to S124, S1
The process advances to 25. As shown in FIG. 9A, if the position difference does not exceed the predetermined range even if the position difference is large to some extent, the process proceeds from S126 to S127 and S128,
Next, the center point 73 of the recognition window to start scanning
Is determined as an appropriate predicted position, the process returns to S113C, and the processing steps from S113C to S128 are repeated several times.

As shown in FIG. 9B, when the above-mentioned position difference further exceeds the predetermined range, the processing proceeds from S127 to S129, and the center point 73 of the recognition window is set to a new specific point [X, Y]. The same operation is repeated with the position search area 72 as the center. However, when the image of the specific part of the subject does not come around the new specific point [X, Y], in S113C, S11 is executed.
The process proceeds to 4-S117. Note that before the center of the recognition window shifts to a new specific point [X, Y], S1
Also in 13A to S113C, if there is no feature data of the registered image as a result of the image recognition, similarly to S114
The process proceeds to S117.

In S112, ts happens to be 1
If it is the moment when 0 msec has elapsed, the process proceeds to S113C. However, normally, the process proceeds to S113A, and the image display surface in the recognition window 72 centered on all the specific points [X, Y] on the image display surface is sequentially compared and the matching level is lowered. The feature data of the registered image is detected by image recognition. When the feature data is detected, in S113B, the center point 73 of the recognition window is sequentially scanned in the position search area 72 of the specific point [X, Y] at this time, and the display image in the recognition window 72 is compared with the target image. The feature data of the registered image is detected by image recognition with a high matching level.

If the display image of the specific portion of the subject does not come in the recognition window, S113A, S113B, S113
In any of the processing steps of 113C, S114 to S1
The processing returns to S112 via S17. In particular, when a display image of a specific portion of the subject is present in the recognition window and the feature data of the registered image is detected with a medium matching level, the specific portion of the subject, for example, the hand 1a is moved to the video camera. When moving closer to or farther from 6, the display image of the specific portion of the subject becomes an out-of-focus display image, or the size of the display image of the specific portion of the subject changes. As a result, even in the image recognition of the medium matching level, the characteristic data of the registered image cannot be detected, and a state where recognition is not possible, that is, a state where the specific portion of the subject is not detected is set. Similarly, when the hand 1a is hidden from the screen,
It becomes unrecognizable. Therefore, in these cases, S
In 113C, the process proceeds from S114 to S117.
After that, if it becomes recognizable, the process proceeds from S113A, S113B to S118.

Referring to FIG. 15, timer processing 1 will be described. This timer process 1 is an interrupt process.
In S141, it is determined whether or not t1 is 0. If t1 is 0, the process proceeds to S144, where t1 is 0
If not, the process proceeds to S150. S shown in FIG.
When the feature data of the registered image is detected in 113C (however, 100 msec has passed since the feature data was not detected last time, and t1 becomes 0 in S119).
S144) is set in S144
The process proceeds to.

In S144, the center point 73 of the recognition window is input in S105 of FIG.
Then, it is calculated which side of the “pitch changing direction” set on the image display surface is set on the image display surface, a ratio to the reference length is calculated, and the process proceeds to S145.
In this flowchart, unlike FIG. 4 to FIG. 6, only one sounding region row 41 is set in the left-right direction of the image display surface, and a band-like shape independent of the vertical direction orthogonal to the sounding region row 41 is set. You have set the pronunciation area. In the mode in which the sound emission area is set in correspondence with the absolute position (absolute coordinates) of the image display surface, the distance in the horizontal direction of the image display surface is set as the reference length, and the center point 73 of the recognition window from the left end of the image display surface is set. The distance to the position (ie, the position of the specific portion) may be divided by the reference length. On the other hand, in the mode in which the sound emission area is set at a relative position with respect to the sound emission area 13 at the time of image registration, a predetermined reference length is set in a direction in which the pitch is changed from the sound emission area 13 at the time of registration as a reference. The distance from this to the position of the center point 73 of the recognition window may be divided by the reference length, with the sounding area 13 at the time of registration serving as the reference as the origin.

In S145, it is determined whether or not this ratio is a predetermined ratio.
The process advances to S46, and if not the predetermined ratio, S150
Processing proceeds to Here, the determination as to whether the ratio is the predetermined ratio will be described with reference to FIGS. 4 to 6. If the center point of the recognition window 12 is present in any of the sounding regions 13 in the sounding region sequence, it is determined. It is to be. In S146, a note name corresponding to the ratio (sound generation area) is set in the parameter K, and first, in S119 of FIG.
The set data obtained by combining with the value of the t2 ′ parameter stored in is taken as the kc parameter, and
This kc parameter is registered in the sequence data Q (p) of the pointer p, and the process proceeds to S147.

Here, the value of the t2 'parameter is the value shown in FIG.
3, the value of t2 before being set to 0 in S119. This value of t2 'indicates the elapsed time from when the specific portion of the subject no longer exists at the previous position until 100 msec has elapsed and when the original position or a new position of the specific portion of the subject is detected again. Value. S
Only when the processing of step S119 is performed, S141 of FIG.
Is determined to be yes, and this position is determined in S144 and S14.
If it is determined in step 5 that the area is a sound generation area, the data t2 'at that time is sequentially registered as automatic performance data in the sequence data Q (p) together with the pitch data, and is also sounded in real time in step S149. Therefore, the center position of the specific portion of the subject has moved, and the user passed through or stayed in another sounding region within 100 msec from when the image of the specific portion of the subject could not be recognized last time. If so,
In this embodiment, no sound is instructed by intentionally ignoring. In this way, an inadvertent elapsing sound when the hand is moved quickly and carelessly is not generated, and is not taken in as automatic performance data.

Under the consideration described above, at S146, t2 'is fetched as the kc parameter together with the current pitch K. Note that t2 'is the elapsed time from when the specific portion of the subject no longer exists at the position where the specific portion of the subject has been detected. If it stops outside the sounding area during the movement, it is the time from when it no longer exists in this area. However, in a normal operation of the user, since a specific portion of the subject is moved along the pitch arrangement direction, the user rarely stops outside the sound generation area. Therefore, t2 'is the time from when the pronunciation of a certain pitch name was previously instructed to when the pronunciation of a new pitch name is instructed this time. Further, as shown in FIG.
Since the parameter is suppressed to a maximum of 4 seconds, t2 ′
Is also reduced to a maximum of 4 seconds. As a result, even if the time between when the pronunciation of the previous note name was previously instructed and when the pronunciation of the new note name is instructed is longer than 4 seconds,
It is forcibly stored as a time interval of 4 seconds. When it is desired to exclude a time when the vehicle stops outside the sounding area as an erroneous operation, the count of t2 'is stored in a register in S146 of FIG. The difference may be calculated and this may be set as t2 '.

In S147, t2 'is reset, and the process proceeds to S148. In S148, a circle of a predetermined radius centered on the center point of the recognition window 12 is displayed on the screen as a confirmation mark (51 in FIG. 7).
The process proceeds to 149. In S149, the musical tone of note name K is generated for a predetermined duration (gate time), and the process proceeds to S150. In S150,
determining whether the value of the t2 parameter is equal to or greater than n2,
If n2 or more, the process proceeds to S152, where n2
If it is less than the predetermined value, the process proceeds to S151. S151
In, the count value is incremented by adding 1 to the value of the t2 parameter, and the process proceeds to S153. In S152, the value of the t2 parameter is set to n2 and S1
The process proceeds to 53.

In S153, it is determined whether or not the value of the t1 parameter is equal to or more than n1, and if it is not less than n1, the process proceeds to S155, and if it is less than n1, the process proceeds to S154. In S154, 1 is added to the value of the t1 parameter to count up, and the interrupt routine ends. In S155, t1
The value of the parameter is set to n1, and the interrupt routine ends. Changes in the values of the t1 parameter and the t2 parameter are as shown in FIGS. 18 (b) and 18 (c).

The timer s process will be described with reference to FIG. This timer s process also starts operating at interrupt timing at predetermined time intervals. In S161, 1 is added to the ts parameter, and the process proceeds to S162. S16
In 2, it is determined whether or not the value of the ts parameter is equal to or greater than n. If the value is equal to or greater than n, the interrupt routine is terminated. Set and end the interrupt routine. The time change of the value of the ts parameter is shown in FIG.
8 (a).

Here, with reference to FIGS. 12 to 14 and FIG. 15, a description will be additionally given of how the recognition window tracks a specific portion of the subject and the tone is generated. The process proceeds from S141 in FIG. 15 to S144 because it is determined in S118 in FIG. 13 that the time corresponding to n1 or more has elapsed since the specific portion of the subject no longer exists at the previous position, and S11 is determined.
Go to No. 9, where t1 = 0 is set. The time determined by the value of n1 is 100 in the above example.
msec, but decided in consideration of the speed of moving the hand, etc.,
50 msec to 100 msec is preferable. Only when the specific part of the subject is currently in the sounding area, S
The determination at 145 is yes, and the process proceeds to S146.
That is, the time data t2 'parameter that matches the conditions of the processing steps in the flowchart is fetched into the sequence data Q (p) in S146. When the mode is shifted from the registration mode to the performance processing mode, FIG.
When the process proceeds from S141 to S149 of S14,
At 9, the tone of the note name assigned to the tone generation area at the time of registration is pronounced. In the example described with reference to FIG. 1B, since the center of the recognition window is always located in the sound generation area at the time of registration, no is set in S145.

Finally, the automatic performance reproduction process will be described with reference to FIG. In S171, W / R is 2
Is determined, and when it is 2, the process proceeds to S172, and when it is not 2, the process returns to the main routine.
In S172, the sequence data Q (p) is read from the first p = 1. At this time, the tone K of a certain note name is read out, and at the next pointer p + 1, the value of the t2 'parameter recorded together with the next note name K is read out, and the process proceeds to S173. However, when there is no next kc data at the position of the next pointer p + 1, t2 ′
= 0. In S173, a musical tone having the pitch name K is generated, and the process proceeds to S174. In S174,
It is determined whether or not the next kc data exists in the sequence data Q (p + 1). If so, the process proceeds to S175; otherwise, the process proceeds to S176.

In S176, the read t2 '
It is determined whether or not the time equal to the value has elapsed. If the time has not elapsed, the process returns to S176. If the time has elapsed, the process returns to S172. In S175, the same processing as in S107 shown in FIG. 12 is performed, that is, W / R is set to 0 and the mode is switched to the performance mode.
The value n is set in the ts parameter, the value n1 is set in the t1 parameter, the flag F1 is set to 0, and the process returns to the main routine.

In the above description, the sequence data Q (p) is read sequentially from the beginning to the end. , And a favorite phrase is designated by numerical input using a keyboard or a mouse controller, and only the sequence data Q (p) for the designated period is read out to produce a sound, or a material for composing is displayed. Alternatively, the sequence data Q (p) only for the period specified in the range may be stored in the memory again.

In the above description, the specific part of the subject is 1
However, two or more specific portions may be provided. In this case, for example, a plurality of flows for detecting the position of each specific portion in the time division process under the same conditions as described above are simultaneously executed. Alternatively, a plurality of general-purpose computers are used to detect the positions of a plurality of specific portions of the subject by individual computers, and one of the computers collects position detection information of the plurality of specific portions of the subject. It is only necessary to give a musical tone pronunciation instruction.

In the case where two specific parts are provided, when the movement of the first specific part satisfies a predetermined movement condition, an instruction is given to produce a tone assigned to the tone generation area in the first tone color. In addition, when the movement of the second specific portion satisfies a predetermined movement condition, the second tone assigned to the tone generation area is sounded with a second tone different from the first tone. Can be instructed. Alternatively, when the movement of the first specific portion satisfies a predetermined movement condition,
Instructing the first performance part to produce the musical tone assigned to the pronunciation area, and when the movement of the second specific portion satisfies a predetermined movement condition, the tone of the musical tone assigned to the pronunciation area is given. It is possible to instruct that the pronunciation is to be performed in a second performance part different from the first performance part.
In addition, while detecting the movement position of the second specific portion,
When the movement of the first specific portion satisfies a predetermined movement condition, the pitch assigned to the sound generation area is identified, and the identified pitch is determined according to the direction of change in the position of the second specific portion. To shift the pitch by one octave.

In the above description, a tone is generated by assigning a tone name (pitch) to a plurality of tone generating regions on the image display surface. Further, different timbres may be assigned to a plurality of tone generation areas on the image display surface. If the operator simply confirms that the specific portion is located in the sounding area, the same tone may be assigned to a plurality of sounding areas.

[0098]

As is clear from the above description, according to the present invention, there is an effect that a moving state of how a specific portion of a subject has moved can be confirmed by a musical tone. The tone control device of the present invention is used together with a tone generator or used as a tone generator integrally with a tone generator. However, if the operator does not intend to play the instrument, he or she can produce a desired tone. It does not become a musical instrument. As a result, the operator is motivated to perform well, which leads to rehabilitation in which the limbs are moved. Therefore, it can be used as a support device for recovering the physical ability of rehabilitation or a support device for improving the physical function. At the same time, since it also has the function of an electronic musical instrument that produces musical sounds in response to gestures, it is possible to enjoy rehabilitation. It can also be expected to enhance musical creativity and dance skills. Even as an input device of an electronic musical instrument, it is possible to easily input a pitch at a hand position or the like, so that even a person who has never touched the musical instrument can play. Not only the hands, but also the movements of the head (neck), elbows, legs, the movement of the cards and sticks held in the hands, etc. can be performed. In addition, since the musical tones to be generated can be stored including the sounding time intervals, the performance can be input and recorded at the same time, and then the phrase being played can be taken out again to easily compose the music.

[Brief description of the drawings]

FIG. 1 shows a first embodiment of a musical tone control device according to the present invention.
FIG.

FIG. 2 shows a second embodiment of the musical sound control device according to the present invention.
FIG.

FIG. 3 is a schematic diagram of a hardware configuration of an embodiment of a musical sound control device according to the present invention.

FIG. 4 is a first explanatory diagram of a sound generation region of the musical tone control device according to the embodiment of the present invention;

FIG. 5 is a second explanatory diagram of a sound generation area according to the embodiment of the tone control device of the present invention.

FIG. 6 is a third explanatory diagram of a sound generation area according to the embodiment of the tone control device of the present invention.

FIG. 7 is an explanatory diagram of a tone control function according to an embodiment of the tone control device of the present invention.

FIG. 8 is a first explanatory diagram of image recognition in one embodiment of the musical sound control device of the present invention.

FIG. 9 is a second explanatory diagram of image recognition in one embodiment of the musical sound control device of the present invention.

FIG. 10 is a flowchart of a main routine according to an embodiment of the tone control device of the present invention.

FIG. 11 is a flowchart of a parameter and mode setting process according to an embodiment of the tone control device of the present invention.

FIG. 12 is a flowchart of an image recognition registration process according to an embodiment of the tone control device of the present invention.

FIG. 13 is a first flowchart of a performance process according to an embodiment of the tone control device of the present invention.

FIG. 14 is a second flowchart of a performance process according to an embodiment of the musical tone control device of the present invention.

FIG. 15 is a flowchart of a timer process 1 according to the embodiment of the musical sound control device of the present invention.

FIG. 16 is a flowchart of a timer process s according to an embodiment of the musical tone control device of the present invention.

FIG. 17 is a flowchart of an automatic performance reproduction process according to an embodiment of the tone control device of the present invention.

FIG. 18 shows a time parameter ts and t
FIG. 4 is an explanatory diagram showing a time change of one parameter and a t2 parameter.

[Explanation of symbols]

1 operator, 1a hand, 2 computer body, 3
Keyboard, 4 image display device, 5 speaker, 6 video camera, 11 display image of operator, 11a is display image of hand, 12 is recognition window, 13 is sound generation area, 21
A position detection unit, 21a an image storage unit for a subject specific portion,
21b Image recognition unit, 22 sounding instruction unit, 23 sounding region setting unit, 51 confirmation mark, 71 specific point, 7
2 position search area, 73 center point of recognition window, 7
4 Moving area around the recognition window

Claims (10)

[Claims] 1. A musical tone control device for detecting the position of a moving operation object and controlling the generation of musical tones, comprising: a position detection means for detecting a position of the operation object; and an output of the position detection means. Determining whether or not to emit the musical tone in accordance with at least both the movement time from an arbitrary reference position of the operating object to the current position of the operating object and the current position of the operating object; And a sound instructing means for instructing sounding of the musical sound when the musical sound is caused to sound. 2. A position detecting means for detecting a position of a specific portion of the subject on an image display surface on which the subject is photographed, and at least one sounding region is set on the image display surface. Sounding area setting means for assigning musical sounds to be sounded, and an output of the position detecting means is inputted, and a predetermined time elapses after it is determined that the specific portion does not exist in a certain area on the image display surface. Later, when it is determined that the specific area exists in another area on the image display surface different from the certain area, and when it is determined that the other area belongs to the sound generation area, the specific part is currently present. And a tone generation instruction means for instructing the tone generation assigned to the existing tone generation area. 3. The sounding instructing means determines that the position of the specific part is in the same area as the certain one area after the predetermined time has elapsed, and the certain area belongs to the sounding area. 3. The tone control device according to claim 2, wherein when it is determined that the tone is assigned to the tone generation area in which the specific portion is present, the tone control is performed. 4. The tone generation area setting means sets a plurality of tone generation areas in a predetermined direction on the image display surface, and generates a tone having a pitch according to an order of the tone generation areas along the predetermined direction. The musical tone control device according to claim 2, wherein: 5. The musical tone control device according to claim 4, wherein the sounding area setting means sets the predetermined direction according to an external input signal. 6. A position detecting means for detecting positions of first and second specific portions of the subject on an image display surface on which the subject is photographed, and at least one sounding area is set on the image display surface. Inputting an output of the position detecting means, wherein the first specific portion is no longer present in a certain area on the image display surface. After a predetermined time has passed since it was determined that the other region is different from the certain region on the image display surface, and when it is determined that the other region belongs to the sound generation region. Instructions that the tone assigned to the tone generation area in which the first specific portion is present is to be produced by the first timbre, and that the second specific portion is in the image table. After a predetermined time elapses after it is determined that the one area does not exist in the area, the area is determined to exist in another area on the image display surface different from the one area, and the other area is determined. Is determined to belong to the sounding region, the tone assigned to the sounding region where the second specific portion is present is sounded with a second tone different from the first tone. And a sound instructing means for instructing the sound control. 7. A position detecting means for detecting positions of first and second specific portions of the subject on an image display surface on which the subject has been photographed, and at least one sounding area is set on the image display surface. Inputting the output of the position detecting means, wherein the first specific portion is no longer present in a certain area on the image display surface. After a predetermined time has passed since it was determined that the other region is different from the certain region on the image display surface, and when it is determined that the other region belongs to the sound generation region. Instructs the first performance part to sound a tone assigned to the sounding area in which the first specific part is present, and the second specific part uses the image table. After a predetermined time elapses after it is determined that the one area does not exist in the area, the area is determined to exist in another area on the image display surface different from the one area, and the other area is determined. Is determined to belong to the sounding area, the tone assigned to the sounding area where the second specific portion is present is sounded in a second performance part different from the first performance part. A tone control device, comprising: a sounding instruction means for instructing. 8. A position detecting means for detecting positions of first and second specific portions of the subject on an image display surface on which the subject has been photographed, and a plurality of sounding regions in a predetermined direction on the image display surface. A sounding area setting means for setting and assigning a musical tone having a pitch according to an order of the sounding areas along the predetermined direction; and an output of the position detecting means, and inputting a position of the second specific portion. A change direction is detected, and after a predetermined time elapses after it is determined that the first specific portion does not exist in a certain area on the image display surface, the image display different from the certain one area is performed. When it is determined that the first specific portion exists in another sound-generating area, and the other area is determined to belong to the sound-generating area, And a sound instructing means for instructing sounding of the musical tone by octave-shifting the identified pitch and shifting the identified pitch in accordance with the direction of change in the position of the second specific portion. Music control device. 9. A storage unit, wherein the storage unit stores time lapse information from the previous time of the sounding instruction to the present time each time the sounding instruction unit instructs sounding of the musical tone. The musical tone control device according to any one of claims 2 to 8, wherein the musical tone control device is stored together with information on the musical tone to be instructed. 10. A position detecting means for detecting a position of a specific portion of the subject on an image display surface on which the subject has been photographed, and at least one sounding region is set on the image display surface. Sounding area setting means for assigning musical sounds to be sounded, and an output of the position detecting means is inputted, and a predetermined time elapses after it is determined that the specific portion does not exist in a certain area on the image display surface. Later, when it is determined that the specific area exists in another area on the image display surface different from the certain area, and when it is determined that the other area belongs to the sound generation area, the specific part is currently present. Sounding instructing means for instructing sounding of a musical sound assigned to the existing sounding area, and a recording medium storing a musical sound control program for causing a computer to function .