JP2005099068A - Musical instrument and musical sound control method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a musical instrument not requiring a person to move hands and feet for playing like existing musical instruments and welfare musical instruments in order for the person with four paralyzed limbs and a person with four disabled limbs to enjoy playing of the musical instrument. <P>SOLUTION: In the musical instrument, the images of the face and expressions of the mouth part, tongue, eyes and nose, etc. of a player, are analyzed, a feature quantities is extracted and musical sound is controlled by the feature quantities. Also, in the musical instrument, the image of the mouth part, the tongue, the nose or a part of the face extracted from video signals or a graphic indicating the size, shape or position of the image is displayed together with a musical sound template. Further, in the musical instrument, the operation of arranging the prescribed part of the image or the graphic on the musical instrument template is received and the size and shape of the image or the graphic or the position of the graphic are related to the attribute of the musical sound to be generated. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention is a welfare musical instrument that allows a paralyzed person or a physically handicapped person to enjoy playing a musical instrument, and does not require the use of the extremity for performance, and does not require the operator necessary for performance to be worn About.

With existing musical instruments and welfare instruments, it is essential to move the limbs for performance. There are no known limb paralyzed or handicapped persons who can enjoy playing musical instruments. On the other hand, a technique is known in which performance control information is provided to an electronic musical instrument using sensors attached to the wrist, elbow, shoulders, and feet of a human body (see, for example, Patent Document 1).
JP-A-10-097250 (first page, FIG. 1)

  In order for a person with limb paralysis or a person with a limb disability to enjoy playing a musical instrument, an instrument that does not require moving hands and feet is necessary, such as existing musical instruments and welfare instruments. The present invention provides a musical instrument that does not use limbs for performance and does not attach an operator necessary for performance to the body.

In order to solve the above problems, the musical instrument of the present invention employs the following means.
(1) An image capturing unit that captures a face of a performer or a part thereof, an image analysis unit that analyzes a video signal output from the image capturing unit and extracts a feature value, and generates a musical sound signal based on the feature value A musical instrument with a musical sound synthesis unit.
(2) The player's lip is extracted from the video signal, the height information and width information of the opening of the lip is extracted as the feature amount, and the height information and the width information The musical instrument according to (1), which controls generation of a musical tone signal.
(3) The musical instrument according to (2), wherein one of pitch and volume of the musical tone signal is controlled by the height information, and the other is controlled by the width information.
(4) The musical instrument according to (1), wherein a player's nose position information is extracted as the feature value from the video signal, and the generation of the musical tone signal is controlled by the nose position information.
(5) Two-dimensional position information in the vertical and horizontal directions of the performer's nose is extracted as the feature value from the video signal, and the pitch and volume of the musical sound signal are extracted from the vertical position information. (1) The musical instrument according to (4), wherein one is controlled and the other is controlled based on the lateral position information.
(6) The musical instrument according to (1), wherein position information of a player's tongue is extracted as the feature value from the video signal, and generation of the musical sound signal is controlled by the position information of the tongue.

(7) Two-dimensional position information in the vertical and horizontal directions of the performer's tongue is extracted as the feature value from the video signal, and the pitch and volume of the tone signal are extracted from the vertical position information. One side is controlled and the other side is controlled by the lateral position information, or three-dimensional positional information of the player's tongue in the vertical direction, horizontal direction, and depth direction is extracted. The musical instrument according to (6), wherein any one of a pitch, a volume, and a tone color of the musical tone signal is controlled.
(8) Extracting opening / closing pattern information of a player's heel or lip as the feature amount from the video signal, and controlling the attribute of the musical sound signal or controlling the operation of the musical instrument based on the opening / closing pattern information (1) The musical instrument according to (1).
(9) An image of a part of the lip, tongue, nose, or face extracted from the video information, or a graphic indicating the size, shape, or position of the image, together with a musical sound template The musical instrument according to (1), which is displayed on the display unit.
(10) Accepting an operation to place a predetermined portion of the image or the graphic on the musical sound template, and the size or shape of the graphic or the position of the graphic and the attribute of the generated musical sound The musical instrument according to (9), wherein the musical instrument is associated with the musical instrument.

(11) Analyzing a video signal of a face or a part thereof and extracting a feature quantity, and a set of procedures for controlling generation of a musical sound signal based on the feature quantity, or from the video signal, Extracting the lip and extracting the height information and the width information of the opening portion of the lip as a feature amount, and generating a musical sound signal by either the height information or the width information According to a set of procedures to be controlled or a procedure for extracting the lip from the video signal and extracting height information and width information of the opening of the lip as features, and the height information, A set of procedures for controlling one of the pitch and volume of the signal and controlling the other based on the width information, or a procedure for extracting position information of a player's nose as a feature value from the video signal; The position information of the A musical tone is obtained from a set of procedures for controlling the player, or a procedure for extracting two-dimensional position information in the vertical and horizontal directions of the player's nose as a feature value from the video signal, and the vertical position information. A set of procedures for controlling one of the pitch and volume of the signal and controlling the other based on the lateral position information, or a procedure for extracting the position information of the performer's tongue as a feature value from the video signal; and From the set of procedures for controlling the generation of the musical tone signal based on the position information of the tongue, or from the video signal, two-dimensional position information of the player's tongue in the vertical and horizontal directions is extracted as a feature amount. From the procedure and the position information in the vertical direction, one of the pitch and volume of the tone signal is controlled, and the other is controlled by the position information in the horizontal direction, or from the video signal, as a feature amount The vertical direction of the performer's tongue A set of procedures for extracting three-dimensional position information in the horizontal direction and the depth direction, and a procedure for controlling one of the pitch, volume, and tone color of the musical tone signal according to any one of the position information, or the video signal From the procedure for extracting the opening / closing pattern information of the player's kite as the feature quantity, and the set of procedures for controlling the attribute of the musical tone signal or the operation of the musical instrument based on the opening / closing pattern information. A musical sound control method that executes a set of procedures.
(12) A lip portion, tongue, nose, or part of face image extracted from the video information, or a graphic indicating the size, shape, or position of the image is displayed on the display unit together with the musical sound template. The musical sound control method according to (11), further comprising a procedure.
(13) An operation for placing the image or a predetermined part of the graphic on the musical sound template is accepted, and the size or shape of the graphic or the position of the graphic and the attribute of the generated musical sound are received. (10) The musical tone control method according to (12), further comprising:
(14) A program that causes a computer to perform the procedure according to any one of (11), (12), and (13).
(15) A recording medium on which a program for causing a computer to perform the procedure described in any one of (11), (12), and (13) is described.

  According to the musical instrument in the present invention as described above, a paralyzed person or a physically handicapped person can play a musical instrument without moving his limbs. Moreover, it is not necessary to attach an operation element required for performance to the body. A paralyzed person or a handicapped person can play a musical instrument as if singing.

Hereinafter, embodiments of a musical instrument of the present invention will be described with reference to the drawings. In addition, when the component which attached | subjected the same code | symbol performed in embodiment performs the same operation | movement, description may be abbreviate | omitted again.
(Embodiment 1)
FIG. 1 is a diagram showing an embodiment of a musical instrument of the present invention. The present invention includes an imaging unit 2 that captures a face of a performer 1 or a part thereof, an image analysis unit 3 that analyzes a video signal output from the imaging unit and extracts a feature amount from a face or facial expression, and The musical instrument includes a musical tone synthesis unit 4 that generates a musical tone signal based on the feature amount.

In FIG. 1, the imaging unit 2 captures the face of the player 1 and supplies a video signal of the face to the image analysis unit 3. The image analysis unit 3 performs face image analysis as will be described later on the video signal, detects a feature amount, and supplies it to the musical tone synthesis unit 4. The musical tone synthesis unit 4 outputs a musical tone signal according to the feature amount. The musical sound signal is converted into a sound by a headphone speaker or a speaker, and is listened to by the performer 1 and surrounding people. In FIG. 1, the headphone speaker and the speaker are omitted.
First, it is assumed that the display unit 5 and the operation unit 6 in FIG. 1 are not provided.
The image analysis unit 3 recognizes the lip portion of the player 1 by image analysis, measures the position of the lip, each sample point at each position set on the lip, the opening area, etc. in real time, and calculates the displacement as pitch data. , Convert to volume data and play. It should be noted that only the lip position may be used for performance means.

  FIG. 2A is an image example of the lip extracted by the image analysis unit 3. In a state where the mouth is open like the lip 13, W1 is the width of the opening. H1 is the height of the opening. In this example, the mouth is widened. In the lip portion 14 of FIG. 2B, the width and height are W2 and H2, respectively, in the case where the mouth is wide open vertically and is gently swollen sideways. The image analysis unit 3 recognizes and extracts such lips, and extracts feature amounts such as width information W and height information H.

  FIG. 3 is an example of the image analysis unit 3 and the tone synthesis unit 4. The image analysis unit 3 includes an H detection unit 31, a W detection unit 32, a tongue position detection unit 33, a right eye detection unit 34, and a left eye detection unit 35. The H detector 31 detects the height H of the opening in FIGS. 2 (A) and 2 (B). The W detection unit 32 detects the width W of the opening of the images of the lip portions 13 and 14 in FIGS. The height data H of the opening is supplied to the terminal T1 of the musical tone synthesis unit 4 and controls the musical tone scale. The width W data of the opening is supplied to the terminal T2 of the musical tone synthesizing unit 4 and controls the volume of the musical tone, that is, the volume.

The height W of the opening is divided every 5 mm, H = 5 to 10 mm is assigned, H = 10 to 15 mm is assigned, and H = 15 to 20 mm is assigned to Mi. It becomes 1 octave at 5-45 mm. An opening width H of 30 mm or less is assigned to the mezzo piano (mp) of the musical score, 30 to 40 mm is assigned to the mesoforte (mf), and 40 mm or more is assigned to the forte (f). When the H data is applied to the terminal T1 as the scale data, the music synthesizer 4 generates a musical signal of a corresponding scale. When W data is applied to the terminal T2, a corresponding tone signal is generated. When H = 0 to 5 mm, that is, when the mouth is almost closed, the generation of the tone signal is stopped. This is because it seems that it is not an easy operation to reduce the left and right sides of the mouth in order to keep the musical sound without leaving the musical sound and to keep W at a small value.
Note that the pitch may be controlled by the width information W and the volume may be controlled by the height information H. In this case, the volume becomes zero when the mouth is closed.
When the generation of musical sounds is controlled by the shape of the lip as in this embodiment, even if the face is moved a little on the rhythm, the generated musical sounds can be kept unchanged if the lip is kept as it is. The player 1 is less likely to be restrained by his face.

  In FIG. 3, the tongue position detection unit 33 detects the position of the tongue when the mouth is open, and outputs the position information. Based on the center position of the opening, it is detected whether the tongue is in the vertical or horizontal position, and tongue coordinate data, which is two-dimensional tongue position information, is supplied to the terminal T3. The tongue coordinate data can be used to control various characteristics of the tone signal. It can be used to soften or harden the tone of the tone signal. Move your tongue to make it vibrato. The imaging unit 2 may be a stereoscopic camera to detect whether the tongue is in front or in the back, and supply three-dimensional positional coordinate information of the tongue to be used for controlling sound.

In FIG. 3, a right eye detection unit 34 and a left eye detection unit 35 detect conscious blinks of the left and right eyes. When the eyes are blinked a plurality of times or when the eyes are pinched for a predetermined time or more, a detection signal is supplied to the terminals 4 and 5. A code such as a simple Morse code can be determined, and the opening / closing pattern information of the kite can be obtained from the left and right code detection signals to switch the tone color of the musical tone signal and the instrument sound. Ignore momentary short blinks that occur physiologically. When one eye is closed, the sound volume may be set to zero. Note that the opening / closing pattern information of the kite can be used not only for control and selection of musical sound attributes, but also for selection and switching of various functions of musical instruments.
The continuously changing H data output from the image analyzing unit 3 may be converted into scale data by the musical tone synthesizing unit 4, or the continuously changing H data represents the scale name in the image analyzing unit 3. It may be supplied to the musical tone synthesis unit 4 after being converted into musical scale data. The resulting scale is a scale separated by semitones according to the average rate.
The technique for generating and controlling the musical tone in the musical tone synthesizing unit 4 is well known in the field of electronic musical instruments and will not be described in detail.

(Embodiment 2)
Next, an embodiment in which the display unit 5 is provided so that the performer 1 can practice the lips and check during performance will be described with reference to FIG.
In FIG. 1, image data of a part of the face and musical tone template data are supplied to the display unit 5 from the image analysis unit 3, and the display unit 5 displays them. Other parts are the same as those in the first embodiment.

FIG. 4A is an example of an image displayed on the display unit 5. In FIG. 4A, a mesh-like musical sound template is displayed vertically and horizontally. As shown in FIG. 4A, the musical sound template is assigned a musical scale in the vertical direction in the first quadrant, and a dynamic symbol of p, mp, and mf in the horizontal direction, as shown in FIG. 4A. . The second to fourth quadrants are subject to left, right, up and down with respect to the first quadrant. An image of the lip of the performer 1 is displayed on the musical tone template. The image analysis unit 3 detects the center position of the lip part, supplies display data so as to match the center position of the musical tone template, and displays the lip part. In the case of the lip image in FIG. 4A, the scale is “Le” and the strength symbol is “Forte”. The player 1 is watching the image of his / her lip on the display unit 5. You can adjust and adjust the scale and volume. It is also possible to practice the practice of musical instruments on a daily basis while checking how much of the scale sound is generated when the mouth is opened.
The image analysis unit 3 handles the image data of the lip portion and the musical tone template data therein, matches the center point data of the lip portion image with the central point data of the musical tone template, and sets the size of the opening to the musical tone template. Evaluation may be made to calculate W and H, and the scale and volume may be determined.
In the musical tone template of FIG. 4A, for the sake of simplicity, the scale is C major, and the sharp and flat semitones are omitted. All the chromatic scales may be allocated by increasing the number of lines of the musical tone template.
(Embodiment 3)

Next, a function for setting the scale range and the strength range of the generated musical tone according to the player 1 in the scale / strong / weak range setting mode will be described. That is, an operation for placing a predetermined portion of the image of the lip on the musical tone template is accepted, and by accepting the operation, the size, shape, or specific part of the image is associated with the attribute of the musical tone to be generated I do.
In general, there are individual differences in the size of the performer's lip and the range in which the mouth can be opened. The performer 1 can eliminate this individual difference by adjusting the distance to the imaging unit 2. However, individual differences in the size of the lip may be corrected and normalized as follows. A case where the operation unit 6 shown in FIG. 1 is used will be described.

First, the performer 1 presses a scale / dynamic range setting mode button (not shown) on the operation unit 6 to set the image analysis unit 3 to a scale / dynamic range setting mode. The player 1 presses a maximum scale determination button (not shown) of the operation unit 6 in a state where the lip portion is maximally widened in the vertical direction. By this operation, the image analysis unit 3 determines to arrange the upper lip of the lip image at the uppermost position of the musical tone template in FIG. 4A and the lower lip at the lowest position of the template. Next, the player 1 presses a strength symbol determination button (not shown) of the operation unit 6 in a state where the lip portion is widened to the maximum in the lateral direction. By this operation, the image analysis unit 3 determines to arrange the right end of the lip image at the right end position of the musical tone template and the left end of the lip image at the left end position of the musical tone template. The original lip image is processed and displayed with the magnification changed in the vertical direction and the horizontal direction. Thereafter, the lip image is processed according to this magnification, the position is determined on the musical sound template, and the scale and strength are determined. Further, the lip image is displayed according to this magnification. By this operation, the scale and strength can be controlled in accordance with the personality of how to open the lip of the performer 1 and individual differences.
Instead of changing the vertical / horizontal magnification difference of the lip display, the musical sound template may be reduced or expanded in the vertical or horizontal direction to match the size of the lip of the player 1.
When performing a music performance, a scale / strength range setting mode cancel button (not shown) may be pressed to cancel the scale / strength range setting mode.
In the case of a person with a physical disability, a caregiver may operate if the button operation of the operation unit 6 is difficult. If the image analysis unit 3 detects a specific pattern of blinking eyes of the performer 1 and detects the selection of the scale / strength range setting mode and the setting operation of the scale and strength, the operation of the operation unit 6 is performed. Even if there is no sound, the scale and strength range can be set.
(Embodiment 4)

  In the third embodiment, it is preferable that the image pickup unit 2 is provided with an automatic focus adjustment function so that the lip image is not blurred even when the player 1 approaches or moves away from the image pickup unit 2. The distance information between the subject obtained from the automatic focus adjustment function is supplied to the image analysis unit 3, and even if the performer 1 approaches the imaging unit 2 and the lip image becomes large, the scale is high and the sound is strong. The magnification of the lip image may be further corrected so as not to occur. In this way, the distance between the performer 1 and the imaging unit 2 is not affected by the scale or strength, and the distance information is used to determine the tone of the musical tone and the depth of vibrato, etc. You may make it control the property of a sound.

(Embodiment 5)
The player 1 moves his / her face closer to or away from the imaging unit 2 without performing normalization of individual differences in the size of the lips and the magnification correction based on the distance information described in the third and fourth embodiments. Thus, the scale and volume may be controlled.

(Embodiment 6)
In the above description, the H data that changes continuously is assigned to the scale notes of the C major major scale, and the music with the average rate scale is played. However, the H data corresponds to the continuous pitch as it is oriental. You may also be able to play national or ethnic music.

(Embodiment 7)
Of the tongue coordinate data described in the first embodiment, the horizontal position data of the tongue may be used as the scale data. It is sufficient to make a high scale sound when the tongue is on the right side. When the sound volume is controlled by position data in the front-rear direction of the tongue and the tongue is in the back of the throat, the sound is low, and the sound is loud enough to protrude forward. The vertical direction of the tongue may control various characteristics of sound such as timbre.
In the second embodiment, not only the display of the lip but also the position of the tip of the tongue is displayed. In the third embodiment, instead of the vertical and horizontal widths of the lip, It is only necessary to set the scale range and the strength range using the up / down / left / right position setting buttons.
This method is suitable for a player who can control the position of the tip of the tongue more easily than the control of the mouth opening.

(Embodiment 8)
The nose position detection unit is provided in the image analysis unit 3 in FIG. 3, the player's nose tip position data is extracted as position information from the video signal, and the pitch of the musical tone signal or the pitch is determined by the nose position information. You may control strength. Two-dimensional position information in the vertical and horizontal directions of the performer's nose is extracted and used to control the musical sound.

In the second embodiment, as shown in FIG. 4B, instead of displaying the lip image, the image of the tip of the nose or the point N indicating the tip is displayed on the musical tone template. In this case, the musical sound template has a center position that is one octave or two octaves in the middle of the scale generation range and the middle of the dynamic symbol, mp, or mf. The leftmost position may be a silent state off. In the example of FIG. 4B, an X mark is also displayed at the position of the point N recognized as the tip of the nose. Further, the central position of the musical sound template is the boundary between fa and so, and the boundary between p and mp.
In the musical tone template of FIG. 4B, for the sake of simplicity, the scale is C major, and the chromatic halftones with sharp or flat are omitted. All the chromatic scales may be allocated by increasing the number of lines of the musical tone template.

Based on the same concept as the normalization of the lip size in the third embodiment, an operation of placing the tip portion of the nose image on the musical tone template is performed, and the association between the nose position and the generated musical tone attribute is performed. Do. First, the performer 1 places the face on a bed or a wheelchair in a normal position, and sets the nose position N0 at this time to the center position of the musical tone template in FIG. The coordinates are set at 3. When this setting is finished, the nose is moved upward with the face slightly upward, and this position Nh is set as the position corresponding to the upper so that is the highest scale sound. When this setting is finished, the face is then moved slightly to the right from the normal position and the nose is moved to the right, and this position Nf is set as the position corresponding to the maximum volume and the strong symbol ff. The scale / strength range setting mode ends with the above settings. The operation unit 6 may be provided with buttons for these setting operations.
After the scale / strength range setting mode is completed, the performer 1 can perform music by controlling the scale and sound intensity by slightly moving his / her face up / down and left / right. When the performance is over, you can turn the face a little to the left and stop the sound. In the example of FIG. 4B, the lower symbol is pronounced with an intensity of mf.
Note that the coordinate direction may be assigned to the left and right direction, and the up and down direction may be assigned to dynamic symbols.
It is not necessary to move the lip. If it is difficult to keep the mouth open and control the opening accurately, this embodiment allows music to be played more easily.

The W and H data of the lip and the opening area may also be detected and used for another musical tone control. The control of the strength symbol may be controlled according to the degree of opening regardless of the position of the nose. The nose position control is used for the scale control only for the left and right sides, and when making a loud sound, the mouth is opened slightly larger. Face orientation and mouth size are easy to control independently. It can also be said to be a movement sensation that matches the proportion of human musical sensation. You can usually perform with your mouth closed and your mouth open only when you want to generate a musical tone.
In general, it is difficult to move the position of the nose from one musical scale to another, and it tends to be accompanied by continuous pitch changes like portamento. This is now a musical expression. If portamento is not preferred, it is easy to generate the next scale after closing the mouth and muting the sound.

When the performance is performed so that the mouth is closed immediately after opening the mouth, the sound may be gradually attenuated rather than being immediately turned off. Percussive sounds such as piano and guitar can be played. The rising of the sound, the tightness of the sound, and the timbre may be controlled by the speed at which the mouth is opened or closed. If you close your mouth and leave the previous sound, change the position of your nose to determine the sound of a new scale, and open the mouth to make the next sound. Can be put out. Pseudo and multi-voice music performances can also be performed.
If the face is turned to the left or right, the lip image changes and affects the volume of the sound. Therefore, the head may be moved left and right while looking forward. When the face turns to the left or right, the angle may be detected by image analysis and used to control the musical sound. Conversely, the deformation of the lip image may be corrected based on the angle data.
It should be noted that, as in the case of the sixth embodiment, continuous pitches may be generated instead of discretely assigning the nose position data to the chromatic scale.
In addition, without performing the scale / strength range setting on the musical tone template according to the position of the nose as described above, the position of the imaging unit 2 is set to an appropriate position in front, back, left, and right in front of the player 1, You may make it determine the range of a musical scale, and the range of a dynamic symbol.

(Other embodiments and supplements)
In each of the above embodiments, when the image analysis and image recognition performance of the image analysis unit 3 is insufficient, or the position of the lip, tongue, or nose of the player 1 is difficult to recognize. Although it is not very desirable, it is possible to facilitate image analysis and image recognition of the image analysis unit 3 by attaching a label of a small mark on the top / bottom / left / right position of the lip, the tongue, or the tip of the nose. May be. The landmark may be a substance that responds to a specific wavelength but is not visible to the general public.

The display unit 5 displays an image of a part of the lip, tongue, nose, or face extracted from the video information. However, without displaying an actual video, the image of the face of the player 1 is displayed. Then, a figure indicating the size, shape, or position may be drawn, and the artificial figure may be displayed together with the musical sound template. It is only necessary to associate the size, shape, or position of the graphic with the attribute of the generated musical sound by an operation of placing a predetermined portion of the graphic on the musical sound template. The musical sound templates are not limited to those shown in FIGS. 4 (A) and 4 (B).
The musical tone control signals output from the image analysis unit 3 of FIG. 3 to the musical tone synthesis unit 4 may be supplied to the musical tone synthesis unit 4 in the form of MIDI signals well known in the electronic musical instrument field. In this case, since the signals for controlling the musical tone output from the image analyzing unit 3 are collected into one signal, the terminals T1 to T5 of the musical tone synthesizing unit 4 are combined into one.

  The musical instrument of the present invention can be configured on the notebook computer by connecting a camera to the notebook computer and making the operation of the above-described embodiments of the image analysis unit 3 in the form of a computer program. The musical tone synthesis unit 4 can use a musical tone synthesis function provided on a notebook personal computer. 4A and 4B can be displayed on the liquid crystal display portion of the notebook computer. In addition, since a mobile phone is often equipped with a camera and a musical sound synthesis function, the mobile phone can be implemented by installing the function of the image analysis unit 3 as software on such a mobile phone. It can be used as a musical instrument of the invention.

  In other words, the processing of the musical instrument of the present invention in all the above embodiments may be realized by software. Then, this software may be distributed by software download or the like. Further, this software may be recorded and distributed on a recording medium such as a CD-ROM. The software includes an imaging instruction step for instructing a computer to photograph the player's face or a part thereof, and an image analysis step for analyzing a video signal that is an imaging result corresponding to the instruction in the imaging instruction step and extracting a feature amount. And a program for executing a tone synthesis step for generating a tone signal based on the feature amount. In addition, for the above program, a musical tone template is created on the computer by using a lip, tongue, nose, or part of face image extracted from the video signal, or a figure indicating the size, shape, or position of the image. It may be a program for further executing the step of displaying together. Furthermore, a step of accepting an operation for placing a predetermined part of an image or a graphic on the musical sound template with respect to the program, and the size of the image or the graphic, the shape or the graphic by receiving the operation. A program for further executing the step of associating the position with the attribute of the generated musical sound may be used.

  The musical instrument according to the present invention is useful for a paralyzed person or a physically handicapped person to enjoy playing a musical instrument, can be realized on various information devices, and is an instrument used by a healthy person. It is also possible.

The block diagram of one Embodiment of the musical instrument of this invention The figure explaining the feature-value of the player's mouth of the musical instrument of this invention The block diagram of the principal part of one Embodiment of the musical instrument of this invention The figure which shows the example of a display of one Embodiment of the musical instrument of this invention

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Player 2 Image pick-up part 3 Image analysis part 4 Musical sound synthesis part 5 Display part 6 Operation part 31 H detection part 32 W detection part 33 Tongue position detection part 34 Right eye detection part 35 Left eye detection part T1-T5 Input terminal

Claims (17)

An imaging unit for photographing the performer's face or a part thereof;
An image analysis unit that analyzes a video signal output from the imaging unit and extracts a feature amount;
A musical instrument comprising a tone synthesis unit that generates a tone signal based on the feature amount. The image analysis unit
The player's lip is extracted from the video signal, and height information and width information of the opening portion of the lip is extracted as the feature amount,
The musical tone synthesis unit
The musical instrument according to claim 1, wherein the generation of the tone signal is controlled by the height information and the width information. The musical tone synthesis unit
The musical instrument according to claim 2, wherein one of a pitch and a volume of the musical tone signal is controlled by the height information, and the other is controlled by the width information. The image analysis unit
From the video signal, the player's nose position information is extracted as the feature amount,
The musical tone synthesis unit
The musical instrument according to claim 1, wherein the generation of the musical tone signal is controlled by the position information of the nose. The image analysis unit
From the video signal, two-dimensional position information in the vertical and horizontal directions of the performer's nose is extracted as the feature amount,
The musical tone synthesis unit
The musical instrument according to claim 4, wherein one of pitch and volume of the musical tone signal is controlled by the position information in the vertical direction, and the other is controlled by the position information in the horizontal direction. The image analysis unit
From the video signal, the position information of the player's tongue is extracted as the feature amount,
The musical tone synthesis unit
The musical instrument according to claim 1, wherein generation of the musical tone signal is controlled based on position information of the tongue. The image analysis unit
From the video signal, two-dimensional position information in the vertical and horizontal directions of the performer's tongue is extracted as the feature amount,
The musical tone synthesis unit
The musical instrument according to claim 6, wherein one of pitch and volume of the musical tone signal is controlled by the position information in the vertical direction, and the other is controlled by the position information in the horizontal direction. The image analysis unit
Extract the three-dimensional position information of the player's tongue in the vertical, horizontal and depth directions,
The musical tone synthesis unit
The musical instrument according to claim 6, wherein any one of the pieces of positional information controls any one of a pitch, a volume, and a timbre of the musical tone signal. The image analysis unit
From the video signal, as the feature amount, the open / close pattern information of the performer's heel or lip is extracted,
The musical tone synthesis unit
The musical instrument according to claim 1, wherein an attribute of the musical sound signal or an operation of the musical instrument is controlled based on the opening / closing pattern information. A display unit;
An image of a part of a lip, tongue, nose, or face extracted from the video signal, or a graphic indicating the size, shape, or position of the image is displayed on the display unit together with a musical sound template. 1. The musical instrument according to 1. An operation for placing the image or a predetermined portion of the graphic on the musical tone template is accepted, and the size of the image, the shape of the graphic, or the position of the graphic is generated by the reception of the operation. The musical instrument according to claim 10, wherein the musical instrument is associated with a musical sound attribute. An imaging step for photographing the performer's face or part thereof;
An image analysis step of analyzing a video signal that is an output in the imaging step and extracting a feature amount from a face or an expression;
A musical tone control method comprising a musical tone synthesis step for generating a musical tone signal based on the feature amount. A step of displaying an image of a part of the lip, tongue, nose, or face extracted from the video signal, or a graphic indicating the size, shape, or position of the image on the display unit together with the musical sound template. The musical sound control method according to claim 12. Receiving an operation of placing a predetermined portion of the image or the graphic on the musical sound template; and receiving the operation, the size of the image, the shape of the graphic, or the position of the graphic; 14. The musical tone control method according to claim 13, further comprising a step of associating with an attribute of a musical tone to be generated. On the computer,
An imaging instruction step for instructing photographing of the performer's face or a part thereof;
An image analysis step of analyzing a video signal which is an imaging result for an instruction in the imaging instruction step and extracting a feature amount;
A program for executing a tone synthesis step for generating a tone signal based on the feature amount. On the computer,
In order to further execute a step of displaying an image of a part of the lip, tongue, nose, or face extracted from the video signal, or a graphic showing the size, shape, or position of the image together with the musical sound template. The program according to claim 15. Receiving an operation of placing a predetermined portion of the image or the graphic on the musical sound template; and receiving the operation, the size of the image, the shape of the graphic, or the position of the graphic; The program according to claim 16, further executing a step of associating with an attribute of a generated musical sound.

Images