JPH10143155A - Formation of virtual musical instrument - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a method for formation of a virtual musical instrument which forms the virtual musical instrument on a computer screen. SOLUTION: An object 111 (musical instrument) which sets the occurrence of a formation event, graphics 112 (pallet) which simulate the keyboard musical instrument and graphics 113 (paintbrush) interlocked to a mouse are displayed on the screen. Button-like regions which generate different intervals are arrayed on a pallet 112. A user brings the paintbrush 113 to his desired interval and selects the interval by clicking the mouse. Next, the user brings the paintbrush onto the musical instrument and moves the paintbrush while pushing the mouse button, thereby successively setting the region where MIDI information is generated. The selected contents are checked by the user as the corresponding sound is generated at the time of clicking with the mouse 131. As a result, the user is capable of designing the musical instrument of the free shape and timbre meeting his own taste.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for creating a virtual musical instrument by a computer.

[0002]

2. Description of the Related Art As a means for inputting information, a system (window system) that uses a device such as a mouse capable of visually inputting coordinate values on a screen functionally divided into rectangular areas (windows) is known. Currently, it is mainstream in many computers. In developing an interface with a user in such a system, not only the function but also the visual design on the screen is important. In recent years, an environment in which such a development process itself can be visually performed has been established.

In exchanging information between a computer and a music sound source device, standard data in the form of MIDI has been used for a long time. This makes it possible to create and operate music information that does not depend on the type of computer or sound source.

At present, there are many software (music software) for creating data for driving a sound source. Data created by music software can be reproduced many times by automatic performance by a computer. There are two types of data input means in music software: input from a MIDI-compatible instrument (mainly a keyboard) and input on a computer. In the latter case, there is software that has an interface such as clicking a figure imitating a keyboard with a mouse.

[0005] In recent years, with the dramatic improvement in the computing power of a computer, a technique for presenting three-dimensional information on a computer screen to a user has been generalized. In addition,
Many software for creating three-dimensional information have been developed. Furthermore, a device (data glove) that can detect the three-dimensional movement of the user's hand and input it to the computer, a technology that displays a stereoscopic image according to the change in the user's line of sight, etc. A system that allows users to experience virtual (virtual three-dimensional space) is also possible.

[0006]

In order to respond to various user preferences when using a computer, an interface design having a higher degree of freedom than a current functional division of a screen mainly including a rectangular area is required. Required. In addition, in order for the user to satisfy his or her taste, it is desirable that the user himself can design. Therefore, software that can easily design such an interface is required.

[0007] In conventional music software, a function for allowing a computer to output MIDI information in real time by a user is provided as an interface at the time of data creation.
Therefore, the act itself is not a performance in a daily sense. In order to further diversify a user's relationship with music using a computer, it is necessary to use the computer as an interface for performing.

[0008] Conventionally, three-dimensional information has been mainly used for visual expression. On the other hand, music occupies an important position in human expression activities. In order for human expression activities to be more comprehensively performed on systems that handle 3D information,
A system configuration that includes music activities is required. Conventionally, the input of the musical instrument of the MIDI information is mainly a keyboard musical instrument. In order for a computer to contribute more to human creative activities, it is desirable to realize musical input in a more free form.

[0009]

In order to solve the above-mentioned problems, in the present invention, the function of a computer screen is designed using an interface of drawing software existing as a conventional technique. Then, an area for outputting MIDI information is set on the computer screen using the interface of the drawing software. This allows the user to
You can create and play instruments in a three-dimensional space.

[0010]

BEST MODE FOR CARRYING OUT THE INVENTION

(Embodiment 1) In this embodiment, a rectangular area on a computer screen is
This software sets the area where the computer can output MIDI information to an external sound source with a mouse click. Thereby, a figure imitating a musical instrument is created on the screen.

FIG. 1 shows an example of a user's work procedure in the creation process.
(A) to (c) are shown. On the screen, a target 111 for setting creation event occurrence (hereinafter, referred to as an instrument), a figure 112 imitating a keyboard instrument (hereinafter, referred to as a palette), and a figure 113 (hereinafter, referred to as a paintbrush) linked to a mouse are displayed. Is done. Pallets 112
Above, button-shaped areas that generate different pitches are arranged. On the screen 110, the user brings the paintbrush to the pitch he / she wants and clicks the mouse,
Select that pitch. The selected content is confirmed by the user by generating a corresponding sound when clicked. Next, a paintbrush is brought over the musical instrument, and by moving the paintbrush while holding down the mouse button, an area for generating MIDI information, such as the screen 120, is set. The set area is confirmed by the user by coloring the area. The work of the user at this time is exactly the same as that of ordinary paint software. By setting the transparency, a plurality of timbres can be mixed on a musical instrument. The number of musical instruments and pallets is arbitrary, and musical instruments created by the user himself can be used as pallets.

When the user clicks on the pallet where no sound is produced, the creation process ends. In this case, as in the case of the screen 130, the graphic linked to the mouse becomes a graphic 131 different from the paintbrush. In this state, clicking on the musical instrument generates a corresponding musical tone. It is also possible to set so that a special effect such as continuous pitch change or vibrato can be added to the timbre by moving (dragging) the mouse while holding down the button. The setting of the mouse drag effect is performed for each instrument.

The basic data structure and algorithm of the software of this embodiment are shown in FIGS.

As shown in FIG. 2, the panel is data representing a musical instrument and a palette. This software assumes operation on a window system, and each panel is realized as one window. The panel includes:
The state indicating whether the panel is a musical instrument or a palette, a list of areas for generating MIDI information, and a process performed at the time of mouse dragging are included as data.

In addition, data related to the window system is also included, but this is omitted. Each area on the panel is defined by a set of coordinate values on the panel that accept mouse clicks and data (color and MIDI data) required by a program executed at the time of clicking.
A plurality of MIDI data is assigned to each area as a list. The paintbrush data stores the current state of the paintbrush, and there is only one for one application process. The content of the area most recently clicked is stored in the paintbrush. The state field in the paintbrush indicates whether the content is an area on a musical instrument or an area on a pallet, thereby distinguishing between a creation mode and a performance mode. The other fields are the transparency (how to add) when the brush pattern and the tone at the time of painting are defined in multiples.

Similar to a program on a normal window system, the software of the present invention executes a procedure defined for a panel when an event is generated by a mouse operation. The three events recognized as events are mouse click shown in FIG. 3, dragging the mouse while pressing the button shown in FIG. 4, and releasing the button shown in FIG. However, when the mouse comes out of the panel area, the same operation as releasing the button is performed.

As shown in FIG. 6, a key note on is sent to the external sound source when the mouse is clicked, and a key note off is sent when the mouse is released as shown in FIG. Further, as shown in FIG. 8, in the case of a musical instrument for which processing at the time of mouse dragging is set, vendor information and the like are sent. In the example of FIG. 8, when the mouse is moved up and down, the pitch changes continuously. The content output to the external sound source is performed by referring to a paintbrush.

The setting of the area at the time of musical instrument creation is performed by adding the locus of the binary image brush pattern on the panel to the area as shown in FIG. Paintbrush transparency is 0
If not, the overlap with the previously set area is checked. If there is an overlap, another area is created with color and MIDI data added according to the transparency. On the same panel, if the contents of the areas match, the areas are managed as the same area even if they are not continuous areas.

FIG. 10 shows an example of a musical instrument actually created by a user. Various melodic shapes can be obtained by moving the mouse. By performing repetitive operations, it is possible to perform percussion instruments.

(Embodiment 2) This embodiment is an example in which the present invention is realized on the system shown in FIG. 11 which allows the user to experience virtual reality. The image display device 1180 is provided with a computer 1110 so that a user can experience a sense of depth of an object in a virtual space.
Displays an image generated using the image generation device 1140 on both eyes of the user. Headphones 1160, MIDI calculator 1110
The sound image generated using the sound source 1130 is presented. On the other hand, the position and direction of the user's head are input to the computer 1110 via the device 1170, and the movement of the user's hand is input to the computer 1110 via the I01150 via the device 1190. Computer 1110 has an external storage device
1120 is connected. On this system, the user can experience a virtual space that changes in conjunction with the movement of the user in an audiovisual manner. These techniques are described in, for example, JP-A-6-47170.

The creation of the shape of a three-dimensional object in a virtual space that is to be a virtual musical instrument is performed by three-dimensional shape creation software (modeler) in a general computer graphics system. The shape data given by the modeler is composed of surface elements obtained by dividing the object surface into polygons.

FIGS. 12 (a), 12 (b), 13 (a),
(B) illustrates a user's work in creating a virtual musical instrument. FIG. 12 is displayed in screen 1210,
This is a panel for executing commands such as virtual space setting, file input / output, and system termination. In the process of creating the musical instrument, the following objects are presented to the user in the virtual space.

1. An object 1311 to be created. Hereinafter, it is called an instrument.

2. A virtual instrument 1312 prepared in advance that is referred to for creation. Hereinafter, it is called a pallet.

3. Object 1 that moves in conjunction with the movement of the user's hand
313. Hereinafter, it is called a hand model.

These are functionally similar to the first embodiment, except that 3 has a data structure of an object in the same virtual space as the other two. If there is only one user, there are one or two hand models, and more if multiple users share the same virtual space.

The user arranges a desired tone color on the surface of the musical instrument by performing an operation of applying a paint on the pallet to the musical instrument. The specific procedure is as follows.

1. Palette selection. As shown in the screen 1220 of FIG. 12, by pressing the button of palletizing, by selecting an appropriate one from the already existing virtual instruments,
Put the virtual instrument in a pallet state.

2. Selection of MIDI information. As shown on a screen 1310 in FIG. 13, a certain sounding area on the palette is touched by the hand model. As a result, the corresponding MIDI information is output to the external sound source, and the color of the touched portion on the hand model changes.

3. Setting of the sounding area. As shown on a screen 1320 in FIG. 13, the instrument is touched by the color-changed part on the hand model. A sounding area for generating the selected MIDI information is set on the surface of the instrument in contact with the changed color part. As in the first embodiment, it is possible to mix sounds on a musical instrument.

4. Deletion of sound generation area. When erasing the area once set, the pallet 1321 for erasure shown in FIG. 13 is used. The operation is the same as the normal area setting. This has the attribute of transparency like ordinary data, and it is also possible to partially lower the volume of the sound generation area.

The created virtual musical instrument is played by contacting a hand model. During the performance process,
Volume data is calculated from the speed at which the contact between the objects occurs, and this is output to the external sound source as MIDI information together with the timbre data set in the sounding area. FIG.
As shown in (a) and (b), while a part of the hand model is sunk into the instrument 1411 after the contact, the temporary embellishment of the timbre, for example, Vibrato is applied deeply when the inset is large.
When the recessed portion moves in parallel with the contact surface 1421, the pitch can be continuously changed. These functions are set for each instrument.

When objects in the virtual space collide with each other, their spatial arrangement changes according to pseudo-physical rules. This is similar to a normal virtual reality system. In this system, in addition to this, MIDI information is generated by the contact of the object with the sounding area. The user can not only play directly with the hand model, but also create more unexpected music by contact between objects. If a hand model deformed in advance by an external modeler is used, it is possible to easily perform a performance that is impossible in reality.

The basic data structure and algorithm of the software of this embodiment are shown in FIGS.

The entity in the virtual space is the panel 12 in FIG.
Except for 11, all are represented by the data of the object in FIG. Each object has data on the shape required for the three-dimensional representation. The shape data includes surface elements and vertex data constituting each object. In this software, since the hand model performs a joint motion, it also has joint information and the like as data. Other ordinary objects are simple rigid models. The mass of the attribute of an object indicates the difficulty of movement due to contact with another object (however, it has no meaning in contact with a hand model). In this software, the deformation of the object is not considered. The stiffness of the attribute indicates how much other objects do not sink. This is important when performing the processing shown in FIG. If it is too hard, the hand model will not get inside the instrument and the instrument will move quickly. Conversely, if the hardness is 0, it will not start moving when it comes into contact with other objects.

The other fields are similar to the data structure of the panel shown in FIG. However, the state of the paintbrush is added as the state of the object. This is because a mouse cursor (hand model) in the first embodiment is also represented as object data. Further, in the present embodiment, the pronunciation area is managed as a set of face elements. On each surface element, a normal vector from the inside to the outside of the object is defined. In FIG. 17, two volume parameters in the MIDI data are used to calculate the actually output volume from the collision speed of the object.

The last contact processing data in FIG. 15 is for managing the state of contact between the sounding area and another object. FIG. 16 shows a basic algorithm of the management method. The outline of the basic processing is as follows.

When a vertex on a certain object passes through a plane element in a certain area for the first time, a set of the area and the object is stored together with the plane element and the vertex. At this time, the contact processing of FIG. 17 is called.
The face elements and vertices at the time of the first contact are used in the calculation of the volume and the drag processing in FIG. When the object no longer spatially intersects with the area, the release process of FIG. 18 is called,
The pair is removed from the contact management table. Note that detection of contact between objects causing a change in spatial arrangement and the like are not shown in FIG. 18 because they are the same as in a normal virtual space system. The area setting process in FIG. 20 is basically the same as that in FIG. 9 of the first embodiment.

[0039]

According to the present invention, a functional area having a free shape can be easily designed on a computer screen. This allows the user to easily configure a computer interface that suits his / her taste.

According to the present invention, it is possible to easily design a figure imitating the function of a musical instrument having a free shape on a computer screen. This also provided the user with the entertainment of playing music through a computer.

According to the present invention, it becomes possible for a user to create and play a musical instrument having a free shape in a virtual three-dimensional world. This has provided new possibilities for musical creativity activities through the user's computer.

[Brief description of the drawings]

FIGS. 1A to 1C are schematic diagrams showing operations for creating a figure simulating a musical instrument using the software of the first embodiment.

FIG. 2 is a data structure diagram of software according to the first embodiment.

FIG. 3 is a processing diagram when a mouse is clicked by the software of the first embodiment.

FIG. 4 is a processing diagram at the time of mouse dragging by the software of the first embodiment.

FIG. 5 is a processing diagram at the time of mouse release by the software of the first embodiment.

FIG. 6 is a key note-on processing diagram using software according to the first embodiment.

FIG. 7 is a diagram showing a key note off processing by software of the first embodiment.

FIG. 8 is a sound source control diagram at the time of dragging by the software of the first embodiment.

FIG. 9 is a diagram illustrating a setting process of a sound generation area in software according to the first embodiment.

FIG. 10 is an exemplary diagram of a graphic created by the software of the first embodiment.

FIG. 11 is a configuration diagram of a system that realizes virtual reality according to a second embodiment.

FIGS. 12A and 12B are pallet selection diagrams using software according to the second embodiment.

FIGS. 13A and 13B are musical instrument creation diagrams using software according to the second embodiment.

FIGS. 14A and 14B are diagrams showing a method of causing a musical tone to change temporarily when playing a musical instrument using the software of the second embodiment.

FIG. 15 is a data structure diagram of software according to the second embodiment.

FIG. 16 is a processing diagram of a main part of software according to the second embodiment.

FIG. 17 is a processing diagram of contact between a sound generation area and an object by software according to the second embodiment.

FIG. 18 is a processing diagram of the software according to the second embodiment when a sound generation area and an object are separated.

FIG. 19 is a processing diagram of the software according to the second embodiment when an object is immersed in a sound generation area.

FIG. 20 is a diagram illustrating a tone generation area setting process using software according to the second embodiment.

Claims (5)

[Claims] 1. A virtual musical instrument for setting an area on a computer screen including a coordinate point capable of generating a program execution instruction by specifying a coordinate point using an apparatus for specifying a coordinate point on the computer screen. The method of creating a virtual musical instrument, wherein the setting of the area is performed by drawing an outline of a figure corresponding to the area on a screen. 2. A virtual musical instrument for setting an area on a computer screen including a coordinate point capable of generating a program execution instruction by specifying a coordinate point using an apparatus for specifying a coordinate point on the computer screen. The method of creating a virtual musical instrument, wherein the area is set by simulating an operation of applying a paint with a paint by the coordinate point specifying device. 3. Specifying a coordinate point using an apparatus for specifying a coordinate point on a computer screen allows the computer to receive MIDI data.
A plurality of areas on the computer screen including coordinate points capable of generating an execution instruction of a program for outputting information to an external sound source are set, and a figure simulating a musical instrument playable by the coordinate point specifying device is displayed on the screen. A method for creating a virtual instrument, comprising: 4. Using a virtual three-dimensional space realized by software capable of displaying a plurality of three-dimensional objects formed on a storage device of a computer,
MI contact the computer by contact with different 3D objects
It is characterized in that an area for generating an execution instruction of a program for outputting DI information to an external sound source is set, and a contact between the three-dimensional objects is generated to create a three-dimensional object simulating a musical instrument that can be played. How to create a virtual instrument. 5. A three-dimensional object comprising: an external device for detecting a motion in a real three-dimensional space outside; and software for presenting the motion detected by the device as a motion of a three-dimensional object in a virtual space. A virtual musical instrument characterized by performing an act of simulating a work of applying paint with paint on a surface, and outputting MIDI information to an external sound source to a computer by contact with a different three-dimensional object.