JP4736771B2 - Sound effect generating device, sound effect generating method, and computer program - Google Patents

Description

  The present invention relates to a sound effect generating device, a sound effect generating method, and a computer program for generating a sound effect when an interaction such as collision or contact between objects occurs in a three-dimensional virtual space, and in particular, between virtual objects. The present invention relates to a sound effect generating device, a sound effect generating method, and a computer program that generate a realistic sound effect when the above-described interaction occurs while maintaining the real-time property required by the VR system.

  In 3D graphics technology, 3D objects are treated as a collection of numerous polygons, that is, approximate mathematical models, textures that express the texture are mapped on the surface, and light source calculations are performed to create shadows and shades on the texture. By providing the above, it is possible to generate a real and three-dimensional high-definition image.

  In addition, development of computer animation for automatically generating animation of a three-dimensional object on a computer by applying morphing and other methods has been actively performed. Recently, due to advances in rendering technology, the reality of computer animation has been greatly improved, and it has been applied to various fields using virtual reality (VR), including computer games.

  While the reality on the drawing surface of computer animation has increased, there is still room for improving the reality of other elements such as the movement, touch, and sound of objects. Further, in the VR system, speeding up of processing and a method for applying the system to the system are problems, and among them, an important requirement is to achieve both real-time characteristics and high quality images.

  This specification deals with a method for improving the reality and the sense of reality of computer animation by artificially synthesizing sound effects when interaction between objects such as collision and contact occurs.

  With regard to automatic sound generation using computer technology, for example, a musical sound synthesizer that synthesizes musical sound waveforms by simulating the sound generation mechanism of various natural musical instruments has been proposed (for example, see Patent Document 1). In the apparatus, a drive signal of the physical model sound source is generated using the excitation envelope signal generated by the envelope signal generation unit, and an amplitude envelope is added to the musical sound signal using the amplitude envelope signal generated by the envelope signal generation unit. To. Also, even when the amplitude envelope is retriggered, the excitation envelope is not retriggered, and a constant drive signal is input to the physical model sound source so that the excitation is sustained. Therefore, it is possible to easily perform sound generation and performance control, and to perform performance such as one note and one note crescendo without causing a sound delay. However, this device can automatically generate a sound similar to a natural musical instrument, but it solves the problem of what kind of sound effect should be given to interaction between objects such as collision or contact in a three-dimensional virtual space. Not what you want.

  Also, a three-dimensional object presentation device that feeds back sound generated when a physical object is scratched or hit in response to an interactive command input from a user via a three-dimensional viewer has been proposed. (See, for example, Patent Document 2). However, in this apparatus, the sound generated when the original object described as the 3D model data is hit or rubbed is attached to the 3D model data and reproduced according to the interaction from the user. However, it does not generate sound according to the degree of impact at the time of collision or the mechanical quantity of the colliding opponent, and lacks reality when applied to a VR system.

  In addition, an object interference expression device that expresses the degree of interference of a plurality of objects with an impact sound has been proposed (see, for example, Patent Document 3). However, in this device, for example, impact sound data when an object of the same material collides is stored in advance for each case of impact of large, medium, and small, and obtained as a result of an impact deformation analysis simulation. Depending on the degree of impact, some of the impact sound data is reproduced and output as it is, so that the user can audibly recognize the degree of interference of the object, but the sound is inaccurate and sufficient reality It seems that it cannot give a feeling.

  Currently, the synthesis method of the sound source adopted in the VR system basically relies on a simple method of generating pre-sampled speech, and synthesizes realistic sound supported by a physical model. I can't do it.

  On the other hand, real-time processing is important in a system using a three-dimensional virtual space. In an application such as a computer game, it is assumed that a large number of objects interact with each other at the same time. However, if an excessively precise physical model is constructed, real-time processing becomes difficult. In other words, there is a demand for a sound effect generation algorithm that is simple but has a high effect of creating reality.

Japanese Patent Laid-Open No. 9-305177 (Japanese Patent No. 3044375) JP 2001-256510 A JP 2004-252756 A

  An object of the present invention is to generate a realistic sound effect when an interaction such as collision or contact between objects occurs in a three-dimensional virtual space while maintaining the real-time property required by the VR system. Another object is to provide a sound effect generating device, a sound effect generating method, and a computer program.

  A further object of the present invention is to provide an excellent sound effect generating device capable of synthesizing a realistic sound effect when an interaction such as collision or contact between objects occurs in a three-dimensional virtual space by supporting a physical model. And a sound effect generation method, and a computer program.

The present invention has been made in consideration of the above-mentioned problems, and a first aspect thereof is a sound effect for generating a sound effect associated with a physical interaction between objects in a virtual space where a plurality of virtual objects coexist. A generating device,
Interaction detecting means for detecting an object in which physical interaction occurs in the virtual space;
A dynamic quantity calculating means for calculating a dynamic quantity acting on the object by the interaction using a dynamic model that determines the behavior between the objects in which the physical interaction works based on physics;
Using the calculated mechanical quantity, a sound effect generating means for generating a sound effect generated in the object in accordance with the interaction;
It is provided with the effect sound production | generation apparatus characterized by the above-mentioned.

  With the recent advancement of information technology, VR systems using virtual reality have been applied in various ways. However, while the reality on the drawing side of computer animation has increased, other factors such as sound effects are not realistic. There is still room for improvement. In many systems, basically, sound data that has been recorded or generated in advance is synchronized with the timing of physical interaction in the virtual space, such as collisions between objects, and the sound is simply output. It seems that it lacks elaboration and cannot give enough sense of reality.

  On the other hand, according to the present invention, collision between objects in the three-dimensional virtual space and other physical interactions are calculated using dynamic simulation, and a realistic and precise object is obtained based on the obtained dynamic quantity. Movements and touch, and sound effects at the time of collision can be derived.

  The operation space is a space for describing the relationship between the force acting on the object and the generated acceleration. Using the operation space physical quantities such as the operation space inertia inverse matrix and the operation space bias acceleration, a relational expression between the acceleration and force of the operation space can be described. Then, the acceleration and force satisfying the linear complementarity problem consisting of the constraints imposed when physical interaction such as collision between objects works is obtained by using these operation space inertia inverse matrix and operation space bias acceleration. Based on the applied force, it is possible to determine a collision force acting on the object, and further a mechanical quantity such as velocity, acceleration, impulse, and energy.

  If the operation space inertia inverse matrix is obtained by calculation as defined, the calculation amount is enormous, and there is a problem that it is not suitable for the real-time property required for the VR system. Therefore, in the present invention, by using a dynamic model that determines the behavior of an object existing in a three-dimensional virtual space based on physics, a forward dynamics calculation FWD that obtains a generalized acceleration from a generalized force is executed, thereby performing an operation. The spatial inertia inverse matrix and the operational space bias acceleration were calculated at high speed.

  For example, the operation space physical quantity calculation means sets the operation space in the direction of action of the vertical reaction force and friction force acting on the virtual object by physical interaction, and the mechanical quantity calculation means sets the restitution coefficient specified between the virtual objects. Collisions that work on virtual objects when physical interactions occur by setting linear complementarity problems using constraints to achieve repulsion and appropriate frictional forces and solving the linear complementarity problems Mechanical quantities such as force can be determined.

  Further, the operation space physical quantity calculation means sets an operation space in the relative position deviation direction between virtual objects, and the mechanical quantity calculation means linearly uses a constraint condition for setting the relative positions between virtual objects. By setting a complementarity problem and solving the linear complementarity problem, it is possible to determine a mechanical quantity such as a collision force acting on a virtual object when a physical interaction occurs.

  Further, when the virtual object on which the physical interaction works is a link structure configured by connecting a plurality of rigid bodies such as a humanoid character, the dynamic quantity calculation means includes each of the link structures The linear complementarity problem may be set using a constraint condition regarding the position or posture of the link.

  The sound volume is basically governed by an envelope waveform that envelopes the vibration waveform of the sound. Therefore, in the sound effect generating device according to the present invention, the sound effect generating means determines the amplitude of the envelope waveform based on the mechanical quantity acting on the object, so that there is a realistic effect with a physical sense and realism. Generated sound. For example, when a large collision occurs, the absolute value of the amplitude of the envelope waveform increases.

  Each object has vibration attributes including a sound effect attenuation characteristic and a natural vibration waveform. The sound effect generating means adds an attenuation characteristic to the envelope waveform based on the attenuation characteristic of each object that has undergone physical interaction, and synthesizes the natural vibration waveform of each object with the envelope waveform, thereby creating a sense of presence. It is possible to calculate a precise sound effect with

  In the virtual space, the sound at the current time can be considered as the sum of impulse responses of all the collisions that have occurred in the past. In view of this, a generated sound list management unit for registering information on each sound effect generated for each physical interaction as a past sound is provided, and the past sounds registered in the generated sound list management unit are summed on the time axis. Thus, the sound in the virtual space at the current time may be calculated.

  In practice, the volume of the collision sound decreases according to the envelope waveform as time elapses from the collision of the object, and becomes unintelligible eventually. That is, it is not necessary to go back all past times, and it is only necessary to calculate a collision for a finite time that can substantially affect the sound at the current time. Therefore, the generated sound list management unit may perform a process of excluding a sufficiently attenuated past sound from the generated sound list.

According to a second aspect of the present invention, there is provided a computer so as to execute a process for generating a sound effect accompanying a physical interaction between objects in a virtual space in which a plurality of virtual objects coexist on a computer system. A computer program written in a readable format for the computer system
An interaction detection procedure for detecting an object in which physical interaction occurs in the virtual space;
A dynamic quantity calculation procedure for calculating a dynamic quantity acting on an object by interaction using a dynamic model that determines the behavior between objects in which physical interaction works based on physics;
A sound effect generation procedure for generating a sound effect generated in the object in accordance with the interaction using the calculated mechanical quantity;
Is a computer program characterized in that

  The computer program according to the second aspect of the present invention defines a computer program described in a computer-readable format so as to realize predetermined processing on a computer system. In other words, by installing the computer program according to the second aspect of the present invention in the computer system, a cooperative action is exhibited on the computer system, and the sound effect according to the first aspect of the present invention is achieved. The same effect as that of the generating device can be obtained.

  According to the present invention, it is possible to generate a realistic sound effect when an interaction such as collision or contact between objects occurs in a three-dimensional virtual space while maintaining the real-time property required by the VR system. The sound effect generating apparatus, the sound effect generating method, and the computer program can be provided.

  In addition, according to the present invention, it is possible to synthesize a sound effect having a reality when an interaction such as collision or contact between objects occurs in a three-dimensional virtual space, and to synthesize the sound effect with excellent support of a physical model. An apparatus, a sound effect generation method, and a computer program can be provided.

  In addition, according to the present invention, sound effects when an interaction between objects such as a collision and a contact occurs are artificially synthesized by a mathematical model based on physics, and the reality and sense of presence of a computer animation can be obtained. An excellent sound effect generation device, sound effect generation method, and computer program that can be improved can be provided.

  With the recent improvement in computing power, real-time mechanics-based CG animation, in which forward kinematics simulation is executed in the background and a three-dimensional object is driven based on the calculation result, is becoming realistic. According to the sound effect generation method of the present invention, it is possible to realize a game capable of presenting the reality of physical interaction such as the reality of the movement of a three-dimensional object, pressing and grabbing, in cooperation with a dynamic simulation. In addition, according to the sound effect generation method of the present invention, it is possible to generate sound effects with physical grounds in real time, interactively and efficiently, which contributes to further improving the reality of CG animation. Will.

  Other objects, features, and advantages of the present invention will become apparent from more detailed description based on embodiments of the present invention described later and the accompanying drawings.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

System Configuration FIG. 1 schematically shows a hardware configuration of a three-dimensional virtual space presentation device 100 according to an embodiment of the present invention. The apparatus 100 has a function of presenting the state of a three-dimensional virtual space constructed on a computer to a user via a speaker 104, a display 105, and a haptic device (haptic device) 106, for example, a computer. -It can be configured as a game machine or a part thereof.

  The speaker 104 outputs sound related to an event that occurs in the three-dimensional virtual space, the display 105 presents visual information of the three-dimensional virtual space, and the haptic device 106 touches an object in the three-dimensional virtual space. Presents haptic information that can be felt. According to the three-dimensional virtual space presentation device 100 according to the present embodiment, these input / output forms are used in a coordinated manner, as if the three-dimensional virtual space constructed on the computer is a real space. It can be presented to the user with high reality and presence.

  The CPU 101 expands and executes a computer game or other application stored in the ROM 103 or an external storage device (not shown) on the RAM 102, thereby executing temporal changes in the three-dimensional virtual space, sound information, video information, Force information is sequentially calculated and written to the RAM 102. Sound information, video information, and force information are presented to the user via the speaker 104, the display 105, and the haptic device 106, respectively. Further, each of the devices 104 to 106 described above is connected by a high-speed bus 107 having a transfer speed sufficient to ensure real-time properties, and is configured to be able to exchange information with each other.

  The three-dimensional virtual space presentation apparatus 100 according to the present embodiment is particularly characterized in that information about sound is presented in real time and with high reality, and a sound effect when an interaction between objects occurs is physically generated. It can be artificially synthesized by mathematical models based on science to improve the reality and realism of computer animation. Hereinafter, a method for calculating information related to sound, which is executed by the CPU 101, will be mainly described.

  FIG. 2 schematically shows a functional configuration of a program that is executed on the CPU 101 and calculates information related to sound.

  The state of the virtual three-dimensional space to be processed in the three-dimensional virtual space presentation device 100 is stored in the ROM 103 or an external storage device (not shown) as three-dimensional object data relating to each virtual object existing in the space. This three-dimensional object data includes (1) geometric attributes such as position, orientation, dimensions, and connection relations, (2) mechanical attributes such as mass, inertia tensor, friction, and coefficient of restitution, and (3) damping characteristics and inherent characteristics. It is also assumed that at least data relating to vibration attributes such as vibration waveforms is held.

  The forward dynamics calculation unit 21 calculates a mechanical quantity related to the behavior of each object in a three-dimensional virtual space by forward dynamics calculation (FWD) as a core calculation used for so-called dynamic simulation. Specifically, the forward dynamics calculation unit 21 uses the geometric attributes and dynamic attributes of each object stored as three-dimensional object data, based on Newton-Euler dynamics, The acceleration generated in the object is calculated from the force acting on the outside of the object, and the simulation of the movement of the object is performed. And the position and attitude | position in the three-dimensional virtual space of each object are changed according to the result of forward dynamics calculation. By reflecting changes in position and orientation in the animation, each object exhibits realistic movement based on the laws of mechanics. Details of the forward dynamics calculation method will be described later.

  The collision detection unit 22 detects a collision of each object in the three-dimensional object data and calculates a set of collision points on the object surface. The collision detection algorithm for realizing the present invention is not particularly limited. For example, a collision between objects is performed using a contact shape calculation method described in Japanese Patent Application No. 2005-289598 already assigned to the present applicant. Can be calculated.

  The collision force calculation unit 23 calculates an external force that acts on the collision point calculated by the collision detection unit 22. The calculation method of the collision force can be determined using, for example, the operation space physical quantity and the linear complementarity problem, and details of this point will be described later.

  An object to which a collision or other external force is applied emits a collision sound based on the natural vibration waveform of the object, and the volume of the collision sound having the natural vibration waveform as time elapses from the time of the collision. Is treated as decaying. The damping characteristic and the natural vibration waveform are included in the three-dimensional object data as the vibration characteristic of the object (described above). Further, as shown in FIG. 3, a waveform obtained by synthesizing the natural vibration waveform of the object and the envelope waveform is a collision sound emitted from the collided object.

  In the present embodiment, the envelope waveform calculation unit 24 has an effect on all the objects for which a collision is detected by the collision detection unit 22 according to physical quantities such as external force and speed change of the object obtained by the collision force calculation unit 23. Calculate the envelope waveform of the sound. As shown in FIG. 3, the envelope waveform is a waveform that dominates the volume of the sound effect, and its absolute value increases when a large collision occurs. The degree of attenuation is determined by a method described later in addition to the vibration attribute of each object in the three-dimensional object data. By reflecting the result of the dynamic calculation in the envelope waveform in this way, it is possible to effectively generate a sound effect having a physical basis. Therefore, it is possible to expect a synergistic effect to improve virtual reality in addition to presenting a real motion based on the law of mechanics to the object using the dynamic simulation result by the forward dynamics calculation unit 21.

  The natural vibration waveform setting unit 25 sets natural vibration waveforms related to all collisions. The natural vibration waveform of each object corresponds to the timbre emitted by the object. As shown in FIG. 3, the natural vibration waveform has a higher frequency than the envelope waveform, and has a large calculation amount when calculated based on a physical model. For this reason, in this embodiment, it is assumed that the vibration attribute data of each object is registered as one of the static natural vibration data. For example, a metal natural vibration waveform is held in a metal object, and a plastic natural vibration waveform is held in a plastic object as a vibration attribute of each object. The natural vibration waveform setting unit 25 sets a natural vibration waveform related to a collision based on vibration attributes of individual objects in these three-dimensional object data.

  As described above, the sound generated by the collision of the object is determined by the envelope waveform and the natural vibration waveform of the object. Therefore, in the present embodiment, a set of the envelope waveform set by the envelope waveform setting unit 24 and the natural vibration waveform set by the natural vibration waveform setting unit 25 is used to configure a sound generated by collision of an object in the virtual space. Treated as generated sound information. The generated sound list management unit 26 registers the generated sound information obtained sequentially as a past sound in the past generated sound list. In addition, since the volume of the collision sound decreases according to the envelope waveform as time elapses from the collision of the object and eventually becomes inaudible, processing for sufficiently excluding past generated sound information from the past sound list is performed.

  The sound effect generated by each collision is obtained by a composite wave of the envelope waveform and the natural vibration waveform. The envelope waveform calculation unit 24 calculates the envelope waveform according to the collision force obtained by the collision force calculation unit 23, but the collision force calculation unit 23 can calculate the collision force at a high speed by a method based on a dynamic simulation. The envelope waveform can be calculated in real time. In other words, since realistic sound effects can be output in real time, a sense of reality can be given to the three-dimensional virtual space.

  The generated sound synthesizing unit 27 synthesizes the envelope waveform and the natural vibration waveform based on each generated sound information registered in the past generated sound list, and obtains the generated sounds accompanying all object collisions. All the generated sounds are superimposed on the time axis, and the generated sound in the three-dimensional virtual space at the current time is calculated. As described above, the generated sound list management unit 26 excludes the generated sound information that is no longer audible after a sufficient time from the collision from the past generated sound list. No processing is required.

  The sound output unit 28 converts the synthesized waveform obtained by the generated sound synthesis unit 27 into a speaker drive signal, and outputs the sound as a sound effect of an object that has caused an interaction in the three-dimensional virtual space.

  Note that in the three-dimensional virtual space presentation device 100 according to the present embodiment, a physical quantity related to a collision between objects is obtained from a dynamic simulation by the forward dynamics calculation unit 21, and the object is transmitted to the user via the hapfix device 106. It is possible to present a force related to the interaction. However, since force presentation is not directly related to the gist of the present invention, description thereof is omitted in this specification.

Real-time calculation of collision force using dynamic simulation In the three-dimensional virtual space presentation device according to the present embodiment, the envelope waveform of the sound effect is calculated based on the dynamic quantity such as the collision force obtained using the dynamic simulation result. Therefore, by multiplying the natural vibration waveform of the collision object with the envelope waveform, it is possible to generate a sound effect with a physical presence and having a physical presence. Furthermore, by introducing a dynamic simulation method capable of calculating a high-speed impact force, real-time sound effects can be generated. Here, a configuration method of the forward dynamics calculation FWD and a collision force calculation method by dynamic simulation will be described in detail.

  In many fields, such as force control, dynamic simulation, computer animation, posture editing, and external force estimation, there is an important concept called “operational space”. This operation space is a space for describing the relationship between the force acting on the object and the generated acceleration. The operation space is basically given by a matrix called an operation space inertia inverse matrix, and an expression method that links force and acceleration can be obtained.

  Using the operation space physical quantities such as the operation space inertia inverse matrix and the operation space bias acceleration, a relational expression between the acceleration and force of the operation space can be described. Then, the acceleration and force satisfying the linear complementarity problem consisting of the constraints imposed when physical interaction such as collision between objects works is obtained by using these operation space inertia inverse matrix and operation space bias acceleration. Based on the applied force, it is possible to determine a collision force acting on the object, and further a mechanical quantity such as velocity, acceleration, impulse, and energy.

  Here, in the calculation as defined, there is a problem that it is not suitable for real-time processing because the calculation amount of the operation space inertia inverse matrix is very large. Therefore, in this embodiment, by using a dynamic model that determines the behavior of an object existing in a three-dimensional virtual space based on physics, by executing a forward dynamics calculation FWD that obtains a generalized acceleration from a generalized force, The operation space inertia inverse matrix and the operation space bias acceleration were calculated at high speed.

  For example, it is known that the equation of motion of the entire link structure formed by connecting a plurality of rigid bodies such as CG characters can be expressed in the form of the following expression (1).

  Here, τ is a generalized force corresponding to the generalized variable q, b is gravity / Coriolis force, f is an external force acting on the operation space, H is an inertia matrix for the joint space of the entire structure, and J is an operation space. Jacobian. The above equation (1) can be modified as follows.

  However, c is the operation space bias acceleration (acceleration generated in the operation space when no external force is applied), and is expressed by the following equation.

Further, Λ ⁻¹ in the above equation (2) is a matrix called an operation space inertia inverse matrix, and as shown in the following equation, the operation space inertia inverse matrix is a matrix that associates force and acceleration at an arbitrary part, It is an important physical quantity that enables applications in many fields, such as force control, dynamic simulation, computer animation, posture editing, and external force estimation.

  In the above equation (2), considering that the left side is obtained from the right side, it is regarded as a problem for obtaining acceleration in the operation space when external force, generalized force, gravity, force related to velocity product (Coriolis force, etc.) is applied. be able to. Although the point which calculates | requires the acceleration in operation space differs from normal forward dynamics calculation, it can be regarded as a kind of forward dynamics calculation. The forward dynamics calculation FWD has a joint model as a dynamic model of a link structure and the speed of each joint, joint torque τ as a generalized force, gravity g, and external force f as parameters. Can be expressed in However, g is a gravitational acceleration.

According to this forward dynamics calculation FWD, from the generalized variable q corresponding to the dynamic model of the link structure and force information acting on the link structure such as the generalized force τ, gravity g, and external force f, the link structure The acceleration generated at each point can be obtained. Here, in the above equation (5), under the constraint conditions in which all of the input parameters of the forward dynamics calculation FWD except for the generalized variable q and the external force f are 0, the force related to gravity, joint force, and velocity product ( The acceleration generated in the operation space in a situation where no Coriolis force or the like is generated can be obtained. That is, the bias acceleration c = 0 in the equation (3). Furthermore, it is understood that the i-th column of the operation space inertial inverse matrix Λ ⁻¹ can be obtained by executing the calculation under the condition that f = e _i , that is, the unit external force is applied only to the i-th operation space. Therefore, if the operation of the following equation (6) is executed for all rows i, the entire operation space inertial inverse matrix Λ ⁻¹ can be obtained.

  Further, by executing the forward dynamics calculation FWD of the above equation (5) with the external force f = 0 and executing it under the constraint condition that only the force generated at the joint, the velocity of the joint, and gravity act, the following equation (7) The operation space bias acceleration c can be extracted as shown in FIG.

In short, by combining the above formulas (6) and (7) and executing the forward dynamics calculation FWD using the generalized variable q as a dynamic model, the operation space inertia inverse matrix which is an important physical quantity in the operation space. That is, Λ ⁻¹ and the operation space bias acceleration c can be obtained.

  The calculation method of the inverse operation space inertia matrix described above is based on the fact that multiple objects coexist in a virtual environment and a physical simulation such as collision and contact between them is reproduced on a computer in real time. This is effective when manipulating the objects inside. The dynamic calculation of the dynamic simulation can be executed by applying a force between the objects as an external force to the forward dynamic calculation FWD.

  FIG. 4 illustrates a state in which physical interaction such as collision and contact is applied between objects. In FIG. 4 (A), collisions between virtual objects are detected at the locations indicated by (a) and (b) in the same figure, but the external force at that time is the repulsion of the restitution coefficient specified between the objects and appropriate Therefore, it can be determined by solving the linear complementarity problem expressed by the following equations (8) to (10) so as to realize a proper friction.

Here, f _Ni and f _Fi existing in the above formulas (9) and (10) respectively represent an external force and a frictional force in the normal direction at the point of action i due to the constraint on the contact of the object with the outside world. μ _i is the coefficient of friction at the point of action.

The above equations (8) to (10) are called linear complementarity problems. If the operation space inertia inverse matrix Λ ⁻¹ and the bias acceleration c are known, by solving this linear complementarity problem, the x acceleration and force f satisfying these can be determined. Since the mathematical solution itself of the linear complementarity problem is described in, for example, “Fast Contact Force Computation for Non-Penetic Rigid Bodies” (SIGGRAPH 94, pp. 23-34, 1994), description thereof is omitted here.

An operation space is set in the direction of action of the vertical reaction force and friction force at the parts (a) and (b) where the object collides, and the operation space inertia inverse matrix Λ ⁻¹ and the bias acceleration c are increased at high speed by the method described so far. This external force can be obtained accurately in a short time. Therefore, it is possible to execute precise dynamic simulation in real time even in a complicated environment.

  In FIG. 4B, a haptic device D1 is connected to such a real-time dynamic simulation environment, and the object P1 (proxy) in the virtual environment is based on the position of the tip of the haptic device D1. In the example, the position is changed and the force applied to the object P1 is fed back to the haptic device D1.

In this case, the operation space (c) is set with respect to the relative position deviation so that the position of the object P1 matches the command position of the haptic device D1. Similarly to the example shown in FIG. 4A, the operation space (d) is set in the acting direction of the vertical reaction force and the frictional force with respect to the collision between the object P1 and another object. These constraints are also formulated as linear complementarity problems expressed by the above equations (8) to (10). The operation space inverse matrix Λ ⁻¹ having the largest amount of calculation in this configuration can be calculated in real time by the method according to the present embodiment, and the feeling of touching an object in the virtual environment can be expressed on the haptic device D1. It becomes possible to reproduce in real time and precisely.

  Further, in FIG. 4C, in addition to the configuration shown in FIG. 4B, constraints (L1 to L4) regarding positions and postures are provided at various positions of an object (for example, a humanoid character) in the virtual space. It shows the configuration that can be imposed.

  Position and posture constraints can be expressed by equation conditions as shown in equation (8). The operation space may be set in three directions parallel to the world coordinate system at positions where position and posture constraints are set. Further, by applying such constraints on the position and orientation to the relative position and orientation between the object P2 and the object, and setting the operation space (e) here, the object in the virtual space is used by using the haptic device D2. By pulling or twisting a predetermined position of the object and feeding back the force acting on the object P2 to the haptic device D2, the reaction force pulled or twisted in the virtual environment is precisely You can also feel it.

These problems can also be formulated using the operation space and reduced to the linear complementarity problem expressed by the above equations (8) to (10). In the configuration shown in FIG. 4C, the portion with the largest amount of calculation is a calculation portion of the operation space inverse matrix Λ ⁻¹ , but according to the calculation method described above, calculation can be performed in real time. Therefore, for example, the user can perform motion control, posture editing, and the like of the character while feeling in real time the same force as touching an actual object to edit the posture. In addition, such force-based posture editing can also remove the unstable behavior near the singularity often seen in inverse kinematics.

Automatic generation of sound effects based on mechanical quantities In the three-dimensional virtual space presentation device 100 according to the present embodiment, an envelope waveform that envelops the vibration waveform of the sound effects is formed from the result of the dynamic simulation, and the natural vibration of the colliding object has By combining with the waveform, a realistic sound effect with a physical basis is generated. Here, the calculation method will be described in detail.

  The collision detection unit 22 detects an object collision occurring in the three-dimensional object data and calculates a set of collision points on the object surface. Further, the collision force calculation unit 23 calculates the collision force acting on each collision i calculated by the collision detection unit 22 by applying a dynamic simulation (described above).

The envelope waveform calculation unit 24 calculates the envelope waveform of the sound effect for each collision i detected by the collision detection unit 22 according to the physical quantity obtained by the collision force calculation unit 23. If the collision i occurs at time t = 0, the envelope waveform E _i (t) of the sound generated by this collision can be expressed as the following impulse waveform.

In the above equation, E _0i is the rising volume of the sound at which the collision i occurs, and the envelope waveform calculation unit 24 determines this value according to the magnitude of the collision calculated by the collision force calculation unit 23, thereby Or the reality of the sound effect accompanying a contact can be improved. Further, λ _i is a parameter representing the sound attenuation level, and is stored as a vibration attribute (described above) of the three-dimensional object data. Then, assuming that the attenuation parameters of the two objects that have caused the collision i are λ _1i and λ _2i , the attenuation parameter of the collision i can be estimated as an average of the following equation.

Also, the magnitude of the sound generated by the collision is closely related to the magnitude of the collision force. Therefore, as shown in the following equation, a value obtained by multiplying the collision force f _i generated by the collision _{i by} an appropriate gain K _i can be set as the rising volume.

The natural vibration waveform setting unit 25 sets a natural vibration waveform for each collision i. The natural vibration waveform of each object corresponds to the timbre emitted by the object. Assuming that the natural vibration waveforms of the two objects that have caused the collision i are I _1i (t) and I _2i (t), respectively, the natural vibration waveform I _i (t) at which the collision i is generated is expressed as the following equation. Can be estimated.

The sound generated for the collision i is determined by the envelope waveform E _i (t) and the natural vibration waveform I _i (t). Assuming that the natural vibration waveform I _i (t) shown in the above equation is normalized, the sound wave S _i (t) generated by the collision i is expressed by the following equation.

In the present embodiment, a set of envelope waveform E _i (t) and natural vibration waveform I _i (t) is treated as generated sound information constituting sound generated by collision of objects in virtual space, and generated sound list management is performed. The unit 26 registers the generated sound information obtained sequentially as a past sound in the past generated sound list.

The sound S (t) at time t can be considered as the sum of impulse responses of all collisions that have occurred in the past. FIG. 5 shows a state in which impulse responses of individual collisions are synthesized on the time axis. That is, if time t _i has elapsed since the occurrence of the collision i, the sound S (t) is expressed by the following equation.

  Actually, the sound volume of the collision sound decreases according to the envelope waveform as time elapses from the collision of the object, and becomes unintelligible eventually. That is, it is not necessary to go back all past times, and the above equation (16) may be calculated for a finite time collision that can substantially affect the sound at the current time. For this reason, the generated sound list management unit 26 performs a process of excluding those in which the envelope waveform is sufficiently attenuated among the previously generated sounds. Then, the generated sound synthesizer 27 synthesizes the envelope waveform and the natural vibration waveform based on each generated sound information registered in the past generated sound list, thereby obtaining the generated sound associated with all object collisions. All these generated sounds are superimposed on the time axis, and the generated sound in the three-dimensional virtual space at the current time is calculated.

  By generating sound effects in the virtual three-dimensional space by the above procedure, sound effects such as collision sounds and friction sounds can be presented realistically including the texture. Furthermore, when the vibration attributes of all the objects are the same, Expression (16) can be simplified to the following recurrence expression. Here, Δt represents the execution cycle of this calculation.

  Therefore, by executing the above equation (17) for each collision exhibiting the same vibration attribute, the calculation amount can be reduced as compared with the equation (16).

  FIG. 6 shows a processing procedure for synthesizing a sound effect generated by collision or contact of an object in the 3D virtual space presentation device 100 according to the present embodiment in the form of a flowchart. This processing procedure is realized in a form in which the CPU 101 executes a predetermined program.

  First, the collision detection unit 22 detects a collision of each object in the three-dimensional object data, and calculates a collision point set (step S1).

  Subsequently, the external force acting on all the collision points is calculated by the collision force calculation unit 23 (step S2).

  Subsequently, the forward dynamics calculation unit 21 executes forward dynamics calculation, advances the time by a minute time, and updates the position and orientation of the three-dimensional object (step S3).

  The envelope waveform calculation unit 24 sets an envelope waveform of a collision sound generated for each collision (step S4). The rise value of the envelope waveform is given by equation (13), and its attenuation characteristic is given by equation (12). Both values are stored as parameters defining the envelope waveform.

  The natural vibration waveform setting unit 25 sets the natural vibration waveform of the collision sound generated for each collision (step S5). Since the natural vibration waveform of the collision sound generated by the collision between the two objects is given by Expression (14), the natural vibration waveform stored in the vibration attribute of each of the two corresponding objects in the three-dimensional object data is stored. A pointer to the vibration waveform is stored.

  Subsequently, the generated sound information stored in steps S4 and S5 is stored as a structure, and is registered in the past generated sound list as generated sound information (step S6).

  The processes in steps S4 to S6 are executed for all the collision points detected by the collision detection unit 22 in step S1.

  Subsequently, among the generated sound information registered in the past generated sound list, processing for excluding from the past generated sound list those whose envelope waveform value at the current time is smaller than a sufficiently small threshold (step S7). ). As a result, only the generated sound that affects the current sound is reflected in the following sound effect generation process.

Subsequently, the generated sound synthesizer 27 synthesizes the envelope waveform E _i (t) and the natural vibration waveform I _i (t) of the past sound and all the past sounds S _i (t) based on the above equation (16). ) To calculate the current generated sound (step S8).

  The sound calculated in step S8 is converted into a driving signal for the speaker 104 in the sound output unit 28 and output from the speaker 104 (step S9).

  As described above, by repeating the processes of steps S1 to S9, it is possible to generate a realistic sound effect based on the dynamic simulation result.

  The present invention has been described in detail above with reference to specific embodiments. However, it is obvious that those skilled in the art can make modifications and substitutions of the embodiment without departing from the gist of the present invention.

  In this specification, an embodiment in which a collision between objects in a three-dimensional virtual space and other physical interactions are calculated using a dynamic simulation, and a realistic sound effect is generated based on the obtained dynamic quantity. I have explained to the center. However, the gist of the present invention is not limited to this, and the movement and tactile sensation of an object in which physical interaction has occurred are similarly realistic to the user based on the mechanical quantity obtained from the dynamic simulation. Feedback can be realized in real time.

  In short, the present invention has been disclosed in the form of exemplification, and the description of the present specification should not be interpreted in a limited manner. In order to determine the gist of the present invention, the claims should be taken into consideration.

FIG. 1 is a diagram schematically illustrating a hardware configuration of a three-dimensional virtual space presentation device 100 according to an embodiment of the present invention. FIG. 2 is a diagram schematically showing a functional configuration of a program for calculating information related to sound, which is executed on the CPU 101. FIG. 3 is a diagram for explaining a mechanism in which a waveform obtained by synthesizing a natural vibration waveform and an envelope waveform of an object becomes a collision sound emitted from the collided object. FIG. 4 is a diagram illustrating a state in which physical interaction such as collision or contact is applied between objects. FIG. 5 is a diagram showing a state in which impulse responses of individual collisions are synthesized on the time axis. FIG. 6 is a flowchart showing a processing procedure for synthesizing a sound effect generated by collision or contact of an object.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 ... Three-dimensional virtual space presentation apparatus 101 ... CPU
102 ... RAM
103 ... ROM
DESCRIPTION OF SYMBOLS 104 ... Speaker 105 ... Display 106 ... Haptic device 21 ... Forward kinematics calculation part 22 ... Collision detection part 23 ... Collision force calculation part 24 ... Envelope waveform calculation part 25 ... Natural vibration waveform setting part 26 ... Generated sound list management part 27 ... Sound synthesis unit 28 ... Sound output unit

Claims (12)

A sound effect generator for generating sound effects associated with physical interaction between objects in a virtual space where a plurality of virtual objects coexist,
Interaction detecting means for detecting an object in which physical interaction occurs in the virtual space;
A dynamic quantity calculating means for calculating a dynamic quantity acting on the object by the interaction using a dynamic model that determines the behavior between the objects in which the physical interaction works based on physics;
Using the calculated mechanical quantity, a sound effect generating means for generating a sound effect generated in the object in accordance with the interaction;
Comprising
The mechanical quantity calculating means includes
Forward dynamics calculation means for executing forward dynamics calculation for obtaining acceleration generated in the object based on external force by physical interaction;
An operation space physical quantity calculation means for obtaining an operation space inertia inverse matrix and an operation space bias acceleration by executing a forward dynamics calculation by the forward dynamics calculation means using the dynamic model,
Linear complementarity problem consisting of relations between the acceleration and force of the operation space described using the inverse operation space inertia matrix and the operation space bias acceleration, and the constraints imposed when the physical interaction between objects works Is calculated using the operation space inertia inverse matrix and the operation space bias acceleration obtained by the operation space physical quantity calculation means, and calculates the mechanical quantity acting on the object based on the obtained force.
A sound effect generator characterized by the above. A sound effect generator for generating sound effects associated with physical interaction between objects in a virtual space where a plurality of virtual objects coexist,
Interaction detecting means for detecting an object in which physical interaction occurs in the virtual space;
A dynamic quantity calculating means for calculating a dynamic quantity acting on the object by the interaction using a dynamic model that determines the behavior between the objects in which the physical interaction works based on physics;
Using the calculated mechanical quantity, a sound effect generating means for generating a sound effect generated in the object in accordance with the interaction;
Three-dimensional object data holding means for holding data relating to vibration attributes including the attenuation characteristics of sound effects and natural vibration waveforms possessed by each object;
Comprising
The sound effect generating means assigns an attenuation characteristic to the envelope waveform that envelops the acoustic vibration waveform based on the attenuation characteristic of each object that has undergone physical interaction, and the natural vibration waveform that each object has to the envelope waveform. Calculate the sound effect by synthesizing,
A sound effect generator characterized by the above. A sound generation list management unit for registering information of each sound effect generated for each physical interaction as a past sound,
Summing the past sounds registered in the generated sound list management unit on the time axis to calculate the sound in the virtual space at the current time,
Sound effect generating apparatus according to claim 1 or 2, characterized in that. The generated sound list management unit performs a process of excluding sufficiently attenuated past sounds from the generated sound list.
The sound effect generator according to claim 3 . Treat physical objects as collisions or touches between objects in virtual space,
The mechanical quantity calculation means calculates at least one of a collision force, velocity, acceleration, impulse, or energy as a mechanical quantity acting on the object by physical interaction.
Sound effect generating apparatus according to claim 1 or 2, characterized in that. A sound effect generation method for generating a sound effect associated with a physical interaction between objects in a virtual space where a plurality of virtual objects coexist,
An interaction detection step of detecting an object in which physical interaction occurs in the virtual space;
A dynamic quantity calculation step for calculating a dynamic quantity acting on the object by the interaction using a dynamic model that determines the behavior between the objects in which the physical interaction works based on physics;
A sound effect generation step for generating a sound effect generated in the object in accordance with the interaction using the calculated mechanical quantity;
Have
The mechanical quantity calculating step includes:
A forward dynamics computation step for performing forward dynamics computation to obtain acceleration generated in the object based on external force due to physical interaction;
An operation space physical quantity calculation step for obtaining an operation space inertia inverse matrix and an operation space bias acceleration by executing a forward dynamics calculation by the forward dynamics calculation step using the dynamic model;
Linear complementarity problem consisting of relations between the acceleration and force of the operation space described using the inverse operation space inertia matrix and the operation space bias acceleration, and the constraints imposed when the physical interaction between objects works Is calculated using the operation space inertia inverse matrix and the operation space bias acceleration obtained by the operation space physical quantity calculation step, and calculates the mechanical quantity acting on the object based on the obtained force.
A sound effect generation method characterized by the above. A sound effect generation method for generating a sound effect associated with a physical interaction between objects in a virtual space where a plurality of virtual objects coexist,
An interaction detection step of detecting an object in which physical interaction occurs in the virtual space;
A dynamic quantity calculation step for calculating a dynamic quantity acting on the object by the interaction using a dynamic model that determines the behavior between the objects in which the physical interaction works based on physics;
A sound effect generation step for generating a sound effect generated in the object in accordance with the interaction using the calculated mechanical quantity;
A three-dimensional object data holding step for holding data relating to vibration attributes including sound effect attenuation characteristics and natural vibration waveforms of each object;
Have
In the sound effect generation step, an attenuation characteristic is imparted to an envelope waveform that envelops an acoustic vibration waveform based on the attenuation characteristic of each object that has undergone physical interaction, and the natural vibration waveform of each object is included in the envelope waveform. Calculate the sound effect by synthesizing,
A sound effect generation method characterized by the above. A generated sound list management step of registering information of each sound effect generated for each physical interaction as a past sound,
Sum the past sounds registered as the generated sound list on the time axis to calculate the sound in the virtual space at the current time,
The sound effect generation method according to claim 6, wherein the sound effect is generated. A process of excluding sufficiently attenuated past sounds from the generated sound list,
The sound effect generation method according to claim 8. Treat physical objects as collisions or touches between objects in virtual space,
In the mechanical quantity calculating step, as a mechanical quantity acting on the object by physical interaction, at least one of collision force, velocity, acceleration, impulse, or energy is calculated.
The sound effect generation method according to claim 6, wherein the sound effect is generated. A computer program written in a computer-readable format so as to execute a process for generating a sound effect associated with a physical interaction between objects in a virtual space in which a plurality of virtual objects coexist, The computer,
Interaction detecting means for detecting an object in which physical interaction occurs in the virtual space;
A mechanical quantity calculating means for calculating a mechanical quantity acting on an object by interaction using a dynamic model that determines a behavior between objects in which physical interaction works based on physics;
A sound effect generating means for generating a sound effect generated in the object in accordance with the interaction, using the calculated mechanical quantity;
Function as
The mechanical quantity calculating means includes
Forward dynamics calculation means for executing forward dynamics calculation for obtaining acceleration generated in the object based on external force by physical interaction;
An operation space physical quantity calculation means for obtaining an operation space inertia inverse matrix and an operation space bias acceleration by executing a forward dynamics calculation by the forward dynamics calculation means using the dynamic model,
Linear complementarity problem consisting of relations between the acceleration and force of the operation space described using the inverse operation space inertia matrix and the operation space bias acceleration, and the constraints imposed when the physical interaction between objects works Is calculated using the operation space inertia inverse matrix and the operation space bias acceleration obtained by the operation space physical quantity calculation means, and calculates the mechanical quantity acting on the object based on the obtained force.
A computer program characterized by the above. A computer program written in a computer-readable format so as to execute a process for generating a sound effect associated with a physical interaction between objects in a virtual space in which a plurality of virtual objects coexist, The computer,
Interaction detecting means for detecting an object in which physical interaction occurs in the virtual space;
A mechanical quantity calculating means for calculating a mechanical quantity acting on an object by interaction using a dynamic model that determines a behavior between objects in which physical interaction works based on physics;
A sound effect generating means for generating a sound effect generated in the object in accordance with the interaction, using the calculated mechanical quantity;
Three-dimensional object data holding means for holding data relating to vibration attributes including the attenuation characteristics of sound effects and natural vibration waveforms possessed by each object;
Function as
The sound effect generating means assigns an attenuation characteristic to the envelope waveform that envelops the acoustic vibration waveform based on the attenuation characteristic of each object that has undergone physical interaction, and the natural vibration waveform that each object has to the envelope waveform. Calculate the sound effect by synthesizing,
A computer program characterized by the above.

Images