EP1319207A2 - Freely specifiable real-time control - Google Patents

Abstract

Freely specifiable real-time control of animated graphics, video and/or audio sequences with the aid of controllable virtual objects. In order to control the movement of a virtual articular object in real-time, a model of a moving parameter (106), in addition to a surface model (108) defined for said object control signals (116) are inputted, using a 3D input device, in real-time. Additionally, the skeletal modified data is calculated on the basis of the parameterised control signals (116) and at least one parameterised moving model (106). The surface is calculated on the basis of the skeletal modified data and at least one surface model (108). Cyclic repetition of said steps enables a real-time movement control to be obtained.

Description

Freely specifiable real-time control

BACKGROUND AND FIELD OF APPLICATION OF THE INVENTION

The invention relates to a method for graphic visualization and real-time control of virtual and / or real objects for generating and / or influencing image or sound sequences, with the aid of which objects or real objects, such as robots or the like, are shown on a screen can be manipulated or controlled in a comfortable and reliable manner (quasi) in real time by interactive control commands of a user in their properties and / or actions. In particular, the present invention relates to a method for animation and motion control of a three-dimensional (real or virtual) joint object in real time, to a method for freely specifiable control of animated graphics, video or audio data with the aid of optical or acoustic parameters, on a computer software product for performing such a method and on a system for real-time motion control of virtual and / or real objects.

It should be noted that “objects” in the sense of the invention can be virtual objects, but also real objects such as robots or other remote-controlled objects with several degrees of freedom.

In the context of interactive communication between man and machine, problems often arise which are based on the insufficient adaptation of the machine to the properties of the recording, processing and output of information by the human user. On the one hand, this mismatch can result in information that the user can no longer handle. flood, especially when several tasks have to be done. On the other hand, if the user is under-challenged, for example in a highly automated system in which humans only have a control function, the monotony of the work situation leads to a drop in performance and incidents due to a lack of practice in such situations can no longer be controlled. The lack of consideration of the knowledge and level of training of the user of a machine should also be mentioned here. In many cases, human behavior, for example in the selection, evaluation and linking of information, in decision-making, in problem-free as well as in the planning and execution of actions, is still insufficiently taken into account and supported if it is is the design of technical systems.

The systems currently used to display and control objects in virtual environments increasingly take into account the ability of humans to record and process information, but they have one major disadvantage: when entering control commands for directly influencing the displayed scene, this is Users still rely on conventional methods for manually entering information, such as using a mouse, trackball, joystick, graphics tablet with stylus or touchscreen. The input mechanisms required for this must first be learned by the user in order to be able to be carried out at an appropriate speed of reaction. On the other hand, the innate or already existing learned human skills for communication by means of acoustic signals (e.g. speech) or optical signals (e.g. facial expressions, gestures, gestures and movements) are only insufficiently taken into account when entering information for controlling objects. For the adaptation of technical systems to humans, prior knowledge about their properties, their behavior patterns, their skills and their level of knowledge is necessary. When exchanging information between a user and an information system, the sensory, cognitive and motor properties of humans are of particular interest.

On the side of the sensory properties of humans predefined by the sensory channels, conventional machines and devices for outputting information essentially address the following perception channels:

- the visual channel (eyes) through optical signals,

- The auditory channel (ears) through acoustic signals and

- The tactile channel (sense of touch) through haptic signals.

Following the processing of the signals in the brain (cognition), the following channels are essentially available on the side of the motor properties of humans specified by the output channels:

- arm, hand and finger or leg and foot motor functions as well as body, head, eye or mouth movements, i.e. physical movements, gestures, gestures and facial expressions to generate mechanical or optical signals,

- Speech motor skills for the generation of acoustic signals.

Signals can be entered into an information system via these channels in order to trigger a desired action by the system. An ideal medium for communication between a user and an information system should be tailored to the sensory and perceptual as well as the motor skills as well as to the specific properties of the human user. The information should be structured in such a way that an optimal match between the representation of the information output and the mental model of the user is achieved: If the information to be displayed to the user is presented in such a way that, for example, his spatial perception is addressed users deal with amazingly complex amounts of information per unit of time. Likewise, the information system should be able to record, understand and process as many types of information sent by a user as possible and convert them into corresponding actions. This has the advantage that the user can react more efficiently and quickly to new events and situations. Ease of use and appropriateness of the task are therefore typical features of such an ideal communication medium. These characteristics can be expressed as follows:

- correspondence between type, scope and output speed and presentation of the information given with the sensory properties of the human user

Taking into account all information channels of the user when recording, recognizing and interpreting received control signals from the user,

- easy to learn and intuitive operation of the medium,

- high bandwidth of information transfer to the brain and high throughput of information, - dynamic adaptation of the application to the individual properties, skills, tasks, work and organizational techniques of the user,

- use of a natural interaction language with high semantic content,

- reliability, robustness and maintainability of the medium,

- social acceptance of the medium in the population,

- Consideration of health, ergonomic and safety-related aspects etc.

The aim of developing suitable interfaces between man and machine is to start from the properties of human communication channels and skills in order to provide devices, interaction techniques and interfaces that guarantee effective mutual communication via these channels. To achieve this goal, so-called "virtual realities" (VR) are particularly suitable. The term "virtual reality" (VR) is the computer-based generation of an intuitively perceptible or empfindbaren scene consisting of its ς _f rafischen presentation and interaction for the user. A virtual environment enables a user to access information that would otherwise not be available at the given place or time. It relies on natural aspects of human perception by using visual information in three spatial dimensions. This information can, for example, be changed in a targeted manner or enriched with further sensory stimuli. Control is essential the perspective in real time and the possibility of active

The user of the system influences the scene shown.

5 When navigating through virtual environments, the user can use the control mode that is natural for him. These can be, for example, corresponding arm or leg movements, movements for positioning the head or eyes, rotational movements of the body or bell directed at an object. By using the user's existing skills to control, the cognitive load during the interaction between man and machine can be reduced. This can increase the bandwidth of communication between man and machine and improve the operability of the machine. While the conventional forms of human-machine communication control the machine in a command-oriented manner, no specific commands have to be learned and used again when controlling objects in virtual environments: the computer "passively" observes the user and reacts in an appropriate manner based on the user's eye, head and / or hand movements under real-time conditions.

Manipulating the properties and influencing the actions of objects in a depicted scene requires a complicated interplay of sensors, cognitive processing and motor skills, which are influenced by many factors (individual behavior patterns and skills, experience, environmental influences, etc.). Interactions in a virtual world pose additional difficulties. In order to control, manipulate or influence objects, a reflex-like or cognitive sensory-motor feedback is particularly important, such as that of receptors in the skin, kinesthetic sensations, the sense of balance, as well as visual and / or acoustic sensations comes. In many cases this results in a necessary one

Redundancy that is not always the case with VR applications.

Due to the often insufficient sensory feedback

VR applications also make it difficult to learn motor skills.

In the case of commercial VR applications, a distinction is made between systems in which the user is fully integrated in the virtual environment ("immersion") and systems that only offer a "window" to virtual reality. In addition to the well-known forms of human-machine communication such as

- Direct manipulation of objects through manual fine motor operations (pointing, touching, gripping, moving, holding etc.),

- formal interaction languages (programming languages, command languages and formal query languages),

- natural language interaction,

- gestural interaction using non-verbal symbolic commands (facial expressions, gestures, gestures, movements) and

- hybrid task-oriented forms of interaction

one can also understand virtual realities as a new form of human-machine communication. As the name "Virtual Reality" already suggests, a certain degree of reality of the representation is necessary for this: The user should be provided with the sensory information that is required to process a task or to achieve a goal.

The visual perception not only provides information about the position, movement, shape, structure, contour, texture, color or pattern of objects etc., but also information about the relative body position of the viewer and his Movements and the nature of the three-dimensional environment. Synthetically generated environments can be made more realistic by simulating as much of the information as possible in natural environments (movement parallax, vanishing points of perspective, spatial depth and plasticity, lighting and shadows, concealment, gloss, mirroring and diffuse reflection, etc.) , How much and which information should be presented depends on the task at hand. The differences between the real and virtual world determine how realistic the simulation is perceived.

The visual information must be simulated by a computer in order to implement virtual realities. Similar aspects are relevant as in painting. The computer-aided simulation of three-dimensional worlds usually simulates the projection of individual light beams. ^' The starting point of such a simulation is the specification of the environment to be simulated. To do this, the individual objects with their properties and their location must be defined. The intensities of individual pixels are then calculated for visualization and projected onto the output medium.

With the help of these simulations, completely new ways of learning and practicing can be realized (examples: vehicle or aircraft simulator), on the other hand, it is always abstracted from certain aspects of the real world. VR applications therefore simultaneously enrich and limit the user's experience.

VR systems basically consist of sensors and actuators and their coupling. Important hardware components include the following: - "Displays" for the presentation of the virtual environment: As part of the visual presentation, today mainly monitors, "Head Mounted Displays" (HMD), "Binocular Omni-Oriented Monitors" (BOOM) and projection systems; However, auditory or tactile displays are also used, which react to acoustic or manual user input.

- Positioning and orientation systems for recording the location and perspective of the user: Here, a distinction is made between determining the absolute position ("Position Tracking") and measuring the deflection of the joints ("Angle Measurement"). Electromagnetic, kinematic, acoustic, optical and image processing procedures are used. .

- Interaction and manipulation systems for acting and reacting to the user in the virtual environment: Pointing devices (2D or 3D mice, trackballs, joysticks etc.) or tactile devices (touchscreens, electromagnetic graphics tablets with stylus, etc .) used; So-called "data gloves" with diffraction and pressure sensors are also increasingly being used. Voice control should also be mentioned in this context.

- Calculation systems and software for creating the virtual environment under real-time requirements.

- Networks for the integration of different users, through which new forms of cooperation can develop.

The various technical variants of home or head-based systems for the visualization of virtual realities are summarized in English as 'Visually Coupled Systems' (VCS). They consist of the following important components:

1. a display attached to the head or helmet,

2. a device for determining the user's head and / or eye movements,

3. a source of visual information that depends on the user's head and / or line of sight.

When using such _^ system for VR applications both information can be simultaneously performed from the real and from the virtual environment. One speaks of "See-Through Displays" for the representation of enriched realities.

Tracking head movements is an important part of VR applications. Usually the position and orientation of the head in the room are determined, advanced systems can also follow the direction of the gaze. Most systems use either ultrasound, magnetic or light energy to communicate between the head-mounted transmitters and the receivers. Important technical data that play a role in the selection of these systems are:

the number of degrees of freedom for the directions of movement which can be registered and tracked,

- the detectable angular range,

- the static accuracy (vibration sensitivity),

- the resolving power, - the reliability,

- the data throughput and the screen sampling frequency,

- the interface to the computer as well

- further performance aspects.

VR applications can be successfully used in practice in a number of different areas. Some possible applications are outlined below as examples.

- Use in the training area: By learning to deal with (virtual) objects, interactive demonstrations, visualization of abstract concepts, virtual training in behavior in dangerous situations, virtual research into distant places or epochs, knowledge can be imparted, creative skills can be trained and behavioral patterns can be trained.

- Use in driving and flight training in appropriate simulators: By using simulators, behavior can be trained, especially in emergency situations.

- Use in the field of computer games: The possibility of navigating through a virtual scene and the possibility of specifically controlling and influencing virtual objects creates a realistic impression, which can significantly increase the attractiveness of a computer game for the user.

The technologies available today for entering information into a data processing system can be divided into four groups according to the sensors used: 1. mechanical input systems (eg keyboards, mice, trackballs and joysticks), 2. electrical input systems (eg tactile displays and graphics tablets), 3. optical input systems (eg light pen) and

4. Acoustic input systems (e.g. voice input and interpretation systems).

In the following, the aids for inputting information which are used according to the current state of the art and which are used to control objects in the area of VR applications will be briefly discussed.

Conventional input systems such as keyboards, mice, trackballs and joysticks are widely used today. They are used to control cursors, mouse pointers, etc., for example to be able to navigate through a virtual scene or to move virtual objects on the screen. The disadvantage of these input systems is that they require a storage area (i.e. a fixed location) in order to be operated in an efficient manner.

With a touchscreen, on the other hand, it is possible to point directly at objects depicted on the screen without the need for additional space-consuming additional devices on the desk. Low-resolution touchscreens have 10 to 50 positions in the horizontal and vertical directions and use a horizontal and vertical row of infrared LEDs and photo sensors to build up a grid of invisible light rays right in front of the screen. When the screen is touched, both vertical and horizontal light beams are interrupted. The current finger position can be determined from this information. Another known embodiment of touch-sensitive information input devices is the capacitively coupled touch panel. This provides a resolution of approximately 100 positions in each direction. If a user touches the conductive coated glass plate of the touchscreen with a finger, the current finger position can be determined based on the change in impedance. Other high-resolution panels use two minimally spaced, transparent layers. One is coated with a conductive material, the other with a resistance material. These two layers touch through the pressure of the finger, and the current finger position can then be determined by measuring the resulting voltage drop. A lower-resolution and cheaper variant of this technology uses a grid of fine wires instead of these layers.

According to the state of the art, various solutions for the problem of real-time motion control of virtual objects are available today, each of these solutions being optimized for a specific application. Each of these solutions therefore has certain limitations. In order to be able to begin to explain some of the most important of these solutions, it is necessary to briefly address their most important aspects.

One possibility for real-time motion control of virtual or real objects has recently emerged from the fact that input devices for computers have become known which enable the simultaneous input of control signals of several degrees of freedom that are independent of one another. The possibilities created by this far exceed those that exist, for example, when using a mouse, which can only be controlled two-dimensionally (e.g. on the shelf of a desk). It is also known to use a mouse, for example, with additional switches to provide, however, these switches have the disadvantage that they do not allow the input of analog data, but rather are limited to binary data (on / off).

Various input devices are also known from the prior art which can generate analog control signals of different degrees of freedom that are independent of one another, each of these analog signals thus being able to be used as a parameter value of a control. For example, US-A-5, 757, 360 discloses an egg-shaped input device for computers which can be moved freely in space by a hand of the user, determines its current positions, directions of movement, speeds and accelerations, and these kinematic data are transmitted wirelessly transmits to the computer. An analogous sequence of movements is shown in

Form of a movement pattern is identified, from which movement commands are derived and converted into an animated graphic representation. The movement patterns are automatically recognized with the help of a pattern recognition algorithm. Control commands are also generated. The disadvantage of this method is that it cannot be freely specified, since the user's movement sequences, which are recorded by the input device in an analog manner, correspond to corresponding movement sequences of stored movement sequences of an animated graphic representation and can only be represented as such.

Input devices which have manually operated force-torque sensors are known, for example, from the patent documents DE 36 11 336 C2, DE 37 64 287 and EP 0 979 990 A2. From the last-mentioned patent specification, it is also known to use such a force-torque sensor for controlling a real or virtual mixing or control desk, for example in order to create and design novel color, light and / or sound compositions. Here- in turn, the intuitive spatial control in three translational and three rotational degrees of freedom can be advantageously transferred to a continuous spatial mixing or control of a large number of optical and / or acoustic parameters.

Manually controllable input systems that allow navigation in three dimensions are successfully used in a number of different technical application fields. Such an application field is represented, for example, by control devices for controlling the functions of electronic musical instruments (especially in the case of synthesizers and master keyboards), which have a so-called MIDI interface (“Musical Instrument Digital Interface”).

Such a three-dimensional control device for electronic musical instruments is provided, for example, by the D-beam controller integrated in the keyboards EM-30 and EM-50 from Roland. By using an extremely sensitive infrared light beam, the hand and / or body movements of a user can be detected without contact above a surface coated with infrared sensor elements. Thanks to the D-Beam technology, these acoustic movements of the user's hand and / or body can be used to modify or control individual acoustic parameters of recorded improvisations or compositions in real time. The D-Beam controller converts the user's movements into MIDI signals and analog control signals.

Since one of the preferred exemplary embodiments of the present invention is also based on a control device for controlling stored parameterized audio data with the aid of parameterized control signals which can be controlled using controllable virtual objects via a MIDI interface. be transferred to at least one electronic musical instrument, the aspects of the MIDI standard that are important for understanding the invention are briefly described below.

The MIDI interface is a digital data transmission format between electronic musical instruments, computers and peripheral devices. The MIDI standard is widely used today and has been used by many musicians and composers since its introduction in 1983.

MIDI opens up a very efficient method of displaying audio data, and this makes MIDI a very attractive data transfer protocol not only for composers or artists, but also for a variety of computer applications that are able to generate sound patterns , such as multimedia applications or computer games. Thanks to the publications of the General MIDI System Specification, the most common PC / MIDI interfaces are widely recognized by users today. MIDI is also supported by the Microsoft Windows operating system and other operating systems. Due to the development and marketing of inexpensive synthesizers, the MIDI standard is becoming increasingly popular as the number of applications increases.

MIDI was originally developed to be able to couple two or more keyboards from different manufacturers. At the time, no one foresaw that complete music productions could be created using sequencer systems using the MIDI data format. Today, MIDI is mainly used as a transmission medium to replace or supplement digitized audio data in computer games or multimedia applications. MIDI was standardized by the MIDI Manufacturers Association (MMA), which includes all manufacturers of digital musical instruments worldwide. This committee defines the norm binding for all members, among other things also the command structure of the MIDI protocol defined in the MIDI standard. Without this standard, incompatibilities among devices from different manufacturers would result.

In contrast to the transmission of analog audio data, when transmitting sound patterns via MIDI interface from one or more keyboards to a computer (or in the opposite direction), only bit sequences (so-called "MIDI events") are transmitted, which contain the significant acoustic parameters of the pieces of music played or to be played on the individual keyboards in electronically readable

Include form. These programming instructions include MIDI-Se- quences, for example, instruct the synthesizer, which tracks are to be recorded, which solo or loading ς _f leitinstrumente to be used for an arrangement and which musical parameters are transmitted. Acoustic parameters are understood to mean, for example, pitches, note or pause values, volumes, tempos, articulation instructions, timbres, pedal, vibrato, chorus, reverb, overtone and / or other special effects. In the following, these acoustic parameters will be referred to as "MIDI game information". The playback of the audio data on a keyboard is therefore not an analog recording of pieces of music previously recorded on a keyboard, but the exact reproduction of the recording process itself. The polyphonic voices of a playback synthesizer are at least partially occupied.

Compared to the use of sampled audio data stored on a floppy disk or CD-ROM, the generation of sound patterns (sounds) using MIDI Synthesizers have many ^~ advantages. One of these advantages relates to the storage space required to store the parameterized audio data. Files in which digitally sampled audio data are normally saved in a PCM format (such as ".WAV 'files) are usually quite large. This is particularly the case for long pieces of music recorded in stereo quality at a high sampling rate In contrast, MIDI files are extremely small, for example files in which high-quality sampled audio data are stored in stereo quality contain approximately 10 Mbytes of music played per minute, while a typical MIDI sequence has a size of less than 10 kBytes per Minute of music is played back, because MIDI files - as already mentioned - do not contain the sampled audio data, but only the programming commands required by a synthesizer to generate the desired sound.

Since the MIDI game information does not convey any immediate information about the type of audio data represented, individual acoustic parameters of this MIDI game information can subsequently be exchanged as desired. This also has great advantages:

- A composer can subsequently instrument or rearrange his work in a variable manner.

- The errors that may have occurred during the import

(e.g. "wrong tones") can be corrected afterwards.

- Several synthesizers can, for example, play the same voice (unison) to achieve more sonority and much more The additional possibility of being able to edit pieces of music recorded on one or more synthesizers or other electronic musical instruments on the screen, i.e. to be able to change, add, delete, move or transpose individual notes or pauses, groups of notes or entire staves, considerably simplifies the work of a composer ,

A complete MIDI word usually consists of three bytes. The so-called status byte is sent first. It is a notification of what type of message it is. The status byte is followed by two data bytes, which contain information about the respective content of the message. The following example refers to the MIDI representation for "switching on" a tone of medium pitch (c ¹ ), which should sound at a medium volume (Italian: "Mezzoforte", mf):

The first bit ("Most Significant Bit", MSB) in the binary representation of the status byte is always assigned the value "1", with data bytes the MSB always has the value "0". In this way, status and data bytes can be clearly differentiated. For the decimal values Z _s for a status byte, the following applies as a result of the "1" in the MSB: Z _s e [128ι ₀ ; 255 ₁₀ ]. Since the MSB is set to "0" for the data bytes, which means that it can no longer be used as a value indicator, the data Bytes only seven bits each, so that consequently for the

Decimal values Z _Di and Z _{D2 of} the two data bytes apply: Z _D1 e [0 ₁₀ ; 127 ₁₀ ] and Z _D2 e [0 _lo ; 127 ₁₀ ]. These 128 different "Note On" or "Note Off combinations are completely sufficient, since 128 different pitches - arranged in an equal-tempered, chromatic tone scale (ie at intervals of semitone intervals) - the range (ambitus) of a modern one Concert grand with 88 keys far exceeds.

In order to be able to address individual devices within a MIDI system in a targeted manner, there are a total of 16 MIDI channels. The MIDI channel of the transmitter must be identical to that of the respective receiver. The basic principle here is that a MIDI data line transports all game information on all 16 MIDI channels, whereby the connected tone generators select the messages that are intended for them. With the help of the last four bits of the status byte, the address A _{k of} a selected MIDI channel k (where ke {0 ₁₀ , ..., 15ι ₀ } or A _k e {0000 ₂ , ..., 1111 ₂ }) transferred. This means that only the first four bits of the status byte contain the status information of a MIDI sequence (eg "Note On", "Note Off" etc.).

For the example of the "Note On" command described above, this means specifically that the first channel (k = 0) with the address A = 0000 _{2 was} selected. If a byte of such a MIDI word is received by a synthesizer, the MSB is first used to check whether there is a status byte or a data byte. The receiver must also check all status bytes with regard to their channel training, which they receive, instructed by the respective MIDI channel setting, etc. Discovered ^'-To a MIDI event with its intended address, the recipient decrypts the status byte following data bytes and generates the appropriate tones. This principle can be compared to the reading of a choir singer or instrumentalist who, when singing or playing a piece of music, only reads the voice intended for him from a polyphonic set of sounds, an arrangement or a score.

The functions of two of the most common control devices found in synthesizers and master keyboards are briefly described below: the "pitch bend wheel" for the continuous detuning of the pitches of the tones of the keys of a synthesizer and the "modulation wheel" for modulating the tone characteristics the pitches of tones of struck keys on a synthesizer.

The MIDI data format provides for the smallest possible interval between different pitches of equal-tempered halftone steps or "chromas" (ie enharmonially equivalent intervals such as excessive primes, small seconds or double-reduced thirds). In order to achieve a continuous frequency-Va ^¬ riation (engl .: "pitch bend"), so-called are pitch bend wheel data is required to be generated with a respective control instrument of the synthesizer (the "pitch bend wheel") can. It is usually a wheel or (more rarely) a joystick that can be moved in four directions. With synthesizers equipped with these pitch bend wheels, the pitches of the Keys that are depressed on the keyboard of the synthesizer are detuned by rotating the pitch bend wheel in the direction of higher or lower frequencies. Normally, detunings of up to an equal-tempered whole tone can be generated in the direction of higher or lower frequencies. The pitch bend wheel is usually equipped with a return mechanism that springs back to the middle or normal position when the wheel is released. This position corresponds to the pitches of the depressed keys on the basis of the floating mood.

With the help of a modulation function, which can usually be carried out with another control instrument of the synthesizer (the "modulation wheel"), the tones of the keys struck on the keyboard of the synthesizer can optionally be provided with a vibrato effect. Alternatively, this control instrument can also be used to achieve other effects, for example to mofify the sound brightness or the resonance of the played tones by changing their overtone spectrum. If the modulation wheel is turned to a first point of impact, the depth of the effect is minimal; if it is turned up to an opposite second anchor point, the effect depth is maximum.

In order to be able to explain the control of individual MIDI functions with the aid of parameterized control signals, the most important of the necessary MIDI controllers will be briefly discussed below.

With the help of the first data byte of a MIDI word maximum of 128 different controller addresses and up to 128 different play or other MIDI functions can be spoken to ^¬. The second data byte is responsible for the control area. For example, with the help of Frequency modulations responsible controller No. 1 from

A vibrato or tremolo effect can be added to synthesizers. The following table lists the most common controllers with their numbers (addresses) and designations:

Addresses 12 to 31 are not assigned and offer the user the possibility of freely assigning MIDI functions. Depending on the nature of the respective synthesizer, these addresses can sometimes be assigned very unusual physical parameters, such as the oscillator frequency or pulse width of the generated vibrations.

The controllers 32 to 38 serve to more finely resolve the range of values of the controller addresses 1 to 6. The same is done by controller numbers 39 to 63 for addresses 7 to 31.

The game aids described so far are characterized by a common property: they can be (in 128 or more Step by step). The generic term for this

Controller is "Continuous Controller". In contrast, there are other controllers that perform switch functions and are therefore generally called "switch controllers". The following table gives an overview of the most important of these controllers:

A closer look at the second data byte of controller 64 reveals only two values, namely

00000000, (= 0 ₁₀ ) for "Pedal Off" and (= 127ιo) for "Pedal On".

In principle, however, the MIDI data format allows a more differentiated interpretation of the value range 1 ₁₀ to 126ι ₀ . For example, a multi-stage sustain effect can also be provided. For synthesizers that allow this multi-stage sustain effect, the decay phase of sound events with a "half" pedal depressed is reduced accordingly compared to the decay phase of sound events of the same pitch and volume with the pedal depressed to the stop.

Normally, the controller addresses, for example for the modulation wheel, the holding pedal or sills that can be operated with the foot to influence the dynamics, are already defined. If one of these game aids is used, the sent controller address is fixed. However, modern synthesizers and master keyboards also have freely definable controllers, ie any controller number can be assigned to the intended play aids (pedals, sills, wheels and sliders). Some tone generators in turn allow the controller functions to be redefined in the device itself. The controller number received can be freely assigned to an internal function.

A large number of different technologies and processes are used today to produce sound with the aid of electronic musical instruments, computers or sound cards. Two widely used techniques of sound generation using electronic musical instruments are frequency modulation (FM synthesis) and the use of wavetables (WAV synthesis).

FM synthesis is a process used by sound cards and computers to mimic the tones of acoustic musical instruments by electronic means. Sounds generated with the help of FM synthesis are easily recognizable as such - in contrast to sound patterns that were generated by WAV synthesis with the help of wavetables. Master keyboards and synthesizers that have wavetables are relatively expensive, but are often preferred by many professional and amateur musicians because of their high quality playback. The sound patterns are generated by wavetables that combine and / or reproduce stored, digitally sampled original sounds ("Samples") of acoustic musical instruments. Compared to FM synthesis, in which electronic sounds are generated with the help of the computer, wavetable sounds appear much more realistic. With the help of sound cards, the possibilities of conventional computers for generating audio signals and sound patterns can be expanded. Sound cards are essential for any application that uses sound effects. In order to be able to translate analog audio data into digital computer language, sound cards have appropriate devices for digitizing analog sounds. The sound generation by a sound card is based on either an FM synthesis or a WAV synthesis.

OBJECT OF THE PRESENT INVENTION

Based on the above-mentioned prior art, is the present invention dedicated to _? the task of providing comfortable and reliably working methods by means of which the user is able to actively control virtual and / or real objects, the existing skills of the user being used to transmit information. In particular, the movement control of objects in real time or the real time control of electronically generated audio signals is to be simplified.

This object is achieved according to the invention by the features of the independent claims. Advantageous exemplary embodiments which further develop the idea of the invention are defined in the dependent patent claims.

SUMMARY OF THE PRESENT INVENTION

To achieve the object defined in the preceding section, the invention proposes an efficient method for graphic visualization and real-time control of virtual or real objects for generating and / or influencing image or sound sequences Objects or real objects displayed on a screen can be manipulated, controlled or influenced in real time in a comfortable and reliable manner by interactive control commands of a user in their properties and / or actions. In particular, the present invention includes a method for animation and motion control of a joint object in (quasi) real time, a method for freely specifiable control of optical or acoustic parameters, a computer software product for executing such a method and a system for real-time Motion control of virtual or real objects.

The present invention relates in particular to input devices which can generate control signals for various degrees of freedom that are independent of one another, and this in each case by means of analog signals. Each of these analog signals can thus be used as a parameter value when controlling virtual objects.

In contrast to conventional manually operated, mechanical or touch-sensitive input mechanisms via keyboard, mouse, trackball, joystick, graphics tablet with stylus, tactile displays, the technologies used to input the control commands issued by the user are devices for recording , Recognition, interpretation and processing of three-dimensional movements of a user. The user is therefore no longer dependent on the presence of additional hardware devices for the manual input of control commands. The inputted information can be evaluated instead or additionally using methods of signal or pattern recognition. Furthermore, the type of input method can be tailored to the individual skills of the user. One aspect of the present invention is that reference patterns for animation and motion control of objects, for designing the shape, color and structure of the surfaces of virtual objects and for generating and / or influencing image or sound sequences are stored in advance. Depending on the control signals of the user, these reference patterns can then be called up in real time as parameter values for the generation and / or influencing of animation effects, for the surface design of the objects, for the control or targeted manipulation of video or audio data. The respective reference patterns can be stored in a standardized manner, so that, based on such a real-time control, the above-mentioned parameter values are changed by a module-like exchange of the respective reference patterns at a corresponding interface of the

Real-time control can be done with relatively little effort.

1.Procedure for the animation and movement control of a joint object in real time by activation with the aid of parameterized movement patterns

According to the underlying invention, it is possible to dispense with completely reprogramming a real-time movement sequence; rather, known movement patterns can be made available via a standardized interface in a parameterized form of real-time movement control.

A preferred exemplary embodiment of the underlying invention is based on a method for real-time motion control of a joint object. First, at least one parameterized movement pattern is defined for at least one surface pattern for the object. If control signals are then fed into the real time motion control entered, this calculates skeletal

Change data (i.e. the position changes of the contours of a virtual object or a framework, consisting of the hinge points of a virtual object connected by edges) as parameter values based on the control signals and at least one parameterized movement pattern. The skeleton change data reflect translatio and / or rotatochange changes in the position or orientation of the joints of the object. Finally, the surface of the virtual joint object is calculated on the basis of the skeleton change data and at least one surface pattern. By repeating the last-mentioned steps in close succession, one obtains (depending on the control signals of the user) a movement control of the object which can no longer be distinguished by a viewer from a movement control in real time.

By exchanging or redefining a parameterized movement pattern and / or surface pattern, movement control of a novel joint object can be achieved with very little effort.

According to the exemplary embodiment of the underlying invention, control signals for at least six independent degrees of freedom can be input simultaneously and processed in real time. These are, for example, control signals for three translatio and three rotatio degrees of freedom in a virtual three-dimensional vector space.

Based on the determined instantaneous position vectors of the input device, the first and second time derivatives of the control signals (ie the speed, acceleration, angular velocity or angular acceleration vectors) can be determined according to the invention and processed. The controller thus determines the change in the control signals over time, so that, in contrast to the already mentioned US Pat. No. 5,757,360, for example, no separate acceleration sensor is required. The acceleration can rather be determined by evaluating the development of the control signals with the help of the motion control itself. The assignment of the control signals or their time derivatives as parameter values for the stored movement patterns can be freely specifiable.

2.Procedure for real-time control of acoustic parameters of recorded audio data with the aid of parameterized sound patterns ..- ^•

In addition or alternatively, in a further exemplary embodiment of the underlying invention, audio data can also be specifically influenced in real time with the aid of parameterized control signals from the user. Digitally stored, parameterized sound patterns, such as the acoustic parameters of (according to the so-called MIDI standard) digitally recorded pieces of music, which can be controlled via the parameterized control signals of the user, can be specifically modified. These sound patterns can then be called up as parameter values in real time, depending on the respective control signals of the user.

3.Procedure for the freely specifiable control of animated graphics, video and / or audio data

According to a further exemplary embodiment of the present invention, a method for freely specifiable control of animated graphics, video and / or audio data using optical or acoustic parameters is provided. For the Generation and / or influencing of these image or sound sequences is at least one predefined parameterized movement, surface, sound and / or lighting pattern stored as a reference pattern. For this purpose, control signals of different degrees of freedom are evaluated by an input device and used as parameter values for generating or influencing animated graphics, video and / or audio data with the aid of at least one stored reference pattern. The assignment of the control signals of a degree of freedom to a specific reference pattern can be freely specified by the user.

In addition, temporal derivations of the control signals can optionally be assigned to a specific reference pattern. The coupling parameters of the assignment, such as the damping and the amplification, can also be freely adjustable by the user.

4. Computer software product for freely specifiable control of animated graphics, video and / or audio data

According to a further aspect of the present invention, a computer software product is provided, with the aid of which a method according to one of the exemplary embodiments described above can be carried out.

5. System for real-time motion control of virtual and / or real objects

According to the invention, a system may be provided for real-time motion control objects that has a memory in which are stored in the at least one parameterized Bewegungsmu ^¬ edge and at least a surface pattern of the object. This system can also have an input device for inputting control signals for real-time motion control. tion. Furthermore, a motion generator can be provided in order to calculate skeleton change data on the basis of the control signals as parameter values and at least one parameterized motion pattern. The skeleton change data reflect translational and / or rotational changes in the position or orientation of the joints of the object. In addition, a surface generator can be provided which calculates the surface of virtual joint objects on the basis of their skeleton change data and a selected surface pattern.

The input device can be designed for the simultaneous input of control signals for at least six independent degrees of freedom. It can be designed in particular for the input of control signals for three translational and three rotary degrees of freedom in a three-dimensional vector space. In addition, a unit for determining and processing the time derivatives of these control signals can also be provided.

In order to enable the user to control stored audio data in real time, a sound generator can also be provided for generating sound signals by linking digitally parameterized sound patterns with digital control signals that are present in parameterized form.

In addition, a unit for the optional assignment of the control signals or their derivatives can also be provided as parameter values for the parameterized movement, surface, sound and / or lighting patterns.

6. System for freely specifiable real-time control of animated graphics, video and / or audio sequences In a further exemplary embodiment of the present invention, a system for freely specifiable real-time control of animated graphics, video and / or audio sequences is provided. The system has a memory in which at least one predefined parameterized movement, surface, plan and / or lighting pattern is stored as a reference pattern in order to generate and / or influence animated graphics, video and / or audio sequences in real time. The system also has an arithmetic unit for evaluating control signals of different degrees of freedom from an input device as parameter values for generating and / or influencing animated graphics, video and / or audio sequences using at least one stored reference pattern. In addition, an adjustment unit is also provided in order to enable a freely selectable assignment of control signals of one degree of freedom to a specific reference pattern. This setting unit can optionally allow time derivatives of the control signals to be assigned to a specific reference pattern. Furthermore, it can achieve a freely selectable setting of coupling parameters of the assignment, such as, for example, the temporal damping and the amplification.

As an average person skilled in the art can readily recognize, the underlying invention is not only limited to the exemplary embodiments described above. In addition to the features disclosed therein, numerous modifications and variations are possible without substantially deviating from the scope of the present invention, as disclosed in the patent claims.

BRIEF DESCRIPTION OF THE DRAWINGS

Further properties, features, advantages and expediencies of the underlying invention result from the dependent dependent patent claims and from the following description of the preferred exemplary embodiments of the invention, which are illustrated in the following drawings. Show here:

->

FIG. 1 shows a simplified block diagram to illustrate the input and output signals of the freely specifiable real-time controller 102 for animated graphics, video and / or audio sequences according to the preferred embodiment. Example of the underlying invention,

FIG. 2 shows a detailed block diagram to illustrate the components and the input and output signals of the freely specifiable real-time controller 102 for animated graphics, video and / or audio sequences,

FIG. 3 shows a flowchart to illustrate the actions performed by the freely specifiable real-time controller 102 for animated graphics, video and / or audio sequences,

FIG. 4 shows a block diagram to illustrate the control of individual or several acoustic parameters of digitized sound patterns by a system, consisting of a computer, the control unit 401 of which is conductively connected to at least one electronic musical instrument 402 via a MIDI interface and a 3D input device 104 for the input of parameterized ones Control signals for real-time motion control of virtual objects that can be displayed on a screen

and

FIG. 5 shows a block diagram to illustrate the control of individual or several acoustic parameters of digitized sound patterns by a system, consisting of a computer which has an electronic tone generator 502 and a sound card 510, and a 3D input device 104 for input of parameterized control signals for the Real-time motion control of virtual objects that can be displayed on a screen.

DETAILED DESCRIPTION OF THE INVENTION

The functions of the assemblies contained in a first exemplary embodiment of the present invention, as shown in FIGS. 1 to 5, are described in more detail below.

With reference to FIG. 1, the present invention will first be explained schematically. The human-machine interface is represented by the input device 104. In particular, this is a so-called 3D input device, which can generate control signals 116 for six degrees of freedom, which are independent of one another, if it is manipulated accordingly by a user. These degrees of freedom include, for example, three translational degrees of freedom, which are referred to below as x, y and z, and three rotational degrees of freedom, which are referred to below as φ _κ , φ _y and φ _z . The variables x, y and z denote the orthogonal axes of a three-dimensional Cartesian coordinate system. This leaves describe themselves mathematically as a three-dimensional vector space V, which is represented by an orthonormal basis consisting of the orthonoreal unit vectors e _x , e _y and e ₂ . If these variables are summarized in vector notation, then at each discrete point in time n (after omitting the units) results:

the location vector x (n): = [x (n), y (n), z (n)] ^τ e <-? ° ³ and the direction of rotation vector φ (n): = [φ _x (n), φ (n ), φ _z (n)] ^τ e * -> _° ³ .

The transformed location vector x _τ (x, Δx), which results after the translation of a point object P with the location vector x by the path differences Δx, Δy and Δz in the direction of the respective axes x, y and z, respectively, in this vector space V are simply expressed by adding the location vector x and a linear combination of the orthonormal unit vectors e _x , e _y and e _z :

x _τ (x, Δx): = x + Δx with Δx: = Δx-e _x + Δy-e _y + Δz-e. V Δx, Δy, Δz e ^« 5>».

The variables φ _κ , φ _y and φ _z denote the direction of rotation of virtual objects around the x, y and z axes of this coordinate system. The transformed location vector x _R (x, Δφ), which results after the rotation of a point object P with the location vector x by the angles of rotation Δφ _x , Aφ _γ and Δφ _z , can be expressed in this vector space V as follows:

x. _R (x, Δφ _x , Aφ _yf Aφ _z ): = R _z (Δφ _z ) ^• R _y (Δφ _y ) ^• R _χ (Δφ _χ ) • x with Δφ: = [Δφ _* , Δφ _yι Aφ _z ] ^τ V Aφ _y ., Aφ _yr Δφ _z e ^ », where the rotation matrices R _x (Δφ _x ), R _y (Δφ _y ) and R _z (Δφ _z ) are defined as follows:

R _* (Δφ

cos (Δφ _y ) 0 Sliln (Δφ _y )

R _y (Δφ _y ) 1 0 and

-sin (Δφ _y ) 0 cos (Δφ _y ^y ) 'J

^' cos (Δφ _z ) -sin (Δφ ₂ ) 0

R _z (Δφ ₂ ) sin (Δφ _z ) cos (Δφ _z ) 0

0

Of course, gen using Tastensteuerun ^¬ or switches relatively simple yet more free-added degrees of freedom, for example. It should be noted that switches or keys generally generate binary control signals (on / off), whereas the three translatory degrees of freedom x, y or z or the three rotational degrees of freedom φ _x , φ _y or φ mentioned above _{z can} each result in analog control signals which are then available, for example with byte-wise coding in 2 ⁸ = 256 stages, as digital signals for further processing.

Since the translational three, and the three rotational free ^¬ degrees of freedom x, y, z, _φ κ, φ _y, and _z as "analog signals" φ or in 256 steps quantized digital signals are to be considered, the present invention may in accordance with the temporal change of these control signals 116 by the freely specifiable real-time control 102 for animated graphics, video and / or audio sequences, which is then described in more detail below. zen can be evaluated. In particular, it is therefore possible to use the three-dimensional vectors for at any discrete point in time n (after omitting the units)

the speed v (n): = [x (n), y (n), z (n)] ^τ e «s *» ³ , the acceleration a (n): = [x (n), y (n) , z (n)] ^τ e \ - *> ³ , the angular velocity ώ (n): = [φ _x (n), φ _y (n), φ _z (n)] ^τ e ^'

and the angular acceleration (n): = [φ _x (n), φ _y (n), φ _z (n)] ^τ e ss *> ³

the degrees of freedom x, y, z, φ _xr φ _y and φ _z, respectively, and, if necessary, to use them as additional degrees of freedom, regardless of the absolute value of the respective control signals 116. The input device 104 does not require any further sensors (for example speed or acceleration sensors).

Thus, while the control signals 116 are fed on the input side from the 3D input device 104 to the freely specifiable real-time controller 102 for animated graphics, video and / or audio sequences according to the present invention, the latter outputs animation output signals 120 depending on these control signals 116 With the help of which a 3D animation 114 can be generated, for example, on the screen of a computer. This 3D animation 114 may include:

controllable movement patterns 106 of at least one movable joint body, controllable surface patterns 108 of the joint body and / or its surroundings,

controllable sound patterns 110 and / or controllable lighting patterns 112 for the joint body and / or its surroundings.

It should be noted that the generation of this SD animation 114 is carried out in real time with respect to the control signals 116. In contrast to some known examples according to the prior art, it is therefore normally not necessary to save sound or image sequences in advance in order to then trigger them at the push of a button.

A major advantage of the present invention is that for a newly to be generated 3D animation 114 does not have to be done a completely new programming, as is the case in the prior art ^'.. Rather stored motion patterns 106, surface pattern 108, Sound patterns 110 and / or lighting patterns 112 are connected in an interface-like manner to the freely specifiable real-time control 102 for animated graphics, video and / or audio sequences. By exchanging or re-specifying at least one of the named patterns, a completely new 3D animation 114 can be generated in a simple manner. The programming effort for creating a new 3D animation 114 is thus significantly reduced.

Since the brain can process extremely complex amounts of information per unit of time in various animal species and in humans in the course of evolution with regard to the spatial perception and processing of movement patterns, in a preferred exemplary embodiment of the underlying invention, these learned and inherited skills are easily accessed In that known movement patterns are made available in a modular manner via a standardized interface of the freely specifiable real-time control 102 for animated graphics, video and / or audio sequences. It should be noted that the 3D input device 104 represents only one example of an input device for generating control signals 116. In principle, any other type of input device is also suitable, although the 3D input device 104 has the great advantage that the input of control signals 116 of different degrees of freedom can take place in a particularly intuitive manner. With regard to the specific embodiments of such 3D input devices 104, reference is made to the prior art cited in the introduction to the description. Such products are commercially available from LogiCad3d, for example.

Referring now to FIG. 2, the structure of the freely specifiable real-time control 102 according to the invention for animated graphics, video and / or audio sequences will now be explained in detail. The arrows 116 and 120 shaded in gray show the real-time input and output of signals from or to the freely definable real-time control 102, namely the control signals 116 (x, y, z, φ _κ , φ _y and φ _z ) from the 3D input device 104 and the animation output data 120.

As can be seen from FIG. 2, the control signals 116 are fed to a free specification module 202. One function of the free specification module 202 is to use the absolute values of the entered “analog data” x, y, z, φ _x , φ _y and φ _z to derive the first time derivatives (velocities or angular velocities) or the second temporal derivatives ( Accelerations or angular accelerations) can be determined without the aid of an additional sensor (acceleration sensor), for example, the speed and the acceleration with respect to the “analog data” mentioned under the control signals 116. A second important function of the free specification module 202 is to control the control signals 116 of the various degrees of freedom for the components for generating and / or influencing the graphics, image or sound sequences, ie the motion generator 204, the surface generator 206, the acoustic generator 208 or the Lighting generator 210 to assign. For example, a mask can be provided for the user during the initialization of the real-time control 102 according to the invention for animated graphics, video and / or audio sequences, which enables the user to access the free specification module 202 in such a way that the user has a degree of freedom Control signals 116 can assign one of the above-mentioned components - for generating and / or influencing the graphics, image or sound sequences.

In addition, with the help of this free specification module 202, the user can freely set the coupling parameters for the assignment of control signals to stored reference patterns, such as, for example, to carry out spectral filtering for damping or amplifying the signal components of audio signals in predeterminable frequency ranges. The adjustable coupling parameters determine the temporal coupling and the damping or amplification of the change in the corresponding reference pattern depending on the assigned control signal and thus the signal response behavior of the system. Depending on the specification by the user, the free specification module 202 has the effect that the coupling parameters of the assignment - such as the damping or amplification - are freely adjustable. If, for example, the coupling parameters are changed with regard to the movement pattern of a walking movement of the joint object, it is possible to switch continuously from a dynamic to a limping walking movement. According to the invention, different moods or emotions can also be expressed as a result Facial expressions of a depicted face. The

Movement patterns can be linked to mood patterns as follows:

The user can therefore continuously select between the walking movement of a young athlete and that of an old man and, coupled with this, easily select different characters and / or moods.

In terms of a higher-value assignment, it is also possible to select the character (young / old etc.) and / or the mood (funny / sad etc.) instead of setting individual coupling parameters in the course of the specification, with the free specification module 202 then automatically sets the individual coupling parameters (e.g. for the walk, face, etc.). Instead of setting individual coupling parameters, this higher-value (linked) assignment thus enables the selection of a character and / or a mood. The choice of mood can influence different patterns, so that, for example, the The soundtrack, the scene lighting and the movement can be changed. Thus, an action by the user can automatically change several coupling parameters of one pattern as well as several coupling parameters of different patterns (movement, surface, image and / or sound). Thus, you can create choreographies that control real-time arrival of a character figure and background USAGE ^¬.

Depending on the specification by the user, the free specification module 202 assigns the corresponding control signals 116 as follows:

a motion generator 204 as parameterized motion control signals 212,

as parameterized surface control signals 214 to a surface generator 206,

- as parameterized acoustic control signals 216 to an acoustic generator 208 and / or - as parameterized lighting control signals 218 to a lighting generator 210.

The motion generator 204 combines the motion control signals 212 assigned to it with at least one predefined parameterized motion pattern 106 for generating motion animation output signals 220. Thus, different motion patterns of the virtual joint object (eg walking in different directions, stooping or Jumping etc.) can be controlled in real time.

The surface generator 206 links the surface control signals 214 assigned to it, the motion animation output signals 220 generated by the motion generator 204, and data from at least one predetermined parameterized surface pattern 108 for generating surface outputs. signals 222. Thus, for example, a structural,

Changes in the color, saturation or brightness of certain surfaces of a moving object and / or its surroundings can be initiated.

The acoustic generator 208 links the acoustic control signals 216 assigned to it with at least one parameterized, predefined sound pattern 110. The product of this connection is the sound output signals 224, which represent part of the animation output signals 120. The gespeicher ^¬ th sound patterns may for example be parameterized digital audio files.

Similarly, the illumination generator 210 linked to its assigned illumination control signals 218 with at least one pre-defined parameterized illumination pattern 112 to produce illumination output signals 226. The illumination pattern 112 may in this case a virtual Whether ^¬ ject and / or affect its surroundings.

As already stated, the predefined movement patterns 106, surface patterns 108, sound patterns 110 and lighting patterns 112 are each parameterized, so that the corresponding assigned control signals can be used as parameter values by the movement generator 204, the surface generator 206, the acoustic generator 208 or the lighting generator 210 , perform a real-time control of the corresponding pattern depending on the control signals 116 from ^¬, said real-time control of the corresponding pattern in the corresponding change in the erzeug ^¬ th animation data 220, 222, 224 and 226 is reflected, the total of the animation - Output signals 120 can be summarized. The user can by manipulating the

Input device 104 and the corresponding assignment by the free specification module 202 (which was set by the user himself or in advance by the manufacturer) in real time the movement patterns 106, the surface patterns (geometries, textures, etc.) 108, the sound patterns 110 and / or control the lighting pattern 112. On the one hand, it can trigger predefined patterns, for which purpose a switch on the input device 104 can be used, for example. On the other hand, it can use the "analog drive signals" x, y, z,

(|) _x , φ _y or φ ₂ perform real-time control of the above-mentioned patterns.

The function of the generators 204, 206, 208 and 210 is to be explained in detail using the motion generator 204:

It is assumed that a joint body should be controlled in real time during the animation. In this case, the movement generator 204 continuously calculates the new position and orientation of the joints of the joint body depending on the movement control signals 212.

A movement pattern 106 is thus defined, for example, as follows: First, skeletal data are defined that describe the type and position of the joints of the joint body. In addition to the position of the joints (position), the type of joints (which can be described by the type and number of degrees of freedom) is also defined. For each joint j of the J joints of a virtual joint object under consideration, the following string S _\ can be defined, for example:

S: = {namβj, position- _} , degrees of freedom-j,

Maximum_Amplitudej, Attenuationj, ...} for 1 <j <J

A movement sequence then consists of a list or a function that determines which of the defined joints moves when and how, depending on the parameterized control signals. A sequence of movements can thus consist, for example, of a list of the joints to be “processed.” For each cycle, the list or a function is used to calculate which new position and / or orientation of the joints results from the individual contributions of the various joints depending on the control signals.

At a point in time n, the position of a joint, defined by the coordinates x, y, z, and its orientation, defined by the angular coordinates φ _{x ι} φ _y or φ ~, are calculated as follows:

This equation (1) is therefore calculated for each joint defined in the skeletal data. The new position and orientation of a joint at time n is thus calculated from the position in the previous cycle n-1 plus the new contributions which are dependent on the control signals

Δx-, Δy_, Δz-, Δφ _x , Δφ _y and Δφ _{Z /} .. Each joint j thus makes a contribution to the change

Position and orientation of each joint, including itself.

The contributions of each joint j to the displacement of the position or orientation of every other joint and the same joint are calculated using the following formula:

[ΔXJ, Δy- _j , ΔZ _J , Δφ _{Kι i} , Aφ _y , ₃ , Aφ _Zι ] ^τ : = fj (x, y, z, φ _xl φ _yr φ _z , x, y, z, φ _x , φ _y , φ _z , x, y, z, φ _x , φ _y , φ ₂ ) (2) with f _j : <- » ¹⁸ → ^ ⁶ V js {1, 2, 3, ..., J}.

The vector-valued function fj represents a representation, such as a certain movement, e.g. the course of a human being can be reproduced in a standardized manner.

3, the sequence according to the present invention will finally be explained again. Steps 301a, 301b and 301c relate to a setting and assignment priority in the free specification module 202. This process can either be defined by the user or else in advance. After starting in step 301 so the user and / or a programmer can by releasing specification module 202, the individual control signals 116 of the various degrees of freedom different patterns (movement patterns 106, Upper ^¬ surface pattern 108, sound patterns 110 and / or illumination patterns 112) to assign. In a step 301c, the coupling parameters between the control signals and the changes in the corresponding patterns that are actually carried out in real time can be set. These coupling parameters can relate, for example, to the attenuation and the amplification of sound patterns. When a real-time animation starts in step 302, the control signals are then forwarded from the SD input device 104 to the real-time controller 102 for animated graphics, video and / or audio sequences in a step 304. The control signals are then evaluated in a step 306, the first and second time derivatives of the “analog control signals” being determined, for example. Finally, in a step 308, the control signals of the different degrees of freedom are assigned to the movement patterns 106, surface patterns 108, already mentioned above.

Sound patterns 110 and / or lighting patterns 112 executed.

With regard to the lighting pattern 112, it should be noted that both the lighting situation for an animated articulated object itself and the lighting of the background or different scenes can be changed in real time by corresponding assignment and input of corresponding control signals 116.

Referring to the exemplary embodiment of the underlying invention shown schematically in FIG. 4, a method or a system can be provided for controlling individual or several acoustic parameters of digitized sound patterns 410, which uses the following devices:

a computer, the control unit 401 of which is conductively connected to at least one electronic musical instrument 402 via a MIDI interface 404a + b and

an input device 104 for input of parameterized control signals 116 for real-time motion control of virtual objects that can be displayed on a screen. The digitized sound patterns 410 can be digitally stored in the memory of at least one electronic musical instrument 402 in accordance with the MIDI standard and generated or played back on at least one electronic musical instrument 402 in accordance with the MIDI standard. Individual or more of these acoustic parameters can be generated or influenced according to the invention by controlling the virtual objects by moving the input device 104 in real time.

As shown in Figure 5, alternatively, the underlying _^ invention, a method and a system of individual or for controlling a plurality of acoustic parameters of digitized sound pattern 512 according to a variation of this embodiment can be provided which det the following devices USAGE ^¬:

a computer, the control unit 501 of which is connected to at least one sound card 510 and to at least one tone generator 502, and

an input device 104 for input of parameterized control signals 116 for real-time motion control of virtual objects that can be displayed on a screen.

The digitized sound patterns 512 can be stored digitally in the memory of at least one sound card 510 of the computer and can be generated and played on at least one tone generator 502 of the same computer. One or more acoustic parameters of the digitally stored sound patterns 512 can According to the invention by means of the virtual objects Steue ^¬ tion are generated by moving the input device 104 in real-time or influenced. The controllable virtual objects, which are controlled to generate or influence individual or more acoustic parameters of the digitally stored sound patterns 410 or 512 by moving the input device 104, can be, for example, the control elements, buttons,

Act switches, knobs and / or sliders of a virtual mixer, with the help of which, for example, individual frequency ranges of the overtone spectrum of a stored, digitized sound pattern 410 or 512 can be specifically attenuated or amplified.

The input device 104 can be, for example, a wireless control medium that can be moved in three dimensions (“3D mouse”) and which is capable of simultaneously parameterized control signals 116 of movements in at least six independent degrees of freedom to process in real time.

The controlled acoustic parameters of a specific digitized sound pattern 410 or 512 can be, for example, pitches, note or pause values, volumes, tempos, articulation instructions, timbres, pedal, vibrato, chorus, reverb, overtone and / or other special effects.

Parameterized control signals 116 of movements of the input device 104 can be entered in three translational and three rotational degrees of freedom. In addition, it can be provided according to the invention that time derivatives of the parameterized control signals 116 can also be determined and processed. The assignment of parameterized control signals 116 of one degree of freedom to one or more parameters of a specific digitized sound pattern 410 or 512 can be freely specified according to the invention. Optionally, the time derivations can be assigned to one or more parameters of a specific digitized sound pattern 410 or 512 in accordance with the parameterized control signals 116. The coupling parameters of the assignment of parameterized control signals ilβ can also be freely specified.

The meaning of the symbols provided with reference symbols in FIGS. 1 to 5 can be found in the attached list of reference symbols.

LIST OF REFERENCE NUMBERS

No, symbol

100 simplified block diagram to illustrate the input and output signals of the freely specifiable real-time controller 102 for animated graphics, video and / or audio sequences according to the preferred exemplary embodiment of the underlying invention

102 freely definable real-time control of animated graphics, video and / or audio sequences

10 '3D input device for _. Input of the control signals 116

106 parameterized movement patterns for joint objects

108 parameterized surface patterns (geometries, textures) for joint objects

110 parameterized wedge patterns

112 parameterized lighting patterns for articulated objects and the background of virtual scenes ^'

114 3D animation, displayed aμ-f computer screen

116 control signals from the 3D input device 104 for the freely specifiable real-time control 102 for animated graphics, video and / or audio sequences

118 Transfer of the parameterized movement patterns 106, surface patterns 108, sound patterns 110 and / or lighting patterns 112 as reference patterns to the corresponding generators 204, 206, 208 and 210 in the freely specifiable real-time controller 102 for animated graphics, video and / or audio sequences

120 animation output signals from real-time controller 102 for animated graphics, video and / or audio sequences for 3D animation 114

120a animation output signals from the control unit 401 of a computer for the 3D animation 114

120b status signals from the 3D animation 114 for the Steuerein ^¬ ness 401 computer

200 detailed block diagram illustrating the components as well as the input and output signals of the free specific ^¬ fizierbaren real-time control 102 for animated graphics, video and / or audio sequences

202 free specification module for assigning the control signals or their time derivatives as parameter values for the parameterized movement pattern 106, surface pattern 108, sound pattern 110 or lighting pattern 112

204 motion generator for calculating the skeleton change data of joint objects on the basis of the control signals 116 as parameter values and at least one parameterized movement pattern 106 each, the skeleton change data translational and / or rotational changes in the position or orientation of the Play joint objects

206 surface generator for calculating the surface virtually No . symbol

eller joint objects based on their skeletal change data and at least one surface pattern 108 each

208 Acoustic generator for generating sound sequences, based on parameterized sound patterns 110

210 lighting generator for generating lighting output signals, starting from parameterized lighting patterns 112

! 12 motion control signals from free specification module 202 to motion generator 204

214 surface drive signals from free specification module 202 to surface generator 206

216 sound drive signals from the free specification module 202 for the acoustic generator 208

218 lighting control signals from the free specification module 202 to the lighting generator 210

220 motion animation output signals from motion generator 204 to surface generator 206

222 surface output signals - from surface generator 206

224 sound output signals from the acoustic generator 208 as part of the animation output signals 120

226 lighting output signals from lighting generator 210

300 flowchart illustrating the zifizierbaren from the free spe ^¬ real-time control 102 performed for animated graphics, video and / or audio sequences Actions

301a Start of the setup phase [= preparation of a real-time animation)

301b presets, initializations and assignments for the free specification module 202

301c parameter presetting (e.g. damping or amplification)

301d End of the setup phase

302 start of the control phase start of a real-time animation)

104 Input of the control signals

106 Evaluation of the control signals

Allocation of the control signals

310 Generation of movement data or influencing of movement patterns 106 with the aid of the animation output signals 120

312 generate surface data or influencing of surface patterns 108 using the animation Ausgabesi ^¬ gnale 120

314 generation of sound data or influencing of Klangmu ^¬ star 110 by means of the animation output signals 120

Generating illumination data or influencing of Be ^¬ leuchtungsmustern 112 using the animation Ausgabesi ^¬ gnale 120

400 individual block diagram illustrating the control ^¬ ner or more acoustic parameters of digitized sound pattern by a system consisting of a computer- No . Symbol ter, the control unit 401 of which is conductively connected to at least one electronic musical instrument 402 via a MIDI interface and a 3D input device 104 for input of parameterized control signals for real-time motion control of virtual objects that can be displayed on a screen

401 control unit of a computer

402 master keyboard or synthesizer according to the MIDI standard (here: keyboard with 73 keys, i.e. with an ambitus of six octaves)

404a MIDI / IN connector (input socket) of the master keyboard or synthesizer 402

404b MIDI / OUT connector (output socket) of the master keyboard or synthesizer 402

406 Loudspeaker connection socket of the master keyboard or synthesizer 402

40! speaker

410 memory of the master keyboard or synthesizer 402, in which a database with digitized sound patterns is stored

412a MIDI / IN signal from the control unit 401 of the computer to the master keyboard or synthesizer 402

412b MIDI / OUT signal from the master keyboard or synthesizer 402 to the control unit 401 of the computer

414 Transfer of the digitized sound pattern 410 to the master keyboard or synthesizer 402 as a reference pattern

500 block diagram to illustrate the control of individual or several acoustic parameters of digitized sound patterns by a system, consisting of a computer which has an electronic tone generator 502 and a sound card 510, and a 3D input device 104 for input of parameterized control signals for real-time motion control of virtual objects that can be displayed on a screen

501 Control unit of the computer, which has an electronic tone generator 502 and a sound card 510

502 electronic tone generator of the computer according to the MIDI standard

504a MIDI / IN signal input of the electronic tone generator

504b MIDI / OUT signal output of the electronic tone generator

506 Computer speaker connector

speaker

510 computer sound card

512 memory of the sound card 510, in which a database with digitized sound patterns is stored

514a parameterized MIDI drive signals from the control unit 501 of the computer to the sound card 510

514b MIDI / IN signal from sound card 510 to electronic tone generator 502 No symbol

514c MIDI / OUT signal from the electronic tone generator 502 to the control unit 501 of the computer

Claims

claims

1. Method for real-time motion control of a joint object, in which the following steps are carried out: a) definition of at least one parameterized movement pattern (106) for the joint object, b) definition of at least one parameterized surface pattern (108) for the joint object, c) input of parameterized control signals (116) by means of a 3D input device (104), d) calculation of skeletal change data on the basis of the parameterized control signals (116) and on the basis of at least one parameterized movement pattern (106), the skeleton change data reflecting translational and / or rotational changes in the position or orientation of the joints of the joint object, e) calculating the surface of the joint object on the basis of the skeleton change data and on the basis of at least one surface pattern (108) , steps c), d) and e) being repeated cyclically in order to perform a real-time movement step control of the joint object depending on the control signals (116).

2. The method according to claim 1, characterized in that sound signals (224) are further generated by linking stored digital, parameterized sound patterns (110) with parameterized control signals (116).

3. The method according to any one of the preceding claims, characterized in that time derivatives of the parameterized control signals (116) can be determined and processed.

4. The method according to any one of the preceding claims, characterized in that the assignment of the parameterized control signals (116) or their derivatives to the movement patterns (106), surface patterns (108) or sound patterns (110) can be freely specified.

5. The method according to any one of the preceding claims, characterized in that further parameterized lighting patterns (112) are stored, which are used to generate lighting effects by means of corresponding links with parameterized control signals (116).

6. Method for freely specifiable real-time control of animated graphics, video and / or audio sequences, characterized in that

at least one predefined parameterized movement pattern (106), surface pattern (108) and / or sound pattern (110) is stored for generating movement, surface and / or sound data,

- Parameterized control signals (116) of different degrees of freedom are evaluated by a 3D input device (104) and to generate movement data (310), surface data (312) and / or sound data (314) with the aid of the stored movement patterns (106), surface patterns (108) and / or sound patterns (110) can be used and

- The assignment of parameterized control signals (116) of a degree of freedom to a specific movement pattern (106), surface pattern (108) and / or sound pattern (110) can be freely specified.

7. The method according to claim 6, characterized in that further time derivatives of the parameterized control signals (116) a certain movement pattern (106), Surface patterns (108) and / or sound patterns (110) can be assigned.

8. The method according to claim 6 or 7, characterized in that coupling parameters of the assignment of parameterized control signals (116), such as damping and amplification, can be freely specified.

9. Computer software product, characterized in that it implements a method according to one of the exemplary embodiments described above when it is loaded in a memory of a computer.

10. System for real-time motion control of virtual or real articulated objects, the system having a memory in which are stored:

- at least one parameterized movement pattern (106) and - at least one parameterized surface pattern (108) of a joint object,

a 3D input device (104) for input of parameterized control signals (116) for real-time motion control,

- A motion generator (204) for calculating skeleton change data on the basis of the parameterized control signals (116) and on the basis of at least one parameterized motion pattern (106), the skeleton change data translational and / or rotational changes in position or orientation of the joints of the joint object, as well

- A surface generator (206) for calculating the surface of the joint object on the basis of the skeleton change data and the parameterized surface pattern

(108).

11. System according to claim 10, ς _f ek indicates by a sound generator (210) for generating sound signals by linking at least one digitally parameterized sound pattern (110) with parameterized control signals (116).

12. System according to one of claims 10 or 11, characterized by a free specification module (202) for determining and processing time derivatives of the parameterized control signals (116).

13. System according to one of claims 10 to 12, characterized by a free specification module (202) for optional assignment of the parameterized control signals (116) or their derivatives for the movement data (310), surface data (312) and / or sound data (314).

14. System for freely specifiable real-time control for animated graphics, video and / or audio sequences, comprising - a memory in which for the generation of movement data (310), surface data (312), sound data (314) and / or lighting data ( 316) at least one predefined parameterized movement pattern (106), surface pattern (108), sound pattern (110) and / or lighting pattern (112) is stored, and

- A computing unit for evaluating parameterized control signals (116) of different degrees of freedom from a 3D input device (104) for generating movement data (310), surface data (312), sound data (314) and / or lighting data (316) with the aid of the stored evidence - pattern (106), surface pattern (108), sound pattern

(110) and / or lighting pattern (112), further comprising a free specification module (202) for the freely specifiable assignment of parameterized control signals (116) each with a degree of freedom to a specific movement pattern (106), surface pattern (108), sound pattern (110) and / or lighting patterns (112).

15. System according to claim 14, characterized in that the free specification module (202) furthermore the assignment of time derivatives of the parameterized control signals (116) to a specific movement pattern (106), surface pattern (108), sound pattern (110) and / or lighting tion pattern (112).

16. System according to claim 14 or 15, characterized in that the free specification module (202) enables a freely specifiable ^• setting of coupling parameters of the assignment of parameterized control signals (116), such as damping and amplification.

17. A system for real-time control of acoustic parameters of digitized sound patterns (410), comprising:

- A computer, the control unit (401) with at least one electronic musical instrument (402) via a MIDI interface (404a + b) is conductively connected and

- A 3D input device (104) for input of parameterized control signals (116), the digitized sound pattern (410)

- Are digitally stored in the memory of at least one electronic musical instrument (402) and

- Can be generated and played on at least one electronic musical instrument (402), characterized in that individual or more acoustic parameters of the digitally stored sound pattern (410) are obtained by moving the

3D input devices (104) can be generated or influenced in real time.

18. System according to claim 21, characterized in that for generating or influencing individual or more acoustic parameters of the digitally stored sound pattern (410) control elements, buttons, switches, rotary knobs and / or sliders of a virtual mixer are provided.

19. System according to claim 18, characterized in that the controlled acoustic parameters of a specific digitized sound pattern (410) are pitches, note or pause values, volumes, tempos, articulation instructions, timbres, pedal, vibrato, chorus. , Reverb, overtone and / or other special effects.

20. System according to one of claims 17 to 19, characterized in that time derivatives of the parameterized control signals (116) can be determined and processed.

21. System according to one of claims 17 to 20, characterized in that the assignment of parameterized control signals (116) of a degree of freedom to one or more parameters of a specific digitized sound pattern (410) can be freely specified.

22. A system for real-time control of acoustic parameters of digitized sound patterns (512), comprising:

- A computer, the control unit (501) of which is connected to at least one sound card (510) and to at least one sound generator (502), and

- A 3D input device (104) for entering parameterized control signals (116) for real-time motion control of objects, the digitized sound patterns (512) - being digitally stored in the memory of at least one sound card (510) of the computer and

- can be generated on at least one tone generator (502) of the same computer and played back, characterized in that individual or a plurality of acoustic parameters of the digitally stored sound patterns (512) generated by a movement of the 3D input device (104) in real time or influenced ^¬ that can.

23. The system according to claim 22, characterized in that time derivatives of the parameterized control signals (116) can also be assigned to one or more parameters of a ^certain digitized sound pattern (512).

24. System according to any one of claims 22 or 23, characterized in that coupling parameters of the assignment of parameterized control signals (116) can be freely specified.

25. A method for real-time control of acoustic parame ^¬ tern digitally recorded sound patterns (410 or 512) - are digitally stored in the memory of at least one sound card (510) of a computer and / or at least one electronic musical instrument (402) according to the MIDI standard and - on at least one tone generator (502) of the same computer or at least one electronic musical instrument (402) can be played according to the MIDI standard, the control unit (401 or 501) of the computer

- With at least one sound card (510) and at least one tone generator (502) or with at least one electronic

Musical instrument (402) via a MIDI interface (404a + b or 504a + b) 'is conductively connected and

- Conductively connected to a 3D input device (104) for entering parameterized control signals (116) for real-time motion control of virtual objects, characterized in that individual or more acoustic parameters of the digitally stored sound patterns (410 or 512) are controlled of the virtual objects can be generated or influenced in real time by moving the 3D input device (104).

26. The method according to claim 25, characterized in that the virtual objects which are moved to generate or influence individual or more acoustic parameters of the digitally stored sound patterns (410 or 512) are the control elements, buttons, switches, Knobs and / or sliders of a virtual mixer is.

27. The method according to any one of claims 25 or 26, characterized in that the controlled acoustic parameters of a certain digitized sound pattern (410 or 512) are pitches, note or pause values, volumes, tempos, arti- calculation instructions, timbres, pedal, vibrato, chorus,

Hall, overtone and / or other special effects.

28. The method according to any one of claims 25 to 27, characterized in that coupling parameters of the assignment of parameterized control signals (116) can be freely specified.