EP1609307B1 - Image reproduction device - Google Patents

Abstract

The invention relates to an image reproduction device comprising an input unit (1) which is used to receive image data and which is connected to an image data source (2), and an output unit (3) which is used to emit image data and which is at least indirectly connected to the input unit (1). The output unit is embodied in such a way that it supplies image data to at least one image display screen (4). A processing unit (6, 9) comprises an input for receiving at least one temporally changeable parameter (5). The processing unit (6, 9) is connected to the input unit (1) and the output unit (3), and is embodied in such a way that the image data received by the input unit (1) is continuously modified according to parameters in partial regions defined by masks, and the image data is supplied to the output unit (3).

Description

The present invention relates to an image reproducing apparatus having an input unit for receiving image data connected to an image data source and an output unit for outputting image data at least indirectly connected to the input unit and configured to output image data to an image display such as a screen issue.

Such systems are widely known, for example, females are used to reproduce images. Frequently used as an image data source is a video recorder which plays back image data stored on a video cassette and which receives and reproduces a monitor. Such a picture display system consisting of video recorder and television monitor has the disadvantage that it can only reproduce recorded picture information. The image playback system is not able to react to external influences. Image reproduction systems that can at least partially respond to external influences are known from public transport. For example, monitors are placed in subway stations to present advertisements to passers-by. These images are simply played from an image data source that stores the images.

However, when a train enters, the viewer is presented with an image on the monitors that draws his attention to this event. For example, the message "Attention, a train enters" appears on the monitor. Interactivity with the environment is achieved by having the monitor receive its information from a second image data source in the event of an incoming train. This image data source contains the image data representing the warning message to be issued. Ultimately, therefore, only existing image data are reproduced, only the order and the time in which these image data are reproduced can be influenced by the environment, because it is switched between two image data sources. The ability of this image display device to respond to its environment is therefore still severely limited.

Another common application of interactive imaging is computer games. The player is exposed to images that suggest a situation to which he has direct influence by means of input devices (typically joystick, keyboard, or computer mouse) to solve the tasks given by the situation , The situation can be taken from a wide variety of areas, modeled on the real world as accurately as possible, or highly abstract in appearance and content.
Out US-A-4090 216 a device is known in which, depending on the brightness of the environment, a processing of brightness or contrast of a telephoto image takes place.
It is an object of the present invention to provide a device for image reproduction with extended and changed possibilities.

The object is achieved by a device having the features of claim 1. The dependent claims relate to preferred embodiments.

An image reproducing apparatus comprises: an input unit for receiving image data connected to an image data source and an output unit for outputting image data at least indirectly connected to the input unit and configured to output image data to an image display such as a screen; a processing unit having an input for receiving at least one temporally variable parameter originating externally from outside the device, the processing unit being connected to the input unit and the output unit and being designed to receive image data received from the input unit in subregions defined by masks as a function of the parameters continuously change and output to the output unit. At least one of the parameters is deprived of direct influence by a user or viewer.

The image display receives the image data in the appropriate form from the output unit, which in turn is connected to the processing unit. This receives the time-variable parameter via an input. The image data originating from the image data source are continuously changed by the processing unit in partial areas depending on the parameter. It is thus possible for the device for image reproduction to respond to the parameters in such a way that the reproduced images are changed in partial areas. The change in the images thus not only causes the display of a completely different image, but also the display of an image changed in some areas. Also, the reproduced images no longer correspond to an image originating from an image data source, but are changed from it. The image data reproduced after processing are new and their appearance is determined solely by the dependence between the incoming parameters and the associated change of the image data by the processing unit.

The continuous dependencies of the image processing effects on the input parameters are coordinated via a sequence control and optionally varied in time according to a user. The temporal variation can be both spontaneously during the ongoing processing, as well as in advance. The sequence control is designed in such a way that the incoming measurement results and images are optimized depending on the performing medium, its image size, refresh rate, color depth and contrast range. For the same input data, so different images are generated, for example, as regards the richness of detail, yet convey the same message. This will be further illustrated in the context of the dissemination of image data over networks.

The device according to the invention is thus able to respond in a much more differentiated manner to external events, as is known from the prior art.

Although post-production systems such as Photoshop, Flame and others are known from the prior art. These are intended to edit stored on a storage medium image data by means of a personal computer targeted. The effects of input commands for image editing are displayed on the screen of the PC. Thus, the PC receives no time-varying parameters such as temperature or brightness of a particular environment via an input, but commands via a keyboard or other input means for targeted and predictable editing of images. Furthermore, the image changes produced by these image processing programs are usually pre-produced as effects. The continuous change of image parameters such as color and brightness as a function of an external parameter can not be made visible by means of these programs. Instead, initial and final states of image parameters are calculated in these post-production systems. The intermediate states are either pre-produced dynamically, achieved by effect filters or determined by a mixture of the initial and final states. However, a recalculation in the strict sense does not take place. Such solutions are sufficient as long as the information to contain the image data as the final product is predictable. However, the requirements change as soon as unpredictable events affect the displayed image. In this case, the change of the image data must be continuous depending on the parameters. That is, a parameter value P (t) received at a certain time t causes a change in the image data BD (t) received at the same time t. If the change of the image data takes place for a limited time sequence of successive image data and parameter values as described above, there is a continuous change of the image data as a function of the parameter. The manipulation of the image data by the processing unit thus represents a procedural animation of the image data in the area of the masks. The masks can be fixed or can themselves be changed depending on the parameters, so that the masks themselves undergo a procedural animation.

Preferably, the received parameters represent events outside the device. An external event is any form of temporal change that is not generated by the device itself but is received via the input of the processing unit. Any conceivable physical variable such as brightness, volume or temperature can serve as a parameter. Furthermore, the parameters may also represent events such as the landing of an aircraft, the current value of a stock index or the number of accesses to an Internet site. Thus, the image processing apparatus is set in relation to its environment. The changes in the reproduced image in their own way reflect the events outside the device. For the unbiased viewer, the observation of the image is of high entertainment value, especially if it itself can manipulate the image changes as part of the environment of the device, and generally serves to intuitively capture information.

In a preferred embodiment, the processing unit is configured to change the image data by performing computational processes that realizes the change of the image data by the parameters. The one of the processing unit The calculation process performed is based on an algorithm that determines how the parameters affect the image data. Thus, it is impossible to predict which changes of the image data will occur, because the unknown timing of the parameter determines the change of the image data. The actual changes that occur are also a function of the time-varying parameter and thus unique for each value of the parameter. With prior knowledge of the incoming parameter, the temporal changes of the image data are determined, but since the knowledge of this parameter is generally unknown, the altered image data is unpredictable.

The processing unit of the device according to the invention is preferably designed such that it changes the received image data on the basis of the parameters in real time. Such a processing unit, if necessary with the components of the device directly connected with it, is also referred to below as "real-time system". By changing the image data in real time, it is understood that if it is impossible for a human observer to detect a delay in processing. Such a real-time change thus represents for an observer a temporally direct relationship between the received parameters and the reproduced image. The device is thus able to reflect the current events in a different form. By looking at the reproduced image, the current external events can be derived by thought. The reproduced image thus represents a decipherable code, that is to say a kind of language, which represents the external events.

The processing unit can also be designed such that it changes the image parameters of the received image data such as color, brightness, contrast, gradation or sharpness. In this way, the image parameters are related to the parameters that represent the environment of the device. The appearance of a changed portion of the rendered image thus represents a complex representation of different events. Besides such individual pixels or pixels directly descriptive parameters, the processing unit can also be configured to change from a plurality of pixels composite image objects by influencing their position in the (virtual) space, their surface texture or the like. The image objects can also be calculated three-dimensionally.

In preferred variants of the processing unit, the processing unit is designed either to time-transform image objects of the temporally variable (control) parameter or also to regenerate new, not only naturalistic but also artificial appearing image objects as a function of the time-variable parameter and to add them to the incoming image data ,

Furthermore, the processing unit is preferably designed to influence the dynamics of image sequences or image sequences, either by direct, time-controlled influencing of the incoming image data or by processing dependent on the temporally variable (control) parameter-dependent processing and stored sequences.

A special embodiment of the invention is characterized in that the output unit is designed to output the changed image data to image displays via the Internet or television picture transmitter. Thus, it is possible to output the image data to a plurality of image displays at different locations simultaneously.

Preferably, the device according to the invention comprises a control unit which is connected to the sensor and the output unit and is suitable for controlling the sensor, the control unit receiving image data from the output unit and controlling the sensor in dependence on the image data received from the output unit. This returns the output, changed images with the image data source.

According to a preferred embodiment, the device comprises an input unit connected to the processing unit for determining the processing processes of the processing unit. This allows an "author" to specify how the parameters change the image data. Because the computing processes performed by the processing unit can be determined via the input unit. Furthermore, it can be provided that the input unit has a user interface that visualizes the computing processes of the processing unit. The author can thus check at any time, which computational processes are performed by the processing unit. Particularly in connection with the use of image masks, the speed and intensity of the image change can be determined by the input unit. specify by fading in and out of the masks. The input unit can also be an authoring system mentioned below.

The computing processes of the processing unit can also serve to simulate the movement of liquids, gases, physical particles or solids. The parameters received by the sensor thus produce changes that appear to the viewer of the image as the movements of real fluids, gases, or solids. In addition to lifeless objects such as crystals, solid bodies also include living objects such as animals, humans and plants. The illustrated image changes create the illusion of a real movement, such as that of a water jet.

To simulate flowing substances such as gases, liquids or solid erosive substances, the Navier-Stokes equation is preferably used. The Navier-Stokes equation describes the motion of flowing substances whose velocity of motion is well below the speed of sound. The observed in everyday life liquids such as water droplets are described for example by the Navier-Stokes equation. Using the Navier-Stokes equations, the previously mentioned, artificial image objects can also be generated.

Preferred embodiments of the present invention will be described below with reference to the accompanying drawings.

Fig. 1

eine Vorrichtung zur Bilddatenwiedergabe gemäß einem ersten bevorzugten Ausführungsbeispiel,

Fig. 2

eine Bearbeitungseinheit der Vorrichtung zur Bilddatenwiedergabe gemäß dem ersten Ausführungsbeispiel,

Fig. 3

eine Vorrichtung zur Bilddatenwiedergabe gemäß einem zweiten Ausführungsbeispiel,

Fig. 4

eine Bearbeitungseinheit der Vorrichtung zur Bilddatenwiedergabe gemäß dem zweiten Ausführungsbeispiel, und

Fig. 5

eine schematische Übersicht über ein alternatives Echtzeit-Bildatenverarbeitungssystem.

Show it:

Fig. 1

an image data reproducing apparatus according to a first preferred embodiment;

Fig. 2

a processing unit of the image data reproducing apparatus according to the first embodiment;

Fig. 3

a device for image data reproduction according to a second embodiment,

Fig. 4

a processing unit of the image data reproducing apparatus according to the second embodiment, and

Fig. 5

a schematic overview of an alternative real-time image processing system.

The following is the device in Fig. 1 described. The apparatus according to the first embodiment includes an image data source 2. As the image data source 2, any storage medium for images such as video cassettes, photographs, DVD disks, and hard disks is contemplated. The image data is reproduced by the input unit 1. The input unit can be the image data sources corresponding playback devices such as VCR, DVD drive or the read head of a hard drive can be used. The input unit 1 transmits the reproduced image data to the processing unit 6. At the same time, the processing unit 6 receives temporally variable information from a sensor 5. As the processing unit 6, an electronic computer capable of performing various operations on the received image data is usually employed perform. First, the processing unit 6 selects from the received image data a partial area defined by a mask. The manipulation of the image data occurs only within the selected mask. The position and size of the mask can be predetermined, so that the processing unit 6 only has to recognize the image data to be processed on the basis of the mask. However, it can also be provided that the processing unit 6 automatically determines a mask by means of predetermined mask recognition methods. Furthermore, the processing unit 6 may be configured to determine the masks in dependence on the parameters received from the sensor 5. As a sensor, any type of detector can be used, which detects such variables as temperature or brightness of a particular environment. It is also possible to use a plurality of detectors combined with one another so that the sensor 5 can receive a plurality of different information and forward it to the processing unit 6. The different information reaches the processing unit 6 as time-variable parameters. The image data transmitted by the input unit 1 are generally not changed by the processing unit 6 when the processing unit 6 does not receive information from the sensor 5.

The processing unit 6 calculates new image data by means of a predetermined arithmetic operation from the received parameters and the image data in the masks to be changed. The dependence of the change of the image data on the received parameters can be chosen arbitrarily. In order to determine the application of the arithmetic operations performed by the processing unit 6, an input unit 7 is provided. This input unit 7 allows a user to determine how the processing unit 6 changes the image data in dependence on the parameters, that is, which arithmetic operations are used and to what extent. As input unit 7 may be provided a keyboard or an electronic pen.

The image data is finally sent from the processing unit 6 to an output unit 3 issued. The output unit ensures that the image data is output to an image display 4 in a suitable form. In particular, it can be provided to output the image data to a plurality of image displays 4 in different forms. If the transmission takes place on image displays 4 via the Internet, the output unit ensures that the image data are continuously retrievable as compressed data packets from an Internet site. The retrieval of the data packets is carried out automatically by means of suitable software, which ensures that the image data are displayed on the corresponding screen. As image display 4, any type of monitor or projector can be used. A control unit 8 for controlling the sensor 5 is provided which also receives the image data output from the output unit 3. By predetermined operations, the control unit 8 converts the received image data into control commands for controlling the sensor 5. By these commands, the control unit 8 changes, for example, the position or orientation of the sensor 5. Further, the control of the sensor 5 may be to increase or decrease its sensitivity. The image data changed as a function of the parameters of the sensor 5 thus in turn cause a change in the parameters received via the sensor 5 via the control unit 8.

Fig. 2 shows a block diagram of the processing unit 6 according to the first embodiment. Therein, a mask recognition unit 10 is provided, which ensures that subareas to be processed of the image data received by the input unit 1 are selected. For this purpose, the mask recognition unit 10 is connected to the input unit 1. The mask recognition unit 10 selects the image data to be processed on the basis of predetermined mask recognition methods and transmits them to a mask processing unit 11. The remaining image data are buffered by the mask recognition unit 10 in an image data memory 12. The mask processing unit 11 receives, in addition to the image data to be processed, time-varying parameters from the sensor 5, and calculates new image data depending on the received image data and the received parameters.

If the mask processing unit 11 does not receive parameter information from the sensor 5, it outputs the image data obtained from the mask recognition unit 10 unchanged to a mixing unit 14. This is connected to the mask processing unit 11 and the image data memory 12 and reassembles the image data unchanged from the mask processing unit 11 with the image data stored in the image data memory 12. The composite image data in this case correspond to the image data originally transmitted from the input unit 1 and are then transmitted from the mixing unit 14 to the output unit 3.

On the other hand, when the mask processing unit 11 receives temporally variable parameters from the sensor 5, it computes new image data for the mask area. The algorithm for calculating the new image data is stored in an operation memory 13. The mask processing unit 11 accesses the operation memory 13 to perform the calculation process for calculating new image data stored therein. The operation memory 13 is in turn connected to the input unit 7, which is provided to determine the operations of the mask processing unit 11 by storing the operations to be performed via the input unit 7 in the operation memory 13. The mask processing unit 11 transmits the newly calculated image data to the mixing unit 14. This composes the newly calculated image data with the image data stored in the image data memory 12, so that an image data set is produced which differs in the mask area from the image data input via the input unit 1. This image data set is in turn output from the mixing unit 14 to the output unit 3. Finally, the operation memory 13 is also provided to store the mask detection operations to be performed by the mask recognition unit 10. These operations can also be written into the operation memory 13 via the input unit 7. The mask recognition unit 10 is connected to the operation memory 13 and executes the mask recognition operations stored therein.

Fig. 3 shows the second embodiment of the present invention. In this figure, the features corresponding to the first embodiment are denoted by the same reference numerals. Instead of the processing unit 6, a processing unit 9 is shown, which has extended possibilities. This processing unit is in detail in Fig. 4 shown. However, the processing unit 9 still causes a change in the incoming image data depending on the parameters. A difference to the in Fig. 1 described device is that the control unit 8 has a second input, via which it receives the unchanged image data from the input unit 1. The control unit 8 is also designed to control the sensor 5 both in response to the changed image data from the output unit 3 and in dependence on the unchanged image data from the input unit 1. In particular, the control unit 8 can be designed such that it collects a difference between the unchanged image data and the changed image data and controls the sensor as a function of the difference. This difference corresponds to the change of the image data in the area of the masks. The control of the sensor is thus a function of the change of the image data.

Fig. 4 shows the processing unit 9 of the second embodiment in detail. The same components of the processing unit as in Fig. 6 are indicated by the same reference numerals. In addition to a first mask processing unit 11, the mask processing unit Fig. 2 corresponds, a second mask processing unit 15 is provided. This is connected to a mask memory 17 and a parameter memory 16, in which the parameters derived from the sensor are stored. The second mask processing unit 15 accesses these memories to change the image data in the mask area. The change of the image data is carried out according to the principle of the superimposition of so-called image layers. Image data suitable for the overlay are stored in the mask memory 17. Depending on the values of the parameter originating from the sensor stored in the parameter memory 16, image data stored in the mask memory 17 by the second mask processing unit 15 for the image mask to be changed in each case as specified and, if appropriate, processed as a function of the parameter. The image data obtained in this way are available at the output of the mask processing unit 15 and can be mixed in the mixing unit 14 with the image data output in real time by the first mask recognition unit 11. In this way it is possible, depending on the parameter originating from the sensor, to achieve cross-fading effects between the image data stored in the mask memory 17 and the image data originating from the first mask recognition unit 10 within image sections defined by the respective masks. The degree of the cross-fading is already determined by the second mask processing unit 15, in which the image data originating from the mask memory 17 is amplified or attenuated as a function of the parameter stored in the parameter memory 16. The image data originating from the input unit 1 are already amplified or attenuated in the first mask processing unit 11 as a function of the parameter originating from the sensor, so that the already mentioned cross-fading effects result from superposition of the image data processed in this way for certain masks. The modification of image data in the manner described last by superimposition with image data previously stored in the mask memory 17 is associated with a lower computation outlay than the completely new calculation of image data for specific masks, so that the embodiment of FIG FIG. 4 is suitable to realize the real-time image manipulation as a function of the parameter with less computational effort. In addition, the input unit 7 may be directly connected to the layer memory 17 and the mask processing unit 15, for example, to make the mask processing dependent on the input image signal.

• Die Berechnung der Bildströme in Echtzeit ist aufwändig, wobei der Rechenaufwand natürlich von der verwendeten Bildgröße und -wiederholfrequenz abhängt.
• Daher wird bei hohen Auflösungen ein zentraler Server mit großer Rechenleistung die Bilder berechnen. Bei niedrigerer Auflösung ist auch eine lokale Berechnung mit preiswerteren Systemen denkbar.

In the following, aspects of the technical realization will be explained in more detail:

• The calculation of the image streams in real time is complex, the computational effort of course depends on the image size and frequency used.
• Therefore, at high resolutions, a central server with high computing power will compute the images. At lower resolution, a local calculation with cheaper systems is conceivable.

Particularly interesting are distributed systems, which contain a portion of the logic in each single display, while the majority is concentrated on a server. This will be described separately.

1. 1) Hochleistungsgrafikrechner:
   • Solche Rechner - zur Zeit insbesondere die von sgi - enthalten spezielle Hardwarebeschleuniger, die die benötigten Rechenoperationen übernehmen. Über einen genormte Schnittstelle (OpenGL) wird die Grafikhardware programmiert. Diese Lösung bietet große Flexibilität, auch was die eventuelle zusätzliche Darstellung dreidimensionaler Bildinhalte betrifft.
2. 2) Cluster von "normalen" PCs:
   • Der Vorteil dieser Lösung liegt in der nahezu beliebigen Erweiterbarkeit durch Hinzufügen weiterer Rechner. Es existieren auch für diese Systeme Grafikbeschleuniger und mit derselben Programmierschnittstelle (OpenGL), wie bei den Hochleistungsgrafikrechnern, was eine eventuelle Übertragung vereinfacht und auch die Möglichkeit hybrider Systeme eröffnet.
3. 3) Bildeffektkarten und Videohardware, wie sie für den digitalen Videoschnitt verwendet werden:
   • Der Vorteil liegt in einem moderaten Preis bei hoher Leistung auf speziellen Gebieten. Der Nachteil ist die Begrenzung auf sehr bestimmte Auflösungen und relativ geringe Flexibilität, da die Effekte sehr genau auf die Karten abgestimmt sein müssen. Komponenten einer Serienproduktion in höheren Stückzahlen können allerdings durchaus diesem Bereich entnommen sein.
4. 4) Signalprozessorsysteme (insbesondere DSP, digitale Signalprozessoren):
   • Diese bieten extrem hohe Rechenleistung bei allerdings begrenztem Datenein- und -ausgang, was sich jedoch in näherer Zukunft aller Voraussicht nach ändern wird.

Basically, different architectures exist with respective advantages and disadvantages to calculate the pictures or parts of them:

1. 1) High performance graphics calculator:
   • Such computers - currently especially those of sgi - contain special hardware accelerators, which take over the required arithmetic operations. A standardized interface (OpenGL) is used to program the graphics hardware. This solution offers great flexibility, also with regard to the possible additional representation of three-dimensional image content.
2. 2) Clusters of "normal" PCs:
   • The advantage of this solution lies in the almost arbitrary expandability by adding more computers. Graphics accelerators and the same programming interface (OpenGL) also exist for these systems, as in the high-performance graphics computers, which simplifies any possible transmission and also opens up the possibility of hybrid systems.
3. 3) Image effect maps and video hardware as used for digital video editing:
   • The advantage is a moderate price with high performance in special areas. The disadvantage is the limitation to very specific resolutions and relatively little flexibility, since the effects must be very accurately matched to the cards. However, components of a series production in larger quantities can certainly be taken from this area.
4. 4) Signal processor systems (in particular DSP, digital signal processors):
   • These offer extremely high computing power, but with limited data input and output, which is likely to change in the near future.

Preferably, the processing unit is adapted to OpenGL.

OpenGL is a standardized interface that allows you to program a variety of graphics operations. The operations are performed either in software on the central processor or on the graphics card, depending on the system configuration. OpenGL is standardized and exists for a variety of different systems, so programs can be transferred with relatively little effort if the standard is adhered to. OpenGL is also a quasi-standard for three-dimensional image processing, but its potential in the two-dimensional area is hardly exhausted.

• Veränderung der Geometrie der Körper oder der Texturkoordinaten für Verzerrungen der Videobilder
• Lagen (layer) mehrerer teilweise transparenter Körper übereinander für Blendeneffekte
• Lookup Tables und Farbkorrekturmatrizen während der Texturdarstellung für Farbeffekte
• Beleuchtung: gleichmäßig, gerichtet, punktförmig und vieles mehr
• Multiple Texturen auf einem Körper
• Unschärfen und andere matritzenbasierte Effekte wie Eckenerkennung während der Texturdarstellung
• Entfernungsabhängige Nebeleffekte zur Farbkorrektur
• Partikelsysteme aus kleineren texturierten Objekten oder verzerrte Texturen zur Darstellung von Wirbeln oder anderen physikalischen Phänomenen.

In principle, OpenGL initially offers commands to display three-dimensional bodies. However, images - so-called textures - can be placed on these bodies and manipulated in a variety of ways. This provides powerful tools for changing two-dimensional images:

• Change the geometry of the bodies or the texture coordinates for video image distortions
• Layers (layer) of several partially transparent bodies one above the other for glare effects
• Lookup tables and color correction matrices during texture rendering for color effects
• Lighting: even, directional, punctiform and much more
• Multiple textures on a body
• Blurs and other matrice-based effects such as corner detection during texture rendering
• Distance-dependent fog effects for color correction
• Particle systems of smaller textured objects or distorted textures to represent vertebrae or other physical phenomena.

If all of this is done in hardware (architecture 1), then, depending on the equipment but different, a limitation of the simultaneously executable effects is given. That is, the effects must be mapped to the so-called graphics pipeline in such a way that a real-time behavior is possible. An additional unit is provided to plan these dependencies in advance and to map effects to the corresponding hardware.

For both 1) and 2), as the complexity of the effects increases, multiple systems can be switched either in parallel or in series, which entails disadvantages in terms of complexity and maintainability.

An important role in the realization of the concepts presented here plays the transmission over networks. A network may include, in addition to direct, wired connections, radio or satellite links, or the like.

For cost and maintenance reasons, it is preferably provided to centrally set up the computer systems that are necessary for the image generation: The hardware that is used to generate the data in real time is complex and requires a high degree of technical infrastructure and reliability all components; this requires a certain amount of space. However, the various displays should be as mobile and undemanding as possible in order to be very flexible everywhere.

A computer system can certainly record any number of displays with content, which considerably reduces the cost per recorded display. In order to still ensure a high degree of integration of the viewer in the projected events, a local processing unit is provided in the display, which makes a local adjustment of the global image streams with relatively little computational and financial outlay. This method will be described in more detail later.

Real-time compression can significantly reduce transmission rates and costs, but places high demands.

The transmission over narrow-band networks, whereby the Internet in particular plays an important role, places further requirements. Although there are techniques to transfer multimedia content, in particular moving images and sound, even on the Internet to transmit compressed, but the data rates are too low to achieve high image resolution.

Thus, images must be seen in smaller resolutions, which requires a fundamental adaptation of the image processing effects by the process control. The same input parameters and image sequences result in variable image resolution to similar and meaningful but - for example, the richness of detail, the color depth or the rate of change - yet different output images.

It is also envisaged that a well-equipped and correspondingly expensive server would be able to run a number of terminals to reduce the cost per terminal. However, this has a detrimental effect on the integration of the viewer in the events shown, which is the charm of the described system.

However, this disadvantage can be avoided: Each terminal receives a comparatively simple and cheap own image processing unit to the from the Server to enrich coming, elaborately designed material with interactive and local aspects. For this purpose, for example, in the server image stream at least one level is defined, which is changed in accordance with the image composition by the display. Thus, only explicitly released areas in the local image processing unit are processed in order to integrate locally acquired data.

Finally, a specific, alternative system for real-time procedural manipulation of image data and variants thereof with reference to FIG. 5 be explained.

For the system for real-time procedural manipulation basically two components have to be distinguished: the authoring system and the real-time system.

The authoring system (input unit) consists of a configuration module and a processing module. The configuration module specifies the hardware of the system. This includes the number of screens, the specification of the input data (measuring stations, music, interaction components), etc. The interactive editing module maps procedures (CFD, image processing) and masks to the actual video stream on a frame basis, which in turn are linked to the input data. In this way, the video image can be edited creatively. In addition, the specified sequences must already be simulated in the processing module. Both the data generated in the configuration module and in the processing module are saved in a specially designed data format. As a result, any scripts and configurations can be generated and reused at any time.

In the real-time system (processing unit), a new video data stream is composed of one or more video input streams, CFD, and procedurally controlled image manipulation. In addition to the video input data, the real-time system reads the configuration and the editing file.

Unlike the real-time system, temporal boundary conditions play a minor role in the authoring system. Interactive response times are perfectly adequate. Important in the case of the authoring system, however, is the user interface, which can even be completely dispensed with in the case of the real-time system, depending on the application. A user interface for the real-time system is only interesting if the system is to respond directly to the users. This is e.g. This is the case if one of the output devices contains a touch screen that allows the user to control the images, or if the video should respond to voice input.

• Video/Bildeingangsmodul 20, Eingabemodul 21, insbesondere Messwerterfassungsmodul, CFD-Simulator 22 (CFD: Computational Fluid Dynamics), Volumenvisualsierungseinheit 23, Bildverarbeitungseinheit 24 (OpenGL basiert), Ausgabeeinheit 25, Bilddatenspeicher 26 und zentrales Kompositionsmodul 27.

The real-time system is composed in the embodiment of the following components:

• Video / image input module 20, input module 21, in particular measured value acquisition module, CFD simulator 22 (CFD: Computational Fluid Dynamics), volume visualization unit 23, image processing unit 24 (OpenGL based), output unit 25, image data memory 26 and central composition module 27.

In the composition module, the input data streams converge. On the basis of the specification in the editing file, the composition module builds a 3D scene with video textures and generates and animates masks on the frames of the input video, eg with the aid of the stencil buffer. For procedural animation of image content, the composition module communicates with the input module, the CFD module, the volume visualizer, and the image processing unit. With the aid of the manipulation modules, the 3D scene description is modified accordingly and finally rendered and the image is forwarded to the image output unit. The image output unit can be omitted if, for example, the image data is passed directly from the frame buffer via a VGA interface to a corresponding output device. Otherwise, appropriate compression algorithms and archiving methods can be implemented as required. The input unit usefully encapsulates the different input data streams (scalar values) through a normalized interface.

As a technical basis, a graphics computer is used, which supports video textures. In addition, the hardware must have a sufficiently fast bus and a sufficiently fast disk access to store the output video stream.

In addition to hardware support, the software interfaces also play a significant role in the selection of a system. In addition to the hardware support, various manufacturers offer the corresponding software support in the form of an extension of the standard OpenGL instruction set with commands for video texturing and image processing in hardware. These possibilities (also in the form of expansion cards) exist on practically all platforms. The most important differences are here regarding the resolution and the number of video streams that can be processed in real time.

Desirable hardware features are: support of DVE (Digital Video Editing), many image processing operations in hardware, such as Texture filtering, convolution, histogram creation, color matrix and linear color space conversion, lookup tables with up to 4096 color entries, atmospheric effects, e.g. Fog.

It is also important to consider the data rates. Since the pre-produced sequences are preferably 10-12 minutes long, they can not be held in RAM but must be loaded to process external media.

When using 1024 x 1280 pixels video image resolution results in an uncompressed data stream of 93.75 MB / s for a color depth of 8x8x8 bits, ie 16 million colors and 25 frames per second. Compared to 29.66 Mbytes / sec for uncompressed PAL, this is more than a factor of 3. If you use the quite usual 4: 2: 2 coding for the colors, you get PAL at 19.78 Mbytes / sec and at a resolution from 1024 x 1280 on considerable 62.5 Mbytes / sec. Without further compression, this data stream can therefore not be read from disk, if one of a max. Transfer rate of 40 Mbytes / sec for an ULTRA SCSI adapter. That is, in addition to all possible calculations, the video stream must be decompressed in real time. If several layers are to be mixed, then at the time of mixing the layers must be available as uncompressed video streams. Because of the high bandwidth required, preferably fast memory accesses of a shared memory architecture come into question here. A hardware bandwidth of 80OMB / sac peak performance is basically sufficient, for example, to load 5 video data streams from memory, merge them into one processor (when only performing very basic operations) and write them back to memory as a data stream.

For example, these operations can be implemented with realistic effort when using POSIX threads and the synchronization primitives available there. POSIX threads are supported by the IRIX operating system of Silicon Graphics Inc. (SGI). Since the necessary data transfer rates are very high at 5 layers and 1024 x 1280 resolution, a great deal of care must be taken when implementing software of any kind, since the synchronization effects of each instruction must be considered.

The loading of a (video data) stream can usually be carried out much more efficiently than that of multiple streams, for example repositioning the hard disk head occur less frequently, etc. Also other operations, such as reading the data on so-called "slave" computers and transfer to the "Master" calculators such as via Fast Ethemet remain in a similar range. To limit the bandwidth for video streaming, it is useful to compress the video data, depending on the hardware.

Claims (18)

1. Device for the procedural real-time manipulation of image data streams comprising an input unit (1) for receiving image data streams, which is connected to an image data source (2), and an output unit (3) for outputting image data, which is connected to the input unit (1) at least indirectly and is designed to output image data to at least one image display (4), comprising a processing unit (6; 9) having an input for receiving at least one temporally variable, external parameter derived from ambient variables originating from outside the device, such as the ambient temperature, for example, wherein the processing unit (6; 9) is connected to the input unit (1) and the output unit (3) and is designed to change image data received from the input unit (1) in a manner dependent on the parameters in the image parameters and to output them to the output unit (3), wherein the processing unit (6; 9) is designed in such a way that it changes the received image data by carrying out computational processes which determine the change in the image data by means of the parameters, wherein the device comprises an input unit (7) - connected to the processing unit - for determining the computational processes of the processing unit, and wherein the processing unit (6) comprises a mask recognition unit (10), which ensures that partial regions to be processed of the image data received from the input unit (1) are selected, and wherein the mask recognition unit (10) is connected to the input unit (1) and, on the basis of predetermined mask recognition methods, selects the image data to be processed from the incoming image data stream and communicates them to a mask processing unit (11), and wherein the mask processing unit (11) receives temporally variable parameters alongside the image data to be processed and calculates new image data depending on the received image data and the received parameters, and wherein the processing unit (6; 9) is designed to change the masks depending on the parameters, such that the masks themselves experience a procedural animation.
2. Device according to Claim 1, characterized in that the processing unit (6; 9) is designed to continuously change the image data received from the input unit (1) depending on the parameters in such a way that even an abrupt change in the parameter brings about a gradual change in the image data, and to output them to the output unit (3).
3. Device according to Claim 2, characterized in that the processing unit (6; 9) is designed to continuously change the image data received from the input unit (1) by transformation proceeding from predetermined image information.
4. Device according to Claim 1, characterized in that the processing unit (6; 9) is designed to change the image data received from the input unit (1) by generating new, artificial image information.
5. Device according to Claim 1, characterized in that the processing unit (6; 9) is designed to change the received image data pixel by pixel.
6. Device according to Claim 1, characterized in that the parameters represent events taking place outside the device.
7. Device according to any of the preceding claims, characterized in that the processing unit (6; 9) is configured in such a way that it changes the received image data in real time.
8. Device according to any of the preceding claims, characterized in that the processing unit (6; 9) is configured in such a way that it changes image parameters of the received image data such as colour, brightness, contrast, gradation, sharpness.
9. Device according to any of the preceding claims, characterized in that the processing unit (6; 9) is configured in such a way that it changes the sequential speed of image sequences, the speed of image transformations, form profiles, positioning and extent of three-dimensional bodies, textures of image objects, combinations - predetermined by stylizations - of parameters determining the appearance of image objects, a depth representation, or image objects in the form of geometrical effects.
10. Device according to any of the preceding claims, in particular according to Claim 3, characterized in that the processing unit (6; 9) is designed to change a received image data by superimposition with previously stored image data within image regions defined by masks.
11. Device according to any of the preceding claims, characterized in that the output unit (3) is designed to output the changed image data to the image displays (4) via a network such as the Internet, television picture transmitters or computer networks.
12. Device according to any of the preceding claims, characterized by a detector (5) connected to the input of the processing unit (6; 9) and serving for measuring the image-influencing temporally variable parameters.
13. Device according to Claim 12, characterized in that the detector (5) is designed such that it cannot be influenced in its state by a viewer of the image data output to the output unit (3).
14. Device according to Claim 12 or 13, characterized by a control unit (8), which is connected to the detector (5) and the output unit (3) and is suitable for controlling the detector (5), wherein the control unit (8) receives image data from the output unit (3) and controls the detector (5) depending on the image data received from the output unit (3).
15. Device according to Claim 1, characterized in that the input unit (7) is designed to determine the computational processes on the basis of an algorithmic rule system.
16. Device according to Claim 1, characterized in that the input unit (7) has a user interface that visualizes the computational processes of the processing unit (6; 9).
17. Device according to Claim 1, characterized in that the processing unit (6; 9) is designed to simulate the movement of liquids, gases, particles, physical particles or solid bodies by means of the computation processes.
18. Device according to Claim 17, characterized in that the processing unit (6; 9) is designed to simulate the movement of flowing substances by means of the Navier-Stokes equation.

Images