DE10125309C1 - Method and arrangement for controlling audiovisual media content - Google Patents

Abstract

The invention relates to a method and an arrangement for controlling the playback of media content, in particular video, sound, music and / or voice signals, in information or entertainment systems supported by at least one computer with peripheral devices and display, playback and sound devices , in which the physical presence of at least one user is determined by at least one interactive input and the kinematic behavior of the user, in particular his dwell time, speed and / or acceleration, is determined, this behavior triggering at least one visual display and / or sound system or is ended. DOLLAR A The invention has for its object to provide a method and an arrangement of the type mentioned, with which it is possible to chain different parts of a time-linear media content, such as a sound film, independent of time and play back in real time according to the user's interest. DOLLAR A This problem is solved in that at least one digitizable audio and / or image information is used as a data container and in its representation at the same time as a user interface, and that the information is broken down into virtual segments and the segments are divided into scenes and sub-scenes, and that from meta information linked to the segments and stored in the background, the information of these scenes regardless of their

Description

The invention relates to a method for controlling audiovisual content, especially video, sound, music and / or voice signals in from at least one computer peripheral devices and display, playback and Sound devices support information and / or Entertainment systems where the physical present at least one user by at least one interactive Input determined and the kinematic behavior of the User, especially his residence time, speed and / or acceleration, is determined, with this Behavior at least one visual display and / or Sound system triggered, controlled and ended becomes.

The invention further relates to an arrangement for controlling the Playback of audiovisual content, especially video, Sound, music and / or speech signals, with at least one Input means for recording the presence and determining the behavior of at least one user, a computer to Processing of the detected signals, and display, Playback and sound equipment.  

From DE 196 53 682 A1 and DE 196 54 944 A1 is one Control for a system, in particular for a computer or a room with sensor means for detecting a Position of at least one trigger, in particular one Cursors on the display of the computer or a person in the Space, in a 1-, 2- or 3-dimensional area (Sensor area) of the system known. By the trigger is at least one function of the system, in particular one optical display and / or sound system, can be activated. There are Means for recording the dwell time of the at least one Trigger in the sensor area, the dwell time Information can be used to influence the system.

Furthermore, DE 195 38 436 A1 describes a device for Use during dance events revealed at which by image evaluation the position and / or the movement of the Dancing is determined. With the determined data controlled visual and / or acoustic events.

US 5 012 334 A describes a video image bank for Storage of a large number of moving video image sequences and for automatic access to one or more of the saved image sequences. It is video storage means that contain a library of the moving video image sequences, Library index media, which is an index of Video storage means include, access means to Library index means for automatic identification an initial sentence of the image sequences, display means for Display of a motionless single image for each moving Video image sequence of the first image set, means for selection one of the video sequences from the first image set is available.  

In US 5 220 648 A is a high speed research system for an image data storage system proposed that essentially a storage means for the sequential storage of a variety of images, first Control means, selection means, research means, Display means and second control means comprises. Each of the Images is provided by research information data and one Image number identified.

Furthermore, US Pat. No. 4,538,188 A describes a method and a Device for reading segments from one Image source material stored on at least one storage medium is saved.

All of these known solutions use conventional ones separate databases in which, for example, programs for Control of heating, lighting or sound systems, Games, multimedia applications or image and Research system data are stored permanently. After this by input, be it by direct command input using a mouse or by a non-contact activity of a User, for example the expiration of a selected one Commercials has started, the user can play no longer influence within the film presentation and only has the option to stop movie playback if this no longer interests him. In all this is omitted Rule because the viewer is concerned about the operating area Disinterested long ago. The selected program part continues and provides for the next viewer often one, at least psychological, Hurdle because this is the previously selected program level must go back to one supposed for him to be able to select an interesting new part of the program. The  So viewers easily lose that from the start Interested in the operation of such Information system and therefore ignore it.

EP 0 871 177 A2 describes a spatial, non-temporal and nonlinear multimedia method, in which in multimedia materials stored in a memory as objects shown, one or more object (s) from memory for Display selected on a console, viewed dynamically is or will be and at least one of these objects is edited on the console to create an edited object generate and then the edited objects on a Machining area of the console can be placed around a To create action sequence that is visible on the console becomes.

Furthermore, a video system is described in US Pat. No. 5,434,678 described in which a direct access to Video technologies and video architectures for the automated selective search of non-sequential parallel saved transitions and overlaps of video segments is integrated from a simple variable program source.

The previously known information and Entertainment systems are therefore only limited for the control the reproduction of audiovisual content such as advertising, Information and training films suitable because the films run in a linear manner and the amount of data to be transferred is very high get big to the places of the film with interesting content. A consideration the user does not respond to the content.

In this prior art, the object of the invention based on a method and an arrangement of the beginning specified type, with which it is possible to differentiate  Parts of a time-linear media content, for example, of a sound film, to chain independently of time and in real time according to user interests play.

This task is accomplished by a method of the type mentioned at the beginning Genus with the characterizing features of claim 1 and by an arrangement with the characteristic features of claim 19 solved.

Advantageous refinements of the method are Removable subclaims.

The method according to the invention is particularly noteworthy in that it becomes possible to play back audiovisual content running in a linear fashion, detached from their timeline through the activity of at least one user control interactively. The audiovisual content, for example that of a sound film, is by the The method according to the invention is so made up and prepared that he can move away from his timeline. To The digital sound film is divided into individual segments at least a video frame, encoded with meta information and the individual segments are linked together so loosely that from the combination of in the meta information drawn paths and received user input sensible presentation stimulating further input is generated.

By coding the individual segments, the user is no longer forced to use individual different image sequences select, but the inventive method interprets the behavior of the user and presents it to the user individual users as well as a group of users scene or sequence of scenes appropriate to behavior.  

The inventive method has the advantage that it Presentation of a variety of media formats such as writing, Image, sound, video formats, VR panoramas, animations and TV shows supported.

By determining the statistical or empirical Audience behavior it opens up the possibility of To create program formats that automatically re edit, d. H. an ongoing television program of any length constantly renews itself consciously by proven segments placed and less popular replaced by others. The method according to the invention has the particular advantage to react directly to spectator habits and one Qualify broadcast or playback.

In addition, the automatically reediting allows Presentation of media content by the invention Process music tracks using interactive video clips and Banner advertising through influenceable video sequences replace. The video clips or video sequences do not have to can be reproduced in an identical form, but can depending on the user's response be put together.

Some possible uses of the method according to the invention are listed below by way of example, the order of which does not constitute a ranking:

• - Use in a museum as an information system for various fields of knowledge;
• - entertainment and information in public spaces, for example airport terminals, foyers, etc .;
• - navigation systems in public transport;
• - Use as an audiovisual musical instrument;
• - Use in presentation and advertising, for example Display boards, spots, clips, banner advertising on the Internet etc., for sales rooms, trade fairs, public events;  
• - Multimedia applications, for example interactive audiovisual shows,
• - Applications for theater and opera stages;
• - security systems.

Further advantages and details emerge from the following description with reference to the attached drawings.

The invention is intended to be based on an exemplary embodiment are explained in more detail.

Show it:

Fig. 1 is a diagram of the method according to the invention,

Fig. 2 is a diagram of the arrangement according to the invention with representation of the signal processing and

Fig. 3 shows the working steps according to the inventive method.

Figs. 1 and 2 illustrate the sequence of the method according to the invention and a variant of an arrangement for carrying out the method according to the invention with a schematic representation of the signal flow. The arrangement essentially consists of a commercially available fast computer 1 with a hard disk memory, an optional network 2 assigned to the computer 1 , an audio / video playback device 7 , a display 8 and a sound system 9 . Sensors 10 that operate in a contactless manner are positioned near or on the display 8 . The arrangement according to the invention optionally includes further sensors 11 , at least one camera 12 and conventional input means 13 such as a mouse, touchscreen or keyboards.

In this example, the method works with a digitized, time-linear audio film, which is stored on the hard disk memory of the computer 1 for fast data access. Of course, it is also possible to feed the digital sound film via the external network 2 . In addition to image data 4 and sound data 5 , the digitized sound film also contains information 3 on the playback behavior which is necessary for the time-linear playback.

In order to be able to use the digitized sound film for the method according to the invention, the sound film is broken down into virtual segments 6 , ie the film product finished in the conventional sense is re-edited. The segmentation according to the method according to the invention is explained below with reference to FIG. 1.

The original film is in the form of a digitized sound film file before, whose starting point a over time to the end point g is shown.

The virtual segment 6 is determined by a variable number of digitized image and sound frames of the time-linear sound film, which are marked as belonging together via index information. A segment comprises at least one video frame and is not limited in its length, that is to say the total number of frames belonging to a segment.

The segment is defined by its first and last frame. On the storage side, the segment can be part of a higher-level file, for example the film file, and does not have to be a separate file. The segment is identified with meta information 3 , ie encoded by individual byte structures representing the digital content within the digital video frame.

The scene is arranged hierarchically above the segments, which is a summary of segments. At least one segment is needed to make a scene define their length, i.e. the number of segments, again need not be limited.

The scene defines the initial playback behavior of the Segments and the mode of user interaction. Consists for example a scene from more than one segment, so the transitions between the segments are available as fixed points Available. With a visual representation of the media content these transitions do not have to be visible.

The corresponding meta information of the scene such as initial playback specifications, definition of parameters that can be influenced by the user, connection scenes, associated sound samples etc. are encoded in the media data. The method according to the invention reads these out when jumping into a scene and reconditions its input and playback modules based on this information. Once this conditioning has taken place, the method according to the invention is ready to have parameters manipulated in real time by user input. Both the type of input, for example in the form of a contactless sensor 10 or direct input using a keyboard, as well as the manipulable parameters such. B. playback speed, immediate film position, direction, volume, scene-specific parameters can be defined separately for each scene.

The segment boundaries serve as possible connection points for Subsequent scenes, definable areas around the segment boundaries can release certain decision phases. The selection  is taken depending on the scene depending on the definition.

A special form of the segment is the breakpoint (see FIG. 1), which defines one without media content and depends on the user input, on a random basis or by other factors such as e.g. B. determines the time of day, allows a multiplication of the connection scenes.

The segmentation takes place in several steps. First the film is divided into scenes and these in turn Sub-scenes or segments. The segments so defined will be addressed via indices so that their start and end points are available as parameters.

The film is therefore as small as possible Can be broken down into segments. The individual pictures and Sound segments are thus available if necessary continuous addressing for immediate access available regardless of their order.

After segmentation and addressing, the film is available as a numbered image / sound sequence and also as a data container 21 , in which, in addition to the combination of individual segments, the direction and speed of play can also be freely adjusted.

To play back the scenes or sub-scenes (pictures and sounds), the computer-based video / audio playback device 7 is used, which allows the scenes or sub-scenes to be played back at a frame rate of up to 50 frames per second, regardless of the context, direction and speed.

Conventional monitors, displays or video projectors 8 are used to display the video images. The sounds are made audible to the user with the sound system 9 consisting of audio amplifier and loudspeaker.

At least one sensor 10 is positioned on or near the display 8 , which detects the presence of at least one user and detects its activity in the detected area.

Depending on the sensor type and measurement method, the range is about 1.5 to 100 m with an opening angle of up to 180 °. Optical and / or acoustic sensors are used, preferably infrared, ultrasonic or laser sensors, Microphones, cameras, laser scanners, microwave systems or Radar systems.

Each sensor can of course also be a simple one Rangefinder be executed. The distance measurements are at a time interval of 1 to 200 Performed milliseconds. Different types of sensors too combining is also possible.

A set of rules 14 is stored in the form of codes 15 on the data storage medium of the computer 1 and carries out the work steps of the inventive method determined by the code 15 . The set of rules 14 forms the link between the data streams generated by the user and the correspondingly prepared digital sound film. It provides an expandable collection of specific video playback and user data processing functions, which can be linked to different program sequences or variations using controllable variables (variables).

The segmentation data, navigation path markings and addresses stored in the segments form the framework parameters for the playback behavior, on the basis of which the control system 14 constantly adapts a constellation to the current state and prepares the subsequent steps resulting therefrom.

Corresponding behavior patterns of average users are stored in the data memory of the computer 1 and are used to determine deviations in the determined behavior pattern of the users by comparing the stored characteristic reference behavior and the determined behavior pattern. If the set of rules 14 recognizes that new characteristic properties appear in the behavior of the user, the characteristic reference behavior is expanded by these properties and the change is recorded in the data memory.

The rule set 14 interprets the determined behavior pattern of the users by comparing them with the characteristic reference behavior as soon as a threshold value that exceeds the behavior properties is determined.

The detectable behavior pattern includes the position that Speed of movement, the direction of movement, the Movement intensity, the movement path, the movement pattern, repetition of movements as well as gestures and Facial expressions of one or more users.

The respective subsequent step is then selected by a selection of virtual segments or scenes or Sub-scenes according to the previously described interpretation and the prevailing state of an ongoing Presentation of segments.

With the representation and perception of the display of the segments or the sound and video frames close the loop and the user is again prompted to take action to generate a new system reaction.  

The rule set 14 - as shown schematically in FIG. 3 - carries out the following individual work steps.

As soon as sensors 10 and 11 detect a user in their detection area, contactless object detection begins with distance measurements and other measurements that characterize user behavior. Feedback control loops continuously adapt the sensor area relevant for the application to the user's radius of movement. The movement of the user is determined over an adjustable period of time. Depending on the result of this determination, the set of rules 14 adapts the threshold values necessary for triggering certain subsequent steps to the visitor behavior. This enables, for example, a calm visitor with slight natural head, hand or body movements to be shown the same media event as a group of strongly moving children.

Through this dynamic adjustment of the sensor sensitivity the user behavior is also possible to attract visitors bring out critical sensor boundary areas by in A more attractive direction towards the non-critical area content offer is placed.

The control system 14 reads out the sensor signals present on the input side, depending on the sensors used, between 5 and 1000 Hz. Assuming this supporting sensor hardware, the readout frequency of the sensors can be changed dynamically on the hardware side but also on the software side.

With an input filter 16, the control system 14 filters out certain signals or data that are irrelevant or disruptive to the execution of the method according to the invention. This includes noise caused by technology or the environment, fluttering or trembling of user data, strong short-term pulse jumps and pulses above, below and / or between certain threshold values.

By comparing values measured in quick succession In addition to determining the distance, information about the Direction of movement, speed, acceleration and Length of stay of the visitor won.

The sensor evaluation data are over a certain period of time recorded and with or according to the reference behavior pattern examined. Recognizing such a pattern triggers for example an operating step in the segment display out.

The recorded sensor evaluation data are also included the data from another measurement period or the current incoming data compared. That determined Comparison result can, provided there is a reference behavior corresponds to an operating step in the Trigger segment display.

The recorded control data are classified by the control system 14 on the basis of adjustable threshold values as unambiguous control commands and, depending on the application requirements, any number of steps such as, for example, B. "Standstill short", "Standstill medium", "Standstill long", "slight movement forward", "slight movement backwards", "slight movement left", "slight movement right", "strong movement forward", "strong backward movement", "strong left movement" or "strong right movement". Of course, it is also possible to combine these steps accordingly.

In addition to the step distribution of the control data explained above, the data stream generated by the user movement is alternatively or in parallel used to directly control the sound film files or segments. The control system 14 smoothes and modifies the data streams in order to achieve smooth playback of the segments. The modifications can relate, for example, to the reversal of the direction of the data and / or their mixing with other generated data.

These other data can be prepared control sequences 17 or enrichment data 18 . The prepared control sequences 17 ensure that the interaction remains varied, interesting and meaningful for the user. Repetitions within a preset period of time are excluded, or scenes appear different by changing the playback speed, the entry and exit points for the scene, color effects and sounds, even if the same segments are used.

The set of rules 14 determines that the media presentation of the segments or scenes remains in flow and that there are no abrupt transitions between the segments. This is achieved in that only certain connection points or connection areas of the segments allow a scene change.

If the user requests a change and the segments are not in the vicinity of a connection point or area at this moment, the control system 14 ensures that the corresponding connection point is quickly but gently approached before switching to the appropriate segment. Furthermore, real-time media effects such as cross-fades or prepared intermediate films can be inserted at the switching points, which enable a smooth transition.

By means of the camera 12 , video images or sounds recorded on site or recorded live can be integrated into the presentation sequence of the segments, scenes or sub-scenes by means of the rules 14 . For example, an additional increase in the entertainment effect is achieved when the image of the respective user appears in the presentation sequence.

For the integration of live recordings, certain segments (breakpoint) are reserved and addressed in advance in the sound film described as a data container. This enables the rules 14 , the real-time call and the program-specific modulation of the live images and sounds in the same way as the pre-produced segments.

The control commands of the user signals categorized via the threshold values are compared in a real-time matrix 19 with the segment that is currently playing. The next step is generated by taking both factors into account. Such a procedure ensures that the user activity and the respective state of the media presentation is always responsible for the respective subsequent step and never only one of these two factors alone.

Preset control mechanisms, which act like an optimization filter 20 , optimize the system performance in that only the required part of the generated data is used for control, even with high user activity.

The frame rate is matched to human perception, ie the method according to the invention works with a frame rate of approximately 15 frames per second, because humans can no longer perceive a frame rate of more than 15 frames per second individually. If a user provokes a high playback speed through his activity, then the set of rules 14 slows the audiovisual presentation down to viewing habits which the person can perceive without being overloaded.

Switching between non-contact sensors and conventional input tools are based on content dependent in the meta information Information, the user behavior determined in each case and the state of the system. With one on the input side Temporary shutdown of the sensors is also possible to relieve the system.  

List of the reference symbols used

1

computer

2

Optional external network

3

Playback information, metadata

4

image data

5

sound data

6

Segments of the sound film

7

Audio / video playback device

8th

Display, monitor, projector

9

sound system

10

sensors

11

Optional additional sensors

12

Optional camera

13

Commercial input devices

14

regulations

15

code

16

input filters

17

Prepared control sequences

18

enrichment data

19

Real Time Matrix

20

optimization filter

21

container film

Claims (23)

1. A method for controlling the playback of media content, in particular video, sound, music and / or voice signals, in at least one computer with peripheral devices and display, playback and sound devices support information or entertainment systems in which the physical presence of at least one user is determined by at least one interactive input and the kinematic behavior of the user, in particular his residence time, speed and / or acceleration, is determined, with this behavior triggering, controlling and ending at least one visual display and / or sound system is characterized in that at least one digitizable audio and / or image information is used as a data container and in its representation at the same time as a user interface, and that the information is broken down into virtual segments and the segments are divided into scenes and sub-scenes, and that with the segments linked meta information stored in the background, the information of these scenes is re-edited, re-chained and reproduced in real time regardless of their original relationships, directions and speeds as a function of a change in the determined behavior of the user. 2. The method according to claim 1, characterized characterized that the virtual segment from is composed of at least one video frame.   3. The method according to claim 1, characterized characterized that sound films, video films, VR Panoramas or television programs as digitisable Information is used. 4. The method according to claim 1 to 3, characterized by the following steps occurring before the playback:

• a) detecting the change in the behavior of the at least one user, preferably the direction of movement, movement intensity, movement path, movement pattern and repetition of movements, by means of the at least one input with an associated storage medium in which a reference behavior characteristic of the behavior of an average user is digitally stored;
• b) determining the deviations in the determined behavior pattern of the user by comparing the stored characteristic reference behavior and the determined behavior pattern;
• c) extension of the characteristic reference behavior of the average user by new determined properties, provided that significant deviations are identified by the comparison;
• d) interpretation of the determined deviations of step b) by a set of rules stored in the storage medium as soon as a threshold value exceeding the behavioral properties is determined;
• e) Selection of the virtual segments or scenes or sub-scenes according to the interpretation after step d)
• f) triggering a command for playback in a decision matrix weighing the respective state of an ongoing presentation and the determined behavior of the user.

5. The method according to claim 1 to 4, characterized characterized that as meta information those on playback behavior, on subsequent steps, Image / sound links, media / user links and for Identification can be used. 6. The method according to claim 1 to 5, characterized characterized that the meta information be stored within a video frame. 7. The method according to claim 1 to 6, characterized characterized that certain segments Connection points or connection areas for one Scene changes can be assigned. 8. The method according to claim 7, characterized marked that at the connection points or connection areas for an abrupt transition from one Scene to another an accelerated start of the Connection point is made, intermediate films or real-time media effects, preferably fades, be inserted. 9. The method according to claim 1 to 6, characterized characterized that individual segments by a pre-addressing for the reproduction of foreign content be reserved. 10. The method according to claim 9, characterized characterized that as foreign content on site recorded or live recorded video images or sounds be used.   11. The method according to claim 1 to 10, characterized characterized that the rules are constantly with the attached meta information of the segments to the determined behavior patterns is automatically adjusted. 12. The method according to claim 4, characterized in that the following sequential work steps are carried out with the set of rules:

• Reading out the signals detected by the at least one input with an adjustable sampling frequency corresponding to the type of the respective input,
• - filtering out disturbing and / or irrelevant signals,
• - recording the relevant signals over a certain period of time,
• Comparison of signals recorded in quick succession to determine the position, direction of movement, speed, acceleration and length of time of the user in the sensor area,
• Comparing the recorded signals with those of another measurement period or with the behavior pattern of the average user to trigger the playback of segments,
• - Converting the sensor signals into control commands for qualification and categorization into switching commands and for simultaneous direct control of the playback of segments.

13. The method according to claim 12, characterized characterized that the sampling frequency is changed dynamically depending on the input data.   14. The method according to claim 1 to 13 thereby characterized that the at least one Interactive input without contact using sensors is carried out. 15. The method according to claim 14, characterized characterized that as optical and / or acoustic sensors, preferably infrared, ultrasonic or laser sensors, microphones, cameras, laser scanners, Microwave systems or radar systems are used. 16. The method according to claim 1 to 14, characterized characterized that the at least one interactive input by activating a touchscreen or a mouse is performed. 17. The method according to claim 14 to 16, characterized marked that the contactless input combined with the touching input. 18. The method according to claim 1, characterized characterized that in virtual segments decomposed digitized information in its original Form remains intact and can be reproduced. 19. Arrangement for performing the method according to claim 1 to 18, with at least one input means for detecting the presence and behavior of at least one user, a computer connected to the input means with peripheral devices for processing the signals and the output signals and display, playback and sound devices, characterized in that at least one data container ( 21 ) of digital audio and / or image information containing virtual information segments is provided, which is provided with a computer-based audio / video device ( 7 ) for the context, direction and speed-independent reproduction of the information segments via a set of rules ( 14 ) is connected to control playback depending on user behavior. 20. The arrangement according to claim 19, characterized in that the input means is at least one non-contact detecting sensor ( 10 ). 21. The arrangement according to claim 19, characterized in that the sensor ( 10 ) is an optical and / or acoustic detector, preferably infrared, ultrasound or laser detector, microphones, a camera, a microwave system or a radar system. 22. The arrangement according to claim 19, characterized in that the at least one sensor ( 10 ) is a laser scanner with an opening angle of 180 ° and a range of at least 1.0 m to a maximum of 100 m. 23. The arrangement according to claim 19, characterized in that the at least one sensor ( 13 ) is a touchscreen or a mouse.