DE102021126741A1 - Process for the aggregated output of media content - Google Patents

Abstract

The invention relates to a method for the aggregated output of media content. By analyzing media content from various sources, the method according to the invention enables the recognition, acquisition and comparison of clusters within this media content, the indexing of common and/or overlapping media content, the selection and aggregation of individual content (E) and the generation of an output of the aggregated media content (F) an automated adaptation of different media content to the needs of individual users or consumers. The invention also relates to a computer program and a system for carrying out the method.

Description

Background of the Invention

The invention relates to a method for the aggregated output of media content. The invention also relates to a computer program and a system for carrying out the method.

State of the art

When consuming media content (e.g. news, podcasts, audio and video streams), users are often trapped in a certain window of time because the media content has a fixed length. For example, a cinema film lasts at least 90 minutes and news programs often around 30 minutes, although the duration of the film or program cannot be changed by the consumer. Sometimes, however, a user does not have the entire time window available, but only 80 minutes for a film or 20 minutes for the news, but still wants to consume the film or the news. In such cases, the user has hitherto only been able to either interrupt the playback of the media content before it actually ends or to skip certain sections that are considered rather uninteresting.

In addition, when consuming media content, it often turns out that there is duplication or the reproduction of unnecessary information. News from different sources often contain the same topics, with the differences usually only being marginal. This also applies to documentaries on a specific topic, for example, since a lot of information is duplicated and the unique information is often not worth watching a whole documentary over, say, 90 minutes again. Even when creating trailers for films, a lot of content has to be compressed for a short time, which is currently still mostly done manually in laborious detail work. The work mentioned is still usually carried out by people (e.g. editors and editors). However, this is time-consuming and labour-intensive and usually does not allow the consumer's wishes and specifications to be taken into account.

Description of the invention

The object of the invention is to provide a method and a computer program that executes this method, which enable media content to be adapted to the needs of individual users or consumers quickly and in a manner that saves on personnel.

1. A) Auswählen mindestens eines Quelldatensatzes, der Nutzdaten umfasst, die zumindest bildlich, akustisch und/oder schriftlich wiedergebbare Medieninhalte umfassen;
2. B) Analysieren der Medieninhalte und Erfassung von Clustern innerhalb dieser Medieninhalte;
3. C) Vergleichen der Cluster und Ermitteln von mindestens einem Gesamtcluster, das gemeinsame und/oder sich überschneidende Medieninhalte umfasst;
4. D) Verschlagwortung des Gesamtclusters und Erzeugung von diesem Gesamtcluster zugeordneten Metadaten und/oder Schlagwörtern und Speichern dieser Metadaten und/oder Schlagwörter des Gesamtclusters in mindestens einer Datenbank;
5. E) Auswählen von Gesamtclustern aus der Datenbank anhand der Metadaten und/oder Schlagwörter und Zusammenstellen der ausgewählten Gesamtcluster zur Erzeugung mindestens eines Medieninhaltsaggregats;
6. F) Erstellen mindestens eines Zieldatensatzes, der Nutzdaten umfasst, die das Medieninhaltsaggregat mit zumindest bildlich, akustisch und/oder schriftlich wiedergebbaren Medieninhalten umfassen; und
7. G) Übertragen des Zieldatensatzes an mindestens ein Endgerät zur Wiedergabe des Medieninhaltsaggregats.

According to the invention, the object is achieved by a computer-implemented method, which includes the following steps:

1. A) Selecting at least one source data set that includes user data that includes media content that can be reproduced at least visually, acoustically and/or in writing;
2. B) analyzing the media content and detecting clusters within this media content;
3. C) comparing the clusters and determining at least one overall cluster that includes common and/or overlapping media content;
4. D) Indexing of the overall cluster and generation of metadata and/or keywords assigned to this overall cluster and storage of this metadata and/or keywords of the overall cluster in at least one database;
5. E) Selecting overall clusters from the database using the metadata and/or keywords and assembling the selected overall clusters to generate at least one media content aggregate;
6. F) Creation of at least one target data record that includes user data that includes the media content aggregate with media content that can be reproduced at least visually, acoustically and/or in writing; and
7. G) Transmission of the target data set to at least one terminal for playback of the media content aggregate.

By means of the method according to the invention, an adaptation or packaging of media content to the needs of individual users or consumers, which would currently still have to be carried out in laborious detailed work, can be automated and thus carried out quickly and in a personnel-saving manner. According to the invention, a method or system is made available that enables the automated aggregation of media content based on user settings or specifications, with one or more sources for a specific content (topic) being able to be processed based on specific requirements. For this purpose, the media content from one or more (data) sources is first analyzed, for example with regard to content, audio sequences and video scenes, and then categorized and prioritized. Subsequently, selected content is then summarized (aggregated) based on specified requirements or specifications and finally output to the user or consumer. For example, "News from Germany in 5 minutes" (= Nutz requirement) from different sources are analyzed in terms of content, duplications are removed and only the unique information from all sources is summarized. If the initially generated aggregate exceeds the time frame of 5 minutes, the messages with the highest priority, for example those that occur most frequently in all sources, are used first. This advantageously saves a lot of consumption time with the same information content. By analyzing media content from different sources, the method according to the invention thus makes it possible to identify, record and compare clusters within this media content, index common and/or overlapping media content, select and aggregate individual content and generate an output of the aggregated media content an automated adaptation of different media content to the needs of individual users or consumers.

In an advantageous embodiment of the invention, external data sources, preferably publicly available metadata sources (e.g. IMDB, Amazon ratings, Gracenote for music, etc.), can be used to enrich the media content in step A). In this way, the number of source data records or pieces of information available for selection can be increased significantly and in a meaningful way.

• B1) Aufteilen der Nutzdaten durch Trennen der bildlich, akustisch und/oder schriftlich wiedergebbaren Medieninhalte; und
• B2) Getrenntes Analysieren der unterschiedlichen Medieninhalte und Erfassung von Bild-/Video-, Ton- und/oder Text-Clustern innerhalb der jeweiligen Medieninhalte.

Es wird also ein „Inhalts-Splitting“ durchgeführt, so dass die unterschiedlichen Medieninhalte bzw. Dateiformate (z.B. Video, Sound, Untertitel) getrennt analysiert und „geclustert“ werden können. Die Erkennung und Erfassung der Medieninhalt-Cluster kann dabei beispielsweise anhand der folgenden Parameter bzw. Analyseverfahren datenspezifisch durchgeführt werden:

• - „Video-Clustering“: Farbänderungen, Helligkeit, Geschwindigkeit der Bewegung, Inhalts- bzw. Objekterkennung, Gesichtserkennung und/oder „Setting“- bzw. Szene-Erkennung;
• - „Sound-Clustering“: Musik, Amplitude, Frequenz-Erkennung, Stimmen-Segmentierung, Sprechererkennung, Geschlechtererkennung und/oder Stim m ungsanalyse;
• - „Untertitel-Clustering“: Dialoganalyse, Spracherkennung (z.B. „Natural Language Understanding“ (NLU), Inhaltserkennung) und/oder Wortkomplexitätsbestimmung.

In a further advantageous embodiment of the invention, it is provided that step B) comprises the following steps:

• B1) splitting up the user data by separating the media content that can be reproduced visually, acoustically and/or in writing; and
• B2) Separate analysis of the different media content and acquisition of image/video, sound and/or text clusters within the respective media content.

A "content splitting" is carried out so that the different media content or file formats (e.g. video, sound, subtitles) can be analyzed and "clustered" separately. The detection and recording of the media content clusters can be carried out in a data-specific manner, for example, using the following parameters or analysis methods:

• - "Video clustering": color changes, brightness, speed of movement, content or object recognition, face recognition and/or "setting" or scene recognition;
• - “Sound Clustering”: music, amplitude, frequency detection, voice segmentation, speaker detection, gender detection and/or mood analysis;
• - “Subtitle clustering”: dialogue analysis, speech recognition (e.g. “Natural Language Understanding” (NLU), content recognition) and/or word complexity determination.

Step C) can advantageously include comparing the clusters of the different media content, so that common and/or overlapping media content can be determined in video, voice and text files, for example, in order to generate an overall cluster that contains the common and/or or overlapping media content of the different media or file formats (content reproduced visually, acoustically and/or in writing).

In a further advantageous embodiment of the invention, it is also provided that in step C) specific patterns in the clusters are recognized and compared with one another with the aid of at least one algorithm, preferably an algorithm for machine learning (“Machine Learning Algorithm”), with the at least one overall cluster is identified based on common and/or overlapping patterns.

In a further advantageous embodiment of the invention, it is provided that the selection of overall clusters from the database in step E) is at least partially based on at least one specification that is selected from the group consisting of at least one user specification, at least one user-specific filter setting, at least one preset restriction or preference and at least one recommendation filter. In this way, the selection of the media content can be specifically adapted to the wishes, needs and specifications of the user or consumer on the one hand and to legal restrictions and/or restrictions chosen by the user on the other hand.

In a further advantageous embodiment of the invention, it is provided that the creation of at least one target data record in step F) includes the generation of the media content to be output based on at least one criterion that is selected from the group consisting of the time length of the playback (time-specific selection of the individual cluster pieces , in order to adhere to the duration of content playback specified by the user (e.g. 10 min.) as precisely as possible), the weighting and selection of the individual clusters based on user preferences (based on the cluster keywording, a decision is made as to which cluster pieces are used. For this purpose, for example, a confidence value be determined, which indicates how high the rating for a certain metadata feature that was determined for a cluster piece) and the weighting based on at least one recommendation filter (e.g. cluster pieces that are more highly weighted or rated by other users can be prioritized over low or non-rated cluster pieces).

In a further advantageous embodiment of the invention, it is also provided that the media content in the target data record is time-compressed in comparison with the at least one source data record. Using the method according to the invention, for example, a cinema film with a length of 100 minutes or a news program with a length of 20 minutes can be reduced to a length of, for example, 60 minutes for the film or 10 minutes for the news program without parts that are important for the consumer or information is lost. The film or the news is specifically adapted based on the user's specifications, so that he/she can consume the parts or information that are important to him/her in the desired time.

The target data set can, for example, be in the form of at least one text, image, audio, video and/or other media file or can be converted into one. In this way, the target data record, which contains the media content or information requested by the user, can be displayed and/or played back by a device (e.g. smartphone, tablet or desktop PC or notebook) in a suitable manner. The user receives the processed information in a form that is easy for him/her to absorb, with all possible file formats being able to be used.

The object is also achieved by a computer program which comprises computer-executable instructions which, when the program is executed by a computer, cause the latter to execute the method according to the invention described above. The invention also includes a computer-readable storage medium, which includes computer-executable instructions which, when executed by a computer, cause the computer to carry out the method according to the invention. The invention also relates to a computer-readable data carrier on which the computer program according to the invention is stored.

1. a) Mittel zur Analyse und zum Vergleichen von Medieninhalten aus mindestens einem Quelldatensatz mittels mindestens eines in mindestens einer Datenverarbeitungseinrichtung implementierten Algorithmus';
2. b) Mittel zum Erzeugen von Metadaten und Speichern dieser Metadaten in mindestens einer Datenbank;
3. c) Mittel zum Auswählen und Aggregieren von Medieninhalten aus mindestens einem Inhaltsspeicher basierend auf den in der Datenbank gespeicherten Metadaten; und
4. d) Mittel zur Übertragung mindestens eines Zieldatensatzes über die Datenverbindung an das Endgerät.

The invention also includes an arrangement or a system for data processing, which includes means for carrying out the steps of the method according to the invention. The object is achieved in particular by a system for transmitting aggregated media content to at least one terminal device via at least one data connection, which includes the following means:

1. a) means for analyzing and comparing media content from at least one source data set using at least one algorithm implemented in at least one data processing device;
2. b) means for generating metadata and storing this metadata in at least one database;
3. c) means for selecting and aggregating media content from at least one content store based on the metadata stored in the database; and
4. d) Means for transmitting at least one target data record via the data connection to the terminal.

In an advantageous embodiment of the invention, it is provided that the source data set is processed and analyzed using at least one algorithm implemented in a central data processing device. A method or system is thus provided which, for example by means of cloud integration, uses additional algorithms which run on the central data processing device (server) in a computationally intensive and/or data-intensive manner. The computationally and data-intensive algorithms can be, for example, speech recognition, text recognition, image data evaluation, content analysis and the like. The automatic speech recognition systems can include, for example, acoustic models such as Hidden Markov Models, Natural Language Understanding (NLU) and/or neural networks. The automatic text recognition systems can include, for example, "optical character recognition" (OCR), "intelligent character recognition" (ICR) and/or "intelligent word recognition" (IWR). In addition, machine learning algorithms can be used.

The terminal can be, for example, a mobile telecommunications terminal (eg smartphone or tablet), a personal computer (eg desk or laptop) or a smart TV. At least one control device of the mobile telecommunications terminal is preferably coupled to at least one display device, which shows the user the media content and information contained in the target data record. This display can, for example, take place visually via a display and/or acoustically via a loudspeaker (eg as a voice message). The display can, for example, be part of the end device (e.g. smartphone, tablet, laptop) or a separate monitor. The transmission of the data from at least one control device of the terminal device to the display device can, for example, also take place wirelessly, for example by means of Bluetooth, WLAN, ZigBee, NFC, Wibree or WiMAX.

For the purposes of the invention, the term “user data” refers to data in a data record that does not contain any control or log information. User data within the meaning of the invention are, for example, text, characters, images and sounds.

In the context of the invention, the term “metadata” refers to structured data that contains information about characteristics, properties and technical parameters of other data, in particular user data, and provides corresponding control and/or protocol information.

For the purposes of the invention, the term “file format” describes the type of data and how this data is technically stored within a file. A file format thus defines e.g. the meaning and interpretation of data within a file, so that knowledge of the file format is essential for interpreting the information stored in a file. The file format is required for the interpretation of the information stored in a file, applications that can interpret this file being assigned to a file by the operating system using the file format. Most file formats are identified by a one to three character identifier (such as ".dat" or ".exe"). To determine the file format, metadata that exactly defines the file format (see above) is usually stored or transmitted together with the file. For example, file formats can be categorized by the type of content they contain (e.g., text, image, audio, and video formats).

In the context of the invention, the term “keywording” (also called “indexing”) refers to the creation and collection of keywords and metadata for user data (documents), i. H. the assignment of descriptors to a data set (document) in order to index the (media) content contained therein. Keywording can be used as controlled indexing, i. H. with a thesaurus or keyword catalogue, or free indexing, d. H. with unspecified descriptors.

• 1 zeigt ein Ablaufdiagramm zur Durchführung der ersten Schritte einer beispielhaften Ausführungsform des erfindungsgemäßen Verfahrens (Die Referenzen A) bis D) bezeichnen die jeweiligen Schritte des erfindungsgemäßen Verfahrens).
• 2 zeigt ein Ablaufdiagramm zur Durchführung der letzten Schritte einer beispielhaften Ausführungsform des erfindungsgemäßen Verfahrens (Die Referenzen E) und F) bezeichnen die jeweiligen Schritte des erfindungsgemäßen Verfahrens).

The invention is explained in more detail below by way of example with reference to the figures and embodiments described below.

• 1 shows a flowchart for carrying out the first steps of an exemplary embodiment of the method according to the invention (references A) to D) denote the respective steps of the method according to the invention).
• 2 shows a flowchart for carrying out the last steps of an exemplary embodiment of the method according to the invention (references E) and F) denote the respective steps of the method according to the invention).

Description of advantageous and exemplary embodiments of the invention

1) Analysis of media content (Figure 1):

First, source data sets are selected that include user data with media content that can be reproduced visually, acoustically, and/or in writing (step A). Different data sources (internal and/or external) can be accessed. To enrich the media content in step A), publicly available metadata sources (e.g. IMDB, Amazon ratings, Gracenote for music, etc.) can also be accessed.

A so-called "content splitting" is then carried out so that the different media content or file formats (e.g. video, sound, subtitles) can be analyzed separately and "clustered" (step B or steps B1 and B2). The detection and recording of the media content clusters can be carried out in a data-specific manner, for example, using the following parameters or analysis methods:

Video Cluster Detection:

metadata:

Overall brightness over time, change in brightness histogram, color values over time, color composition of a scene, detection of movement in a scene, speed of movement across the screen (distance/"frame"), and/or recognition of faces in a scene.

Description:

Video cluster detection is an iterative process that starts with coarse-grained clustering based on simple changes like color, brightness, etc. After the coarse analysis, another pass is made using content detection within the previously defined timeframes to determine more complex scene details. Face recognition and object recognition are used to determine what is happening in a time frame and whether the setting has changed (e.g. inside, outside, room, cars, bus, train etc.).

Sound Cluster Detection:

metadata:

Music detection, "beats" per minute, loudness, music genre, frequencies over time, speech detection, gender detection, natural language processing (NLP) for identification, content of a dialogue, scene content, word complexity assessment, and/or sentiment analysis of dialogue.

Description:

Audio cluster detection uses the audio track to determine what is happening in a scene. NLU is used to analyze speech content. Voices are analyzed to detect the number of speakers in a scene and their gender, including the speaker's mood. Directional audio can be used to determine relevant content from background content. Dialogue runs mainly through the center speaker. Scenes in which music is played can be specified. Music can be analyzed to identify scene type (e.g., action, romance, erotic).

Subtitle Cluster Detection:

metadata:

Dialog analysis, NLU / context, word complexity and/or language level rating.

Description:

When multimedia content contains subtitles, an algorithm analyzes the text for specific intent and context, as well as additional descriptions of the audio being played, such as laughter or other sounds. People with hair loss are shown to improve the viewing experience. In addition, the context of a dialogue between people is analyzed to determine the beginning and end of a cluster. Finally, a language level rating is applied to meet users' requirements in terms of language difficulty and complexity of the described topic for cluster generation.

The recognized clusters are then compared (cluster comparison or "cluster matching") and at least one overall cluster is determined that includes common and/or overlapping media content (step C):

Cluster matching:

The main goal of cluster matching is to find general patterns recognized by the various cluster analyzes and to generate common clusters (overall clusters) based on this.

Description:

• Kombinieren und Generieren einer „Urliste“ aus allen Clustern (Video, Ton, Untertitel), Berechnung des „Interquartilsabstand“ für jedes Cluster (Der Interquartilsabstand ist die Differenz zwischen dem oberen und dem unteren Quartil einer Stichprobe. Sortiert man eine Stichprobe der Größe nach, so gibt der Interquartilsabstand an, wie breit das Intervall ist, in dem die mittleren 50 % der Stichprobeelemente liegen.) und dynamische Anpassung von „Quantil“ für die Zählung von Clustern und Zeitrahmen. Der „Machine Learning“-Algorithmus wird verwendet, um Überschneidungen in den gegebenen Funktionen zu finden und einen gemeinsamen Zeitrahmen und Verschlagwortungen für ein Gesamtcluster zu finden.

Generate frequency distribution for all specified clusters:

• Combine and generate a "primary list" from all clusters (video, sound, subtitles), calculate the "interquartile range" for each cluster (The interquartile range is the difference between the upper and lower quartile of a sample. Sorting a sample by size, (e.g., the interquartile range indicates how wide the interval is in which the middle 50% of the sample elements fall.) and dynamically adjusts Quantile for counting clusters and time frames. The "machine learning" algorithm is used to find overlaps in the given functions and to find a common time frame and keywords for an overall cluster.

Requirements for a cluster: minimum/maximum duration of a cluster and/or overlap of at least x% of the specified metadata.

Content of metadata:

Time indexes, video type and audio content (genre, music, speed, content, people, complexity, volume, shots), scene index (shot change), day/night/weather, FSK rating for indexed areas, background -/Ambient Content and/or Main/Focus Content.

After the overall cluster has been indexed and metadata and/or keywords assigned to this overall cluster have been generated and this metadata and/or keywords of the overall cluster have been stored in at least one database (step D), the media content clusters processed in this way (overall cluster) are aggregated.

2) Aggregation of the tagged media content clusters (Figure 2):

aggregation:

• Nutzerpräferenzen: Basierend auf dem Nutzerprofil wie z.B. Alter, Geschlecht, Sprachniveau usw. werden die Unordnungen ausgewählt.
• Benutzerdefinierte Filter: Anpassung der Nutzereinstellungen und Inhaltsbeschreibung wie z.B. Genre, benötigte Dauer und Art der Szenen; die Cluster werden ausgewählt.
• Einschränkungen: GEMA, Geo-Location-Blocking, FSK und Sonstiges. Gesetzliche Beschränkungseinstellungen werden bei der Cluster-Generierung berücksichtigt.
• Empfehlungsfilter: Community-basierte Empfehlungen werden bei der Cluster-Generierung berücksichtigt.

Based on the following criteria, the identified media clusters are presented to the user or consumer and aggregated:

• User Preferences: Based on the user profile such as age, gender, language level, etc., the disorders are selected.
• Custom Filters: Customization of user settings and content description such as genre, required duration and type of scenes; the clusters are selected.
• Restrictions: GEMA, geo-location blocking, FSK and others. Legal restriction settings are taken into account during cluster generation.
• Recommendation filter: Community-based recommendations are taken into account during cluster generation.

The keyworded media content clusters (overall clusters) are selected from a database based on the metadata and/or keywords based on the user query and combined to generate a media content aggregate (step E). To generate the content output, a target data record is created from this, which includes user data that includes the media content aggregate with the media content that can be reproduced, for example, visually, acoustically and/or in writing (step F). This target data record is then transmitted to at least one end device for playback of the media content aggregate, i.e. it is made available to a consumer, for example as a "stream" (step G).

The method and system according to the invention can be used, for example, to watch a film or a documentary in a specific time frame. For example, "a documentary about space in 60 minutes" could be requested. Then the system will access various sources on this topic and analyze the media content. Duplications are removed and the important information is put together (aggregated) so that at the end exactly 60 minutes of documentation is generated so that the user can fill in his/her remaining time perfectly. With the help of the system according to the invention, there are also many other possible applications, such as compiling the best car racing scenes from all the "Fast and the Furious" films or cutting a trailer for the film "Avatar" as desired.

Claims (11)

Computer-implemented method, comprising the following steps: A) Selecting at least one source data set that includes user data that includes media content that can be reproduced at least visually, acoustically and/or in writing; B) analyzing the media content and detecting clusters within this media content; C) comparing the clusters and determining at least one overall cluster that includes common and/or overlapping media content; D) Indexing of the overall cluster and generation of metadata and/or keywords assigned to this overall cluster and storage of this metadata and/or keywords of the overall cluster in at least one database; E) Selecting overall clusters from the database using the metadata and/or keywords and assembling the selected overall clusters to generate at least one media content aggregate; F) Creation of at least one target data record that includes user data that includes the media content aggregate with media content that can be reproduced at least visually, acoustically and/or in writing; and G) Transmission of the target data set to at least one terminal for playback of the media content aggregate. procedure after claim 1 , characterized in that external data sources, preferably publicly available metadata sources, are used to enhance the media content in step A). procedure after claim 1 or 2 , characterized in that step B) comprises the following steps: B1) splitting up the user data by separating the media content that can be reproduced visually, acoustically and/or in writing; and B2) Separately analyzing the different media content and capturing image/video, sound and/or text clusters within the respective media content. procedure after claim 3 , characterized in that step C) comprises comparing the clusters of the different media contents. Procedure according to one of Claims 1 until 4 , characterized in that in step C) specific patterns in the clusters are recognized and compared with one another with the aid of at least one algorithm, preferably an algorithm for machine learning, the at least one overall cluster being determined on the basis of common and/or overlapping patterns . Procedure according to one of Claims 1 until 5 , characterized in that the selection of total clusters from the database in step E) is based at least in part on at least one specification selected from the group consisting of at least one user specification, at least one user-specific filter setting, at least one preset restriction or preference and at least one recommendation filter. Procedure according to one of Claims 1 until 6 , characterized in that the creation of at least one target data record in step F) comprises generating the media content to be output based on at least one criterion selected from the group consisting of the time length of the playback, the weighting and selection of the individual clusters based on user preferences and the weight based on at least one recommendation filter. Procedure according to one of Claims 1 until 7 , characterized in that the media content in the target data set is temporally compressed in comparison with the at least one source data set. Computer program comprising computer-executable instructions which, when the program is executed by a computer, cause the latter to carry out the method according to one of Claims 1 until 8th to execute. Computer-readable data carrier on which the computer program claim 9 is saved. System for the transmission of aggregated media content to at least one end device via at least one data connection, comprising the following means: e) means for analyzing and comparing media content from at least one source data record using at least one algorithm implemented in at least one data processing device; f) means for generating metadata and storing this metadata in at least one database; g) means for selecting and aggregating media content from at least one content store based on the metadata stored in the database; and h) Means for transmitting at least one target data record via the data connection to the terminal.

Images