WO2021023397A1 - Method and device for enriching multimedia content through metainformation - Google Patents
Method and device for enriching multimedia content through metainformation Download PDFInfo
- Publication number
- WO2021023397A1 WO2021023397A1 PCT/EP2020/025354 EP2020025354W WO2021023397A1 WO 2021023397 A1 WO2021023397 A1 WO 2021023397A1 EP 2020025354 W EP2020025354 W EP 2020025354W WO 2021023397 A1 WO2021023397 A1 WO 2021023397A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- video
- content
- audiovisual program
- linear
- audiovisual
- Prior art date
Links
- 238000000034 method Methods 0.000 title abstract description 12
- 238000001514 detection method Methods 0.000 claims abstract description 14
- 238000012163 sequencing technique Methods 0.000 claims abstract description 9
- 230000001360 synchronised effect Effects 0.000 claims abstract description 4
- 230000002452 interceptive effect Effects 0.000 claims description 14
- 230000000007 visual effect Effects 0.000 claims description 7
- 230000001131 transforming effect Effects 0.000 claims description 6
- 230000002123 temporal effect Effects 0.000 claims description 4
- 238000000605 extraction Methods 0.000 claims description 2
- 238000007792 addition Methods 0.000 claims 2
- 230000006870 function Effects 0.000 abstract description 3
- 238000013527 convolutional neural network Methods 0.000 description 3
- 238000013135 deep learning Methods 0.000 description 2
- 239000000284 extract Substances 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 230000005236 sound signal Effects 0.000 description 1
- 238000013518 transcription Methods 0.000 description 1
- 230000035897 transcription Effects 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/85—Assembly of content; Generation of multimedia applications
- H04N21/854—Content authoring
-
- G—PHYSICS
- G11—INFORMATION STORAGE
- G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
- G11B27/00—Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
- G11B27/02—Editing, e.g. varying the order of information signals recorded on, or reproduced from, record carriers
- G11B27/031—Electronic editing of digitised analogue information signals, e.g. audio or video signals
-
- G—PHYSICS
- G11—INFORMATION STORAGE
- G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
- G11B27/00—Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
- G11B27/10—Indexing; Addressing; Timing or synchronising; Measuring tape travel
- G11B27/19—Indexing; Addressing; Timing or synchronising; Measuring tape travel by using information detectable on the record carrier
- G11B27/28—Indexing; Addressing; Timing or synchronising; Measuring tape travel by using information detectable on the record carrier by using information signals recorded by the same method as the main recording
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
- H04N21/23424—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving splicing one content stream with another content stream, e.g. for inserting or substituting an advertisement
Definitions
- the present invention is a system capable of automating the sequencing and enrichment of a linear audiovisual program or stream of images of any kind (television shows, films, documentaries, series, or any other audiovisual or educational programs).
- the method and the device consist of a Smart-Video platform giving users access to additional content, services and features for arranging and browsing content, search, sharing and e-commerce relating to the content of the video.
- the platform is made up of B pillars:
- An editing tool called the Editor, for generating a temporal description file, data content and interface models for users.
- a video player or "player” for using the video and associated data. Thanks to these B pillars, the use of the platform generates a Smart-Video, a video whose content, functions and navigation mode are enhanced, allowing the user to personalize his journey and his experience within the same. video. By its modular nature, the video player can benefit all third-party applications, OTT platforms (such as My Canal, BBC
- the first patent concerning a data source enrichment process on the creation of new data from sources and text analyzes, but does not specify that these are audiovisual sources.
- the second relates to a method and apparatus for enriching multimedia contents with meta-information. It adds metadata about the original media content.
- the media are physical and the invention relates to a methodology for the manual use of metadata.
- Audiovisual soundtrack and its broadcasters Presentation of the invention
- the present invention endowed with a strong analytical capacity, makes it possible to automatically detect a large number of determining elements allowing the collection of metadata relating to the content of the moment analyzed.
- the metadata is synchronized with the content. For this, all the frames of a video are analyzed, 1 second of video corresponds to a number of frames ranging from 25 to 30 frames. For example, if it is detected that a personality appears on the 23rd frame of the 5th second of the video, it is then possible to attach additional information about this personality from the 00: 00 timecode : 05: 23.
- the present invention relates to the first pillar of the platform, the objective of which, by enriching metadata, is to simplify, by automating as much as possible, the addition of content and / or interactive elements to an audio-visual program.
- This automated system is capable of analyzing and then transforming a linear audiovisual program or image stream of any kind (TV shows, films, documentaries, etc.) into an interactive audiovisual program that can be viewed on a mobile phone, any screen or connected device. in particular, tablet, Apple TV, Android TV, and / or smart TV.
- the Editor then allows you to step through a video stream frame by frame, check the results of automatic analysis, add data and place it at selected times.
- This enriched content is integrated into the initial video and reformatted to achieve a personalized user experience.
- Chapter creation consists of indexing the content of a video with the aim of browsing it in a non-linear way as would be possible by cutting chapters provided on DVD media or by creating a playlist grouping together only the relevant sequences of the video. multiple videos under a pop-up list. This also makes it possible to locate and navigate temporally during the playback of the stream. In addition, adding metadata to these chapters offers the possibility of performing search queries within the video itself and not just on the title of the files.
- This technology will make it possible to analyze and transform videos accessible to the public, live or in deferred time, coming from any hypertext or search engine, from any content whatever its origin, from databases of any type.
- the contents to be added can be of different types and are defined during the creation of the models which are to the desired user experience, which can differ depending on the format (Film, Magazine, Documentary) but also on the program itself: it can These may be examples of biographies, video extracts (trailer, musical clip extract, archive of programs already broadcast ...), or even purchasing features for the sale of concert tickets, books, or others.
- the result can then be viewed from an application (mobile phone, tablet, smart TV, TV OS, website or any screen connected to the Internet) allowing users to benefit from the enriched experience and consult the added data.
- an application mobile phone, tablet, smart TV, TV OS, website or any screen connected to the Internet
- the device consists of a Smart -Video platform which includes, on the one hand, an editor making it possible to edit, publish, share, and provide access to enriched content, on the other hand to generate a temporal description file of synchronized metadata, and finally a video player for using Smart-Video.
- the metadata are accessible from codes and functionalities (API, SDK) making the video player replaceable and the device compatible with any other player or technological platform in an interoperable manner.
- the method automates as much as possible the actions which were until then carried out by the intervention of a human being capable of visualizing and understanding the editorial meaning of a video. This is the case for the following actions:
- the platform is capable of automatically detecting chapters and video, audio or multimedia events, from Deep Learning solutions ("deeplearning”) or neural networks but also from algorithms in the field of signal processing, and to create a "video event” from the initial video on your own. It can also add “video events” of other types depending on the context, content and narration detected.
- deeplearning Deep Learning solutions
- neural networks also from algorithms in the field of signal processing, and to create a "video event” from the initial video on your own. It can also add “video events” of other types depending on the context, content and narration detected.
- the automatic detection solution more commonly known as the “automation engine” is divided into 2 blocks, one intended for splitting the video into chapters, the other for placing interactive content.
- Each block is based on a set of already known techniques and algorithms meeting the specific needs of the platform: the detection of “video events” and their placements corresponding to types of content.
- FIG. B illustrates an example of steps in the case where the video includes audio or visual cues making it possible to identify the cut between two chapters: 1) First splitting following the detection of the audio cues using the so-called cross-correlation method . The comparison of two signals can find similarities between the latter two. Each similarity found is then accompanied by a probability score. The closer this score is to 1, the stronger the similarity; 2) Adjustment thanks to the detection of visual cues in the segments of the previous cut;
- the present invention makes it possible to describe in the form of rules, as a function of the program, the sequence of the algorithms to be used in order to detect the relevant timecodes.
- the automatic placement of interactive elements also relies on cues such as text embedded in the video or keywords resulting from the transcription of the audio track of the stream into text.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Business, Economics & Management (AREA)
- Marketing (AREA)
- Computer Security & Cryptography (AREA)
- Television Signal Processing For Recording (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention is a system capable of automating the sequencing and the enrichment of a linear audiovisual program or stream of images of any kind (television programs, films, documentaries, series, or any other audiovisual or educational programs). The method and the device consist of a smart video platform giving users access to additional content, services and functionalities for content navigation and layout, searching, sharing and e-commerce in relation to the content of the video. The audiovisual content and user experience are thus enriched. A powerful analysis capability allows this system to collect a high number of metadata on the basis of which the video content is able to be enriched and sequenced. These metadata are synchronized, and the placement of additional content and sequencing may thus be positioned with very high accuracy. The software, which we call the Editor, makes it possible to run through a video stream frame by frame, to adjust the metadata that are automatically generated, to add data and/or functional objects and to place them at chosen times. The interface of this system is formed of two parts, the first designed for the addition, editing and management of the content, the second being used to create and label sequences (chapters), marking points and/or functional objects. It is also used to place content on the timeline. The representation of the data is therefore generated using this Editor or through automatic detection. Finally, a video reader or "player" makes it possible to read the resulting video enriched with its associated data and functions. Under these conditions, this is a module that allows the reader to be included in a third-party application offering consumers an enriched audiovisual experience. It is either autonomous or remains in the universe and the ecosystem of audiovisual rights holders and broadcasters.
Description
DESCRIPTION : DESCRIPTION:
Procédé et appareil d'enrichissement de contenu multimédia par des meta- informations Method and apparatus for enriching multimedia content with meta-information
Domaine technique Technical area
La présente invention est un système capable d'automatiser le séquençage et l'enrichissement d'un programme audiovisuel linéaire ou flux d'images de toute nature (émissions de télévision, films, documentaires, séries, ou tout autres programmes audiovisuel ou éducatif). Le procédé et le dispositif consistent en une plateforme Smart-Vidéo donnant aux utilisateurs un accès à des contenus additionnels, des services et des fonctionnalités d'agencement et navigation de contenus, de recherche, de partage et d'e-commerce relatifs au contenu de la vidéo. The present invention is a system capable of automating the sequencing and enrichment of a linear audiovisual program or stream of images of any kind (television shows, films, documentaries, series, or any other audiovisual or educational programs). The method and the device consist of a Smart-Video platform giving users access to additional content, services and features for arranging and browsing content, search, sharing and e-commerce relating to the content of the video.
La plateforme est constituée de B piliers : The platform is made up of B pillars:
1) Un outil d'édition, appelé l'Éditeur, permettant de générer un fichier de description temporelle, des contenus de données et des modèles d'interface pour les utilisateurs. 1) An editing tool, called the Editor, for generating a temporal description file, data content and interface models for users.
2) Des fichiers représentant le support généré à partir de cet Éditeur. 2) Files representing the media generated from this Editor.
3) Un lecteur de vidéo, ou « player » permettant d'utiliser la vidéo et les données associées.
Grâce à ces B piliers, l'utilisation de la plateforme génère une Smart-Vidéo, vidéo dont le contenu, les fonctions et le mode de navigation sont enrichis, permettant à l'utilisateur de personnaliser son parcours et son expérience au sein même de la vidéo. De par sa nature modulaire, le lecteur vidéo peut 5 bénéficier à toutes applications tierces, plateformes OTT (tels My Canal, BBC3) A video player, or "player" for using the video and associated data. Thanks to these B pillars, the use of the platform generates a Smart-Video, a video whose content, functions and navigation mode are enhanced, allowing the user to personalize his journey and his experience within the same. video. By its modular nature, the video player can benefit all third-party applications, OTT platforms (such as My Canal, BBC
Player, MyTFl), OSTV, ou sites internet, mais également être utilisé de manière autonome. Player, MyTFl), OSTV, or websites, but also be used stand-alone.
Etat de la technique antérieure State of the prior art
10 On connaît l'état de la technique d'approche interactive décrite ci-dessus. On peut en citer deux exemples : The state of the art of the interactive approach described above is known. We can cite two examples:
Un brevet WO2009115695, déposé le 25 février 2009 et publié le 24 septembre 2009 par MAIM ENRICO Français et un brevet WO2013079768 15 déposé le 17 octobre 2012 et publié le 6 juin 2013 par NOKIA Corp US. A patent WO2009115695, filed February 25, 2009 and published September 24, 2009 by MAIM ENRICO French and a patent WO2013079768 15 filed October 17, 2012 and published June 6, 2013 by NOKIA Corp US.
Le premier brevet concernant un procédé d'enrichissement de source de données sur la création de nouvelles données à partir de sources et d'analyses de textes mais ne précise pas qu'il s'agit de sources audiovisuelles. Le second 20 concernant un procédé et appareil d'enrichissement de contenus multimédias par des méta-informations. Il ajoute des métadonnées sur les contenus multimédias d'origine. Les supports sont physiques et l'invention porte sur une méthodologie de l'utilisation manuelle des métadonnées. The first patent concerning a data source enrichment process on the creation of new data from sources and text analyzes, but does not specify that these are audiovisual sources. The second relates to a method and apparatus for enriching multimedia contents with meta-information. It adds metadata about the original media content. The media are physical and the invention relates to a methodology for the manual use of metadata.
25 En l'état, il n'existe aucun procédé automatisé d'édition permettant l'enrichissement et le découpage des média vidéo en séquences et ou évènements fonctionnels ou offrant une solution intégrée de back-office, de lecteur vidéo enrichie de contenus, de fonctionnalités et services (vidéo et expérience utilisateur enrichie) à destination de tout détenteur de droits25 As it stands, there is no automated editing process allowing the enrichment and division of video media into functional sequences and / or events or offering an integrated back-office solution, a video player enriched with content, features and services (video and enriched user experience) intended for all rights holders
BO audiovisuels et ses diffuseurs.
Présentation de l'invention Audiovisual soundtrack and its broadcasters. Presentation of the invention
La présente invention dotée d'un forte capacité d'analyse permet de détecter automatiquement un grand nombre d'éléments déterminants permettant la collecte de métadonnées en rapport avec le contenu du moment analysé. Les métadonnées sont synchronisées avec le contenu. Pour cela, toutes les trames d'une vidéo sont analysées, 1 seconde de vidéo correspond à un nombre de trames allant de 25 à 30 trames. A titre d'exemple, s'il est détecté qu'une personnalité apparaît sur la 23ème trame de la 5ème seconde de la vidéo, il est alors possible d'attacher des informations complémentaires au sujet de cette personnalité à partir du timecode 00 :00 :05 :23. The present invention, endowed with a strong analytical capacity, makes it possible to automatically detect a large number of determining elements allowing the collection of metadata relating to the content of the moment analyzed. The metadata is synchronized with the content. For this, all the frames of a video are analyzed, 1 second of video corresponds to a number of frames ranging from 25 to 30 frames. For example, if it is detected that a personality appears on the 23rd frame of the 5th second of the video, it is then possible to attach additional information about this personality from the 00: 00 timecode : 05: 23.
La présente invention concerne le premier pilier de la plateforme, laquelle a pour objectif, par l'enrichissement de métadonnées, de simplifier, en automatisant au maximum, l'ajout de contenus et/ou d'éléments interactifs à un programme audio-visuel. The present invention relates to the first pillar of the platform, the objective of which, by enriching metadata, is to simplify, by automating as much as possible, the addition of content and / or interactive elements to an audio-visual program.
Ce système automatisé est capable d'analyser puis de transformer un programme audiovisuel linéaire ou flux d'images de toute nature (émissions de TV, films, documentaires ...) en un programme audiovisuel interactif consultable sur téléphone mobile, tout écran ou appareil connecté notamment, tablette, Apple TV, Android TV, et/ou smart TV. This automated system is capable of analyzing and then transforming a linear audiovisual program or image stream of any kind (TV shows, films, documentaries, etc.) into an interactive audiovisual program that can be viewed on a mobile phone, any screen or connected device. in particular, tablet, Apple TV, Android TV, and / or smart TV.
Son interface, tout en parcourant un flux vidéo trame par trame, grâce à des algorithmes procédant par extraction, se compose de deux parties, la première destinée à l'ajout, l'édition et la gestion des contenus d'origine, quelle qu'en soit la source interne ou externe, enrichi d'informations ou de fonctionnalités interactives, la seconde servant à la création et à la labélisation de chapitres, séquences événements ou points de marquage étant des contenus éditoriaux contextuels et pertinents. Elle sert également au placement de ces contenus et/ou objets fonctionnels et l'enrichissement de métadonnées.
Des indices audio ou visuels, les métadonnées, permettent d'organiser le séquençage, premier découpage suite à la détection d'indices audio ou visuels de la vidéo, puis de les ajuster et de les affiner à partir des indices ou scènes de la vidéo pour parvenir à une thématique ou une interactivité partielle. Its interface, while browsing a video stream frame by frame, thanks to algorithms proceeding by extraction, is composed of two parts, the first intended for the addition, the edition and the management of the original contents, whatever. either the internal or external source, enriched with information or interactive features, the second used for the creation and labeling of chapters, event sequences or marking points being contextual and relevant editorial content. It is also used for the placement of this content and / or functional objects and the enrichment of metadata. Audio or visual cues, metadata, make it possible to organize the sequencing, the first cutting following the detection of audio or visual cues from the video, then to adjust and refine them from the cues or scenes of the video to achieve a thematic or partial interactivity.
L'Éditeur permet ensuite de parcourir un flux vidéo trame par trame, de vérifier les résultats de l'analyse automatique, d'ajouter des données et de les placer à des temps choisis. The Editor then allows you to step through a video stream frame by frame, check the results of automatic analysis, add data and place it at selected times.
Ces contenus enrichis sont intégrés à la vidéo initiale et reformatés pour obtenir une expérience d'utilisateur personnalisée. This enriched content is integrated into the initial video and reformatted to achieve a personalized user experience.
La création de chapitres consiste à indexer le contenu d'une vidéo dans le but de la parcourir d'une manière non linéaire comme le permettrait le découpage de chapitres fourni sur un support DVD ou la création de liste de lecture regroupant seulement les séquences pertinentes de plusieurs vidéos sous une liste contextuelle. Cela permet également de se situer et de naviguer temporellement durant la lecture du flux. De plus, l'ajout de métadonnées à ces chapitres offre la possibilité d'effectuer des requêtes de recherche au sein même de la vidéo et non plus simplement que sur le titre des fichiers. Chapter creation consists of indexing the content of a video with the aim of browsing it in a non-linear way as would be possible by cutting chapters provided on DVD media or by creating a playlist grouping together only the relevant sequences of the video. multiple videos under a pop-up list. This also makes it possible to locate and navigate temporally during the playback of the stream. In addition, adding metadata to these chapters offers the possibility of performing search queries within the video itself and not just on the title of the files.
Ce procédé d'indexation et de séquençage est illustré dans la [Fig.l] This indexing and sequencing process is illustrated in [Fig.l]
Cette technologie permettra d'analyser et de transformer des vidéos accessibles au public, en direct ou en temps différé, provenant de tout moteur hypertexte ou de recherche, de tout contenu quel qu'en soit l'origine, de bases de données de tout type...
Les contenus à ajouter peuvent être de différents types et sont définis lors de la création des modèles qui sont à l'expérience utilisateur souhaité, lesquels peuvent différer en fonction du format (Film, Magazine, Documentaire) mais aussi du programme proprement dit : il peut s'agir à titre d'exemples de biographies, d'extraits vidéo (bande annonce, clip musical extrait, archive d'émissions déjà diffusés...), ou encore de fonctionnalités d'achat pour la vente de places de concerts, de livres, ou autres. This technology will make it possible to analyze and transform videos accessible to the public, live or in deferred time, coming from any hypertext or search engine, from any content whatever its origin, from databases of any type. ... The contents to be added can be of different types and are defined during the creation of the models which are to the desired user experience, which can differ depending on the format (Film, Magazine, Documentary) but also on the program itself: it can These may be examples of biographies, video extracts (trailer, musical clip extract, archive of programs already broadcast ...), or even purchasing features for the sale of concert tickets, books, or others.
Le résultat est ensuite consultable depuis une application (téléphone mobile, tablette, smart TV, TV OS, site web ou tout écran connecté à internet) permettant de bénéficier de l'expérience enrichie et de consulter les données ajoutées. The result can then be viewed from an application (mobile phone, tablet, smart TV, TV OS, website or any screen connected to the Internet) allowing users to benefit from the enriched experience and consult the added data.
Le dispositif consiste en une plateforme Smart -Vidéo qui comprend, d'une part, un éditeur permettant d'éditer, de publier, de partager, de donner accès à des contenus enrichis, d'autre part de générer un fichier de description temporelle de métadonnées synchronisées, et enfin un lecteur vidéo permettant d'utiliser la Smart-Vidéo. The device consists of a Smart -Video platform which includes, on the one hand, an editor making it possible to edit, publish, share, and provide access to enriched content, on the other hand to generate a temporal description file of synchronized metadata, and finally a video player for using Smart-Video.
Les métadonnées sont accessibles depuis des codes et des fonctionnalités (API, SDK) rendant le lecteur vidéo remplaçable et le dispositif compatible avec tout autre lecteur ou plateforme technologique de manière interopérable. The metadata are accessible from codes and functionalities (API, SDK) making the video player replaceable and the device compatible with any other player or technological platform in an interoperable manner.
Dans ces conditions, le procédé automatise un maximum les actions qui étaient jusqu'alors effectuées par l'intervention d'un être humain capable de visualiser et de comprendre le sens éditorial d'une vidéo. Il en est ainsi des actions suivantes :Under these conditions, the method automates as much as possible the actions which were until then carried out by the intervention of a human being capable of visualizing and understanding the editorial meaning of a video. This is the case for the following actions:
- visionnage d'un flux vidéo « timeline » ; - viewing of a “timeline” video stream;
- identification d'un passage pouvant être augmenté, enrichi d'informations ou de fonctionnalités interactives ; - identification of a passage that can be increased, enriched with information or interactive features;
- ajout de contenu et de données en rapport avec le passage identifié ;
- division en chapitres, séquençage, marquage ; - addition of content and data related to the identified passage; - division into chapters, sequencing, marking;
- placement de modèle temporel de contenus, d'objets interactifs et/ou fonctionnels « vidéo événement » - placement of a temporal model of content, interactive and / or functional “video event” objects
- association d'un modèle de « vidéo événement » à un des contenus gérés par la base de données. - association of a “video event” model with one of the contents managed by the database.
Aussi la plateforme est capable de détecter automatiquement des chapitres et des évènements vidéo, audio ou multimédia, à partir de solutions d'Apprentissage Profond (« deeplearning ») ou réseaux neuronaux mais également à partir d'algorithmes du domaine du traitement du signal, et de créer seul un « vidéo événement » à partir de la vidéo initiale. Elle peut aussi ajouter des « vidéo événements » d'autres types en fonction du contexte, du contenu et de la narration détectée. Also the platform is capable of automatically detecting chapters and video, audio or multimedia events, from Deep Learning solutions ("deeplearning") or neural networks but also from algorithms in the field of signal processing, and to create a "video event" from the initial video on your own. It can also add “video events” of other types depending on the context, content and narration detected.
La solution de détection automatique, plus communément appelée « moteur d'automatisation » se divise en 2 blocs, l'un destiné au découpage de la vidéo en chapitres, l'autre aux placements de contenus interactifs. The automatic detection solution, more commonly known as the “automation engine” is divided into 2 blocks, one intended for splitting the video into chapters, the other for placing interactive content.
Elle peut être représentée selon le schéma décrit dans la [Fig.2] It can be represented according to the diagram described in [Fig.2]
Chaque bloc s'appuie sur un ensemble de techniques déjà connues et d'algorithmes répondant aux besoins spécifiques de la plateforme : la détection de « vidéo événements » et leurs placements correspondant à des types de contenus. Each block is based on a set of already known techniques and algorithms meeting the specific needs of the platform: the detection of “video events” and their placements corresponding to types of content.
Le séquençage individualisé offre à l'utilisateur la possibilité de sélectionner tout ou partie des chapitres d'une vidéo à visualiser avec un repère de temps permettant de se situer lors de la visualisation.
Pour la détection automatique des timecodes permettant le séquençage en chapitres et le placement de vidéo événements, un ensemble d'algorithmes permet de réduire au maximum la fenêtre de recherche de ces derniers. Cet ensemble d'algorithmes est ensuite adapté en fonction du programme et des éléments recherchés. Individualized sequencing offers the user the possibility of selecting all or part of the chapters of a video to be viewed with a time mark allowing them to position themselves during viewing. For the automatic detection of timecodes allowing sequencing in chapters and the placement of video events, a set of algorithms makes it possible to reduce the search window for these as much as possible. This set of algorithms is then adapted according to the program and the elements sought.
La [Fig. B] illustre un exemple d'étapes dans le cas où la vidéo comporte des indices audios ou visuels permettant d'identifier la coupure entre deux chapitres : 1) Premier découpage suite à la détection des indices audio en utilisant la méthode dite de cross-corrélation. La comparaison de deux signaux peut retrouver des similarités entre ces deux derniers. Chaque similarité retrouvée est alors accompagnée d'un score de probabilité. Plus ce score est proche de 1, plus la similarité est forte ; 2) Ajustement grâce à la détection d'indices visuels dans les segments du découpage précédent ; The [Fig. B] illustrates an example of steps in the case where the video includes audio or visual cues making it possible to identify the cut between two chapters: 1) First splitting following the detection of the audio cues using the so-called cross-correlation method . The comparison of two signals can find similarities between the latter two. Each similarity found is then accompanied by a probability score. The closer this score is to 1, the stronger the similarity; 2) Adjustment thanks to the detection of visual cues in the segments of the previous cut;
3) Affinage à partir de la détection de scènes dans le montage vidéo. 3) Refinement based on the detection of scenes in the video edit.
La présente invention permet de décrire sous forme de règles, en fonction du programme, l'enchaînement des algorithmes à utiliser dans le but de détecter les timecodes pertinents. The present invention makes it possible to describe in the form of rules, as a function of the program, the sequence of the algorithms to be used in order to detect the relevant timecodes.
La liste des algorithmes mise à disposition est la suivante : The list of algorithms available is as follows:
- Détection de similitudes audio par « cross-corrélation » ; - Détection et reconnaissance d'un signal audio à partir du modèle de réseaux de neurones convolutifs Youtube-8M ; - Detection of audio similarities by "cross-correlation"; - Detection and recognition of an audio signal from the Youtube-8M convolutional neural network model;
- Détection de similitudes d'une partie visuelle par SSIM ; - Detection of similarities of a visual part by SSIM;
- Détection et reconnaissances de visages à partir de l'implémentation de l'algorithme Eigen-Faces ;
- Détection et reconnaissance de visages à partir de différents modèles de réseaux de neurones convolutifs : ResNet 50, InceptionVB, DensNet, YOLOv2, MobileNet-SSD, MTCNN ; - Face detection and recognition based on the implementation of the Eigen-Faces algorithm; - Face detection and recognition from different models of convolutional neural networks: ResNet 50, InceptionVB, DensNet, YOLOv2, MobileNet-SSD, MTCNN;
- Classification d'images à partir de différents modèles de réseaux de neurones convolutifs : ResNet50, InceptionVB, MobileNet. - Classification of images from different models of convolutional neural networks: ResNet50, InceptionVB, MobileNet.
Le placement automatique des éléments interactifs s'appuie également sur des indices tels que des textes incrustés dans la vidéo ou des mots clés issus de la transcription en texte de la piste audio du flux.
The automatic placement of interactive elements also relies on cues such as text embedded in the video or keywords resulting from the transcription of the audio track of the stream into text.
Claims
1. Système automatisé capable d'analyser puis de transformer un programme audiovisuel linéaire ou flux d'images de toute nature (émissions de TV, films, documentaires ...) en un programme audiovisuel interactif consultable sur téléphone mobile, tout écran ou appareil connecté notamment, tablette, Apple TV, Android TV, et/ou Smart TV caractérisé en ce que son interface, tout en parcourant et analysant le programme audiovisuel linéaire ou le flux vidéo d'origine trame par trame, grâce à des algorithmes procédant par extraction, servant à détecter et labelliser, sur la base de l'expérience utilisateur souhaitée et définie au préalable, des chapitres, séquences événements ou points de marquage étant des contenus éditoriaux contextuels et pertinents, ainsi qu'à l'ajout et au placement de ces contenus et/ou objets fonctionnels à tous endroits dans le programme audiovisuel linéaire ou le flux vidéo d'origine, ces ajouts et placements étant déduits automatiquement de l'expérience utilisateur souhaitée. 1. Automated system capable of analyzing and then transforming a linear audiovisual program or image stream of any kind (TV shows, films, documentaries, etc.) into an interactive audiovisual program that can be viewed on a mobile phone, any screen or connected device in particular, tablet, Apple TV, Android TV, and / or Smart TV characterized in that its interface, while browsing and analyzing the linear audiovisual program or the original video stream frame by frame, using algorithms proceeding by extraction, serving to detect and label, on the basis of the desired user experience and defined beforehand, chapters, event sequences or marking points being contextual and relevant editorial content, as well as for the addition and placement of this content and / or functional objects at all places in the linear audiovisual program or the original video stream, these additions and placements being automatically deduced from the user experience s desired.
2. Système automatisé capable d'analyser puis de transformer un programme audiovisuel linéaire ou flux d'images de toute nature en un programme audiovisuel interactif consultable sur téléphone mobile, écran ou appareil connecté, selon la revendication 1, caractérisé en ce que des indices audio ou visuels, les métadonnées, permettent d'organiser le séquençage et la coupure entre deux chapitres, premier découpage suite à la détection d'indices audio ou visuels de la vidéo, puis de les ajuster et de les affiner à partir des indices ou scènes de la vidéo pour parvenir à une thématique ou une interactivité partielle conformément à l'expérience utilisateur souhaitée.
2. Automated system capable of analyzing and then transforming a linear audiovisual program or image stream of any kind into an interactive audiovisual program that can be viewed on a mobile phone, screen or connected device, according to claim 1, characterized in that the audio indices or visual, metadata, make it possible to organize the sequencing and the cut between two chapters, first cutting following the detection of audio or visual clues of the video, then to adjust and refine them from the clues or scenes of video to achieve a thematic or partial interactivity in accordance with the desired user experience.
3. Système automatisé capable d'analyser puis de transformer un programme audiovisuel linéaire ou flux d'images de toute nature en un programme audiovisuel interactif consultable sur téléphone mobile, écran ou appareil connecté, selon les revendications 1 et 2, caractérisé en ce que les contenus et/ou objets fonctionnels additionnels sont intégrés à la vidéo initiale et reformatés pour l'expérience d'utilisateur souhaitée. 3. Automated system capable of analyzing and then transforming a linear audiovisual program or image stream of any kind into an interactive audiovisual program that can be viewed on a mobile phone, screen or connected device, according to claims 1 and 2, characterized in that the Additional content and / or functional objects are integrated into the initial video and reformatted for the desired user experience.
4. Système automatisé capable d'analyser puis de transformer un programme audiovisuel linéaire ou flux d'images de toute nature en un programme audiovisuel interactif consultable sur téléphone mobile, écran ou appareil connecté, selon les revendications 1 à 3, caractérisé en ce que cette technologie permet d'analyser et de transformer tout contenu quel qu'en soit l'origine, en ce compris des vidéos accessibles au public, en direct ou en temps différé, provenant de tout moteur hypertexte ou de recherche, et des bases de données de tout type. 4. Automated system capable of analyzing and then transforming a linear audiovisual program or image stream of any kind into an interactive audiovisual program that can be viewed on a mobile phone, screen or connected device, according to claims 1 to 3, characterized in that this technology makes it possible to analyze and transform any content whatever its origin, including videos accessible to the public, live or in deferred time, coming from any hypertext or search engine, and databases of Every type.
5. Système automatisé capable d'analyser puis de transformer un programme audiovisuel linéaire ou flux d'images de toute nature en un programme audiovisuel interactif consultable sur téléphone mobile, écran ou appareil connecté, selon les revendications 1 à 3, caractérisée en ce que ce système consiste en un dispositif de plateforme Smart-Vidéo comprenant, d'une part, un éditeur pour éditer, publier, partager et donner accès à des contenus enrichis, d'autre part un fichier de description temporelle de métadonnées synchronisées, généré via l'éditeur vidéo, et enfin un lecteur vidéo pour l'utilisation de la Smart-Vidéo et des données associées par le public.
5. Automated system capable of analyzing and then transforming a linear audiovisual program or image stream of any kind into an interactive audiovisual program that can be viewed on a mobile phone, screen or connected device, according to claims 1 to 3, characterized in that this The system consists of a Smart-Video platform device comprising, on the one hand, an editor for editing, publishing, sharing and providing access to enriched content, on the other hand a temporal description file of synchronized metadata, generated via the video editor, and finally a video player for the use of Smart-Video and associated data by the public.
6. Dispositif de plateforme Smart-Vidéo, selon la revendication 5, caractérisé en ce que les métadonnées sont accessibles depuis des codes et des fonctionnalités (API, SDK) rendant le lecteur vidéo remplaçable et la lecture des programmes interactifs interopérable avec tout autre lecteur, plateforme technologique ou tout autre dispositif digital.
6. Smart-Video platform device, according to claim 5, characterized in that the metadata are accessible from codes and functionalities (API, SDK) making the video player replaceable and the reading of interactive programs interoperable with any other player, technological platform or any other digital device.
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| FR1908943A FR3099674B1 (en) | 2019-08-03 | 2019-08-03 | METHOD AND APPARATUS FOR ENRICHING MULTIMEDIA CONTENT WITH META-INFORMATION |
| FRFR1908943 | 2019-08-03 |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| WO2021023397A1 true WO2021023397A1 (en) | 2021-02-11 |
Family
ID=69572030
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| PCT/EP2020/025354 WO2021023397A1 (en) | 2019-08-03 | 2020-07-31 | Method and device for enriching multimedia content through metainformation |
Country Status (2)
| Country | Link |
|---|---|
| FR (1) | FR3099674B1 (en) |
| WO (1) | WO2021023397A1 (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2009115695A1 (en) | 2008-02-25 | 2009-09-24 | Enrico Maim | Method for enriching data sources |
| WO2010055242A1 (en) * | 2008-11-13 | 2010-05-20 | France Telecom | Method for cutting multimedia content, and corresponding device and computer program |
| WO2013079768A1 (en) | 2011-11-30 | 2013-06-06 | Nokia Corporation | Method and apparatus for enriching media with meta-information |
| US20140337127A1 (en) * | 2013-04-23 | 2014-11-13 | Brightcove, Inc. | Client bridge |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8249423B2 (en) * | 2008-05-08 | 2012-08-21 | Sony Ericsson Mobile Communications Ab | Electronic devices and methods that insert addressable chapter marks relative to advertising content in video streams |
-
2019
- 2019-08-03 FR FR1908943A patent/FR3099674B1/en not_active Expired - Fee Related
-
2020
- 2020-07-31 WO PCT/EP2020/025354 patent/WO2021023397A1/en active Application Filing
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2009115695A1 (en) | 2008-02-25 | 2009-09-24 | Enrico Maim | Method for enriching data sources |
| WO2010055242A1 (en) * | 2008-11-13 | 2010-05-20 | France Telecom | Method for cutting multimedia content, and corresponding device and computer program |
| WO2013079768A1 (en) | 2011-11-30 | 2013-06-06 | Nokia Corporation | Method and apparatus for enriching media with meta-information |
| US20140337127A1 (en) * | 2013-04-23 | 2014-11-13 | Brightcove, Inc. | Client bridge |
Also Published As
| Publication number | Publication date |
|---|---|
| FR3099674A1 (en) | 2021-02-05 |
| FR3099674B1 (en) | 2022-04-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| KR101994592B1 (en) | AUTOMATIC VIDEO CONTENT Metadata Creation METHOD AND SYSTEM | |
| US7620551B2 (en) | Method and apparatus for providing search capability and targeted advertising for audio, image, and video content over the internet | |
| US8204317B2 (en) | Method and device for automatic generation of summary of a plurality of images | |
| US20150301718A1 (en) | Methods, systems, and media for presenting music items relating to media content | |
| US20080187231A1 (en) | Summarization of Audio and/or Visual Data | |
| US20110035382A1 (en) | Associating Information with Media Content | |
| JP2014032656A (en) | Method, device and program to generate content link | |
| JP2004533756A (en) | Automatic content analysis and display of multimedia presentations | |
| US20110289099A1 (en) | Method and apparatus for identifying video program material via dvs or sap data | |
| US10665267B2 (en) | Correlation of recorded video presentations and associated slides | |
| KR20060008897A (en) | Method and apparatus for summarizing a music video using content analysis | |
| Jiang et al. | Automatic consumer video summarization by audio and visual analysis | |
| KR20060129030A (en) | Video trailer | |
| RU2413990C2 (en) | Method and apparatus for detecting content item boundaries | |
| US20120239689A1 (en) | Communicating time-localized metadata | |
| EP2104937A1 (en) | Method for creating a new summary of an audiovisual document that already includes a summary and reports and a receiver that can implement said method | |
| WO2021023397A1 (en) | Method and device for enriching multimedia content through metainformation | |
| Tseng et al. | Hierarchical video summarization based on context clustering | |
| JP2001075992A (en) | Method and system for sound retrieval and computer- readable recording medium | |
| JP2006157688A (en) | Significance label providing method, apparatus, and program to video scene | |
| Agnihotri et al. | Music videos miner | |
| Carmichael et al. | Multimodal indexing of digital audio-visual documents: A case study for cultural heritage data | |
| Peeters et al. | A Multimedia Search and Navigation Prototype, Including Music and Video-clips. | |
| JP2010157080A (en) | System, method and program for retrieving content relevant information | |
| Saudagar et al. | A comparative study of video splitting techniques |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| 121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 20747341 Country of ref document: EP Kind code of ref document: A1 |
|
| NENP | Non-entry into the national phase |
Ref country code: DE |
|
| 32PN | Ep: public notification in the ep bulletin as address of the adressee cannot be established |
Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 02.06.2023) |
|
| 122 | Ep: pct application non-entry in european phase |
Ref document number: 20747341 Country of ref document: EP Kind code of ref document: A1 |