EP4335111A1

EP4335111A1 - Computer-implemented method for delivering audio-visual media on-demand

Info

Publication number: EP4335111A1
Application number: EP22748259.3A
Authority: EP
Inventors: Boris BORZIC; Elmahdi SADOUNI
Original assignee: Centre National de la Recherche Scientifique CNRS; CY Cergy Paris Universite; Ecole Nationale Superieure de lElectronique et de ses Applications ENSEA
Current assignee: Centre National de la Recherche Scientifique CNRS; CY Cergy Paris Universite; Ecole Nationale Superieure de lElectronique et de ses Applications ENSEA
Priority date: 2021-07-08
Filing date: 2022-07-06
Publication date: 2024-03-13
Also published as: FR3125193A1; WO2023280946A1

Abstract

This computer-implemented method for delivering audio-visual media on-demand allows one or more digital video files to be sequenced and the sequences generated by the sequencing to be indexed, by dividing virtually the one or more digital video files into digital virtual sequences, each bounded virtually by two sequence time markers. The method is intended to produce and select automatically virtual sequences of each digital video file, the file fragments corresponding to the virtual sequences then being able to be extracted from the digital video files in question to be viewed or recorded in a new digital video file.

Description

COMPUTERIZED DE-LINEARIZATION PROCESS

AUDIOVISUAL

FIELD OF THE INVENTION The present invention relates to the field of identification and automated processing of digital data, in particular digital video files.

The invention relates more specifically to a computerized process for the audiovisual de-linearization of digital video files. TECHNOLOGICAL BACKGROUND

The amount of information generated in today's society is increasing exponentially. Moreover, data is made available in multiple dimensions on different digital media, such as video stream, audio stream, and text stream.

This mass of multimedia information poses significant technological challenges in terms of how multimedia data can be integrated, processed, organized and indexed in a semantically meaningful way to facilitate efficient retrieval. Usually, a content structure is designed by the data producer before the data is generated and saved. To enable future content-based retrieval, such intended semantic structure (or metadata) must be transmitted with the content to users when the content is delivered. In this way, users can choose what they want based on the description of this metadata. For example, each book or magazine is published with its table of contents, through which users can find the page number (index) where the desired information is printed by simply jumping to the page. Such indexing of highly structured content a priori therefore allows rapid access to specific parts of documents and the constitution of aggregates of document sequences, such as playlists in the case of audio files.

This structuring is rarely provided in the case of video data. For example, for a film intended for the cinema, it is not common practice to provide the indications allowing access to the different sequences composed by the filmmaker.

A large number of video files cannot be structured a priori. This is the case, for example, of events filmed live, the course of which cannot be predicted before the production of the digital video file.

Finally, the indexing defined a priori by the producer may not be relevant from the point of view of the user whose search criteria are not always known a priori either. In the case of digital video files, due to the difficulty of accessing relevant indexing, the practice is therefore to label the digital video file as a whole, so that the metadata associated with a digital video file are global, such as name, creation date, file format, viewing time. A set of metadata provides access to a digital video file as a whole when a search for audiovisual content is performed. These metadata are therefore “global”.

It is known to enrich the "global" metadata associated with a digital video file with additional metadata, but this metadata is always managed at the global level of the file to facilitate access to the video via a search engine. For example, it is possible to retrieve information such as the author, the actors, the composer of the soundtrack of a film or comments from spectators on the Internet and to complete the initial metadata with this metadata. Such enrichment allows more efficient access to a digital video file via a search engine.

To also allow access to a relevant sequence of a given digital video file, several a posteriori indexing methods can be envisaged, in particular manual indexing. However, these methods are long and tedious. In the field of video content research, the use of automatic indexing methods has thus become essential.

The difficulty with video content is that it is not self-descriptive, unlike text media.

EU document EP3252770A1 proposes a process for the identification and automatic post-processing of audiovisual content. In this process, a formal description of the content of the digital video file is provided by an operator, such as a script in the case of a film. After extracting the image (i.e. containing visual data) and audio streams from the audiovisual data, these two parts of the audiovisual data are broken down into a set of successive fragments. Furthermore, the formal description of the digital video file is broken down into logical parts. A dialog pattern is generated from the audio stream only. An association of the audiovisual data with the corresponding formal description is achieved by associating logical parts of the formal description to the set of audiovisual data fragments, using the dialogue pattern. A digital video file can then be indexed and then manipulated based on this association.

Ue document US6714909B1 is another example in which a method of automating the multimodal indexing process is proposed. A process comprises the following steps:

- separate a multimedia data stream into audio, visual and textual components; - segmenting the audio, video and text components of the multimedia data stream based on semantic differences, the frame-level features being extracted from the segmented audio component into a plurality of sub-bands;

- identifying at least one target speaker using the audio and video components;

- identifying text semantic boundaries for at least one of the identified target speakers to generate semantically consistent text blocks;

- generate a summary of the multimedia content based on the audio, video and textual components, the semantically coherent text blocks and the identified target speaker;

- deriving a topic for each of the semantically consistent text blocks based on a set of topic category models;

- generating a media description of the media event based on the identified target speaker, the semantically consistent text blocks, the identified topic and the generated summary.

The method described in document EP3252770A1 has the disadvantage of requiring the provision of a formal description of the digital video file. The method described in the document US6714909B1 has the disadvantage of requiring that the content of the audio streams and or texts of the digital video file be semantically structured, that is to say that it is a question of being able to reconstitute an audio content which makes sense by extracting and aggregating footage from a given video. It cannot therefore be implemented to aggregate sequences from different video files or for semantically weakly structured video files. The invention thus aims to propose an automated method of analysis, indexing and editing of a set of digitally possibly weakly structured video files on criteria defined by the user and without a priori indexing of the content of these files.

SUMMARY OF THE INVENTION

Thus, the invention relates to a computerized process for audiovisual de-linearization allowing sequencing of one or more digital video files and indexing of the sequences resulting from the sequencing, by virtually cutting by time stamping the digital video file(s) into virtual sequences, each virtual sequence being defined by two sequence time stamps and associated descriptors.

The method comprises the following steps: a. receiving one or more digital video files to be analyzed; b. indexing each of the digital video files in a primary index by means of associated primary endogenous descriptors making it possible to identify each digital video file; vs. automatic extraction of audio, image, and text data streams from each digital video file; d. by means of a plurality of computerized devices implementing an automatic learning algorithm chosen and/or trained for a typology of digital video files defined beforehand, automatic analysis, file by file, of each of the digital video files, according to the four modalities: image modality, audio modality, text modality, action modality making it possible to identify the groups of successive images forming a given action, the analysis automatically producing one or more unimodal cut-out temporal markers for each of the modalities, one or several descriptors being associated with each of the unimodal cutting time markers, e. automatic production, following the analysis of each of the digital video files, of virtual cutting candidate sequence time markers, with the aim of delimiting virtual sequences, and of the descriptors associated with these cutting candidate sequence time markers virtual, which are:

- either unimodal cutting time markers of the digital video files, and which are called at the end of this step unimodal candidate sequence time markers;

- or, for each of said digital video files taken in isolation, the time codes corresponding to the unimodal virtual cutting time markers are compared and, each time that at least two unimodal cutting time markers resulting from different analysis modalities are separated by a time interval less than a main predetermined duration, a multimodal candidate sequence time marker, mathematically related to the at least two unimodal sequence markers, is created; f. for each of said digital video files analyzed, according to a lower limit and an upper limit defined to determine the minimum duration and the maximum duration of each sequence, with respect to the typology of the digital video file(s),

- automatic selection, among the unimodal or plurimodal candidate sequence temporal markers, of pairs of sequence markers,

- each pair having a start of sequence marker and an end of sequence marker, so that the duration of each sequence selected is between said lower and upper limits,

these pairs of sequence markers being associated with the descriptors associated with the said selected candidate sequence temporal markers, these descriptors therefore being referred to as “secondary endogenous descriptors”; g. indexing, in a secondary index which is in a relationship of inheritance with respect to said primary index, of all the pairs of sequence markers and of the associated descriptors allowing the identification of each sequence, the virtual sequences being identifiable and capable of being searched for less by the secondary endogenous descriptors and the primary endogenous descriptors. Thanks to these provisions, it is possible to sequence a digital video file in sequences presenting a semantic coherence according to one to four different modalities, in the form of virtual sequences delimited by pairs of sequence time markers and indexed by secondary descriptors associated with these sequence time stamps as well as the primary descriptors associated with the digital video file from which the sequences originate. The space in memory used for these sequences corresponds to the space necessary to store the pairs of temporal markers and the associated secondary descriptors. It is in this that the sequencing is said to be virtual.

According to one embodiment, the computerized process for audiovisual de-linearization is characterized in that a video extract associated with a virtual sequence, obtained by viewing the file fragment delimited by the two sequence markers of the virtual sequence has a unit of meaning (in other words a semantic coherence) which results from the automatic analysis of each digital video file according to the four modalities and from the virtual cutting in relation to this analysis.

By virtue of this arrangement, the virtual sequences can be extracted and the video extracts corresponding to the virtual sequences can be viewed by a user who will perceive its semantic coherence and will be able to attribute an overall meaning to it.

According to one embodiment, at least one of the two sequence markers of each pair of sequence markers selected in step f is a plurimodal candidate sequence temporal marker and is then called a plurimodal sequence marker, and advantageously each sequence marker of each selected sequence tag pair is a multimodal sequence tag.

In this way, the overall meaning of the sequence is supported by several modalities and advantageously four modalities. In the latter case, semantic consistency is therefore obtained on the text modality, the action modality, the audio modality and the image modality.

Advantageously:

- the more a cut has a large number of endogenous descriptors, the more this video extract is likely to be retained in the playlist following the search for the user who would present these endogenous descriptors

-and the greater this number of endogenous descriptors present by different modalities of common results (and in this case we speak of plurimodal descriptors) the more this video extract is likely to be retained in the playlist which will be described later following the search for the user who would present these endogenous descriptors.

In general, the more the cutting markers are multimodal, the more the video extracts are cut with a fine grain size. According to one embodiment, for each video extract, the so-called endogenous descriptors are derived from the same modality, or from one or more modalities different from the modality or modalities from which are derived for the start and end temporal cutting markers sequence of the video extract

In a particular embodiment, in step f, two types of plurimodal sequence markers are distinguished:

- a plurimodal sequence marker created from four unimodal temporal cutting markers resulting from the four different modalities separated two-by-two by a time interval less than the main predetermined duration is called main plurimodal sequence marker and

- a plurimodal sequence marker created from two or three unimodal temporal cutting markers resulting from as many modalities among the four modalities, separated two-by-two by a time interval less than the main predetermined duration is said marker of secondary multimodal sequence.

According to one embodiment, at least one of the tags of each pair of sequence tags is a main multimodal sequence tag.

Thanks to this arrangement, the overall meaning of the sequence is supported by four modalities.

According to one embodiment, the action modality is a modality of at least one of the two sequence markers of the pair of sequence markers selected.

Thanks to this arrangement, the semantic coherence of a sequence is at least underpinned by the action modality, which plays a special role in many video files. For example, in the field of sport, the sequence obtained will be coherent from the point of view of sporting actions.

According to one embodiment, weights are assigned to each of the modalities for the production of candidate sequence markers in step e and/or the selection of sequence markers in step f.

Thanks to this arrangement, the semantic coherence of a sequence can be underpinned in various proportions, possibly adapted to video typologies, by the four modalities. For example, in the field of sport, we can assign a higher weight to the action modality. In the field of online courses, we can assign a higher weight to the text modality.

According to one embodiment,

- for digital video files in the field of sport, the weight of the action modality is greater than that of the image modality, itself greater than the weight of the text and audio modalities,

-for video files with high informational content by speech, the weight of the text modality is greater than that of the other three modalities. Thanks to this arrangement, the semantic coherence of a sequence can be adapted to a video typology such as a video in the field of sports or to a video with high informational content such as a documentary or an online course.

According to one embodiment, a weight is assigned to the secondary endogenous descriptors as well as to the primary endogenous descriptors to characterize their importance in the sequences, and this weight is greater for the secondary endogenous descriptors than that of the primary endogenous descriptors.

The different weights of the endogenous and exogenous descriptors make it possible, when formulating a sequence search query formulated later, to play different roles for these two types of descriptors. In particular, if the weight of endogenous descriptors is greater than that of exogenous descriptors, the results of a sequence search will be based more on endogenous descriptors than on exogenous descriptors.

According to one embodiment, the secondary endogenous descriptors are said to be “unimodal” when they correspond to a single modality and are said to be “multimodal” when they are detected for several modalities.

Thanks to this arrangement, it is possible to distinguish the descriptors subtended by a single or by several modalities, which can be useful during a search for a video file sequence in which one wishes to make these two play different roles. types of descriptors. To this end, according to one embodiment, information on the unimodal or multimodal character of a given secondary endogenous descriptor is kept during the indexing process. For example, if the image modality gives the “thermodynamics” descriptor, and the text modality also gives the “thermodynamics” descriptor, then we can create a “thermodynamics” plurimodal descriptor (which comes from the two previous descriptors and is therefore more robust on the interest of viewing this extract we are interested in thermodynamics).

According to one embodiment, step f of the method presents these sub-steps, for each digital video file, to produce the sequences: i) - selection of a last end-of-sequence marker, in particular multimodal, from the end digital video file,

-and determination of the presence of a multimodal sequence marker at a time code between two extreme time codes, calculated by subtracting the lower limit from the time code of the selected end marker and by subtracting the upper limit from the time code of the selected end marker,

- selection of the plurimodal marker as the last sequence start marker if the presence is confirmed,

-otherwise, determination of the presence of a unimodal marker whose modality depends on the typology of the digital video file between the two extreme time codes - selection of the unimodal marker as the last sequence start marker if the presence is confirmed,

otherwise, the last sequence start marker is designated by the subtraction from the time code of the last end marker selected from the upper limit; ii), step i) is repeated to select a penultimate sequence start marker, the sequence start marker selected at the end of the previous step i playing the role of last sequence end marker selected at the start of the previous step i; iii) sub-step ii) is repeated in this way until the start of the digital video file.

Thanks to this arrangement, the convergence of the sequencing is ensured.

According to one embodiment, the main predetermined duration is less than 5 seconds, and optionally the maximum duration of each selected sequence is equal to two minutes. so that the candidate sequence markers are close enough in time and the sequencing is fine enough.

If the sequencing is fine enough, it is possible to constitute virtual sequences whose duration is limited by a relatively low upper limit. Thus, according to one embodiment, the duration of the selected virtual sequences is limited by an upper limit. For example, the time between the two markers of a sequence marker pair is less than 2 minutes, 1 minute, or 30 seconds.

According to one embodiment, at least one additional step of enriching the indexing of the virtual sequences by exogenous secondary descriptors is carried out in step g.

Thanks to this arrangement, the sequencing can be repeated to end up with finer sequencing, since additional - exogenous - information has been added.

According to one embodiment, the secondary descriptors by means of which the identified sequences are indexed are enriched with a numerical or lettered indicator, such as an overall score of a digital collection card, calculated for each sequence from the secondary descriptors the virtual sequence and/or the primary descriptors of the digital video file in which the sequence was identified.

Thanks to this provision, the results of a subsequent sequence search in the secondary index can be ordered on the basis of this encrypted or lettered indicator.

According to one embodiment, the action modality comprises the sub-modalities: {detection of change of shots, detection of action according to a typology of digital video files}, and each of the sub-modalities of the action modality makes it possible to generate a game particular of unimodal cut-out time markers.

Thanks to this arrangement, as many unimodal cutting time marker sets as there are sub-modalities (a modality not containing a sub-modality being counted as a unique submodality) can be obtained, so that the sequencing will make it possible to produce coherent sequences according to N submodalities, N being between one and the total number of submodalities, the sequencing being able to identify multimodal sequence markers based on 1 to N sub-modalities. The sequencing is therefore finer and presents a greater variety of points of view than in the case where the sub-modalities of the same modality are not distinguished.

According to one embodiment, the analysis according to the audio modality comprises noise detection, music detection and/or transcription of speech into a text stream.

Thanks to this arrangement, the different aspects of audio modality can be taken into account for the search for unimodal slice markers.

According to one embodiment, the analysis according to the image modality includes the sub-modalities {shape or object recognition; plan aggregation; optical character recognition}, and each of the sub-modalities of the image modality makes it possible to generate a particular set of unimodal descriptors.

Thanks to this arrangement, the different aspects of the image modality can be taken into account for the search for unimodal cut markers, on the same principle as what has been described for the sub-modalities of the action modality.

The invention also relates to a computerized method for the automatic production of an ordered playlist of video extracts from digital video files, with a data transmission stream, the digital video files being indexed in a primary index stored in a documentary database containing the digital video files with primary descriptors, the digital video files having been, beforehand and by means of the computerized process of de-linearization according to one of the preceding embodiments, cut virtually by time stamping into virtual sequences which are defined by two sequence time markers forming a pair of sequence markers and by associated secondary descriptors, the pairs of virtual sequence markers and the associated secondary descriptors being stored in a secondary index stored in a documentary database, the index secondary being in inheritance relation with the primary index these index being accessible via a graphical interface. The computerized process of research and automatic production of a playlist of video extracts includes:

1. the formulation of at least one search query;

2. transmitting said search query to a search server associated with the database;

3. the determination and reception from the documentary database, in response to the transmitted search query, of the search result which is an automatic list of pairs of time stamps of sequences and of the associated descriptors, according to an order which is a function of the descriptors associated with each virtual sequence and the formulation of the search query, the virtual sequences being identifiable and able to be searched by the secondary descriptors and the primary descriptors;

4. redisplay and viewing from a virtual remote control of the playlist which presents all the video extracts associated with the automatic ordered list of time stamp pairs received during step 3, without creating a new digital video file, the virtual remote control allowing navigation on the playlist, each video extract from the playlist being associated with a virtual sequence, and being called upon viewing of the playlist via the data transmission stream from the digital video file indexed in the primary index in which the virtual sequence indexed in the secondary index has been identified.

In the computerized process for the automatic production of a playlist of video extracts,

- the stored digital video files have been sequenced, and the virtual sequences of the digital video files have been indexed in the secondary index before the search criteria are formulated and before the search result is received by the client by means of the sequencing process as described above;

- the ordered automatic playlist is a list of video sequences of the digital video file(s) each corresponding to a virtual sequence of a digital video file, according to an order which is a function of the secondary descriptors associated with each sequence and primary descriptors associated with each video file digital. Thanks to this arrangement, it is possible to select one or more sequences of digital video files obtained at the end of the process for sequencing one or more digital video files, that is to say in an automated manner without required that the user view one or more digital video files in their entirety.

This selection can be made by means of a search query and the search is carried out in the secondary index containing the secondary descriptors of the sequences, which is linked to the primary index containing the primary descriptors of the digital video files from which the sequences.

According to one embodiment, when determining the search result: - in a sub-step 1), the method determines according to the search query and the descriptors of the virtual sequence(s), whether the virtual sequences are essential (the number of descriptors is relevant) or ornamental (the number of descriptors is not relevant with respect to the criterion defined for the essential virtual sequences);

- in a sub-step 2) when the pairs of virtual sequence time stamps constituting the automatic list are identified in a single digital video file, the method produced via the transmission stream is an exhaustive playlist video extracts associated with all the essential virtual sequences, or a summary with a selection of video extracts associated with the essential virtual sequences according to criteria specified by the user,

when the pairs of virtual sequence time markers constituting the automatic list are identified in several digital video files, the method produces via the transmission stream a playlist of video extracts associated with the so-called "zapping" virtual sequences, of these digital files with a selection of the essential virtual sequences associated with the video extracts according to criteria specified by the user.

According to one embodiment of the computerized method for automatically producing an ordered playlist of video extracts from digital video files,

- when the pairs of virtual sequence time markers constituting the automatic list are identified in a single digital video file, the method produces via the transmission stream a summary playlist with a selection of video extracts from this digital video file according to criteria specified by the user during his search,

when the pairs of virtual sequence time markers constituting the automatic list are identified in several digital video files, the method produces via the transmission stream a playlist of video extracts associated with the so-called “zapping” virtual sequences, of these digital files with a selection of video extracts according to criteria specified by the user during his search.

According to one embodiment, the computerized method for automatically producing a playlist of video extracts allows, after automatic production of an ordered playlist of video extracts from digital video files, the following navigation operations from the virtual remote control and from the data transmission stream:

- playing, stopping and resuming the current excerpt from the playlist which includes all the video excerpts associated with the automatic list obtained in step 3;

- pointing of an extract in the playlist which includes all the video extracts associated with the automatic list obtained in step 3 by fast forward or fast reverse;

- temporary output of the extract from the playlist which includes all the video extracts associated with the automatic list obtained in step 3 to view the original digital video file of the extract, without temporal constraints linked to the start time markers and end of the virtual sequence associated with the video extract.

Advantageously, this comprises a single navigation bar for all the video extracts arranged one after the other on the playlist, according to the order of the sequence markers according to the user's request (which presents the descriptors associated with the markers cutting in the secondary index).

Thanks to this arrangement, it is possible from a sequence identified as interesting for the user with respect to his search criteria to proceed, at the user's choice, to reading the rest of the file in which this sequence has been identified, or to move on to another sequence identified as interesting.

According to one embodiment, the method for automatically producing an ordered playlist of video extracts from digital video files allows the following additional operation: d. new temporary output from the viewing of the original digital video file of the extract being played from operation c), to view during step d) a summary created automatically and prior to this viewing from this single original digital file.

According to one embodiment, the method for automatically producing an ordered playlist of video extracts from digital video files allows the following additional operation: e. recording of browsing history on the playlist of video sequences and creation of a new digital file which is this browsing history.

According to one embodiment, the search query formulated in step 1 is multi-criteria, and combines a search on the full text, a faceted search and in that the criteria for carrying out the order for the automatic playlist include criteria chronological and/or semantic and/or relevance.

This arrangement makes it possible to formulate search queries as varied as possible, including with suggestions based on facets or criteria, and to obtain an ordered list of results.

According to one embodiment of the method for automatically producing an ordered playlist of video extracts from digital video files, the search query formulated in step 1 is carried out automatically on the basis of one or more criteria specified by the user chosen from a list comprising: the desired duration of an automatic playlist as well as semantic criteria.

In this way, the search for sequences in digital video files can be fully automated based on minimal search criteria.

According to one embodiment of the computerized method for automatically producing an ordered playlist of video extracts from digital video files, the search query formulated in step 1 is carried out by a conversational robot.

According to one embodiment, the computerized method for automatically producing an ordered playlist of video extracts from digital video files comprises a viewing step in which the user displays on a first screen a video extract from the playlist, and descriptors of the virtual sequence associated with the video extract on a second screen synchronized with the video extract. According to one embodiment, the computerized method for automatically producing an ordered playlist of video extracts from digital video files comprises a viewing step in which the descriptors associated with the virtual sequences are displayed on the extracts. Thanks to these arrangements, the user can view, at the same time as the video extracts, the descriptors on the basis of which the method has considered the sequence as relevant with respect to the search query. In this way, the user can both assign a global meaning to the video extract and compare it to the global meaning which could be attributed to it on the basis of the descriptors which have been automatically associated with it.

According to one embodiment of the computerized process for the automatic production of an ordered playlist of video extracts from video files, the technology used is ElasticSearch®.

According to one embodiment of the computerized process for the automatic production of an ordered playlist of video extracts from digital video files, access to the video files is done in “streaming” mode.

The invention further relates to an automatic list of pairs of sequence markers and associated descriptors resulting from the computerized method of automatically producing an ordered playlist of video extracts from digital video files, presenting endogenous and exogenous descriptors consistent with the request of research.

According to one embodiment, in the automatic list of pairs of sequence markers and associated descriptors resulting from the computerized method of automatically producing an ordered playlist of video extracts from video files, all the virtual sequences (therefore all the pairs sequence markers) have, as end-of-sequence marker, at least one main multimodal sequence marker or sequence marker resulting from three modalities.

According to one embodiment, in the automatic list of pairs of sequence markers and associated descriptors resulting from the computerized process for the automatic production of an ordered playlist of video extracts from video files, the end of sequence marker of each pair of sequence time markers corresponding to each virtual sequence is derived at least from the action modality.

According to one embodiment, in the automatic list of pairs of sequence markers and associated descriptors resulting from the computerized method for the automatic production of an ordered playlist of video extracts from video files, the sequence time markers are determined by an approach multimodal by automatic analysis, file by file, of each of said one or more digital video files, according to at least two of the four modalities: image modality, audio modality, text modality, action modality.

According to an embodiment of the automatic list, at least two sequence time markers are determined randomly or unimodally.

The invention also relates to a computerized method of editing with virtual cutting without creating a digital video file, from the computerized method of automatic production an ordered playlist of video clips from digital video files comprising the following steps:

I. automatic production of at least one ordered playlist of video extracts from digital video files and recording of the at least one automatic ordered list of sequence time stamp pairs and associated descriptors resulting from this production, without file creation digital video;

II. navigation on at least one automatic playlist of video extracts from digital video files, per data transmission stream

III. selection by the user of one or more virtual sequences associated with the at least one automatic playlist of video extracts from digital video files to produce a new playlist of video extracts whose order can be modified by the user.

According to one embodiment, the computerized method of assembly with virtual cutting comprises the following steps:

- modification of the automatic playlist of video extracts by adding and/or removing video extracts from the playlist;

- modification of one or more video extracts by extending or reducing the duration of the virtual sequences associated with the video extracts of the playlist, by moving the start and end markers of each virtual sequence;

- modification of the video extracts by a visual effect or a sound effect.

Thanks to this arrangement, a new video can be edited in a very automated way, without manipulation of digital video files to aggregate or cut them. Editing saves memory and computation time since it is based on the manipulation of sequence markers.

According to one embodiment of the browsing method, the playlist of video extracts is generated automatically by a computerized method of searching and automatically producing a playlist having ordered video extracts according to one of the embodiments described above. The invention further relates to the use of video extracts or a playlist of video extracts obtained by the computerized method of research and automatic production of a playlist, or by the editing method according to one of the embodiments described above, in a social network or in a search engine or to constitute a new digital video file. The invention finally relates to a computerized system comprising:

- At least one acquisition module for one or more digital video files;

- At least one splitter module;

- At least one multimodal analysis module;

- At least one sequencing module generating sequences of indexed digital video files; - At least one search module comprising a client making it possible to formulate a search query for the implementation of the steps:

1. One or more digital video files to be analyzed are received via the acquisition module;

2. Each of said digital video files is automatically indexed in a primary index, based on the endogenous, so-called primary, descriptors of said digital video file;

3. The audio, image and text data streams are extracted from each of the digital video files;

4. By means of a plurality of computerized devices implementing an automatic learning algorithm chosen and/or trained for a typology of video files defined beforehand and contained in the multimodal analysis module, a file analysis is carried out by file of each of said one or more digital video files according to the four modalities: image modality, audio modality, text modality, action modality, the analysis automatically producing one or more unimodal cutting time markers for each of the modalities, one or more descriptors being associated with each of the single-mode slice time markers;

5. At the end of the analysis of each of the digital video files, candidate sequence time markers are provided, with the aim of determining virtual sequences, and the descriptors associated with these candidate sequence time markers, which are :

- either unimodal cutting time markers of said digital video files which are called at the end of this step unimodal candidate sequence time markers;

- either, for each of said digital video files taken in isolation, the time codes corresponding to said unimodal cutting time markers are compared and, each time that at least two unimodal cutting time markers resulting from different analysis modalities are separated by a time interval less than a main predetermined duration, a plurimodal candidate sequence temporal marker, in mathematical connection with the at least two unimodal cut markers, is created;

6. For each of said analyzed digital video files, a lower limit and an upper limit are defined according to the type of said digital video file for the duration of a sequence and pairs of sequences are automatically selected from the candidate sequence markers. sequence markers, called start and end of sequence markers, each pair of sequence markers having a start of sequence marker and an end of sequence marker, such that the duration of each sequence retained is between said lower limits and superior, these pairs of sequence markers being associated with the descriptors associated with the said selected candidate sequence temporal markers, these descriptors therefore being referred to as “secondary endogenous descriptors”;

7. We index by means of the sequencing module in a secondary index which is in relation of inheritance compared to the primary index, all the pairs of sequence markers by means of the associated descriptors allowing the identification of each sequence, the sequences being identifiable and able to be searched at least by the secondary endogenous descriptors and the primary endogenous descriptors;

8. A search query for sequences of digital video files is formulated using the search module; each of the modules comprising the necessary calculation means, each of the modules other than the dispatcher module communicating with the dispatcher module and the dispatcher module managing the distribution of the calculations between the other modules.

According to one embodiment of the computerized system, this system further comprises at least one module for enriching the primary descriptors of the digital video files and/or the secondary descriptors of the virtual sequences of digital video files by exogenous complementary descriptors.

According to one embodiment of the computerized system, this system further comprises video editor module communicating with the research module.

BRIEF DESCRIPTION OF DRAWINGS

Embodiments of the invention will be described below with reference to the drawings, briefly described below:

Fig. 1 represents a flowchart of a device making it possible to implement the method of analysis, sequencing and indexing of the sequences of a digital video file.

Fig. 2a represents a first step in sequencing a digital video file according to the four modalities: image, audio, text and action.

Fig. 2b represents a second step of sequencing a digital video file according to the four modalities: image, audio, text and action.

Fig. 2c represents a third step of sequencing a digital video file according to the four modalities: image, audio, text and action.

Fig. 3 represents the different interactions between the modules and the services of the computerized process in connection with the possible actions of the user.

Fig. 4 represents the steps of an iteration of the method for sequencing a video file on the basis of four modalities. Fig. 5a represents a graphical interface 55 for editing or viewing a playlist.

Fig. 5b shows another embodiment of a graphical interface for editing or viewing a playlist.

Fig. 6 schematically represents the effect of the manipulation of the virtual remote control on the playlist.

Fig. 7a shows a third embodiment of a graphical interface 55.

Fig. 7b shows a fourth embodiment of a graphical interface 55.

Fig. 8 shows a fifth embodiment of a graphical interface 55.

Fig. 9 shows a sixth embodiment of a graphical interface 55.

Fig. 10 shows a seventh embodiment of a graphical interface 55.

Fig. 11 shows an eighth embodiment of a graphical interface 55.

Fig. 12 shows a ninth embodiment of a graphical interface 55.

In the drawings, identical references designate identical or similar objects.

DETAILED DESCRIPTION

The invention relates to a method for the analysis, sequencing and multimodal indexing of digital audiovisual data. The format of the audiovisual data is not limited a priori. By way of example, the digital video file formats MPEG, MP4, AVI, WMV of the ISO/IEC standard can be considered.

The audiovisual data may be available on the Internet, on a public or private digital video library, or even provided individually or in a group by a particular user.

Metadata is integrated into the audiovisual document, in particular technical metadata: compression level, file size, number of pixels, format, etc. cataloging: title, year of production, director, ...

This metadata will be referred to as "global" metadata insofar as it is associated with the digital video file as a whole.

In general, as will be seen below, it is not necessary for the digital video file to be structured for the audiovisual de-linearization method according to the invention to work. A digital video file without any cataloging metadata can be sequenced automatically by the method according to the invention without human intervention. This is one of the strengths of the method compared to the sequencing methods of the prior art.

In particular, even if the audiovisual de-linearization process can be implemented on structured digital video files, such as those used in "broadcast" type distribution processes, it is particularly relevant in the case of a video file unstructured or weakly structured digital material, such as those generally available on the Internet or used in "multicast" type broadcasting processes, for example YouTube® videos.

The method comprises several steps traversed in a non-linear manner, requiring its implementation on a computerized device 8 for sequencing a digital video file, an embodiment of which is shown in FIG. 1, comprising several modules:

- An acquisition module 1, allowing the recovery of one or more video files from various sources and their indexing by means of so-called primary descriptors in a primary index;

- A module 2 splitter;

- A module 3 of multimodal analysis;

- An optional metadata enrichment module 4;

- A sequencing module 5 generating virtual sequences (or even virtual fragments) of the digital video file(s) and indexing them in a secondary index by means of secondary descriptors;

- A research module 6, comprising the client making it possible to carry out a search on the sequences generated by the module 5 for one or more digital video files.

- Optionally an enrichment module 4.

- Optionally a module 7 video editor comprising a graphical interface allowing manipulation of virtual sequences produced following a search for virtual sequences by module 5.

We will speak of a “virtual sequence” or equivalently of a “virtual fragment” of a digital video file. A virtual sequence of digital video file (to simplify in the following: sequence of digital video file, even sequence) designates a virtual fragment of the initial digital video file, of shorter duration than that of the initial file, in which the succession of images between the beginning and the end of the fragment is exactly the same as that of the initial digital video file (or original, or in which the virtual sequence was identified) between the two corresponding instants, without a specific new digital video file to the sequence is constituted at the physical level.

A virtual sequence of a digital video file is therefore constituted solely by the data of a pair of sequence time markers, comprising a start of sequence marker and an end of sequence marker.

Each time stamp corresponds to a particular timecode in the original digital video file.

When a virtual digital video file sequence is identified, are only stored in memory, for example in a documentary database, the pair of corresponding sequence time markers as well as the descriptors allowing its indexing and thus access to the virtual sequence by a search in the index.

A virtual digital video file sequence is therefore systematically indexed by means of one or more semantic descriptors, called secondary descriptors. The space in storage memory used to memorize these "virtual" sequences corresponds to the space necessary to store the pairs of markers temporal and the associated secondary descriptors. This is what sequencing is called virtual.

In other words, it is not necessary to create a new digital video file per virtual sequence, which would be a copy of a fragment of the original digital video file, in which the sequence was identified.

The sequencing and indexing method according to the invention is therefore particularly inexpensive in terms of memory.

A virtual sequence of digital video file allows in a second time, in particular according to the needs of the user, the extraction of a "real" fragment of a digital video file, that is to say the constitution of a "video clip" of a digital video file.

The constitution of a video extract from a digital video file can for example take the form of modifications in the random access memory of a processor by viewing the content between the two sequence markers of the chosen virtual sequence, in particular in streaming, in particular after a decompression stage. This visualization of the video extract does not require the constitution of a new digital video file and directly calls up the passage or the fragment of the original digital video file thanks to the virtual sequence.

The constitution of a video extract can possibly in certain cases materialize in a storage memory by the recording of the fragment of digital video file associated with the virtual sequence in the form of a new digital video file which can be of smaller size than that of the digital video file in which the corresponding virtual sequence has been identified.

The acquisition module 1 makes it possible to copy from various storage sources and to record on a suitable storage device one or more digital video files that one wishes to analyze.

The storage device may contain other files already acquired and its content is increased as the device is used. Preferably, the storage device allows access to the video file in “streaming” mode.

In particular, it is possible to download thematic videos to be analyzed via Web connectors, on the basis of a search query formulated on a Internet search. It is also possible to copy all or part of the digital video files from another storage device, such as a USB key or an archive server for example. The set of digital video files acquired by the module 1 can be homogeneous from a content point of view or heterogeneous.

We can for example consider acquiring digital video files on a date criterion, like all video files filmed on a specific day. In this case, the set of digital video files will have no a priori reason to be homogeneous from the content point of view. Or, one or more digital video files can be acquired based on a combination of keywords. For example, one can consider acquiring all the digital video files corresponding to league 1 football matches in France for a given year. All the files then present content related to football. By way of example, the operation of the process will be described several times on this particular case of football. It is important to note that this example, homogeneous in the sense defined above, is in no way limiting and only serves to promote understanding of the process. The process can be implemented in any field (sport, online courses, scientific conferences, television news, amateur videos, cinema, etc.) or even in several fields at the same time. We will speak equivalently of domain or typology of the digital video file. A domain or even a typology can in particular be described using semantic descriptors.

The different modules are made up of physical or virtual machines, and therefore of one or more processors. The machines are organized into farms (“cluster” in English). The device comprises at least one master node (“master” in English) which interacts with a plurality of “worker” nodes called “workers”. Each of the nodes, master and "workers", encapsulates at least the applications, storage resources, means of calculation necessary for the realization of the task or tasks to which it is dedicated.

Any container orchestration solution that automates the deployment and scaling of the management of containerized applications can be considered for the creation of this “cluster”. By way of non-limiting example, the ElasticSearch® technology, available in Open Source, may be used.

The digital video files acquired by the module 1 are therefore stored, for example in a documentary database, and they are further indexed in a so-called "primary" index, making it possible to find and access each of the digital video files in his outfit.

The primary index is for example contained in the documentary database. The indexing of a given digital video file in the primary index is done by means of so-called “primary” descriptors. This is for example all or part of the metadata of the digital video file. The database is document-based, as opposed to relational, in the sense that searching the database is not based on a relational model or limited to an SQL-like language based on algebraic operators, such as this will be described later.

Each digital video file acquired by the acquisition module 1 is transmitted to the dispatcher module 2 which is a master node. The dispatcher module 2 receives and distributes the requests on the "worker" nodes suitable for the execution of the requests and available for this execution.

Possible redundancies of "worker" nodes may be considered but will not be described in detail here.

After receiving a digital video file, if the metadata of the digital video file allows it, the dispatcher module 2 can launch a preliminary and optional step of enriching the metadata at the level of the enrichment module 4.

The enrichment module 4, which is a "worker" node, is in particular connected to external databases, such as databases (4a) that are free to access and use (Open Data), web services (4b) or other databases (4c), private in particular.

For example, from the metadata of a digital video file of a football match containing information on the date, the place and the teams present, one can imagine recovering, thanks to the enrichment module 4, data such as the names of the players, the colors of the jerseys, or even a possible textual sequence of the match if it exists. However, this preliminary step is not essential for the implementation of the method and it may not be executed or may not result in any effective enrichment of the metadata initially associated with the digital video file.

The method is based on techniques of automatic de-linearization of the digital video file based on the content. By delinearization is meant the discovery and/or recognition of underlying structures in a digital file, in particular a digital video file, without human intervention. The de-linearization is, in the context of the invention, based on the content of the digital file, including the metadata, enriched or not beforehand.

Just after the acquisition of the digital video file or after the preliminary enrichment step, the dispatcher module 2 can initially trigger four analyzes at the level of the multimodal analysis module 3.

Multimodal analysis module 3 is a “worker” node on which four different computerized devices are implemented, each implementing an automatic learning algorithm. These are, for example, four different neural networks. These neural networks analyze the digital video file with different viewpoints in parallel.

Each of these neural networks is chosen appropriately to extract temporal markers of potential cutting of the digital video file into sequences having coherence, i.e. meaning, with respect to a particular point of view of analysis .

The image stream (equivalently video stream) of the digital video file can be considered, among other things, as an ordered collection of images. We can therefore assign a sequence number to each image, allowing it to be found within the digital video file.

Within the meaning of the invention, a cutting time marker corresponds to a sequence number, or equivalently to a given instant during the viewing of the video, the dates being able to be identified with respect to the initial instant corresponding to the first image digital video file. In particular, a cutting marker is associated with a time code (“timecode”).

The neural networks used may in particular be convolutional neural networks (“Convolutional Neuronal Network”, CNN) and/or recurrent.

Each of these neural networks contains several successive layers of neurons, so as to be able to undergo a learning phase of the deep learning type ("deep leaming"), unsupervised, semi-supervised or supervised, preferably pre-trained before its implemented in device 8.

The role of supervision may be more or less important depending on the method of analysis. For example, the analysis of the text and sound streams may, in one non-limiting embodiment, be carried out by a neural network having undergone an unsupervised learning phase, and the analysis of the image stream may implement a network neurons that have undergone a supervised or semi-supervised learning phase.

The number and type of layers are chosen according to the type of analysis to be performed.

A digital video file includes components (also called “flows”) images (or equivalently video), sound (or equivalently audio) and text placed in a container. A digital video file may contain several audio streams and/or several image streams.

The text type stream has things like metadata, subtitles, transcription of the audio stream as text where possible, etc.

It is possible to analyze each of these components, or streams, of the file separately.

The first neural network, called an analyzer according to the image modality (3a), is configured to carry out an analysis of the image flow, image by image. It can in particular carry out analyzes of the type: detection of objects, shapes, color, texture, detection of similar images, ocerization. The analyzer according to the image modality (3a) analyzes the content of each image of the file to be analyzed pixel by pixel. It is, among other things, equipped with an object detection algorithm, preferably capable of analyzing a video stream in real time while maintaining good predictive performance (algorithm available under the name “Yolo3” for example). The analyzer following the image modality (3a) extracts a set of primitives which take into account certain representations such as the contour, the texture, the shape, the color, then it aggregates the results in a single signature allowing the calculations of similarity in particular to through a hybridization between Deep Leaming and unsupervised clustering algorithms (“K Nearest Neighbors”, KNN).

From a set of primitives on the outline, the texture, the shape, the color (shape recognition) the algorithm aggregates the results in a signature allowing similarity calculations in particular through a hybridization between Deep Leaming algorithms and unsupervised clustering (KNN) (plane aggregation).

It also has a feature to search for symbols such as emoticons, which can be added to the digital video file before it is analyzed by user interaction.

In a particular embodiment, the image modality gives rise to an analysis according to at least three sub-modalities:

- Detection of objects, shapes

- Recognition of text in images (stopwatches, scores, text on players' shirts, text in teaching presentation slides, etc.) and analysis of this text ("ocerization")

- shot aggregation: similar shots detected in images analyzed one by one are grouped together.

The second neural network is a so-called sound analyzer network (3b) or equivalently an analyzer according to the audio modality or according to the sound modality. It is equipped with an audio track separator and an activity detector for speech, noise, music, ...

It allows the analysis of the audio stream by carrying out, for example, a detection of musical sequences, dialogues or at least words, noise, silence, or even a detection of sound environments...

The third neural network (3c) is a text flow analyzer or equivalent analyzer depending on the text modality, for example metadata, subtitles when available, or text obtained after a "speech" type text extraction to text” on the basis of known voice recognition technologies, or even “video tagging” information described later.

Using NLP (Natural Language Processing) algorithms implemented on the text (e.g. from speech-to-text transcription) ("speech to text")), the analyzer following the text modality (3c) cuts sentences, paragraphs into units of meaning translating a change of subject, or the continuation of an argument according to models of the analysis of the speech.

The analyzer following the text modality (3c) can also, via an automatic language processing (T.A.L) platform, possibly Open Source, extract semantic metadata to feed structured fields from the full text coming from module 4, for example from web sources and/or social networks.

The fourth neural network (3d) is an analyzer of the video stream as a whole, in order to create cutting markers based on dynamic notions, such as the notion of action or shot changes. This modality of analysis will be called equivalently action modality or event modality.

The role of this analyzer following the action modality (3d) is to define a typology of actions for the digital video file to be analyzed, possibly without prior knowledge of these actions.

In the table tennis example, the actions could include the phases of actual play as opposed to the phases during which the players are not playing, for example: waiting for the next serve, picking up the ball, ...

Specific actions, such as a forehand or an offensive or defensive backhand can be identified.

The analyzer following the action modality (3d) first detects the changes of shots. It should be noted that the changes of shots are generally not made randomly by an editor, so they can carry rich information, which can be found at least partially thanks to this detection of the changes of shots. The characteristic images of each plane are then sent to the analyzer according to the image modality (3a). In a second step, the information returned by the analyzer according to the image modality (3a) is analyzed in the analyzer according to the action modality (3d) by an action detection algorithm.

For example, a dense pose estimation system can be implemented, which associates the pixels of two successive images based on the intensities of the different pixels to match them with each other. the other.

Such a system can perform “video tracking” without sensors having been positioned on the animated objects/subjects present in the video content. In particular, it is possible to detect parts of the human body and therefore to follow the trajectory of a given football player, for example.

A stock bank can be set up with a view to a supervised learning phase, thanks in particular to this estimation. In the table tennis example, the analysis of a player's arm gesture on a set of digital video files each containing a sequence of well-identified offensive forehands allows the neural network to recognize, based on the successive positions of a player's arm, an offensive forehand in a video file that was not used for training.

An offensive forehand ("topspin"), whose gesture is made from bottom to top, is for example different from a defensive forehand ("cut"), whose gesture is made from top to bottom.

Actions can be defined outside the context of sport. In the field of political news videos, a handshake between two subjects can be an action in the sense of the invention, and a neural network can learn to recognize such an action.

In the field of education, the fact that a teacher writes on a board can constitute an action.

The analyzer following the action modality (3d) can also exploit the sound associated with the images. Thus, for educational videos, an interruption in the flow of the speaker can be indicative of a change of action in the sense of these videos, that is to say the passage from one sequence of the course to another sequence.

The analyzer following the action modality (3d) can also exploit "video tagging" information, i.e. metadata of the keyword type added manually to the digital video file, when they are relevant from the point of view of view of the actions that have been identified.

In a particular embodiment, the action modality gives way to at least two sub-modalities:

- The first sub-modality is the analysis (or equivalently the detection) of shot changes

- The second sub-modality is action detection in the sense of a typology, such as a typology of digital video files or gesture or motion.

The method can include the phase of training the neural networks on a set of video files associated with a particular domain, for example a set of video files relating to a particular sport, or a particular scientific field. It can also be implemented on neural networks previously trained for a domain chosen by G user for example.

At the output of the multimodal analysis module 3, at least four sets of unimodal cutting time markers, each originating from a modality, or even from a sub-modality of a modality, can be provided for the digital video file, each unimodal cutting time markers being associated with a set of semantic descriptors, called endogenous unimodal descriptors.

It is recalled that in particular, the analyzers according to the image (3a) and action (3d) modalities can provide sets of unimodal temporal markers according to several sub-modalities. For example, different unimodal cutting temporal markers can be identified according to one or more of the sub-modalities: change of planes,

- aggregation of shots (by similarity, for example from the same camera)

- detection of objects, shapes,

- ocerization.

If we distinguish the sub-modalities, it is therefore possible to obtain more than four sets of unimodal cutting time markers.

It is also of course possible to reduce the number of modality or sub-modality of analysis to provide less than four sets of unimodal cut-out time markers. In this case, the sequencing will be less fine.

Within the meaning of the invention, a descriptor is a term, which may be a common noun or a proper noun, an adjective, a verb, a phrase, a compound word or a group of words, and which represents a concept. Only descriptors or combinations of descriptors can be used for indexing. The non-descriptors may, however, be used in the formulation of the search request at the level of module 6 of research and assembly.

The descriptors can optionally be defined in a thesaurus specific to the device or come from existing thesaurus. A descriptor therefore makes it possible, in documentary language, to specify the content of the digital video file when it is associated with the digital video file as a whole, or of a sequence of digital video file when it is associated with the latter.

The analysis step can be performed based on minimal metadata. The following schematic example helps to understand the different steps of the process. Let's assume that a user of the device wants to analyze a video:

- that he does not know a priori, that he does not wish to view beforehand

- for which it only has a meaningless file name

- whose audio track does not allow the extraction of significant textual content. For example, it contains only noise without identifiable words, or background music without words and unrelated to the image content.

Typically, the example digital video file is an "example 1" amateur video file, made during a football match and in a very noisy sound environment so that any words cannot be highlighted in the noise of background. A first analysis by module 3 of multimodal analysis makes it possible to bring out a few descriptors of the ball, football, jersey type (and their colors), names of certain players, football stadium soundscape, corresponding to a relatively coarse sequencing after processing of the results of module 3 of multimodal analysis by module 5 of sequencing which will be described later. The dispatcher module 2 can optionally enrich the unimodal descriptors identified and associated with the unimodal cutting time markers by exogenous descriptors, either by transmitting them to the enrichment module 4, or from the descriptors already identified and stored in the device itself. , especially in the primary and secondary indexes.

In the case of "Example 1", by searching the Internet for data containing the keywords "ball, football, players' names", additional descriptors, or equivalent exogenous descriptors, such as "match, goal, half-time, ...” may be added. Such exogenous descriptors can also be found on the device's database if it has already analyzed other video files such as football matches.

If the enrichment module 4 has intervened, the dispatcher restarts an analysis step by the multimodal analysis module 3 on the basis of these enriched descriptors. This new step generates more unimodal cut-out time markers and/or more adapted to the analyzed video. For example, a second stage of analysis of the "example 1" video following the enrichment of the descriptors by the enrichment module 4 will make it possible to obtain a sequencing on the basis of the two halves and the goals scored if these events are identified.

Module 3 of multimodal analysis used a priori can be "generalist", i.e. adapted to digital video files whose content is as varied as possible, or even specialized by learning on an ad hoc video game.

If you want to analyze videos from a sports point of view, a multimodal analysis module 3 dedicated to and trained in this area, or even in a specific sport, can be implemented. But it is possible to analyze the same video with several 3 multimodal analysis modules dedicated to several different domains to obtain different sequencing, or to use a set of 3 modules to change the choice of the 3 multimodal analysis module as the metadata are enriched to move towards a multimodal analysis module 3 increasingly adapted to the content of the digital video file, on which the device had no a priori knowledge of the domain of the content.

In this last case, a redundancy of module 3 of multimodal analysis is therefore necessary, each of the modules 3 of multimodal analysis being adapted to a particular and/or general field.

In a particular embodiment, the multimodal analysis module 3 can only analyze the file according to two methods, for example if one of the streams of the file is not usable, or if one wishes to favor these two modalities.

At the end of a step in module 3 of multimodal analysis, and of a possible step of intermediate enrichment in module 4 of enrichment, the temporal markers of unimodal cutting and the endogenous, and possibly exogenous, associated unimodal descriptors are transmitted by the dispatcher to the module 5 of sequencing.

Sequencing module 5 is also a “worker” module. The sequencer synthesizes all the information collected by the dispatcher to create homogeneous, coherent and relevant sequences, if possible according to several of the points of view used in module 3 of multimodal analysis at the same time.

In the example shown in Figs. 2a, Fig. 2b and Figs. 2c, the horizontal axis represents the time axis for the digital video file, that is to say the order of appearance of the various images which constitute it; the unimodal slice time markers associated with the image modality are for example represented on the top line, the unimodal slice time markers associated with the audio visual modality on the line, just below, then again below the time markers of unimodal cutouts associated with the textual modality, and finally the unimodal cutout temporal markers associated with the action modality are represented on the bottom one.

At the end of the sequencing, the sequencing module 5 proposes candidate sequence time markers.

A candidate sequence time stamp is:

- either a multimodal candidate sequence time marker,

- or a unimodal candidate sequence marker.

To create a plurimodal candidate sequence temporal marker, one proceeds as follows: if at least two unimodal cutting temporal markers from different modalities are identified as temporally close, a plurimodal candidate sequence temporal marker, in mathematical relation with these temporal cutting markers unimodal, is created.

The temporal proximity is defined with respect to a time criterion T2 specified beforehand: two (or more) of unimodal cutting temporal markers are considered as temporally close if they are separated two-by-two by a duration less than a duration predetermined T2, called main.

A plurimodal sequence temporal marker is created in mathematical connection with the unimodal cutout markers which underlie its creation according to a rule fixed beforehand.

For example, the candidate multimodal sequence time stamp is identical to the single-mode slice time stamp from the audio modality. Or again, it can correspond to the time marker closest to the mean of the time codes of the n unimodal cutting time markers identified as temporally close. A unimodal candidate sequence time stamp is created based on a single modality. In this case, it is said to be a unimodal candidate sequence time marker and identical to the identified unimodal cut-out time marker.

Figure 2a represents the decomposition of a digital video file according to the four modalities: image, audio, text and action.

In this figure, two candidate sequence time markers 21 plurimodal are detected in this case according to four modalities.

When the four modalities have identical timecodes or timecodes evaluated as close, a candidate sequence marker called main because stemming from four modalities is detected. Candidate sequence markers are therefore said to be “main” when they come from the four modalities. The two candidate sequence temporal markers 21 of FIG. 2a are therefore principal multimodal.

Endogenous plurimodal descriptors, called “main” because they come from the four modalities, are associated with each of the 21 main plurimodal candidate sequence temporal markers identified.

FIG. 2b represents the breakdown of the same digital video file as for FIG. 2a according to the four modalities: image, audio, text and action.

This decomposition leads initially to the detection of three main candidate sequence temporal markers 21 , resulting from four different modalities.

Multi-modal, but only three-modality candidate sequence temporal markers 22 can be identified.

When three modalities have identical timecodes or timecodes rated as close, a sequence marker is identified. This plurimodal candidate sequence marker is said to be secondary because it is plurimodal but stems from less than four modalities. The secondary plurimodal candidate sequence marker is associated with endogenous plurimodal descriptors, called secondary because they are plurimodal but come from less than four modalities.

In all cases, a multimodal candidate sequence marker, whether primary or secondary, can be associated with endogenous multimodal (or equivalently multimodal) descriptors, derived from the unimodal descriptors associated with the unimodal cut-off temporal markers of all the modalities which made it possible to select the multimodal marker.

The descriptors are said to be "endogenous" when they come from the sequencing of the digital video file by the sequencing module (5) but not from an enrichment step by the module (4) from information exogenous to the video file digital.

Four secondary candidate multimodal cut-out time markers 22 from three modalities can be seen in Figure 2b. When only two modalities have identical time codes or time codes evaluated as close, a proximity threshold being able to be predetermined, a multimodal candidate cutting marker called "secondary", because multimodal but resulting from less than four modalities, is identified, to which are associated endogenous multimodal descriptors, called secondary because multimodal but resulting from less than four modalities, in a second step.

This case is represented in FIG. 2c, still for the same digital video file as in FIG. 2a. The sequencing allows the detection in a first stage of main plurimodal candidate sequence markers 21, in a second stage of secondary plurimodal candidate sequence markers 22 resulting from three modalities, then in a third stage of secondary plurimodal candidate sequence markers 23.

Preferably, the multimodal candidate cut markers are therefore initially chosen by temporal proximity out of four modalities, which leads to the choice of the main multimodal candidate sequence markers 21.

If the criterion of temporal proximity over at least four different modalities or submodalities results in insufficient sequencing, secondary multimodal sequence markers 22 or 23 can be selected based on a combination of two or three modalities.

The sequencing is considered “insufficient” on automatically assessable criteria. For example, if at least one time interval separating two successive candidate sequence markers has a duration greater than a predetermined duration, called the threshold duration T1, defined for example in relation to the total duration of the digital video file or absolutely, the sequencing is insufficient.

Once the candidate sequence time markers have been identified, a selection is made from among these candidate sequence markers to constitute one or more pairs of sequence markers, each comprising a start of sequence marker and an end of sequence marker.

In one embodiment, the duration of a sequence is, to do this, limited by a minimum duration D 1 and by a maximum duration D2 which depend on the type of digital video file to be sequenced.

Then, a last end of sequence marker can be, to initialize the constitution of pairs of sequence markers, placed from the end of the digital video file, either exactly at the end of the file, or for example at the level of a candidate sequence time stamp provided it is separated by a time interval less than a predetermined threshold from the end of the file.

Then, it can be considered to proceed with iterations of the following steps: - A multimodal candidate sequence marker separated by a duration between the durations and D1 and D2 of the last end-of-sequence marker is sought. If it exists, it is effectively retained as the last sequence start marker and associated with the last sequence end marker to constitute the last pair of sequence markers, which delimits the last virtual sequence.

If a multimodal candidate sequence marker is found at a duration less than D 1 from the last end-of-sequence marker, it can thus be decided not to retain it because the sequencing would result in sequences that are too short for them to be really of interest.

- Otherwise, if no multimodal candidate sequence marker is identified below the D2 duration, a unimodal candidate sequence marker separated by a duration between the durations and DI and D2 of the last end-of-sequence marker is sought . If it exists, it is selected as the last start-of-sequence marker and combined with the last end-of-sequence marker to form the last pair of sequence markers, which delimits the last virtual sequence.

- Failing this, a last sequence start marker is created, separated by a duration D2 from the identified cutting marker, so as to ensure the convergence of the process.

- Then the search process is repeated to select the penultimate sequence start marker, the last sequence start marker playing the role of penultimate sequence end marker in the algorithm described just above. above.

- And so on until the beginning of the digital video file is reached.

Each time a pair of sequence markers comprising a start of sequence marker and an end of sequence marker is formed, a sequence is therefore virtually formed.

In a particular embodiment, at least one of the sequence tags of each pair of sequence tags is multimodal. Optionally, the two sequence markers of each pair of sequence markers are multimodal.

This arrangement makes it possible to ensure that the identified sequences have a semantic coherence defined by several modalities.

In a particular embodiment, still with the aim of increasing the fineness of the sequencing while maintaining high semantic consistency, at least one of the sequence markers of each pair of sequence markers is main multimodal.

In a particular embodiment, weights can be assigned to the different modalities according to the typology of the digital video file. For example, for “sport” type videos, the action modality can play a more important role in the sequencing if its weight is higher. The weights of the different modalities can optionally be chosen according to the nature of the content analyzed (known a priori or detected as the iterations progress) and/or the video file search criterion formulated by a user of the device 8. Each virtual sequence of digital video file can be indexed in a secondary index by means of the endogenous descriptors, and if necessary exogenous, associated with the start of sequence marker, as well as those associated with the end of sequence marker. The descriptors associated with the start of sequence marker and/or with the end of sequence marker are said to be “secondary” in the sense that they are associated with a digital video file sequence and no longer with the digital video file as a whole. They allow the sequence marker pair to be indexed in the secondary index.

The secondary index is in a relationship of inheritance with the primary index so that the primary endogenous descriptors, associated with the digital video file, are also associated with the identified sequence.

The inheritance relationship is to be understood in the sense of computer science, in particular object-oriented programming: the sequences of a digital video file are "daughters" of this digital file in the sense that if the digital video file is indexed means of endogenous and, where appropriate exogenous, primary descriptors, the sequence inherits these primary descriptors and can therefore be searched in the index not only on the basis of the secondary descriptors which characterize it but also on the basis of the primary descriptors which characterize the digital video file of which it is a "daughter".

Alternatively, the minimum duration of a video file sequence is not fixed a priori but a video file sequence (or equivalently a pair of sequence time stamps) is retained in the secondary index only if it is associated with a sufficient number of descriptors, for example for there to be a significant probability of finding this sequence at the end of a search query.

As we saw above, in the event that it is not possible to find plurimodal sequence markers, unimodal sequence markers can be selected, before an enrichment step and a new iteration of the process of sequencing for example.

Unimodal sequence markers then play the same role as multimodal sequence markers in the indexing process, i.e. the corresponding sequences are indexed on the basis of the associated unimodal descriptors. This scenario is not sought in itself, but makes it possible to ensure the convergence of the sequencing process.

According to one embodiment, information on the unimodal or multimodal character of a given secondary endogenous descriptor is kept during the indexing process. Thanks to this arrangement, it is possible to distinguish the multimodal secondary descriptors from the unimodal descriptors, which can be useful when searching for a video file sequence in which it is desired to make these two types of descriptors play different roles.

In a variant, the analysis of a digital video file is not carried out backwards, but by starting by selecting a first initial sequence marker, then a first end sequence marker and so on until the file has been completely scanned starting from the beginning of the file.

At the end of the video dismantling process, or even of de-linearization, which it carries out, the sequencer therefore indexes in a secondary index all the validated virtual sequences, that is to say all the virtual sequences identified and delimited by a marker of sequence start and an end of sequence marker retained by the sequencing module 5, each of which is associated with a set of endogenous and, where appropriate, exogenous secondary semantic descriptors.

It should be noted that a sequence time marker can be associated by default with the first image and/or the last image, so as to ensure the sequencing of the entire file.

It should also be noted that a preliminary step of reducing the digital video file can be carried out so as to proceed with the sequencing only on the fragments of digital video file of interest.

One can for example imagine eliminating automatically, thanks to specialized neural networks, the fragments corresponding to advertising sequences, or fragments of an amateur digital video file too dark for them to deserve to be kept. . This step reduces the file sequencing time.

The secondary descriptors selected at the end of the sequencing step are secondary because they are not associated with a digital video file in its entirety, like "global" metadata or generally like "primary" descriptors, but they are associated to a particular sequence.

The sequencing module 5 may optionally be a cluster of sequencers, this arrangement making it possible to distribute the requests to the various sequencers of the cluster according to the increase in load of the device.

The process is iterative, i.e. the secondary descriptors associated with a virtual sequence can be enriched by a search for so-called "exogenous" secondary descriptors, such as sequence descriptors already existing in the descriptor database. of the device and/or through the enrichment module 4, before a new sequencing is restarted in order to achieve finer sequencing, on the basis of the endogenous and exogenous primary and secondary descriptors identified. It is also possible to proceed, before the sequencing of a digital video file, to a step of enrichment of the primary endogenous descriptors of this digital video file by exogenous descriptors, also called primary by means of the enrichment module 4 . A digital video file is therefore indexed in the primary index by means of endogenous and, where appropriate, exogenous primary descriptors.

According to one embodiment, information on the exogenous or endogenous character of a given primary or secondary descriptor is kept during the indexing process. Thanks to this arrangement, it is possible to distinguish the endogenous descriptors from the exogenous descriptors, which can be useful when searching for a video file sequence in which one wishes to make these two types of descriptors play different roles.

In the case of "example 1", if the sequences have been defined at the end of a first sequencing step on the basis of the schedule identified for the goals and half-time, it is possible for example to find the corresponding match on the Internet and to enrich the endogenous secondary descriptors of each sequence on the basis of textual information on this match.

Then a new analysis by module 3 of multimodal analysis and a refined sequencing by module 5 of sequencing can be carried out.

Fig. 4 gives a schematic representation of the steps of an iteration of the sequencing process of a video file on the basis of four modalities.

These back-and-forth steps between the multimodal analysis modules 3 and the sequencing modules 5, orchestrated by the dispatcher module 2, can be repeated in a controlled manner either on the basis of a limitation of the number of iterations, or on the basis of a sufficiently fine sequencing of the digital video file. One can for example stop the process when at least one candidate sequence marker has been identified for any specified time interval t, a few seconds as an example. We have seen that the digital video files acquired by module 1 were indexed in a so-called “primary” index, allowing access to the digital video file as a whole. The sequencing module 5 indexes the identified sequences of the digital video file in a so-called “secondary” index.

The process of indexing digital video file sequences is of the parent/child type: the dispatcher's index points to the general information of the digital video file, therefore to the so-called "primary" index, while the sequencer creates an indexing "secondary" inherited. In one embodiment, the primary and secondary indexes are multi-field and mutually feed each iteration. For example, a step of sequencing the video of a football match can cause N sequences to emerge, the k-th of which is associated with a descriptor that is “half-time”. The “half-time” information is relevant both for the sequence k but also for the entire video file. The primary indexing of the video file can therefore be enriched with the half-time information and the date of this half-time in the file.

In a second iteration of the sequencing, if for example it is known that three goals are to be sought and that these four goals are identified before the first half, the information for which is contained in the primary index, it will be possible to associate the second half sequences that could come close to a goal to offensive actions with no goal achieved. The secondary index is then enriched with this information. And so on.

In summary, wildcard information can populate the primary index from the secondary index, and character information initially identified as generic and becoming particularly relevant to a particular sequence can populate the secondary index from the primary index.

The invention therefore makes it possible to go down, thanks to this indexing process, to a much finer grain size in a search for content in digital video files than what is permitted by the indexing processes currently implemented for this type of files, as well as a two-level sequence search possibility according to the two nested dimensions created by the two indexes.

It is understood that after at least one passage in the modules 3 of multimodal analysis and 5 of sequencing followed by a stage of enrichment of the descriptors via the module 4 dispatcher, an automated indexing of the sequences identified in the digital video file - l "Secondary" indexing - can be obtained in the absence of any prior knowledge of the content of this digital video file, and even if the audio and textual contents do not make it possible to obtain relevant descriptors at the origin.

It is well understood that this secondary indexing is dynamic, that is to say that it can be enriched and refined: as the analyzes of videos of the same domain are carried out, the corpus of relevant descriptors associated to this domain on the basis of which the multimodal analysis module 3 can analyze a digital video file increases. As a result, the first analyzed digital video file can be re-analyzed after analyzing N other digital video files to refine its sequencing.

It is also understood that the secondary indexing can be carried out according to various points of view according to the video search requests carried out by the user on the video library already analyzed. In other words, an initial point of view chosen for secondary indexing is not absolutely limiting and can always be modified on the basis of a particular search. For example, a digital video file could have been created manually by aggregating two video files to give a digital video file containing a football sequence containing, among other things, a spectacular football goal followed by a rugby sequence containing, among other things, a spectacular rugby. Analyzing this digital video file in sports mode would yield two sequences, one sequence (a) for football and one sequence (b) for rugby, but there is no reason why the sequencing should be suitable for football rather than in rugby or vice versa.

If during a search via the search module 6 described below, on the basis of keywords associated with football, the sequence (a) is presented among the search results among other videos, the dispatcher can relaunch an analysis of the video (a) on descriptors adapted to football, to obtain a sequencing and an indexing more adapted to this particular sport. But he can repeat the same process at another time in the context of rugby.

It is therefore a dynamic indexing, not requiring a priori knowledge of the content of the digital video file and being refined and enriched as the device is used.

Once the criterion for stopping the iterations has been fulfilled for at least one digital video file, the search module 6 contains a “client”, which allows a user to access the various sequences of the video files analyzed by formulating a search query .

The research module 6 therefore constitutes the so-called “front-end” level of the device, that is to say through which the end user interacts with the device, while modules 1 to 5 constitute the so-called “back-end” level. -end”, i.e. not visible to the end user of the device.

The research module 6 can communicate with a video editor module 7, comprising an interface for creating, editing and viewing video extracts corresponding to virtual sequences.

The search module 6 allows the user at least to formulate a search query and to visualize the result.

When the server of the documentary database receives the request thus formulated in the client, a search, by keywords in particular, is carried out on the sequences of video files thanks to the association {primary index, secondary index} based on a inheritance link and thanks to the sets of descriptors that have been associated with each sequence of each digital video file during secondary indexing.

The query is not an a priori query based on a relational database language, although this possibility could be envisaged. This is a query of the type used by search engines, i.e. the query can combine a full-text, faceted search based on the descriptors present in the primary and secondary and numerical (for example, sorting can be done on chronological type criteria).

The search query can be formulated by a user in a user interface or else by a conversational robot (“chatbot” in English).

The search result is then displayed in the graphical interface of the search and editing module 6 and it does not appear in the form of a list of video files but of a list of sequences of video files, classified in order of relevance.

The principle is therefore that implemented for website search engines, which allow direct access to the pages that make up the websites, or for the constitution of playlists from a set of audio files in which tracks or chapters are predefined. However, if this principle is natural for these two types of media, highly structured and designed to be indexed, it is not used for any type of digital video file in general, for which the choice has historically been made to index them in their globality due to the complexity of their sequencing.

The device makes it possible in summary to constitute a search engine for digital video file sequences, the sequencing of video files on which the search is carried out being dynamic, that is to say to be created or modified or adapted at the end of formulating a new search query.

Thus, if we take the example of football matches again, if the user wishes to obtain a video made up of all the goals scored by the number 11 of the team which won Ligue 1 in a given year in France, this is possible thanks to the process described here, from the only data of the complete video files of the league 1 matches in France and without any manual intervention of sequence selection in each of the video files.

In the field of online courses, it is also possible to compose a video made up of sequences of videos from different video files, each dealing with the subject of limited developments, but by selecting only the portions of video files that deal with development. of Taylor-Lagrange. This represents a considerable time saving, since it is no longer necessary to view all the relevant video files when only portions (sequences) of these video files are really suitable for the search query formulated.

The search result may include several sequences from several different video files and/or several sequences from the same digital video file.

It should also be noted that at least in the first case, the notion of temporal coherence between the sequences of video files resulting from the search is absent, which goes far beyond the capabilities of current video search engines. The chaptering is then a chaptering straddling several digital video files.

The temporal consistency of the original sequences may not be respected, even in the case where the sequences forming the list returned in response to the search query come from the same original digital video file, since this is the relevance of the sequences with respect to the search criterion which fixes their order of appearance in this list. The relevance of the sequences in relation to the search criterion is for example evaluated according to logical and mathematical criteria, which make it possible to assign a score to each sequence according to a query. The sequences are then presented in descending order of score. Prior filtering steps (language, geographical origin, dates, etc.) may be provided.

In a particular embodiment, during indexing, a higher weight is assigned to the secondary descriptors than to the primary descriptors so that the search result is based more on the content of the sequence than on the content of the video file digital as a whole.

Thanks to the indexing architecture (primary and secondary), a user can therefore perform several tasks dynamically from full-text search functionalities, semantic concepts, themes or multi-criteria filters/facets.

The research module 6 can comprise a user interface, such as a computer, a tablet, a smartphone for example.

The video editor module 7 can include a user interface, such as a computer, a tablet, a smartphone for example.

The user interface can be common to modules 6 and 7.

The user can in particular, via one or other of these interfaces: from each virtual sequence, extract the virtual sequence from the digital video file to produce a video extract that he can view , such as streaming, or saving as a new digital video file. In the case where a video extract is displayed, it can optionally simultaneously display the endogenous and/or, where appropriate exogenous, secondary and/or primary descriptors associated with the extracted sequence.

- Make a summary from a video file (either by "natural language Processing" for online courses, or by image recognition to summarize sports sequences);

Build playlists by associating similar sequences and/or sequences responding to a query, these sequences potentially coming from different original video files and organized in the playlist according to a criterion other than a temporal criterion; - Create a virtual edit by associating similar sequences and/or response sequences to a request, these sequences potentially being from different original video files and organized in the playlist according to a criterion other than a temporal criterion;

- Navigate within the playlist or the new video thus edited, since the latter are automatically chaptered thanks to the secondary indexing system. In particular, it is possible to trigger the playback of a chapter of your choice or even to interrupt and resume the dynamic scrolling of the video extracts from an appropriate graphical interface.

- Synchronize the video extracts with a "second screen" of the dashboard type presenting enriched information from metrics or statistics, resulting from a calculation of indicators extracted from the video extracts. Data analysis can then possibly be coupled with video analysis. The dashboard can also present other information, such as definitions or "find out more" from the encyclopedic web, geographical maps, graphs...

The user interface can comprise a graphical interface 55 comprising a zone 52 dedicated to formulating the search query and displaying its results, a zone for viewing video extracts (screen 1, reference 53), a second zone display (or even screen 2, reference 54), synchronized with screen 1 and a virtual remote control zone 51.

When a playlist is obtained, in a particular embodiment, each end of sequence marker of each virtual sequence associated with an extract from the playlist is: main plurimodal or

- secondary multimodal and resulting from three modalities.

This arrangement makes it possible to increase the semantic consistency of the playlist as a whole and its consistency with respect to the search criterion formulated.

Navigation can, thanks to the primary and secondary indexing system, be extended outside the selected playlist: it is for example possible, from a given sequence of the playlist, to extend the playback of the digital video file from which from the sequence beyond this sequence by moving the start and/or end of sequence markers.

Visual effects, such as, in a non-exhaustive way, slow motions, enlargements, repetitions, can be applied to the playlist, either during viewing, an addition of text, a freeze frame, etc., or for the editing a new digital video file. Sound effects, such as, but not limited to, modifying a background sound, adding a commentary or another sound, can be applied to the playlist, either during viewing or for editing. a new digital video file. Building a playlist or editing a new video can be fully automated from the formulation of the search query. However, as the system behaves like a virtual playhead which moves dynamically from sequence to sequence, at any time, if the graphic interface of module 6 gives it the possibility, the user can act on the playlist or the new video.

In one embodiment, the graphical interface of the video editor module 7 thus offers navigation options in the form of an improved video player allows access to the summary when the search result is an entire video or an interactive zapping within the selected and aggregated sequences. One embodiment of such a graphical interface 55, for editing or viewing a playlist, can be viewed in FIG. 5a. Selectable descriptors are positioned to the left of playlist viewing screen 1, the playlist can be displayed above screen 1, the descriptors related to the user's search are displayed above the playlist . Virtual remote control 51 is located below the playlist. A second screen linked to the video extract corresponding to the virtual sequence being viewed is located to the right of the playlist and allows you to display graphics or other useful information linked to the playlist.

Fig. 5b shows another embodiment of the graphical interface of the device 8 in which selectable descriptors are positioned to the left of the screen for viewing the playlist, the playlist is viewed in screen 1 (reference 53), the descriptors related to the user's search are located above the playlist and the virtual remote 51 is located below the playlist

Fig. 6 represents the actions performed when using each button of the virtual remote control on an example of a playlist created from three digital video files, the playlist being composed by way of example of three different extracts.

The virtual remote control comprises for example at least 5 virtual buttons.

The al button allows viewing of the video extract corresponding to the current sequence and stopping viewing.

When button a2 is pressed, the playback of the video extract corresponding to the sequence being viewed will be extended in the original digital video file beyond the duration provided for this sequence, a second press of button a2 while viewing has not yet exceeded the time limit provided for the sequence cancels the first press of button a2, a second press of button a2 when viewing the digital video file outside the time limit provided, stops viewing of the original digital video file and resumes the playlist at the next sequence.

Button a3 allows you to return to the start of the sequence preceding the sequence currently being viewed. The a4 button allows you to return to the start (at the timecode of the start marker) of the sequence currently being viewed.

Button a5 stops viewing the current sequence and starts playing the next sequence.

Other virtual buttons are likely to be added:

- A button (“-N s”), which allows you to go back N seconds in the digital video file of the current sequence, allowing you to review a sequence or to see N seconds before the start marker of the current virtual sequence;

- A virtual button ("+ Ns"), this button allows you to advance N seconds ahead of the digital video file of the current sequence allowing you to skip a sequence or see 10 seconds after the end marker of the virtual sequence In progress.

Virtual buttons allow you to interact with the sequence start and end markers in the background.

The virtual remote control therefore allows flexible navigation within the automatic playlist of video extracts from digital files, the user being able to view the selected extracts at will in the order of the playlist or in an order that suits him better or even extending the viewing an extract before or after the cut markers, without the files associated with each extract being created and having to be opened and/or closed to switch from one extract to another. The comfort and browsing potential are therefore considerably improved compared to what is possible with a “static” playlist within the meaning of the prior art.

Figs. 7a and Figs. 7b represent two examples of graphical interface 55.

FIG. 7a represents a graphic interface of the computerized method, comprising a first screen 53 for viewing the playlist, a second screen 54 for a graphic linked to the sequence being viewed and a virtual remote control 51 located below the two screens to navigate in the playlist (in which the video extracts are arranged one after the other), as well as a button used to put the playlist in full screen.

FIG. 7b represents a graphic interface 56 of the computerized method, comprising a first screen 53 for viewing the playlist, a second screen 54 for putting messages in connection with the video or for communicating with other users, a virtual remote control 51 located below the two screens to navigate in the playlist and a button used to put the playlist in full screen.

When a search result includes only virtual sequences identified in one and the same digital video file, the playlist made up of extracts based on this search result can be exhaustive. It may also contain only extracts considered essential with respect to search criteria specified by the user.

In particular, a score can be defined to classify the virtual sequences of digital video files into two categories: "essential" and "ornamental" according to the number of descriptors found.

When a search result includes virtual sequences from different digital video files, the playlist made up of extracts based on this search result may contain only the extracts associated with virtual sequences identified as essential with respect to criteria user-specified searches.

The concept of summary can be defined in relation to a particular domain. In the case of sport, and in particular football, the summary can be built from keywords provided by the user or defined beforehand, for example {goal, yellow card, red card, change of player, mid- time}, the relevant sequences being presented in the temporal order of the initial digital video file from which they originate.

It is possible to navigate in this playlist or the new video by selecting or deselecting certain scenes, with a view to real-time video editing, for example through a graphical interface comprising a menu bar and control buttons that can be activated by a mouse click, such as "start", "fast forward", "stop", "select chapter", ...

The search is possible in "full text" mode (or even "full text") and in "faceted" search mode, with optional semi-automatic completion. Faceted answers help refine search criteria and are combined with full-text words.

For example, for the example of the field of football, it is possible to create a playlist including goals from corners of all the league 1 teams in France over a year at home in the last 1/4 hour of a match in a duration of around 10 times less than that required on a professional platform (Instat/Dartfish/Sportscode coupled with Opta/Bombstats data providers), and the playlist is only made up of relevant match sequences and not all of them matches.

Thanks to the inheritance indexing system, the video files (in the previous example, the matches) from which the sequences originate are known. It is therefore possible to provide an option to view all or part of the original video files of the sequences if necessary.

The interfacing between module 6 "front-end" and the "back-end" level composed of modules 1 to 5 can be done whatever the support of module 6 (computer, tablet, smartphone, etc.) possibly without use a proprietary application. That is in particular achievable with technologies accessible in Open Source, such as the React JavaScript library.

Optionally, the device can be integrated into a social network, and offer two user profiles: the creators of video files by editing using the video editor module 7 and the viewers (“followers”) who follow these creators.

The browsing history on a playlist of excerpts from digital video files obtained according to the invention can be recorded. It can then be shared in a social network or used to semi-automatically edit a new digital video file.

Fig. 8 represents a graphic interface of the device 8 comprising a screen for the representation of a mental map ("mindmap" in English) of a directory of sequences or automatic lists or extracts or playlist recorded by the user, a part of the backups being public and the other part private, below this screen several tabs are selectable: Mindmap, Chatbot, Search by facet, Social network and video editor.

Fig. 9 represents a graphic interface 56 of the device 8, comprising a screen for the representation of the interactive Chatbot making it possible to carry out a search for playlists or sequences through a discussion by keyword, below this screen several tabs are selectable: Mindmap, Chatbot, Facet Search, Social Network and Video Editor.

Fig. 10 represents a graphic interface of the device 8, comprising a screen for the representation of the search by facet, grouping descriptors under other more general descriptors, making it possible to search by tree structure, below this screen several tabs are selectable: Mindmap, Chatbot, Facet Search, Social Network and Video Editor.

Fig. 11 represents a graphic interface of the device 8, comprising a screen for the social network integrated into the invention, the users share the playlists found or created, below this screen several tabs are selectable: Mindmap, Chatbot, Search by facet, Network social and video editor.

Fig. 12 represents a graphic interface of the computerized device 8, comprising a screen for editing video, the user can modify the order of the extracts and integrate the extracts he wishes into a playlist, below this screen several tabs are selectable: Mindmap, Chatbot, Facet Search, Social Network and Video Editor.

LIST OF REFERENCE SIGNS

1: acquisition module

2: splitter module

3: multimodal analysis module 3 a: analyzer according to the image modality 3b: analyzer according to the audio modality 3 c: analyzer according to the text modality 3d: analyzer according to the action modality 4: enrichment module

4a: database free to access and use 4b: web services 4c: other type of database 5: sequencer module 6: client

7: video editor module

8: computerized device for sequencing digital video files

21: main multimodal candidate sequence marker

22: tri-modality secondary multimodal candidate sequence marker 23: bi-modality secondary multimodal candidate sequence marker

51: virtual remote control

52: area dedicated to formulating the search query and displaying its results

53: viewing area for video extracts (screen 1)

54: display area synchronized with screen 1 55: graphical user interface

Claims

1. Computerized process of audiovisual de-linearization allowing sequencing of one or more digital video files and indexing of the sequences resulting from the sequencing, by virtually cutting by time stamping the digital video file(s) into virtual sequences, each virtual sequence being defined by two sequence time stamps and associated descriptors, the method comprising the following steps: a. receipt of one or more digital video files for analysis; b. indexing of each of said digital video files in a primary index by means of endogenous, so-called primary, associated descriptors making it possible to identify each digital video file; vs. automatically extracting audio, image, and text data streams from each of said digital video files; d. by means of a plurality of computerized devices implementing an automatic learning algorithm chosen and/or trained for a typology of digital video files defined beforehand, automatic analysis, file by file, of each of said one or more digital video files, according to the four modalities: image modality, audio modality, text modality, action modality making it possible to identify the groups of successive images forming a given action, the analysis automatically producing one or more unimodal cut-out temporal markers for each of the modalities , one or more descriptors being associated with each of the unimodal cutting time markers, e. automatic production, following the analysis of each of said one or more digital video files, of candidate sequence time markers, with the aim of delimiting virtual sequences, and descriptors associated with these candidate sequence time markers, which are :

- either unimodal cutting time markers of said digital video files, and which are called at the end of this step unimodal candidate sequence time markers;

- either, for each of said digital video files taken in isolation, the time codes corresponding to said unimodal cutting time markers are compared and, each time that at least two unimodal cutting time markers resulting from different analysis modalities are separated by a time interval less than a duration predetermined principal (T2), a multimodal candidate sequence temporal marker, in mathematical connection with the at least two unimodal cut markers, is created; f. for each of said digital video files analyzed, according to a lower limit and an upper limit defined to determine the minimum duration and the maximum duration of each sequence, with respect to the typology of the digital video file(s),

- each pair of sequence markers having a start of sequence marker and an end of sequence marker, such that the duration of each sequence retained is between said lower and upper limits,

these pairs of sequence markers being associated with the descriptors associated with the said selected candidate sequence temporal markers, these descriptors therefore being referred to as “secondary endogenous descriptors”; g. indexing, in a secondary index which is in a relationship of inheritance with respect to said primary index, of all the pairs of sequence markers by means of the associated descriptors allowing the identification of each sequence, the virtual sequences being identifiable and able to be searched at least by the secondary endogenous descriptors and the primary endogenous descriptors.

2. Computerized method of audiovisual de-linearization according to claim 1 characterized in that a video extract associated with a virtual sequence, obtained by viewing the file fragment delimited by the two sequence markers of the virtual sequence has a unit of meaning which results from the automatic analysis of each digital video file according to the four modalities and from the virtual cutting in relation to this analysis.

3. A computerized audio-visual de-linearization method according to claim 1 or claim 2, wherein at least one of the two sequence markers of each pair of sequence markers selected in step f, is a multimodal candidate sequence time marker and is then said to be a plurimodal sequence marker, and advantageously each sequence marker of each pair of sequence markers selected is a plurimodal sequence marker.

4. Computerized method of audiovisual de-linearization according to one of claims 1 to 3, in which in step f, two types of plurimodal sequence markers are distinguished:

- a multimodal sequence marker created from four unimodal cut-out temporal markers from the four different modalities separated two-by-two by one time interval less than said main predetermined duration (T2) is said to be the main plurimodal sequence marker and

- a multimodal sequence marker created from two or three unimodal cutting temporal markers resulting from as many modalities among the four modalities, separated two-by-two by a time interval less than said main predetermined duration (T2) is said secondary multimodal sequence marker.

5. Computerized method of audiovisual de-linearization according to one of claims 1 to 4, in which at least one of the markers of each pair of sequence markers is a main multimodal sequence marker.

6. Computerized method of audiovisual de-linearization according to one of claims 1 to 5, in which the action modality is a modality of at least one of the two sequence markers of the pair of sequence markers selected.

7. Computerized process for audiovisual de-linearization according to one of claims 1 to 6, in which weights are assigned to each of the modalities for the production of the candidate sequence markers in step e and/or the selection of the markers of sequence in step f.

8. Computerized method of audiovisual de-linearization according to claim 7, in which:

- for video files with high informational content via speech, the weight of the text modality is greater than that of the other three modalities.

9. Computerized method of audiovisual de-linearization according to any one of claims 1 to 8, in which a weight is assigned to the secondary endogenous descriptors as well as to the primary endogenous descriptors to characterize their importance in the sequences, and this weight is more larger for secondary endogenous descriptors than for primary endogenous descriptors.

10. Computerized method of audiovisual de-linearization according to any one of claims 1 to 9, in which the secondary endogenous descriptors are said "unimodal" when they correspond to a single modality and are said to be "multimodal" when they are detected for several modalities.

11. Computerized method of audiovisual de-linearization according to any one of claims 1 to 10, in which step f presents these sub-steps, for each digital video file, to produce the virtual sequences: i) - selection of a last end-of-sequence marker, in particular multimodal, from the end of the digital video file,

- and determination of the presence of a multimodal sequence marker at a time code between two extreme time codes, calculated by subtracting the lower limit from the time code of the selected end marker and by subtracting the upper limit from the time code of the selected end marker,

- otherwise, determination of the presence of a unimodal marker whose modality depends on the typology of the digital video file between the two extreme time codes

- selection of the unimodal marker as the last sequence start marker if the presence is confirmed,

otherwise, the last sequence start marker is designated by subtracting the last selected end marker from the upper limit from the time code; ii) step i) is repeated to select a penultimate sequence start marker, the sequence start marker selected at the end of the previous step i playing the role of last sequence end marker selected at beginning of the previous step i; iii) sub-step ii) is repeated in this way until the start of the digital video file.

12. Computerized method of audiovisual de-linearization according to any one of claims 1 to 11, in which said main predetermined duration (T2) is less than 5 seconds, and optionally said maximum duration of each selected sequence is equal to two minutes.

13. Computerized method of audiovisual de-linearization according to any one of claims 1 to 12, in which at least one additional step of enriching the indexing of the virtual sequences by exogenous secondary descriptors is carried out in step g.

14. Computerized method of audiovisual de-linearization according to one of the preceding claims, in which the secondary descriptors by means of which the identified sequences are indexed are enriched with a numerical or lettered indicator, such as an overall score of a card of digital collection, calculated for each virtual sequence from the secondary descriptors of the sequence and/or the primary descriptors of the digital video file in which the sequence was identified.

15. Computerized method of audiovisual de-linearization according to any one of claims 1 to 14, in which the action modality comprises the sub-modalities {detection of change of shots, detection of action according to a typology of digital video files}, and in that each of the sub-modalities of the action modality makes it possible to generate a particular set of unimodal cutting time markers.

16. A computerized sequencing and indexing method according to any preceding claim, wherein the analysis according to the audio modality comprises noise detection, music detection and/or transcription of speech into a text stream .

17. Computerized sequencing and indexing method according to any one of the preceding claims, in which the analysis according to the image modality comprises the sub-modalities { shape or object recognition; plan aggregation; optical character recognition}, and in that each of the sub-modalities of the image modality makes it possible to generate a particular set of unimodal descriptors.

18. Computerized method for automatically producing an ordered playlist of video extracts from digital video files, with a data transmission stream, the digital video files being indexed in a primary index stored in a documentary database containing the video files digital, with primary descriptors, the digital video files having, beforehand and by means of the computerized process of audiovisual delinearization according to any one of Claims 1 to 17, been cut virtually by time stamping into virtual sequences which are defined by two sequence time markers forming a pair of sequence markers and by associated secondary descriptors, the pairs of virtual sequence markers and the associated secondary descriptors being stored in a secondary index stored in a documentary database, the secondary index being in an inheritance relationship with the primary index, these indexes being accessible via a graphical interface, the method comprising:

1. the formulation of at least one search query;

2. transmitting said search query to a search server associated with said database;

3. determining and receiving from the documentary database of said server, in response to said transmitted search request, the search result which is an automatic list of pairs of time stamps of sequences and associated descriptors, according to a order which is a function of the descriptors associated with each virtual sequence and of the formulation of the search query, the virtual sequences being identifiable and able to be searched by the secondary descriptors and the primary descriptors;

4. the display and viewing from a virtual remote control of the playlist which presents all the video extracts associated with the ordered automatic list of pairs of time markers and received during step 3, without creating a new video file digital, the virtual remote control allowing navigation on the playlist, each video extract from the playlist:

- being associated with a virtual sequence, and

- being called when viewing the playlist, via the data transmission stream from the digital video file indexed in the primary index, in which said virtual sequence indexed in the secondary index has been identified.

19. A computerized method for automatically producing an ordered playlist of video extracts from digital video files according to claim 18, in which:

20. A computerized method for automatically producing an ordered playlist of video extracts from digital video files according to claim 18 or claim 19, in which the method allows the following navigation operations from the virtual remote control and the data transmission stream: a. playing, stopping and resuming the current extract of the playlist which comprises all the video extracts associated with the automatic list obtained in step 3; b. pointing to an extract in the playlist that includes all the video extracts, by fast forward or fast reverse; vs. temporary output of the extract from the playlist which includes all the video extracts, to view the original digital video file of the extract, without temporal constraints linked to the start and end time markers of the virtual sequence associated with the video extract.

21. A computerized method for automatically generating an ordered playlist of video clips from digital video files according to claim 20, wherein the method allows the following additional operation: d. new temporary output from the viewing of the original digital video file of the extract being played from operation c), to view during step d) a summary created automatically and prior to this viewing from this single original digital file.

22. Computerized process for the automatic production of an ordered playlist of video extracts from digital video files according to one of claims 20 and 21, in which the process allows the following additional operation: e. recording of browsing history on the playlist of video sequences and creation of a new digital file which is this browsing history.

23. Computerized method for the automatic production of an ordered playlist of video extracts from digital video files according to one of claims 18 to 22, in which said search query formulated in step 1 is multi-criteria, and combines a search on the full text, a faceted search and in that the criteria for achieving the order for said automatic playlist include chronological and/or semantic and/or relevance criteria.

24. Computerized method for the automatic production of an ordered playlist of video extracts from digital video files according to one of claims 18 to 23, in which said search query formulated in step 1 is carried out automatically from one or more user-specified criteria chosen from a list comprising: the desired duration of an automatic playlist as well as semantic criteria.

25. Computerized method for the automatic production of an ordered playlist of video extracts from digital video files according to one of claims 18 to 24, in which the said search query formulated in step 1 is carried out by a conversational robot.

26. Computerized method for the automatic production of an ordered playlist of video extracts from digital video files according to one of claims 18 to 25, comprising a viewing step in which the user displays on a first screen a video extract from the playlist, and descriptors of the virtual sequence associated with the video extract on a second screen synchronized with the video extract.

27. Computerized method for the automatic production of an ordered playlist of video extracts from digital video files according to one of claims 18 to 26, comprising a viewing step in which the descriptors associated with the virtual sequences are displayed on the extracts.

28. Computerized process for the automatic production of an ordered playlist of video extracts from digital video files according to one of claims 18 to 27, in which the technology used is ElasticSearch.

29. Computerized process for the automatic production of an ordered playlist of video extracts from digital video files according to one of claims 18 to 28, in which the process accesses the video files in “streaming” mode.

30. Automatic list of pairs of sequence markers and associated descriptors resulting from the method of computerized process for the automatic production of an ordered playlist of video extracts from digital video files according to one of claims 18 to 29, presenting endogenous descriptors and exogenous consistent with the search query.

31. An automatic list of sequence marker pairs and associated descriptors according to claim 30, in which the sequence time markers are determined by a multimodal approach by automatic analysis, file by file of each of said one or more digital video files, according to at least two of the four modalities: image modality, audio modality, text modality, action modality.

32. Automatic list of pairs of sequence markers and associated descriptors according to one of claims 30 and 31, in which at least two sequence time markers are determined randomly or unimodally.

33. Computerized method of editing with virtual cutting without creating a digital video file, from the computerized method for the automatic production of an ordered playlist of video extracts from digital video files according to one of claims 18 to 29, comprising the steps following:

II. navigation on the at least one automatic playlist of video extracts from digital video files per data transmission stream;

III. selection by the user of one or more virtual sequences associated with at least one automatic playlist of video extracts from digital video files, to create a new playlist of video extracts whose order can be modified by the user.

34. Computerized method of assembly with virtual cutting according to the preceding claim 33, comprising one of the following steps:

- modification of said new automatic playlist of video extracts by adding and/or removing video extracts to said new playlist;

- modification of one or more video extracts by extending or reducing the duration of the virtual sequences associated with the video extracts of the said new playlist, by moving the start and end markers of each virtual sequence;

- modification of the video extracts by a visual effect or a sound effect.

35. Use of the video extracts or of a playlist of video extracts obtained by the method for automatically producing an ordered playlist of video extracts from digital video files according to one of claims 18 to 29, or by the method of assembly with virtual cutout according to any one of Claims 33 to 34, in a social network or in a search engine.

36. Computerized system including: i. At least one module (1) for acquiring one or more digital video files; ii. At least one dispatcher module (2); iii. At least one multimodal analysis module (3); iv. At least one sequencing module (5) generating sequences of indexed digital video files; v. At least one search module (6) comprising a client making it possible to formulate a search query, for the implementation of the steps:

1. One or more digital video files to be analyzed are received via the acquisition module (1);

3. The audio, image and text data streams are extracted from each of said one or more digital video files;

4. By means of a plurality of neural networks chosen and/or trained for a typology of video files defined beforehand and contained in the multimodal analysis module (3), a file-by-file analysis is carried out of each of said one or several digital video files according to the four modalities: image modality, audio modality, text modality, action modality, the analysis automatically producing one or more unimodal cutting time markers for each of the modalities, one or more descriptors being associated with each unimodal cut-out time markers;

5. At the end of the analysis of each of said one or more digital video files, candidate sequence time markers are provided, with the aim of determining virtual sequences, and the descriptors associated with these candidate sequence time markers , which are :

- either unimodal cutting time markers of said digital video files, which are called at the end of this step unimodal candidate sequence time markers;

- either for each of said digital video files taken in isolation, the time codes corresponding to said unimodal cut-out time markers are compared and, each time that at least two unimodal cut-out markers from different analysis modalities are separated by an interval of time less than a main predetermined duration (T2), a multimodal candidate sequence marker is created in mathematical connection with the at least two unimodal cut-out temporal markers;

6. For each of said analyzed digital video files, a lower limit and an upper limit are defined according to the type of said digital video file for the duration of a sequence and an automatic selection is made from among the markers of candidate sequence of the pairs of sequence markers, called start and end of sequence markers, each pair of sequence markers presenting a start of sequence marker and an end of sequence marker, so that the duration of each sequence retained is between said lower and upper limits, these pairs of sequence markers being associated with the descriptors associated with the said selected candidate sequence temporal markers, these descriptors therefore being called “secondary endogenous descriptors”;

7. By means of the sequencing module (5), in a secondary index which is in a relationship of inheritance with respect to said primary index, all the pairs of sequence markers are indexed by means of the associated descriptors allowing G identification of each sequence, the sequences being identifiable and able to be searched at least by the secondary endogenous descriptors and the primary endogenous descriptors; 8. A search query for sequences of digital video files is formulated by means of the search module (6); each of said modules (1) to (6) comprising the necessary calculation means, each of said modules (1), (3), (5) and (6) communicating with said dispatcher module (2) and said dispatcher module (2) managing the distribution of calculations between said modules (1), (3), (5) and (6).

37. System according to claim 36 further comprising at least one module (4) for enriching the primary descriptors of the video files and/or secondary descriptors of the virtual sequences of digital video files by exogenous complementary descriptors.

38. A system according to claim 36 or claim 37 further comprising a video editor module (7) communicating with the search module (6).