EP1845521A1 - System for movie dubbing - Google Patents

Abstract

The system has a displaying unit e.g. television, for displaying a sequence of images or video sequence, and lip-sync band containing a text of cues and marks, on a screen simultaneously, where the marks permits the synchronization of the cues with respect to the images. A voice transformation and synthesizing engine restores an intelligible sound signal from the lip-sync band for forming dubbed cue. A adjusting unit re-edits a word of each cue according to the band.

Description

The present invention relates to a film doubling system.

The dubbing of a film is to adapt the dialogue of a film to a language other than the original one. To do this, the original soundtrack must be replaced by a new soundtrack. The ultimate goal is then that the movement of the lips of the characters of the film coincides at best with the words pronounced in the new language. Generally, the manufacture of a dubbing consists in using a screen on which are projected on the one hand the images of the film and on the other hand a rhythm band, still called rythmo band or band of dialogue, containing the text of the replicas of the characters in the target language. This rhythm band, generally arranged below the images, scrolls continuously and synchronously with the images. Actors can then read the text that scrolls out loud as it appears, and their interventions are simultaneously recorded to ensure synchronism with the lip movements of characters appearing in the image. The document FR 2,765,354 describes such a dubbing system.

A disadvantage associated with this type of method is that it is very difficult for the person in charge to adapt the dialogues to the images to get a concrete idea of the quality of his work compared to the images. Thus, to verify that the text of the rythmo band that he proposes is perfectly adapted to the expressions and movements of lips of the characters of the film and to be sure that the comedian will be able to interpret his text properly, he will generally have to wait for the performance of actor (s) in charge of dubbing. This often results in a number of errors that will need to be taken again after registration. These errors then require a new reflection of the person in charge of the adaptation compared to the texts that it must propose followed by a new performance of the actors. In the business these corrections are commonly called "retakes". Such a method of work can therefore take a lot of time to achieve a final quality soundtrack and is thus very expensive.

An object of the present invention is to overcome the disadvantages of the aforementioned prior art.

In particular, an object of the present invention is to facilitate the work of the person in charge of adapting the lyrics of the film. Another object of the present invention is to reduce the costs associated with establishing a dubbing.

An object of the present invention is further to ensure a maximum quality of the soundtrack doubled with respect to the images of the film.

To achieve these aims, the present invention provides a computer assisted film dubbing system comprising display means for displaying in parallel on at least one screen images and at least one rhythm band containing the text of the replicas. and markers adapted to allow synchronization of the replicas with respect to the images, said system further comprising a speech synthesis and transformation engine capable of reproducing an intelligible sound signal from said rhythm band to form doubled replicas.

Advantageously, said speech synthesis engine generates doubled replicas in accordance with the markers for synchronizing the replicas generated by said speech synthesis engine with the markers of the rhythm band.

Advantageously, said markers of said rhythm band comprise the beginning and the end of a replica, the beginning and the end of sentences, the breaths, the breaks, the labials, the half-lips, etc.

Advantageously, said system may comprise adjustment means able to adapt the speech rate of each doubled replica in agreement with the rhythm band.

Advantageously, said adjustment means make it possible to slow down and / or speed up the throughput of each replica or duplicate replica part without modifying the vocal tone.

Advantageously, said system comprises means for modifying said rhythm band for adding, shifting and / or deleting marks and / or modifying the text of the replicas in order to modify the replicas doubled generated by said speech synthesis engine correspondingly.

Advantageously, the rhythm band comprises at least one graphic illustration strip able to schematically represent the markers of each replica.

The invention will now be further described with reference to the accompanying drawings giving by way of non-limiting example an advantageous embodiment of the invention.

• la figure 1 est un schéma du principe de fonctionnement général d'un système de doublage de film selon la présente invention,
• la figure 2 est une représentation schématique d'un signal audio,
• la figure 3 est une vue schématique du système associant les mouvements de la bouche sur les images, les textes des répliques comme ils apparaîtraient sur la bande rythmo, les repères et la représentation graphique du signal audio généré à partir de la bande rythmo,
• la figure 4 est une vue similaire à la figure 3, montrant une bande rythmo remaniée, ainsi que les modifications sur le signal sonore résultant.

In the figures:

• FIG. 1 is a diagram of the general operating principle of a film doubling system according to the present invention,
• FIG. 2 is a schematic representation of an audio signal,
• FIG. 3 is a schematic view of the system associating the movements of the mouth on the images, the texts of the replicas as they would appear on the rhythm band, the marks and the graphic representation of the audio signal generated from the rhythm band,
• FIG. 4 is a view similar to FIG. 3, showing a reshaped rhythm band, as well as the modifications on the resulting sound signal.

Fig. 1 is a flowchart illustrating the general operating principle of a film liner system according to the present invention. Such a dubbing system generally comprises display means and means for locating the text of the replicas.

The display means are intended to display, broadcast or project on at least one screen of a sequence of images or video sequence and at least one visualization strip of the text of the replicas or rhythm band. Such display means comprise for example a projector, a monitor, a television, a VCR-TV set, a computer, etc. These devices can broadcast the video clip and the rhythm band on one or more screens. The particular arrangement of this / these screen (s) must advantageously allow a good visualization of the contents of the rhythm band that scrolls with respect to the images of the video sequence. To do this, the rhythm band advantageously scrolls in parallel with the video sequence and preferably under said video sequence in order to easily compare the content of the rhythm band to the content of the video sequence.

The locating means make it possible for them to indicate on said rhythm band markers (particularly temporal) relating to said replicas. Such markers may for example indicate the beginning and end of a sentence, the beginning or the end of a breath or a pseudo-verbal sequence, the labials and half labials. Other indications are also possible. Usually, this tracking work can be done manually by annotating a movie tape or via the use of computer software. Such tracking usually leads to the division of the script into small scenes, commonly called loops in the business, which will subsequently be doubled, that is to say, be read and interpreted aloud while being recorded. Such markers are intended to perfectly synchronize the text of the replicas with respect to the images. In other words, these markers can synchronize the replicas of the actors to double compared to the lip movements of the characters in the film.

An example of a film doubling system having particularly advantageous display and registration means is given in the document FR-2 765 354 which is cited as a reference for such a system. This system includes a computer connected to a video recorder, speakers, a keyboard, a display screen and possibly a mouse. The computer includes a central processing unit associated with a program memory for driving a video card, a sound card and the screen interface for generating the synchronized scrolling of a rhythm band containing a plurality of superimposed tracks. Furthermore, a set of keys is provided in order to enter the different marks relating to the replicas of said rhythm band. These different marks are materialized by symbols and / or codes. These markers then make it possible, for example, to locate loop changes, plane changes, the beginning and end of sentences called open mouth, the beginning and end of closed mouth phrases, the presence of labials "M", "P", "B", the presence of half-labial 'F', 'V', 'R', the presence of 'W', 'EU', 'Q', the beginning and end of 'Ah', 'Oh' or also the beginning and the end various expressions such as moans, tears or rales.

In accordance with the present invention, the film dubbing system further incorporates a speech synthesis and processing engine. Such a motor is capable of rendering an intelligible sound signal from the text and markers of the rhythm band to form doubled replicas. This principle of moving from a text to a synthetic voice is generally called "TTS" or "Text to speech" or "text to speech". The interest of such a synthesis and voice transformation engine for film dubbing is twofold. First, by coupling a motor to the system, the person in charge of adapting the text to the images can directly listen to the result of his work by generating a synthetic voice interpreting the replica synchronously with the video sequence. This person can therefore assess the quality of his adaptation work more concretely than if he just reads the text aloud and more effectively than if he had to wait for the performance of the actors in charge of dubbing. He can thus make sure directly of the synchronism of his text with the labials of the characters appearing in the image. Secondly, the voice generated by said engine may, in addition to its adaptation aid, directly serve as the final voice for dubbing the video sequence.

The audio stream generated by the speech synthesis and transformation engine can be redirected either to an acoustic speaker as indicated in FIG. 1, or to a storage device capable of storing this stream. The synthesis technique used can be of different types and can be defined in the advanced settings of the tool.

As shown in FIG. 1, both the video sequence, the rhythm band and the audio stream generated by the synthesis and transformation engine voice can be synchronized using a master clock system. Such a system thus makes it possible to perfectly block said video sequence, said rhythm band and said audio stream so as to synchronize their operations in particular.

Referring now to Figure 2, this figure illustrates, in the form of a graphical representation, an audio signal generated from the phrase "you are beautiful". The synthesized voice must be perfectly synchronous with the rhythm band. As a result, said synthesis and voice transformation engine advantageously generates doubled replicas in accordance with the rhythm band to synchronize the replicas generated by said synthesis and voice transformation engine with the markers. In other words, the information of the rythmo band, and in particular the indicated marks, control said motor. The system may further advantageously comprise means capable of adapting the speech rate of each doubled replica in accordance with the beginning and end of sentence markers of the rhythm band. For example, said means advantageously make it possible to slow down and / or speed up the throughput of each doubled replica without modifying the vocal tone. Such deformation by acting on the speech rate generated by said speech synthesis and transformation engine can be done using an algorithm such as PSOLA or Vocoder. It should be noted that most conventional synthesis and speech transformation engines allow speech rate control, especially when they are compatible with certain standards (VOICE XML, SSML, JSML) such as AT & T Natural Voice 1.4. More concretely, generally, to achieve such a deformation, the user requests the locating means of the rhythm band so as to indicate on said band the various references relating to replicas. The rhythm band is thus cut into several parts according to the indicated marks. Each piece or part of a sequence is then synthesized so that the duration of the signal is equal to the duration between two reference marks framing it. A concrete example of such a deformation will now be described with reference to FIGS. 3 and 4.

Referring to Figure 3, the person in charge of the adaptation first positions the markers of the replicas on the rhythm band based on the reading of the movement of the lips of the character of the film. These marks, represented in the figure by vertical lines surmounted by figurines, subdivided the rhythm band. Thus, such a location generally corresponds to a specific lip or a specific reaction of the character of the film to be distinctly heard in the new soundtrack produced by said engine for synthesis and vocal transformation. The speech synthesis and transformation engine will therefore respect the positioning of the various marks indicated on the rythmo band and will consequently produce a corresponding audio stream.

Referring to Figure 4, the person in charge of the adaptation can move the markers at the rhythm band to perfectly fit the image, including expressions and lip movements of the character (s). To do this, the dubbing system advantageously comprises means for modifying said rhythm band for adding, shifting and / or deleting locating means and / or modifying the text of the replicas in order to modify the doubled replicas generated. by the corresponding synthesis and speech transformation engine. In Figure 4, the person in charge of the adaptation moves a mark of the rhythm band so that the audio signal is then deformed accordingly. As can be seen in the schematic representation of the audio signal, we can notice in this example that there is no change at the beginning of the phrase "You are" while the artificial speaker lingers longer on the end of the sentence. corresponding to the pronunciation of the word "beautiful".

Another example of use of this synthesis and voice transformation engine will now be described. In this example, we will assume that the original video footage shows a character exclaiming "This is nice". The person in charge of the adaptation then places on the rhythm band markers in order to signal on the tape the beginning and the end of the reply to be respected in the sentence to be pronounced. This person then translates the sentence "This is nice" by "C'est beau". However, by rewinding the video recording tape and viewing it while listening to the doubled replica generated by the voice synthesis and transformation engine, said person can detect a defect in his adaptation work. In particular, this person may be shocked by the fact that the lips of the film character do not close on the lip "b" of "It's beautiful". This labial is indeed absent from the original replica in English language. The advantage of the proposed film dubbing system is that the person in charge of the adaptation can directly replace the text "C'est beau" by, for example, the expression "C'est classe". When reviewing the replica dubbed, this person finds that the lips of the character of the film coincide perfectly with the new proposed text. If necessary, in the event of a discrepancy between the duplicated replica and the movement of the lips of the character of the film, it is still possible for him to perfect his work of adding markers on the rhythm band, for example at the moment when the character's mouth of the film opens to pronounce the word "nice" and thus make perfectly coincide the word "class" of the doubled replica. The doubled replica generated by the speech synthesis and transformation engine can then be used as a pronunciation aid for the actor (s) in charge of the dubbing or can be used as a final voice intended to appear on the new voice. soundtrack.

Although the present invention has been described with reference to a particular embodiment, it is understood that a person skilled in the art may make various modifications, without departing from the scope of the present invention defined by the appended claims.

Claims (7)

A film dubbing system comprising display means for displaying in parallel on at least one screen images and at least one rhythm band containing the text of the replicas and markers adapted to allow synchronization of the replicas with respect to the images, characterized in that said system further comprises a speech synthesis and transformation engine capable of rendering an intelligible sound signal from said rhythm band to form doubled replicas. The system of claim 1, wherein said speech synthesis engine generates duplicate replicas in accordance with the markers for synchronizing the replicas generated by said speech synthesis engine with the markers of the rhythm band. A system according to claim 1 or 2, wherein said markers of said rhythm band comprise the beginning and the end of a replica, the beginning and end of sentences, the breaths, the breaks, the labials, the half-labials. System according to any one of the preceding claims, further comprising adjustment means adapted to adapt the speech rate of each replica doubled in agreement with the rhythm band. The system of claim 4, wherein said adjusting means is operative to slow down and / or speed up the throughput of each duplicate replica or replica part without modification of the vocal tone. A system according to any one of the preceding claims, further comprising means for modifying said rhythm band for adding, shifting and / or deleting reference marks and / or modify the text of the replicas to modify the duplicate replicas generated by said speech synthesis engine correspondingly. A system as claimed in any one of the preceding claims, wherein the rhythm band comprises at least one graphic illustration strip adapted to schematically represent the markers of each replica.

Images