NZ532174A - Voice over synchronisation - Google Patents

Abstract

A computer apparatus for displaying a synchronised sound script has at least one symbol representative of a sound event. The computer apparatus comprises a means for generating a graphical user interface on a display screen. The graphical user interface has a control element associated with the at least one symbol. The control element allows adjustment of the synchronisation of the at least one symbol with an original recording. The control element is adapted to be activated by a pointing device to generate a control signal to vary a vocalisation time for the at least one symbol. The computer apparatus also comprises a means responsive to the generation of the control signal by activation of the control element to associate a new vocalisation time for the at least one symbol.

Description

532174 Patents Form No. 5 OurRef: WEJ505049 NEW ZEALAND PATENTS ACT 1953 Complete After Provisional No. 532174/535457 Filed: 6 April 2004 COMPLETE SPECIFICATION VOICE OVER SYNCHRONISATION We, VOICEQ INTERNATIONAL LIMITED, a New Zealand company of 24 Fleet Street, Eden Terrace, Auckland 1003, New Zealand hereby declare the invention, for which We pray that a patent may be granted to us and the method by wh Sh IT l^fl^^f^ijf<ymgd^to bo particularly described in and by the following statement: PT054417494 Office of N.Z. 6 - JUL 2005 RECE ED 300432038 1 VOICE OVER SYNCHRONISATION Field of Invention This invention relates to synchronisation of sound with existing sound or video recordings. Background In the film production industry it is usual to overlay voice or other audio on to a pre-existing recording of sound or video. The expressions "film" and "video" are used throughout this document to refer to audio visual information, or means for recording audio visual information. Therefore these expressions include conventional cinematic films in which a changeable visual image is carried on a film as a succession of pictures (1 per frame of the film) and to include magnetic tape (video tape) which carries a succession of frames each of which carries magnetised information representing a picture in analogue or digital form. The expression also includes other appropriate recording media such as read only memory, optical discs with information recorded in various formats such as DVD, or other forms of digital recordings. Sound signals to accompany the pictures and to be synchronised therewith are typically carried on the film, magnetic material or optical material.

With any film or video, it is important that the changing visual image and the varying sounds that accompany the image are strictly synchronised in order to maintain realism for the user. Well known recording formats mean that synchronism is easily achieved if the visual images and their accompanying sounds are recorded on the film at the same time. However, if the sound is to be added at the film after the visual images have been recorded, it can be very difficult to synchronise the newly recorded sound to the film.

This problem is particularly apparent if the sound to be added is human speech to accompany lip movements of a person in the visual images. The problem is exacerbated where the speaker of the visual images speaks in a first language, and the sound that is to be newly recorded is a second language to be recorded in the place of the first language.

In the film industry it is common to acquire a dubbed translation of the speech track of the film. It is also common for dialogue to be replaced after a film has been shot. This may be due to extraneous noise occurring at the time that the film was being shot, or there may have been sound problems while the film was being shot. Alternatively, performance of the actor's voice 300431152_1 may not have been adequate.

In the past dialogue has been replaced in a recording studio in which the new dialogue is recorded by an actor while being shown the visual images. A number of methods have been used. In the most basic process, the actor wears headphones in a series of audio "beeps" are fed to the headphones immediately prior to moment in which the actor is required to deliver his or her line. This process is laborious, prone to errors, and since there is such a focus on achieving synchronisation, the performance is often neglected.

In another known system, commonly referred to as the "Rythmoband" system, visual images are recorded on a first film or magnetic tape and are projected onto a screen for viewing. A ribbon is wound on a machine in which the linear movement of the ribbon is synchronised with the linear movement of the film or tape.

As a word or sentence is seen to be spoken, an operator writes by hand on the ribbon, and the ribbon is then superimposed on the lower part of the picture either physically or using a projection system. The writing is placed on the ribbon such a way that it coincides with a marker bar at the time the words are seen to be spoken in the pictures.

This system of synchronisation is very long, is tiresome for the participants, and is therefore very expensive.

In yet another system, which is described in French patent number 2765354, film and dubbing equipment is described comprising a control unit, a working screen or display, a keyboard, a video recorder and sound recording apparatus. The display unit is divided into three windows. A scene being played from a film is displayed in one of the windows. In another window, there is a space for dialogue to scroll across the screen. In the third window, there is a list of actors who participate in the seen that is being played in the visual display window.

In practice, although the precise timing of the beginning of words can be synchronised with the visual image, it is frequently desirable to change the timings of individual words or sounds to provide a better match between the new words or sounds being recorded and the visual image. A difficulty with all of these known systems is being able to manipulate the timing of individual words within the line spoken by a particular character. Another disadvantage with these known systems is the considerable time involved in manipulating the textual information to provide it in a form which can easily be read by actors. 300431152 _1 Summary of the Invention It is an object of the present invention to provide an improved apparatus or method for synchronising sounds with a recorded sound track or video, or to at least provide public with the useful choice.

Accordingly the invention broadly consists in computer apparatus for displaying a synchronised sound script having at least one symbol representative of a sound event, the apparatus comprising means for generating a graphical user interface on a display screen, the graphical user interface having a control element associated with the at least one symbol, the control element allowing adjustment of the synchronisation of the at least one symbol with an original recording, the control element being adapted to be activated by a pointing device to generate a control signal to vary a vocalisation time for the at least one symbol, means responsive to generation of the control signal by activation of the control element to associate a new vocalisation time for the at least one symbol.

In a preferred embodiment the graphical user interface displays the at least one symbol and the control element comprises the displayed symbol. The interface may also include a display of sound amplitude of sounds associated with the original recording.

The display of sound amplitudes may be displayed over a time interval corresponding to the voice track.

The control element preferably allows the time for initiation of vocalisation and completion of vocalisation for the symbol to be adjusted.

In one alternative the symbol comprises a sentence. In another alternative the symbol comprises a word.

In other embodiments the symbol comprises a syllable or a plurality of sequential characters. The sound script preferably comprises a voice script for dubbing over the original recording. In a second aspect the invention broadly consists in a method for producing a synchronised 300431152_1 sound script comprising providing a sound script comprising a plurality of sound events for at least part of a scene from an existing audio and/or video recording; determining a time period for vocalisation of the script; determining a vocalisation time for each sound event being the time at which the sound event should be vocalised for the sound event to be substantially synchronised with the scene, and; storing the vocalisation time as a proportion of the time period.

Preferably the method includes the step of adjusting the vocalisation time for a sound event by adjusting the proportion of the time period for that sound event.

In a third aspect the invention consists in a method for producing a synchronised sound script comprising the steps of providing a sound script comprising a plurality of sound events for at least part of a scene from an existing audio and/or video recording; determining a real time period provided for vocalisation of the script; estimating a time period required to vocalise each sound event and cumulatively summing the estimated time periods to provide an estimated time relative to a predetermined time for vocalisation of each sound event; using the estimated time periods to determine a total estimated time for vocalisation of the script; scaling the total estimated time to the real time to provide an adjusted estimated time for vocalisation of each sound event relative to the predetermined time, and; using the adjusted estimated times to allow the sound events in the script to be vocalised substantially in synchronism with the scene.

The method may include the step of using the adjusted estimated time to display the sound events such that the relative time of occurrence of sound events can be visualised.

Preferably the method includes the step of scaling the total estimated time to the real time provides an adjusted estimated time for initiation of vocalisation of each sound event relative to the predetermined time and wherein the sound events are scrolled past a reference mark so that the time at which a sound event reaches the reference marker is the time at which initiation of the sound event should substantially commence for synchronism with the scene.

In further aspects the invention consists in computer apparatus or programs for implementing 300431152 1 the inventive aspects set forth in the foregoing statements.

The invention also broadly consists in any novel feature or combination of features disclosed herein.

Further aspects of the invention, which should be considered in all its novel aspects, will become apparent to those skilled in the art upon reading the following description which provides at least one example of a practical application of the invention.

Brief Description of the Drawings One or more embodiments of the invention will be described bloew by way of example with reference to the accompanying drawings in which: Figure 1 is a diagrammatic view of computer apparatus, Figure 2 is a schematic of a hierarchy of conceptual components, Figure 3 is a part of a script, Figures 4 to 7 are screen shots of a user interface, Figure 8 is a screen shot of an output view.

Detailed description Referring to Figure 1, processing apparatus is shown in the form of a personal computer generally referenced 1 having a visual display unit 2, and input apparatus which may optionally comprise a keyboard 3 and a pointing device such as a mouse 4.

The apparatus 1 may form part of a network, or be connectable to a network 5 (such as the Internet for example) to allow communication or exchange of information with further processing apparatus such as the computer apparatus generally referenced 6.

The apparatus 6 may in one example comprise a server and the apparatus 1 may comprise a workstation.

In a preferred embodiment the computing apparatus 1 comprises a personal computer which may have Internet access, and is able to provide data to a projector or alternative display 7 to 300431152,1 6 allow one or more actors to view the information as will be described further below.

The apparatus 1 includes a processor, a memory and an operating system, and is capable of running an application to perform functions as will be described further below.

With reference to Figure 2, a fundamental structure according to one embodiment of the invention is shown for allowing a record film or soundtrack to be processed to allow further sounds to be synchronised with the original recording. The film may be referred to as a Project 10 and is broken down into one or more scenes 12. The individual scenes 12 can in turn be broken down into one or more lines 14 spoken by a variety of characters which appear in each scene. The lines 14 in total effectively provide a sound script which is referred to in this document as a voice script for the original recording.

A script is provided as an electronic file in a suitable format. In Figure 3 an initial portion of a script is shown in a text file format. However, other formats may be used, such as XML for example, or a mixed media format. Information conveyed by the script includes information on the lines spoken by each character. In a preferred embodiment the script also includes information on at least the timing of initiation of the vocalisation of the line. This timing information may also include information on the time for the completion of vocalisation of the line.

As shown in Figure 3, the format is such that the application can recognise the scenes 12, the lines 14 spoken by each character 16 and the start time 18 (relative to a predetermined datum time) for each line. In a preferred embodiment the start time for each object is measured relative to the parent object. Thus the start time for each line in this example is measured from the beginning of the scene, and the start time for each scene will be measured from the beginning of the project. Those skilled in the art to which the invention relates will realise that other time datums, standards or formats may be used. The electronic script file may be read by the application which then extracts the name for each scene, identifies each of the characters and recalls the character name, and stores each line of each scene.

In the example shown in Figure 3, the application stores the start time of each line. The times are preferably displayed as hours: minutes: seconds: frames, where the number of frames per second is derived from a MIDI time source, or defaults to 24. The supporting software preferably includes a MIDI time source that interfaces with the operating system. The application has a concept of current time, and this time is provided by an interface to a MIDI 300431152_1 time source which is also able to provide a concept of "playing" (by data streaming at a constant and real time pace). This concept of time pushes several events, including: 1. the selection of the current scene, provided editing is not happening 2. the placement of the line showing the current time on the timeline view (as will be described further below) 3. updating of the output view (as will be described further below).

The data which is obtained from the electronic script file can be edited by the application. Editing of the data held by the application preferably occurs on the scene by scene basis.

Each project has the relative file name and an internally held identifier which is unique across all the projects for any given user of the application. Each project also has a native language and a collection of secondary languages that the scenes in the project may be translated to. A project also has a collection of characters, where each character has a name and an associated colour (as will be described further below). Each project also preferably holds a low detail version of a sound wave on an associated multimedia file. This will also be described further below.

Each scene has a name and an internally held identifier which is unique across the project. In a preferred embodiment, scenes have a relative start time from the start of the project and this is also the relative time from the start of the associated multimedia file.

Each line has an internally held identifier which is unique across the scene to which the line relates. Each line is associated with a character and is a start time relative to the start of the scene. Each line has a duration and a collection of words spoken in the native language, with additional collections of words for each of the translated languages (if any). Although reference is made in this document to lines comprising words, those skilled in the art to which the invention relates will appreciate the lines may comprise other symbols which an actor or machine may recognise or use to produce certain sounds.

A graphical user interface according to one embodiment of the invention is shown in Figure 4. Referring to that Figure, the interface provides a timeline view that allows a user to edit lines. A first window 20 displays the title of the relevant scene 12, and the lines 14 in the scene together with the character 16 that is to vocalise each line. As can be seen from the background shading associated with each of the character names 16, each character is assigned a different colour to facilitate quick and easy identification of the different characters. The start time 18 for 300431152_1 initiation of vocalisation of the line by the character is recorded alongside each line, together with the end time 19, and an overall duration 21 for each line. The number of words 22 in each line is also provided. In a preferred embodiment the application can therefore be used to keep track of the number of words in each line, provide some of the number of words read by a given character in each scene, and the number of words for any given character across the entire project. This can be advantageous as some actors who vocalise the lines are remunerated on a per word basis.

Window 25 provides a global list of characters 16 for the project (i.e. for the original recording as a whole, across all scenes). Buttons 26 allow the user to select the lines associated with each character that may be displayed in window 20. Therefore, if the user is only interested in editing lines relating to some but not all of the actors (for example, if some actors cannot be present at a recording session, or if it is only the lines of certain actors that require further doubling or synchronisation), then the information portrayed in window 20 can be simplified to relate to only the actors of interest.

A further window 26 provides a summary of the characters 16 in the relevant scene being portrayed in window 20.

Window 28 shows a timeline 29 with time measured in seconds from left to right across the window. An upper portion of the window portrays the words in each line, being spaced according to the time that which each word is to be vocalised by the relevant character. A reference marker 31 marks the current time and this is recorded numerically, as shown by reference 32, adjacent to the marker.

In the lower portion 33 of window 28 a sound wave 34 is illustrated. This is obtained from sound data relating to the original recording. In a preferred embodiment, the original recording is provided or converted into an electronic file, such as a multimedia file, for example a QuickTime™ file. The sound data from this file is sampled by the application and converted into a sound wave based on sound amplitude. We have found that the sudden increases or "bursts" in amplitude tend to occur at the beginning of individual words in each of the lines 16. For example, this can be seen in the initial sudden increase in amplitude 35 in the sound wave 34 which corresponds to the first word in the first line shown in upper portion 30 of window 28.

The sound wave 34 facilitates editing of the lines, and in particular facilitates synchronisation of words or characters within each line with the soundtrack of the original recording. It therefore 300431152 1 facilitates synchronisation of words with the visual images in the original recording, which assists when other languages are used.

Referring to Figure 5, the scene and the lines portrayed in Figure 4 are shown, but this time each of the lines 14 in English is translated into Italian, with the Italian line 14a shown immediately below the English line. In a preferred embodiment, each scene is provided with the unique identifier (or at least an identifier which is unique within the project), and each line has a unique identifier (either within the scene or within the project). Using this information, the application can allow lines associated with scenes, with characters, or with the project is a whole, to be translated into other languages, possibly by different translators, and then associated with the lines of the original voice script.

In Figure 6, the interface is again shown using the same scene and lines as depicted in Figures 4 and 5, but this time the user has used the language selection button 40, so that the lines 14a now appear as the primary lines in window 20, and also appear in upper portion 30 of window 28. As can be seen from the word count 22, the number of words has changed, and the user has used information regarding the total length of time of each line, and the sound wave 34 to arrange the timing of each word within the line so that it is likely to coincide with the lip movements of characters as portrayed in the visual images of the original recording.

This is further illustrated in Figure 7, where further lines from the same scene as shown in the preceding Figures are illustrated in window 28 i.e. the timeline has been moved along approximately 20 seconds.

A user may advance the timeline by using a pointing device such as a mouse to activate buttons 42, or to drag scroll bar 44. Also, the user may "play" the scene while viewing the visual images for the original recording using the outline view as will be described further below.

Lines may be edited by simply selecting the line (for example by using a pointing device) in window 20 and using a keyboard to edit the words or other symbols in the line. Editing is helpful since it may take a number of attempts to provide translated words, or word timings, which come as close as possible to the lip movements portrayed in the visual images. Also, in some recordings the actors words may not match the actor's original script.

Furthermore, in window 28, the interface provides handles at the beginning 45 and end 47 of 300431152J each line which may be selected by a pointing device such as a mouse, and dragged along the timeline to change the beginning time and/or end time of each line. Furthermore, the line as a whole may be selected and trait along the timeline (i.e. so that the overall duration of the line does not change, but the beginning and end time of the line change). In a preferred embodiment a handle is provided for or associated with each word or syllable so that at least the start time (i.e. the time of initiation of vocalisation of the word or syllable) can be changed. Handles may also be attached to or associated with individual letters or other symbols. In this way, the precise timing of a word may be synchronised with the associated visual image.

The application stores the vocalisation time for each word as a proportion of the overall duration of the line. In this way, the mathematics involved in the editing process is greatly simplified. Accumulation errors due to repeated numerical recalculation of the time values may also be eliminated.

The collection of words comprises an ordered list of strings, each of which has time when the first character takes place expressed as a fraction of the length of an entire line. The order list is organised such that the chronological order of the words cannot be changed.

In order to provide the initial timings for each of the lines, the application reads each word and/or symbol within the line from the initial voice script, and applies a predetermined time to each of the various words and/or symbols in the line. For example, the characters or syllables in each word or the words themselves may be assigned predetermined time. A predetermined pause is provided between words, and predetermined time periods are assigned to things such as commas, stops, fricatives and glottals. These times are summed cumulatively to provide an overall estimated time for vocalisation of the line, and to provide an estimated time for at least the initiation of vocalisation of each word in the line, preferably as a proportion of the overall duration of the line.

Then, knowing the real time duration of the line (for example from being provided with information as to the real end time of the line), the estimated vocalisation initiation times are scaled to fit within the available real time. The lines are then displayed as described above, with reference to window 28, and the user can alter the timings as necessary using a point-and-click device in order to quickly make any adjustments.

The output view for the lines shown in window 28 of Figure 7 is shown in Figure 8. This view may be projected at onto a separate screen, shown on a separate monitor or display, or may 300431152 _1 be introduced as a separate window such as an "always on top" window which overlays part of the display shown in Figure 7. As can be seen in Figure 8, the output view portrays the visual images 46, together with words and symbols from the relevant lines. The lines for each character are displayed in a separate track 48 and 50 which are laid over the images 46. Each track carries the background colour associated with the relevant character. The words are scrolled from right to left across the screen, and the actor associated with the relevant character begins to pronounce the word which appears on each track when they word touches marker bar 52.

The output view is shown to actors who read the lines for the various characters based on the information scrolling across the screen. The sounds produced by the actors are recorded, and the recorded sounds can be used to provide a replacement soundtrack or partial soundtrack for the original recording.

The output view or the timeline view can be viewed at any time, which allows editing of lines to be easily carried out during recording sessions, and also allows the synchronisation operations that are performed during the editing process to be quickly tested.

The words in each line are generated for display in the output view by the application analysing the script dynamically to find timing of words in the next portion the script before they are to be spoken or displayed. The script is then displayed at an appropriate speed so that the initialisation of vocalisation of each word or syllable corresponds with the first character of the word or syllable reaching the reference mark 52 at the appropriate time. The user can select the scroll speed. This will typically correspond to the size of the words on the screen. For example, if a slow scroll speed is chosen, then the words will typically appear in a smaller font, or the characters will be placed closer together. A faster scroll speed will typically result in a larger font or the characters being spaced further apart. Therefore there may be some compromise dependent upon the preference of the actor. Also, as can be seen in Figure 8, the font size or character spacing may be altered dynamically by the application dependent upon the available real time for enunciation of the relevant word or emphasis to be given to the word.

The application also allows actors to choose whether they prefer a constant presentation speed or whether they prefer the words to be presented on the screen of fixed inter- character spacing which results in a varying speed between words. 300431152 1 12 It should be rioted that various changes and modifications to the presently preferred embodiments described herein will be apparent to those skilled in the art. Such changes and modifications may be made without departing from the spirit and scope of the present invention and without diminishing its attendant advantages. It is, therefore, intended that such changes and modifications be included within the present invention.

Throughout this document the word "comprise" and variations such as "comprises" and "comprising" is intended to be interpreted in an inclusive sense. 300431152 1 13

Claims (18)

Claims

1. Computer apparatus for displaying a synchronised sound script having at least one symbol representative of a sound event, the apparatus comprising means for generating a graphical user interface on a display screen, the graphical user interface having a control element associated with the at least one symbol, the control element allowing adjustment of the synchronisation of the at least one symbol with an original recording, the control element being adapted to be activated by a pointing device to generate a control signal to vary a vocalisation time for the at least one symbol, means responsive to generation of the control signal by activation of the control element to associate a new vocalisation time for the at least one symbol.

2. Computer apparatus as claimed in claim 1 wherein the graphical user interface displays the at least one symbol and the control element comprises the displayed symbol.

3. Computer apparatus as claimed in claim 1 or claim 2 wherein the interface includes a display of sound amplitude of sounds associated with the original recording.

4. Computer apparatus as claimed in claim 3 wherein the display of sound amplitudes is displayed over a time interval corresponding to the voice track.

5. Computer apparatus as claimed in claim 1 wherein the control element allows the time for initiation of vocalisation and completion of vocalisation for the symbol to be adjusted.

6. Computer apparatus as claimed in any one of the preceding claims wherein the symbol comprises a sentence.

7. Computer apparatus as claimed in any one of claims 1 to 5 wherein the symbol comprises a word.

8. Computer apparatus as claimed in any one of claims 1 to 5 wherein the symbol comprises a syllable.

9. Computer apparatus as claimed in any one of claims 1 to 5 wherein the symbol comprises a plurality of sequential characters. - 7 NOV 2006 RECEIVED 300607629

10. Computer apparatus as claimed in any one of the preceding claims wherein the sound script comprises a voice script for dubbing over the original recording.

11. A method for producing a synchronised sound script comprising providing a sound script comprising a plurality of sound events for at least part of a scene from an existing audio and/or video recording; determining a time period for vocalisation of the script; determining a vocalisation time for each sound event being the time at which the sound event should be vocalised for the sound event to be substantially synchronised with the scene, and; storing the vocalisation time as a proportion of the time period.

12. A method as claimed in claim 11 including the step of adjusting the vocalisation time for a sound event by adjusting the proportion of the time period for that sound event.

13. A method for producing a synchronised sound script comprising the steps of providing a sound script comprising a plurality of sound events for at least part of a scene from an existing audio and/or video recording; determining a real time period provided for vocalisation of the script; estimating a time period required to vocalise each sound event and cumulatively summing the estimated time periods to provide an estimated time relative to a predetermined time for vocalisation of each sound event; using the estimated time periods to determine a total estimated time for vocalisation of the script; scaling the total estimated time to the real time to provide an adjusted estimated time for vocalisation of each sound event relative to the predetermined time, and; using the adjusted estimated times to allow the sound events in the script to be vocalised substantially in synchronism with the scene.

14. A method as claimed in claim 13 including the step of using the adjusted estimated time to display the sound events such that the relative time of occurrence of sound events can be visualised.

15. A method as claimed in claim 14 wherein the step of scaling the total estimated time to the real time provides an adjusted estimated time for initiation of vocalisation of each sound event relative to the predetermined time and wherein the sound events are scrolled past a reference mark so that the time at which a sound event reaches the reference marker is the 300431152 1 15 time at which initiation of the sound event should substantially commence for synchronism with the scene.

16. Computer apparatus programmed to implement the method as claimed in anyone of claims 11 to 15.

17. Software to implement the method as claimed in anyone of claims 11 to 15.

18. Computer apparatus substantially as herein described with reference to any embodiment shown in Figures 2 to 8 of the drawings. CSPEC4415923 Intellectual Property Office ef N.z. " 1 NOV 20U 300607629