JP2007295218A - Nonlinear editing apparatus, and program therefor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a nonlinear editing apparatus which can make images and voices correspond to text data without requiring human work and can generate editing data for editing the images and voices in conjunction with the editing of the text data. <P>SOLUTION: The nonlinear editing apparatus is provided with: a text corresponding means 32 for recognizing a voice and generating text data made to correspond to time information; a display means 50 for displaying the text data and images while making them corresponding to the time information; a text editing means 41 for editing the text data; and an editing data generation means 44 for generating editing data on the basis of the time information corresponding to the edited text data. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a nonlinear editing apparatus that generates editing data for editing video and audio in conjunction with text data and a program thereof.

In recent years, video and audio editing has been performed by playing back a tape on which video or the like has been recorded, copying the desired location to another tape, writing the video or the like as digital data in a storage device such as a hard disk, Shifting to non-linear editing that is performed by a computer.
This non-linear editing system performs non-linear editing such as “copy”, “cut”, “paste”, etc., on the GUI (Graphical User Interface) of a computer terminal by an operator. By doing the above, edit data composed of the start point and end point of video and the like is created. When the edited video or the like is reproduced, the nonlinear editing system reads out and outputs the video or the like between desired start points and end points from the storage device based on the edited data. Thus, the non-linear editing can reduce the time required for the editing as compared with the linear editing.
However, even in the case of non-linear editing, at the editing stage and the stage of confirming the editing content, it is necessary for the operator to reproduce the video and so on, so that it takes a lot of time for editing. .

Therefore, a technique is disclosed in which a sentence (text), which is text data, is associated with a video frame, and the text is edited to edit the corresponding video (see Patent Document 1). In this technique, a sentence that is text data and a start frame number and an end frame number of a video corresponding to the sentence are associated in advance, and the sentence is selected or the order of the sentences is changed. The video access order can be arbitrarily changed.
Thus, the operator can easily edit the video by editing the text data with a small amount of information without editing the video itself having a large amount of information and difficult to edit.
JP-A-9-237486 (paragraphs 0086 to 0089, FIG. 5)

However, according to the above-described conventional technology, it is necessary to associate a sentence that is text data with a video frame in advance, so that it takes a lot of time for input work for the association. There was a problem.
Further, in the conventional technique, since a plurality of video frames are associated with one sentence, there is a problem in that a video editing unit becomes a sentence unit and fine video editing cannot be performed.

  The present invention has been made to solve the above-described problems. Video and audio are associated with text data without human intervention, and the video is linked with editing in units of characters of text data. Another object of the present invention is to provide a text-linked non-linear editing device and a program thereof that can generate edit data for editing audio and voice.

  The present invention was devised to achieve the above object. First, the nonlinear editing apparatus according to claim 1 edits the video and audio based on the video and audio to which time information is added. A non-linear editing apparatus that generates editing data for performing the above-described processing, and includes a text association unit, a display unit, a text editing unit, and an editing data generation unit.

In such a configuration, the non-linear editing apparatus recognizes a voice to which time information corresponding to a video is added by a text association unit, and generates text data associated with the time information. As a result, the text data and the video and audio are associated with each other through time information (time code). At this time, the text data can be associated with the time code in units of at least one character.
In addition, the non-linear editing device divides the display area into a text display area for displaying text data and a video display area for displaying video, and displays the text data and video in association with time information by the display means. Display on the device. As a result, the audio corresponding to the video is displayed as text data in conjunction with the display (playback) of the video.

Then, the non-linear editing device edits the text data by the text editing means based on the operation of the operator. At this time, the editing of the text data corresponds to the editing contents of the video and audio associated with the time information as they are. Then, the nonlinear editing device generates editing data by the editing data generation unit based on the time information corresponding to the text data edited by the text editing unit.
Since the text data and the video are associated with each other by time information, the text data can be edited by editing the video.

  Further, in the nonlinear editing apparatus according to claim 2, in the nonlinear editing apparatus according to claim 1, the text association unit associates the time information with each word break recognized by speech recognition. Text data is generated, and the text editing unit edits the text data in units of words.

In such a configuration, the nonlinear editing apparatus generates text data in which time information is added for each break of a word as a speech recognition result when the text association unit converts speech into text data by speech recognition.
Then, the non-linear editing device edits the text data in units of words by the text editing means. In this way, by setting the minimum unit of editing as a word unit, it is possible to avoid editing a voice in the middle of a word that has no meaning as a voice.

  The nonlinear editing apparatus according to claim 3 is a nonlinear editing apparatus that generates editing data for editing the video based on the video to which time information is added, the text data input means, The configuration includes an association unit, a display unit, a text editing unit, and an editing data generation unit.

In such a configuration, the nonlinear editing apparatus inputs text data corresponding to the video from the outside by means of text data input means. This text data is a manuscript or the like in which sound corresponding to a video is digitized in advance.
Then, the non-linear editing apparatus associates the time information added to the video for each word break based on the instruction of the operator by the text association unit with the text data input by the text data input unit.
In addition, the non-linear editing device divides the display area into a text display area for displaying text data and a video display area for displaying video, and displays the text data and video in association with time information by the display means. Display on the device.

  Then, the non-linear editing device edits the text data by the text editing means based on the operation of the operator. Then, the nonlinear editing device generates editing data by the editing data generation unit based on the time information corresponding to the text data edited by the text editing unit.

  Furthermore, the non-linear editing apparatus according to claim 4 is the non-linear editing apparatus according to any one of claims 1 to 3, wherein the edit data generating means uses a unique control character in the text data. A configuration for recognizing the start point and the end point by describing the start point and end point of the identified edit, and generating the edit data based on time information corresponding to the start point and end point; did.

In such a configuration, the non-linear editing apparatus supports text data by describing unique control characters indicating the editing start point (in point) and end point (out point) in the text data in the text editing means. From the attached time information, the time of the start point and the end point can be recognized.
Then, the edit data generating means generates edit data with the time of the start point and end point as video and audio cut points.

  The nonlinear editing device according to claim 5 is the nonlinear editing device according to any one of claims 1 to 4, wherein the editing data generation means uses a control character unique to the text data. The character super character string is recognized by describing the identified character super character string, and the character super character string is included in the edit data based on time information corresponding to the insertion position of the character super character string. The reproduction information is added.

In such a configuration, the non-linear editing apparatus describes the character super-character from the time information associated with the text data by describing the character super-character string identified by the unique control character in the text data in the text editing means. The time to play can be recognized.
Then, the editing data generation means adds the character string of the character superimposition and the time for reproducing the character superimposition to the editing data.

  Furthermore, the nonlinear editing device according to claim 6 is the nonlinear editing device according to any one of claims 1 to 5, further comprising video editing means for editing the video, wherein the text editing means includes: The text data and the time information associated with the text data are edited based on the time information corresponding to the video edited by the video editing means.

  In such a configuration, the nonlinear editing apparatus edits the video by the video editing unit, so that the text editing unit can edit the text data based on the time information associated with the video.

  The non-linear editing program according to claim 7, wherein a computer is used to generate editing data for editing the video and audio based on the video and audio to which time information is added. , Display means, text editing means, and edit data generation means.

In such a configuration, the non-linear editing program recognizes the voice to which the time information corresponding to the video is added by the text association means, and generates text data associated with the time information. As a result, the text data and the video and audio are associated with each other through time information (time code).
In addition, the non-linear editing program divides the display area into a text display area for displaying text data and a video display area for displaying video, and displays the text data and video in association with time information. Display on the device. As a result, the audio corresponding to the video is displayed as text data in conjunction with the display (playback) of the video.
Then, the non-linear editing program edits the text data based on the operation of the operator by the text editing means.
Then, the non-linear editing program generates edit data by the edit data generating means based on the time information corresponding to the text data edited by the text editing means.

The present invention has the following excellent effects.
According to the first and seventh aspects of the present invention, it is possible to associate video and audio with text data without manual intervention and to edit video and audio at an arbitrary position of the text data. Thus, the editing result can be confirmed by characters without watching the video and audio, and the video and audio editing time can be shortened.

  According to the second aspect of the invention, since editing is performed in units of words, it is not necessary to edit the voice in the middle of a word that has no meaning as a voice, and the efficiency of the editing operation can be improved.

  According to the invention described in claim 3, it is possible to edit the video at an arbitrary position of the text data. As a result, the editing result can be confirmed by characters, so that the video editing time can be shortened.

  According to the fourth aspect of the present invention, it is possible to set the start and end points of video and audio editing simply by inserting unique characters into text data. As compared with the case where the end point is set, the editing operation can be performed in a short time.

  According to the fifth aspect of the present invention, it is possible to generate edit data into which a character super is inserted in a series of text data editing operations. This eliminates the need to search for a place to insert a character superimpose by reproducing the video, and can therefore generate edit data in which the character superposition is set in a short time.

  According to the sixth aspect of the present invention, since the video and text data are associated with each other by time information, the user interface for generating edit data by editing the video as in the past, and editing the text data By doing so, it is possible to coexist with a user interface for generating edit data, and it is possible to improve the convenience for the operator.

Embodiments of the present invention will be described below with reference to the drawings.
[Outline of nonlinear editing equipment]
First, the outline of the nonlinear editing apparatus according to the present invention will be described with reference to FIG. FIG. 1 is a diagram for explaining the outline of the nonlinear editing apparatus according to the present invention, and shows a display screen displayed by the nonlinear editing apparatus.
The non-linear editing apparatus 1 generates editing data for editing video and audio, based on audio to which a time code (time information) corresponding to the video is added.
The non-linear editing apparatus 1 inputs video and audio associated with a time code, and displays text data obtained by recognizing the input audio in a text display area T of the display screen D. Further, the nonlinear editing device 1 displays the input video in the video display area M of the display screen D.

Then, when the operator edits the text data displayed in the text display area T, the non-linear editing device 1 generates edit data for editing video and audio in conjunction with editing of the text data. .
As described above, the nonlinear editing apparatus 1 associates text data with video and audio with the same time code, so that nonlinear editing can be performed by editing text data.
The configuration and operation of the nonlinear editing apparatus according to the present invention will be described below.

<< First Embodiment >>
[Configuration of nonlinear editing device]
First, the configuration of the nonlinear editing apparatus according to the first embodiment of the present invention will be described with reference to FIG. FIG. 2 is a block diagram showing the configuration of the nonlinear editing apparatus according to the first embodiment of the present invention.
As shown in FIG. 2, the nonlinear editing apparatus 1 generates editing data for editing video and audio based on the video and audio to which time information is added.
Here, the nonlinear editing apparatus 1 includes a control unit 10, a storage unit 20, an input unit 30, an editing unit 40, a display unit 50, and an output unit 60.

  The control means 10 controls the entire nonlinear editing apparatus 1. The control means 10 displays a menu screen or the like (not shown) on the screen and executes an operation selected by the operator. Here, the control unit 10 controls the input unit 30, the editing unit 40, the display unit 50, and the output unit 60. Then, the control means 10 operates the input means 30 when inputting the video / audio to be edited, and operates the editing means 40 and the display means 50 when performing editing work, thereby editing the editing data or the like. Is output, the output means 60 is operated.

  The storage means 20 stores externally input video / audio, various data used in the nonlinear editing apparatus 1, or editing results. Here, the storage unit 20 includes a voice recognition data storage unit 21 and an edit data storage unit 22.

The voice recognition data storage means 21 stores various data used for voice recognition such as a language model and an acoustic model used in the voice recognition means 321 of the text association means 32 described later. It is a general storage device.
The language model models the appearance frequency, connection probability, and the like of output sequences (words, morphemes, phonemes, etc.) learned from a large amount of speech data. As this language model, for example, a general “N-gram language model” can be used.
The acoustic model is obtained by modeling a feature amount for each phoneme learned in advance from a large amount of speech data using a “hidden Markov model”. As this acoustic model, a single acoustic model may be used, or a plurality of models may be used for each acoustic type (for example, for each person).

The editing data storage means 22 stores video and audio material data to be edited and editing results, and is a general storage device such as a hard disk. Further, it is assumed that a time code is added to the video and audio stored in the edit data storage means 22. It is assumed that the time code added to the audio is associated with a video frame (video frame) as “hour: minute: second: video frame number”.
Here, the storage means 20 is composed of the voice recognition data storage means 21 and the edited data storage means 22 by separate hard disks, but may be composed of the same hard disk or the like.

  The input unit 30 inputs video / audio based on the control from the control unit 10 and generates text data corresponding to the time code. Here, the input unit 30 includes a video / audio input unit 31 and a text association unit 32.

  The video / audio input means 31 inputs video and audio to be edited from the outside. The video / audio input means 31 stores the input video and audio in the edit data storage means 22.

  The text association means 32 recognizes speech and generates text data corresponding to the time code. Here, the text association unit 32 includes a speech recognition unit 321.

  In addition to a general speech recognition function for converting speech into text data, the speech recognition means 321 adds a video / audio time code in association with text data as a recognition result. Note that the voice recognition unit 321 performs voice recognition using a language model or an acoustic model stored in the voice recognition data storage unit 21. Here, the voice recognition unit 321 includes an analysis unit 321a, a similarity calculation unit 321b, and a search unit 321c.

  The analysis unit 321a extracts a framed waveform by applying a window function (such as a Hamming window) to a speech waveform of speech, and extracts various feature amounts by performing frequency analysis on the waveform. . For example, a cepstrum coefficient or the like, which is a value obtained by inverse Fourier transform of the logarithm of the power spectrum of a framed waveform, is used as the feature amount. In addition to the cepstrum coefficients, general audio feature quantities such as a mel frequency cepstrum coefficient (MFCC), an LPC (Linear Predictive Cording) coefficient, logarithmic power, and the like can be used as the feature quantity. Note that the analysis unit 321a adds the time code added to the video / audio at the time of extracting the feature value to the feature value.

  The similarity calculation unit 321b analyzes the similarity between the feature amount analyzed in the analysis unit 321a and input in time series and the phoneme modeled by the acoustic model stored in the voice recognition data storage unit 21 ( Probability value). The similarity calculation unit 321b outputs the time code added to the feature amount to the search unit 321c together with the phoneme and the similarity (probability value).

The search unit 321c searches for a connected output sequence candidate from the language model stored in the speech recognition data storage unit 21, and recognizes the output sequence having the maximum probability value as a recognition result (text data). Is output as Here, the search means 321 c stores the text data as the recognition result in the edit data storage means 22. The search means 321c adds a time code for each break of words based on the time code added by the analysis means 321a when sequentially outputting text data.
By configuring the voice recognition means 321 in this way, the text data that has been voice-recognized is added with a time code corresponding to each frame of the video as shown in FIG. This text data is output by the display means 50 to a display device (not shown).

  The editing unit 40 generates editing data by editing the video and audio stored in the editing data storage unit 22 and the text data in conjunction with each other based on the control from the control unit 10. . The edit data is one or more sets of time code start points and end points indicating at least sections to be extracted for reorganization in video and audio to be edited. Here, the editing unit 40 includes a text editing unit 41, a video editing unit 42, an audio editing unit 43, and an editing data generation unit 44.

The text editing unit 41 edits text data stored in the editing data storage unit 22. Here, the text editing unit 41 edits the text data by performing an editing operation on the text data displayed by the display unit 50 via an input device such as a mouse or a keyboard (not shown). I do.
That is, the text editing unit 41 performs partial replacement or deletion of text data by the operator performing “copy”, “cut”, and “paste” operations. The text editing unit 41 edits text data in units of words. As a result, editing is performed in units of time codes associated with the text data.

  The text editing unit 41 displays a cursor indicating the editing location on the screen at the same time as the text data. When the operator moves the cursor with a keyboard or the like, the text editing unit 41 is associated with the character corresponding to the cursor position. The corresponding time (code) is displayed by notifying the video editing means 42 of the obtained time code.

  Further, the text editing means 41 sets information for editing in the text data when the operator inputs a predetermined control character or the like in the text data. The text data editing result in the text editing unit 41 is output to the editing data generation unit 44.

  Here, with reference to FIG. 4 (refer to FIG. 2 as appropriate), control characters serving as editing information will be described. FIG. 4 is an explanatory diagram for explaining the control characters, and shows text data displayed on the display screen of the nonlinear editing apparatus.

For example, as shown in FIG. 4, by enclosing an arbitrary character string of text data with a predetermined unique control character (edit identification character, for example, “/”), video and audio are included in the text data. Edit points (start point [in point], end point [out point]) for knitting are set.
At this time, it is also possible to add a predetermined unique control character (effect identification character) that specifies the effect of the effect at the editing point. That is, as a video effect, characters that identify “fade”, “wipe”, “dissolve”, and the like, and their time are added. For example, a character for specifying “fade” (for example, “FO”) and a number indicating the time (for example, “3”) are surrounded by effect identification characters (for example, “<<”, “>>”). As a result, the effect of rendering the video at the editing point is set as a character string (for example, “/ << FO3 >>”) in the text data. In the example of FIG. 4, fade-out is shown in 3 seconds at the end point.
Note that “wipe”, “dissolve”, and the like can be set by setting other control characters in advance.

  Also, by inserting a predetermined unique control character (character super section identification character, character super identification character) and an arbitrary character string (character super character string) into the text data, A character super start point and a character super end point for displaying the character super may be set. For example, by enclosing an arbitrary character string of text data with character super section identification characters (for example, “[”, “]”), a time section for displaying a character super is set in the text data. In addition, by inserting a character super-identification character (for example, ““ ”,“ ””) in that section, a character super character string is set in the text data. To do.

In addition, a silent section may be set by setting a predetermined unique control character (silent designation character, for example, “_”). In this case, it is assumed that one character of the silence designation character is a silence time of a predetermined time length.
Also, for example, by specifying an area for any character string of text data (video / audio separation specification), video and audio are separated in the section of this area, and only video is used for editing It is good to do.

Thus, the text editing means 41 can set various editing contents by inserting predetermined control characters or the like into the text data. The control characters shown in FIG. 4 are examples, and other characters can be used. In addition to the editing information described with reference to FIG. 4, an arbitrary character string is inserted into a predetermined control character (comment designation character, for example, “(”, “)”). May set an irrelevant comment. Thus, the operator can use the comment as a “memo” and can use it as reference information when editing.
Returning to FIG. 2, the description of the text editing means 41 will be continued.

The text editing unit 41 further includes a time display unit 411 and a keyword search unit 412 to assist in editing the text data.
The time display means 411 visualizes the time code associated with the text data. For example, the time display unit 411 displays a time code corresponding to the position by instructing an arbitrary position of the text data via an input device such as a mouse (not shown). At this time, the time display unit 411 may display the time code of the video and audio material data to be edited and the time code corresponding to the edited video and audio as the time code.

  Further, the time display means 411 associates an arbitrary character string of the text data with a mouse or the like so that the video / audio reproduction time corresponding to the character string in the drag area is associated with the text data. It may be calculated from the time code and displayed. Thus, the operator can confirm the video / audio time after editing from the text data.

The keyword search means 412 searches for an arbitrary character string (keyword) from the text data. That is, the keyword search unit 412 displays an input screen for inputting a keyword on the display screen, and searches for the keyword from the text data stored in the edit data storage unit 22 when the keyword is input by the operator. To do.
Thus, the operator can search for a part to be edited by inputting a keyword without searching for a part to be edited by actually reproducing video and audio.

  The video editing means 42 is for editing the video stored in the editing data storage means 22. Here, the video editing unit 42 edits the video by allowing the operator to edit the video displayed by the display unit 50 via an operating device such as a jog / shuttle controller (not shown). Do. For example, the video editing means 42 plays and stops the video in units of frames by the operator via the jog / shuttle controller, and sets the editing points (start point, end point). Alternatively, when an operator presses an operation button or the like on the display screen with a mouse or the like, video playback / stop and edit point setting are performed.

At this time, the video editing means 42 inserts an editing identification character (see FIG. 4) at a location corresponding to the time code of the editing point with respect to the text data. As a result, even when an edit point is set depending on the video, the text data is edited in conjunction with the video.
When a scene of a video desired by the operator is displayed, the scene may be displayed by moving the cursor on the timeline associated with the time code.

  The voice editing unit 43 edits the voice stored in the editing data storage unit 22. The audio editing unit 43 performs audio editing in association with the time code of the video edited by the video editing unit 42. Furthermore, the voice editing unit 43 also edits the voice based on the time code of the text data edited by the text editing unit 41.

The edit data generation means 44 is for editing video and audio based on the control characters for editing inserted in the text data (see FIG. 4) and the time code associated with the text data. Edit data is generated.
That is, the edit data generation means 44 searches the various control characters described with reference to FIG. 4 to generate edit contents corresponding to each control character as edit data.

  For example, the edit data generation unit 44 searches the text data for an edit identification character indicating an edit point (start point, end point), and sets the time code set in the word immediately after the first edit identification character, It is the time code of the starting point. The edit data generation means 44 uses the time code corresponding to the frame of the video immediately before the time code set in the word immediately after the second edit identification character as the end time code.

Further, when the text data includes an effect identification character for specifying the effect, the editing data generating means 44 specifies the specified effect from the time code corresponding to the position where the effect identification character is inserted. Write the code for effect to edit code.
In addition, when the text data includes a character super section identification character, the edit data generation unit 44 uses the time code set in the word immediately after the first character super section identification character as the character super start point. Time code. The edit data generation means 44 uses the time code corresponding to the frame of the video immediately before the time code set in the word immediately after the second character super section identification character as the time code at the character super end point. . Then, the edit data generation unit 44 sets a character string surrounded by character super identification characters between the character super section identification characters as a character super character string.

Further, when the text data includes a silence designation character, the edit data generation means 44 describes the content of editing the video in the edit data only for the silence section indicated by the silence designation character.
The generated edit data is stored in the edit data storage unit 22.

Here, a specific example of the edit data generated by the edit data generating unit 44 will be described with reference to FIG. FIG. 5 is a data structure diagram of edit data generated by the edit data generating means. Here, “number” indicates the serial number of the edit data, and is a serial number assigned in order from the top of the edit data.
“Edit target” is information for specifying an edit target. Here, “VA” indicates that the editing target is both video and audio, and “V” indicates that the editing target is only video. “Edited content” is information for specifying the edited content for the editing target. Here, the operation content for extracting (cutting) the video at the start point and end point of the edit point is indicated by “C”. Furthermore, as another operation, fade-out is indicated by “FO”. The start point and end point indicate the start and end time codes to be edited.

Also, to add character super information to the edit data, as shown by “number 015”, the “edit target” is the video “V” and the “edit contents” is the identification character “S” indicating the addition of the character super. "Is added to the character superscript character string, and the character super display time is described as the start point and end point.
As described above, the edit data generation means 44 can generate the edit data shown in FIG. 5 from only the edited text data by the operator.
Returning to FIG. 2, the description of the configuration of the nonlinear editing apparatus 1 will be continued.

  The display means 50 divides the display area into at least a text display area for displaying text data and a video display area for displaying video, and associates the text with the time code with respect to a display device (not shown). It displays (outputs) data and video. Here, the display unit 50 includes a text display unit 51, a video / audio display unit 52, and an editing time axis display unit 53.

The text display means 51 displays text data in the text display area of the display device. The text display means 51 displays text data in conjunction with the time code of the video reproduced by the video / audio display means 52. While the video is being reproduced, the text data is displayed based on the time code. Scroll to display. Further, the text display means 51 displays the cursor C _T (see FIG. 1) at the position of the text data indicating the current editing position.

  The video / audio display means 52 displays video in the video display area of the display device. Further, the video / audio display means 52 outputs audio to an audio output device such as a speaker (not shown). The video / audio display means 52 notifies the text display means 51 of the time code during the reproduction of the video / audio.

The video / audio display means 52 displays an operation button for reproducing and displaying the video as an icon in addition to displaying the video. For example, as shown in FIG. 1, operation buttons such as “rewind”, “play”, “fast forward”, “stop”, and setting buttons for setting a start point and an end point are displayed. Is pressed, the video editing means 42 edits the video.
The video / audio display means 52 displays a timeline associated with the time code of the video, and the cursor C _M (on the time axis corresponding to the scene (frame) corresponding to the time code of the currently displayed video. 1) is displayed.

The editing time axis display means 53 displays the relationship between the video and audio material data to be edited, the edited video and audio, the added character super, and the time in the editing time axis display area of the display device. A timeline visualized along the axis is displayed. The editing time axis display means 53 refers to the video, audio, and editing data stored in the editing data storage means 22, and indicates the start and end points of the video and audio for all time areas of video and audio. By calculating the position on the time axis, a video and audio timeline is displayed in the editing time axis display area L as shown in FIG.
The editing time axis display means 53 displays the cursor C _L (see FIG. 1) on the time axis corresponding to the scene (frame) corresponding to the time code of the currently displayed video.

  The output means 60 outputs editing data or the like as video / audio editing results. Here, the output unit 60 includes an edit data output unit 61 and a text data output unit 62.

  The edit data output means 61 outputs edit data stored in the edit data storage means 22 as an edit result. This edit data is used as data for actually editing video / audio.

The text data output means 62 outputs the text data stored in the edit data storage means 22 as an editing result. The text data may include time code, or may be data of only a character string that does not include time code. As a result, the operator can check the editing result with text data in addition to video and audio, thereby simplifying the editing checking operation.
This text data can also be used for other purposes as metadata and caption data for video and audio.

By configuring the nonlinear editing apparatus 1 as described above, editing data for editing video and audio can be generated in conjunction with text data. This makes it possible to perform an editing operation that has been performed over time while viewing video and audio by a simple operation of editing text data.
The non-linear editing apparatus 1 can be operated by a non-linear editing program that causes a general computer to function as each means described above.

[Operation of non-linear editing device]
Next, the operation of the nonlinear editing apparatus 1 will be described with reference to FIGS. Here, the operation of the nonlinear editing device 1 will be described by dividing it into three operations: video / audio input operation, editing operation, and output operation. FIG. 6 is a flowchart showing the video / audio input operation of the nonlinear editing apparatus. FIG. 7 is a flowchart showing the editing operation of the nonlinear editing apparatus. FIG. 8 is a flowchart showing the output operation of the nonlinear editing apparatus.

(Input operation)
First, the input operation of the nonlinear editing device 1 will be described with reference to FIG. 6 (refer to FIG. 2 as appropriate).
First, the nonlinear editing apparatus 1 inputs video and audio to be edited from the outside by the video / audio input means 31 of the input means 30 and stores them in the edit data storage means 22 (step S1).

After that, the non-linear editing device 1 converts the sound into text data associated with the video / audio time code by the speech recognition unit 321 of the text association unit 32.
That is, the nonlinear editing apparatus 1 extracts a framed waveform by applying a window function to the speech waveform of the speech stored in the edited data storage unit 22 by the analyzing unit 321a, and frequency-analyzes the waveform. Thus, the feature amount is extracted (step S2). Further, the analyzing unit 321a associates the feature quantity with the time code by adding the time code added to the voice at the time of extracting the feature quantity to the feature quantity (step S3).

  Then, the nonlinear editing device 1 uses the similarity calculation unit 321b to measure the similarity between the feature amount extracted in step S3 and the phoneme modeled by the acoustic model stored in the speech recognition data storage unit 21. (Probability value) is calculated (step S4).

  Further, the non-linear editing device 1 uses the search means 321c to search for a connected output series candidate from the language model stored in the speech recognition data storage means 21, and recognizes the output series having the maximum probability value. In addition to the result (text data), the time code is associated with each word break based on the time code associated in step S3 (step S5).

Then, the non-linear editing device 1 stores the text data to which the time code is added for each word break by the text association unit 32 in the editing data storage unit 22 (step S6).
With the above operation, the nonlinear editing apparatus 1 generates text data corresponding to each word in the video / audio time code.

(Editing action)
Next, the editing operation of the nonlinear editing apparatus 1 will be described with reference to FIG. 7 (refer to FIGS. 1 and 2 as appropriate). Here, an operation in which the operator edits text data in the text display area T will be described.

  First, the non-linear editing device 1 analyzes the operation of the operator input via the input device such as a mouse or a keyboard (not shown) by the text editing means 41 (step S11).

Here, the operator has performed the operation, when an operation of moving the cursor C _T of the text display region T (Yes in step S12), the nonlinear editing apparatus 1, the movement of the cursor C _T of the text display region T Along with this, the playback position of the video / audio is moved (step S13).
At this time, nonlinear editing apparatus 1, the text editing means 41, a time code of the text data corresponding to the position of the moved cursor C _T, read from the editing data storing means 22, the video editing unit 42, the image display area M The video to be displayed is moved to the scene corresponding to the time code. The editing time axis display means 53, the cursor C _L to be displayed on the editing time axis display area L, is moved to the position of the scene (frame) corresponding to the time code.

Further, when the operation performed by the operator is an operation for editing text data in the text display area T, that is, “copy”, “cut”, “paste”, and edit points (start point, end point) are set. (Yes in step S14), the nonlinear editing apparatus 1 performs video / audio editing in response to partial replacement or deletion of text data (step S15).
At this time, the non-linear editing apparatus 1 reads the time code of the text data edited by the text editing unit 41 from the editing data storage unit 22, and the video editing unit 42 edits the video corresponding to the time code of the text data. After that, the edited video is displayed in the video display area M. The voice editing unit 43 edits the voice corresponding to the time code of the text data. Then, the edit data generation means 44 adds the time code corresponding to the position where the edit identification character inserted in the text data is inserted as edit points (start point, end point) to the edit data.

Further, when the operation performed by the operator is an operation for changing a transition, that is, an operation for setting a character and time for identifying “fade”, “wipe”, “dissolve”, etc. in the text data. In the case (Yes in step S16), the nonlinear editing device 1 adds the type and time of the transition to the editing data (step S17).
At this time, in the non-linear editing apparatus 1, when the text editing means 41 receives a character (effect identification character) indicating the effect of the image in the text data, the editing data generation means 44 inserts the effect identification character. From the time code corresponding to the current position, a code for performing the specified effect is added to the edit code.

In addition, when the operation performed by the operator is an operation of adding a character super, that is, an operation of inserting a character string (character super character string) sandwiched between character super identification characters into text data (Yes in step S18). The nonlinear editing device 1 adds the character string of the character superimpose and the display time thereof to the editing data (step S19).
At this time, in the non-linear editing device 1, when a character super character string sandwiched between character super identification characters is input to the text data in the text editing unit 41, the character super identification character is inserted by the editing data generation unit 44. A code indicating that the character super character string is displayed in the time section indicated by the time code corresponding to the two positions is added to the edit code.

When the operation performed by the operator is an operation for displaying time information of text data (Yes in step S20), the nonlinear editing device 1 displays the corresponding time on the display screen based on the time code. (Step S21).
At this time, the non-linear editing apparatus 1 displays a time code corresponding to the position on the display screen by instructing an arbitrary position of the text data by the time display means 411 via an input device such as a mouse. Alternatively, the time display means 411 calculates a time corresponding to the character string in the drag region by dragging an arbitrary character string of the text data with a mouse or the like from the time code associated with the text data, The calculated time, that is, the reproduction time is displayed on the display screen.

When the operation performed by the operator is an operation for searching for a keyword (Yes in step S22), the nonlinear editing device 1 searches for the keyword in the text data (step S23).
At this time, the nonlinear editing apparatus 1 searches the keyword from the text data stored in the editing data storage unit 22 by the keyword search unit 412.
Here, (Yes in step S24), and the keyword search means 412 when succeeding in searching keywords, the movement position of the cursor C _T, is set to a position of a search result keyword, the process goes back to step S12, the keyword Is displayed in the text display area T of the display screen D.

  Then, the nonlinear editing apparatus 1 determines whether or not the end of the editing operation has been instructed (step S25). If the end has not been instructed (No in step S25), the nonlinear editing apparatus 1 returns to step S11 and continues the operation. On the other hand, when the end is instructed (Yes in step S24), the editing ends. The instruction to end the editing operation is given when the control unit 10 displays a menu screen (not shown) and the operator selects the end of editing.

With the above operation, the nonlinear editing apparatus 1 can generate editing data in conjunction with text data.
Here, the operation of generating the edit data in conjunction with the text data has been described. However, since the video / audio and the text data are associated with each other by the time code, the video / audio can be edited. Then, edit the text data accordingly and generate the edited data.

(Output operation)
Next, the output operation of the nonlinear editing apparatus 1 will be described with reference to FIG. Here, it is assumed that the control unit 10 displays a menu screen (not shown) and the operator selects desired data.

First, the nonlinear editing apparatus 1 analyzes the instruction content selected by the operator by the output means 60 (step S31).
Here, when the instruction content selected by the operator is output of edit data (Yes in step S32), the non-linear editing device 1 uses the edit data output means 61 to edit the edit data stored in the edit data storage means 22. Data is read and output (step S33).

When the instruction content selected by the operator is output of text data to which time information is not added (Yes in step S34), the non-linear editing device 1 uses the text data output means 62 to edit the edit data storage means 22. Only character information is extracted from the text data stored in and output (step S35).
When the instruction content selected by the operator is output of text data to which time information is added (Yes in step S36), the non-linear editing device 1 uses the text data output means 62 to store the edit data storage means 22. The stored text data is read and output as it is (step S37).

  Then, the nonlinear editing apparatus 1 determines whether or not the end of the output operation has been instructed (step S38). If the end has not been instructed (No in step S38), the non-linear editing apparatus 1 returns to step S31 and continues the operation. On the other hand, when the termination is instructed (Yes in step S38), the output operation is terminated. The instruction to end the output operation is given by the control means 10 displaying a menu screen (not shown) and the operator selecting the output end.

With the above operation, the non-linear editing apparatus 1 can output text data in addition to editing data. In addition, in the output of this text data, by making it possible to select whether or not to add a time code, the text data can be used as metadata or subtitle data in addition to being used for checking the edit data. It becomes possible.
The configuration and operation of the nonlinear editing device 1 have been described above, but the present invention is not limited to this. The configuration of another nonlinear editing apparatus will be described below.

<< Second Embodiment >>
With reference to FIG. 9, the structure of the nonlinear editing apparatus based on 2nd Embodiment of this invention is demonstrated. FIG. 9 is a block diagram showing the configuration of the nonlinear editing apparatus according to the second embodiment of the present invention.
The non-linear editing apparatus 1 described with reference to FIG. 2 generates text data to which a time code is added when video / audio is input and voice recognition is performed. After the data is generated, the time code is set.
That is, the non-linear editing apparatus 1B is different from the input means 30 of the non-linear editing apparatus 1 described in FIG. 2 in the input means 30B, and the other configurations are the same. Therefore, the components other than the input unit 30B are denoted by the same reference numerals as those of the nonlinear editing apparatus 1 described with reference to FIG.

  The input unit 30B includes a video / audio input unit 31 and a text association unit 32B. The video / audio input means 31 has the same configuration as that of the video / audio input means 31 of the nonlinear editing apparatus 1 (FIG. 2), and thus the description thereof is omitted.

  The text associating unit 32B generates text data by recognizing speech, and associates video / audio with text data by setting a time code for the text data. Here, the text association unit 32 includes a voice recognition unit 321B and a time allocation unit 322.

  The speech recognition unit 321B converts speech into text data, and performs speech recognition using a language model or acoustic model stored in the speech recognition data storage unit 21. The speech recognition unit 321B is a general speech recognition unit that does not have a function of adding a time code to text data during speech recognition, like the speech recognition unit 321 described in FIG.

  The time assigning means 322 sets time information (time code) for each word break in the text data generated by the speech recognition means 321B. Here, the time allocating means 322 sets the time code at the designated position of the text data by inputting the designated position divided by the word unit of the text data and the time code assigned to the designated position. Note that the time allocation means 322 sets time codes at positions divided by other word units by setting two or more time codes in the text data.

  Here, with reference to FIG. 10 (refer to FIG. 9 as appropriate), an operation in which the time allocation unit 322 sets the time code in the text data will be described. FIG. 10 is an explanatory diagram for explaining the operation of setting the time code performed by the time allocating means.

FIG. 10A shows the content of the text data after the speech recognition unit 321B recognizes the speech. Here, the text data is divided (T1 to T5) for each word.
Here, the time allocation means 322 inputs the designated position and time code of the text data, and sets the time code at the designated position. In the example of FIG. 10, the time allocation unit 322 displays the text data of FIG. 10A by setting the first time code between “flower” and “no” (T1). As shown in FIG. 10B, a time code (TA) is added to the position of T1.

Further, the time allocating unit 322 sets the second time code between “cell” and “from” (T3) in the text data, so that the time allocating unit 322 is configured as shown in FIG. As shown, the time code (TB) is added to the position of T3.
At this time, the time allocation means 322, based on the time code (TA and TB) set in T1 and T3, sets the time code (TC, TD and TE) to the position (T2, T4 and T5) between other words. ) Is added (automatic allocation). For example, the time allocation unit 322 sets the time code (TC, TD, and TE) by linearly interpolating the already set time code (TA and TB).

The linear interpolation performed by the time allocating unit 322 may simply linearly interpolate the time code based on the number of characters of each word. Further, when the number of phonemes of each word is known by the speech recognition unit 321B, linear interpolation may be performed based on the number of phonemes of each word. In addition, when the time length of each word is known by the speech recognition unit 321B, linear interpolation may be performed based on the time length.
Note that the time allocation unit 322 reassigns the time code when the time code is set again at a position where the time code has already been set. Thereby, the accuracy of the time code can be increased.

  The configuration and operation of the nonlinear editing apparatus 1B have been described above. The time allocation unit 322 is incorporated into the nonlinear editing apparatus 1 described with reference to FIG. The time code set by the operator may be added to the text data so that the operation is switched as appropriate.

<< Third Embodiment >>
Next, the configuration of a nonlinear editing apparatus according to the third embodiment of the present invention will be described with reference to FIG. FIG. 11 is a block diagram showing the configuration of the nonlinear editing apparatus according to the third embodiment of the present invention.
In the non-linear editing apparatus 1 (see FIG. 2) and the non-linear editing apparatus 1B (see FIG. 9), text data is generated by recognizing the input voice. It is good also as inputting.
That is, the non-linear editing apparatus 1C is different in the input means 30C from the input means 30 and 30B of the non-linear editing apparatuses 1 and 1B described with reference to FIGS. 2 and 9, and the other configurations are the same. Therefore, the components other than the input unit 30C are denoted by the same reference numerals as those of the nonlinear editing apparatuses 1 and 1B described in FIG. 2 and FIG.

The text data input means 33 is for inputting text data such as a document that has been digitized in advance from the outside. The text data input means 33 stores the input text data in the edit data storage means 22.
The text association unit 32C associates the video with the text data by setting a time code for the text data input by the text data input unit 33. Here, the text association unit 32C includes a time allocation unit 322. The time allocation means 322 is the same as that already described with reference to FIG.
By configuring the non-linear editing apparatus 1C in this way, an inexpensive configuration that does not perform speech recognition can be achieved.

It is explanatory drawing for demonstrating the outline | summary of the nonlinear editing apparatus which concerns on this invention. 1 is a block diagram illustrating a configuration of a nonlinear editing device according to a first embodiment of the present invention. It is a data structure figure which shows the structure of the text data which added the time code. It is explanatory drawing for demonstrating a control character. It is a figure which shows an example of the edit data which an edit data generation means produces | generates. It is a flowchart which shows the video / audio input operation of a nonlinear editing apparatus. It is a flowchart which shows the edit operation | movement of a non-linear editing apparatus. It is a flowchart which shows the output operation | movement of a non-linear editing apparatus. It is a block diagram which shows the structure of the nonlinear editing apparatus which concerns on 2nd Embodiment of this invention. It is explanatory drawing for demonstrating the operation | movement of the time code setting which a time allocation means performs. It is a block diagram which shows the structure of the nonlinear editing apparatus which concerns on 3rd Embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1, 1B, 1C Non-linear editing apparatus 10 Control means 20 Storage means 30 Input means 31 Image | video audio input means 32 Text matching means 321 Speech recognition means 33 Text data input means 40 Editing means 41 Text editing means 42 Video editing means 43 Audio editing Means 44 Editing data generation means 50 Display means 60 Output means

Claims (7)

A non-linear editing device that generates edit data for editing the video and audio based on the video and audio to which time information is added,
A text association means for recognizing the speech and generating text data associated with the time information;
Display means for dividing the display area into a text display area for displaying the text data and a video display area for displaying the video, and displaying the text data and the video on a display device in association with the time information. When,
Text editing means for editing the text data based on an operator's instruction;
Edit data generating means for generating the edit data based on time information corresponding to the text data edited by the text editing means;
A non-linear editing apparatus characterized by comprising: The text association means generates the text data by associating the time information for each break of a word recognized by speech recognition,
The non-linear editing apparatus according to claim 1, wherein the text editing unit edits the text data in units of words. A non-linear editing device that generates editing data for editing the video based on the video with time information added thereto,
Text data input means for inputting text data corresponding to the video;
Text associating means for associating the time information with each word break based on the operator's instruction to the text data input by the text data input means;
Display means for dividing the display area into a text display area for displaying the text data and a video display area for displaying the video, and displaying the text data and the video on a display device in association with the time information. When,
Text editing means for editing the text data based on an instruction from the operator;
Edit data generating means for generating the edit data based on time information corresponding to the text data edited by the text editing means;
A non-linear editing apparatus characterized by comprising: The edit data generating means
Based on the time information corresponding to the start point and end point by recognizing the start point and end point by describing the start point and end point of editing identified by a unique control character in the text data The nonlinear editing apparatus according to claim 1, wherein the editing data is generated. The edit data generating means
By describing a character super character string identified by a unique control character in the text data, the character super character string is recognized, and based on time information corresponding to the insertion position of the character super character string, 5. The nonlinear editing apparatus according to claim 1, wherein reproduction information of the character super character string is added to the editing data. 6. Video editing means for editing the video,
The text editing unit edits the text data and the time information associated with the text data based on the time information corresponding to the video edited by the video editing unit. The nonlinear editing device according to claim 5. In order to generate editing data for editing the video and audio based on the video and audio to which the time information is added,
A text association means for recognizing the voice and generating text data associated with the time information;
Display means for dividing the display area into a text display area for displaying the text data and a video display area for displaying the video, and displaying the text data and the video on a display device in association with the time information. ,
Text editing means for editing the text data based on an instruction from the operator;
Edit data generating means for generating edit data for editing the video based on time information corresponding to the text data edited by the text editing means;
Non-linear editing program characterized by functioning as

Images