JP4341390B2 - Error correction method and apparatus for label sequence matching and program, and computer-readable storage medium storing label sequence matching error correction program - Google Patents

Description

  The present invention relates to an error correction method and apparatus for label sequence matching, a computer program, and a computer-readable storage medium storing a label sequence matching error correction program, and more particularly for comparing amino acid sequences of proteins and the like. FIELD OF THE INVENTION The present invention relates to a label sequence matching error correction method and apparatus for interactively correcting a matching error that occurs in label sequence matching, a program, and a computer-readable storage medium storing a label sequence matching error correction program. .

  Matching (or association) of a series of labels is used in many situations. This is a technique used when associating similar sentences with each word so as to minimize the deviation and comparing the amino acid sequences of proteins.

  In recent years, with the development of multimedia technology, video and audio can be handled on a computer. Along with this, matching between different types of media is requested so that character information such as text of a manuscript or script and an audio signal (usually a video program includes audio) spoken by a speaker in the video program are associated. This has also been attempted (for example, see Non-Patent Documents 1 and 2).

The matching problem can be mathematically regarded as a minimization problem under a certain cost function. Here, A and B are set as labels. Let a be an element of A and b be an element of B (a∈A, b∈B). A original sequence S _A = {a ₁ , a ₂ , a ₃ ,..., A _n } (a _i εA, 1 ≦ i ≦ n) is a label sequence of A, and B original sequence S _B = { Let b ₁ , b ₂ , b ₃ ,..., b _m } (b _i εB, 1 ≦ i ≦ m) be a label string of B.

Let w be the matching result of the label string S _A and the label string S _B. w is a mapping which maps the integer [1, n] to an integer [1, m], the i-th label string _{S A} label _{a i} and, the label string _{S B} of w (i) th label _{b w ( i)} means that it corresponds. In general, some restrictions are imposed on w. For example, the restriction that the order of the labels is not reversed (if i <j, w (i) ≦ w (j), or the inclination restriction that the value of w does not change abruptly (w (i + 1) ≦ w (i) + Α and α are constants).

  Let c (a, b) be the cost for associating label a with label b. The cost c (a, b) is a function of a and b, and takes a smaller value as a and b are similar, and takes a minimum value 0 when they completely match.

The cost C of the matching result w between the label string S _A and the label string S _B is expressed by the following formula 1.

と定義する。最小のコストＣを与えるｗを、ラベル列Ｓ_Ａとラベル列Ｓ_Ｂの最適マッチング結果ｗ_０と定義する。Ｃ（ｗ）は、マッチング結果をスカラー量に写す関数であり、コスト関数と呼ぶことにする。よって、２系列のマッチングを求める問題は、コスト関数の下での最小化問題に帰着する。

It is defined as The w that gives the minimum cost C is defined as the optimum matching result w ₀ between the label sequence S _A and the label sequence S _B. C (w) is a function that copies the matching result to a scalar quantity, and is called a cost function. Therefore, the problem of obtaining two series of matching results in a minimization problem under a cost function.

  A general minimization algorithm can be used to find the match. For example, a full search or an annealing method can be used (see, for example, Non-Patent Document 3). In particular, when there is no reversal of the order of arrangement, a method called DP matching by high-speed dynamic programming (dynamic programming, DP) is often used.

  These minimization algorithms seek the best solution under a given cost function.

  Note that the minimization problem is also called a cost minimization problem, an energy minimization problem, an optimization problem, or the like. Also, since the problem is to maximize the cost if the sign is reversed, the minimization problem and the maximization problem are essentially the same problem.

  Here, some terms used in this specification are defined.

  “Correspondence is“ correct ”or“ error ”” refers to a result of a person examining and determining the correspondence. The matching result calculated by the cost minimization method is always correct in the sense of minimizing the cost, but humans who use the matching result judge correctness based on another unique standard, so the result is not necessarily obtained. It is not acceptable. In addition, when associating speech signals with text, there may be cases where errors are mixed in the speech recognition results. In such cases, even if matching is performed to minimize costs, errors may occur. It is considered that it is included, and the correctness cannot be determined by evaluating the cost, and it must be determined by human judgment.

Matching by a mechanical procedure such as a cost minimization method is called “automatic matching”. Although it has the meaning of being executable on an automatic machine, it does not necessarily mean that it must be executed on a machine.
2000 Final Report on Broadcasting Software Production Technology Research and Development Project for Visual and Visually Impaired Persons, Communications and Broadcasting Organization Masatake Tanimura, Hiroshi Nakagawa, “Time Synchronization Method of Drama Video Audio Track and Scenario Dialogue”, Information Processing Society of Japan, 118th Research Report on Intelligence and Complex Systems, SIG-ICS, no. 188-4, pp. 25-31 Richard O. Duda, Peter E. Hart and David G. Stork, Translated by Morio Onoe, Pattern Identification, New Technology Communications

  However, the above-described method for obtaining matching as a cost function minimization problem does not work even if an attempt is made to obtain matching of different media sequences.

  Consider as an example the correspondence between a script character string and a sequence of speech recognition results of uttered speech. In such a case, the utterance is not always made according to the manuscript. In the course of program production, words different from the script may be inserted or deleted, and the order of words may be changed. Also, with the current technical level, it is difficult to eliminate errors in speech recognition. In reality, it is necessary to assume that speech recognition results include errors. Therefore, the script character string and the speech recognition result do not always coincide with each other.

  Even in such a case, the best matching result can be obtained in the sense of minimizing the cost function, but this is not necessarily acceptable to the user. If the criterion of the matching result desired by the user is constant and can be formulated, it will be possible to bring the matching result closer to the matching result desired by the user by reflecting them in the cost function. However, it may be difficult to formulate the user's wishes. In addition, there are cases where the user's desire is different each time matching is performed, and it is not allowed to generate a work for formulation each time. Therefore, in reality, a process of correcting the matching result as each user desires is required. In the matching correction process, the user compares the program with the script and corrects the matching result while judging whether the matching result is correct or incorrect.

  In addition, there may be a need for global correction, which creates further problems. For example, assuming a case where the matching result desired by the user is deviated by one label, all the deviated labels must be corrected, which increases the number of steps.

  In addition, since it is difficult to know where and how many labels should be corrected by visual inspection, if you try to correct the matching result simply, visually check the matching result from the beginning to the end. The procedure will be to determine whether the response is correct and correct, and to respond to all correct responses to the correct response destination.

On the other hand, without using the automatic matching, if attempted matching initially manually and visually label string S _A and the label string S _B from beginning to end, the procedure of adapt to the correct corresponding destination for all corresponding Let's go.

Then, what is the difference between the time T ₁ required for correcting the error after executing automatic matching and the time T ₂ required for manually matching without using automatic matching? When T1 is formulated, Equation 2 is obtained.

T ₁ = t _auto + Nt _check + n _miss t _bind.
Here, t _auto is the time required for automatic matching, N is the number of correspondences, t _check is the time required to determine whether each correspondence is correct, n _miss is the number of incorrect correspondences, and t _bind is The time required to find and associate the correct correspondence destination for one correspondence. When formulating latter time T _2, so that the Equation 3.

T ₂ = Nt _check + Nt _bind
Estimate roughly the difference between T ₁ and _{T 2.} Since t _auto can be executed by a machine, it is regarded as being very small and approximated by zero. In addition, since the work for checking correctness by visual inspection will be simpler and less time-consuming than the work for associating with the correct correspondence destination, t _check is approximated by 0 _assuming t _check << t _bind . Then, the ratio T ₁ / T ₂ between them is at most about n _miss / N. Here, as mentioned above, considering that there are cases where the number of correspondence errors is small and the number of steps to be corrected may increase, the case of manual correction using a simple procedure using automatic matching The difference between the manual work load and the manual work load when matching is performed manually is not very small in the former case. At the same time, it is considered that it is still effective if there is a method for correcting the matching result manually, which requires a short time or a method with a small number of steps.

  As can be understood from the above discussion, even if a method that can be executed at high speed by a machine called automatic matching is used, if it is unavoidable to introduce matching errors, the speed advantage will be fully enjoyed in the end. There is a problem that it will not be possible.

  The present invention has been made in view of the above points, and in a correction process for matching, an error correction method and apparatus for label series matching and a program for reducing the number of work required for the correction, and a matching error correction program for a label series It is an object of the present invention to provide a computer-readable storage medium that stores information.

  FIG. 1 is a diagram for explaining the principle of the present invention.

The present invention (claim 1), the label string _{S A = {a 1, a} 2, a 3, ..., a n} in individual elements of the label string _{S B = {b 1, b} 2, b 3 ,..., B _m } in a label sequence matching error correction method using an information processing apparatus that corrects a matching result w of a label sequence that associates any element of
Using the cost c (a, b) taking a small value enough to label a and labels b are similar, the cost minimization method that minimizes the sum C of the label string S _A viewed from the cost c (a, b) by obtains the matching results w and the minimum cost C, the matching process to be stored in remembers means (step 1),
A large cost c _i is given to the correspondence between the label a _i and the label b _{w (i)} corresponding to the label a _i in the matching result w, and the sum C _i of the cost c (a, b) reflecting the cost c _i is given. if by the cost minimization method to minimize Lima etching results were obtained w _i and the minimum cost C _i, said minimum cost C _i is small and, ing large distance d between the matching result w and w _i one or a correction candidate selection process for selecting a plurality of candidate a _i (step 2),
The matching result w is output to the display means, the candidate a _i is highlighted,
Get the correctness determination result indicating whether the corresponding OPERA author or al candidate a _i and b _{w (i)} is correct or incorrect (Step 3),
If the correspondence between the candidates a _i is incorrect, a large cost c is given to the correspondence between the labels a _i and b _{w (i)} , while if the correspondence between the candidates a _i is correct, the labels a _i and b _{w (} a cost giving process (step 4) in which a small cost c is given to the correspondence of _i) and the cost c is stored in the storage means;
Based on the cost c of a change in cost imparting process is performed again to repeat the matching process (Step 1) subsequent processing (step 5) repeating process, the.

Further, the present invention (Claim 2), in the cost granting process,
When the operator determines the correspondence between the labels a _i and b _{w (i)} , if the correspondence is incorrect and the label b _j is selected as the correct correspondence destination of the label a _i ,
A large cost c is given to the correspondence between the labels a _i and b _{w (i)} , and a small cost c is given to the correspondence between the labels a _i and b _j , and the two costs c are stored in the storage means.

  FIG. 2 is a principle configuration diagram of the present invention.

The present invention (claim 3), the label string _{S A = {a 1, a} 2, a 3, ..., a n} in individual elements of the label string _{S B = {b 1, b} 2, b 3 ,..., B _m } , a label sequence matching error correction device that corrects the matching result w of the label sequence in which any of the elements is associated ,
Using the cost c (a, b) taking a small value enough to label a and labels b are similar, the cost minimization method that minimizes the sum C of the label string S _A viewed from the cost c (a, b) by obtains the matching results w and the minimum cost C, the matching processing unit 111 to be stored in the memorize means,
Giving a large cost c _i to the correspondence between the label a _i and the label b _{w (i)} corresponding to the label a _i in the matching result w, and minimizing the sum C _i of the cost c (a, b) reflecting the cost c _i if by the cost minimization method Lima etching results were obtained w _i and the minimum cost C _i to, said minimum cost C _i is small and, ing large distance d between the matching result w and w _i 1 Correction candidate selection means 112 for selecting one or a plurality of candidates a _i ,
It outputs the matching result w on the display means, to highlight the candidate a _i, interface means for acquiring correctness determination result indicating whether the corresponding OPERA author or al candidate a _i and b _{w (i)} are correct or incorrect 113,
If the correspondence between the candidates a _i is incorrect, a large cost c is given to the correspondence between the labels a _i and b _{w (i)} , while if the correspondence between the candidates a _i is correct, the labels a _i and b _{w (} a cost giving unit 114 that gives a small cost c to the correspondence of _i) and stores the cost c in a storage unit;
Based on the cost c changed by the cost giving means 114 , the matching processing means 111, the correction candidate selecting means 112, the interface means 113, and the repeating means 117 that repeats the process of the cost giving means 115 are included.

Further , the present invention (Claim 4) is provided in the cost giving means 115 .
When the operator determines the correspondence between the labels a _i and b _{w (i)} , if the correspondence is incorrect and the label b _j is selected as the correct correspondence destination of the label a _i ,
A large cost c is given to the correspondence between the labels a _i and b _{w (i)} , and a small cost c is given to the correspondence between the labels a _i and b _j , and the two costs c are stored in the storage means.

The present invention (Claim 5) is a label sequence matching error correction program for causing a computer to function as each means constituting the label sequence matching error correction apparatus according to claim 3 or 4.

The present invention (Claim 6) is a computer-readable storage medium storing the label series matching error correction program according to Claim 5.

  According to the present invention, when there is an error in the matching result, the error can be corrected in the order of the influence of the correction. Therefore, matching without error or few errors without correcting the error brute force. Matching results can be obtained with less effort.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  The present invention aims to quickly reach an error-free state by correcting errors in the order of the influence of correction, instead of correcting all errors brute-force when correcting errors in matching results. It was made by an idea.

  FIG. 3 is a flowchart showing an outline of the operation according to the embodiment of the present invention.

  First, automatic matching is performed and temporarily stored in the storage means (step 101).

  Next, the automatic matching result is analyzed and corrected to search for a correspondence that greatly changes the automatic matching result and greatly reduces the matching cost (step 102).

  A person visually determines whether the searched correspondence is correct (step 103). If the correspondence is incorrect, a correct correspondence destination is searched for and a high degree of coincidence (that is, low cost) is given to the correct correspondence. Alternatively, a high cost (that is, low coincidence) is given to the erroneous response (step 104).

  The process is repeated until it is determined that the error has disappeared, or until it is determined that the error has been reduced and the correct answer has been approached (step 105).

  According to this procedure, since it is not necessary to visually find where the error to be corrected is, the burden of human work is reduced. In addition, since the error having a large influence is corrected and matching is performed again, it is expected that the error is more correct than the other errors due to the ripple effect, the number of correction procedures is reduced, and the human work load is reduced.

  Further, if it is determined that the number of errors has decreased and the user has reached an acceptable range, correction may be stopped midway even if the answer is not correct.

In the above step 104, in order to evaluate the magnitude of the change in matching, the distance d (w ₁ , w ₂ ) between the matching results w ₁ and w ₂ is defined as, for example, Equation 4, and the magnitude of d is Can be evaluated. δ _ij is Kronecker's δ, which is 1 when i = j, and 0 otherwise.

本明細書に記載する方法では、２つのラベル列Ｓ_Ａ＝｛ａ_１，ａ_２，ａ_３，…，ａ_ｎ｝、Ｓ_Ｂ＝｛ｂ_１，ｂ_２，ｂ_３，…，ｂ_ｍ｝のマッチング結果ｗ：［１，…，ｎ］→［１，…，ｍ］を求める方法であって、
（１）コスト最小化法によりマッチング結果ｗと最小コストＣを求め、
（２）１∈［１，…，ｎ］について、ラベルａ_ｉとｂ_ｗ（ｉ）の対応に高いコストを与えたと仮定してコスト最小化法によって、ラベル列Ｓ_Ａ，Ｓ_Ｂのｗ_ｉと最小コストＣｉを求めた場合に、Ｃｉが小さく、かつ、マッチング結果距離ｄ（ｗ，ｗ_ｉ）が大きくなるような１個もしくは複数のｉを選び、
（３）上記の選んだｉについて、ラベルａ_ｉとｂ_ｗ（ｉ）の対応の正誤を目視により判定し、
（４）上記ラベルａ_ｉとｂ_ｗ（ｉ）の対応が誤りの場合には、ラベルａ_ｉとｂ_ｗ（ｉ）の対応に高いコストを与え、
（５）（４）で変更されたコストを加味して再度コスト最小化法によりマッチング結果ｗと最小コストＣを求め、上記（２）から（５）の手順を繰り返すマッチング方法について説明する。

In the methods described herein, two labels column _{S A = {a 1, a} 2, a 3, ..., a n}, S B = {b 1, b 2, b 3, ..., b m} A matching result w: [1,..., N] → [1,.
(1) Find the matching result w and the minimum cost C by the cost minimization method,
(2) 1∈ [1, ... , n] for, by a label _{a i} and _{b w} cost minimization method assuming gave corresponding to high cost of _(i), the label string _S A, the _{S B w i} And the minimum cost Ci, one or a plurality of i is selected such that Ci is small and the matching result distance d (w, w _i ) is large,
(3) For the above selected i, the correctness of the correspondence between the labels a _i and b _{w (i)} is determined visually,
(4) If the correspondence between the labels a _i and b _{w (i)} is incorrect, a high cost is given to the correspondence between the labels a _i and b _{w (i)} ,
(5) A matching method in which the matching result w and the minimum cost C are obtained again by the cost minimization method in consideration of the cost changed in (4), and the procedures (2) to (5) are repeated will be described.

In the present invention, in the above matching method, in step 103, the control of the correspondence between the labels a _i and b _{w (i)} is determined, and if the correspondence is incorrect, the correct correspondence b of a _{i A} matching method is described in which _j is selected, and in step 104, a high cost is given to the correspondence between labels a _i and b _{w (i)} and a low cost is given to the correspondence between labels a _i and b _j .

  In the following, a description will be given by taking as an example a situation in which a speech section of a video program is associated with a script line.

  FIG. 4 shows a system configuration according to an embodiment of the present invention.

  The system shown in FIG. 1 includes an information processing apparatus 101, a display apparatus 102, and an input apparatus 103.

  The operator corrects the matching result while interacting with the system. As a dialog interface, the display device 102 is used to present information to the operator, and the input device 103 is used to input the operation of the operator.

  FIG. 5 shows an example of a screen displayed on the display processing device according to the embodiment of the present invention. The screen shown in the figure is a so-called graphical user interface, and buttons and the like are virtually arranged.

  An area 201 in FIG. 5 represents a display area of the screen. An area 202 is an area for displaying an utterance section of a program. Each of the utterance sections 206 to 209 represents one utterance section, and displays a representative image of the video of the program while the utterance is being made. The representative image is created, for example, by extracting a video at the utterance start time as a still image. In the present embodiment, it is described that the utterance section is represented by the representative image, but the method of representing the utterance section is not limited to this, and other methods may be used. For example, the start time and end time of the utterance section may be displayed. The utterance sections 206 to 209 are buttons, and by operating the utterance sections 206 to 209, the video (including audio) of the utterance section is reproduced. The scroll bar 203 is for displaying a desired utterance section in the program. The area 204 is an area for displaying a script line.

  Each of the lines 210 to 215 represents a line of lines, and a sentence of the line is displayed. The scroll bar 205 is for displaying a desired line in the program. An arrow 216 represents the correspondence between the speech segment and the speech. For example, if the utterance section 206 and the dialogue 210 are connected by an arrow, it indicates that the utterance section 206 and the dialogue 210 correspond to each other. An area 217 is an area for outputting a message that the system presents to the operator. Buttons 218 and 219 are for the operator to respond to the system when the system inquires the operator of the correctness of the correspondence. The button 218 is for answering that the correspondence is correct, and the button 219 is for answering that the correspondence is incorrect. The button 220 is a button for ending the matching result correction process.

  Prior to the description of the matching example, preparation of input data will be described.

  FIG. 6 is a diagram showing an overall flow of work for associating a program and a script in one embodiment of the present invention.

In the figure, a video file 301 is a file obtained by digitizing a program to be associated and converting it into an MPEG format or the like. A script 303 is obtained by digitizing a script to be associated with a text file. The speech segment extraction processing 306 receives the video file 301 as an input, extracts a speech segment, and outputs a speech segment column 302. The label a _i (1 ≦ i ≦ n) is a label representing the i-th utterance section. n is the total number of utterance sections extracted in the program. For example, “Kenichi Minami, Akito Akutsu, Hiroshi Hamada, Yoshinobu Tonomura“ Video indexing using sound information and its application ”, Science theory D-II, vol. J81-D-II, no. 3, pp. 529-537, 1998 "can be used. The serif extraction process 307 takes the script 303 as an input and outputs a serif string 304. In other words, the phrase means one utterance section represented on the script, but is interpreted as, for example, one utterance surrounded by square brackets (“”) representing an utterance sentence. The method of dividing the lines is not limited to this, and may be divided according to other criteria. The label b _i (1 ≦ i ≦ m) is a label representing the i-th line. m is the total number of lines in the script. The matching process 310 receives the video file 301, the audio segment sequence 302, and the serif sequence 304 as input, and obtains and outputs a matching result 305 between the utterance segment sequence 302 and the serif sequence 304 while interacting with the operator. The matching result is represented by the symbol w. Only the utterance section string 302 and the serif string 304 are essential for performing the matching, but it is convenient if the operator can refer to the video when determining the correctness of the correspondence, so the video file 301 is also input. is doing.

  Prior to performing matching, it is necessary to determine a cost evaluation method and an automatic matching method.

Various methods can be used to construct the label and the label cost c (a _i , b _j ). For example, the label is constructed as follows. Assume that the duration of the utterance in the utterance section a _i is t _i , and the number of mora in the line b _j is r _j . References: Masatake Tanimura and Hiroshi Nakagawa, “Time Synchronization Method for Drama Video and Audio Tracks and Scenarios”, Information Processing Society of Japan, 118th Research Report on Intelligence and Complex Systems, SIG-ICS, no.188-4, pp .25-31 ", it is known that the duration of the utterance interval is approximately proportional to the number of mora in the speech. The proportionality constant (“duration” / “number of mora”) is α. And the cost between labels is defined like Formula 5.

ラベル間のコストの定義は、この方法に限らず、例えば、音声認識技術を利用して発話の音声信号とセリフとの類似度を用いてもよいし他の方法を用いてもよい。

The definition of the cost between labels is not limited to this method. For example, the similarity between the speech signal of speech and the speech may be used by using a speech recognition technique, or another method may be used.

Since the present invention has a feature of dynamically changing the cost value, the cost function needs to be modifiable. To achieve this, a hint list is used. The hint list is a list of pairs (speech interval index, serif index, cost). Initially, the hint list is empty. When evaluating the cost of labels a _i and b _j , search the hint list, and if there is a pair in the hint list whose utterance interval index is i and serif index is j, Let cost values be the costs of a _i and b _j . If it does not exist in the hint list, the cost is calculated by Equation 5. A cost function that allows you to overwrite these cost values,

と表す。

It expresses.

  The automatic matching method uses a general solution for the nonlinear minimization problem. For example, the annealing method (annealing method) can be used (literature: Richard O. Duda, Peter E. Hart and David G. Stock, supervised by Morio Onoe, pattern identification, New Technology Communications).

  Next, the detailed configuration of the information processing apparatus 101 in FIG. 4 will be described before explaining the error correction method for label sequence matching results of the present invention.

  FIG. 7 shows the configuration of a matching error correction apparatus according to an embodiment of the present invention.

  The apparatus shown in the figure corresponds to the information processing apparatus 101 shown in FIG.

  The apparatus 101 shown in the figure includes a matching processing unit 111, a correction candidate selection unit 112, an interface unit 113, an input determination unit 114, a cost assignment unit 115, a hint list 116, and an end determination unit 117.

  The matching processing unit 111 performs the matching processing 308 in FIG. 6 and stores the matching result in a storage unit such as a memory.

  The correction candidate selection unit 112 selects a correction candidate to be presented to the operator. The correction candidate is a candidate for correspondence that is expected to change the matching result more greatly by performing the correction, and to reduce the matching cost.

  The interface unit 113 exchanges information with the display device 102 and the input device 103.

  The input determination unit 114 determines which button has been pressed by the operator.

  The cost assigning unit 115 assigns a cost to the cost function in the input determining unit 114 based on the judgment from the operator (correspondence is incorrect and the correspondence is correct).

  The hint list 116 is a storage unit such as a memory, and is a list including a set of an utterance section, an index, a serif index, and a cost.

  The end determination unit 117 determines whether or not the processing for all the correction candidates has ended, or whether or not the operator has pressed the end button.

  FIG. 8 is a flowchart of matching error correction processing according to the embodiment of the present invention.

  Step 401) First, a matching result w between the speech section and the speech is obtained by automatic matching in the matching processing unit 111. The cost of correspondence between the i-th speech segment and the j-th line

とし、マッチングのコストＣを最小化するようなマッチング結果ｗを求め、このときの最小コストをＣ_ｍｉｎとする。最小化には焼きなまし法などを用いることができる。

And a matching result w that minimizes the matching cost C is obtained, and the minimum cost at this time is defined as C _min . An annealing method or the like can be used for minimization.

そして、得られたマッチング結果の状態をインタフェース部１１３を介して表示装置１０２に表示する。表示の方法は、領域２０２中に、発話区間ごとに代表画像を表示し、領域２０４の中にセリフ毎にセリフの文面を表示する。対応している発話区間ａ_ｉとセリフｂ_ｗ（ｉ）の間には、領域２１６に矢印を表示する。なお、発話区間ａ_ｉとセリフｂ_ｗ（ｉ）が領域２０２、２０４内にない時は、矢印を表示せずともよい。代表画像は、映像ファイル３０１から、発話区間ａ_ｉの発話開始時刻の画像を静止画として抽出し、これを代表画像とする。

The state of the obtained matching result is displayed on the display device 102 via the interface unit 113. In the display method, a representative image is displayed for each utterance section in the area 202, and a sentence text is displayed for each line in the area 204. An arrow is displayed in the area 216 between the corresponding utterance section a _i and the dialogue b _{w (i)} . When the speech section a _i and the speech b _{w (i)} are not in the areas 202 and 204, the arrow may not be displayed. As the representative image, an image at the utterance start time of the utterance section a _i is extracted from the video file 301 as a still image, and this is used as the representative image.

  Step 402) In the correction candidate selection unit 112, the correspondence of P correction candidates is selected. The correspondence that is a candidate for the correction is a correspondence that is expected to change the matching result more greatly and reduce the matching cost by correcting the correspondence. Details of the selection method will be described later.

  As a result of the processing, it is assumed that the index values of the P speech sections are stored in X [1] to X [P] of the array X. P need not be a constant, and may be different each time the process is executed. P may be 1.

  Step 403) Assign 1 to the variable p.

  Step 404) The index X [p] of the correction candidate of the p-th utterance interval is substituted into the variable i.

Step 405) The arrow connecting the speech segment a _i and the speech b _{w (i)} is highlighted on the display device 102. The highlighting method may be changed in color, blinked, or other methods. When the speech section a _i and the speech b _{w (i)} are not in the areas 202 and 204, they are scrolled so that they are displayed together. Thereafter, a corresponding correct / incorrect input request is displayed in the message output area 217. The input request is displayed, for example, “Is this correspondence correct?”.

  Step 406) The information processing apparatus 101 stands by until any button on the screen is pressed.

While the information processing apparatus 101 is on standby, the operator is prompted by the input request in step 405, and examines the utterance section a _i and the line b _{w (i)} pointed to by the highlighted arrow. If necessary, the representative display image is pressed, the video (including audio) of the utterance section is reproduced, the utterance content is heard, and the identity with the speech b _{w (i)} is determined. If necessary, the scroll bars 203 and 205 may be operated to check before and after the speech segment and before and after the speech. When the operator determines that this correspondence is correct, the operator presses the “correct” button 218, and when the operator determines that the error is correct, the operator presses the “false” button 219. The case where the “confirm all” button 220 is pressed will be described later.

  When the operator presses any button, the process proceeds to step 407.

  Step 407) The input determination unit 114 determines whether the pressed button is the “correct” button 218. If so, the process proceeds to step 411. Otherwise, the process proceeds to step 408.

  Step 408) It is determined whether or not the pressed button is the “wrong” button 219. If so, the process proceeds to Step 412. Otherwise, the process proceeds to step 409.

  Step 409) It is determined whether the pressed button is the “End” button 220. If so, the process proceeds to Step 410, otherwise returns to Step 406 and waits until another button is pressed. .

  Step 410) The current matching result w is output and the process is terminated.

Step 411) When the process shifts to this step, the operator has determined that the correspondence between the utterance section a _i and the speech b _{w (i)} is correct.

に有利なコストを付与する。そのために、ヒントリスト１１６に、組（ｉ，ｗ（ｉ），ε）を挿入する。εは、定数であり、小さな数である。例えば、ε＝０とする。その後、ステップ４１７に移行する。
ステップ４１２） 当該ステップに処理が移行した場合は、発話区間ａ_ｉとセリフｂ_ｗ（ｉ）の対応が誤りと操作者が判定したのであるから、

To give a favorable cost. For this purpose, a set (i, w (i), ε) is inserted into the hint list 116. ε is a constant and a small number. For example, ε = 0. Thereafter, the process proceeds to step 417.
Step 412) When the process moves to the step, the operator determines that the correspondence between the speech section a _i and the speech b _{w (i)} is incorrect.

に不利なコストを付与する。そのため、ヒントリスト１１６に、組（ｉ，ｗ（ｉ），Ω）を挿入する。Ωは、定数であり、充分大きな数である。例えば、その後、ステップ４１３に以降する。

Is disadvantageous. Therefore, the set (i, w (i), Ω) is inserted into the hint list 116. Ω is a constant and is a sufficiently large number. For example, the process proceeds to step 413 thereafter.

Step 413) The cost giving unit 115 requests the operator to select which of the correct lines corresponding to the utterance section a _i is prompted. For example, the message output area 217 displays “Please select the correct line corresponding to this utterance section”.

  Step 414) The information processing apparatus 101 waits until one of the lines is selected.

While waiting in this step, the operator is prompted by the input request in step 413 and examines the speech segment a _i pointed to by the highlighted arrow and the line of speech. If necessary, the representative image is pressed, the video (including audio) of the utterance section is reproduced, and the utterance content is listened to. The scroll bar 205 is operated to search for and select a corresponding line. For example, the selection operation is performed by pressing a corresponding line portion of the display area 204.

  When the operator presses any line, the process proceeds to step 415.

Step 415) The cost assigning unit 115 obtains the serif index j. The line selected by the operator is represented as b _j .

Step 416) Since the operator determines that the correspondence between the utterance sections a _i and the lines b _j is correct, the cost granting unit 115

に有利なコストを付与する。そのために、ヒントリスト１１６に、組(ｉ，ｊ，ε)を挿入する。その後、ステップ４１７へ移行する。

To give a favorable cost. For this purpose, the set (i, j, ε) is inserted into the hint list 116. Thereafter, the process proceeds to step 417.

  Step 417) The value p + 1 is substituted into the variable p.

  Step 418) Compare p and P, and if p ≦ P, the correction candidates have not been exhausted, and the process goes to Step 404. Otherwise, since the correction candidates are exhausted, the process proceeds to step 419.

  Step 419) In the matching processing unit 111 again, automatic matching is executed. The cost function uses Equation 6 as in the previous case, but since the contents of the hint list 116 are different from the previous time, a matching result w different from the previous matching is obtained. Thereafter, the process returns to step 402 to select a correction candidate again.

  Next, details of the processing in step 402 will be described.

  FIG. 9 is a flowchart of correction candidate selection processing according to an embodiment of the present invention.

  Step 501) The list L is emptied.

  Step 502) Assign 1 to the variable j.

  Step 503) It is checked whether or not there is a pair whose utterance section index is j in the hint list 116. If it exists, the utterance section j means that the operator has already determined whether it is correct or not, so there is no need to enter it as a correction candidate, so the following processing is skipped and the process proceeds to step 509. To do. If not, the process proceeds to step 504.

  Step 504) Insert the pair (j, w (i), Ω) into the hint list 116. In this way, a situation is assumed in which the correspondence between the utterance section j and the speech w (j) is regarded as an error.

Step 505) The matching processing unit 111 executes automatic matching. The automatic matching method is the same as in steps 401 and 419. Here, the matching result is w _j and the minimum cost is C _j .

Step 506) A correction candidate fitness H _j is obtained. H _j is an amount for evaluating whether the utterance section j is suitable as a correction candidate. In this embodiment,
H _j = C _j −k _d (w, wj)
Evaluate by Here, k is a predetermined constant. Value of about H _j cost matching is less reduced, matching the value of the result w and w _j and distances larger the H _j of the smaller. Note that the method of evaluating H _j is not limited to this, and other methods may be used.

Step 507) Insert the pair (j, H _j ) into the list L.

  Step 508) The pair (j, w (j), Ω) is deleted from the hint list 116, and the original state is restored.

  Step 509) The value j + 1 is substituted into the variable j.

  Step 510) When j is compared with the number n of utterance sections, if j ≦ n, the process returns to Step 503, and if not, the process proceeds to Step 511.

  Step 511) The list L is sorted in ascending order of the correction candidate fitness Hj. As a result, pairs with smaller values of Hj are arranged at the top of the list.

  Step 512) P pairs are extracted from the head of the list L, and the value j of the speech segment index is stored in the array X in order.

  The details of the processing in step 402 have been described above.

  In the above embodiment, the correction candidate is selected by full search. However, other search algorithms may be used as long as an equivalent result can be obtained.

  In the present embodiment, an example of giving a cost advantageous for correct handling in steps 411 and 416 has been shown, but instead, a cost disadvantageous for handling other than correct handling may be given.

  Also, in step 412, in order to reflect that the correspondence of a specific label is incorrect, a method of diversifying the cost of correspondence of the label has been shown, but instead, the label does not correspond. You may make it incorporate a restriction condition in automatic matching. For that purpose, Ω is set to infinity, or both label columns are divided before and after the label, the label columns before the label are matched, the label columns after the label are matched, and the result You may carry out by the method of couple | bonding.

The flowchart of FIG. 8 is an operation including Claim 1 (Claim 3 ) and Claim 2 (Claim 4 ), but only the processing related to Claim 1 (Claim 3 ) is executed. Is executed by omitting step 413 to step 416.

  As described above, by using the processing of the present invention, when there is an error in matching, the error can be corrected in the order of the influence of the correction. Therefore, matching without error without correcting the error brute force, Alternatively, a matching result with few errors can be obtained with less effort.

  The operations shown in FIGS. 8 and 9 can be constructed as a program, installed in a computer such as the information processing apparatus 101 and executed, or distributed via a network.

  Further, the constructed program can be stored in a portable storage medium such as a hard disk, a flexible disk, or a CD-ROM connected to the computer, and installed in the computer.

  The present invention is not limited to the above-described embodiment, and various modifications and applications can be made within the scope of the claims.

  The processing of the present invention is not limited to the matching of the speech section and the speech, but the matching of the sentence manuscript and the generated audio signal, the matching of the shot of the video program and the script cut, etc. It is applicable to a wide range.

It is a figure for demonstrating the principle of this invention. It is a principle block diagram of this invention. It is a flowchart which shows the outline | summary of operation | movement in one embodiment of this invention. 1 is a system configuration diagram according to an embodiment of the present invention. It is an example of the screen displayed on the display processing apparatus in one embodiment of this invention. It is a figure which shows the whole flow of the operation | work of matching of the program and a script in one embodiment of this invention. It is a block diagram of the matching error correction apparatus in one embodiment of this invention. It is a flowchart of the matching error correction process in one embodiment of the present invention. It is a flowchart of the correction candidate selection process in one embodiment of the present invention.

Explanation of symbols

101 Information Processing Device 102 Display Device 103 Input Device 111 Matching Processing Unit, Matching Processing Unit 112 Correction Candidate Selection Unit, Correction Candidate Selection Unit 113 Interface Unit, Interface Unit 114 Input Determination Unit 115 Cost Giving Unit, Cost Giving Unit 117 Repeating Unit, End determination unit 201 Display area 202 Area for displaying program utterance section 203 Scroll bar 204 Area for displaying script lines 205 Scroll bar 206 to 209 Speech section 210 to 215 Line segment 216 )
217 Fields 218 and 219 for outputting a message to be presented to the operator by the system Button 220 for the operator to respond to the system Button 220 for finishing the process of correcting the matching 301 Video file 302 Audio section string 303 Script 304 Serif string 305 Matching 306 Speech recognition processing 307 Line extraction processing 308 Matching processing

Claims (6)

Label string _{S A = {a 1, a} 2, a 3, ..., a n} in individual elements of the label string _{S B = {b 1, b} 2, b 3, ..., b m} either In a method for correcting a matching error of a label sequence, using an information processing apparatus that corrects a matching result w of a label sequence in which elements of
Using the cost c (a, b) taking a small value enough to label a and labels b are similar, the cost minimization method that minimizes the sum C of the label string S _A viewed from the cost c (a, b) by obtains the matching results w and the minimum cost C, a matching processing step of storing in the memorize means,
Giving corresponding to higher cost _{c i} labels _{a i} and labels _{b w} corresponding to the label _{a i (i)} in the matching result w, the sum C of the cost _{c i} is reflected in the cost c (a, b) when due to the cost minimization method to minimize the _i Lima etching results sought w _i and the minimum cost C _i, said minimum cost C _i is small and the distance d between the matching result w and w _i one large ing or a modified candidate selection step of selecting a plurality of candidate a _i,
Outputting the matching result w to the display means, highlighting the candidate a _i ,
Get the correctness determination result indicating whether the corresponding OPERA author or al candidate a _i and b _{w (i)} are correct or incorrect,
If the correspondence between the candidates a _i is incorrect, a large cost c is given to the correspondence between the labels a _i and b _{w (i)} , while if the correspondence between the candidates a _i is correct, the labels a _i and b _{w (} a cost giving process of giving a small cost c to the correspondence of _i) and storing the cost c in the storage means;
Based on the cost c changed in the cost granting process, repeating the process after the matching process again,
A method for correcting a matching error of a series of labels, characterized in that: In the cost granting process,
When the operator determines the correspondence between the labels a _i and b _{w (i)} , if the correspondence is incorrect and the label b _j is selected as the correct correspondence destination of the label a _i ,
Together give pre Symbol label a _i and b _w corresponding to a large cost c of _(i), giving a small cost c in correspondence of the label a _i and the label b _j, and stores the two said cost c in the storage means The method for correcting a matching error in a label sequence according to claim 1. Label string _{S A = {a 1, a} 2, a 3, ..., a n} in individual elements of the label string _{S B = {b 1, b} 2, b 3, ..., b m} either A label sequence matching error correction device that corrects a matching result w of a label sequence that associates elements of
Using the cost c (a, b) taking a small value enough to label a and labels b are similar, the cost minimization method that minimizes the sum C of the label string S _A viewed from the cost c (a, b) by obtains the matching results w and the minimum cost C, a matching processing means for storing in the memorize means,
A large cost c _i is given to the correspondence between the label a _i and the label b _{w (i)} corresponding to the label a _i in the matching result w, and the total C _i of the cost c (a, b) reflecting the cost c _i is obtained. if by the cost minimization method to minimize Lima etching results were obtained w _i and the minimum cost C _i, said minimum cost C _i is small and it increases the distance d between the matching result w and w _i one or a correction candidate selecting means for selecting a plurality of candidate a _i that,
And outputs to the display unit the matching result w, wherein the candidate a _i highlight acquires correctness determination result indicating whether the corresponding OPERA author or al candidate a _i and b _{w (i)} are correct or incorrect Interface means;
If the correspondence between the candidates a _i is incorrect, a large cost c is given to the correspondence between the labels a _i and b _{w (i)} , while if the correspondence between the candidates a _i is correct, the labels a _i and b _{w (} a cost giving means for giving a small cost c to the correspondence of _i) and storing the cost c in a storage means;
Based on the cost c changed by the cost giving means, a repeating means that repeats the matching processing means, the correction candidate selecting means, the interface means, and the cost giving means again,
An apparatus for correcting a matching error of a series of labels, characterized by comprising: The cost giving means is
When the operator determines the correspondence between the labels a _i and b _{w (i)} , if the correspondence is incorrect and the label b _j is selected as the correct correspondence destination of the label a _i ,
Together give pre Symbol label a _i and b _w corresponding to a large cost c of _(i), giving a small cost c in correspondence of the label a _i and the label b _j, and stores the two said cost c in the storage means The label sequence matching error correction device according to claim 1. 5. A label sequence matching error correction program for causing a computer to function as each means constituting the label sequence matching error correction device according to claim 3. A computer-readable storage medium storing the label series matching error correction program according to claim 5.

Images