JP2003345809A - Database constructing system, passage retrieving device, database constructing method, and program therefor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To register a news picture associated with a character string such as a telop character string appearing in the news picture in a database together with the character string as an index when the database from which the news picture is retrieved is generated. <P>SOLUTION: A speech writing device 2 writes a news speech in a character string. A character recognizing device 1 detects a character appearance section wherein the character string appears in the news picture and recognizes the character string. The retrieving device 3 finds similarity between words in a speech writing result corresponding to the character appearance section detected by the writing device 2 and retrieves the passage similar to the character string recognized by the character recognizing device 1 from the speech writing result by using the similarity. A registering device 4 relates and registers the character recognition result of the character recognizing device 1 and a news picture corresponding to the passage retrieved by the retrieving device 3 in a database 5. <P>COPYRIGHT: (C)2004,JPO

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a database construction technique for constructing a database for retrieving news images, and more particularly to a character string such as a telop character string appearing in a news image and the character string in the news image. The present invention relates to a technique of registering a part to be registered in a database in association with the part to be performed.

[0002]

2. Description of the Related Art In recent years, a large amount of news video information has begun to be generated due to multi-channel broadcasting and the like, and there has been a demand for viewers to watch only news programs of interest. As a conventional technique for such a request, for example, there is known a system in which an index is added to a news video and registered in a database so that a user can view only an interesting news video based on the index. ing. By the way, when registering news videos by adding an index to the database,
Conventionally, a keyword is extracted from a character string such as a telop character string in a news video, and the similarity between this keyword and each sentence in the news speech transcription result is determined using the similarity between words in a thesaurus. A news video corresponding to a sentence having a degree higher than the threshold is registered in the database using the character string as an index.
In the past, news transcripts were manually transcribed,
We are building a thesaurus.

[0003]

In the conventional system described above, based on one keyword extracted from a telop character string or the like, it is associated with a sentence in a transcription result using a thesaurus, and a key sentence is obtained. , There is a risk that unrelated sentences may be extracted.
For this reason, there is a risk that news videos unrelated to the index may be registered in the database.

[0004] Further, since it is determined on a sentence basis whether or not to register the corresponding news video in the database, there is a problem that the context of the registered news video may be difficult to understand.

[0005] Further, since a thesaurus constructed manually and requiring a long time to construct is used, a time difference from a broadcast news video signal is likely to occur. At different times, the thesaurus relationships are changing. For example, Koizumi is now Prime Minister, but at times he was not. The similarity between Koizumi and the Prime Minister in the thesaurus when it was not so is low, but the similarity is high now that Koizumi is Prime Minister. In other words, at different times,
The thesaurus relationships are changing. Using a thesaurus that does not respond to change will adversely affect the system.

Further, since news speech transcribed manually is used, a great deal of labor is required.

Therefore, an object of the present invention is to eliminate the danger that unrelated news videos are registered in a database, and not to use the similarity between words in a thesaurus but to use the similarity between words in a search device. By using the degree of similarity automatically calculated by the degree calculation unit, it is possible to avoid the problem of the time difference with the news voice, and instead of using the news voice transcribed manually, by the news voice transcription device The aim is to reduce the effort by using the transcribed news audio.

Japanese Unexamined Patent Application Publication No. 2002-14973 discloses a technique for recognizing character information such as a telop character string in a video and calculating a similarity between two images based on the similarity of the recognized character information. Although described, it is not a technique for calculating the similarity between a character string such as a telop character string and a transcription result of news audio.

[0009]

In order to achieve the above object, a database construction system according to the present invention comprises: a voice transcriptor for writing a news voice corresponding to a news video into a character string; A character recognition device for recognizing the character string while detecting an appearing character appearance section, and a similarity between words in a speech transcription result corresponding to the character appearance section detected by the character recognition device are obtained. A search device that searches for a passage similar to the character string recognized by the character recognition device from the speech transcription result using a degree, and a recognition result of the character recognition device and a passage searched by the search device. A registration device for associating with a corresponding news video and registering the news video in a database.

[0010] More specifically, in the database construction system according to the present invention, the search device may be arranged so that the search device obtains a similarity between words in a speech transcription result corresponding to a character appearance section detected by a character recognition device. A similarity calculation unit; and a passage search unit configured to search the speech transcription result for a passage similar to the character string recognized by the character recognition device using the similarity.

[0011]

The above-mentioned configuration is made by paying attention to the point that the telop character string and the CG flip character string appearing in the news video summarize the contents of the news program. In the above-described configuration, a passage search is performed on a transcription result of a news voice using all words in a telop character string or a CG flip character string recognized by the character recognition device. On the other hand, in the conventional system, a search result is obtained for each transcript of a news sound in units of one sentence using one word. For this reason, there is a danger that an unrelated sentence is extracted due to the influence of the thesaurus of one word, and an unrelated news video is registered in the database. In the above configuration, not only one word but all words in the character string are used.
It removes the danger of extracting unrelated sentences due to the influence of the thesaurus of one word, thereby eliminating the risk that unrelated news videos will be registered in the database. it can. Further, by using the search result in passage units, it is possible to solve the problem that the context of news videos is difficult to understand.

Further, in the above-described configuration, the inter-word similarity calculation unit of the search device uses the inter-word inter-word transcript based on the character transcription section of the character appearance section detected by the character string appearance section detection unit of the character recognition device. Learn similarity. That is,
Rather than using the similarity between words in a manually constructed thesaurus as in the prior art, the similarity between words is calculated directly from the data to be analyzed, so that the problem of time difference can be avoided.

[0013]

Embodiments of the present invention will now be described in detail with reference to the drawings.

FIG. 1 is a block diagram of an embodiment of a database construction system according to the present invention.
, A transcription device 2, a search device 3, a registration device 4, and a database 5.

The character recognizing device 1 uses a telop character string or CG appearing in a news video based on a news video signal.
It has a function of character recognition of a character string such as a flip character string, a function of detecting a character string appearance section in which a character string appears, and the like.

A character recognition device 1 having such a function
Are a character string appearance section detection unit 11 and a character string region extraction unit 1
2, a binarization processing unit 13 and an OCR unit 14.

The character string appearance section detecting section 11 detects a character string appearance section in which a character string such as a telop character string or a CG flip character string appears in a news video based on the news video signal, and detects the character string appearance section. Function to output the character string appearing section information to the speech transcriptor 2, a function to detect a still section in which the character string is stationary, and to output the still section information indicating the still section to the character string area extraction unit 12. Having.

The character string region extracting section 12 has a function of extracting a character region in a frame based on the news video signal indicated by the static section information from the news video signal.

The binarization processing unit 13 includes a character string area extraction unit 1
It has a function of binarizing the character area extracted in step 2 with other areas.

The OCR unit 14 has a function of performing character recognition based on the processing result (binarization result) of the binarization processing unit 13.

The voice transcription device 2 has a function of transcribing a news voice signal separated from a news video signal into a character string, and a voice transcription result in a character appearance section of a voice transcription result in the retrieval device 3. It has a function of outputting to the inter-word similarity calculator 31 and the like.

The search device 3 obtains the similarity between words in the speech transcription result in the character appearance section, and uses this similarity to extract the character recognition device 1 from the speech transcription result.
Has a function of searching for a passage similar to the character string recognized by.

The search device 3 having such a function includes an inter-word similarity calculation unit 31 and a passage search unit 32.

The inter-word similarity calculation unit 31 has a function of calculating the similarity between words in the speech transcription result in the character appearance section.

The passage retrieval unit 32 uses the similarity between words obtained by the inter-word similarity calculation unit 31 to extract passages similar to the character string recognized by the character recognition device 1 from the speech transcription result. Has the function of searching for

The registration device 4 has a function of registering the news video corresponding to the passage searched by the passage search unit 32 in the database 5 in association with the character string recognized by the character recognition device 1.

[0027]

Next, the operation of this embodiment will be described.

First, the operation of the character string appearance section detection section 11 in the character recognition device 1 will be described.

The appearance patterns of character strings in news videos can be roughly classified into the following three types.

-Appears gradually from the head and stops for a certain period of time. -Appears while moving and stops for a certain period of time. -Appear simultaneously from the beginning to the end and stop for a certain period of time.

Even if a telop character string or a CG flip character string that is appearing is detected and character recognition is performed, it does not make sense as a character string. Focusing on the feature that the character string or the CG flip character string is stationary for a certain period of time, character recognition is performed in a section where the character string is stationary (stationary section). The character string appearance section detection unit 11 is used to detect a still section. As shown in FIG. 2, a stationary section is a section in which a character string is stationary in a constant state.
In FIG. 2, a character appearance frame is a frame where a character starts to appear, a character disappearance frame is a frame before the character disappears, and a character appearance section is a section from the time when the character starts appearing to a point before the character disappears. In this embodiment, the character string appearance section detection unit 11 is also used to detect a character appearance section.

When detecting a character appearance section and a stationary section, the character string appearance section detection section 11 uses a plurality of types of rectangular areas defined on the frames shown in FIGS.

FIG. 3A shows a rectangular area obtained by dividing a frame into eight equal parts. FIG. 3B shows six rectangular regions of the same shape which straddle two rectangular regions in FIG. 3A, and FIG. 3C shows the same rectangular region which straddles four rectangular regions in FIG. FIG. 3 shows three rectangular regions of the shape,
FIG. 3D shows four rectangular regions of the same shape that straddle the two rectangular regions in FIG.

The character string appearance section detecting section 11 firstly makes the current frame a candidate for a character appearance frame based on the distance between the luminance histogram of the current (interest) frame in the news video signal and the luminance histogram of the previous frame. It is determined whether or not (S41 in FIG. 4). The processing in step S41 will be described in detail below. Note that the previous frame, the current frame, and the subsequent frame are defined in FIG.
That is, the previous frame is a frame one frame before the current frame, and the subsequent frame is a frame one frame after the current frame.

In step S41, the character string appearance section detection unit 11 first uses the first rectangular area of the rectangular areas shown in FIGS. Perform processing. Now, for example, assuming that the upper left rectangular area 301 in FIG. 3A is the first rectangular area in use order, the character string appearance section detection unit 11 determines the luminance histogram of the upper left rectangular area 301 of the current frame. And the upper left rectangular area 301 of the previous frame corresponding to the rectangular area 301
Distance Diff between the luminance histogram of (H _1, H ₂₎ obtained using the following equation (1) (the maximum value of the difference between the frequency of luminance histogram of the power and the previous frame of the luminance histogram of the current frame).

[0036]

In equation (1), H ₁ (i) represents the frequency of luminance i in the previous frame, H ₂ (i) represents the frequency of luminance i in the current frame, and abs represents the absolute value.

Thereafter, the character string appearance section detecting section 11 determines whether or not the distance Diff (H ₁ , H ₂ ) exceeds a predetermined threshold. If the threshold value is exceeded, it is determined that the current frame is a candidate for a character appearance frame (S
42 is YES), the process of step S44 is performed. On the other hand, if the threshold value is not exceeded, the order of use is the next rectangular area (for example, the upper right rectangular area 302 in FIG. 3A).
To perform the same processing as the processing described above. If the distance Diff (H ₁ , H ₂ ) does not exceed the threshold even when the last rectangular area (for example, the lowermost rectangular area 303 in FIG. 3D) is used, the current frame is It is determined that it is not a character appearance frame candidate (S42: NO), the current frame is advanced by one (S43), and the process of step S41 is performed again. That is, the process of step S41 is performed again with the subsequent frame as a new current frame.

In step S44, it is determined whether or not the current frame is a character appearance frame based on the edge position coincidence between the current frame and the subsequent frame. The edge position coincidence indicates the degree to which the position and direction of the edge coincide. The processing in step S44 will be described in detail as follows.

In step S44, the character string appearance section detecting section 11 first focuses on the first rectangular area 301 in the current frame t in the order of use.
Each pixel existing within _{(x s, y s) ~} (x e, y e) edge component E _t with respect to the direction n of the _{(x s, y s, n} ) ~E t (x e, y e, n) Is calculated by the calculation shown in the following equation (2). The pixel (x _s , y _s ) has the coordinate value
A pixel existing at (x _s , y _s ) represents the upper left pixel of the rectangular area 301. The pixel (x _e , y _e ) is a pixel existing at the coordinate value (x _e , y _e ), and represents the lower right pixel of the rectangular area 301.
Further, x is, has a relationship of _{x s ≦ x ≦ x e,} y has a relationship of y _s ≦ y ≦ y _e. Further, the character string appearance section detection unit 11 determines that the use order in the subsequent frame t + 1 is the first rectangular area 3.
The edge component of each pixel existing in 01 is also calculated using equation (2).

[0041]

In equation (2), Rt (x, y), Gt (x, y), Bt
(x, y) represents the red, green, and yellow shading levels of the pixel (x, y). n represents four directions (0, 45, 90, and 135 degrees) in the pixel (x, y) as shown in FIG. In other words, the string appearance section detection unit 11, each pixel _{(x s, y s) ~} (x e, y e) for each performs a calculation shown in equation (2) as n = 1, 2, 3, 4 , Each pixel (x _s , y _s ) to (x _e , y _e )
Edge components E _t (x _s , y _s , 1) to E _t (x
_{e, y e, 1),} E t (x s, y s, 2) ~E t (x e, y e, 2), E t (x s, y s, 3) ~E t
(x _e , y _e , 3) and E _t (x _s , y _s , 4) to E _t (x _e , y _e , 4) are calculated.

Thereafter, the character string appearance section detecting section 11 performs processing.
Each pixel (x_s, y_s) ~
(x_e, y_e) Apply the following equations (3) and (4) to each
And each pixel (x_s, y_s) ~ (X_e, y_e) In each direction (n = 1,2,3,4)
Edge_t(x_s, y_s, n) ~ Edge_t(x _e, y_e, n). One
In other words, the edge component E of the pixel (x, y)_t(x, y, n) is the threshold th₁Than
Edge if big_t(x, y, n) = 1, threshold th₁Be less than
Edge_t(x, y, n) = 0. For example, Edge_t(x, y, 3) = 1
If there is, an edge exists in the direction n = 3 of the pixel (x, y).
It can be said that. Further, the character string appearance section detection unit 11
Equations (3) and (4) are also applied to each pixel of frame t + 1
And perform the same processing.

[0044]

Next, the character string appearance section detecting section 11 uses the following equation (5) to calculate each pixel (x _s , y _s ) to (x _e ,) in the rectangular area 301 to be processed. y _e ), it is checked whether an edge exists in each direction n (n = 1, 2, 3, 4) in both the current frame t and the subsequent frame t + 1. For example, the current frame t and the subsequent frame t +
In both 1, pixel edge exists in the direction n = 4 (x, y) for _{Ecor t (x, y, 4} ) becomes 1, is not the case the pixel (x, y)
_{Ecor t (x, y, 4} ) for is zero.

[0046]

Thereafter, the rectangular area 30 to be processed is
Each pixel in _{1 (x s, y s)} ~ (x e, y e) by performing the calculation shown in equation (6) with respect to each pixel _{(x s, y s) ~} (x e, y _e ) for each of them, the degree to which the direction of the edge of the current frame t and the direction of the edge of the subsequent frame t + 1 coincide with each other Eco _t (x _s , y _s ) to Eco r
Calculate _t (x _e , y _e ). For example, the Eco for a pixel (x, y) whose edge direction matches in two directions of n = 1 and n = 4
r _t (x, y) is 2.

[0048]

Thereafter, the rectangular area 30 to be processed is
Each pixel in _{1 (x s, y s)} ~ (x e, y e) the edge position coincidence Ecor in the square area 301 of the rear and the current frame frame by summing the calculated result of the equation (6) for ( nex
Find t). Then, this edge position coincidence Ecor (next)
Is compared with a predetermined threshold value, and if the edge position coincidence Ecor (next) exceeds the threshold value, it is determined that the current frame is a character appearance frame (S45: YE
S), the process of step S46 is performed. On the other hand, if the edge position coincidence Ecor (next) does not exceed the threshold, the same processing as the above-described processing is performed using the next rectangular area 302 in the use order. Last rectangular area 3
If the edge position coincidence Ecor (next) does not exceed the threshold value even if 03 is used, it is determined that the current frame is not a character appearance frame (S45: NO), the current frame is advanced by one (S43), and again The process of step S41 is performed.

In step S46, the edge position coincidence Ecor (previous) in a certain rectangular area corresponding to the current frame and the previous frame, and the edge position coincidence in the certain rectangular area corresponding to the current frame and the subsequent frame are determined. Ec
or (next), based on the edge position coincidence ratio,
It is determined whether or not the current frame is the first frame of the still section. The processing in step S46 will be described in detail below.

In step S46, the character string appearance section detection unit 11 calculates the edge position coincidence ratio by performing the calculation shown in the following equation (7) on the rectangular area 301 whose processing order is the first. I do.

[0052]

As shown in FIG. 7, a telop character string
When the CG flip character string begins to appear, the edge position coincidence ratio is greater than 1, and the value decreases as the stationary state is approached. It takes a value near 1 in the stationary section, and sharply decreases in the character lost frame. Therefore, the character string appearance section detection unit 11 checks whether or not the edge position coincidence ratio is within a predetermined error range with respect to 1 to determine whether or not the current frame is the first frame of the still section. A determination is made (S46). If it is determined that the current frame is the first frame of the stationary section (S47: YES), the process of step S50 is performed after the current frame is advanced by one (S49). On the other hand, when the edge position coincidence ratio is not within the error range, the same processing as the above-described processing is performed for the rectangular area 302 whose use order is the second. If the edge position coincidence ratio does not fall within the error range even when the last rectangular area 303 used in the use order is used, it is determined that the current frame is not the first frame of the still section (S47 is N).
O) After the current frame is advanced by one (S48), the process of step S46 is performed again.

In step S50, it is determined based on the edge position coincidence ratio shown in equation (7) whether or not the previous frame is a character lost frame in a still section. Step S
The process 50 is described in detail as follows.

In step S50, the character string appearance section detection unit 11 performs the calculation shown in the equation (7) on the rectangular area 301 whose processing order is the first, thereby processing the current frame and the previous frame. An edge whose order is the ratio between the edge position coincidence Ecor (previous) in the first rectangular area and the processing order of the current frame and the subsequent frame is the ratio of the edge position coincidence Ecor (next) in the first rectangular area. Calculate the position coincidence ratio.

As shown in FIG. 7, in the character lost frame, the edge position coincidence ratio sharply decreases. Therefore,
The character string appearance section detection unit 11 sets the edge position coincidence ratio smaller than 1 by a predetermined value or more with respect to the rectangular area whose edge position coincidence ratio is determined to be within the error range with respect to 1 in step S46. It is determined whether or not the previous frame is a character lost frame by checking whether or not the frame has disappeared (S50). If it is determined that the frame is a character lost frame (S51: YES), the process of step S52 is performed. On the other hand, if the edge position coincidence ratio is within the error range, it is determined that the previous frame is not a character lost frame (S51: NO), and after the current frame is advanced by one (S49), step S50 is performed. Return to the processing of.

In step S 52, the character appearance section information indicating the character appearance section is output to the voice transcription device 2, and the static section information indicating the static section is output to the character string area extraction unit 12. As the character appearance section information and the still section information, for example, frame numbers such as the m-th frame to the n-th frame can be used. Also, the frame number can be converted into time information based on the number of frames per unit time, and this time information can be used as character appearance section information.

The character string appearance section detecting section 11 repeats the above processing until there is no news video signal (YES in S53).
Until it becomes). If it is detected in step S53 that the news video signal has ended, the character string appearance section detection unit 11 notifies the audio transcript device 2 and the passage search unit 32 of the fact.

Next, the operation of the character string area extraction unit 12 will be described. The character string region extraction unit 12 first removes regions other than characters by background removal, and then extracts character string regions based on local linear density. The method of removing the background will be described below. In a plurality of frames in a section where a character is stationary, the character region has the following characteristics in terms of time.

(1) The character region has stable edge components. (2) Hue, luminance, and saturation of the character area hardly change.

Therefore, background removal using a plurality of frames is performed based on these features. First, the character string region extraction unit 12 creates a background-removed image based on the change in the degree of coincidence of the edge position (FIG. 8, S81). The processing in step S81 will be described in detail as follows.

In step S81, the character string region extraction unit 12 determines whether each pixel (x _sf , y _sf ) to (x _ef , y) exists in a certain frame t in a still section (N frames). _ef ) for each of the pixels
_{(x sf, y sf) ~} (x ef, y ef) edge component E _t for each direction n (n = 1,2,3,4) of the _{(x sf, y sf, n} ) ~E t (x ef , y _ef , n). Note that the pixel (x _sf , y _sf ) represents the upper left pixel of the frame,
Pixel (x _ef , y _ef ) represents the lower right pixel of the frame. Also,
The still section can be specified by the still section information passed from the character string appearance section detection unit 11.

[0063]

In equation (8), Rt (x, y), Gt (x, y), Bt (x,
y) represents the red, green, and yellow shading levels of the pixel (x, y), respectively. Incidentally, x is, it has a relationship of _{x sf ≦ x ≦ x ef,} y has a relationship of y _sf ≦ y ≦ y _ef. In addition, n is shown in FIG.
As shown in (4), four directions (0 degree, 45 degrees) in the pixel (x, y)
Degrees, 90 degrees, 135 degrees).

Thereafter, the character string appearance section detection unit 11 uses the equations (9) and (10) to calculate each pixel (x _sf , y _sf ) to (x _ef , y
_ef ) Edge vector Edge _t (x _sf , y _sf , n) to Edge _t (x _ef ,
y _ef , n). _{E t (x, y, n} ) if is greater than the threshold th _{2,
Edge t (x, y, n} ) = 1, if the threshold value th ₂ or _{less, Edge t (x, y,} n) =
It becomes 0. For example, a pixel (x, y) where Edge _t (x, y, 2) = 1 is
It can be said that an edge exists in the direction n = 2.

[0066]

Further, the character string region extraction unit 12 performs an operation (AND) shown in the following equation (11) on each pixel (x _sf , y _sf ) to (x _ef , y _ef ) in each frame of the still section. Processing)
For all frames in the stationary section, the direction n (n = 1or2
A pixel (x, y) having an edge at or3or4) is obtained. For example, if the pixel (x, y) has an edge in the direction n = 3 in all the frames in the stationary section, the calculation result of Expression (11) becomes Edge (x, y, 3) = 1. If at least one frame has no edge in the direction n = 3, Edge (x, y, 3) =
It becomes 0.

[0068]

Thereafter, the character string region extraction unit 12
By performing the operation shown in the following expression (12) on (x _sf , y _sf ) to (x _ef , y _ef ), the edge position coincidence Edge (x _sf , y _sf ) which is the sum of the elements of the edge vector is obtained. Calculate Edge (x _ef , y _ef ).

[0070]

Further, the character string area extraction unit 12 calculates the following equation (1)
3) and (14), each pixel (x _{science fiction}, y_{science fiction}) ~ (X_ef, y_ef)
The value of Ecor (x_{science fiction}, y_{science fiction}) ~ Ecor (x_ef, y_ef) Is determined. That is,
Edge position coincidence Edge (x, y) is the threshold th_ThreeIf it is larger than
Pixel (x, y) is assumed to be the edge of the character area,
The value Ecor (x, y) is set to 1.

[0072]

When the processing in step S81 is completed, the character string area extraction unit 12 performs the processing in step S82. In step S82, a background-removed image is generated based on a change in luminance in the RGB space. The processing in step S82 will be described in detail as follows.

In step S82, the character string region extraction unit 12 performs the following processing on the pixel for which Ecor (x, y) = 1 in step S81 (the pixel extracted as an edge portion). .

First, a combination of two consecutive frames t and t + 1 in a stationary section (N frames) (a combination of the first frame and the second frame to a (N−1) th frame and a Nth frame). In each case (combination with the Nth frame), each pixel to be processed ( _xss , _yss ) to
(x _es , y _es ) is calculated for each pixel (x _ss , y _ss ) in the frame t by performing the operation shown in the following equation (15)
~ (X _es, y _es) and the RGB values of the corresponding pixel (x _ss, y _ss) in the frame _{t + 1 ~ (x es,} y es) the difference (distance) between the RGB values of C
_t (x _ss , y _ss ) to C _t (x _es , y _es ) are obtained.

[0076]

Thereafter, by performing the calculation represented by the following equation (16), the luminance change amount of each of the processing target pixels (x _ss , y _ss ) to (x _es , y _es ) is calculated.

[0078]

[0079] Then, equation (17), (18) using in each processing target image _{(x ss, y ss) ~} (x es, y e s) the respective values Colo
Determine r (x _ss , y _ss ) to Color (x _es , y _es ). That is, if the luminance variation is the threshold value th ₄ below, the pixel (x, y) determines that hits the edge portion of the character region, the value Color (x, y) is referred to as 1. Greater than the threshold value th ₄ contrast, value Color a (x, y) 0
And

[0080]

Thereafter, the character string area extraction unit 12 calculates the expression (1
Only the pixel (x, y) satisfying the conditions in both (3) and (17) is extracted (S83). As a result, the background in which the color is changed or moving is removed, and a background-removed image in which the edge portion of the character is extracted is generated.

Thereafter, the character string area extracting section 12 performs a character string area extracting process based on the local line density (S84). The processing in step S84 will be described in detail below.

By the background removal processing in steps S81 to S83, character regions with small changes in luminance and stable edges are extracted. These regions are represented by figures (for example, white box images). A region other than a telop having an area may be included. In order to remove these and extract only the character area, the feature that a character is a set of line segments is used. In the image from which the background has been removed, the line density within a certain area centered on each pixel is calculated. According to this method, the line density is higher in the character area than in the area such as the figure. Formula (19) shows a formalization of this processing.

[0084]

In equation (19), VLD (i, j), HLD (i, j)
Is a process for obtaining a vertical and horizontal change point, respectively, and is defined by the following equations (20) and (21), respectively. Also, w (i, j | x, y) is a weight for the position of the pixel (i, j) centered on (x, y) and is defined by the following equation (22). D (i, j | x, y) in the equation (22) is a coordinate (i, j) and a coordinate (x, y)
And is defined by the following equation (23).

[0086]

The right side of the equation (20) is expressed by F (i, j) and F (i,
j + 1), and the right side of equation (21) is F
It represents the exclusive OR of (i, j) and F (i + 1, j). F (i, j) represents the presence / absence of an extraction point at the coordinates (i, j) of the background-removed image (Yes; 1, No; 0).

As a result, the value of the line density at the pixel (x, y) is obtained. Lastly, if the linear density of each pixel is higher than the average linear density of the surrounding constant area and equal to or larger than the threshold value, the pixel is set as a pixel of a character area, and a connected area of pixels is extracted as a character area.

When the character string area extraction unit 12 extracts a character area, the binarization processing unit 13 operates. Binarization processing unit 13
Performs a binarization process on the character region extracted by the character string region extraction unit 12 using a floating ternarization process. First, a moving average image is obtained by obtaining an average grayscale value in a rectangular mask area centered on each pixel, and a floating average is obtained.
A binary image is obtained by a ternarization process. Two types of rectangular mask regions, a horizontally long (x direction) and a vertically long (y direction), are used, and a binary image is generated for each. Finally, a binary image is obtained by performing an AND process on the two types of generated binary images. Formulas (24) to (27) show formulas of the floating ternary processing.

[0090]

2X and 2Y represent the lengths of the sides of the rectangular mask area, and I (x, y) represents the luminance value of the pixel at the coordinates (x, y). k is the image B of the character area with higher brightness than the background
This is a constant used for determining a margin when generating an image D (x, y) of (x, y) and a character area having lower luminance than the background. Equation (24) represents the average value of the luminance in the rectangular mask area centered on the pixel (x, y). In equation (25), the corresponding pixel (x, y) between the original image and the moving average image
Is obtained. Therefore, a difference image can be obtained from this equation.

As defined by the equations (26) and (27), if the gray value of the pixel (x, y) in the difference image is larger than kσ (σ is the standard deviation of the image I), the luminance becomes higher than the background. Is high in the character area image B (x, y) = 1, and the pixel (x,
If the gray value of y) is smaller than -k [sigma], the character region image D (x, y) = 1 having a lower luminance than the background. By generating an AND image of the character region image B having a higher luminance than the background and the character region image D having a lower luminance than the background, it is possible to perform the floating ternary processing on the character string region image. Image B and Image D
When performing the AND process, the sign of the image D is inverted before the AND process.

When the binarized character area image is generated by the binarization processing section 13, the OCR section 14 operates. OCR
The unit 14 performs character recognition using the character area image binarized by the binarization processing unit 13, and passes the character recognition result to the passage search unit 32. The OCR unit 14 includes a binarization processing unit 13
Performs the above-described processing every time a binarized character area image is generated. The passage search unit 32 stores the character recognition result every time it is passed from the OCR unit 14. As the OCR unit 14, a commercially available OCR can be used. The OCR used is Win Reader Pro Ver5.0, and the recognition method is as follows. Cut out characters and extract edge features of character lines. Divide into 7 x 7 blocks and take a weighted direction histogram in four directions for each block. The histogram is smoothed, and the features are compressed from 196 dimensions to 64 dimensions by canonical discriminant analysis. In recognition, a linear discriminant function for calculating a distance from a standard pattern is used.

Next, the operation of the voice transcription device 2 will be described. The audio transcription device 2 transcribes the news audio signal separated from the news video signal into a character string,
The speech transcription result and time information (information indicating how many seconds after the head of the news speech signal the speech transcription result corresponds to the news speech signal) are stored in association with each other. Transcription of news speech is performed by large vocabulary continuous speech recognition using a language model and an acoustic model.
The language model used was 45 months for the Mainichi Newspaper CD-ROM version (91
(January-September 1994), 20K vocabulary
Back-off bigram. Wit for back-off smoothing
Using ten-Bell estimation, cut-off for bigram
Is 1. The acoustic model is a male unspecified speaker HMM, and is a cross-word triphone model that also considers phoneme context dependencies between words. For learning, we used 21782 utterances of 137 male speakers in the acoustic corpus of newspaper articles read by the ASJ. The acoustic features include 39-dimensional feature parameters (12-dimensional M
FCC power and respective Δ, ΔΔ coefficients) were used.

Further, each time the character appearance section information is passed from the character string appearance section detection section 11, the voice transcription device 2 stores it. When the end of the news video signal is notified from the character string appearance section detection unit 11, the section indicated by the stored character appearance section information in the stored voice transcription results. Is passed to the inter-word similarity calculation unit 31. If the character appearance section information passed from the character string appearance section detection unit 11 is based on a frame number, the frame number is converted into time information, and the corresponding speech transcription result is converted using the time information. It is specified and passed to the inter-word similarity calculation unit 31.

Next, the operation of the search device 3 will be described. The search device 3 includes the inter-word similarity calculation unit 31 and the passage search unit 32 as described above. The inter-word similarity calculation unit 31 learns the inter-word similarity based on the voice transcription result in the character appearance section among the voice transcription results of the voice transcription device 2. Instead of using the similarity between words in a thesaurus constructed manually as in the related art, the similarity between words is calculated directly from the data to be analyzed, so that the problem of time difference can be avoided.

When the inter-word similarity calculation unit 31 receives the speech transcription result corresponding to each character appearance section detected by the character string appearance section detection unit 11 from the speech transcription device 2, the speech transcription unit 2 Find similarity between existing words. This processing will be described in detail below.

The distance (similarity) between the word w _i and the word w _j is calculated on the Euclidean space of the TF, IDF, and mutual information of each of the words w _i and w _j with respect to the speech transcription result P _m . Can be expressed as the following equation (28). P _m
Is a transcription result of a news speech signal in the m-th character appearance section detected by the character string appearance section detection unit 11, and is input from the speech transcription apparatus 2. TF is the frequency that the word w _i appears in P _m , IDF
(w _i ) is a value related to the number of character appearance sections in which the word w _I appears, and is defined by the following equation (29). Mutual information, by knowing the word w _i, and that the amount information obtained for P _m, represented by the following formula (30). This information amount is obtained by calculating the word w from the information amount i (P _m ) that P _m originally had.
It is defined as the difference from the information amount i (P _m | w _i ) that P _m still has after knowing _i . Mutual information by the equation (30) becomes smaller if the independent P _m of the occurrence probability P (P _m) and the occurrence probability of a word w _i P (w _i) is a large value if the dependent .

[0099]

When the end of the news video signal is notified from the character string appearance section detecting section 11, the passage search section 32 searches the stored character recognition results (recognition results of each telop character string and the like) by a search query. Qk (k is a subscript indicating the number of the character recognition result), and the following processing is performed. First, analysis sections (for example, one utterance is defined as one analysis section) are sequentially taken from the head of the transcription result of the news speech, and the analysis section is shifted while shifting the analysis section Pl (1 is the number of the analysis section). And a similarity (X _k , X _l ) between the search query Qk. The similarity (X _k , X _l ) is obtained by the following equation (31).

[0101]

Next, the passage search section 32 detects an analysis section having a high degree of similarity as a passage candidate section. Thereafter, a section in which the passage candidate sections are continuous is defined as a passage as a search result, and the passage and the OCR unit 14 are used.
Is output to the registration device 4. The passage search unit 32 performs the above-described processing for all stored search questions.

The registration device 4 registers the character string (search question) recognized by the OCR unit 14 and the news video signal corresponding to the passage searched by the passage search unit 32 in the database 5 in association with each other. For example, the news video signal is registered in the database 5 using the search question as an index. Here, the news video signal corresponding to the passage can be obtained, for example, as follows. A transcription result stored in the audio transcription device 2 in association with the time information is searched to find a portion having the same content as the passage. Thereafter, time information stored in association with the searched portion is obtained, and the time information is converted into a frame number. Then, a news video signal corresponding to the passage is obtained based on the frame number.

FIG. 9 is a block diagram showing an example of the hardware configuration of the database construction system. The computer 91 receives a news video signal and a news audio signal, a recording medium 92, and a database 93 (FIG. 1).
Database 5).
The recording medium 92 is a disk, a semiconductor memory, or another recording medium, and stores a program for causing the computer 91 to function as a database construction system. This program is read by the computer 91, and by controlling its operation,
1, a character recognition device 1, a voice transcription device 2, a search device 3, and a registration device 4 shown in FIG.

[0105]

According to the present invention, a passage search is performed for a transcription of a news voice using all words in a telop or CG flip character string recognized by a character recognition device. Therefore, there is an effect of reducing the risk of extracting an unrelated sentence due to the influence of the thesaurus of one word. Therefore, it is possible to reduce the risk that news videos unrelated to the database are registered. Further, since the search result is set in passage units, an effect that the context of the result can be easily understood is obtained. Therefore, the news video can be registered in the database in such a manner that the context is easy to understand.

The inter-word similarity calculation unit of the search device learns the inter-word similarity based on the transcription of the speech signal in the time section detected by the character string appearance section detection unit in the character recognition device. is there. That is, the present invention calculates the inter-word similarity directly from the data to be analyzed, instead of using the inter-word similarity in a manually constructed thesaurus as in the conventional technique, and thus avoids the time difference problem. effective.

According to the present invention, the news voice transcribed by the voice transcriptor is used instead of the news voice transcribed manually, so that the labor can be reduced as compared with the related art.

[Brief description of the drawings]

FIG. 1 is a block diagram of an embodiment of the present invention.

FIG. 2 is a diagram illustrating a character appearance section and a still section.

FIG. 3 is a diagram for explaining a rectangular area used by a character string appearance section detection unit 11;

FIG. 4 is a flowchart illustrating a processing example of a character string appearance section detection unit 11;

FIG. 5 is a diagram for explaining a previous frame, a current frame, and a subsequent frame.

FIG. 6 is a diagram for explaining a direction n with respect to a pixel (x, y).

FIG. 7 is a diagram illustrating a change in an edge position coincidence ratio.

FIG. 8 is a flowchart showing a processing example of a character string area extraction unit 12;

FIG. 9 is a block diagram illustrating a hardware configuration when a database construction system is realized by a computer.

[Explanation of symbols]

1 .... Character recognition device 11 ... character string appearance section detection unit 12 ... Character string area extraction unit 13 binarization processing unit 14 OCR section 2. Voice transcription device 3. Search device 31: Inter-word similarity calculator 32 ... passage search section 4: Registration device 5 ... Database

Continued on the front page (51) Int.Cl. ⁷ Identification code FI Theme coat II (Reference) H04N 7/03 7/035 (72) Inventor Yasuo Ariki 1-5 Yokotani, Seta-Oe-cho, Otsu-shi, Shiga Prefecture Ryukoku University F Terms (reference) 5B075 ND12 NK10 NK13 NK21 NK32 UU40 5C052 AA01 AC08 CC20 DD04 5C063 CA23 DA03

Claims (11)

[Claims] An audio transcriptor for writing a news voice corresponding to a news video into a character string, a character recognition device for detecting a character appearance section where the character string appears in the news video and recognizing the character string. The similarity between words in the speech transcription result corresponding to the character appearance section detected by the character recognition device is obtained, and the character recognition device recognizes the word from the speech transcription result using the similarity. A search device that searches for a passage similar to the character string; and a registration device that associates a recognition result of the character recognition device with a news video corresponding to the passage searched by the search device and registers the news image in a database. A database construction system characterized by: 2. The database construction system according to claim 1, wherein said character string is a telop character string or a CG flip character string. 3. The database construction system according to claim 1, wherein the search device determines a similarity between words in a speech transcription result corresponding to a character appearance section detected by the character recognition device. A passage similarity calculator that searches for a passage similar to the character string recognized by the character recognition device from the speech transcription result using the similarity calculated by the inter-word similarity calculator. And a database construction system. 4. The database construction system according to claim 3, wherein the inter-word similarity calculator calculates a word appearance frequency in a character appearance section, a number of character appearance sections in which the word appears, a word and a character appearance section. A database construction system having a configuration for calculating a degree of similarity between words based on a mutual information amount and a transcription result corresponding to (i). 5. The database construction system according to claim 4, wherein said character recognition device has a configuration for recognizing said character string in a stationary section of the character string. 6. A news voice corresponding to the news video corresponding to a character appearance section detected by a character recognition device that recognizes the character string while detecting a character appearance section where a character string appears in the news video. The similarity between words in the transcription result of the voice transcription device transcribed to the column is determined, and the similarity is used to determine a passage similar to the character string recognized by the character recognition device from the voice transcription result. A passage search device, comprising: means for searching for a passage. 7. The passage retrieval apparatus according to claim 6, wherein the character string is a telop character string or a CG flip character string. 8. A voice transcription step of writing a news voice corresponding to the news video into a character string, a character recognition step of detecting a character appearance section where the character string appears in the news video and recognizing the character string. The similarity between words in the speech transcription result corresponding to the character appearance section detected in the character recognition step is obtained, and the character recognition step recognizes the word from the speech transcription result using the similarity. A search step of searching for a passage similar to the character string; and a registration step of associating a recognition result of the character recognition step with a news video corresponding to the passage searched in the search step and registering the news image in a database. Characteristic database construction method. 9. The database construction method according to claim 8, wherein the character string is a telop character string or a CG flip character string. 10. A program for causing a computer to function as a database construction system, comprising: a sound transcriptor for writing a news voice corresponding to a news video into a character string; and a character string appearing in the news video. A character recognition device that detects a character appearance section and recognizes the character string, obtains a similarity between words in a speech transcription result corresponding to the character appearance section detected by the character recognition device, and calculates the similarity. A search device that searches for a passage similar to the character string recognized by the character recognition device from the speech transcription result using the search result; news corresponding to the recognition result of the character recognition device and the passage searched by the search device. A program for functioning as a registration device that registers images in the database in association with video . 11. The program according to claim 10, wherein the character string is a telop character string or a CG flip character string.