JP3386102B2 - Video search method and apparatus - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an apparatus for retrieving an image, and more particularly, to a method and an apparatus for retrieving an image using a telop arrangement included in the image.

[0002]

2. Description of the Related Art There is a method of searching for a necessary portion from a large amount of videos based on the contents of the videos, but a method of using a telop included in the videos as a search key is drawing attention. This is because the telop is often composed of characters and symbols, has a clear meaning, has a typical appearance pattern in the arrangement of the telop, and has a characteristic that the contents of the video are well reflected.

In the prior art, an image in the telop area is extracted from the video, the characters included in the telop are recognized, and the video is searched by comparing the character information as a search key (eg, Mogi et al., "News"). Article Indexing Based on Character Recognition in Video ", IEICE Technical Research Report IE95-153, 1996, etc.). In this method, edge extraction is performed on the image by first-order differentiation,
The edge image is projected in the horizontal and vertical directions to cut out the rectangular area where the telop exists. Character recognition is performed by calculating Gabor feature amounts, which are outputs of eight types of Gabor filters, at 64 positions in each character region of the telop, and projecting a total of 512-dimensional feature amounts on the subspace.

[0004]

With the above-described conventional video search apparatus, it is impossible to search based on the position where the telop appears. Further, since the telop is interpreted by character recognition, a costly character recognition process is required. In addition, since the character recognition rate is not so high, the search efficiency was not very good. Further, since the target image for character recognition requires high quality, there is a drawback that the processing cost for extracting the image in the telop area with high quality becomes high.

An object of the present invention is to provide a video search device for searching video at a low processing cost based on the spatial layout of telops appearing in a video, using the spatial layout of telops as a search key.

[0006]

According to a first video search method of the present invention, a telop detection step of detecting the position of a telop from video data and position data indicating the spatial arrangement of a telop to be searched are input as search keys. There is a search key input step, a selection step for controlling display / non-display of video data based on the output of the telop detection step and the comparison result of the search key, and a display step for displaying the video data.

The second video retrieval method of the present invention is a record for recording, as index information, a telop detection step of detecting a telop position from video data, and a combination of the video data and position data indicating a spatial arrangement of the telop as index information. Step, a search key input step of inputting position data indicating the spatial arrangement of the telop to be searched as a search key, and a selection step of selecting video data from the recording unit based on the comparison result of the recorded index information and the search key And a display stage for displaying video data.

The first video search device of the present invention is, in the video search device, a telop detection unit for detecting a telop from video data, and a search for inputting positional information of the spatial arrangement of the telop to be searched as a search key. It has a key input unit, a selection unit for controlling display / non-display of video data based on the output of the telop detection unit and the comparison result of the search key, and a display unit for displaying the video data.

Further, the second video retrieval device of the present invention is
In the video search device, a telop detection unit that detects a telop from the video data, a recording unit that records a combination of the position data of the video data and the spatial layout of the telop as index information, and a search key for the spatial layout of the telop that you want to search. A search key input section for inputting, the selection section for selecting video data from the recording section based on the comparison result between the index information of the recording section and the search key, and the display section for displaying the video data.

The present invention also includes the search key input section having an input section for inputting a range of spatial arrangement of telops to be searched by drawing a figure using the input means.

It is a preferred embodiment that the selecting section compares the position data of the spatial arrangement of the telop with the search key, and selects the one existing or overlapping the former inside the latter.

Further, the display unit displays the frame image of the video image selected by the selection unit and the spatial arrangement position data of the telop in the frame image, and the display unit is selected by the selection unit. What displays an image corresponding to a frame is also included in the present invention.

[0013]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Next, embodiments of the present invention will be described with reference to the drawings. First, three types of methods for expressing the spatial arrangement of telops will be described with reference to the drawings.

FIG. 2 shows a first spatial layout expression method of telops. This method is expressed by the combination of the coordinates of the upper left corner of the circumscribed rectangle of the telop and the dimensions of width and height. In FIG. 2, considering a rectangle R circumscribing the telop, which is partitioned by the rectangle, a combination of coordinates (x, y) of the upper left apex of the rectangle R, width w, and height h, (x, y,
w, h). The advantage of this expression method is that one telop can be expressed by four numerical values. The disadvantage is that in the case of a telop having a complicated shape, an extra area other than the telop is included.

A second method for expressing the spatial arrangement of telops is shown in FIG. This method is a method of expressing a telop using a bitmap in which the block in which the telop exists is 1 and the other blocks are 0. In FIG. 3, the telop screen shown in FIG. 3 (a) is divided into M × N blocks to form the bitmap shown in FIG. 3 (b) consisting of M × N elements. Then, the bit corresponding to the block in which the telop exists is set to 1, and the other bits are set to 0. The advantage of this method is that the smaller the block, the more accurate the telop can be represented. The disadvantage is that smaller blocks require more memory for bitmaps.

FIG. 4 shows a third spatial layout expression method of telops. This method is a combination of the above two expression methods. That is, a small bitmap corresponding to the circumscribed rectangle is created, and the block containing the telop is set to 1
And the other values are set to 0.

In FIG. 4, the circumscribed rectangle (x, y, w, h) of the telop shown in FIG. 4A and the corresponding FIG.
The telop is represented by a small bitmap (4 × 2 blocks in this case) shown in (b). The advantage of this method is that the telop can be expressed just enough compared to the first method, and the amount of memory used is smaller than that of the second method.

In the following examples, unless otherwise specified,
Any one of the above three types of telop expression methods is used.

As a method of expressing the circumscribed rectangle, a method of using a combination of the coordinates of the upper left vertex and the coordinates of the lower right vertex can be considered, but it is substantially the same as the method described above.

Next, the arrangement of the telop as the search key will be described with reference to the drawings.

The appearance of the telop in the image has a certain tendency depending on the custom and the intention of the producer. That is, as shown in FIG. 5, the time is displayed in the upper left part A, the place and date are displayed in the upper right part B, the person's name and place are displayed in the central lower part C, and the title, description, dialogue, etc. are displayed in the lower part D. Interpreted lines tend to be displayed respectively. The present invention uses this tendency to search the video.

Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 shows a first example of the video retrieval apparatus of the present invention.
It is a block diagram showing an example of.

The input video data may be continuous frame images or may be H.264. It may be encoded by H.261 or MPEG. The telop detection unit 1 detects a telop from the video data and outputs the telop area data. A spatially existing range of a telop to be searched for is input to the input unit 3 as a search key. Selector 2
Compares the spatial arrangement of telops based on the telop area data with the search key. The switch is switched based on the comparison result to decide whether to send the video data to the display unit. The display unit 4 displays the input video data.

The telop detection unit 1 can be implemented by a method of using telop information given to a video in advance, or an automatic method of extracting a telop portion by processing the video by using image processing or image recognition technology. There are two possible detection methods.

As an automatic telop detection method, for example, a method using the property that the image area of the telop has a higher brightness than the background portion and an edge can be easily extracted (for example, "Mogi et al.," News Video " Indexing Based on Character Recognition in Text ", IEICE Technical Research Report IE95
-153, 1996), or MPEG, etc., among the video data encoded using the correlation between frames, the pixel encoded using the correlation between frames and without motion compensation is temporal. The existing telop detection means shown in a method of extracting a spatially concentrated portion (Japanese Patent Application No. 8-131898 “moving image telop extraction method and apparatus”) may be used.

A flow chart of the latter method is shown in FIG. In this method, encoded video data is input for each frame (step 61), and it is determined whether the frame is an encoded frame by using the correlation between the frames (step 62).
Targeting a frame coded using the correlation between frames, the coded macroblocks using the correlation between frames and without using motion compensation are counted within a constant counting section (step 63, 64), the telop portion is selected based on the result of comparison with the threshold value (step 65),
The process moves to the next counting section (step 66).

FIG. 7 shows a block diagram of a telop detection device based on this method. In the frame discrimination unit 71,
A coded frame is extracted from the coded video data using correlation between frames. The coefficient unit 72 determines the number of appearances of macroblocks coded without correlation and using motion correlation among the macroblocks of this frame for each macroblock position by a constant counting interval. Count within. The counting result becomes a matrix and is input to the selection unit 73. Based on the comparison result with the given threshold value, a matrix representing the telop area is output. In the connected area creation unit 74,
The connected areas of the telop area matrix are obtained, and the respective unconnected areas are output as telop areas.

FIG. 8 is a block diagram showing an embodiment of the search key input section of the first embodiment. Here, the input device 82 is assumed to be a pointing device such as a mouse, a digitizer, a joystick, or a touch panel. The input device 82 is used to draw a rectangle or an arbitrary closed curve figure. The combination of the coordinates (x, y) of the upper left vertex of the circumscribed rectangle of the drawn figure, the width w, and the height h is stored in the circumscribed rectangle field R of the index information register 81. Further, by writing a value in the bitmap field B in which the inside of the figure is 1 and the other is 0, the value of the index information register 81 becomes a search key.

Either the circumscribing rectangle or the bitmap may be omitted depending on the type of expression method of the spatial arrangement of the telop as the search key. Alternatively, the search key may be visually expressed by displaying the bitmap on the display device 83 such as a CRT.

9 and 10 show display examples of the search key input section. FIG. 9 shows a search key drawn by a rectangle. FIG. 10 shows a search key drawn by a closed curve figure. The drawing surfaces 91 and 101 can freely draw a rectangle or a closed curve figure with a pointing device. The hatched lines are the drawn figures. Two modes, rectangular and closed curve, are displayed in the two icons 9 at the bottom left.
2, 93; 102, 103.

FIG. 11 shows the selection unit 2 of the first embodiment of the present invention.
It is a block diagram showing. By using the telop area data, the spatial arrangement data of the telop is index information register 1
It is stored in the telop field T of 11. The telop field is further divided into a circumscribed rectangular field R and a bitmap field B, but either one may be omitted depending on the method of expressing the spatial arrangement of the telop. The search key created by the search key input unit 3 is stored in the search key register 113. The value of the index information register 111 and the value of the search key register 113 are compared by the comparison unit 112. The comparison result controls the switch as 0N / 0FF information.

In the comparison unit 112, (1) the case where the range K expressed by the search key includes the spatial arrangement D of the telop, and (2) the range K expressed by the search key and the spatial arrangement D of the telop D There are two types of comparison methods when and overlap. Each case is shown in FIG. In (1), the comparison result is true only when D is included in K, and is false otherwise. In (2), it becomes true when D and K at least overlap, and becomes false only when they do not overlap at all.

First, in the case of (1), the determination is made as follows. When the spatial arrangement of telops is expressed as a circumscribed rectangle, D is expressed as (Xd, Yd, Wd, Hd) and K is expressed as (Xk, Yk, Wk, Hk). Then, the conditional expression Xk ≦ Xd∧ (Xk + Wk) ≧ (Xd + Wd) ∧ (Yk
It is possible to determine the case where D is included in K by ≦ Yd) ∧ (Yk + Hk) ≧ (Yd + Hd).

When the spatial arrangement of telops is expressed as a bit map, it is possible to determine the case where D is included in K by the bit operation D∧K = D.

Similarly, considering the case of (2), in the case of the circumscribed rectangle expression, the conditional expression Xk ≦ (Xd + Wd) ∧Xd ≦ (Xk + Wk) ∧Yk ≦
It can be determined by (Yd + Hd) ∧Yd ≦ (Yk + Hk).

In the case of the bit map expression, it can be determined by the bit operation (the logical product of D and K is 0) D∧K ≠ φ.

FIG. 13 shows an example of display on the display unit according to the first embodiment of the present invention. Here, an image matching the search key is displayed.

FIG. 14 is a block diagram showing a second embodiment of the video search device of the present invention. In this embodiment, a recording unit 22 for recording the combination of the spatial arrangement of video data and telop as index information is added to the first embodiment, and the selection unit 23 can select the index information. .

Telop detection section 21 and search key input section 24
Can use the same device as in the first embodiment.

FIG. 15 is a detailed block diagram of the recording unit 22 according to the second embodiment of the present invention. The input video data is stored in the frame field F of the index information register 151. At this time, it is possible to store the frame image and the frame number. Using the input telop area data, the spatial arrangement data of the telop is stored in the telop field T of the index information register 151.

The telop field is further divided into a circumscribed rectangular field R and a bitmap field B, but either one may be omitted depending on the method of expressing the spatial arrangement of the telop. The value of the index information register 151 is recorded on the information recording medium 152. Further, as the information recording medium 152, a semiconductor memory, a magnetic disk device, a CD, a magnetic tape or the like may be used.

FIG. 16 is a detailed block diagram of the selection unit 23 according to the second embodiment of the present invention. The index information is sequentially read from the information recording medium 161 (identical to the information recording medium 152) recorded by the recording unit 22 and stored in the index information register 162. The search key created by the search key input unit 24 is stored in the search key register 164. The value of the telop field T of the index information register 162 and the value of the search key register 164 are compared by the comparison unit 163. As the comparison method in the comparison unit 163, the above method can be used. Comparison result is 0N / 0FF
Controls the switch as information, and the index information register 162
It is determined whether the value of is output to the display unit 25.

FIG. 17 shows an example of display by the display unit 25 according to the second embodiment of the present invention. In the display unit 25, the selection unit 2
Using the value of the frame field of the index information selected by 3, the list of frame images of the video data is CR
Display on T etc. The frame image may be appropriately reduced and displayed. Further, the value of the telop field of the search information may be used to highlight the area in which the telop exists.
As a method of highlighting, there are a method of drawing a circumscribing rectangle with a prominent color such as red, and a method of increasing the luminance value of the telop area portion and decreasing the luminance value of other portions.

Further, on the display unit 25, the frame images displayed in a list may be designated by an input means such as a mouse to reproduce and display the video from the corresponding frame on the CRT.

The present invention is not limited to the above embodiments, but can be variously modified and implemented within the scope of the invention. For example, in the search key input sections 3 and 24,
It is also possible to search for an image by registering some search keys corresponding to a typical telop arrangement and selecting from them.

[0046]

The present invention is an apparatus for retrieving video, including a telop detection section for detecting a telop from video data, a search key input section for inputting the spatial arrangement of the telop to be searched as a search key, and a telop. The output of the detector,
Since there is a selection unit that controls the display / non-display of video data based on the comparison result of search keys and a display unit that displays video data, the spatial arrangement of telops is used as the search key, and the spatial arrangement of telops that appear in the image is displayed. Based on the above, there is an effect that an image can be selected and displayed.

Since there is no need for character recognition processing or extraction of high-quality telop area images, it is possible to select and display an image at low processing cost.

Further, the second invention is, in an apparatus for retrieving video, a telop detection section for detecting a telop from video data, a recording section for recording the spatial arrangement of the video data and the telop as index information, and a telop to be searched. Has a search key input section for inputting the spatial arrangement of the as a search key, a selection section for selecting video data from the recording section based on the comparison result between the index information of the recording section and the search key, and a display section for displaying the video data. Thus, there is an effect that the spatial arrangement of the telops is used as a search key and the images can be searched and displayed as a list based on the spatial arrangement of the telops appearing in the image. Further, since there is no need for character recognition processing or extraction of high-quality telop area images, there is an effect that videos can be searched and displayed in a list at low processing cost.

The present invention also has an input unit for inputting the range of the spatial arrangement of the telop to be searched by drawing a figure using the input means. Therefore, there is an effect that the spatial arrangement of the telop can be used as the search key.

Further, the present invention compares the spatial arrangement of telops with the search key, and the former has a selection unit for selecting the one existing inside the latter or overlapping, so that the space of the telop appearing in the image is displayed. The effect is that the video can be selected based on the physical arrangement.

Further, the present invention has an effect that the searched video can be displayed by having the display unit for displaying the frame image of the video selected by the selection unit and the video corresponding thereto.

[Brief description of drawings]

FIG. 1 is a block diagram of a first embodiment of a video search device of the present invention.

FIG. 2 is a diagram showing a method of expressing a spatial arrangement of a first telop.

3A and 3B are diagrams showing a method of expressing a second spatial arrangement of telops, where FIG. 3A is a telop on the screen and FIG. 3B is a bitmap representation.

4A and 4B are diagrams showing a third method of expressing the spatial arrangement of telops, where FIG. 4A is a telop on the screen, and FIG. 4B is an circumscribing rectangle and a bitmap.

FIG. 5 is a diagram showing a spatial arrangement of a typical telop.

FIG. 6 is a flowchart of a video telop detection method.

FIG. 7 is a block diagram of a video telop detection device.

FIG. 8 is a block diagram of a search key input unit 3 according to the first embodiment.

FIG. 9 is a display example of a search key input unit.

FIG. 10 is a display example of a search key input unit.

FIG. 11 is a block diagram of a selection unit 2 according to the first embodiment of this invention.

FIG. 12 is a diagram illustrating a method of comparing a spatial arrangement of telops and a search key.

FIG. 13 is an example of display by the display unit 4 according to the first embodiment of the present invention.

FIG. 14 is a block diagram of a second embodiment of the video search device of the present invention.

FIG. 15 is a block of a recording unit 22 according to the second embodiment of the present invention.

FIG. 16 is a block diagram of a selection unit 23 according to the second embodiment of this invention.

FIG. 17 is an example of display by the display unit 25 according to the second embodiment of the present invention.

[Explanation of symbols]

1 Telop detector 2 Selector 3 Search key input section 4 Display 21 Telop detector 22 Recording section 23 Selector 24 Search key input section 25 Display 71 Frame discriminator 72 Coefficient part 73 Selector 74 Connected area creation unit 81 Index information register 82 Input device 83 Display 91 Drawing surface 92 icons 93 icon 101 drawing surface 102 icon 103 icon 111 Index information register 112 Comparison section 113 Search key register 151 Index information register 152 Information recording medium 161 Information recording medium 162 Index information register 163 comparison section 164 Search Key Register

─────────────────────────────────────────────────── ─── Continuation of the front page (72) Inventor Yayoi Arakura 3-19-2 Nishishinjuku, Shinjuku-ku, Tokyo Nihon Telegraph and Telephone Corporation (56) Reference JP-A-9-322173 (JP, A) JP-A-7-192003 (JP, A) JP-A-8-317301 (JP, A) (58) Fields investigated (Int.Cl. ⁷ , DB name) H04N 5/76-5/956 H04N 7/24 -7/68

Claims (8)

(57) [Claims] 1. A video search method, a telop detection step of detecting a position of a telop from video data, a search key input step of inputting position data indicating a spatial arrangement of a search target telop as a search key, and a telop detection step. An image search method comprising: a selection step of controlling display / non-display of video data based on a result of comparison between the output of the above and a search key, and a display step of displaying the video data. 2. A video search method, a telop detection step of detecting a telop position from video data, a recording step of recording a combination of video data and position data indicating a spatial arrangement of the telop as index information, and a telop to be searched. The search key input step of inputting position data indicating the spatial arrangement of the search key as a search key, the selection step of selecting video data from the recording section based on the comparison result of the recorded index information and the search key, and the video data A video search method, comprising: a display step of displaying. 3. A video search apparatus, a telop detection unit for detecting a position of a telop from video data, a search key input unit for inputting position data indicating a spatial arrangement of a telop to be searched as a search key, and a telop detection unit. A video search device, comprising: a selection unit for controlling display / non-display of video data based on a result of comparison between the output and the search key, and a display unit for displaying the video data. 4. A video search device, a telop detection unit for detecting a telop position from video data, a recording unit for recording a combination of video data and position data indicating a spatial arrangement of telop as index information, and a telop to be searched. The search key input section that inputs position data indicating the spatial arrangement of the image as a search key, the selection section that selects the video data from the recording section based on the comparison result of the index information of the recording section and the search key, and the video data display And a display unit for displaying the image. 5. The search key input unit according to claim 3, wherein the search key input unit can input position data indicating a range of spatial arrangement of search target telops by drawing a figure by using an input unit. Video search device. Wherein said selection unit, by comparing the spatial arrangement of the telop and the search key, whether the former is present in the interior of the latter, or any of claims 3 to 5 to select one overlapping 1 The video search device according to item. 7. The display unit displays the frame image of the video selected by the selection unit and the spatial arrangement of telops in the frame image, according to any one of claims 3 to 6. Video search device described. 8. The video search device according to claim 3, wherein the display unit displays a video corresponding to the frame selected by the selection unit.