JPH03282779A - Animation picture retrieving device - Google Patents

Abstract

PURPOSE:To retrieve a desired picture without previously adding a key word by using arbitrary picture information forming an animation picture stored in an animation picture storing means as retrieval instructing information. CONSTITUTION:Retrieval instructing information is inputted by an input means 1 in order to retrieve desired picture information from an animation picture storing means 4 storing an animation picture comprising time series picture information. The collation between the retrieval instructing information by the input means 1 and the animation picture stored in the animation picture storing means 4 is carried out by a collating means 3. As for the retrieval instructing information inputted by the input means 1, characters, numeric characters, graphics and picture itself are used. Thus, the animation picture can be retrieved by the arbitrary retrieval instructing information without previously adding a key word to the animation picture.

Description

[Detailed Description of the Invention] [Objective of the Invention] (Industrial Application Field) The present invention provides a method for searching for a predetermined image from a medium storing moving images. The present invention relates to a moving image search device that performs a search using instruction information.

(Prior Art) In conventional video image search devices that search for desired image information from video images recorded on storage media such as video tapes and CDs (compacts, discs), similar to many databases, the It is necessary to add keywords in advance. Enter this keyword when searching. Then, the presence or absence of the input keyword is checked, and if the keyword is present, the image is displayed. An example of the storage format of the keyword and the corresponding frame number is shown in FIG. This is the videotape roll number 1001° and frame number 1002. Keyword 1003. Recording port 100
Consists of 4. Now, when a user instructs us to search for "giant" as a keyword, keyword 1 in Figure 1θ
It is checked whether the same keyword "giant" exists in the column 003, and if there is the same keyword, the search is considered successful and, for example, that frame is output. In the example of FIG. 10, this corresponds to the first frame of the first volume. If the user instructs playback from the output frame number, the user can view the match or its performance news as image information with the keyword "Giants". However, in the keyword-added video image search device, it is not possible to search using keywords that have not been added in advance.

For example, in the example shown in Figure 10, if you want to search by the name of the stadium where the game was played and enter "Tokyo Dome," the search will fail because the keyword does not include the stadium name. In order to solve the problems of such a keyword addition type video image search device, a method has been proposed in which a still image such as a map is searched using a sketch image. (Akio Okazaki et al. Human Interface J 198
9. Vol, 4N & RP, 181-P, 188). This search using sketch images is effective when the positional relationship of target objects is clear, such as on a map. However, there was a problem in that it was not possible to search for randomly recorded video images such as TV programs such as baseball games and pop songs, or videotapes of children's sports days and school performances. .

(Problems to be Solved by the Invention) As described above, in the conventional moving image search device, it is necessary to add information such as keywords in advance in order to perform a search. In addition, it was possible to search for objects such as maps, where the positional relationship of objects is fixed, by using sketches without adding keywords in advance, but there are quite a few objects that can be searched using this method. The problem was that it was limited.

The present invention is intended to solve the above-mentioned problems,
An object of the present invention is to provide a moving image search device that performs a search based on arbitrary search instruction information when searching for desired image information from a moving image.

[Structure of the Invention] (Means for Solving the Problems) In order to achieve the above-mentioned object, the moving image retrieval device of the present invention provides: (4) a moving image search device for storing moving images consisting of time-series image information; The apparatus includes: an image storage means, an input means for inputting search instruction information, and a collation means for comparing the search instruction information inputted by the input means with the moving image stored in the moving image storage means.

The input means inputs arbitrary image information constituting a moving image stored in the moving image storage means as search instruction information.

Note that the image information refers to characters and numbers that constitute the moving image stored in the moving image storage means.

The figure 9 is the image itself, etc.

(Operation) According to the moving image search device of the present invention configured as described above,
In order to search for desired image information from a moving image storage means that stores moving images consisting of time-series image information, human input of search instruction information is performed using an input means. The search instruction information input by the input means is compared with the moving image stored in the moving image storage means by a matching means. Here, the search instruction information inputted by the input means includes characters and numbers constituting a moving image stored in the moving image storage means.

These include figures and images themselves. In other words, it is possible to search for moving images using arbitrary search instruction information without adding keywords to moving images in advance.

(Example) Hereinafter, an example of the present invention will be described with reference to the drawings.

FIG. 1 is a block diagram of a moving image search device according to an embodiment of the present invention.

A moving image storage section 4 consisting of a medium for storing time-series image information such as a videotape or CD; a display output section 2 consisting of a monitor etc. that displays the results of playing back the moving image storage section 4; and a moving image storage section 4. An input section consisting of an image input means such as a scanner for directly inputting an image to be searched for, or a pointing device such as a mouse for indirectly indicating a part of the image displayed on the display output section 2. 1, and a key image storage section 6 for holding key images input or instructed from the input section 1. The pattern matching section 5 calculates the degree of correlation (similarity) between images using a composite similarity method, etc., and the image processing section 7 calculates the degree of correlation (similarity) between images using a composite similarity method, etc., and the image processing section 7 performs spatial transformation such as affine transformation, edge extraction, etc. The information management unit 3 is used to perform preprocessing necessary to calculate the degree of correlation between images, and the information management unit 3 manages the exchange of this information and executes the actual search. That is, the moving image storage means is the moving image storage section 4, and the input means is the input section 2. The key image storage section 6 is the matching means for the pattern matching section 51 and the image processing section 7. This is the information management section 3. Hereinafter, according to a flowchart for explaining the operation of the information management section 3 shown in FIG. The operation of the image retrieval device will be described. When the information management unit 3 receives an instruction to start a search from the input unit 1, it sends the key image stored in the key image storage unit 6 to the image processing unit 7, and first calculates brightness, calculates color tone, and extracts edges ( 7) Perform the following (step 5201): Calculate the brightness of the key image. Now, let B be the brightness of the baseball player's face, which is the search key. The color tone calculation is to find the color that has the largest area in the key image. For example, if the face of a human (baseball player) is the key, the skin tone (close to brown) will be the color tone. This color tone is designated as C. Edge extraction is obtained by binarizing the key image, and for example, the outline of a human face, eyes, nose, etc. as shown in FIG. 3 are extracted as lines. Next, the moving image stored in the moving image storage section 4 is called out frame by frame. For this purpose, the initial value of the variable i is set to 0 (step 5202), and 1 is issued to sequentially call from 1 frame (step 8203).
. Note that a frame is a still image that is the minimum unit of an image, and an image for one second is composed of, for example, 30 frames. Each frame is slightly different, and these frames come together to form an image, and the images come together to form a moving image. Then, it is determined whether or not the first frame exists (step 5204), and if there is no frame, the recording medium is converted (8), the search is terminated, and other processing is performed. Now, since it is iml and the i-th frame exists, proceed to the next step. Here, the image processing unit 7 performs color tone calculation for the i-th frame (step 8205).

The pattern matching unit 5 determines that the color tone C of the key image is
Check whether it is included in the frame (step 820
B) If it is included, then the image processing unit 7 performs normalization using the brightness B of the key image in order to match the brightness with the key image (step 5207).

Note that if the i-th frame does not include tone C of the key image, 30 frames are skipped in order to search from an image 1 second later (step 5211). Thereafter, edges are extracted from the portion including tone C (step 8208). The pattern matching unit 5 calculates the degree of correlation between the key image and the edge image of the i-th frame.

At this time, the image processing unit 7 appropriately performs affine transformation to facilitate calculation of the degree of correlation.

For example, the result of edge extraction from the skin color portion of the i-th frame is shown in FIG. The center part is the pitcher's arm, and the lower left part is the butter's face and bat. Here, it is determined whether the key image has a high correlation with the edge image (step 8209).

Even if the center edge image is affine transformed, the correlation with the key image in Figure 3 is mostly O, but the lower left edge has a high correlation due to rotation, so this frame The number is stored, for example, as in FIG. 5 (step 5210). Note that if the degree of correlation between the key image and the edge image is not high, 30 frames are skipped as in step 8206 (step 5211). For example, 30 frames are skipped from the image 1 second after step 8211 to perform a search (step 8212).

The above processing is performed up to the first frame, that is, the last frame, and when the search is completed, the information management section 3 displays it on the display output section 32 while referring to the frame number in FIG. 5 from the moving image storage section 4. At this time, for example, if you display from the 108th frame, there is a risk that you will miss the frame containing the key image of interest, so for example, 0.5
Control is performed to display, for example, 1.5 seconds (45 frames) from the 93rd frame, which is 15 frames back, which corresponds to seconds. By controlling the display in this way,
You can avoid missing frames containing raw images. In the above embodiments, the degree of correlation is determined using edge images, but it is also possible to determine the degree of correlation using monochrome images or the like. Note that when a monochrome image is used, color tone calculation is not necessary.

Second Embodiment In the first embodiment, even if the same face
The edge image changes significantly. For this reason, there are many cases that cannot be absorbed by affine transformation.

The second embodiment is included in moving images in order to solve such problems. For example, numbers such as jersey numbers and line figures such as rectangles can be recognized as characters and symbols.

FIG. 6 shows a block diagram of a moving image search device according to an embodiment of the present invention.

The difference from FIG. 11 is that a pattern dictionary 9 for recognizing characters, numbers, and line figures, and a key (11) and pattern storage section 8 for storing the results of recognizing key images using the pattern dictionary 9 have been added. The point is that

As described above, the operation of the information management section 3 shown in FIG. 7 will be explained using the above-mentioned configuration as an example of a case where a jersey number as shown in FIG. 8 is input from the input section 1 to find a baseball player. The operation of the moving image retrieval device of the present invention will be described according to a flowchart. The information management section 3 sends the key image to the pattern matching section 5. The key image storage section 6 stores the key image shown in FIG. Then, the pattern matching unit 5 performs recognition using the pattern dictionary 9 (by the composite similarity method, etc.), and stores the result in the key pattern storage unit 8 (step 8701).

In addition, when using the composite similarity method, the pattern dictionary 9
A standard vector is stored in , and the similarity is evaluated by calculating the composite similarity method with the vector of the key image. The key pattern storage unit 8 contains characters (rlJ rnJ
raJ rgJ raJrsJ rhJ ri
J rmJ rag) and the positional relationship therebetween. This is shown in FIG. As shown in FIG. 9, character number 9011 character (12) 902 and position 903 are stored. In addition, position 903
is positional information on coordinates assumed for the entire key image (not shown).

Next, as in Figure 2, the variable i that changes the frame number
settings and check the frame (steps 8702~
5704). When it is determined in step 5704 that there is an i-th frame, a variable i that counts the character number stored in the key pattern storage unit 8 and a variable k that counts the number of characters that match the key image in one frame are set. Both are initialized (step 8705). Starting with the first character of the characters stored in the key pattern storage unit 8, 1 is added to the characters in order to determine whether the characters are included in the box i-frame (step 870B). Since it is j-1 in step 8706, it is determined whether there is a character in the first position in the key pattern storage section 8 (step 8707). Since the character "1" is stored in the character number 1 of the key button storage section, it is determined whether the character "1" is in the i-th frame (step 8708). If there is a character "1", determine the position of that character (Step 5709), and if the position of the character is correct, count the number of matching characters in the key image and one frame by 1 (Step 5710) Step 8
706 Return. Next, count the character number j (step 70B), the key pattern storage section 8 shown in FIG.
Step 5707 and subsequent steps are performed for the character rnJ of character number 2 stored in . Then, it is determined whether all of the 10 characters stored in the key pattern storage section 8 are in one frame, and step 57 is performed for the 11th character.
Judging by 07. In this case, since there is no 11th character, it is determined whether the number of matching characters in one frame matches the number of characters stored in the key pattern storage unit 8 (step 8711), and if they match, the key is added to the i-th frame. Since the image is included, the frame number is memorized (step 5712), and as in FIG. 2, for example, 80 frames are skipped and the search is performed using the frame one second later. Note that if it is not -j-1 in step 8711, the number of characters stored in the key pattern storage unit 8 and the number of matching characters in one frame are different, and in this case, the key image is not included in this frame. For example, 30 frames are skipped as in FIG. 2. This is done in step 8708.
If the th character is not in the i-frame, or if the
The same applies when it is determined in step 709 that the positions are different.

When the key image search is completed for all frames, the process ends.

Thereafter, display control is the same as in the first embodiment.

According to the second embodiment, it is also possible to automatically assign keywords by converting the searched characters into keywords and storing them in correspondence with frame numbers, for example, as shown in FIG.

Additionally, in order to match code information that exists separately such as a scorebook or script with frames of a moving image, input the code information from the input section 1 and use the format shown in Figure 9 to make the frame containing the code correspond. Can be searched.

[Effects of the Invention] According to the moving image search device of the present invention configured as described above,
A desired image can be obtained without adding keywords in advance.

Furthermore, if the search is performed using letters, numbers, or line figures, the desired image can be more reliably obtained without adding keywords.

[Brief explanation of drawings]

FIG. 1 is a block diagram of the moving image search device of the present invention in the first embodiment, FIG. 2 is a flowchart for explaining the operation of the information management section in the first embodiment, and FIG. FIG. 4 is a diagram showing an example of edge extraction of a key image in the first embodiment. FIG. 5 is an example of frame number storage in the first embodiment. FIG. 6 is a block diagram of a moving image search device of the present invention in a second embodiment. FIG. 7 is a block diagram of a moving image search device of the present invention in a second embodiment. is a diagram showing an example of a key image of the second embodiment, (15
) FIG. 9 is a diagram showing an example of the memory format of the second embodiment, and FIG. 1O is a diagram of a moving image database. DESCRIPTION OF SYMBOLS 1... Input section, 3... Information management section, 4... Moving image storage section, 5... Pattern matching section, 6... Key image storage section, 7... Image processing section, 8...Key pattern storage unit, 9...Pattern dictionary. An example of the memory format of the button memory section is shown below.

Claims (2)

[Claims] (1) A moving image storage means for storing moving images consisting of time-series image information, an input means for inputting search instruction information, and a method for inputting the search instruction information input by the input means and the moving image storage means. a collation means for performing a collation with a stored moving image, and the input means inputs arbitrary image information constituting the moving image stored in the moving image storage means as search instruction information. A moving image search device. (2) The moving image search device according to claim 1, wherein the image information is characters, numbers, figures, images themselves, etc. that constitute the moving images stored in the moving image storage means.