JP2002288185A - Image retrieval system and image accumulation method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image retrieval system, which allows users to more easily register keywords when accumulating multimedia. SOLUTION: Predetermined voice and image patterns, which are keys to be used for retrieval, and one or more pieces of keyword information for the voice pattern and/or the image pattern are stored in advance. Voice data of multimedia data are compared with voice patterns, and then keyword information corresponding to a matched voice pattern is detected. Furthermore, an image pattern corresponding to the keyword information is compared with image data of the multimedia data, and then index information, which connects the keyword information of the matched image pattern to the multimedia data, is generated.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image retrieval system for compressing and storing multimedia data including image data and audio data, storing the data, and delivering the stored multimedia data in response to a request from a client. It is.

[0002]

2. Description of the Related Art Conventionally, in an image retrieval system for multimedia data, an image itself is retrieved by utilizing characteristics of an image from image element information. In recent years, a system that searches for a series of image information based on edge information, color information, and the like in the same manner as a still image, and a search that focuses on individual content appearing in an image by using an image analysis program at the time of registration. As a result, a system for searching for individual objects on the screen is realized. This prior art is described in JP-A-11-338876. In addition, in order to suppress an increase in the amount of information to be searched, an index file including a representative frame and audio data before and after the representative frame is created, so that audio data, image data, and decoding information of information registered in the index data are created. A system for retrieving an image is described in JP-A-11-331779. Japanese Patent Laid-Open No. 2000-2988 discloses a method for directly retrieving compression-encoded data in a storage device for multimedia data including audio and video.
No. 1 publication.

[0003]

However, in these search systems, a user assigns a search keyword when registering video / audio / index data, or prepares and compares search image / audio data separately at the time of search. There is a problem that means for inputting a search keyword, such as searching, is separately required on the user side. In addition, the index file for managing the search data must be manually registered without updating the keyword information in the index file when the multimedia data created by another device is copied, which is troublesome. There is a problem.

[0004] In order to solve the above-mentioned problems, the present invention provides a stored image search system capable of more easily registering information serving as a key to be used for search when storing multimedia data. The purpose is to:

[0005]

An image retrieval system comprises: an encoding unit for compressing and encoding multimedia data including image data and audio data; and a storage unit for storing the compressed and encoded multimedia data. Having.

[0006] The storage means stores in advance a predetermined voice pattern and image pattern which are used as keys for searching. The comparing means compares the audio data of the multimedia data with the audio pattern stored in the storage means when the encoding means compresses and encodes the multimedia data, and compares the image data of the multimedia data with the audio data. And the image pattern stored in the storage means. The creating unit creates index information that associates the matched multimedia data with the identification information of the audio pattern and / or the image pattern as a result of the comparison by the comparing unit.

[0007]

Embodiments of the present invention will be described below with reference to the drawings.

FIG. 1 shows a system configuration diagram of an embodiment of the present invention. In FIG. 1, the stored image search system includes a server device 1, a video pump device 2, a content creation terminal 3, and a client device 8, each of which is constituted by a personal computer, and each device includes a communication unit 7 such as a LAN. Communicate via. The client device 8 selects desired multimedia data and issues a distribution request, so that the server device 1 transmits the requested data of the client device 8 to the video pump device 2. The video pump device 2 has a disk drive unit 5
And the multimedia data created by the content creation terminal 3 can be stored. The user inputs a keyword from the keyboard of the input device 9 of the client device 8. The entered keyword is the server device 1
In the search unit 4 in, it is determined whether a matching keyword exists, and when the keyword matches, multimedia data is delivered to the client device 8. The content creation unit 6 of the content creation terminal 3 creates content data in which a matching keyword is input by comparing the keyword information with pre-registered pattern information when the input multimedia data is compression-encoded and filed. I do. The created multimedia data file is stored in the video pump device 2, and information linking the search keyword information and the file name is recorded in the search information management table in the search unit 4 of the server device 1.

Next, a method of creating content data will be described. FIG. 2 is a block diagram of the content creation unit 6 provided inside the content creation terminal 3 in the present embodiment, and FIG.
FIG. 2 shows a block diagram of a pattern comparison unit 60 and 0.

As shown in FIG. 2, a control unit 80 controls each block. The communication control unit 100 controls an interface with external communication means. The system bus 20 includes data lines for control data, image data, and audio data. Image data from a camera and audio data from a microphone are input to the input unit 10 as multimedia data composed of video and audio. The input image data and audio data are converted from analog signals to digital signals by the A / D unit 20. The digital signalized image data and audio data are subjected to an encoding process according to the encoding method selected by the encoder unit 30 in order to perform compression encoding.

The audio data is transmitted to the audio encoder 3
The data is recorded in the audio memory unit 340 via the control unit 10.
The input image data is temporarily recorded in the video memory unit 320 by the video encoder unit 310 shown in FIG.
Divide into pixels having luminance and color difference, and
It is encoded as one block (macroblock).
In performing the encoding process, a motion vector is calculated for each macroblock, and a motion amount is calculated by a motion amount detection unit 600 that calculates a motion amount per frame. When the change in the calculated motion amount is larger than the threshold α, the control unit 80 starts the pattern comparison of the voice
A start instruction for comparison processing is sent to 0. When the comparison process starts, the control unit 80 causes the audio data extraction unit 610 to perform Fourier transform on the audio data recorded in the audio memory unit 340 and convert the audio waveform into spectral components. An outline of the converted spectrum is called a spectrum envelope, and the spectrum is decomposed from the spectrum into a spectrum envelope and a residual component called a residual signal which is fine structure information of a voice spectrum representing sound source information. The decomposed components are recorded in the multimedia memory unit 40 as a spectrum pattern and a sound source pattern. At the same time, the image data is
The image data recorded in 0 is decomposed into macroblock units for each frame by the image data extraction unit 640, and is recorded in the multimedia memory unit 40 together with image configuration information.
The image / audio data recorded in the multimedia memory unit 40 is recorded until the comparison processing is completed. Further, the video encoder unit 310 stores the time management information value of the data encoded in the encoding process in the multimedia memory unit 4.
Record at 0. The time management information value is an information value indicating when to decode and reproduce image data and audio data.
This is a process for eliminating a sense of incongruity such that the reproduced video and the reproduced audio are deviated from each other due to a delay generated when the image data and the audio data are respectively encoded. In the encoded multimedia data, image data and audio data are decoded based on this information. After recording the image data and the audio data in the multimedia memory unit 40, the recorded audio data is compared with the audio pattern previously recorded in the audio pattern recording unit 630 by the audio pattern comparison unit 620.

The voice pattern storage unit 630 stores in advance keywords corresponding to voice patterns as shown in FIG. In the pattern storage unit 630, the reception unit 635 can receive and register a voice pattern and a keyword. As the voice pattern, a voice pattern corresponding to a plurality of keywords can be registered in advance in addition to a pattern corresponding to an individual keyword. As a result of the comparison, the keyword of the matched pattern is recorded in the keyword extracting unit 670. Even if there are a plurality of keywords, they are all recorded in the keyword extracting unit 670. The control unit 80 selects an image pattern from the image pattern recording unit 660 from the keywords recorded in the keyword extraction unit 670. Keywords corresponding to the image patterns are registered in the image pattern recording unit 660 in advance. As the image pattern, an image obtained by performing edge processing of an image corresponding to a keyword is registered.
Since there may be a plurality of image patterns for one keyword, the control unit 80 may output an instruction to select an image pattern with a high similarity frequency from the image patterns of the keyword detected last time. The image pattern comparison unit 650 compares the image data recorded in the multimedia memory unit 40 sequentially from the image pattern selected by the control unit 80. As a result of the comparison, when there is a matched pattern, the corresponding keyword is detected by the keyword detecting unit 680.
To record. The control unit 80 recognizes that the keyword has been detected, extracts the keyword and the time management information value of the corresponding video / audio unit from the multimedia memory unit 40 as index data, and causes the index data generation unit 700 to record it. In the recorded index data, each time management information value is recorded for each keyword. If the calculated change in the amount of motion is smaller than the threshold α, the control unit 80 does not issue a pattern comparison instruction, and extracts the audio data and the image data until the change in the amount of motion becomes larger than the threshold α. Absent. The control unit 80 controls the voice pattern comparing unit 6
As a result of the comparison in step 20, if there is no matching keyword, the keyword “no voice pattern matching” is recorded in the keyword detecting unit 680, and the time management information value recorded in the multimedia memory unit 40 and the index data generating unit Record at 700. If there is no keyword that matches the image pattern comparing unit, the control unit 80 records the keyword “No image pattern matching” in the keyword detecting unit 680, and together with the time management information value recorded in the multimedia memory unit 40, This is recorded in the index data generation unit 700.

The coded audio data and image data are multiplexed by a multiplexing unit 50. FIG. 4 shows a multiplexing format example. In FIG. 4, image data and audio data are each divided into 48-byte streams called packets. A 5-byte header 160 includes attribute information of image data or audio data, and forms a 53-byte packet. Here, a packet containing image data is called a video packet, and a packet containing audio data is called an audio packet. Multiplexer 5
At 0, time division multiplexing is performed by switching between video and audio packets. When the multiplexing of the multimedia data is completed, the file creation unit 710 uses the time management information of the index data of the index data generation unit 700 as an information value (time stamp value) indicating the reproduction start position of the desired video after the search together with the keyword. Is recorded at the head of the multiplexed multimedia data.

FIG. 5 shows an example of a data format. FIG.
As shown in (1), index data, image data, and audio data are each divided into 48-byte streams called packets. This contains a 5-byte header 1
Reference numeral 50 includes attribute data of index data, image data, and audio data, and forms a 53-byte packet. Here, a packet containing index data is called an index packet, a packet containing image data is called a video packet, and a packet containing audio data is called an audio packet. In the index packet, all keywords of the multimedia data in the same file and a time stamp value corresponding to the keyword are stored. Each of the index packet, the video packet, and the audio packet is created by the file creating unit 710 as a multimedia file with the head of the multiplexed multimedia data as an index packet. When reproducing the multimedia data, only the video packet or the audio packet is to be reproduced in order to read the information of the header part of the index packet and recognize that the packet is not the data to be reproduced.

FIG. 6 shows a flowchart of the multimedia data creation processing.

In FIG. 6, multimedia data is input, and a multimedia data file creation process is started (step 400). The motion amount is acquired from the image data by the motion amount acquisition process (step 4100). FIG.
The outline of the motion amount detection processing is shown in FIG. In FIG. 7, image data from the encoder unit 310 shown in FIG. 3 is alternately stored in the prediction memory 1 (4710) or the prediction memory 2 (4720). The difference between the frames of the image data in the prediction memory 1 or the prediction memory 2 or the image data of the average value of the prediction memory 1 and the prediction memory 2 and the current image data is collected by the difference detection unit 4730, Then, a vector value is calculated by displacing in the vertical direction and setting a position where the difference becomes small as a motion vector. The calculated vector value is converted as a motion amount, and the motion amount is determined for each frame. The acquired amount of motion is compared with the threshold value α in step 4200. If the motion amount is larger than the threshold value α in Step 4200, the pattern comparison process is started (Step 4300). The pattern comparison process is performed by the audio data comparison unit 620 and the image data comparison unit 650 shown in FIG.
The matching keyword is recorded in the keyword detection section 680 (step 4300). An index is created by an index creation process from the keywords compared and extracted by the pattern comparison process (step 4400). FIG. 8 shows an example of the index creation process. The keyword recorded in the keyword detection unit 680 shown in FIG. 3 and the time management information value recorded in the multimedia memory unit 40 shown in FIG. FIG.
In the keyword detecting section 680, the number of detected keywords is recorded at the beginning of the recording section. Unless the change in the amount of movement becomes larger than the threshold value, the pattern comparison / time management information value is not updated. Therefore, the keyword recorded in the keyword detection unit is managed by one time management information value. Until the amount of motion exceeds the threshold and a new pattern comparison process is started, the index data generation unit 700 shown in FIG. 3 records the number of keywords, keywords, and time management information. The keyword resulting from the new pattern comparison is recorded after the previous time management information value before the next pattern comparison process starts. In recording, if the same keyword as the previous keyword exists, the keyword may be deleted and a process of reducing the number of keywords may be performed.

After the index creation processing, step 450
In multimedia data creation end determination processing of 0, the creation end of multimedia data is determined. If the motion amount is smaller than the threshold value α in step 4200,
It is determined that there is no change in the image to be compared, and in the multimedia data creation end determination processing (step 4500), the creation end of the multimedia data is determined. At the end, a file of multimedia data is created by file creation processing (step 4600). In the multimedia creation process, a process of putting the final data amount at the beginning of the index data is performed, a process of dividing the data into 48 bytes as an index packet 151 shown in FIG. 5 is performed, and the multimedia data is combined with the video data and the audio data. Create When the continuation is performed, the process returns to the motion amount acquisition processing in step 4100.

Next, processing for registering a plurality of multimedia data files (search information registration) will be described with reference to FIG. FIG. 9 shows a flowchart of the search information registration process.

In FIG. 9, when the search information registration process is started, the content creation terminal 3 performs a file registration process (step 5100). The file registration process is a process of registering a file name and a title in a multimedia data file created by the content creation terminal 3.
In this method, a user registers a file name and a title using an input device from a keyboard. When the registration of the file name and the title is completed, the content creation terminal 3 transmits the file data to the video pump
Is transmitted to the disk unit 5. When the file transmission to the video pump device 2 is completed, the content creation terminal 3 starts the index data acquisition processing in step 5200. The index data processing reads the index data in the multimedia file and transmits the index data to the search unit 4 of the server device 1. Search information management table registration processing (step 530)
At 0), the index data is registered in the search information management table of the search unit 4 of the server device 1. As shown in FIG. 10, the search information management table includes a keyword name 250 indicating a keyword registered in the index data,
A file name 251 indicating the file name of the target multimedia file of the keyword, a time stamp value 252 corresponding to the time management information indicating a reproduction position of the multimedia data corresponding to the keyword, and a multimedia file input separately at the time of registration It is composed of a title name 253.

At the time of search, a file name including a keyword input from the input device 9 of the client device 8 is selected by referring to the search information management table.
By reading the reproduction position of the file from the time stamp value, it is possible to search for a desired image.
The search information management table can also record search information such as the date of creation of an image.

FIG. 11 is a block diagram showing a multimedia data search information registration process according to the second embodiment. In the second embodiment, a configuration in which multimedia data that has already been created is received and a keyword is registered will be described.

Multimedia data created by a system other than the present system is separated into image data and audio data by a multimedia separation unit 350. The image data is decoded by the video decoder unit 360, and the motion amount at the time of decoding is detected by the motion amount detection unit 600. The amount of movement is the threshold α
If it is larger, the control unit 80 starts the pattern comparison processing as in the case of creating the multimedia data described above. The audio data is decoded by the audio decoder unit 380. When the control unit 80 starts the comparison process,
The audio data is extracted by the audio extraction unit and compared with a previously recorded audio pattern. Similarly, comparison with an image pattern is also performed to detect a keyword and time management information. The detected keywords and time management information are recorded in the index data creation unit 700. Upon completion of the decryption processing of the multimedia data, the file creation unit 710
Retrieves multimedia data created by a system other than the present system from the multimedia data recording unit 400, and uses the time management information of the index data of the index data generation unit 700 as a reproduction start position information value (time stamp value) together with a keyword. Is recorded at the beginning of the extracted multimedia data to create a new file. At this time, due to the creation of the new file, there are two types of multimedia files having the same contents. Even when the creation of a new file is refused, the file generation unit 710 creates an index file in which the file name and the index data information are recorded, and allows the file to be recorded in the search information management table when the file is registered. . This makes it possible to automatically insert a keyword into multimedia data created separately by the system.

[0023]

According to the image search system of the present invention, it is possible to automatically register keywords when creating multimedia without registering keywords or feature data when searching for multimedia data. . Furthermore, by selecting a keyword based on the voice data and the amount of motion, registration can be performed without inserting extra search data.

[Brief description of the drawings]

FIG. 1 is a system configuration diagram of an image search system.

FIG. 2 is a block diagram of a content creation unit.

FIG. 3 is a block diagram of a pattern comparison unit.

FIG. 4 is an explanatory diagram showing an example of a multimedia data multiplexing format.

FIG. 5 is an explanatory diagram showing an example of a multimedia file data format.

FIG. 6 is a flowchart of a file creation process.

FIG. 7 is a block diagram of a motion amount detection process.

FIG. 8 is an explanatory diagram showing an example of creating index data.

FIG. 9 is a flowchart of a search information registration process.

FIG. 10 is an explanatory diagram showing an example of a search information management table.

FIG. 11 is a pattern comparison block diagram of a file created by a system other than the present system.

FIG. 12 is an explanatory diagram of an audio pattern and an image pattern.

[Explanation of symbols]

 Reference Signs List 1 server device 2 video pump device 3 content creation terminal 4 search unit 5 disk unit 6 content creation unit 7 communication means 8 client device 9 input device 10 input unit 20 A / D unit 30 encoder unit 40 multimedia memory unit 50 multimedia Multiplexing unit 60 Pattern comparison unit 70 File creation unit 80 Control unit 90 Memory unit 100 Communication control unit 110 Decoder unit 150 Header unit 151 Index data 152 Video data 153 Audio data 160 Header unit 250 Keyword name 251 File name 252 Time stamp value 253 Title name 300 Encoder input section 310 Video encoder section 320 Video memory section 330 Audio encoder section 340 Video memory section 350 Multimedia separation section 360 Video decoder section 370 Video memory section 380 Audio decoder section 390 Audio memory section 400 Multimedia recording section 600 Motion amount detection section 610 Audio data extraction section 620 Audio pattern comparison section 630 Audio pattern recording section 640 Image data extraction section 650 Image pattern comparison Unit 660 image pattern recording unit 670 keyword extraction unit 680 keyword detection unit 700 index data generation unit 710 file generation unit 4000 file creation process 4100 motion amount acquisition process 4200 threshold value discrimination 4300 pattern comparison process 4400 index creation process 4500 end discrimination 4600 multimedia data Creation processing 4710 Prediction memory 1 4720 Prediction memory 2 4730 Difference detection unit 5000 Search information registration processing 5100 F File registration processing 5200 Index data acquisition processing 5300 Search information management table registration processing.

──────────────────────────────────────────────────続 き Continued on the front page (51) Int.Cl. ⁷ Identification symbol FI Theme coat ゛ (Reference) G10L 15/00 G10L 3/00 551Z (72) Inventor Kiyoshi Ishida 890 Kashimada, Saiwai-ku, Kawasaki-shi, Kanagawa Prefecture Stock Company Hitachi, Ltd.Social & Network Systems Division (72) Inventor Eiichiro Washita 890 Kashimada, Saiwai-ku, Kawasaki-shi, Kanagawa Prefecture Hitachi, Ltd.Social & Network Systems Division (72) Inventor Susumu Takase Kashimada, Sai-ku, Kawasaki-shi, Kanagawa Prefecture 890 Hitachi, Ltd. Social Network System Division (72) Inventor Hitoshi Hara 890 Kashimada, Saiwai-ku, Kawasaki-shi, Kanagawa Prefecture F-term, Hitachi Social Network System Division (Reference) 5B075 ND16 NK02 NK13 NK21 NR02 NR05 NR16 5B082 EA08 GA01 GC0 4 5D015 AA04 CC03 KK02

Claims (6)

[Claims] An image retrieval system comprising: an encoding unit for compressing and encoding multimedia data including image data and audio data; and a storage unit for storing the compressed and encoded multimedia data. Storage means for storing a predetermined audio pattern and image pattern, which are keys to be used at the time, and audio data of the multimedia data when the encoding data is compression-encoded by the encoding means. A comparison unit that compares the audio pattern stored in the storage unit with the image data of the multimedia data and the image pattern stored in the storage unit; Associating multimedia data with identification information of the audio pattern and / or image pattern An image search system comprising: a creation unit that creates index information. 2. The image retrieval system according to claim 1, wherein when the encoding means compresses and encodes the multimedia data, an amount of motion of an image of the multimedia data is not less than a predetermined amount of motion. An image search system, further comprising a detection unit that detects that there is, wherein the comparison unit performs the comparison when the detection unit performs the detection. 3. The image search system according to claim 1, wherein said storage means stores one or more keyword information in advance for said voice pattern and / or said image pattern, and said comparison means includes: The audio data of the multimedia data is compared with the audio pattern stored in the storage unit, keyword information corresponding to the matched audio pattern is detected, and an image pattern corresponding to the keyword information and the multimedia data are compared. An image search system, wherein the index data is compared with image data, and the creating means creates the index information according to the keyword information of the image pattern matched by the comparing means. 4. The image retrieval system according to claim 1, wherein the comparison means accepts registration of index information from a user when the result of the comparison between the audio pattern and the image pattern indicates that the multimedia data does not match. An image retrieval system further comprising a registration unit. 5. An image storage method for compressing and storing multimedia data including image data and audio data, wherein the predetermined audio pattern is used as a key to be used at the time of a search. And one or two or more pieces of keyword information for the audio pattern and / or the image pattern are stored in advance, and the audio data of the multimedia data is compared with the audio pattern. Is detected, an image pattern corresponding to the keyword information is compared with the image data of the multimedia data, and index information for associating the keyword information of the matched image pattern with the multimedia data is created. An image storage method comprising: 6. An image storage method for compressing and encoding multimedia data including image data and audio data and storing the compressed and encoded multimedia data when compressing and encoding the multimedia data. An image storage method, comprising: detecting that an amount is equal to or greater than a predetermined amount of motion; and generating keyword information of the multimedia data when the detection is performed.