JP4053251B2 - Image search system and image storage method - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an image search system for compressing and storing multimedia data including image data and audio data and storing the multimedia data in response to a client request.
[0002]
[Prior art]
Conventionally, in an image search system for multimedia data, an image itself is searched using image features from image element information. In recent years, using an image analysis program at the time of registration, a system for searching for images in a series of image information from edge information, color information, etc., as well as still images, and searching by focusing on individual content appearing in images Thus, a system for searching for individual objects on the screen is realized. This prior art is described in JP-A-11-338876. In addition, in order to suppress an increase in the amount of information to be searched, decoding information of audio data, image data, and information registered in the index data by creating an index file including a representative frame and audio data before and after the representative frame JP-A-11-331779 discloses a system for searching for images. Further, Japanese Patent Laid-Open No. 2000-29881 discloses a method for directly retrieving compressed and encoded data in a storage device for multimedia data including audio and images.
[0003]
[Problems to be solved by the invention]
However, in these search systems, the user inputs a search keyword such as assigning a search keyword when registering video / audio / index data, or searching by separately preparing and comparing search image / audio data during search. There is a problem that means is separately required on the user side. In addition, the index file that manages the search data must be manually registered without updating the keyword information in the index file when multimedia data created by another device is copied. There is a problem.
[0004]
In order to solve the above-described problems, an object of the present invention is to provide an accumulated image retrieval system capable of more easily registering information serving as a key used for retrieval when accumulating multimedia data. And
[0005]
[Means for Solving the Problems]
The image search system includes an encoding unit that compresses and encodes multimedia data including image data and audio data, and a storage unit that stores the compressed and encoded multimedia data.
[0006]
The storage means stores in advance a predetermined voice pattern and image pattern that are keys for use in the search. The comparing means compares the audio data of the multimedia data with the audio pattern stored in the storage means when the encoding means compresses and encodes the multimedia data, and the image data of the multimedia data And the image pattern stored in the storage means. The creation means creates index information that associates the multimedia data that matches the result of the comparison by the comparison means with the identification information of the audio pattern and / or image pattern.
[0007]
DETAILED DESCRIPTION OF THE INVENTION
Embodiments of the present invention will be described below with reference to the drawings.
[0008]
FIG. 1 shows a system configuration diagram of an embodiment of the present invention. In FIG. 1, the stored image retrieval system includes a server device 1, a video pump device 2, a content creation terminal 3, and a client device 8, each of which is constituted by a personal computer, and each device includes a communication means 7 such as a LAN. To communicate through. The client device 8 selects desired multimedia data and issues a distribution request, whereby the server device 1 transmits the request data of the client device 8 to the video pump device 2. The video pump device 2 includes a disk drive unit 5 and can store multimedia data created by the content creation terminal 3. The user inputs a keyword from the keyboard of the input device 9 of the client device 8. For the input keyword, the search unit 4 in the server device 1 determines whether there is a matching keyword. If the keyword matches, the multimedia data is distributed to the client device 8. The content creation unit 6 of the content creation terminal 3 creates content data in which the matched keyword is input by comparing the keyword information with pre-registered pattern information when the input multimedia data is compressed and encoded into a file. To do. The created multimedia data file is stored in the video pump device 2, and information obtained by linking the search keyword information and the file name is recorded in the search information management table in the search unit 4 of the server device 1.
[0009]
Next, a content data creation method will be described. FIG. 2 shows a block diagram of the content creation unit 6 provided in the content creation terminal 3 in the present embodiment, and FIG. 3 shows a block diagram of the encoder unit 30 and the pattern comparison unit 60 of the content creation unit 6.
[0010]
As shown in FIG. 2, the control unit 80 controls each block. The communication control unit 100 performs interface control with external communication means. The system bus 20 includes data lines for control data, image data, and audio data. As multimedia data composed of video and audio, image data from the camera and audio data from the microphone are respectively input to the input unit 10. The input image data and audio data are converted from analog signals to digital signals by the A / D unit 20. The image data and sound data converted into digital signals are subjected to encoding processing in accordance with the encoding method selected by the encoder unit 30 in order to compress and encode.
[0011]
The audio data is recorded in the audio memory unit 340 via the audio encoder unit 310. The input image data is temporarily recorded in the video memory unit 320 by the video encoder unit 310 shown in FIG. 3, and divided into pixel units having luminance and color difference, and a plurality of pixels are divided into one block (macro block). Is encoded as In performing the encoding process, a motion vector is calculated for each macroblock, and a motion amount is calculated by a motion amount detection unit 600 that calculates a motion amount per frame. When the calculated change in the motion amount is larger than the threshold value α, the control unit 80 sends a comparison processing start instruction to the pattern comparison unit 60 in order to start the voice pattern comparison. When the comparison processing starts, the control unit 80 causes the audio data extraction unit 610 to Fourier transform the audio data recorded in the audio memory unit 340 to convert it into a spectral component. The shape of the converted spectrum is referred to as a spectrum envelope, and the spectrum is decomposed into a spectrum envelope and a residual component referred to as a residual signal which is fine structure information of a speech spectrum representing sound source information. Each decomposed component is recorded in the multimedia memory unit 40 as a spectrum pattern and a sound source pattern. At the same time, the image data is recorded in the multimedia memory unit 40 along with image configuration information after the image data recorded in the video memory unit 320 is decomposed into macroblock units for each frame by the image data extraction unit 640. The image / sound data recorded in the multimedia memory unit 40 is recorded until the comparison process is completed. Furthermore, the video encoder unit 310 records the time management information value of the data encoded by the encoding process in the multimedia memory unit 40. The time management information value is an information value indicating information on when image data and audio data are decoded and reproduced. This is a process for eliminating a sense of incongruity that the reproduced video of the decoding and the reproduced audio are shifted due to the delay caused when the image data and the audio data are encoded. In the encoded multimedia data, image data and audio data are decoded based on this information. After the image data and the sound data are recorded in the multimedia memory unit 40, the recorded sound data and the sound pattern recorded in advance in the sound pattern recording unit 630 are compared by the sound pattern comparison unit 620.
[0012]
The voice pattern storage unit 630 stores keywords corresponding to the voice pattern as shown in FIG. In the pattern storage unit 630, the reception unit 635 can receive and register the voice pattern and the keyword. In addition to patterns corresponding to individual keywords, voice patterns corresponding to a plurality of keywords can be registered in advance. As a result of the comparison, the keyword of the matched pattern is recorded in the keyword extraction unit 670. Even if there are a plurality of keywords, all the keywords are recorded in the keyword extraction unit 670. The control unit 80 selects an image pattern from the image pattern recording unit 660 from the keywords recorded in the keyword extraction unit 670. In the image pattern recording unit 660, keywords corresponding to the image patterns are registered in advance. As the image pattern, an image subjected to edge processing of an image corresponding to the keyword is registered. Since there may be a plurality of image patterns for one keyword, the control unit 80 may issue an instruction to select an image pattern having a high similarity frequency from the previously detected keyword image patterns. The image pattern comparison unit 650 compares the image data recorded in the multimedia memory unit 40 sequentially from the image pattern selected by the control unit 80. As a result of the comparison, if there is a matched pattern, the corresponding keyword is recorded in the keyword detection unit 680. The control unit 80 recognizes that the keyword has been detected, extracts the keyword and the time management information value of the corresponding video / audio unit from the multimedia memory unit 40 as index data, and causes the index data generation unit 700 to record them. In the recorded index data, each time management information value is recorded for each keyword. When the calculated change in the motion amount is smaller than the threshold value α, the control unit 80 does not issue a pattern comparison instruction, and extracts the audio data and the image data until the change in the motion amount becomes larger than the threshold value α. Absent. When there is no matching keyword as a result of comparison by the voice pattern comparison unit 620, the control unit 80 records the keyword “no voice pattern match” in the keyword detection unit 680 and the time recorded in the multimedia memory unit 40. It is recorded in the index data generation unit 700 together with the management information value. When there is no keyword matched by the image pattern comparison unit, the control unit 80 records the keyword “no image pattern match” in the keyword detection unit 680, together with the time management information value recorded in the multimedia memory unit 40. This is recorded in the index data generation unit 700.
[0013]
The encoded audio data and image data are multiplexed by the multiplexing unit 50. A format example of multiplexing is shown in FIG. In FIG. 4, image data and audio data are each divided into 48-byte streams called packets. This includes attribute information of image data or audio data in a 5-byte header section 160 to form a 53-byte packet. Here, a packet including image data is referred to as a video packet, and a packet including audio data is referred to as an audio packet. The multiplexing unit 50 performs time division multiplexing by switching between video and audio packets. Upon completion of the multiplexing of the multimedia data, the file creation unit 710 uses the index data time management information of the index data generation unit 700 as an information value (time stamp value) indicating the reproduction start position of the desired video after the search, along with the keyword. And recorded at the beginning of the multiplexed multimedia data.
[0014]
FIG. 5 shows a data format example. As shown in FIG. 5, index data, image data, and audio data are each divided into 48-byte streams called packets. This includes the 5-byte header section 150 including index data, image data, and audio data attribute information to form a 53-byte packet. Here, a packet including index data is referred to as an index packet, a packet including image data is referred to as a video packet, and a packet including audio data is referred to as an audio packet. The index packet stores the keywords of all the multimedia data in the same file and the time stamp value corresponding to the keyword. For each of the index packet, video packet, and audio packet, a file creation unit 710 creates a multimedia file with the beginning of the multiplexed multimedia data as an index packet. When playing back multimedia data, only the video packet or audio packet is played back in order to read the information in the header part of the index packet and recognize that the packet is not data to be played back.
[0015]
FIG. 6 shows a flowchart of the multimedia data creation process.
[0016]
In FIG. 6, multimedia data is input, and multimedia data file creation processing is started (step 400). A motion amount is acquired from the image data by a motion amount acquisition process (step 4100). FIG. 7 shows an outline of the motion amount detection process. In FIG. 7, the image data from the encoder unit 310 shown in FIG. 3 is alternately accumulated in the prediction memory 1 (4710) or the prediction memory 2 (4720). The difference detection unit 4730 collects the difference between frames of the image data in the prediction memory 1 or the prediction memory 2 or the average image data of the prediction memory 1 and the prediction memory 2 and the current image data. Further, the vector value is calculated by shifting in the vertical direction and using the position where the difference becomes small as a motion vector. The calculated vector value is converted as a motion amount, and the motion amount is determined for each frame. In step 4200, the acquired amount of motion is compared with the threshold value α. If the amount of motion is greater than the threshold value α in step 4200, pattern comparison processing is started (step 4300). The pattern comparison process is performed by the audio data comparison unit 620 and the image data comparison unit 650 shown in FIG. 3, and the matching keywords are recorded in the keyword detection unit 680 (step 4300). An index is created by index creation processing from the keywords compared and extracted by the pattern comparison processing (step 4400). FIG. 8 shows an example of index creation processing. The keyword recorded in the keyword detection unit 680 shown in FIG. 3 and the time management information value recorded in the multimedia memory unit 40 shown in FIG. In the keyword detection unit 680 shown in FIG. 3, the number of detected keywords is recorded at the beginning of the recording unit. Since the pattern comparison / time management information value is not updated unless the change in the amount of movement exceeds the threshold value, the keyword recorded in the keyword detection unit is managed by one time management information value. The number of keywords, keywords, and time management information are recorded in the index data generation unit 700 shown in FIG. 3 until the amount of motion exceeds the threshold and a new pattern comparison process is started. The keyword generated as a result of the new pattern comparison is recorded after the previous time management information value before the next pattern comparison process starts. When recording, if the same keyword as the previous one exists, the keyword may be deleted and the number of keywords may be reduced.
[0017]
After the index creation process, the completion of multimedia data creation is determined in the multimedia data creation end judgment process in step 4500. If the amount of motion is smaller than the threshold value α in step 4200, it is determined that there is no change in the comparison target image, and multimedia data creation end determination processing (step 4500) determines the end of multimedia data creation. . At the end, a multimedia data file is created in the file creation process (step 4600). In the multimedia creation process, a process of putting the final data amount at the head of the index data is performed, a process of dividing the data into 48 bytes as the index packet 151 shown in FIG. 5, and the multimedia data together with the video data / audio data. Create When continuing, the process returns to the motion amount acquisition process in step 4100.
[0018]
Next, a process for registering a plurality of multimedia data files (search information registration) will be described with reference to FIG. FIG. 9 shows a flowchart of the search information registration process.
[0019]
In FIG. 9, when the search information registration process is started, a file registration process is performed in the content creation terminal 3 (step 5100). The file registration process is a process of registering the file name and title in the multimedia data file created by the content creation terminal 3. In this case, the user registers a file name and a title with an input device from a keyboard. When the registration of the file name and title is completed, the content creation terminal 3 transmits the file data to the disk unit unit 5 of the video pump device 2 through the communication means 7. When the file transmission to the video pump device 2 is completed, the content creation terminal 3 starts index data acquisition processing in step 5200. In the index data processing, the index data in the multimedia file is read, and the index data is transmitted to the search unit 4 of the server device 1. In the search information management table registration process (step 5300), the index data is registered in the search information management table of the search unit 4 of the server device 1. As shown in FIG. 10, the search information management table includes a keyword name 250 indicating a keyword registered in the index data, a file name 251 indicating a file name of a target multimedia file of the keyword, and a multimedia corresponding to the keyword. It consists of a time stamp value 252 corresponding to time management information indicating a data reproduction position and a title name 253 of a multimedia file which is separately input at the time of registration.
[0020]
At the time of retrieval, a desired image can be obtained by referring to the retrieval information management table to select a file name including the keyword input from the input device 9 of the client device 8 and reading the reproduction position of the file from the time stamp value. Can be searched. In the search information management table, search information such as image creation date can be recorded.
[0021]
FIG. 11 is a block diagram of multimedia data search information registration processing according to the second embodiment. In the second embodiment, a configuration in which already created multimedia data is received and a keyword is registered is shown.
[0022]
Multimedia data created by other than this system is separated into image data and audio data by a multimedia separation unit 350. The image data is decoded by the video decoder unit 360, and the motion amount at the time of decoding is detected by the motion amount detection unit 600. When the amount of motion is larger than the threshold value α, the control unit 80 starts the pattern comparison process as in the case of creating the multimedia data described above. The audio data is decoded by the audio decoder unit 380. When the control unit 80 starts the comparison process, the audio extraction unit extracts audio data and compares it with a previously recorded audio pattern. Similarly, comparison with an image pattern is also performed, and a keyword and time management information are detected. The detected keyword and time management information are recorded in the index data creation unit 700. At the end of the decryption process of the multimedia data, the file creation unit 710 extracts the multimedia data created by other than this system from the multimedia data recording unit 400, and the time management information of the index data of the index data generation unit 700 is A new file is created by recording the information value (time stamp value) of the reproduction start position along with the keyword at the beginning of the extracted multimedia data. At this time, there are two types of multimedia files with the same contents by creating a new file. Even when the creation of a new file is rejected, the file generation unit 710 can create an index file in which the file name and index data information are recorded, and record it in the search information management table when registering the file. . This makes it possible to automatically insert keywords into multimedia data created by a separate system.
[0023]
【The invention's effect】
By using the image search system of the present invention, it is possible to automatically register keywords when creating multimedia without registering keywords and feature data when searching for multimedia data. Furthermore, by selecting a keyword based on voice data and the amount of movement, registration is possible without inserting excessive search data.
[Brief description of the drawings]
FIG. 1 is a system configuration diagram of an image search system. FIG. 2 is a block diagram of a content creation unit. FIG. 3 is a block diagram of a pattern comparison unit. FIG. 4 is an explanatory diagram showing an example of a multimedia data multiplexing format. Explanatory diagram showing an example of multimedia file data format [FIG. 6] File creation processing flow diagram [FIG. 7] Motion amount detection processing block diagram [FIG. 8] Explanatory diagram showing an example of index data creation [FIG. 9] Search information registration processing flow FIG. 10 is an explanatory diagram showing an example of a search information management table. FIG. 11 is a pattern comparison block diagram of files created by other than this system.
FIG. 12 is an explanatory diagram of an audio pattern and an image pattern.
DESCRIPTION OF SYMBOLS 1 Server apparatus 2 Video pump apparatus 3 Content creation terminal 4 Search part 5 Disk unit part 6 Content creation part 7 Communication means 8 Client apparatus 9 Input apparatus 10 Input part 20 A / D part 30 Encoder part 40 Multimedia memory part 50 Multimedia Multiplexer 60 Pattern comparison unit 70 File creation unit 80 Control unit 90 Memory unit 100 Communication control unit 110 Decoder unit 150 Header unit 151 Index data 152 Video data 153 Audio data 160 Header unit 250 Keyword name 251 File name 252 Time stamp value 253 Title name 300 Encoder input unit 310 Video encoder unit 320 Video memory unit 330 Audio encoder unit 340 Video memory unit 350 Multimedia separation unit 360 Video decoder unit 70 Video memory unit 380 Audio decoder unit 390 Audio memory unit 400 Multimedia recording unit 600 Motion amount detection unit 610 Audio data extraction unit 620 Audio pattern comparison unit 630 Audio pattern recording unit 640 Image data extraction unit 650 Image pattern comparison unit 660 Image pattern Recording unit 670 Keyword extraction unit 680 Keyword detection unit 700 Index data generation unit 710 File generation unit 4000 File creation processing 4100 Motion amount acquisition processing 4200 Threshold discrimination 4300 Pattern comparison processing 4400 Index creation processing 4500 End discrimination 4600 Multimedia data creation processing 4710 Prediction Memory 1
4720 Prediction Memory 2
4730 Difference detection unit 5000 Search information registration processing 5100 File registration processing 5200 Index data acquisition processing 5300 Search information management table registration processing.

Claims (5)

In an image search system comprising: encoding means for compressing and encoding multimedia data including image data and audio data; and storage means for storing the compressed and encoded multimedia data.
Storage means for storing a predetermined sound pattern and image pattern that are keys for use in the search, and sound of the multimedia data when the encoding means compresses and encodes the multimedia data As a result of comparing the data with the sound pattern stored in the storage means, comparing the image data of the multimedia data with the image pattern stored in the storage means, and the comparison means. Creating means for creating index information that associates the multimedia data with the audio pattern and the image pattern;
The storage means stores in advance one or more keyword information for the voice pattern and the image pattern,
The comparison unit compares the audio data of the multimedia data with the audio pattern stored in the storage unit, detects keyword information corresponding to the matched audio pattern, and an image pattern corresponding to the keyword information; Compare the image data of the multimedia data,
The creating means creates the index information including keyword information of the image pattern matched by the comparing means;
An image search system characterized by that.   The image search system according to claim 1, wherein when the multimedia data is compression-encoded by the encoding means, it is detected that the motion amount of the image of the multimedia data is equal to or greater than a predetermined motion amount. The image search system according to claim 1, further comprising a detection unit, wherein the comparison unit performs the comparison when the detection is performed by the detection unit. The image retrieval system according to claim 1, wherein the comparison means and the speech pattern and the image pattern and the comparison of results, the multimedia data does not match with the speech pattern, or, in the case that does not match the image pattern in response, the generation means, the data show that matching voice pattern is not an image retrieval system or to create index information including data, a indicating that the image pattern is not matching, characterized in that, . An image storage method in a terminal for compressing and storing multimedia data including image data and audio data,
The terminal stores in advance a predetermined voice pattern, which is a key to be used for search, an image pattern, and one or more keyword information for the voice pattern and the image pattern. A storage unit, a control unit,
The control unit compares the audio data of the multimedia data with the audio pattern, detects keyword information corresponding to the matched audio pattern, detects an image pattern corresponding to the keyword information, and the multimedia data An image storage method comprising comparing image data and creating index information that associates keyword information of a matched image pattern with the multimedia data. The image storage method according to claim 4,
When the control unit compresses and encodes the multimedia data, the controller detects that the amount of motion of the image of the multimedia data is equal to or greater than a predetermined amount of motion,
The image storage method, wherein the control unit creates index information of the multimedia data when the detection is performed.

Images