JP2008199136A - Similar video search apparatus and similar video search method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To compare a reference video with an image and a search target video and the image, change into a suitable feature value, and obtain search results which agrees with human being's similarity feeling. <P>SOLUTION: The apparatus decomposes the video into brightness and hue based on the decoded video, and performs classification by contrast of lightness and darkness for the brightness after performing correction based on a time-space filter. It performs classification of color tone for hue after performing correction by the time-space filter if needed. The feature value is calculated from these classification results, and similar videos are searched based on the agreement of the feature value. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a video search technique, and more particularly to a similar video search apparatus and a similar video search method for searching for a similar scene from a plurality of videos.

  In recent years, with the development of network technology and the increase in capacity of data storage, it has become possible to hold a large amount of video content and view the content based on it. In addition to the large amount of video held by content holders such as broadcasting stations and production companies, the amount of video content will increase further in the future, such as a large amount of stored content due to the spread of hard disk recorders, etc., and a huge amount of video content on the Internet. Therefore, efficient video search technology is required more and more.

  Currently, keyword search is the most common form of video search means. For example, a keyword is assigned in advance for each video scene to be searched, and when searching for a desired video scene, a search is performed using the assigned keyword as a clue. However, in this method, it is necessary to assign a keyword to the video scene in advance, but the operation is generally performed manually. By the way, in addition to the enormous effort required to assign keywords to a large amount of video data, even if keywords are automatically assigned, they are assigned depending on which information in the scene is focused on. Because keywords are different, it is difficult to assign keywords according to consistent criteria.

  As another video search means, there is a method of performing a search using a scene included in a video or an image itself included in the scene as a clue for search. In this method, feature amounts are extracted from individual image data of scenes included in a video to be searched, and similar feature scenes and images included in the scenes are extracted as search results by comparing the feature amounts. .

  In general, information such as the scene included in the video to be searched or the shade, color, texture, and shape, arrangement, and content of the object included in the image, the scene included in the image, A characteristic pattern is shown by an image included in the scene. Therefore, information on colors and edges such as shade, color, texture, shape, arrangement, and content of objects that are image elements is used for searching, classifying, and identifying scenes and images and objects included in the scenes. This is an important clue.

  As a conventional video search method using information on color and edges such as the shade, color, texture, and shape, arrangement, and content of an object included in such a video scene, RGB (as the three primary colors) In color systems such as red, green, and blue (HSV) and HSV (Hues, Saturation, and Luminance in the Munsell color system), the grayscale layout information, edge pattern information, color information histogram, etc. Various techniques for retrieving similar images similar to images have been proposed.

For example, in Patent Document 1 (Japanese Patent Application Laid-Open No. 2000-48181), there is a method of automatically weighting each of a plurality of feature amounts of an image and comparing similarities between images based on the weighted feature amounts. It is disclosed. This method extracts a plurality of feature amounts from each of a reference image and image data to be searched. In this method, the average value C ₀ of the saturation of the reference image, the peak value P ₀ of the spectrum image, and the edge strength E ₀ are exemplified as the feature values. Based on the feature values extracted from the reference image, A weight applied to each feature amount is determined. For example, the weight Wc of the feature amount related to the color and the weight Wt related to the texture are determined. Here, to increase the color weighting W _C of as the average value C ₀ of the saturation of the reference image is large, the texture of the weighting as the peak value P ₀ and the intensity E ₀ of the edge of the spectral image of the reference image is larger W _t Set a larger value.

として定義される。ここで、Ｗ_ｋはキー画像についての第ｋの特徴量についての重みを、Ｒ_ｋはキー画像と検索対象の画像との第ｋの特徴量についての類似度を表し、Σはパラメータｋについての総和を表している。 Similarity R _t of similarity R _C and texture color, degree of similarity by a known method, are respectively separately determined. For example, the color similarity R _C is obtained from the average value C ₀ of the saturation of the key image and the average value C _j of the saturation of the image to be searched, and the texture similarity R _t is obtained from the spectrum image of the key image. It is obtained from the peak value P ₀ , the edge strength E ₀ , the peak value P _j of the spectrum image of the search target image, and the edge strength E _j . The overall similarity _Ra is

Is defined as Here, W _k represents the weight for the k-th feature amount for the key image, R _k represents the similarity for the k-th feature amount between the key image and the search target image, and Σ represents the parameter k. Represents the sum.

In this method, the total similarity _Ra is _calculated as (W _{C RC} + W _t R _t ) / (W from the color weight W _c , the texture weight W _t , the color similarity R _C , and the texture similarity R _{t. C} + W _t ), and it is determined whether or not the total similarity is larger than a predetermined threshold value. If the total similarity is large, it is determined that the key image and the search target image are similar.

  In this way, by using the weighted feature amount calculated according to the feature of the reference image, the reference image emphasizes the color in the reference image where the periodicity of the texture and the strength of the edge are weak and the saturation is high. In a reference image that is low in degree and has a strong texture periodicity and edges, it is possible to search for an image with an emphasis on texture.

  Further, in Patent Document 2 (Japanese Patent Laid-Open No. 5-174072) and Patent Document 3 (Japanese Patent Laid-Open No. 7-46517), similar video using shot length information of video content or scene length as a feature amount is disclosed. Search methods have been proposed. In these methods, a partial video that can be divided by video scene switching (cut points) is defined as one shot, and the time length is used as a feature amount. The comparison of the feature amount is realized by comparing the shot length sequences one by one.

  Further, in the technique disclosed in Patent Document 4 (Japanese Patent Laid-Open No. 2000-123173), image data is divided into a plurality of small areas to detect image grayscale information in each small area, and based on the detected image grayscale information. It is characterized in that the grayscale layout is quantified, and the quantified grayscale layout is extracted as image features of the entire image data or small area units. Then, we propose a method for determining similarity between image data stored in an image database and image data to be compared using image feature quantities that are the quantified shading layout, and increase the number of similar images. Various service menus are provided to enable search based on the above criteria, thereby improving convenience.

  Further, Patent Document 5 (Japanese Patent Laid-Open No. 2002-63208) proposes an image search device and an image search method that can efficiently search for similar images similar to a reference image based on human visual characteristics. Yes. This method provides a hue filter that reacts to different hues, and calculates the difference between the average value and standard deviation of each response output of the hue filter for the reference image and the average value and standard deviation of each response output of the hue filter for the comparative image. This is a method of obtaining the similarity by adding each. This method is devised in consideration of the following reasons.

  An image encoding method using DCT or the like compresses and encodes image data from the viewpoint of signal processing, and only spatial frequency sensitivity is considered among human visual characteristics, such as RGB and HSV. In the image search method based on color space, a similar image that is similar to the reference image is searched based on the similarity from the viewpoint of signal processing, so what is the human visual impression when actually viewing the image? This is because they are often different and a similar image that is similar to the reference image cannot be extracted based on the visual characteristics of the person.

  Further, in Patent Document 6 (Japanese Patent Application Laid-Open No. Hei 8-947714), when searching for a compressed moving image, the processing for extracting motion information from the compressed moving image is reduced, so that the compressed moving image is stored in the search device. Reduces the time from input to search and improves the convenience of the search device, and automatically extracts the size and shape of the moving object as well as the movement of the object appearing in the video However, a technique used as search information has been proposed.

  This is intended for compressed moving images of compression methods called MPEG1 and MPEG2 that have motion vector data generated by a known motion compensation algorithm in order to improve the time-series compression rate. Yes. Usually, in order to obtain motion information, a compressed moving image is decoded and converted into an uncompressed moving image, and then the motion information is extracted. The process of decoding a compressed video into an uncompressed video requires many calculations, and the process of extracting motion information from an uncompressed video is also performed on a block-by-block basis, so many calculations are required. Is required. Therefore, extraction of motion information from a compressed moving image requires a great deal of processing, and it takes processing time from the input of a compressed moving image until search by motion information becomes possible, and can be searched immediately. There is a problem that must not be. In the invention described in Patent Document 6, in order to perform a search based on motion information, a process of decoding the compressed moving image and extracting the motion information after converting the compressed moving image into an uncompressed moving image is not performed. By extracting the motion vectors directly, generating the shape data and the search information, the calculation amount is reduced and the calculation time is reduced, thereby improving the convenience of the compressed moving image search apparatus.

JP 2000-48181 A Japanese Patent Laid-Open No. 5-174072 JP 7-46517 A JP 2000-123173 A JP 2002-63208 A Japanese Laid-Open Patent Publication No. 8-194714 Mitsuo Ikeda, Visual psychophysics, Morikita Publishing, 1975 Visual Society of Japan (ed.), Visual Information Processing Handbook, Asakura Publishing, 2000 Hirota Hotta, Digital Image Evaluation Method and International Standard, Triqueps, 2006 Oyama Tadashi et al. (Edition), new edition Handbook of Sensory and Perceptual Psychology, Seishin Shobo, 1994 Matsushima Hiroyuki (ed.), Basics of TV Program Production Technology, Japan Film and Television Technology Association, 1996 Toshio Morita et al. (Director), TV program production technology-from basics to know-how, Kenrokukan Publishing, 1991

  However, as described above, in the search for similar scenes in a video and similar images included in the scene, it is not a point of searching for a scene that matches the human sense of similarity, but inherently the characteristics of the video and the images. Technological development is being carried out in terms of whether the quantities agree from a mathematical theoretical standpoint. For this reason, in the technique described in the above-mentioned patent document, a video and an image that exactly match in terms of mathematically comparing feature quantities of the video and the image are searched, but a human compares the video. In such a case, there is a problem that a search result consistent with a human sense of similarity is not always obtained. That is, there is a problem that the similarity obtained by the conventional similar image search is different from the actual human sense of similarity.

  For example, a scene where an announcer explains the situation while holding an umbrella in a heavy rain, such as a typhoon relay video, and a rainy street interview, etc. However, it is consistent with the scene where the general person answers the interview from the viewpoint of mathematical comparison of the features that the video and the image originally have, that is, whether the video or the whole image matches. The degree is low.

  However, from the viewpoint of human similarity, the degree of agreement between the two is considered high. Because when we judge that humans are visually similar, we are focusing on the fact that the person wearing an umbrella is reflected. This is because the degree of coincidence is considered high.

  More precisely, whether the rain is strong or weak is not a big problem from a human sense of similarity. On the other hand, in terms of mathematical comparison, it is compared from the viewpoint of image and image features such as how rain falls, whether the person wearing the umbrella is reflected, and the situation around the person wearing the umbrella. It will be.

  Furthermore, as another specific example, in a scene where the announcer is explaining the situation while holding an umbrella in heavy rain, such as a typhoon relay video, the announcer is explaining the situation beside a large rock In this case, it is determined that the images are not similar from the viewpoint of mathematical comparison of the image feature amount inherent to the images and images under the condition that there is a large rock. This is because, from the viewpoint of mathematical comparison of the characteristic amount inherent to the video and the image, whether it is next to a large rock or not is one big factor in similarity judgment and is judged as a different video.

  On the other hand, it is determined that the images are similar based on the human sense of similarity. This is because, similar to the above problem, the human sense of similarity focuses on the person who is holding the umbrella rather than on the difference in behavior around the person holding the umbrella, that is, whether it is next to a large rock or not. The reason is that.

  What can be seen from the above example is that if the viewpoint of mathematical comparison of image feature values inherent to video and images is emphasized, all feature values of video and images are taken into account, which is consistent with the human sense of similarity. It can be seen that there is a problem that there is a case of not.

  In order to solve this problem, it is necessary to combine all the feature quantities of the video and the image with a similar sense in consideration of human visual characteristics. A problem specific to video similarity search that did not need to be considered in still images, that is, human visual characteristics do not emphasize fast moving images, slow moving images, and still images in the same ratio. Rather, it is necessary to mathematically compare these feature quantities after converting them into video and image feature quantities that reflect the knowledge of physiological visual characteristics. Of course, this physiological visual characteristic is not only related to the speed of movement of the image, but also to the color and size of the object shown in the image.

  This means that the spatial frequency characteristics and temporal frequency characteristics, which are knowledge of the results of physiological considerations of human visual characteristics, are reflected in the spatial frequency characteristics and temporal frequency characteristics of the image features inherent to video and images. Is equivalent.

  As described above, it is difficult to accurately search for a video and an image that match the human similarity judgment only by mathematically comparing the image feature amounts inherent to the video and the image. Therefore, it is necessary to provide a means for extracting a similar video / image by converting it into an appropriate feature amount in consideration of human physiological visual characteristics.

  Here, in order to examine a method and apparatus for extracting similar images and images that take into account human physiological visual characteristics, we first consider the viewpoint of human visual characteristics, visual brightness perception characteristics, spatial frequency Characteristics and time frequency characteristics will be described. Human physiological visual characteristics are related to spatial extent (spatial frequency characteristics) and temporal changes (temporal frequency characteristics). Further, the brightness perception characteristic with respect to the illumination light at that time is also related to these two frequency characteristics.

  The Broca-Sulzer effect is well known as a human physiological visual characteristic related to the perception of brightness. The blocker-Salzer effect is a phenomenon in which the brightness perception changes (amplifies or decreases) depending on the duration of the stimulus light. For example, 200 toroland (td) stimulus light has the highest brightness perception around 30 milliseconds. This indicates that there is a duration in which the brightness can be perceived most efficiently even at the same brightness. Furthermore, the blocker-Salzer effect increases as the stimulus light becomes stronger, and the duration of the most effective stimulus light then becomes shorter. When the stimulus light becomes weaker, the effect becomes smaller, and the duration of the most effective stimulus light at that time also becomes longer. It has been confirmed that this phenomenon is more effective in binocular vision. However, it is also suggested that this effect is weakened in the case of too strong stimulus light. In other words, it can be said that human visual characteristics have the brightness that can best perceive the image being shown, and the effect is weakened if it is too dark or too bright (Non-Patent Documents). 1: Blocker Salzer effect; 166 pages, non-patent document 3: Visual time-frequency characteristics; page 15, non-patent document 4: page 338).

  Contrast Sensitivity Function (CSF) obtained from the contrast threshold when light stimulus is given as a sine wave (stimulus of bright and dark waves) to express the spatial frequency characteristics of vision. Often used. As the contrast when measuring the threshold value, a ratio (Michelson contrast) of the amplitude of the sine wave amplitude of the light stimulus to the average luminance is used. The contrast threshold is the minimum contrast that can be perceived by the subject when the amplitude of the stimulation light is changed.

  Contrast sensitivity shows a band-pass characteristic with a maximum sensitivity in the vicinity of 3 to 5 cycles / ° at 9td (Toroland) or higher, but at lower illumination, the contrast sensitivity decreases and the band The characteristic changes from a pass type (band pass) to a low pass type (low pass). The characteristic of the band pass type (band pass) in a bright environment means that there is an effect of enhancing the edge of the image (enhancement of frequency components in the vicinity of 3 to 5 cycle / °). On the other hand, the low-pass characteristics in a dark environment are a mechanism that emphasizes increasing sensitivity in response to collecting signals from a wide range. (Non-patent document 2: Contrast threshold; page 193, Non-patent document 3: Visual spatial frequency characteristics; page 12, non-patent document 4: Spatio-temporal interaction; page 579)

  The time-frequency characteristics of the human visual system have been actively studied by flicker experiments. According to this, when the average luminance of the entire stimulation light is small, it has a low-pass filter characteristic. When the time frequency of flicker photometry (stimulated light with 180 ° phase inversion) is about 5 Hz, the contrast starts to decrease sharply, and when it reaches 20 Hz, no flicker can be seen no matter how much the contrast is increased. Next, when the average luminance of the entire stimulation light is gradually increased, the sensitivity starts to increase in the intermediate frequency characteristic region, and after a peak is reached, the sensitivity decreases and becomes a bandpass filter characteristic. The peak frequency of contrast sensitivity varies depending on the average luminance value of the entire stimulus light, but is between about 10 to 30 Hz.

  Further, the flicker threshold (threshold for feeling flicker) is always determined by the difference luminance from the average luminance regardless of the average luminance of the entire stimulation light. That is, it is shown that the flicker threshold does not change depending on the magnitude of the average luminance, but changes depending on the magnitude of the absolute value of the difference between the stimulus light and the average luminance. In other words, it indicates that the flicker threshold is changed not by the brightness of the entire screen but by the state (combination) of contrast in the screen. Furthermore, the motion of the object in the video has a motion speed that is most suitable for perception, which indicates that a video that is too slow or too fast or an object in the video is difficult to perceive. (Of course, there is also a problem of which object in the video the human is interested in and is gazing at.) (Non-patent document 1: Time frequency characteristics; 180 pages, Non-patent document 2: Contrast sensitivity characteristics; 220 pages, (Non-patent document 3: Visual time-frequency characteristics; page 15, Non-patent document 4: Temporal characteristics of visual system; page 551)

  Next, the spatiotemporal frequency of color vision will be described. The color sensitivity contrast sensitivity function differs from the brightness contrast sensitivity function in the following two points. First, the color contrast sensitivity function has a lower spatial frequency than the luminance contrast sensitivity function, and second, the color contrast sensitivity function has a lower spatial frequency. This means that there is no decrease in sensitivity. That is, the contrast sensitivity function of luminance has a band-pass type (band pass) characteristic, whereas the color contrast sensitivity function is a low-pass type (low pass). For example, according to Mullen et al., The color contrast is more sensitive in the low frequency band of 0.5 cycle / ° or less, and the luminance contrast is more sensitive in the higher frequency band. As compared with the case where the blue-yellow contrast sensitivity function is red-green, the sensitivity attenuation and cut-off at a high spatial frequency occur at a lower spatial frequency.

  Also in the time frequency characteristics, the color contrast sensitivity function is different from the brightness contrast sensitivity function, and there is no decrease in sensitivity at a low time frequency, which is similar to that of the spatial frequency characteristics (non-patent) Reference 2: Spatiotemporal frequency characteristics of color vision; 232 pages; Non-patent document 4: Spatiotemporal characteristics of visual system for chromaticity information; page 592).

  Next, the problem of applying visual characteristics as they are will be described. A case will be described in which image feature amounts inherent to video and images are converted in consideration of human physiological visual characteristics such as temporal frequency characteristics, spatial frequency characteristics, and color perception characteristics. In this case, for example, a method for searching for similar images similar to the reference image by using the shading layout information, edge pattern information, color information histogram, and the like of each image with respect to the image feature amount inherent to the video and the image. When applied, the human physiological visual characteristic is a simple band-pass filter application for the time frequency characteristic and spatial frequency characteristic, and a simple low-pass filter application for the color perception characteristic. This is equal to the image feature amount obtained by extracting only a part of the image feature amount inherent to the video and the image. Therefore, there arises a problem that the similar video / image search performance deteriorates.

  In other words, only objects that have a certain size and are moving at a certain speed in the video and images are extracted as image feature values, reflecting the visual characteristics. On the contrary, the accuracy of the similarity search is deteriorated. Note that “a certain size” means that the spatial frequency is not too high and not too low, and images and images are not large objects that cover the screen and are close to each other. It means that it is not a small object that cannot be seen without seeing it. Further, “a certain degree of speed” means that the object is not an object whose speed is too high to know whether it is moving or an object whose movement is too fast to know.

  As described above, from the viewpoint of human visual characteristics, problems of general similar video and similar image retrieval will be described.

  In Patent Document 2 (Japanese Patent Laid-Open No. 5-174072) and Patent Document 3 (Japanese Patent Laid-Open No. 7-46517), the length of a video content shot or the length of a scene is used as a feature amount. Since the method for adapting the visual characteristics is not specified, the application method of the visual characteristics is unknown.

  The method described in Patent Document 1 (Japanese Patent Application Laid-Open No. 2000-48181) is a technique in which the weights of colors and textures (peak values and edge strengths of spectrum images) are made variable. In addition, since the method of adapting the human visual characteristics is not specified, the application method to the moving image is unknown.

  Patent Document 4 (Japanese Patent Application Laid-Open No. 2000-123173) is a technique for searching for a similar image using a light / dark layout based on the light / dark information of an image as a feature amount. Since the method of adapting the visual characteristics is not specified, the application method to the moving image is unknown.

  Patent Document 5 (Japanese Patent Laid-Open No. 2002-63208) is a method for searching for a similar image similar to a reference image based on human visual characteristics, and calculates an average value and a standard deviation of output values of a hue filter with respect to the reference image. We use as it is. In addition to targeting still images, this method does not use the luminance distribution of the reference image. Furthermore, the application method when adapted to a moving image is unknown.

  In addition, some types of compressed moving images have motion vector data generated by a known motion compensation algorithm in order to improve a time-series compression rate called MPEG. . As an example in which the motion vector data is used for a similar video search, there is Patent Document 6 (Japanese Patent Laid-Open No. Hei 8-947714). This method extracts motion vectors directly from a compressed video, and generates and searches for shape data. The motion vector is intended to generate shape data, but it is a method for adapting human visual characteristics to video search. Is not specified, and the method for application to moving images is unknown.

  As described above, when realizing a method and apparatus for retrieving similar videos and similar images in consideration of human visual characteristics, the human visual characteristics are simply reflected in the similar video and similar image retrieval method and apparatus. If only the visual characteristic correction means that implements the filter is used, the video and the image are equal to the image feature amount obtained by extracting a part of the original image feature amount, and the similar video / image search performance deteriorates. Often occurs.

  In view of the above-described problems of the prior art, the present invention compares the reference video and image with the search target video and image, converts them into appropriate feature amounts, and obtains a search result that matches the human sense of similarity. It is another object of the present invention to provide an image search apparatus, a video and image search method, and a video and image search program.

  In order to solve the above problems, the video is considered from the viewpoint of what the video viewed by human beings is. This consideration is from the viewpoint of how video and images are produced, and how producers and editors such as dramas and variety compose scenes and shots to create video. It is a consideration from the point of view.

  As elements constituting the screen, there are three elements: 1) photo opportunity, 2) composition, and 3) camera angle. An excellent image is said to be a balance of three elements: 1) photo opportunity, 2) composition, and 3) camera angle. Among these elements, the 1) shutter chance is an element that gives power and power to the screen, and the composition of 2) and the camera angle of 3) are elements that give the theme (the theme of the story) itself.

  Here, themes include comical, serious, suspense, documentary, etc., and the visual expression of the theme uses the change of light and shadow to express the atmosphere of the scene, the psychological description of the characters, etc. You can make an impression. Therefore, the way in which the illumination is structured according to the contents of each scene of the video image is the same as the natural image from the creator side and the viewer side. By watching videos broadcast daily, the eyes of watching dramas and watching videos are the same for producers and viewers, and implicit rules on how to create videos for themes have been formed. This can be thought of as a result. In other words, the communication between the producer and the viewer is not absolute, but there are rules that follow certain rules, and this rule can convey the intentions of the producer to the viewers. . For example, if a drama with bright content is a dark video, the viewer is likely to be uncomfortable. If a serious and tense scene has a calm video or a refreshing video, there is a high possibility that the tense situation will not be transmitted to the viewer.

  Furthermore, there is also a problem of whether or not it is the pursuit of realistic expression only by reproducing the natural environment (actual light exposure) with video. “Real” in TV images means that images created with lighting can be seen “likely” through the screen. In other words, whether or not the information that the viewer wants to see is expressed in a range that does not feel unnatural. It is important for drama lighting to accurately capture and convey the performance and facial expression within the atmosphere of the scene, and depending on the content of the scene, there may be dramatic images that emphasize the lighting. In other words, it is one of the indispensable elements to make viewers impactful and symbolic.

  The tone of the video is what determines the impression of the video that the producer side and the viewer side have formed as an implicit rule.

  The tone of the image has a tone based on contrast between light and dark and a tone based on color. The tone based on the contrast between light and dark is based on the area ratio of light and darkness in the video and can be roughly classified as follows.

・ Low key (dark tone)
1) An expression method in which the dark part of the video occupies the majority and accentuates the bright part. 2) Most of the screen is darkened, and it is often seen in night scenes and serious dramas.
Easily obtain emotions such as misery, mystery, and fear. Medium key (average tone)
1) Intermediate expression method of low key and high key, with the area ratio of light and dark parts and middle part averaged. Flat key (flat tone)
1) Representation method with a small area ratio of light and dark parts, many intermediate parts and few shadows ・ High key (bright tone)
2) Method of adjusting brightness by adding dark parts on the whole 3) Bright keys on the whole screen, which can be seen in bright home dramas, quiz programs, etc.-Soft keys (soft tone)
1) An expression method in which the area ratio of light and dark parts is small, and the middle part is emphasized and the boundary between light and dark is changed softly. 2) The opposite of the hard keys, there are many intermediate parts, and the boundary between the light and dark parts changes softly. The tone / hard key (hard tone)
1) An expression method that emphasizes the contrast between bright and dark areas with few intermediate parts. On the other hand, color tones according to color are classified as follows: Pale tone Unification with low-saturation colors with high brightness Tone. Bright and sweet tone / bright tone A clear tone with lower brightness and higher saturation than the pale tone. Bright and cheerful tone. If it is composed of the same color system, it feels very soft, but the accent tends to be insufficient. A strong tension is born, but if you make a mistake in the color balance, it will become muddy. ・ A dull tone with a dull tone. It is a quiet and profound tone, but it is difficult to create this tone only with colored light. ・ Dark tone with darker tone than the dark tone dull tone. ・ Monotone.

  In the drama, the tone of the video is designed based on this contrast. For example, when the same tone is used, the image becomes monotonous and difficult to leave an impression. Incorporating a serious (hard key) tone into a comical drama (high key) may change the flow of the video and produce an impressive effect. In this way, a different flow can be created by combining the individual tone. (Non-patent document 5: What is drama lighting; page 53; Non-patent document 6: video tone; page 92)

  As mentioned above, it is indispensable to consider the viewpoint of video production, that is, how the video is created, and by correcting what the creator intended by visual characteristics, It is possible to extract the impression of the video received by the viewer from the expression as video feature.

  According to the present invention, by providing processing means that takes into account visual temporal frequency characteristics, spatial frequency, brightness perception characteristics, and color vision spatio-temporal characteristics as human visual characteristics, similar images that match human similar sensations are provided. And search for similar images. More specifically, an encoded video decoding unit for decoding compressed video such as MPEG, a video feature amount calculation unit reflecting human physiological visual characteristics, and a prerecorded video feature amount are input. And a video feature amount comparison unit that compares the feature amount of the video.

In order to use the human physiological visual characteristics, the video feature amount calculation unit
1) a luminance / hue separation unit that decomposes the luminance component and hue component of an image included in the video;
2) A luminance time frequency filter unit that corrects luminance components of an image included in a video using temporal visual characteristics with respect to luminance among human physiological visual characteristics;
3) a luminance spatial frequency filter unit that corrects the luminance component of an image included in a video using a spatial visual characteristic with respect to luminance among human physiological visual characteristics;
4) Hue (red-green) time for correcting the hue (red-green) component of the image contained in the video using the visual characteristics of the temporal frequency for the red-green hue among human physiological visual characteristics. Frequency filter section,
5) Hue (red-green) space that corrects the hue (red-green) component of the image contained in the image using the visual characteristics of the spatial frequency for the red-green hue among human physiological visual characteristics. Frequency filter section,
6) Hue (yellow-blue) time for correcting the hue (yellow-blue) component of the image included in the image using the visual characteristics of the temporal frequency for the yellow-blue hue among human physiological visual characteristics Frequency filter section,
7) Hue (yellow-blue) space that corrects the hue (yellow-blue) component of the image contained in the image using the visual characteristics of the spatial frequency for the yellow-blue hue among human physiological visual characteristics. Frequency filter section,
8) A light / dark contrast classifying unit that classifies an image corrected using human physiological visual characteristics related to luminance into a plurality of tones according to the luminance ratio;
9) A color tone classification unit that classifies an image corrected using human physiological visual characteristics related to hue into a plurality of tones according to the ratio of lightness and saturation,
10) It is characterized in that the result of the contrast contrast classification unit and the result of the color tone classification unit are combined into a feature amount for video feature amount comparison.

  Furthermore, the video feature quantity calculation unit receives a DCT coefficient (spatial frequency characteristic), a motion vector (time frequency characteristic), and a decoded video from an encoded video decoding unit that decodes a compressed moving image.

  As a result, it is possible to calculate a video feature amount in consideration of human physiological visual characteristics, and it is possible to perform a video search based on the impression of the video received by the viewer.

  Furthermore, it is possible to reduce the amount of calculation by inputting DCT coefficients (spatial frequency characteristics) and motion vectors (temporal frequency characteristics) from the encoded video decoding unit. The search result can be obtained with

  The present invention may be a program for causing a computer to execute the steps based on each unit, or a computer-readable recording medium for recording the program.

  As described above, according to the similar image search technique of the present invention, there is an excellent effect that a similar image reflecting human physiological visual characteristics can be searched.

  Furthermore, since the video feature quantity is calculated based on the contrast of light and darkness and the tone of color, and similar video is identified based on this feature quantity, it is possible to automatically divide shots with different contrast of light and darkness and tone of color.

  In addition, a plurality of shots (scenes) having similar combinations of continuous light and dark contrasts and color tones can be automatically divided.

  Furthermore, for a video played at high speed, the value of the high speed playback speed is added to the motion vector at the normal playback speed, and the visual characteristics are applied to this method for the motion vector corresponding to this playback speed, Similar video search during high-speed playback can be realized.

  Hereinafter, a similar video search technique according to an embodiment of the present invention will be described with reference to the drawings. 1 to 8 are drawings related to a similar video search technique according to an embodiment of the present invention. In each drawing, the same reference numerals denote the same components. The basic configuration of FIG. 2 is the same as the general configuration shown in FIG. 9, but as shown in FIG. 2, the DCT coefficient (spatial frequency characteristic) and motion vector are transmitted from the encoded video decoding unit. (Time frequency characteristic) is input, and the video feature quantity calculation unit calculates video feature quantity reflecting human physiological visual characteristics. FIG. 9 is a block diagram of encoding processing of compressed moving images such as MPEG, which is a known technique.

  FIG. 1 is a functional block diagram showing a configuration example of a similar image search apparatus according to this embodiment. In the configuration shown in FIG. 1, a video recorded on the hard disk HD 101 or a video recorded on an optical recording medium 102 such as an MO, CD, or DVD is read from the I / O interface IF 103 and the similar video extraction processing unit 107. The similar video is extracted at, and the extracted video is displayed on the display 110 connected to the display interface IF 109. At this time, if audio is included in the video, it is output from a speaker connected to the AV interface IF 108. The video can also be directly input from the communication port via the communication interface IF 106, the Video, Mic 113 or the like connected to the AV interface IF 108, and the like. The input video and similar video extraction processing program is recorded in the memory 104, for example, and the CPU 105 performs processing based on the program. This process may be a hardware process. In FIG. 1, a similar video extraction processing unit 107 is provided as an example of performing similar video extraction processing by hardware processing. The video extracted by the similar video extraction processing unit 107 is temporarily stored in the memory 104 and output to the display 110 via the display interface IF 109.

  Next, regarding the processing contents of the similar video extraction processing unit 107 according to the present embodiment, FIG. 2 is shown as a functional block diagram regarding the processing contents regardless of software processing or hardware processing. The configuration shown in FIG. 2 is basically the same as the conventional similar video processing unit shown in FIG. In order to clarify the difference from FIG. 2, first, FIG. 9 will be described.

  In FIG. 9, an encoded compressed moving image (encoded video) such as MPEG is input. MPEG is encoded by a known technique called entropy coding, and is decoded by a variable length decoding unit 901 by a known technique. At this time, among the decoded signals, the encoding mode and the motion vector are input to the motion compensation unit 904, and motion compensation processing is performed by a known technique. On the other hand, since the inverse quantization unit 902 receives the DCT coefficient quantized at the time of the encoding process, the inverse quantization process is performed here by a known technique. After that, an IDCT unit 903 generates an image included in the video by inverse cosine transform processing (a known technique). However, the image obtained at this time is an image that has not been subjected to motion compensation. Next, in the motion compensation unit 904, motion compensation processing using a motion vector is performed by a known technique and sent to the picture storage unit 905. The previous motion compensation unit 904 performs motion compensation using the images stored in the picture storage unit 905 in the past. The compensated image is sent to the picture rearrangement unit 906 and also to the picture storage unit 905. A video that has been subjected to encoding processing such as MPEG has a compression rate increased by using a known compression method called intra-frame encoding or inter-frame encoding, and there are three picture types in MPEG. These are called an intra picture (I picture), a predictive coded picture (P picture), and a bidirectional predictive coded picture (B picture), and use three pictures to improve the compression rate and the image quality. ing. The picture rearrangement unit 906 performs processing for rearranging these three pictures and returning them to the order before compression.

  The video feature amount calculation unit 911 receives a decoded video of the video before compression. In a general similar video search system, video features are calculated here. For example, in Patent Document 2 (Japanese Patent Laid-Open No. 5-174072), shot length information or scene length is calculated as a feature amount. Thereafter, the feature quantity stored in the video feature quantity DB 913 and the feature quantity of the input video are compared in the video feature quantity comparison unit 912. As a result of the comparison, the image determined to be similar is output as a similar image. Here, reference numeral 900 denotes a video decoding processing unit, and reference numeral 910 denotes an entire similar video extraction processing unit.

  Next, the difference between FIG. 2 and FIG. 9 will be described. In the configuration shown in FIG. 2, the variable length decoding unit 201 corresponds to the variable length decoding unit 901 in FIG. 9, and the same processing is performed by a known technique. Further, the inverse quantization unit 202 is in the inverse quantization unit 902, the IDCT unit 203 is in the IDCT unit 903, the motion compensation unit 205 is in the motion compensation unit 904, the picture storage unit 204 is in the picture storage unit 905, and the picture rearrangement unit. 206 corresponds to the picture rearrangement unit 906, and the same processing is performed by a known technique.

  The similar video search processing unit according to the present embodiment shown in FIG. 2 is different from the similar video search processing unit shown in FIG. 9 in that the motion vector (temporal frequency characteristic) of the variable length decoding unit 201 and the inverse quantization unit 202 are different. The DCT coefficient (spatial frequency characteristic) is sent to the video feature quantity calculation unit 211. Then, the video feature quantity calculation unit 211 uses the DCT coefficient (spatial frequency characteristic), the motion vector (temporal frequency characteristic), and the decoded video to calculate the video feature quantity that reflects human physiological visual characteristics. The feature quantity stored in the video feature quantity DB 213 and the feature quantity of the input video are compared by the video feature quantity comparison unit 212, and an image judged to be similar is output.

  Here, reference numeral 200 denotes a video decoding processing unit, and reference numeral 210 denotes a similar video extraction processing unit.

  Next, a video feature value calculation method in the video feature value calculation unit 211 will be described with reference to FIG. The video feature quantity calculation unit 211 receives the decoded video, the DCT coefficient, and the motion vector, and sends the calculated feature quantity to the video feature quantity comparison unit 212. At this time, the decoded video sent from the picture rearrangement unit 206 is decomposed into luminance and hue (red-green) and hue (yellow-blue) by the luminance / hue separation unit 301, and the luminance is transmitted to the temporal frequency filter 303. Hue (red-green) is sent to the temporal frequency filter 304, and hue (yellow-blue) is sent to the temporal frequency filter 305.

  For example, if the color space of the decoded video is RGB, conversion processing from RGB to the L * a * b * color system via the XYZ color system is performed. Also, if the color space of the decoded video is YUV, after conversion from YUV to RGB, conversion to the L * a * b * color system is performed via the XYZ color system. Of course, conversion from YUV directly to the L * a * b * color system may be performed. The color space conversion formula is, for example, as follows.

YUV to RGB conversion formula:

Conversion formula from RGB to XYZ:

Conversion formula from XYZ to L * a * b *:

  The XYZ color system has a low correlation between the color difference perception and the distance in the color system, and is inappropriate for representing a color difference that is a color difference perceived by humans. The L * a * b * color system is defined as a color function, and the conversion formula is as follows.

  Xn, Yn, and Zn are tristimulus values of illumination light (fully diffuse reflection surface under the D65 light source). X / Xn, Y / Yn, and Z / Zn have values of 0.008856 or more. In addition, when X / Xn, Y / Yn, and Z / Zn are 0.008856 or less, it becomes the following formula | equation.

  The L * a * b * color system is one of the uniform color spaces defined by the CIE in 1976 (a color space intended to allow color differences perceived to be equal in magnitude to correspond to equal distances in the space). Yes, it can measure perception and color differences due to device differences. This color system is recommended in 1976, and is defined in JIS Z8729 in Japan. This CIE 1976 L * a * b * is based on CIE XYZ, and the non-linear relationship of L * a * b * is intended to imitate the logarithmic sensitivity of the eyes. L * means Luminance (luminance), and a * and b * mean chromaticity representing hue and saturation. That is, a * and b * indicate the color direction, a * indicates the red direction, -a * indicates the green direction, b * indicates the yellow direction, and -b * indicates the blue direction.

  In the time-frequency filter 303, correction related to the time-frequency characteristic is performed among the human physiological visual characteristics based on the motion vector (time-frequency characteristic) sent from the variable length decoding unit 201. Further, the spatial frequency filter 306 performs correction related to the spatial frequency characteristic among the human physiological visual characteristics based on the DCT coefficient (spatial frequency characteristic) sent from the inverse quantization unit 202. Thereafter, the luminance information corrected by the human physiological visual characteristic is sent to the contrast contrast classification unit 309.

  In the light / dark contrast classification unit 309, a luminance histogram is created, and based on the light / dark contrast tone classifications (a) to (e) shown in FIG. 4, Low Key, Medium Key, Soft / Flat Key, High Key, and Hard Key. It is classified into each key. As a classification method at this time, clustering by a k-means method which is a known technique, SOM which is a self-organization algorithm using a neural network which is a known technique, or the like can be used. It is also possible to use a known technique, such as a clustering method based on non-line identification using a support vector machine, or a method in which luminance ratios corresponding to white, gray, and black are ruled and classified according to the rule.

  In the time-frequency filter 304, correction related to the time-frequency characteristic among the physiological visual characteristics of human beings is performed based on the motion vector (time-frequency characteristic) sent from the variable length decoding unit 201. Further, the spatial frequency filter 307 performs correction related to the spatial frequency characteristic among human physiological visual characteristics based on the DCT coefficient (spatial frequency characteristic) sent from the inverse quantization unit 202. Thereafter, the luminance information corrected by the human physiological visual characteristic is sent to the color tone classification unit 310.

  In the time-frequency filter 305, correction related to the time-frequency characteristic among the human physiological visual characteristics is performed based on the motion vector (time-frequency characteristic) sent from the variable length decoding unit 201. Further, the spatial frequency filter 308 performs correction related to the spatial frequency characteristic among the human physiological visual characteristics based on the DCT coefficient (spatial frequency characteristic) sent from the inverse quantization unit 202. Thereafter, the luminance information corrected by the human physiological visual characteristic is sent to the color tone classification unit 310.

  In order to correct the time frequency characteristic and the spatial frequency characteristic, a method of tabulating each characteristic value can be used. In the case of performing correction based on the time frequency characteristic, correction is performed by a method such as selecting a correction value from luminance and time frequency values using the motion vector value as the time frequency value and multiplying by the weight coefficient. When correction is performed using the spatial frequency characteristic, correction is performed by a method of selecting a correction value from luminance and temporal frequency values using the DCT coefficient value as a spatial frequency value and multiplying the correction value as a weighting coefficient. Note that normally, a compressed compressed moving image encoded by MPEG or the like can obtain luminance and DCT coefficients for each block such as 8 × 8 pixels or 16 × 16 pixels. In that case, the time frequency and the spatial frequency may be corrected for each block.

  In the color tone classification unit 310, after performing conversion from the L * a * b * color system to the HSV color system in order to perform classification based on lightness and saturation, for example, based on the values of lightness and saturation, for example, Each tone is classified according to the tone classification shown in FIG. As a tone classification method, clustering by a k-means method that is a known technique, SOM that is a self-organization algorithm using a neural network that is a known technique, or the like can be used. Furthermore, it is also possible to use a clustering technique based on non-linear identification using a support vector machine, which is a known technique. In addition, it is possible to form a table of combinations of lightness and saturation values that can be easily estimated and realized by those skilled in the art, and classify them according to the tone classification values at that time. The HSV color space is a known description method in which colors are represented by H (Hue / hue), S (Saturation / saturation), and V (Value / lightness).

  As described above, the video feature values obtained from the contrast contrast classification unit 309 and the color tone classification unit 310 are sent to the video feature value comparison unit 212 in FIG. 2 and compared with the video feature values in the video feature value DB 213. First, an image with a high degree of similarity is output. In general, the feature amount used at this time is determined by the comparison unit in the video feature amount comparison unit 212. For example, the comparison unit is defined as the distance between feature vectors, and the Euclidean distance and the direction cosine distance between vectors, which are known techniques, and the Mahalanobis distance considering the variance between vectors, which is a known technique, are used. Can do.

  As a specific example, for example, when the feature quantity format is defined as follows, classified as Low Key in the contrast contrast classification unit 309, and classified as monotone in the color tone classification unit 310, the following feature vector It becomes.

Feature quantity format = {light and dark contrast feature quantity vector, color tone feature quantity vector}
= {“Low Key”, “Medium Key”, “Soft / Flat Key”, “High Key”, “Hard Key”, “Monotone”, “Pale Tone”, “Dark Tone”, “Bright Tone”, “Dal Tone” "," Bibit tone "}
Feature vector = {light and dark contrast feature vector, color tone feature vector}
= {1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0}

  The above is an example in which the light / dark contrast classification is Low Key = 1, otherwise 0, and the color tone classification is monotone = 1, otherwise 0. Next, the distance between the vectorized feature quantity and the feature vector of an image in the video recorded in the video feature quantity DB 213 shown in FIG. 2 may be calculated by a known technique. Depending on the purpose of the application, when vectorizing the result of light / dark contrast classification and the result of color tone classification, a weighting factor may be used to add a weight as to which is important. For example, when the result of the light / dark contrast classification and the result of the color tone are set to a ratio of 2: 1, a feature vector of the light / dark contrast classification is created at such a ratio (a vector obtained by doubling the feature quantity of the light / dark contrast classification) ) Can also be considered. In such a case, the following feature quantity vector is obtained.

Feature vector = {2 * (light / dark contrast feature vector), color tone feature vector}
= {2, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0}

  For example, when the feature quantity is a Euclidean distance between vectors, a feature vector of the input video is a vector, and a feature vector of one video recorded in the video feature quantity DB is a b vector, the Euclidean distance r is Is calculated as follows.

  Here, ak and bk represent the kth element of the vector. That is, the sum of the squares of the differences between the elements of each vector is related to the magnitude. The direction cosine distance r is calculated as follows.

  Here, a · b represents an inner product, and | a | and | b | represent the magnitudes of the vector a and the vector b, respectively. That is, the angle formed by the vector a and the vector b is expressed as a distance. Next, in order to use the degree of similarity obtained for an image as the degree of similarity of a video, it is only necessary to add the degrees of similarity for all images constituting the video. That is, if the similarity of an i-th image (i is an integer equal to or greater than 1) included in one video recorded in the video feature DB is ri, the similarity R between the input video and the i-th image is It can be calculated as follows.

  Here, the degree of similarity between one entire video recorded in the video feature DB and the entire input video is represented.

  Further, when calculating whether or not a part of one video recorded in the video feature DB is similar to the input video, for example, dynamic programming (a mathematical programming method) which is a well-known technique is used. And the effect of the decision process made over multiple stages with the passage of time can be modeled into a mathematical expression to obtain an optimum condition. By using the start-free dynamic programming method for the image feature vector of the input video and the image feature vector of one video recorded in the video feature DB, it is recorded in the video feature DB. A partial video of one video can be calculated as a similar video. Usually, in this case, a threshold is used to determine similarity. That is, if the calculated similarity is smaller than the threshold, it can be determined that the similarity is high, and if the calculated similarity is larger than the threshold, it can be determined that the similarity is low. This known technique is widely known as a pattern recognition technique.

  Next, the flow of processing in the video feature amount calculation unit 211 will be described using a flowchart. Note that reference numerals 200, 212, and 213 in FIG. 2 are omitted because they are not characteristic portions of the present invention. Also, reference numeral 200 in FIG. 2 has the same configuration as the reference numeral 900 in FIG. 9, and is a functional block diagram for performing encoding processing of compressed moving images such as MPEG. A difference from the known technique is a part for extracting a motion vector from the variable length decoding unit 201 in FIG. 2 and a part for extracting a DCT coefficient from the inverse quantization unit 202. Since these are information extracted by the variable length decoding unit 901 and the inverse quantization unit 902 in FIG. 9, they may simply be input to the video feature amount calculation unit 211 in FIG. 2. If you have knowledge of this technology, you can easily imagine how to extract it.

  The data structure of the MPEG bit stream is composed of a sequence header part and a sequence extension, a GOP header, a picture header, picture data, and extension / user data. In the picture data, specific coding for decoding each picture is performed. Contains parameters. This decoding includes processing such as decoding of DCT coefficients in the block and decoding of motion vectors. This data is encoded with a variable length and is decoded by the variable length decoding unit 201 in FIG. At this time, a motion vector is obtained and sent to the motion compensation unit 205. Also, the DCT coefficient obtained by the inverse quantization unit 202 is sent to the video feature amount calculation unit 211.

  With reference to FIGS. 2 and 8, the flow of processing in the video feature amount calculation unit 211 in FIG. 2 will be described. As shown in FIG. 8, first, in step S801, a decoded video that is an output of a decoding process of the encoded video is input. In step S802, the motion vector of the variable length decoding unit 201 in FIG. 2 is read. Further, in step S803, the DCT coefficient of the inverse quantization unit 202 in FIG. 2 is read.

  Next, the color system of the decoded video input in step S804 is determined. If the color system is RGB, the process proceeds to step S805, where the RGB color system is changed to L: a * b * color system. The color system conversion of the video signal is performed. If the decoded video input in step S804 is not an RGB color system, it is further determined in step S806 whether the decoded video is a YUV color system. If it is the YUV color system (Yes), the process proceeds to step S807, and the inter-color system conversion of the video signal from the YUV color system to the L * a * b * color system is performed.

  If it is determined in step S806 that the system is not a YUV color system (No), the process returns to step S801.

  In step S804 and step S806, the RGB color system and the YUV color system are determined. However, the present invention is not limited to such processing, and color systems such as the YCrCb color system are also targeted. In this case, if it is determined in step S806 that it is not the YUV color system, the process does not return to step S801, the determination of the YCrCb color system is performed, and if it is determined Yes, the YCrCb table is determined. The color system conversion of the video signal from the color system to the L * a * b * color system is performed.

  As described above, by performing the processing in steps S805 and S807, the decoded video input in step S801 can be converted into the L * a * b * color system regardless of the input. it can. Next, in the decoded video converted into the L * a * b * color system, the luminance-related signal is corrected by the time-frequency characteristic using the motion vector information input in step S802 in step S808. Is done. In step S809, correction based on frequency characteristics is performed using the DCT coefficient information input in step S803. Thereafter, in step S810, for example, tone classification is performed based on contrast between light and dark with reference to FIG.

  In addition, in the decoded video converted into the L * a * b * color system, a signal related to the hue (red-green) is obtained in step S811 by using the motion vector information input in step S802. Correction based on frequency characteristics is performed. In step S812, the frequency characteristic correction is performed using the DCT coefficient information input in step S803.

  Further, in the decoded video converted into the L * a * b * color system, a signal related to the hue (yellow-blue) is obtained using the motion vector information input in step S802 in step S815. Correction based on time-frequency characteristics is performed. In step S816, correction based on the frequency characteristics is performed using the DCT coefficient information input in step S803.

  After that, information on the hue (red-green) corrected in step S812 and the hue (yellow-blue) corrected in step S816 are obtained from the L * a * b * color system to the HSV color system in step S813. Color system conversion is performed to obtain information on brightness and saturation. In step S814, classification by color tone is performed using the lightness and saturation information obtained by the conversion in step S813 and using FIG. 5 as a reference.

  After that, using the light / dark contrast classification result in step S810 and the color tone classification result in step S814, feature quantity generation is performed in step S817.

  Through the above processing, the decoded video input to the video feature quantity calculation unit 211 in FIG. 2 is calculated as a feature quantity and is input to the video feature quantity comparison unit 212 in FIG.

  As described above, by comparing the encoded video input in FIG. 2 with the video feature quantity recorded in the video feature quantity DB 213 by the video feature quantity comparison unit 212, a similar video is output. Become.

  Hereinafter, description will be given with reference to FIGS. FIG. 4 is a diagram showing tone classification based on contrast between light and dark, out of two elements constituting a video tone. This contrast of light and dark is based on the area ratio of light and dark in the video, and is a well-known technique (reference 5; page 54, reference 6; page 92) that serves as a guideline for program production. Note that the tone classification based on the area ratio is not precisely defined, and is therefore defined by the ratio of the approximate area ratio of each of white, gray, and black. Therefore, in this embodiment, the soft key and the flat key are treated as the same.

  FIG. 5 is a diagram showing tone (color tone) classification by color among the two elements constituting the tone of the video. The tone based on the color is determined by lightness and saturation, and is a known technique (Reference 6; page 93) to which PCCS (Practical Color Coordinate System) is applied. Note that the tone based on the color is not only classified as shown in FIG. For example, the boundary between a monotone and a bright tone is called a light grayish tone, and the boundary between a monotone and a dull tone is called a grayish tone. In this embodiment, since the target is not a tone such as a printed matter but a display device such as a television, the classification as shown in FIG. 5 can be used as it is.

  FIG. 6 illustrates the human physiological visual characteristics related to luminance, with the horizontal axis representing the spatial frequency ν and the vertical axis representing the time frequency f. A point of equal contrast sensitivity is connected on the basis of the maximum contrast sensitivity value, which is a known characteristic (Non-Patent Document 3: Visual spatio-temporal frequency characteristic; page 17). In the spatial frequency, when f is small, the band pass characteristic is obtained, but when f becomes large (about 20 Hz or more), the low pass characteristic is obtained. Further, in the time-frequency characteristic, when ν is small, it is a band pass characteristic, but when ν becomes large (about 5 CPD or more), it becomes a low pass characteristic.

  Note that it is known that the low-pass characteristic is obtained in both the spatial frequency characteristic and the temporal frequency characteristic in the case of the equal contrast sensitivity having a value smaller than the maximum contrast sensitivity value (Reference 1: Temporal frequency characteristic). 180, document 2: contrast threshold; 193, page 220, document 3: visual spatial frequency characteristics; pages 12, 15; document 4: spatiotemporal interaction; pages 551, 579). In the present embodiment, a human physiological visual characteristic related to luminance, which is a well-known knowledge, is used. For example, the spatio-temporal characteristics of luminance as shown in FIG. 6 can be stored as a table, for example, and luminance correction can be performed using this.

  FIG. 7 is a diagram showing the relative contrast sensitivity of color vision at a spatial frequency. The white-black characteristic is a luminance characteristic, and is shown for comparison with the color vision characteristic. It has been found by experiments so far that the contrast sensitivity function of color vision is different from the contrast sensitivity function of luminance in the following two points. One is that the color contrast sensitivity function has a sensitivity attenuation and cut-off on the high spatial frequency side at a lower spatial frequency than the luminance contrast sensitivity function. Second, the color contrast sensitivity function does not show a decrease in sensitivity at low spatial frequencies. That is, the contrast sensitivity function of luminance has a band-pass type (band pass) characteristic, whereas the color contrast sensitivity function is a low-pass type (low pass). Also, the color contrast is more sensitive in the low frequency band, and the luminance contrast is more sensitive in the higher frequency band. Then, compared to the case where the blue-yellow contrast sensitivity function is red-green, sensitivity attenuation and cut-off at a high spatial frequency occur at a lower spatial frequency.

Also in the time frequency characteristics, the color contrast sensitivity function is different from the brightness contrast sensitivity function, and there is no decrease in sensitivity at a low time frequency, which is similar to that of the spatial frequency characteristics. (Reference 2: Spatiotemporal frequency characteristics of color vision; page 232; Reference 4: Spatiotemporal characteristics of visual system for chromaticity information; page 592)
For example, the spatial characteristics of hues (yellow-blue, red-green) as shown in FIG. 7 can be stored in a table, for example, and hue correction can be performed using this.

  It should be noted that the similar video search apparatus and method according to the present embodiment are not limited to the above-described illustrated examples, and various modifications can be made without departing from the scope of the present invention.

  For example, in FIG. 3, the spatial frequency filter 306 is performed after the temporal frequency filter 303, but this order is not necessarily required, and after performing the spatial frequency filter 306 first, the temporal frequency filter 303 is performed. Also good. Further, it is not always necessary to divide the time frequency filter 303 and the spatial frequency filter 306 into two stages, and they may be performed at a time. In this case, the correction is based on the spatio-temporal frequency characteristic that has both the characteristic of the temporal frequency filter 303 and the characteristic of the spatial frequency filter 306. The same applies to the time frequency filter 304 and the spatial frequency filter 307, the time frequency filter 305, and the spatial frequency filter 308.

  Further, in FIG. 8, the correction by the spatial frequency of step S809 is performed after the correction by the time frequency characteristic of step S808. However, this order may be changed, and after the correction by the spatial frequency of step S809 first. The correction based on the time frequency characteristic in step S808 may be performed. Further, it is not always necessary to divide into two stages of step S808 and step S809, and they may be performed at a time. In this case, the correction is based on the spatio-temporal frequency characteristic having both the temporal frequency characteristic of step S808 and the spatial frequency characteristic of step S809. This process is the same for steps S811 and S812, and also for steps S815 and S816.

  In the present embodiment, the feature amount of the video is calculated using human physiological visual characteristics. This human physiological visual characteristic may give somewhat different results depending on the experimenter. For example, there is a research result example that presents a result that a cut-off on the high spatial frequency side occurs at a low temporal frequency in the temporal frequency characteristics of color perception. This embodiment applies human physiological visual characteristics, and does not feature physiological visual characteristics themselves. Therefore, a practitioner may perform correction based on human physiological visual characteristics as appropriate in a form suitable for the application to be used.

  Further, although the present embodiment describes an apparatus and a method for searching for similar videos using compressed video encoded by MPEG or the like as input, the input video is not necessarily encoded compressed video. There is no need. Usually, an encoding process such as MPEG is a process of generating an encoded compressed moving image by using an unencoded video as an input. In the process of generating the compressed moving image, the motion vector and DCT coefficient shown in FIG. 2 are obtained and embedded in the encoded video signal. That is, in the case of an unencoded video, it is possible to obtain the motion vector and the DCT coefficient by performing the same process as the encoding process such as MPEG which is a known technique. Therefore, in the present embodiment, when an unencoded video is input, the same processing as the encoding technique such as MPEG which is a known technique is performed, and after obtaining the motion vector and the DCT coefficient, What is necessary is just to perform the process by the image | video feature-value calculation part 211 of FIG. Since encoding processing such as MPEG is a known technique, it is described in many documents.

Summary The similar video search technique according to the present embodiment stores a table of spatio-temporal characteristics (temporal frequency characteristics and spatial frequency characteristics) in a memory (or stored in a database on a network) and refers to this. To correct. In order to refer to the table, the input image is decomposed into luminance and hue components. Next, the correction value of the table is applied to the luminance information. The brightness information corrected by the spatio-temporal characteristics is classified by contrast, and the hue is classified by color tone. Thereafter, it is vectorized as a video feature amount, and compared with other input video (similar feature vectors), it is determined whether or not they match.

Therefore, (a) Table of luminance spatio-temporal characteristics
(b) Correction means based on the spatio-temporal characteristics of luminance
(c) Classification means based on contrast table for classification using corrected luminance information
(d) Classification means by color tone classification table using hue
(e) Means for characterizing input video (light / dark contrast, color tone)
(f) It is preferable to have a degree of coincidence calculation means by comparing video feature amounts.

  The hue is corrected as well as the luminance. As an example of the correction means for that purpose, the characteristics as shown in FIGS. 6 and 7 may be tabulated in the same manner as the luminance.

  As described above, according to the similar image search technique according to the present embodiment, there is an excellent effect that it is possible to search for a similar image reflecting human physiological visual characteristics. Furthermore, since the image feature amount is calculated based on the contrast of light and darkness and the tone of color, and similar images are identified based on the feature amount, scene division can be automatically performed for each scene having a different contrast of light and darkness and color tone. . Further, it is possible to automatically divide a plurality of shots (scenes) in which the combination of light and dark contrast and color tone is continuously similar. For example, in a news program or the like, a studio video and an on-the-job video can be automatically divided. Moreover, in the drama, the effect that it can divide | segment into a recollection scene and other scenes is exhibited. In addition, it is assumed that a normally used movie is played back at the same speed as when it was recorded. However, with this method, for a movie played at a high speed, the motion vector of the normal playback speed has a high playback speed. By adding this value and applying this method of determining similarity based on visual characteristics to a motion vector corresponding to this playback speed, it is possible to realize a similar video search during high-speed playback.

  Note that the characteristic at the normal playback speed (FIG. 6) is, for example, that the value of the time frequency is doubled in double speed playback (fast forward). Therefore, at the time of double-speed playback, by performing correction using the value of the corresponding time frequency, it is possible to realize the similarity of the video that humans feel during high-speed playback, and high-speed playback similar video search is possible (in normal playback and Similar video search results differ during high-speed playback).

  The present invention can be used as a similar video search apparatus.

It is a functional block diagram which shows one structural example in the similar image search device by one embodiment of this invention. It is a functional block diagram of a similar video extraction processing unit according to the present embodiment. It is a functional block diagram of the image | video feature-value calculation process part by this Embodiment. It is a figure which shows an example of the contrast contrast classification used in this Embodiment. It is a figure which shows an example of the color tone classification used in this Embodiment. It is a figure which shows the visual physiological spatiotemporal frequency characteristic regarding the brightness | luminance used in this Embodiment. It is a figure which shows an example of the spatial frequency characteristic regarding the color used by this Embodiment. It is a flowchart figure which shows the flow of the image | video feature-value calculation process by this Embodiment. It is a functional block diagram of a general similar image extraction processing unit.

Explanation of symbols

101 ... HD, 102 ... MO, CD, DVD, 104 ... Memory, 105 ... CPU, 106 ... Communication IF unit, 107 ... Similar video extraction processing unit, 108 ... AV IF unit, 109 ... Display IF unit, 200 ... Code 201: variable length decoding unit, 202 ... inverse quantization unit, 203 ... IDCT unit, 204 ... picture storage unit, 205 ... motion compensation unit, 206 ... picture rearrangement unit, 210 ... similar video detection unit 211 ... Video feature value calculation unit, 212 ... Video feature value comparison unit, 213 ... Video feature value DB, 301 ... Luminance / Hue separation unit, 303 ... Luminance information time frequency characteristic filter unit, 304 ... Hue (red-green) Information temporal frequency characteristic filter unit, 305 ... Hue (yellow-blue) information temporal frequency characteristic filter unit, 306 ... Luminance information spatial frequency characteristic filter unit, 307 ... Hue (red-green) spatial time frequency characteristic filter unit, 308 ... Hue (Yellow-Blue) Spatio-temporal frequency characteristics filter unit, 309 ... Light / dark contrast classification unit, 310 ... Color tone classification unit, 900 ... Symbol Video decoding unit, 901 ... variable length decoding unit, 902 ... inverse quantization unit, 903 ... IDCT unit, 904 ... motion compensation unit, 905 ... picture storage unit, 906 ... picture rearrangement unit, 910 ... similar video detection unit, 911 ... Video feature value calculation unit, 912 ... Video feature value comparison unit, 913 ... Video feature value DB.

Claims (18)

A similar video search device for searching for a video similar to the input video from the stored video,
A video correction unit for performing correction based on human physiological visual characteristics with respect to the input video;
A first classification unit that classifies the video based on the contrast in the video after being corrected by the correction unit;
A second classification unit that classifies the video based on color tone in the corrected video,
A feature amount generation unit that generates a video feature amount based on a classification result by the first classification unit and a classification result by the second classification unit;
A similar video search apparatus, comprising: a search unit that searches for similar videos based on the video feature amount. The video correction unit
The similar image according to claim 1, wherein the input image is decomposed into luminance and hue, and the luminance and hue are corrected by visual characteristics of both temporal frequency characteristics and spatial frequency characteristics regarding luminance and hue. Search device.   3. The similar video search apparatus according to claim 2, wherein each of luminance and hue is corrected by referring to a table of the temporal frequency characteristic and the spatial frequency characteristic.   The luminance information corrected by the spatio-temporal characteristics is classified by contrast, and the hue is classified by color tone, and then vectorized as a video feature value, and compared with other input video. 4. The similar video search device according to claim 3, wherein the similarity of the video is calculated.   5. The similar image search device according to claim 1, wherein after the correction based on the physiological visual characteristics of human beings, the classification based on contrast between light and dark and the classification based on color tone are performed.   In the course of decoding the input encoded video, a motion vector and a DCT coefficient are extracted, the motion vector is used for correction by temporal frequency characteristics, and the DCT coefficient is used for correction by spatial frequency characteristics. The similar image search device according to claim 1.   Compensating means for performing correction based on the human physiological visual characteristics, wherein the input image is decomposed into luminance, hue (red-green) and hue (yellow-blue), and the temporal frequency characteristic or spatial frequency characteristic is The similar image search apparatus according to claim 1, further comprising a correction unit based on at least one visual characteristic. A video feature amount calculation unit that calculates a video feature amount that reflects human physiological visual characteristics with respect to the video,
Video feature amount comparison means for comparing a pre-recorded video feature amount with a newly input video feature amount;
A similar video search unit that searches for similar videos based on the comparison of the feature quantities,
The video feature amount calculation unit
A luminance / hue separation unit that decomposes the luminance component and hue component of the image included in the video;
A luminance temporal frequency filter unit that corrects a luminance component of an image included in an image using temporal visual characteristics with respect to luminance among human physiological visual characteristics;
A luminance spatial frequency filter unit that corrects the luminance component of an image included in an image using a spatial visual characteristic with respect to luminance among human physiological visual characteristics;
Hue (red-green) temporal frequency filter that corrects the hue (red-green) component of the image contained in the video using the visual characteristics of temporal frequency for the red-green hue among human physiological visual characteristics And
Hue (red-green) spatial frequency filter that corrects the hue (red-green) component of the image contained in the video using the visual characteristics of the spatial frequency for the red-green hue among human physiological visual characteristics And
Hue (yellow-blue) temporal frequency filter that corrects the hue (yellow-blue) component of the image contained in the image using the visual characteristics of the temporal frequency for the yellow-blue hue among human physiological visual characteristics And
Hue (yellow-blue) spatial frequency filter that corrects the hue (yellow-blue) component of an image contained in the image using the visual characteristics of the spatial frequency for the yellow-blue hue among human physiological visual characteristics And
A light / dark contrast classifying unit that classifies an image corrected using human physiological visual characteristics related to luminance into a plurality of tones according to a luminance ratio;
A color tone classification unit that classifies an image corrected using human physiological visual characteristics related to hue into a plurality of tones according to a ratio of lightness and saturation;
A similar video search apparatus characterized in that a classification result by the contrast contrast classification unit and a classification result by a color tone classification unit are combined into a feature amount for video feature amount comparison.   The video feature amount calculation unit receives DCT coefficients (spatial frequency characteristics), motion vectors (temporal frequency characteristics), and decoded video from an encoded video decoding unit that decodes a compressed moving image. 8. The similar video search device according to 8.   10. The similar video search apparatus according to claim 8, further comprising means for dividing a scene based on the video feature amount.   10. The similar video search device according to claim 8, further comprising means for collecting a plurality of scenes having similar combinations of light and dark contrast and color tone based on the continuity of the video feature amount. A similar video search method for searching a video similar to an input video from stored video,
Correcting for human physiological visual characteristics;
Classifying the input video according to contrast of light and dark;
Categorizing input video according to color tone,
A similar video search method comprising a step of generating a video feature amount based on the classification result of the contrast contrast and the classification result of the color tone. Correction by the human physiological visual characteristics is
13. The similar image search method according to claim 12, wherein the input image is decomposed into luminance and hue, and corrected by at least one visual characteristic of time frequency characteristics or spatial frequency characteristics.   14. The similar image search method according to claim 12, wherein classification based on contrast between light and dark and classification based on color tone are performed after correction based on human physiological visual characteristics.   During the process of decoding the input encoded video, the motion vector and DCT coefficient are extracted, the motion vector is used for correction by temporal frequency characteristics, the DCT coefficient is used for correction by spatial frequency characteristics, and human physiological The similar image search method according to claim 12, further comprising a step of correcting a visual characteristic.   When performing correction based on human physiological visual characteristics, the input image is decomposed into luminance, hue (red-green), and hue (yellow-blue), and at least one of temporal frequency characteristics or spatial frequency characteristics The similar image search method according to claim 12, wherein correction is performed based on visual characteristics.   A program for causing a computer to execute the steps according to any one of claims 12 to 16.   A computer-readable recording medium for recording the program according to claim 17.

Images