JP2010020630A - Video search apparatus, video search method and computer program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a video search apparatus for accurately searching for and listing video data desired by a user. <P>SOLUTION: The video search apparatus receives a search keyword for searching a storage device storing a plurality of video data composed of a plurality of frame images, for specific video data, and searches for video data according to the received search keyword and metadata associated with each video data. The video search apparatus includes: means for generating corresponding metadata for video search for each of the plurality of frame images included in the video data; an evaluation means for evaluating the quality, camera shake, block noise and trend of the video data; a search means for searching for video data including the search keyword in the metadata; and a list generation means for generating a list in which thumbnail images and titles of a plurality of video data are arranged in descending order of video data evaluation. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a moving image search method for searching for moving image data, a moving image search device for executing the moving image search method, and a computer program for causing a computer to function as the moving image search device.

  Video posting sites that store video data uploaded by users so that they can be freely viewed online have been put to practical use. The moving image posting site is configured to search the user-desired moving image data and display a list of thumbnail images by searching for metadata attached to the moving image data, for example, a title keyword. In the list, for example, video data with a large number of viewing times are displayed in ascending order.

  In addition, an electronic album system has been put into practical use in which moving image data uploaded from a user is stored as an electronic album and the electronic album can be browsed online. The electronic album system is configured to search for specific moving image data based on a title, shooting date / time data and the like added to the moving image data, and to reproduce the searched moving image data.

On the other hand, as an apparatus for evaluating the image quality of moving images, an image quality evaluating apparatus that detects image quality deterioration of moving images compressed by MPEG or the like has been proposed (for example, Patent Document 1).
JP 2003-87442 A

However, in the conventional video search method, the search results are displayed in a list without objectively evaluating the image quality, contents, etc. of the video data, so that the video data desired by the user cannot be displayed in a list preferentially. There was a problem.
For example, two pieces of moving image data obtained by photographing the same subject are uploaded. One moving image A has much block noise and camera shake, while the other moving image B has less block noise and no camera shake compared to moving image A. Consider a case like this. In this case, the video B should be preferentially displayed as a list, but if the video A is viewed more frequently than the video B, the video A is preferentially displayed as a list. Although there are video posting sites that accept video data evaluations from users of video posting sites and list videos in descending order of the evaluation, they are based on subjective evaluations by users and are not necessarily users. Cannot display a list of desired video data.

  In addition, in conventional video posting sites, keywords are searched for video data based on the title given to the video data by the user or comments posted on the video data, but the title, comment Etc. are subjectively selected by the user, and there is a problem that the search accuracy is poor.

  The present invention has been made in view of such circumstances, and generates search metadata for each of a plurality of frame images included in moving image data based on the frame image, for example, character information included in the frame image. In addition, by configuring the video data to be objectively evaluated, a video search device and video that can search and display a list of video data desired by the user with higher accuracy than conventional video search methods. It is an object to provide a search method and a computer program.

  Another object of the present invention is to generate metadata for each video scene so that video data desired by a user can be accurately obtained based on metadata that objectively shows the contents of each video scene. An object of the present invention is to provide a moving image search device capable of searching and displaying a list.

  Another object of the present invention is to provide a moving image search apparatus capable of searching and displaying a list of moving image data desired by a user with higher accuracy by configuring the moving image data including more search keywords to be highly evaluated. It is to provide.

  Another object of the present invention is to provide a moving image search apparatus capable of preferentially displaying moving image data having high fashionability as moving image data desired by a user.

  Another object of the present invention is to provide a moving image search apparatus capable of preferentially displaying moving image data with good image quality as moving image data desired by a user.

  Another object of the present invention is to provide a moving image search apparatus that can preferentially display moving image data with small block noise as moving image data desired by a user.

  Another object of the present invention is to provide a moving image search apparatus that can preferentially display moving image data with small camera shake as moving image data desired by a user.

  The moving image search device according to the present invention receives a search keyword for searching for specific moving image data from a storage device that stores a plurality of moving image data composed of a plurality of frame images, and is associated with the received search keyword and each moving image data. In the video search device for searching video data based on the metadata, metadata generation means for generating metadata of the video data based on the frame image for each of a plurality of frame images included in the video data; Obtained by searching with an evaluation means for evaluating the moving picture data based on a frame image included in the data or metadata associated with the moving picture data, a searching means for searching for moving picture data including the search keyword in the metadata If there are multiple video data, each thumbnail image or video data The metadata, characterized in that it comprises a list generating means for generating a list that is arranged on the basis of the evaluation results of each video data.

  In the moving picture search apparatus according to the present invention, the metadata generation means includes means for extracting a character area from the frame image, and character recognition means for recognizing a character from the extracted character area, and is obtained by character recognition. Characters are generated as metadata.

  The moving image search device according to the present invention includes means for identifying a frame image at a boundary at which a video scene is switched by comparing the brightness of each of a plurality of frame images included in the moving image data, and the metadata generation means includes: For each scene, metadata of moving image data is generated based on one frame image constituting each video scene.

  The moving image search apparatus according to the present invention is characterized in that the evaluation means highly evaluates moving image data including more metadata including a search keyword.

  In the video search device according to the present invention, the evaluation unit searches an information resource including metadata associated with the video data from an external device storing a plurality of information resources, and the information resource including the metadata Means for acquiring the number of video data, and the higher the number of the information resources, the higher the evaluation of the moving image data.

  The moving picture search device according to the present invention includes means for storing place name information in which shooting position information indicating a shooting point related to moving picture data is associated with a place name of the shooting point indicated by the shooting position information, and the evaluation means includes When the shooting position information is added to the moving image data, it is specified from the means for specifying the place name of the shooting point based on the shooting position information and the place name information, and the external device storing a plurality of information resources. Means for retrieving an information resource including a place name and acquiring the number of information resources including the place name, and the higher the number of information resources including the place name related to the shooting position information of the moving image data, the higher the evaluation of the moving image data. It is characterized by the above.

  In the moving image search device according to the present invention, the evaluation means searches an information resource having a feature amount of a frame image included in the moving image data from an external device storing a plurality of information resources, and has the feature amount. Means for acquiring the number of information resources, and the higher the number of information resources having the feature amount of the frame image included in the moving image data, the higher the evaluation of the moving image data.

  In the moving image search apparatus according to the present invention, the evaluation unit performs image quality correction for changing a gradation value of a pixel constituting the frame image included in the moving image data, and a gradation value of the pixel constituting the corrected frame image And means for comparing the gradation values of the pixels constituting the uncorrected frame image, and the larger the difference between the gradation values, the lower the evaluation of the moving image data.

  In the moving image search apparatus according to the present invention, the evaluation means detects an edge of block noise from a frame image included in the moving image data, and evaluates the moving image data lower as the block noise edge is larger. It is characterized by.

  In the moving image search apparatus according to the present invention, the evaluation unit includes a unit that calculates a vibration amount of the subject by comparing the subjects included in the end portions of the plurality of frame images included in the moving image data. The larger the amount, the lower the evaluation of the moving image data.

  The moving image search method according to the present invention receives a search keyword for searching for specific moving image data from a storage device that stores a plurality of moving image data composed of a plurality of frame images, and is associated with the received search keyword and each moving image data. In the video search method for searching for video data based on the metadata, the step of generating metadata of the video data based on the frame image for each of a plurality of frame images included in the video data, and included in the video data A step of evaluating the video data based on the frame image or the metadata associated with the video data, a step of searching the video data including the search keyword in the metadata, and a plurality of video data obtained by the search. In some cases, a thumbnail image of each of a plurality of video data or a meta data of each video data. The motor is characterized by having a step of generating a list that is arranged on the basis of the evaluation results of each video data.

  A computer program according to the present invention causes a computer to search video data based on a search keyword and metadata associated with each video data from a storage device that stores a plurality of video data consisting of a plurality of frame images. And generating metadata of the moving image data based on the frame image for each of a plurality of frame images included in the moving image data; and a frame image included in the moving image data or metadata associated with the moving image data A step of evaluating the moving image data based on the step, a step of searching for moving image data including the search keyword in the metadata, and a plurality of moving image data obtained by the search, when there are a plurality of moving image data, each thumbnail image or each moving image The metadata of the data is evaluated for each video data. Characterized in that to perform the steps on a computer to generate a list that is arranged based on.

In the present invention, the metadata generation unit generates metadata of moving image data based on the frame image for each of a plurality of frame images included in the moving image data. The metadata is information used when searching for moving image data, and is generated based on the moving image data. Therefore, the metadata is more objective than the metadata generated by the user.
The evaluation means evaluates the moving image data based on frame images or metadata included in the moving image data. Since the evaluation is performed based on the frame image or the metadata, the objectivity is higher than when the user's evaluation is used as it is.
Then, the search means searches for video data including the search keyword in the metadata, and the list generation means calculates thumbnail images or metadata of the plurality of video data obtained by the search based on the evaluation result of each video data. Generate an ordered list.
Therefore, compared with the conventional moving image search method, it is possible to search moving image data desired by the user with high accuracy and display a list.
Note that the storage device storing the moving image data does not necessarily constitute a moving image search device. In other words, the present invention includes a configuration for retrieving moving image data from a storage device outside the moving image retrieval device.

  In the present invention, a character region is extracted from a frame image, a character is recognized from the extracted character region, and a character obtained by character recognition is generated as metadata. Therefore, compared with the conventional moving image search method, the moving image data desired by the user can be searched objectively and accurately and displayed in a list.

In the present invention, by comparing the brightness of each of the plurality of frame images included in the moving image data, the frame image at the boundary where the video scene is switched is specified, that is, the moving image data is divided into the plurality of video scenes. Strictly speaking, when there are two video scenes arranged in chronological order, the boundary frame image is included in any video scene, but may be included in any video scene. good.
Then, for each video scene, metadata of moving image data is generated based on one frame image constituting each video scene, that is, at least one frame image. Therefore, when a plurality of video scenes are included in the moving image data, it is possible to search the moving image data based on the metadata indicating the contents of each video scene. Therefore, it becomes possible to search the moving image data desired by the user with higher accuracy and display the list.

  In the present invention, a plurality of metadata relating to moving image data is generated. Then, the moving image data including more meta information including the search keyword is more highly evaluated. Therefore, it is possible to search the moving image data desired by the user with higher accuracy and display a list.

In the present invention, an information resource including metadata associated with moving image data is searched from an external device, and the number of information resources including the metadata is acquired. The evaluation means evaluates moving image data higher as the number of information resources including metadata increases.
Information resources are, for example, news-related photographs, images, sounds, HTML documents, XML documents, etc., and reflect the trend, so use it as a standard for evaluating video data from the perspective of the theme's trend can do. That is, when the trend of the theme of the video data is high, the number of information resources related to the video data is large, and when the theme of the video data theme is low, the number of information resources related to the video data tends to be small. There is.
Therefore, it is possible to evaluate moving image data from the viewpoint of fashionability, and to preferentially display moving image data with high fashionability as moving image data desired by the user.
Note that the search of information resources and the process of specifying the number of information resources including metadata may be executed by a video search device, or request the number of searches and information resources from an external information search device. The result may be obtained.

In the present invention, the place name of the shooting point is specified based on the shooting position information added to the moving image data and the place name information in which the shooting position information and the place name indicated by the shooting position information are associated with each other. Then, the information resource including the place name of the shooting point of the moving image data is searched from the external device, and the number of information resources including the metadata is acquired. The evaluation unit highly evaluates the moving image data as the number of information resources including the place name increases. When there is a highly fashionable subject at the video shooting location, there are many information resources related to the video data, and when there is no highly fashionable subject at the video shooting location, the information resources related to the video data Number tends to be small.
Therefore, it is possible to preferentially display the moving image data having high fashionability as the moving image data desired by the user, which is evaluated from the viewpoint of the fashionability of the subject at the shooting point of the moving image data.

In the present invention, an information resource including a feature amount common to the feature amount of the frame image included in the moving image data is searched from the external device, and the number of information resources including the feature amount is acquired. The basic principle is the same as that of the said invention etc., and an evaluation means evaluates moving image data highly, so that there are many information resources including the said feature-value.
Therefore, it is possible to evaluate moving image data from the viewpoint of fashionability, and to preferentially display moving image data with high fashionability as moving image data desired by the user.

In the present invention, the image quality of the frame image included in the moving image data is corrected, and the gradation values of the pixels constituting the frame image after the correction are compared with the gradation values of the pixels constituting the frame image before the correction. To evaluate the video data.
Therefore, it is possible to evaluate moving image data from the viewpoint of image quality, and to preferentially display moving image data with good image quality as moving image data desired by the user.

In the present invention, an edge of block noise is detected by extracting an edge from a frame image included in the moving image data, and the moving image data is evaluated lower as the edge of the block noise is larger.
Therefore, it is possible to evaluate moving image data from the viewpoint of block noise, and to preferentially display moving image data with small block noise as moving image data desired by the user.

In the present invention, the amount of vibration of the subject is calculated by comparing subjects included in the end portions of a plurality of frame images included in the moving image data. The larger the amount of vibration, the lower the moving image data. evaluate.
Therefore, it is possible to evaluate moving image data from the viewpoint of camera shake, and to preferentially display moving image data with small camera shake as moving image data desired by the user.

  According to the present invention, it is possible to search and display a list of moving image data desired by a user with higher accuracy than in a conventional moving image search method.

  According to the present invention, it is possible to accurately search and display a list of moving image data desired by a user based on metadata that objectively shows the contents of each video scene.

  According to the present invention, moving image data including more search keywords can be highly evaluated, and moving image data desired by a user can be searched and displayed in a more accurate manner.

  According to the present invention, it is possible to preferentially display a list of moving image data with high fashionability as moving image data desired by the user.

  According to the present invention, moving image data with good image quality can be preferentially displayed as a list of moving image data desired by the user.

  According to the present invention, moving image data with small block noise can be preferentially displayed as a list as moving image data desired by a user.

  According to the present invention, moving image data with small camera shake can be preferentially displayed as a list as moving image data desired by the user.

Hereinafter, the present invention will be described in detail with reference to the drawings illustrating embodiments thereof.
FIG. 1 is a schematic diagram showing a configuration of a moving image search system according to an embodiment of the present invention. The video search system according to the embodiment of the present invention includes a communication device 4 having an information resource storage unit 5, an information search device 6 for searching for information resources from the communication device 4, and video data search from the video data storage unit 2. The video search device 1 as a video posting site having a function of evaluating the video data obtained by the search from various viewpoints and displaying a list of search results in descending order of evaluation, the mobile phone terminal 7, and the terminal device 8. Each device is connected via a communication network N and configured to transmit and receive various data. The moving image search apparatus 1 according to the present invention generates search metadata for each frame image constituting the moving image data, and further, the moving image data has the theme fashion, image quality, presence of block noise, degree of camera shake, user By comprehensively evaluating based on evaluation, etc., and displaying a list of search results of moving image data in descending order of evaluation, the user can search and display a list of desired moving image data with high accuracy. is there.

  FIG. 2 is a block diagram schematically showing the configuration of the moving image search apparatus 1. The moving image search apparatus 1 is a computer including a CPU 10 that controls the entire apparatus. The CPU 10 includes a ROM 11, a RAM 12, an external storage device 13, an internal storage device 15, an input device 14 such as a mouse and a keyboard, an output device 16 such as a liquid crystal display and a CRT, a communication unit 17, and an I / F unit 18 via a bus 19. Connected through.

The ROM 11 is a non-volatile memory such as a mask ROM or EEPROM that stores a control program necessary for the operation of the computer.
The RAM 12 is a volatile memory such as DRAM or SRAM that temporarily stores various data generated when the arithmetic processing of the CPU 10 is executed.
The external storage device 13 is an optical disk drive that reads data from a recording medium 9 such as a CD-ROM or DVD-ROM in which the computer program 9a according to the embodiment of the present invention is recorded in a computer-readable manner.
The internal storage device 15 is a hard disk, for example, and records the computer program 9a read from the recording medium 9 by the external storage device 13.
The communication unit 17 is an interface for transmitting and receiving information resource search requests, moving image data, and other various data to and from the information search device 6, the communication device 4, the mobile phone terminal 7, and the terminal device 8 via the communication network N. It is.
The I / F unit 18 is connected to a moving image data storage unit 2 that stores moving image data and a map data storage unit 3 that stores map data, and the CPU 10 receives the moving image data via the I / F unit 18. The map data is read and the moving image data is written. The map data includes information indicating the place name of the point indicated by the GPS information.
The CPU 10 implements the moving image search method according to the present invention by reading the computer program 9a stored in the internal storage device 15 into the RAM 12 and executing it.

FIG. 3 is a block diagram schematically showing the configuration of the information search device 6 and the communication device 4. The information search device 6 is a computer including a CPU 60 that controls the entire device. A ROM 61, a RAM 62, an external storage device 63, an internal storage device 65, an input device 64 such as a mouse and a keyboard, an output device 66 such as a liquid crystal display and a CRT, and a communication unit 67 are connected to the CPU 60 via a bus 68. . The ROM 61 and the internal storage device 65 store a control program necessary for the operation of the information search device 6. The CPU 60 reads the control program from the ROM 61 and the internal storage device 65 to the RAM 62 and executes it, thereby executing the communication device 4. The information resource stored in the information resource storage unit 5 is searched.
The communication device 4 is a computer as a web server including a CPU 40 that controls the entire device. The CPU 40 includes a ROM 41, a RAM 42, an external storage device 43, an internal storage device 45, an input device 44 such as a mouse and a keyboard, an output device 46 such as a liquid crystal display and a CRT, a communication unit 47, and an I / F unit 48. Connected through. The I / F unit 48 is connected to an information resource storage unit 5 that stores a plurality of information resources. The information resource is data such as news-related HTML documents, XML documents, photographs, images, karaoke voices, and the like. Note that the types of information resources are not limited to these. The ROM 41 and the internal storage device 45 store control programs necessary for the operation of the communication device 4, and the CPU 40 provides information resources by reading the control programs from the ROM 41 and the internal storage device 45 into the RAM 42 and executing them. Web server function is implemented. The web server function has a function of accepting an information resource transmission request from the information retrieval device 6, the mobile phone terminal 7, the terminal device 8, and the like using the hypertext transfer protocol and transmitting the information resource to the transmission request source.

  The mobile phone terminal 7 shown in FIG. 1 is a camera-equipped mobile phone having a so-called GPS function. The mobile phone terminal 7 includes a CPU (not shown) that controls the entire apparatus. The CPU includes a ROM, a RAM, a storage unit, a communication unit that transmits and receives various data to and from the other mobile phone terminal 7 or the video search device 1, an operation unit such as a number button, a cross key, a determination button, and a menu button. A microphone, a display unit, an audio output unit, an imaging unit, a GPS processing unit, and a clock unit that convert voice transmitted by the user of the mobile phone terminal 7 into an electrical signal and transmit the electrical signal are connected via a bus. The ROM stores a program necessary for the operation of the CPU. The CPU reads out and executes the program stored in the ROM to the RAM, thereby performing transmission / reception of moving image data, a search request for moving image data, and the like. The image pickup unit converts the image formed by the lens into an electric signal such as a CCD or CMOS image sensor, and converts the electric signal converted by the image sensor into digital moving image data. An image processing unit for performing various image processing on the moving image data. The GPS processing unit includes a GPS receiver. The GPS receiver receives radio waves from GPS satellites and measures the current position of the mobile phone terminal 7. When the cellular phone terminal 7 performs an imaging process in the imaging unit, GPS information indicating the latitude and longitude obtained by the GPS processing unit and a clock unit are obtained by measuring the moving image data obtained by the imaging unit. In addition, it is configured such that shooting date / time data and the like are added as metadata and stored in the storage unit.

  The terminal device 8 is a personal computer having a web client function including a CPU (not shown) that controls the entire device.

  FIG. 4 is a flowchart showing a processing procedure of the mobile phone terminal 7 and the moving image search device 1 related to uploading of moving image data. First, the mobile phone terminal 7 reads moving image data from a storage unit (not shown) (step S11), and accepts editing of the title of the read moving image data (step S12). When the editing of the title is completed, the mobile phone terminal 7 transmits the moving image data to which the title is added to the moving image search device 1 (step S13).

  FIG. 5 is an explanatory diagram conceptually showing the data structure of moving image data. For example, the moving image data includes a plurality of frame images in chronological order, a title of the moving image data, metadata generated based on each of the plurality of frame images, GPS information indicating a shooting point of the moving image, Video data evaluation points indicating evaluation. At the time of uploading, metadata and moving image evaluation points are not attached. The title is also a kind of meta information.

  The CPU 10 of the video search device 1 receives the video data transmitted from the mobile phone terminal 7 at the communication unit 17 (step S14). And CPU10 posts the received moving image data on a moving image evaluation site (step S15).

Next, the CPU 10 calls a subroutine related to generation of metadata, and generates metadata based on a plurality of frame images included in the moving image data (step S16). Then, the CPU 10 associates the generated metadata with the moving image data (step S17), stores the moving image data associated with the metadata in the internal storage device 15 (step S18), and ends the processing.
The association in step S17 is performed, for example, by including in the header portion of the moving image data. Note that another method using a table or the like may be employed as long as the moving image data and the metadata can be associated with each other.

  In the above flowchart, the editing of the title is accepted unconditionally, but the moving image search device 1 compares the metadata character string generated in step S16 with the title, and the character string. And when the relevance of the title is low, the mobile phone terminal 7 may be configured to issue a warning. Further, the moving image search device 1 may be configured to transmit the metadata generated in step S16 to the mobile phone terminal 7 and prompt the user to input an appropriate title.

  FIG. 6 is a flowchart illustrating a processing procedure of the moving image search apparatus 1 related to generation of metadata. When a subroutine related to generation of metadata is called, the CPU 10 extracts a character area, that is, a so-called telop area from one frame image included in the moving image data (step S31), and the characters included in the extracted character area Character recognition is performed on the column (step S32). And CPU10 produces | generates as metadata based on the character string obtained by character recognition (step S33). For example, a character string is segmented, a word is extracted, and the word is used as metadata. If no character area is extracted, no metadata is generated for the one frame image.

  Next, the CPU 10 determines whether or not the metadata of all the frame images constituting the moving image data has been generated (step S34). If it is determined that the metadata of all frame images has not been generated (step S34: NO), the CPU 10 returns the process to step S31. When it is determined that the metadata of all frame images has been generated (step S34: YES), the CPU 10 ends the process.

  The video evaluation site is a service that the video search device 1 provides to external devices such as the mobile phone terminal 7 and the terminal device 8. When the moving image search device 1 receives a moving image data transmission request transmitted from the mobile phone terminal 7 or the terminal device 8, the moving image search device 1 searches the moving image data storage unit 2 for moving image data based on a search keyword such as a shooting date and a title. Then, the searched moving image data is transmitted to the transmission request source. Moreover, the moving image search device 1 transmits, to the transmission source, evaluation reception moving image data for receiving evaluation of a moving image evaluation site user (a user who uses the moving image evaluation site) for the provided moving image data together with the moving image data. When the information for evaluating the specific moving image data is transmitted from the mobile phone terminal 7 or the terminal device 8, the moving image search device 1 stores the information for evaluating the moving image data in the internal storage device 15.

In addition, the video search site periodically evaluates the posted video data. Hereinafter, a method for evaluating moving image data will be described.
FIG. 7 is a flowchart showing a processing procedure of the moving image search apparatus 1 related to evaluation of moving image data. The CPU 10 of the moving image search apparatus 1 performs comprehensive evaluation on the moving image data from various viewpoints such as fashion, image quality, presence / absence of block noise, presence / absence of camera shake, and user evaluation.

  When the subroutine is called, the CPU 10 determines whether or not the moving image data to be evaluated is moving image data that undergoes the initial evaluation (step S51).

  When it determines with it being the moving image data which receive initial evaluation (step S51: YES), CPU10 evaluates the image quality of moving image data (step S52). Specifically, a predetermined image quality correction process such as changing the gradation value for one or a plurality of frame images included in the moving image data is performed, and the gradation value of each pixel constituting the corrected moving image data; The gradation value of each pixel constituting the moving image data before correction is compared, and the image quality of the moving image data is evaluated higher as the change in the gradation value of each pixel is smaller. The gradation value of each pixel is, for example, the gradation value of each RGB component or the luminance value of each pixel. In the following description, the gradation value of each pixel is assumed to be 256 gradations. When the change in gradation value of each pixel is 0 before and after correction, it is evaluated as a perfect score.

The predetermined image quality correction processing is, for example, processing for correcting color balance, contrast, etc., and includes so-called smart correction, level correction, contrast correction, color correction, and the like. Specifically, the gradation value of the darkest pixel in the frame image included in the moving image is associated with black (gradation value 0), and the gradation value of the brightest pixel in the frame image is associated with white (gradation value 255). Make corrections. In addition, the gradation values of pixels having other brightness are associated with gradation values of 1 to 254 so that the frequency distribution of each gradation value is not unevenly distributed. For example, when the tone value of the darkest pixel in the frame image is 30 and the tone value of the brightest pixel is 200, the tone value 30 is set to 0 and the tone value 200 is set to 255. Make corrections to convert. And the other gradation values 31-199 in a frame image are matched in the range of 1-254. By correcting in this way, the color balance and contrast can be corrected.
Smart correction, level correction, contrast correction, and color correction cannot be distinguished, but the outline of each correction is as follows. That is, the smart correction is a process for correcting the color balance of the entire frame image and correcting the shadow part (tone value 0) and the highlight part (tone value 255) to be clear. This is a process that corrects only the overall brightness of the frame image without changing the hue of the frame image, and the level correction is a process that corrects the contrast of the entire frame image and the color cast or color cast. The correction is a process of identifying the shadow portion, halftone portion, and highlight portion of the entire frame image and correcting the contrast and color balance.

Next, the CPU 10 evaluates moving image data based on the magnitude of block noise (step S53). Specifically, an edge of block noise is detected from a frame image included in the moving image data, and the moving image data is evaluated lower as the edge of the block noise is larger.
The block noise is noise generated due to a quantization error due to quantization and inverse quantization of moving image data. Since quantization and inverse quantization are performed for each block of 8 × 8 pixels, for example, image discontinuity occurs at the boundary portion of the block. For this reason, a tile-like mosaic pattern occurs in a moving image with large block noise.

Next, the CPU 10 evaluates the moving image data based on the size of camera shake (step S54). Specifically, the CPU 10 calculates the amount of vibration of the frame portion by comparing the frame portions of a plurality of frame images in time series included in the moving image data. For example, the luminance value of each pixel for one horizontal line included in the frame portion of one frame image is stored. Then, the arrangement of the luminance values of the pixels for one horizontal line included in the frame portion of the other frame image is compared with the arrangement of the previously stored luminance, and the horizontal line having a certain correlation is specified. When the luminance arrangement has a certain correlation, it is estimated as a horizontal line constituting a common subject image. Similarly, horizontal lines with high correlation are specified for other frame images. Based on the positional relationship between the horizontal lines in the vertical direction, whether or not camera shake has occurred can be calculated. That is, when the frequency of the horizontal line that vibrates in the vertical direction is greater than or equal to a predetermined frequency, it can be determined that camera shake has occurred. When the frequency is less than the predetermined frequency, it is considered that the user has intentionally moved the camera up and down to take a video. Further, the magnitude of camera shake can be calculated from the magnitude of the horizontal line amplitude.
In addition, although the case where the horizontal line was used was demonstrated, the magnitude | size of the camera shake of the left-right direction is computable by performing the same process also about a vertical line.

When the process of step S54 is completed, or when it is determined that the video data is not subject to the initial evaluation (step S51: NO), the CPU 10 calls a subroutine related to the video data epidemic evaluation to call the video data epidemic. Is evaluated (step S55). Then, the CPU 10 reads the evaluations of the moving image evaluation site users collected from the mobile phone terminal 7 or the terminal device 8 through the moving image evaluation site and stored in the internal storage device 15, and calculates the average value of the evaluations. The user evaluation is performed (step S56), and the process is finished.
By appropriately taking into account the user's evaluation, evaluation elements that cannot be evaluated only with metadata can also be evaluated. However, when it is desired to evaluate the moving image data more objectively, the user evaluation may not be used.

Next, the CPU 10 calculates a moving image evaluation point, that is, a moving image evaluation point of the points calculated in steps S52 to S56, that is, a total score (step S57). The higher the trendiness, image quality, and user evaluation, the higher the video evaluation points. Also, when block noise and camera shake are large, moving image evaluation points are deducted. In the second and subsequent evaluations, the processing of steps S52 to 54 is not executed, but the moving image evaluation points are calculated using the points calculated in steps S52 to 54 during the initial evaluation. This is because the points calculated in steps S52 to S54 do not change with time.
Then, the CPU 10 associates the calculated moving image evaluation point with the moving image data (step S58) and ends the process. The association with the moving image data is performed, for example, by including moving image evaluation points in the header of the moving image data as shown in FIG.

  FIG. 8 is a flowchart showing a processing procedure of a subroutine relating to the trendy evaluation of moving image data. The CPU 10 of the video search device 1 extracts metadata added to the video data to be evaluated (step S71). Then, the CPU 10 transmits request data for requesting a search for an information resource including the extracted metadata character string to the information search device 6 (step S72). The number of metadata may be plural.

  The CPU 60 of the information search device 6 receives the request data transmitted from the video search device 1 (step S73). When the request data is received, the information resource including the extracted metadata character string is received from the communication device 4. Search is performed (step S74). Then, the CPU 60 transmits information resource number data indicating the number of searched information resources to the moving image search device 1 (step S75).

  The CPU 10 of the video search device 1 receives the information resource number data transmitted from the information search device 6 (step S76), and evaluates the video data based on the received information resource number data (step S77). Specifically, the larger the number of information resources indicated by the information resource number data, the higher the moving image data is evaluated, and the smaller the number of information resources, the lower the moving image data is evaluated.

Next, a method for searching for moving image data and displaying a list will be described.
FIG. 9 is a flowchart showing a processing procedure of the moving image search apparatus 1 related to the moving image search. The mobile phone terminal 7 transmits the search keyword and the video search request data to the video search device 1 (step S91).

The CPU 10 of the video search device 1 receives the search keyword and the video search request data transmitted from the mobile phone terminal 7 at the communication unit 17 (step S92), and uses the search keyword as a title and a search keyword. Video data included in the meta information is detected (step S93). For example, when the search keyword is “sakura”, “spring”, or “Kyoto”, the search is performed for video data that includes the characters “sakura”, “spring”, and “Kyoto” in the metadata.
Since the moving image data includes a plurality of metadata, the moving image data including a predetermined number or more of metadata including the search keyword may be searched. In addition, it is not essential that the search keyword is included in the title, and moving image data including a predetermined number or more of metadata including the search keyword may be searched. Furthermore, it may be configured to search for moving image data including at least one metadata including a search keyword.

  Next, the CPU 10 extracts a moving image evaluation point of the moving image data from each searched moving image data (step S94). Then, the CPU 10 generates a list in which thumbnail images and titles indicating the moving image data obtained by the search are arranged in ascending order of moving image evaluation points (step S95), and the communication unit 17 generates list data related to the generated list. The data is transmitted to the mobile phone terminal 7 (step S96).

  The CPU of the mobile phone terminal 7 receives the list data transmitted from the moving image search device 1 (step S97), displays a search result list of moving image data based on the received list data (step S98), and finishes the process. .

  FIG. 10 is a schematic diagram showing a list of search results displayed on the mobile phone terminal 7. As shown in FIG. 10, the search result list is formed by vertically arranging thumbnail images V1, V2,... And titles of a plurality of moving image data obtained by the search. The thumbnail images V1, V2,... Of each moving image data are arranged vertically in the order of the corresponding moving image evaluation points. On the thumbnail image V1 side, the title of the moving image data related to the thumbnail image is displayed, and further, the posting date, the poster, the number of times of reproduction, and the moving image evaluation point of the moving image data are displayed.

The moving image data related to the thumbnail images V1, V2,... Is the same subject photographed in the same manner, but the image quality, camera shake, and block noise are different. The moving image data related to the thumbnail image V1 has good image quality, no camera shake, and no block noise. Therefore, the moving image evaluation point is higher than other moving image data, and is displayed at the top of the search result list. Since the moving image data related to the thumbnail image V2 has poor image quality, the moving image evaluation point is low, and is displayed below the thumbnail image V1. A portion indicated by a broken line of the subject indicates that the contour portion is blurred. Similarly, the moving image data related to the thumbnail image V3 has camera shake, the moving image data related to the thumbnail image V4 has block noise, the moving image data related to the thumbnail image V5 has camera shake, and block noise is also generated. is doing.
As described above, according to the present invention, even moving image data obtained by photographing the same subject is displayed in a list in order from moving image data with good image quality, no camera shake, and no block noise.

  When the user can find the desired moving image data from the list of search results, the user can acquire and reproduce the moving image data from the moving image search device 1 by selecting the corresponding thumbnail image.

  In the moving image search device 1, the moving image search method, and the computer program 9a according to the embodiment configured as described above, for each frame image constituting the moving image data, search metadata is stored based on the frame image. Therefore, the moving image data desired by the user can be searched objectively and accurately compared to the conventional moving image search method.

  Further, as the moving image data desired by the user, moving image data with high fashionability can be highly evaluated, and a list can be preferentially displayed.

  Furthermore, as moving image data desired by the user, moving image data with good image quality can be highly evaluated and preferentially displayed as a list.

  Furthermore, moving image data with small block noise can be highly evaluated as moving image data desired by the user, and a list can be preferentially displayed.

  Furthermore, as moving image data desired by the user, moving image data with small camera shake can be highly evaluated and preferentially displayed as a list.

(Modification 1)
Since the moving image search device 1, the moving image search method, and the computer program 9a according to Modification 1 differ only in the generation method of meta information, the generation method of meta information will be described below.
FIG. 11 is a flowchart illustrating a processing procedure of the moving image search apparatus 1 according to the metadata generation in the first modification. The CPU 10 calculates the average brightness of each of the plurality of frame images (step S111). Then, the CPU 10 calculates a difference in average luminance between adjacent frame images in chronological order, and specifies a frame image having the difference equal to or greater than a predetermined value as a video scene switching location (step S112).

  Next, the CPU 10 extracts a character area from the frame image included in the moving image data of one video scene (step S113), and recognizes the character string included in the extracted character area (step S114). Then, the CPU 10 recognizes a specific shape pattern such as a turtle shell pattern of a soccer ball in the frame image by pattern recognition using the template image (step S115).

  When the process of step S115 is completed, metadata is generated based on the character recognition result or the pattern recognition result (step S116). Note that the moving image search apparatus 1 stores a template image and a search character string in advance in association with each other in the internal storage device 15 and the recording medium 9, and stores a character string corresponding to the template image for which pattern detection has been successful. The character string is read out from the internal storage device 15 as metadata. Further, only one metadata may be generated for each video scene, or a plurality of metadata may be generated for each video scene.

  Next, the CPU 10 determines whether or not metadata for all video scenes has been generated (step S117). When it is determined that there is a video scene for which metadata has not been generated (step S117: NO), the CPU 10 returns the process to step S113. If it is determined that the metadata of all video scenes has been generated (step S117: YES), the CPU 10 finishes the repair.

  In the moving image search method, the moving image search device 1, and the computer program according to Modification 1, even if the moving image does not include character information, meta information is generated to improve the search accuracy of moving image data. be able to.

  Also, it is possible to accurately search and display a list of moving image data desired by the user based on metadata that objectively shows the contents of each video scene.

  Further, when only one metadata is generated for each video scene, it is possible to efficiently generate search metadata for moving image data.

  In addition, although the example which detects the change of a video scene by the change of the average brightness | luminance of a frame image was demonstrated, you may make it detect the change of a video scene by comparing the shape of a luminance spectrum.

(Modification 2)
Since the moving image search apparatus 1, the moving image search method, and the computer program 9a according to the second modification are different only in the moving image data search and list generation method, the moving image data search and list generation method will be described below.
FIG. 12 is a flowchart illustrating a processing procedure of the moving image search apparatus 1 according to the moving image search of the second modification. The mobile phone terminal 7 transmits the search keyword and the video search request data to the video search device 1 (step S211).

  The CPU 10 of the video search device 1 receives the search keyword and the video search request data transmitted from the mobile phone terminal 7 at the communication unit 17 (step S212), and uses the received search keyword as a meta information. Is detected (step S213).

  Next, the CPU 10 extracts a moving image evaluation point of the moving image data from each searched moving image data (step S214). Then, the CPU 10 reevaluates the moving image data based on the number of meta information including the search keyword for each moving image data (step S215). Specifically, the number of all meta information included in one video data and the number of meta information including a search keyword in the meta information are calculated, and the search keyword is included. Calculate the meta information ratio. Then, the higher the ratio, the higher the evaluation that the content of the video data is closer to the search keyword.

  When the process of step S215 is finished, the CPU 10 executes the same process as steps S95 to 98 in steps S216 to 219.

  In the moving image search device 1, the moving image search method, and the computer program 9a according to the modified example 2, it is possible to search and display a list of moving image data desired by the user with higher accuracy.

(Modification 3)
Since the moving image search device 1, the moving image search method, and the computer program 9a according to Modification 3 differ only in the method of evaluating the fashionability of moving image data, a method of evaluating the fashionability of moving image data will be described below.
FIG. 13 is a flowchart showing a processing procedure of a subroutine relating to the trendy evaluation of moving image data in the third modification. The CPU 10 of the video search device 1 extracts metadata added to the video data to be evaluated (step S311). Then, the CPU 10 transmits request data requesting the number of times the information resource is searched using the extracted metadata character string to the information search device 6 (step S312).

  The CPU 60 of the information search device 6 receives the request data transmitted from the video search device 1 (step S313). When the request data is received, the information resource search is performed using the extracted metadata character string. The number of breaks is read from the internal storage device 65 (step S314). Then, the CPU 60 transmits frequency data indicating the read frequency to the moving image search device 1 (step S315).

  The CPU 10 of the video search device 1 receives the frequency data transmitted from the information search device 6 (step S316), and evaluates the video data based on the received frequency data (step S317). That is, the greater the number of times data, the higher the evaluation of moving image data.

  In the third modification example, the moving image data desired by the user can be searched objectively and accurately as compared with the conventional moving image search method as in the above-described embodiment.

(Modification 4)
Since the moving image search device 1, the moving image search method, and the computer program 9a according to the modified example 4 differ only in the method of evaluating the trend of moving image data, the method of evaluating the trend of moving image data will be described below.
FIG. 14 is a flowchart showing a processing procedure of a subroutine related to the trendy evaluation of moving image data in the fourth modification. The CPU 10 of the video search device 1 transmits request data for requesting a search for information resources to the information search device 6 including the video data to be evaluated (step S411).

  The CPU 60 of the information search device 6 receives the request data transmitted from the video search device 1 (step S412). When the request data is received, the CPU 60 of the information search device 6 extracts the feature amount from the video data (step S413). An information resource related to the moving image data is searched based on the feature amount (step S414). Then, the CPU 60 transmits information resource number data indicating the number of searched information resources to the moving image search device 1 (step S415).

As the feature amount, for example, as disclosed in Japanese Patent Laid-Open No. 11-96368, a coefficient obtained by extracting an outline shape of an island from a moving image and expanding a point series of edge pixels of the outline shape by Fourier series expansion is employed. Good. The CPU 60 of the information search device 6 compares the feature amount extracted from the moving image data included in the request data with the feature amount of the moving image data stored in the information resource storage unit 5, that is, the similarity, that is, each coefficient. When the deviation is calculated and the similarity is a predetermined value or more, it is determined that the information resource is related to the video data to be evaluated.
Further, a feature color appearing in a moving image may be adopted as the feature amount. The CPU 60 of the information search device 6 extracts, for example, a color that has a predetermined high saturation and occupies a moving image portion having a predetermined area or more as a feature amount. When the feature amount is used, for example, moving image data of a forest based on the same green color is retrieved as a moving image related to a forest photo.

  The CPU 10 of the video search device 1 receives the information resource number data transmitted from the information search device 6 (step S416), and evaluates the video data based on the received information resource number data (step S417).

  In the modified example 4, as in the above-described embodiment, the moving image data desired by the user can be searched objectively and accurately compared to the conventional moving image search method.

  In the embodiment, the video posting site has been described as an example, but the present invention may be applied to other various services. For example, the present invention may be applied to an electronic album system that provides a list of moving image data, that is, a service for posting photos in a so-called community.

  Further, in the embodiment and the modification, one moving image search apparatus 1 is configured to execute the moving image search process, but needless to say, it may be distributedly processed by a plurality of computers.

  Furthermore, the moving image data evaluation method shown in the embodiment and the modification is an example, and other evaluation methods may be adopted as long as the evaluation method can be objectively evaluated. For example, the moving image search device includes means for recognizing sound data accompanying the moving image data and means for generating metadata based on character information obtained by the sound recognition, and the moving image data is converted based on the metadata. You may comprise so that it may evaluate. In addition, when evaluating moving image data using metadata related to audio data, information resources related to karaoke, for example, information resources including a song title, lyrics, ranking, etc. are searched from a communication device, and moving image data is obtained. You may comprise so that it may evaluate.

  Furthermore, although the information search device is configured to execute the information resource search, the moving image search device may be configured to perform the information resource search processing.

  Furthermore, the embodiments disclosed herein are illustrative in all respects and are not restrictive. The scope of the present invention is defined by the terms of the claims, and includes all modifications within the scope and meaning equivalent to the terms of the claims.

It is a schematic diagram which shows the structure of the moving image search system which concerns on embodiment of this invention. It is a block diagram which shows typically the structure of a moving image search device. It is a block diagram which shows typically the structure of an information search device and a communication apparatus. It is a flowchart which shows the process sequence of the mobile telephone terminal and moving image search apparatus which concern on upload of moving image data. It is explanatory drawing which shows notionally the data structure of moving image data. It is a flowchart which shows the process sequence of the moving image search device which concerns on the production | generation of metadata. It is a flowchart which shows the process sequence of the moving image search apparatus which concerns on evaluation of moving image data. It is a flowchart which shows the process sequence of the subroutine which concerns on the fashionability evaluation of moving image data. It is a flowchart which shows the process sequence of the moving image search apparatus which concerns on a moving image search. It is a schematic diagram which shows the search result list displayed on the mobile phone terminal. 12 is a flowchart illustrating a processing procedure of a moving image search apparatus according to generation of metadata in Modification 1; 10 is a flowchart illustrating a processing procedure of a moving image search apparatus according to a moving image search of a second modification. It is a flowchart which shows the process sequence of the subroutine which concerns on the fashionableness evaluation of the moving image data in the modification 3. It is a flowchart which shows the process sequence of the subroutine which concerns on the fashionableness evaluation of the moving image data in the modification 4.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Movie search device 2 Movie data storage unit 3 Map data storage unit 4 Communication device 5 Information resource storage unit 6 Information search device 7 Mobile phone terminal 8 Terminal device 9 Recording medium 9a Computer program 10 CPU
17 Communication unit 60 CPU
67 Communication Department N Communication Network

Claims (12)

Receiving a search keyword for searching for specific moving image data from a storage device that stores a plurality of moving image data composed of a plurality of frame images, and moving image data based on the received search keyword and metadata associated with each moving image data In the video search device to search,
For each of a plurality of frame images included in the moving image data, metadata generating means for generating metadata of the moving image data based on the frame image;
Evaluation means for evaluating the moving image data based on frame images included in the moving image data or metadata associated with the moving image data;
A search means for searching for video data including the search keyword in metadata;
A list generating means for generating a list in which thumbnail images of each of the plurality of video data or metadata of each video data are arranged based on an evaluation result of each video data when there are a plurality of video data obtained by the search; A moving image search apparatus comprising: The metadata generation means includes
Means for extracting a character region from the frame image;
A character recognition means for recognizing characters from the extracted character area,
The moving image search device according to claim 1, wherein characters obtained by character recognition are generated as metadata. Means for identifying a frame image at a boundary where a video scene is switched by comparing the brightness of each of a plurality of frame images included in the moving image data;
The metadata generation means includes
The moving image search device according to claim 1 or 2, wherein for each video scene, metadata of moving image data is generated based on one frame image constituting each video scene. The evaluation means includes
The moving image search device according to any one of claims 1 to 3, wherein moving image data including more metadata including a search keyword is highly evaluated. The evaluation means includes
Means for retrieving information resources including metadata associated with moving image data from an external device storing a plurality of information resources, and obtaining the number of information resources including the metadata; The moving image search device according to any one of claims 1 to 4, wherein the moving image data is highly evaluated as the number of images increases. Means for storing place name information in which shooting position information indicating a shooting point related to moving image data is associated with a place name of the shooting point indicated by the shooting position information;
The evaluation means includes
In the case where shooting position information is added to the video data, means for specifying the place name of the shooting point based on the shooting position information and the place name information;
Means for retrieving an information resource containing the specified place name from an external device storing a plurality of information resources, and obtaining the number of information resources containing the place name;
The moving image according to any one of claims 1 to 5, wherein the moving image data is highly evaluated as the number of information resources including the place name related to the shooting position information of the moving image data increases. Search device. The evaluation means includes
Means for retrieving an information resource having a feature amount of a frame image included in moving image data from an external device storing a plurality of information resources, and acquiring the number of information resources having the feature amount;
7. The moving image data is highly evaluated as the number of information resources having the feature amount of the frame image included in the moving image data increases. Video search device. The evaluation means includes
Performs image quality correction that changes the tone value of the pixels that make up the frame image included in the video data, and the tone value of the pixels that make up the corrected frame image, and the tone values of the pixels that make up the frame image before correction The moving image search according to any one of claims 1 to 7, further comprising means for comparing values, wherein the moving image data is evaluated lower as the difference between the gradation values is larger. apparatus. The evaluation means includes
9. The block noise edge is detected from a frame image included in the moving image data, and the moving image data is evaluated to be lower as the block noise edge is larger. The moving image search device according to one. The evaluation means includes
By comparing subjects included at the ends of a plurality of frame images included in the moving image data, a means for calculating the vibration amount of the subject is provided. The larger the vibration amount, the lower the evaluation of the moving image data. The moving image search device according to claim 1, wherein the moving image search device is a video search device. Receiving a search keyword for searching for specific moving image data from a storage device that stores a plurality of moving image data composed of a plurality of frame images, and moving image data based on the received search keyword and metadata associated with each moving image data In the video search method to search,
For each of a plurality of frame images included in the moving image data, generating metadata of the moving image data based on the frame image;
Evaluating the video data based on frame images included in the video data, or metadata associated with the video data;
Searching for video data that includes the search keyword in the metadata;
When there is a plurality of video data obtained by searching, the method includes a step of generating a list in which thumbnail images of each of the plurality of video data or metadata of each video data are arranged based on an evaluation result of each video data. A video search method characterized by In a computer program for causing a computer to search for moving image data based on a search keyword and metadata associated with each moving image data from a storage device that stores a plurality of moving image data consisting of a plurality of frame images.
For each of a plurality of frame images included in the moving image data, generating metadata of the moving image data based on the frame image;
Evaluating the video data based on frame images included in the video data, or metadata associated with the video data;
Searching for video data that includes the search keyword in the metadata;
A plurality of video data obtained by the search, generating a list in which thumbnail images of each of the plurality of video data or metadata of each video data are arranged based on the evaluation result of each video data; A computer program that is executed.

Images