JP5380643B2 - Video data verification system - Google Patents

Description

  The present invention creates a collation system that collates browsing video data that can be browsed on a site on a network, and a video database in which a plurality of video data used for collating browsing video data is registered as information. The present invention relates to a method and a registration system and program for registering moving image data in a moving image database.

  In recent years, copyright infringement on the Internet has been regarded as a problem with the digitization of information such as novels, music, movies and photographs and the spread of the Internet. It is possible to reproduce information accurately without losing quality by digitizing information, and it is easy for secondary and tertiary use such as editing and adaptation of information, and public transmission via the Internet. This is because it is easy to use other people's works. In particular, not only organizations such as companies but also individuals are in these environments, making it difficult to prevent and manage copyright infringement.

  Recently, a video sharing service for sharing a video on the Internet has been provided. Copyright infringement has become a major problem in this video sharing service. The video sharing service is a service in which video data uploaded by a certain user is shared so that it can be downloaded to other users so that the video can be disclosed and viewed. This video sharing service posts tens of thousands of video data per day, and the video data currently contains many illegal ones that infringe copyrights. . For example, movies, TV programs, live images, promotional videos, etc. are uploaded without permission from the copyright holder.

  Conventionally, copyright management in video sharing services has been entrusted to the ethics of each user by simply stating that posting of video data that infringes copyright is prohibited in terms of use, etc., and a special censorship system has been established. It wasn't. Even if illegal video data was uploaded, it was only deleted what was pointed out as illegal video data by a third party. Copyright holders downloaded and played uploaded video data, played back, searched for illegal video data by visual inspection, and reported them, but confirmed all the video data that increased by tens of thousands every day That was not realistic. Moreover, even if illegal video data is deleted by reporting, the user often posts it again, and conventional measures have not been effective.

  By the way, as a conventional moving image search technique, most of the techniques are to set a keyword in a moving image, and a searcher inputs a keyword to search for a moving image that matches the keyword. However, in the video search method using keywords, there is a problem that an appropriate video cannot be searched if the set keyword is inappropriate for the video. Another problem is that there are individual differences in keyword settings, and that the keywords do not necessarily match due to differences in individual subjectivity and expressions. If the keywords match, Even if it is a video, there is a problem in that it is searched.

  With respect to a moving image search method using such keywords, Patent Document 1 discloses that a frame image in which a moving image scene changes is extracted and used as a representative frame image from the whole or a part of the representative frame image, the area of a specific color of the image, the representative frame Calculates feature quantities such as the length of time, integral value of sound intensity, time length of sound over a certain intensity, assigns a code corresponding to the feature quantity, and assigns a name to the representative frame by code or code string In addition, a moving image search method is disclosed in which a moving image name is created for a moving image using a frame name sequence corresponding to a representative frame sequence, and collation is performed using the moving image name.

Japanese Patent Laid-Open No. 7-114567

  However, in the current video sharing service, illegal video data filtering has not been sufficiently realized, and an effective video search method is required. The moving image search method of Patent Document 1 described above is intended to search for a matching video scene in a moving image, and is not intended to search illegal moving image data in a moving image sharing service.

  In order to search for illegal video data in the video sharing service, it is necessary to check whether or not the video uploaded to the video sharing service is illegal. For many videos uploaded to the video sharing service, Therefore, it is necessary to check whether each is illegal. The uploaded movie may be the entire original movie, or only a part that is arbitrarily divided may be uploaded. Also, similar videos may be uploaded multiple times, requiring continuous monitoring.

  Patent Document 1 does not disclose a technique for searching for illegal moving image data in such a moving image sharing service. In addition, in the moving image search method of Patent Document 1, in order to collate the moving image with the name string (moving image name) of the frame corresponding to the representative frame sequence, the uploaded moving image is downloaded and the moving image of the uploaded moving image is recorded. The name needs to be calculated and verified. For this reason, since the uploaded video must be downloaded each time, it takes time to search, and download time becomes one of the bottlenecks for speeding up the download. In addition, there is a possibility that the collation cannot be performed accurately depending on finely divided moving image data and how to divide moving images. Furthermore, it is necessary to calculate the name of the moving image with respect to the moving image data updated at any time, and the processing amount of the processing device becomes enormous. In addition, with regard to the feature quantity that is the basis for calculating the name of a moving image, even if it is a different image, if the extracted feature quantity is the same or similar, it will be searched and the authentication accuracy will be reduced. Exists.

  Accordingly, an object of the present invention is to provide a verification system that enables effective censoring of browsing moving image data that can be browsed on a site on a network. It is another object of the present invention to provide a method of creating a moving image database in which moving image data registered as information is increased in the number of registered moving image data per unit capacity and can be easily searched. Furthermore, an object of the present invention is to provide a registration system and a program that improve security against information leakage by a simple method when registering moving image data in a moving image database. In addition, an object of the present invention is to provide copyright management and a new business model using these collation system, moving image database creation method, registration system and program. Note that these purposes are representatively related to each other, and are related or overlapped with each other.

  A collation system according to the present invention is a collation system for collating browsing video data that can be browsed on a site on a network, wherein a plurality of registered video data are registered as information, and the browsing video data And collating means for collating the registered moving image data registered in the moving image database, the browsing moving image data includes a still image displayed as a sample, and the moving image database includes each registered moving image As data, there is a recording medium on which a registration image group consisting of a plurality of registration images generated from a plurality of frame images of a playback video that has played back the registration video data is recorded, and the verification means includes the browsing video A search image generated from a still image for data sample is compared with a registration image recorded on the recording medium of the moving image database. And wherein the Rukoto.

  Furthermore, the collation system of the present invention preferably has a function of extracting a sample still image arbitrarily or randomly from the browsing moving image data. In addition, the registration image group of each of the registered moving image data has a correlation calculation result between one of the registration images included in the registration image group and the other one smaller than a preset threshold value. Is preferred. Furthermore, it is preferable that at least a binary imaging process is performed while the registration image is generated from a frame image. Here, while the search image is generated from a still image for sample of the browsing moving image data, at least binary imaging processing is performed, and the collating means includes the search image and the registration image. Are preferably collated with each other.

  Furthermore, in the collation system of the present invention, image feature information of a frame image corresponding to each registration image is recorded in the moving image database, and the collating means is a still image for a sample of the browsing moving image data. Image feature information may be calculated and collated with the image feature information recorded in the moving image database. In the verification system of the present invention, at least one keyword is set in the registered video data, and at least one keyword data is added to the browsing video data, and the verification means includes the browsing video data The keyword acquired from the keyword data added to the keyword may be collated with the keyword set in the registered moving image data.

  By using the collation system of the present invention, it is possible to collate with the registered moving image data registered in the moving image database for the browsing moving image data that can be viewed on the site on the network. Manage registered video data, protect copyright, or permit permission, and can be censored within the range registered in the video database, and infringing acts caused by uploading illegal video data, Can avoid exercising rights from registrants. Other effects will be described in the following embodiments.

Schematic configuration diagram showing a verification system of the present invention Block diagram schematically showing the flow of data (A) is a figure which shows the data structure of a typical moving image database, (B)-(E) is a figure which shows the data structure of typical registered moving image data, respectively. Movie database creation process flowchart Verification process flowchart Configuration diagram of verification server (A)-(C) is a figure which shows the error rate in each moving image database. The figure which shows the error rate in the collation system which uses the still picture for sample as the picture for search

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following examples. FIG. 1 is a schematic configuration diagram showing a collation system and related system of the present invention, and FIG. 2 is a block diagram schematically showing a data flow. In FIG. 1, a network 11 to which a client group 10 is connected is formed. On the network 11, a site 12 that provides video data to the client group 10 so as to be viewable is established. A collation server 1 that collates the browse image data 13 to be viewed is provided. First, the overall verification system and its business model will be schematically described, and then the specific configuration of the verification system and the like will be described in detail.

  The verification server 1 is provided with a video database 2 in which a plurality of registered video data is registered as information, and a verification unit 3 for verifying the browsing video data and the registered video data registered in the video database 2. The registered moving image data is provided from the registrants 5 and 6 and registered in the moving image database 2.

  The registrants 5 and 6 are, for example, the operator of the site 12, the copyright owner or the producer of the registered moving image data. Here, as shown in FIG. 2, registration data such as the title, viewing conditions, keywords, and image feature information can be registered at the same time for the registered moving image data. The registration data may be arbitrarily registered in the collation server or site instead of the registrants 5 and 6.

  The client group 10 includes a plurality of clients that use the site 12 and is connected to the network 11. The client group 10 can issue a video data browsing request to the server of the site 12. For example, the client group 10 includes users of video sharing sites, paid content members, and the like, and specific terminal devices include personal computers, portable information terminals, and mobile phones.

  The network 11 is a telecommunications circuit network that connects client terminal devices, and includes the Internet, a WAN (Wide Area Network), a LAN (Local Area Network), and the like.

  The site 12 is provided with contents for providing browsing moving image data to the client group 10, and in FIG. 1, contents A, B, and C having different browsing conditions are prepared. Examples of A, B, and C content include completely free content, partially viewable content, and paid content. The site 12 can also set keywords for the posted moving image data when posting. The site 12 may be configured to display at least one still image (generally referred to as a thumbnail image) as a sample for the posted browsing video data.

  The browsing moving image data 13 is preferably in a state that can be browsed at the site 12 after being verified by the verification server 1, but directly to the site 12 without going through the verification server 1 as shown in FIG. 2. It may be configured to be uploaded. When the browsing video data 13 is directly uploaded to the site 12, the verification server 1 may acquire the browsing video data from the site 12 and verify it with the registered video data. In FIG. 1, the browsing moving image data 13 is uploaded from the client group 10 to the server of the site 12 through the network 11. The uploaded browsing moving image data 15 is downloaded to the terminal device of the client through the network 11 based on a browsing request from another client. The browsing moving image data 13 may be in a state that can be browsed by other than the client, or may be transmitted to the site 12 by means other than the network. Further, only a link may be pasted on the site 12, and the browsing moving image data 13 itself may be stored in the terminal device of the moving image data provider. Here, when there is a browsing request from another client via the link pasted on the site 12, it is directly downloaded from the terminal device of the provider to the terminal device of the other client.

  In the collation system of the present invention, in order to perform collation processing at a higher speed, it is preferable that the browsing moving image data has a still image displayed as a sample. The still image displayed as a sample is a frame image or an image obtained by reducing the frame image in the playback moving image of the browsing moving image data to reduce the capacity, and is an image displayed at least on the verification server. Typically, this is the first frame image of the reproduced moving image, but the present invention is not limited to this. Further, the number of still images displayed for sample use may be one or more. It is not only displayed as a thumbnail image of the browsing video data posted on the site 12, but it is not displayed on the site 12, but is downloaded as a sample of browsing video data when there is a browsing request from the client. And still images based on arbitrarily or randomly extracted frame images that are not displayed on the client or the network as will be described later. As shown in FIG. 1, when the browsing moving image data 13 is uploaded from the client group 10 to the site 12 via the verification server 1, from the information processing device of the client group 10 to the information processing device of the verification server 1. The browsing video data 13 is a still image displayed as a sample if the frame image in the playback video of the browsing video data 13 or an image with a reduced capacity is downloaded as an interface between Will have.

  Moreover, it is preferable to add at least one keyword data for facilitating collation and search to the browsing moving image data 13 (see FIG. 2). The keyword data may be arbitrarily selected by the contributor of the browsing moving image data 13, but a keyword specifying the field of the moving image data may be set in advance. Moreover, it is good also as a structure which cannot be contributed to the site 12 unless a keyword is input. This keyword data may be used when a client searches for a moving image in the site 12 or may be used in the moving image database 2 of the matching server 1 to facilitate matching. For example, when registering registered moving image data in the moving image database 2, keywords are recorded as registered data and collated from registered moving image data corresponding to the keyword data, or the moving image database 2 stores registered moving image data by field or keyword. You may register and collate from the area | region applicable to keyword data.

  It is preferable to add identification data to the verified browsing moving image data 14 verified by the verification server 1. The identification data may simply give a history of verification in the verification server 1, or may identify whether or not the browsing video data 13 is registered in the video database 2, and the browsing video data 13 is a video. If it is registered in the database 2, a part of the browsing conditions and registration data may be included.

  For example, when the browsing moving image data 14 to which identification data “verified” is given in the verification server 1 is downloaded to another client and posted again from another client, the verification server 1 recognizes the identification data “verified”. Thus, it is possible to determine whether or not the browsing moving image data 14 to which the identification data is assigned needs to be verified. Further, if the information on whether or not it is registered in the moving image database 2 as identification data is included, it is possible to change the viewing conditions depending on the presence or absence of registration or to manage moving image data on the network. Furthermore, if information such as the title of the moving image and browsing conditions is included in the identification data, finer browsing conditions can be set and managed.

  As browsing conditions, for example, the posted browsing video data 13 is deleted without browsing, paid content, charged according to the number of browsing times, partially viewable, invalid content, etc. Is mentioned.

  In FIG. 1, the browsing moving image data 13 is collated with the moving image database 2 of the collation server 1, and identification data A to C is given according to the collation result, and the site is based on the identification data of A to C. 12 is uploaded to each content of A, B, and C prepared.

  In addition, the structure provided only to what is registered into the moving image database 2 among browsing moving image data may be sufficient as identification data. In addition, when the browsing moving image data is processed according to the matching result in the matching server 1, the identification data may not be added to the browsing moving image data. For example, if the registered video data registered in the video database 2 is not viewed or uploaded at all, the browsed video data that matches the registered video data registered in the video database is deleted in the verification server 1. The browsing moving image data uploaded to the site 12 may not be given identification data. In other words, uploading to the site 12 itself means that the browsing moving image data is not registered in the moving image database. In addition, if only the investigation and the number of postings of moving image data are to be managed, such information may be provided to the registrant in the matching server 1, and the identification data is not added to the browsing moving image data 13. Good.

  Further, when the browsing moving image data 13 is registered in the moving image database 2, the verification server 1 or the site 12 may provide information regarding the moving image data to the registrant who registered the moving image data in the moving image database 2. (See FIG. 2). Examples of information relating to moving image data include the number of postings of registered moving image data, the number of times of browsing (downloading), client information that posted moving image data, and the like.

  In addition, the verification server 1 issues a warning message notifying the client that posted the moving image data registered in the moving image database 2 as the browsing moving image data 13 that posting the moving image data infringes the copyright. You may transmit (refer FIG. 2). By transmitting the warning message in this way, it is possible to raise the copyright awareness of the client group 10.

  By using the collation system as described above, the registrants 5 and 6 register the moving image data that they want to manage as registered moving image data in the moving image database 2, thereby managing the registered moving image data on the network 11 and the copyright. Rights protection or permission can be implemented. In addition, the site 12 can censor the browsed video data within the range registered in the video database 2, and infringement acts caused by uploading illegal video data, as well as exercising rights from registrants. It can be avoided.

  That is, the collation server 1 can provide the registrants 5 and 6 with a service for registering the moving image data to be managed in the moving image database 2 and can also provide a service for acting as part of the management of the registered moving image data. Further, the collation server 1 can provide a browsing video data censoring service within the range registered in the video database 2 to the site 12, and can also provide a service acting as part of the management of the browsing video data. In addition, the registrants 5 and 6 and the site 12 themselves can provide the verification server 1.

  Next, the verification server 1 will be described. The collation server 1 is at least one of collation means 3 that collates a recording medium in which a moving image database 2 in which a plurality of registered moving image data is registered as information is recorded, browsing moving image data, and registered moving image data registered in the moving image database 2. An information processing apparatus that functions as a unit is provided.

  One of the features of the collation system by the collation server 1 of the present invention is that collation between the final browsing moving image data and registered moving image data is performed by comparing images (including digitized image data). There is in point to be called. That is, the collation system of the present invention does not exclude the collation by the conventional keyword or feature amount, but the search image generated from the reproduction video of the registered video data and the reproduction video of the browsing video data. It is indispensable to check directly with the image. Since the images are directly matched, the matching accuracy can be increased. In addition, by registering a registration image group consisting of registration images corresponding to the main frame images of the playback movie of the registered movie data as registered movie data, the search image can be displayed in any scene of the registered movie data. Even if there is, it can be surely searched.

  For this reason, even when collation is performed using only search images generated from still images for browsing video data samples, browsing video data and registration are performed by collating with the registration image group of registered video data. It is possible to determine whether the moving image data matches or is similar. Since the sample still image itself has a very small capacity compared to downloading the browsing moving image data, the download time can be shortened and high-speed collation can be performed. And it can be an effective collation means for a large number of browsing video data uploaded to the site. For example, even if the browsing video data has a capacity of several MB (megabytes) or more, the thumbnail image itself is image data of about 128 pixels (horizontal) × 96 pixels (vertical). It can be overwhelmingly shortened.

  In this way, against a collation system that uses thumbnail images, a malicious client may interfere with the collation by performing some image processing on the frame images that become thumbnail images. For this reason, it is preferable that the verification system further has a function of extracting a sample still image arbitrarily or randomly from the browsing moving image data. In order to realize such a function, an information processing device of a client to be uploaded, an information processing device of a collation server, an information processing device of a site server, etc. An arbitrary frame image or a random frame image may be extracted from the image, and a program that functions as an image generation unit that generates a sample still image from the extracted frame image may be used. Here, it is preferable to generate a plurality of sample still images from a plurality of frame images separated in time. Further, in order to increase the reliability of the collation system, it is preferable that the extraction position of the frame image is a program that cannot be operated by the client, and it is not possible to view which frame image is extracted on the client or the network. It is also effective to do.

  In the moving image database 2, a registration image group including a plurality of registration images generated from a plurality of frame images of a reproduced moving image obtained by reproducing the registered moving image data is recorded as each registered moving image data. That is, as shown in FIG. 3A, a plurality of registration image groups 16 corresponding to the number of registered moving image data are recorded in the moving image database 2, and each of the registration image groups 16 includes a plurality of registration image groups 16. The registration image 17 is included.

  Registering all the frame images of the playback video as multiple frame images will record the largest amount of information in the registered video data. As a result, the calculation time becomes longer and a huge recording capacity is required. It becomes. The video currently used in Japan is 29.97 fps (frames / second) in the NTSC standard for TV broadcasting and 24 fps in movies, so it consists of still images of about 30 frames and 24 frames per second, respectively. ing. Also, since each frame image has pattern information composed of an array of pixels corresponding to the resolution and color information for each pixel, if all frame images are registered as a registration image group, the total image Data capacity becomes extremely large. For example, a full-color (24-bit) image with a resolution of 600 × 480 has a data capacity of 27 MB if simply calculated, and a 90-minute movie (24 fps) has 129,600 frame images. Even if it calculates with one moving image data, it is not a realistic thing at all.

  For this reason, it is very preferable to generate a registration image by performing at least binary imaging processing on the frame image of the reproduced moving image. The binary imaging process is a process for converting each pixel into an image displayed in two colors of white and black (0 and 1). In general moving image data, each pixel has at least a gray scale gradation. Since it has information (8 bits), in many cases color information (24 bits), it is possible to remarkably reduce the data capacity of the registration image by performing binary image processing. In addition, when a binary registration image that has been subjected to binary imaging processing and a binary search image that has also been subjected to binary imaging processing are subjected to correlation operation and collated, full-color images or gray-scale images can be compared with each other. As a matter of course, the calculation speed becomes faster than when the correlation calculation is performed, but the collation accuracy may be improved, and the binary imaging process is an extremely useful process. In addition, since the amount of calculation of the binary image-processed image is light, it is possible to produce a dedicated arithmetic device as hardware using an integrated circuit such as an LSI.

  There are two types of binary imaging processing methods: a fixed threshold method and a region designation method. The former is a method of determining a binarization threshold value with a certain grayscale gradation value, and the latter is a method in which the binary ratio of the entire image is determined first, and the binary value is determined from the density value histogram. This is a method for determining the threshold value. Assuming that the binarization threshold is T, the element of each pixel of the original image is f (i, j), and the element of each pixel of the binary image is g (i, j), the following equation (1) Can be defined.

  If the frame image is color, first, the color information of the color may be removed and changed to a gray scale with only gradation information, and binarized. Further, other image processing may be performed on the frame image or the binary image. For example, image processing for changing the resolution, edge enhancement processing, or the like may be performed before or after the binary imaging processing.

  The binary imaging process is basically an irreversible conversion that only deletes information, unlike the data format conversion process, compression process, and encryption process. That is, the data format conversion processing, compression processing, and encryption processing have information that can be inversely converted in their own or another file in order to restore the original frame image. However, the binary image processing simply discards the color information and gradation information that the original frame image had, and the binary image itself cannot be restored to the original frame image.

  The inability to restore the original frame image seems to be a drawback at first glance, but it is actually effective in preventing information leakage of the original moving image data. In other words, if the original video data from the registrant is provided to the verification server as registered video data for registration in the video database, due to a distribution accident, an accident during transmission of the network, or human information leakage, There is a risk that the original video data will be leaked. For example, the loss when an open movie or an unpublished live video leaks is serious. However, in the registrant's information processing apparatus, if at least binary image processing is performed on the frame image of the registered moving image data and the binary image is provided to the verification server, even if the binary image leaks, Since the video itself cannot be restored to the original video data, the leakage of the original video data can be avoided. A registration system and program for registering moving image data in the moving image database using this effect will be described later.

  In order to reduce the number of registration images, it is preferable to select and register to some extent without excluding similar images and registering them. For this purpose, the correlation calculation is performed between the frame images of the registered moving image data or images (including binary images) obtained by image processing of the frame images, and the result is larger than a preset threshold value. May exclude any one of the images, and if the result is smaller than the threshold value, at least one of the images may be registered. The threshold value is a criterion for the degree of similarity to be selected. If the threshold value is high, the matching accuracy generally improves, but the number of images for registration also increases, and if the threshold value is low, Since the collation accuracy is lowered but the number of registration images is also reduced, it is preferable to obtain an optimum threshold value by an experiment or the like in advance. Further, if the registrant can change the threshold value, the threshold value can be increased for important moving image data, and the threshold value can be set lower for those with low importance. Note that one of the registration images need not be subjected to correlation calculation with all other registration images, and may be smaller than a threshold value as a result of correlation calculation with at least one other registration image.

  As the correlation calculation, for example, the degree of similarity can be calculated by obtaining the inner product, the Euclidean distance, or the correlation coefficient between the registration images. When the elements of the N × N pixel registration images F and G are expressed as a one-dimensional column vector f and column vector g, the inner product can be calculated by Equation (2), and the Euclidean distance can be calculated by Equation (3). The correlation coefficient can be calculated by Equation (4). The calculation of the similarity is not limited to these methods.

  3B to 3E schematically show the data structure of each registered moving image data described above. In FIG. 3B, all the frame images of the reproduced moving image are directly used as a registration image group 16, and each registration image 17 is each frame image and has gradation information and color information. FIG. 3C is obtained by performing binary imaging processing on FIG. 3B, and each registration image 17 is a binary image, so that gradation information and color information are removed. The data capacity of the registration image group 16 can be reduced. FIG. 3 (D) is a screen obtained by performing sorting on FIG. 3 (B). Each registration image 17 is a selected frame image, and a registration image group 16 corresponding to the capacity of the removed frame image. Data capacity can be reduced. FIG. 3E shows the result of performing both binary imaging processing and sorting on FIG. 3B. The binary image processing may be performed first, and then the binary image may be selected (FIG. 3 (B) → (C) → (E)). Processing may be performed (FIG. 3 (B) → (D) → (E)). In FIG. 3E, each registration image 17 is a selected binary image, and the data capacity can be significantly reduced as compared with FIG.

  A method of creating such a moving image database includes a reproduction step of reproducing one of registered moving image data to obtain a reproduced moving image, and a plurality of registrations by performing at least binary image processing on a plurality of frame images of the reproduced moving image. A registration image included in the registration image group between a generation step of generating a registration image, a recording step of recording a registration image group consisting of a plurality of registration images on a recording medium, and a reproduction step and a recording step It suffices to have a sorting step for sorting so that the result of the correlation calculation between one and the other is smaller than a preset threshold value. As described above, any of the generation process and the selection process including the binary imaging process may be performed first, and other image processing and processes are performed between or before and after those processes. Also good. However, in order to increase the calculation speed and accuracy of the correlation calculation process in the selection process, it is preferable to perform the correlation calculation in the selection process after performing the binary imaging process.

FIG. 4 is an example of a flowchart of the moving image database creation process. First, the registered moving image data is acquired and the registered moving image data is reproduced (S41: reproduction step). All frame image groups M _i (i = 1, 2, 3,... N, N are the number of frames) are generated from the reproduced moving image, and binary image processing is performed on all frame image groups M _i to obtain binary image groups Convert to B _i (i = 1, 2, 3,... N, N is the number of frames) (S42 to S43: generation step). Then, as the reference image B _p as a reference of comparison by substituting B _i (initial value i = 1), the reference image B _p and compared the binary image _{B i (i = i + 1} ) and by correlation calculation The correlation value V is acquired (S44 to S46: selection process). If the correlation value V is equal to or greater than the threshold value (No in S47: selection step), it is determined that the reference image B _p and the binary image B _i are similar, and the binary image B _i is registered in the moving image database. The correlation calculation is performed on the reference image B _p and the next binary image B _i (i = i + 1) (S45 to S46). On the other hand, if the correlation value V is less than the threshold value (S47 of Yes: selecting step), the reference image B _p and the binary image B _i is determined to be dissimilar, the reference image B _p (initial value The binary image B ₁ ) is registered in the moving image database (S48: recording process), and the dissimilar binary image B _i is set as the next reference image B _p and the next binary image B _i (i = i + 1) Correlation is performed (No in S49, S44 to S46). In S49, if the binary image B _i is the binary image B _N of the last frame, the frame image B _N is registered and the process ends (S50). Finally, a plurality of reference images registered as dissimilar are recorded as a registration image group B _m (m = 1, 2, 3,... M, M is the number of registration images).

  In the algorithm of the flowchart shown in FIG. 4, the first reference image and the next reference image are subjected to correlation calculation, but the other reference images are not subjected to correlation calculation, and similar images are obtained. There is a possibility that it is registered. In this regard, all other binary images are sequentially correlated with the first reference image, and binary images having a correlation value greater than the threshold value are excluded without being registered in the moving image database. If one of the binary images having a large value is used as the next reference image and sequentially correlated with the other binary images, the number of registration images can be further reduced.

  Furthermore, a registration data recording step of recording registration data such as a title corresponding to the registration image group, a browsing condition, a keyword, and image feature information about a frame image corresponding to the registration image may be included. As image feature information, if the registration image is a binary image or grayscale, and there is a feature in the distribution and area of a specific color in the original frame image, these are recorded as information and searched. Image feature information can also be calculated for the frame image corresponding to the image for use, and the image feature information can be used as a secondary collation.

  The production of the moving image database can be performed only by the verification server, or a part of the process may be performed by the information processing apparatus of the registrant. It is preferable to perform the reproduction process for obtaining the reproduction moving image and the generation process for performing the binary image processing on the frame image by the information processing apparatus of the registrant in order to prevent the information leakage described above. The registration image and the transmission image generated by the registrant information processing apparatus may be transmitted to the verification server through the network, or may be delivered to the verification server offline.

  An example of a registration system using a network will be described. The registration system includes a network, an information processing apparatus connected to the network, and a moving image database server. Then, the information processing apparatus reproduces one of the registered moving image data to acquire a reproduced moving image, and performs at least binary imaging processing on the plurality of frame images of the reproduced moving image to generate a plurality of transmission images. As a transmission image, it is sufficient that at least binary imaging processing is performed on a frame image, and other image processing is also performed, and the state is recorded as a registration image in a moving image database server. Alternatively, it may be recorded after further image processing or the like is performed in the moving image database server. Furthermore, the information processing apparatus transmits a plurality of images for transmission to the server of the moving image database through the network. Since a plurality of images for transmission are converted into binary images by binary image processing, even if they are leaked on the network, they are in an irreversible state for the reproduced moving image, and information leakage of the reproduced moving image can be prevented. . The moving image database server generates a plurality of registration images from the received plurality of transmission images, and records a registration image group including the plurality of registration images on a recording medium. If the received plurality of transmission images can be used as registration images as they are, the moving image database server records the plurality of transmission images as they are as a plurality of registration images on a recording medium.

  Further, the information processing apparatus performs correlation calculation on images based on each frame image of the playback moving image, and excludes one image from the transmission image when the result of the correlation calculation is larger than a preset threshold value. May be. This is performed in the information processing apparatus connected via the network instead of the moving image database server in the selection process, and the total data capacity of the images for transmission can be reduced. The image based on the frame image includes the frame image itself, an image obtained by performing image processing other than the binary imaging process on the frame image, and an image obtained by performing the binary imaging process.

  In order to use such a registration system, the information processing apparatus reproduces registered moving image data to obtain a reproduced moving image, and generates a transmission image by performing at least binary image processing on a plurality of frame images of the reproduced moving image. And a program for causing the image processing unit to function as a transmission unit that transmits a plurality of transmission images to a server of the moving image database through the network may be installed in the information processing apparatus. Further, the information processing apparatus performs correlation calculation between images based on each frame image of the reproduction moving image, and when the result of the correlation calculation is larger than a preset threshold value, as a selection unit that excludes one image May function. Here, the information processing apparatus may function as a threshold value input means so that the registrant can select the threshold value.

  By the way, as a recording medium for the moving image database, a large-capacity hard disk, a holographic recording medium or the like can be used. When a large-capacity hard disk is used for the moving image database and images are collated by arithmetic processing, an information processing apparatus can be used as a collating means. On the other hand, when a holographic recording medium is used, an optical recording unit and a verification unit are required. The collation system, moving image database, registration system and the like described above can be used in any collation means.

  Next, the operation at the time of verification in the verification unit will be described. As already described, in the collation system of the present invention, it is preferable to collate using a search image generated from a still image for sample browsing video data. FIG. 5 shows a case in which, in a moving image database in which a registration image group subjected to binary image processing is recorded, the information processing apparatus uses a search image generated from a still image for browsing moving image data to perform collation. It is a flowchart of a collation process.

First, the information processing apparatus of the verification server acquires a still image for sample browsing video data (S51). Next, the still image is converted into a binary image and converted into a binary search image S (S52). Then, the binary search image S and each registration image in a certain registration image group B _m (m = 1, 2, 3,... M, M is the number of registration images) recorded in the moving image database are sequentially added. The correlation calculation is performed to obtain the maximum correlation value V _max (S53). By comparing the verification threshold maximum correlation value V _max and video data, if the maximum correlation value V _max is greater than the matching threshold (S54 of Yes: verification step) is browsable moving image data registration images It is recognized that it is registered moving image data of a group (S55). If the maximum correlation value V _max is equal to or smaller than the verification threshold value (No in S54: verification process), it is recognized that the browsing moving image data is not the registered moving image data of the registration image group, and the next registration image group. It is determined whether or not there is (S56). When the next registration image group exists (No in S56), the correlation between the search image S and each registration image in another registration image group is sequentially calculated again to obtain the maximum correlation value V _max and collate. Compare with the threshold value (S53-S54). When the next registration image group does not exist (Yes in S56), it is recognized that the browsing moving image data is not registered in the moving image database (S57).

  In the process of converting the still image into a binary image and converting it into a binary search image S (S52), further image processing can be performed. For example, image processing for changing the resolution, edge enhancement processing, or the like may be performed before or after the binary imaging processing. Further, when selecting a registration image group to be collated with the search image S, keyword data of browsing moving image data, image feature information of a still image for a sample, and the like are acquired, and the keyword data is referred to by referring to the registration data. May be preferentially selected, or a registration image group including registration images having the same image feature information may be preferentially selected.

  Furthermore, the information processing apparatus of the verification server may give the verification result as identification data for the browsing moving image data for which verification has been completed and upload it to the site.

  Hereinafter, the verification server 1 using a holographic recording medium will be briefly described. FIG. 6 shows a specific configuration of the verification server 1 using a holographic recording medium, and includes a holographic recording medium 20, an information processing device 30, and an optical system 40. The collation server 1 is provided with an image database 2 and a collation means 3. The holographic recording medium 20 constitutes a part of the image database 2, and the information processing device 30 and the optical system 40 are collation means 3. Configure.

  In FIG. 6, a holographic recording medium 20 is a reflection type recording medium in which a thick hologram recording layer 21 made of a photosensitive material is sandwiched between a surface protective layer 22 and a reflection layer 23. Image data is recorded on the hologram recording layer 21 of the holographic recording medium 20 by interference fringes 24 of information light generated based on the registered image data and reference mark light generated based on the reference mark. If a disk-shaped holographic recording medium 20 is employed, collation can be performed while rotating, and therefore the collation speed can be increased. The holographic recording medium 20 preferably includes an address layer that specifies the position of the interference fringes 24. For example, as the address information, pits may be formed by the uneven shape provided on the surface of the reflective layer 23, and the reflective layer 23 may be used as the address layer. When a glass substrate is used as the surface protective layer 22 of the holographic recording medium 20, shrinkage due to a temperature change or the like can be suppressed. As the reflective layer 23, a metal material such as aluminum can be used.

  The information processing apparatus 30 is connected to a network and performs various types of information processing executed by the verification system. For example, playback of browsing image data and creation of search images, recording and playback of registration data, addition of identification data to browsing video data, determination of collation order based on keyword data, transmission of information on video data, warning message For example, transmission.

  The optical system 40 can record interference fringes on the holographic recording medium 20 and can collate registered image data and browsing image data recorded on the holographic recording medium 20. The optical system 40 includes a hologram laser 41, a mirror 42, a spatial light modulator 43, a polarization beam splitter 44, a first relay lens 45, a mirror 46, a second relay lens 47, a beam splitter 48, and a quarter wavelength. It has a plate 49, an objective lens 50, an aperture 51, and a reference light detector 52, and further has an address laser 60, a beam splitter 61, a mirror 62, and an address light detector 63.

  The hologram laser 41 serves as a light source for information light and reference light when recording interference fringes, and serves as a light source for search light during collation. For example, a high-power laser with a short wavelength such as a blue laser or a green laser. Is preferred. The spatial light modulator 43 has a plurality of pixels and can spatially modulate the light by changing the attribute of the light for each pixel. For example, the spatial light modulator 43 can be a liquid crystal display device or a DMD (Digital Micromirror). Device) can be used. In FIG. 6, a DMD is used as the spatial light modulator 32. The polarization beam splitter 44 transmits one of the orthogonal polarization directions and reflects the other, transmits the information light, the reference light, and the search light directed to the recording medium 20, and refers to the reference light reproduced by the recording medium. Reflected toward the photodetector 52. The first and second relay lenses 45 and 47 form an image displayed on the spatial light modulator 43 on the focal plane of the objective lens. The beam splitter 48 is for directing light from the addressing laser 60 toward the recording medium 20. The quarter-wave plate 49 converts linearly polarized light into circularly polarized light, and the linearly polarized light can be rotated 90 degrees by transmitting twice. The reference light is transmitted by the quarter-wave plate 49 through the polarization beam splitter 44 during irradiation and reflected by the polarization beam splitter 44 during reproduction. The objective lens 50 performs Fourier transform on the image displayed on the spatial light modulator 43 and irradiates the hologram recording layer 21 of the recording medium 20. The aperture 51 has an opening that blocks the search light reflected by the recording medium 20 and allows only the regenerated reference light to pass through the reference light detector 52. The reference light detector 52 detects the regenerated reference light, and particularly preferably detects the light intensity of the reference light. For example, a pin photodiode, a CMOS sensor, or a CCD sensor can be used.

  The light emitted from the hologram laser 41 is reflected by the mirror 42 and is spatially modulated by the spatial light modulator 43 on which the registration image and the search image are displayed to generate information light and search light. Information light and search light are transmitted through the polarization beam splitter 44, relayed by the first and second relay lenses 45 and 47, and reflected by the mirror 46 on the way, through the beam splitter 48 and the quarter wavelength plate 49. The light is transmitted, Fourier-transformed by the objective lens 50, and irradiated on the hologram recording layer 21 of the recording medium 20. The information light interferes with the reference light in the hologram recording layer 21 to record interference fringes. Further, if the search light has a high correlation value with the information light recorded on the interference fringe recorded on the hologram recording layer 21, it is diffracted by the interference fringe and reproduces the reference light. The reproduced reference light is reflected by the reflective layer 24, emitted from the recording medium 20, and in the direction opposite to that during irradiation, the objective lens 50, the quarter-wave plate 49, the beam splitter 48, the first and second The light passes through the relay lenses 45 and 47 and the mirror 46, is reflected by the polarization beam splitter 44, passes through the aperture 51, and enters the reference light detector 52.

  The address laser 60, the beam splitter 61, the mirror 62, and the address photodetector 63 are for specifying the irradiation position from the address layer when the recording medium 20 is provided with the address layer. The light emitted from the light beam is transmitted through the beam splitter 61, reflected by the mirror 61, further reflected by the beam splitter 48, transmitted through the quarter-wave plate 49, and is applied to the address layer of the recording medium 20 by the objective lens 50. Irradiated. Reflected light from the recording medium 20 passes through the optical system in reverse, is reflected by the beam splitter 61, and is detected by the address photodetector 63. As the addressing laser 60, a laser having a relatively long wavelength such as red light is preferably used.

  In the collation server 1 using such a holographic recording medium, when recording an image for registration in the moving image database, the image for registration and the image for reference light are displayed on the spatial light modulator 43 for registration. Information light and reference light spatially modulated by an image and an image for reference light are generated. The information light and the reference light are Fourier transformed by the objective lens 50 and irradiated onto the hologram recording layer 21 of the recording medium 20, whereby the Fourier transformed information light and the reference light interfere with each other in the hologram recording layer 21. Thus, the interference fringes corresponding to the registration image are recorded. When the browsing moving image data is collated, a search image is displayed on the spatial light modulator 43, and search light spatially modulated by the search image is generated. The search light is Fourier-transformed by the objective lens 50 and applied to the interference fringes of the hologram recording layer 21 of the recording medium 20. If the correlation value between the registration image of the information light forming the interference fringes and the search image is high, The search light is diffracted by the interference fringes to reproduce the reference light. That is, the correlation value can be obtained by detecting the intensity of the reproduced reference light. In addition, when the registration image or the search image is a binary image that has been subjected to the binary image processing, it is easy to correspond to each pixel of the spatial light modulator 43, which is also preferable in this respect.

  In the above description, a collinear apparatus that causes the information light and the reference light to interfere on the same axis is used, so that the collation can be performed at high speed. In the above description, the reflective holographic recording medium has been described. However, the present invention can also be realized by a transmissive type in which the reference mark light reproduced by the optical correlation calculation is transmitted and displayed. Further, instead of the collinear method, a two-beam interference method device that separates the optical path of the information light and the optical path of the reference mark light and intersects the recording medium at a certain angle can be realized. For example, the light from the light source 41 is split into two lights by a beam splitter, one light is modulated by a spatial light modulator to generate information light, the other light is deformed to generate reference mark light, The configuration may be such that irradiation is performed so as to intersect with the recording medium.

[Example 1]
As registration moving image data, 10 seconds, to prepare 30 present the video data of 30fps, respectively, and the entire frame image (300 frames) and image processing such as binary image processing to generate a binary image group B _i. The correlation value of autocorrelation (correlation calculation between the same images) is normalized as 100 (au: arbitrary unit), and the threshold values are set to 70 (au) and 60 (au). A moving image database was prepared by performing a selection process. When the threshold value is 70 (au), the number of registration images can be reduced to 2504 for 9000 binary images of 300 frames × 30, and the threshold value is 60 (au). u.), the number of registration images can be reduced to 1580 with respect to 9000 binary images of 300 frames × 30. The number of reductions for each moving image data is shown in Table 1 below.

  Furthermore, the error rate was determined for each of the moving image database in which all binary images were registered, the moving image database in which the threshold value 70 was selected, and the moving image database in which the threshold value 60 was selected. Each of FIGS. 7A to 7C is a diagram showing an error rate in each moving image database. The vertical axis is an error rate, and the horizontal axis is a threshold value related to a normalized correlation value. The error rate is composed of two error curves. One is a registered image rejection rate (FRR) when a registered image is mistakenly recognized as unregistered, and the other records a different image. It is the other image acceptance rate (FAR: False Acceptance Rate) when it is mistaken for the received image. The registered image rejection rate FRR and the other image acceptance rate FAR are each obtained from the correctness / error rate of the collation result when the threshold value is changed. FIGS. 7A to 7C are obtained by extracting one frame image from 30 registered moving image data, creating 30 search images, and checking each of the registered images in each moving image database. The registered image rejection rate FRR and the other image acceptance rate FAR when it is determined that the correlation value matches the registration image when the correlation value is larger than the threshold value.

  The value at which the registered image rejection rate FRR and the other image acceptance rate FAR intersect is a threshold value for which both the registered image rejection rate FRR and the other image acceptance rate FAR are both small. The error rate at that time is expressed as EER (Equal Error When the EER has a certain range, the range is called a threshold region. The smaller the EER, the better. If the EER is 0%, both the registered image rejection rate FRR and the other image acceptance rate FAR are 0%, and no error occurs theoretically. The width of the threshold region indicates the reliability of matching, and the wider the threshold area, the higher the reliability of matching. In order to make it possible to collate similar images, the threshold value may be set to a value smaller than the threshold value region to increase the other image acceptance rate FAR.

  FIG. 7A shows a case of a moving image database in which all binary images are registered, FIG. 7B shows a case of a moving image database in which a threshold value 70 is selected, and FIG. This is a case of a moving image database in which a threshold 60 selection process is performed. 7 (A) and 7 (B), the EER was 0%, and in FIG. 7 (C), an extremely low value of EER 3.3% could be obtained.

[Example 2]
Next, a collation accuracy evaluation experiment of a collation system using a search image generated from a still image as a sample of browsing moving image data was performed. As the moving image database, 30 pieces of moving image data of 10 seconds and 30 fps are prepared as registered moving image data, and all frame images (300 frames) are subjected to image processing such as binary image forming processing, etc., and 9000 pieces of image data are processed. A moving image database in which a binary image group _Bi was generated and all binary images were registered was used. The sample still image size used for the search is 128 × 96 pixels, and for each registered moving image data, a total of 30 sample still images are subjected to image processing such as binary image processing. Was used as a search image. FIG. 8 is a diagram showing the error rate in the verification accuracy evaluation experiment, where the vertical axis is the error rate and the horizontal axis is the threshold value related to the normalized correlation value. The error rate is the same as in FIG. As a result, it was proved that collation with high accuracy of EER 0% was possible.

DESCRIPTION OF SYMBOLS 1 Collation server 2 Image database 3 Collation means 5, 6 Registrant 10 Client group 11 Network 12 Site 13 Browse image data 16 Registration image group 17 Registration image

Claims (7)

A collation system for collating browsing video data that can be viewed on a site on a network,
A moving image database in which a plurality of registered moving image data is registered as information, and a matching means for comparing the browsing moving image data and the registered moving image data registered in the moving image database;
The browsing moving image data has a still image displayed as a sample,
The moving image database includes a recording medium in which a registration image group including a plurality of registration images generated from a plurality of frame images of a reproduction moving image obtained by reproducing the registered moving image data is recorded as each registered moving image data,
The collation system is characterized in that the collation system collates a search image generated from a sample still image of the browsing moving image data and a registration image recorded in a recording medium of the moving image database.   The collation system according to claim 1, wherein the collation system has a function of extracting a still image of a sample arbitrarily or randomly from the browsing moving image data.   The registration image group of each registration moving image data is characterized in that a result of correlation calculation between one of the registration images included in the registration image group and the other is smaller than a preset threshold value. The collation system according to claim 1 or 2.   The collation system according to any one of claims 1 to 3, wherein the registration image is subjected to at least binary imaging processing while being generated from a frame image. While the search image is generated from a sample still image of the browsing moving image data, at least binary imaging processing is performed,
The collation system according to claim 4, wherein the collation unit collates the search image and the registration image by performing a correlation operation. In the moving image database, image feature information of a frame image corresponding to each registration image is recorded,
The collation means calculates image feature information of a still image for sample of the browsing moving image data, and collates it with image feature information recorded in the moving image database. The verification system described in the section. At least one keyword is set in the registered video data,
At least one keyword data is added to the browsing video data,
The said collating means collates the keyword acquired from the keyword data added to the said browsing moving image data, and the keyword set to the said registration moving image data, The any one of Claim 1 thru | or 6 characterized by the above-mentioned. Collation system.

Images