JP2009245314A - Identification system of time-series data, and apparatus of giving personal meta information to moving image - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To accurately discriminate to which a registered person or the other object inputted time-series data belong to. <P>SOLUTION: A plurality of time-series data are inputted to a comparison score calculating means 11. The comparison score calculating means 11, an LLR measuring section 14 and a threshold identifying section 16 perform hypothesis testing for each object, and output an identification result to which object each time-series data belongs to. According to an identification result, a state holding buffer 13 holds a hypothesis state (adoption, rejection, and under testing) for each object. The comparison score calculating means 11 calculates a comparison score only for the hypothesis under testing. The identification result can be used for giving personal meta-information. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an apparatus for identifying time-series data and an apparatus for assigning person meta information to a moving image, and in particular, time-series data for identifying whether input time-series data belongs to a registered person or other object. The present invention relates to a discriminating apparatus and an apparatus for assigning person meta information to a moving image using the same.

  In order to search for an object such as a person appearing in a moving image, it is required to add meta information about the object to the moving image in units of frames. For this purpose, it is necessary to identify to which person or other object the time-series data such as images and sounds belong.

  Non-Patent Document 1 proposes a method of identifying to which of registered objects the input time-series data belongs by a sequential probability ratio test (SPRT).

  Non-Patent Document 2 particularly describes a method of applying SPRT to a speech time-series signal and identifying to whom the input speech time-series signal belongs among registered speakers. .

  SPRT is a sequential decision process that executes hypothesis tests sequentially along the time axis and stops the test when a conclusion is reached. Sequential probability ratio test in hypothesis test theory And the idea of decision theory are introduced.

  SPRT is characterized by being able to identify the trend of input time-series data by sequentially confirming the trend without setting the time point at which a conclusion should be drawn, and efficiently obtaining a reasonable conclusion with less cost and effort. Have.

FIG. 5 is a diagram for explaining the operation of a conventional SPRT. Assuming that the logarithmic likelihood ratio (LLR) of the hypotheses H ₁ and H ₂ that define objects 1 and 2 is LLR (H ₁ ) and LLR (H ₂ ), respectively, the LLR at time t ₀ (H ₁ ) and LLR (H ₂ ) are reset to zero, and hypothesis tests are sequentially executed along the time axis to which time series data is input. If neither LLR (H ₁ ) nor LLR (H ₂ ) exceeds the upper threshold A, neither hypothesis is adopted.

At time t _N1, but LLR (H ₂₎ is not greater than the upper threshold A, when the LLR (H ₁₎ exceeds the upper threshold A, to adopt the hypothesis H ₁ at time t _N1. As a result, the time-series data is identified as belonging to the object 1. Thereafter, LLR (H ₁ ) and LLR (H ₂ ) are reset to zero and the sequential determination process is repeated again.

Then, at time t _N2, but LLR (H ₁₎ is not greater than the upper threshold A, when the LLR (H ₂₎ exceeds the upper threshold A, to adopt a hypothesis H ₂ at time t _N2. This time, the time series data is identified as belonging to object 2. Thereafter, LLR (H ₁ ) and LLR (H ₂ ) are reset to zero and the sequential determination process is repeated.

Conversely, if LLR (H ₁ ) or LLR (H ₂ ) falls below the lower threshold B, the hypothesis H ₁ or H ₂ is rejected. Rejected hypotheses LLRs remain in reset until other hypotheses are adopted.
Kazunori Matsumoto, Kazuo Hashimoto, “Communication Traffic Monitoring System Combining Local Steady Binomial Distribution Model,” IEICE Transactions D, Vol.J84, No.6, pp.800-808, 2001. Takeshi Hamasaki, Hideki Noda, Eiji Kawaguchi, “Adaptive Speaker Identification Using Sequential Probability Ratio Test,” IEICE Technical Report. PRMU, Vol.99, No.710, pp. 9-14, Mar., 2000.

  However, in the conventional SPRT represented by Non-Patent Documents 1 and 2, no special consideration is given to the case where a plurality of time-series data are input simultaneously. When the conventional SPRT is simply applied when a plurality of time-series data are simultaneously input, there is a problem in that the identification results of the respective time-series data are inconsistent.

  For example, when time-series data of a plurality of face images that exist simultaneously in an image are input at the same time and a person is identified based on these time-series data, if a conventional SPRT is simply applied, a plurality of time-series data are the same May be mistaken for a person. This occurs when fluctuations are added to the time-series data, and this is caused by the fact that various fluctuations such as posture, illumination, facial expression, etc. are easily added to the face image.

  Hereinafter, the problem when the conventional SPRT is simply applied to a plurality of time-series data will be described with specific examples. FIG. 6 shows an example of an image of one frame of a TV program video at a time t of a certain scene. This image contains two face images, one face image (a) is front-facing and stable with little movement in the scene, and the other face image (b) is compared in the scene. Suppose that the target movement is intense and unstable. In addition, here, face images of three persons 1, 2, and 3 on the performer list attached to the TV program are registered in advance, and face images detected in units of frames from TV images ( Assume that a) and (b) identify one of the three registered persons 1, 2, and 3.

FIG. 7 is an explanatory diagram of problems in the conventional SPRT. Here, since three registered persons (m = 3) are scheduled, hypotheses H ₁ , H ₂ , and H ₃ that respectively define the persons ₁ , ₂ , and ₃ are prepared, and face images (a ) and (b), LLRs of hypotheses H ₁ , H ₂ , and H ₃ are calculated. FIGS. 7A and 7B show transitions of LLRs of hypotheses H ₁ , H ₂ , and H ₃ calculated for the face images (a) and (b), respectively, and the solid line indicates LLR (H ₁ ) and the broken line. Is LLR (H ₂ ), and the alternate long and short dash line is LLR (H ₃ ).

  When SPRT is performed based on the LLR along the time axis of FIG. 7, the adoption and rejection of hypotheses are decided in the following order (1) to (6).

(1) At time t = t ₁ , LLR (H ₃ ) for the face image (a) falls below the lower threshold B (LLR (H ₃ ) <B), so the hypothesis H ₃ is rejected. Time t = t ₁ and later, only the test of the hypothesis H _1, H ₂ is the face image (a) is subsequently performed.

(2) at time t = t _2, since the LLR for the face image (b) (H ₃₎ is below the lower threshold _{B (LLR (H 3) <} B), the hypothesis H ₃ is rejected. After time t = t ₂ , only the hypothesis H ₁ and H ₂ are continuously tested for the face image (a).

(3) Since LLR (H ₁ ) for the face image (a) exceeds the upper threshold A (LLR (H ₁ )> A) at time t = t ₃ , hypothesis H ₁ is selected. As a result, the face image (a) is identified as that of the person 1.

(4) Since face image (a) is identified as that of person 1 in (3) above, all other hypotheses about face image (a) are rejected. In this case, the hypothesis of H ₂ for the face image (a) is rejected.

(5) time t = t _4, since the LLR (H ₁₎ exceeds the upper threshold A for the face image _{(b) (LLR (H 1} )> A), the hypothesis H ₁ is adopted. As a result, the face image (b) is identified as that of the object 1.

(6) Since face image (b) is identified as that of person 1 in (5) above, all other hypotheses about face image (b) are rejected. In this case, the hypothesis of H ₂ for the face image (b) is rejected.

In the example above, the face image (a) is identified as the person 1 at time t = t _3, is identified as the person 1 in the face image (b) it is also a time t = t _4. That is, the face images (a) and (b) are identified as those of the same person. In this case, the face image (b) is very unstable and unstable, and it takes a long time for the LLR to exceed the threshold value due to this influence, and it is considered that an error has occurred in the identification. In particular, there are various variation factors such as posture and facial expression in the face image, and the probability ratio (LLR) due to the variation element is likely to fluctuate. However, in the SPRT that targets time series data having a variation factor as well as the face image, there is a possibility that the identification result may be inconsistent.

  The inconsistencies in the identification results are caused by the fact that the conventional SPRT does not consider the exclusion between objects and allows a plurality of time-series data to be identified as the same object.

  An object of the present invention is to solve the above-mentioned problems and to provide a time-series data identification device capable of accurately identifying whether input time-series data belongs to a registered person or other object. .

  Another object of the present invention is to provide a person meta information adding apparatus that adds meta information about a person to a moving image using the time-series data identification apparatus.

  In order to achieve the above object, an apparatus for identifying time-series data according to the present invention includes a plurality of identification means for individually identifying which of a registered object each of a plurality of input time-series data includes, In one of the plurality of identification means, when the identification means is identified as belonging to a registered object, the other identification means includes an exclusion means for excluding the object from the identification candidates. There is a feature.

  In the time-series data identification device according to the present invention, the identification unit sets hypotheses that define that the time-series data belongs to an object for each object, and the probabilistic probability and uncertainty of each hypothesis. It is characterized in that a likelihood ratio indicating a ratio is obtained, and which of the registered objects the time-series data belongs to is determined based on the magnitude.

  In the time-series data identification device according to the present invention, the identification unit stores a feature database of each object in advance, a feature database of each object stored in the registration database, and each time-series data. The comparison score calculation means for sequentially comparing the feature quantities of the two and calculating the comparison score, the LLR measurement means for calculating the likelihood ratio using the comparison score calculated by the comparison score calculation means, and the LLR measurement means There is a feature in that a comparison means for comparing the likelihood ratio obtained with a threshold value is provided.

  In the time-series data identification device according to the present invention, the exclusion means includes a state holding buffer indicating whether each hypothesis is being adopted, rejected, or being tested, and the comparison score calculating means includes: It is characterized in that a comparison score is calculated only for a hypothesis that is in the state of being tested.

  In the time-series data identification device according to the present invention, the identification unit identifies that the time-series data belongs to the object when the likelihood ratio of the object exceeds the upper limit value, and the hypothesis It is characterized in that the status is changed to adopted.

  In the time-series data identification device according to the present invention, the identification unit identifies that the time-series data does not belong to the object when the likelihood ratio for a certain object falls below a lower threshold, and the hypothesis It is characterized in that the state is changed to rejection.

  In addition, the time-series data identification device according to the present invention includes an LLR holding buffer that holds a log likelihood ratio of the previous time, the likelihood ratio is a log likelihood ratio, and the identification means holds the LLR holding It is characterized in that the likelihood ratio at the current time is obtained recursively using the likelihood ratio at the previous time held in the buffer.

  The time-series data identification device according to the present invention is characterized in that the identification means obtains a likelihood ratio using a Gaussian density function modeled in advance by learning.

  The apparatus for assigning person meta information to a moving image according to the present invention uses time-series data as a moving image of a face image, an object as a person, and any of the above-described time-series data identification devices and a moving image as a frame unit. The time-series data identification device includes a face index construction unit that sequentially reads and constructs a face index that associates a face image of the same person with a continuous display period thereof, and a person meta information addition unit. The person identification information is sent out, and the person meta information adding means adds the person identification information and the face index as person meta information to each corresponding frame of the moving image.

  Further, in the apparatus for assigning human meta information to a moving image according to the present invention, the time-series data identification device extracts a performer list from program information attached to the moving image and appears in the performer list. Characterized by identifying moving images by focusing only on hypotheses that define people

  In the time-series data identification device according to the present invention, in consideration of exclusivity between time-series data, one of the plurality of identification means is identified as an object in which certain time-series data is registered. In this case, since the object is excluded from the identification candidates by other identification means, a plurality of time-series data is not identified as the same object, and each time-series data can be identified with high accuracy.

  In addition, the apparatus for assigning human meta information to a moving image according to the present invention applies a time-series data identification method that considers exclusivity to a face image, and sets a face index that associates the same face image with its continuous display period. Since it builds and assigns person meta information in units of frames, it is possible to add meta information such as person names to moving images with high accuracy without visual confirmation, and the appearance of a desired person using this meta information as a query The scene can be searched accurately.

  The present invention will be described below with reference to the drawings. First, a time-series data identification device according to the present invention will be described. The time-series data identification device receives a plurality of time-series data as input, identifies which of the objects each of them belongs in advance, and outputs the identification result.

  FIG. 1 is a block diagram showing an embodiment of a time-series data identification device according to the present invention. The time-series data identification device 10 according to the present embodiment includes a comparison score calculation unit 11, a registration database (DB) 12, a state holding buffer 13, an LLR measurement unit 14, an LLR holding buffer 15, and a threshold identification unit 16.

  A plurality of time-series data is simultaneously input to the comparison score calculation unit 11. The plurality of time-series data are processed in parallel by the comparison score calculation unit 11, the LLR measurement unit 14, and the threshold identification unit 16. Therefore, these parts are prepared for the input time-series data. Each of these units can be realized by hardware or software.

  The comparison score calculation unit 11 reads time-series data at arbitrary time intervals, and calculates a comparison score indicating the degree of similarity between the time-series data and each object registered in advance in the registration DB 12. For example, when the time-series data is a face image (moving image), the frames of the face image are read one by one and the comparison score is calculated.

  In calculating the comparison score, the time-series data and the feature amount of the object can be used. In this case, in the registration DB 12, the identification function of each object, the feature amount of each object, and the density function of the score distribution regarding each of the positive example and the negative example are registered in advance. Data registered in advance in the registration DB 12 can be acquired by learning in advance using time-series data with known objects.

  The comparison score calculation unit 11 calculates a comparison score by comparing the feature amount of the input time-series data with the feature amount of each object registered in the registration DB 12. The comparison score indicates a higher value as the input time-series data is more similar to the object. In the following description, it is assumed that the input time-series data is a moving image (face image).

  The state holding buffer 13 holds a state indicating whether each hypothesis at the SPRT is being adopted, rejected, or being tested. The comparison score calculation unit 11 refers to the state held in the state holding buffer 13 and calculates a comparison score only for the hypothesis in the state under test. The state indicating whether each hypothesis is being adopted, rejected or being tested is held in the state holding buffer 13 according to the identification result in the threshold identifying unit 16, as will be described later.

  The LLR measurement unit 14 obtains the LLR at the current time using the comparison score calculated by the comparison score calculation unit 11.

Hypothesis on data z _{t at} time t (t = t ₁ , t ₂ ,..., t _N , t _{N + 1} ,...), where m class hypothesis is H _i (i = 1 to m) Assuming that the probability of H _i is P _i (z _t ), the log likelihood LLR (H _i ) of the hypothesis H _i at time t _N can be obtained by Equation (1). The probability P _i (z _t ) is learned in advance, and is set by modeling as a value that becomes a Gaussian density function with respect to the comparison score.

Also, the LLR at the current time can be obtained recursively using equation (3) using the LLR at the previous time stored in the LLR holding buffer 15. In Equation (3), LLR (H _i ) is the log likelihood of hypothesis H _{i at} the current time t = t _N , and LLR ′ (H _i ) is hypothesis H _{i at the} previous time t = t _N−1 . Log likelihood. Hand side and the third term in Equation (2) represents the LLR Additions at the current time t = t _N.

The threshold identifying unit 16 compares the LLR (H _i ) at the current time obtained by the LLR measuring unit 14 with the two thresholds A and B (A> B), and outputs the identification result. In other words, if LLR (H _i ) obtained by hypothesis testing for each of a plurality of time series data exceeds the upper threshold A, the hypothesis H _i is adopted, and if it falls below the lower threshold B, the hypothesis H _i is rejected. . Note that if LLR (H _i ) is between the upper threshold A and the lower threshold B, no determination is made. The thresholds A and B can be statistically derived from the viewpoint of accuracy represented by Recall and Precision.

For example, if time series data f is tested with hypothesis i, if LLR (H _i ) exceeds upper threshold A, hypothesis i is adopted, and time series data f is the person specified in hypothesis i. Identify. At the same time, change the state of hypothesis i to adopted.

If LLR (H _i ) falls below the lower threshold B, hypothesis i is rejected and the state of hypothesis i is changed to reject. If LLR (H _i ) is between the upper threshold A and the lower threshold B, nothing is done. The state of hypothesis i remains uncertain.

  The LLR holding buffer 15 accumulates the LLR obtained by the LLR measuring unit 14, and the state holding buffer 13 holds the state of each hypothesis i sent from the threshold identifying unit 16.

  FIG. 2 is an explanatory diagram of the SPRT in FIG. Again, it is assumed that the time-series data to be identified is a moving image and the objects are persons 1, 2, and 3 appearing in a TV program.

2 (a) and 2 (b) show the LLR transitions of hypotheses H ₁ , H ₂ and H ₃ obtained for the input facial images (a) and (b) (FIG. 6), respectively. LLR (H ₁ ), the broken line is LLR (H ₂ ), and the alternate long and short dash line is LLR (H ₃ ). When SPRT is performed based on the LLR along the time axis in FIG. 2, the adoption and rejection of hypotheses are decided in the order of (1) to (6) below.

(1) At time t = t ₁ , LLR (H ₃ ) for the face image (a) falls below the lower threshold B (LLR (H ₃ ) <B), so the hypothesis H ₃ is rejected. Time t = t ₁ and later, only the test of the hypothesis H _1, H ₂ is the face image (a) is subsequently performed.

(2) at time t = t _2, since the LLR for the face image (b) (H ₃₎ is below the lower threshold _{B (LLR (H 3) <} B), to reject the hypothesis H _3. Time t = t ₂ and later, only the test of the hypothesis H _1, H ₂ is the face image (a) is subsequently performed. Until the time t = t ₂ is exactly the same as that shown in FIG. 7.

(3) Since LLR (H ₁ ) for the face image (a) exceeds the upper threshold A (LLR (H ₁ )> A) at time t = t ₃ , hypothesis H ₁ is adopted. As a result, the face image (a) is identified as that of the person 1.

(4) Since face image (a) is identified as that of person 1 in (3) above, all other hypotheses about face image (a) are rejected. In this case, the hypothesis of H ₂ for the face image (a) is rejected.

(5) (3) identified a person in the face image (a), since the same person in the same frame does not appear, to reject the hypothesis H ₁ of the face image (b). Time t = t ₃ or later, only the test of the hypothesis H ₂ is for the face image (b) is subsequently performed.

(6) Since LLR (H ₂ ) for the face image (b) exceeds the upper threshold A (LLR (H ₂ )> A) at time t = t ₅ , hypothesis H ₂ is adopted. Thus, the face image (b) is identified as that of the person 2.

  (7) Since the face image (b) is identified as that of the person 2 in the above (6), all other hypotheses about the face image (b) are rejected. In this case, there is no hypothesis rejected for the face image (b).

  As described above, in consideration of the fact that the same person does not appear in the same frame, when identifying certain time-series data as that of a certain person, that person is excluded from identification candidates in other identifications. The other time series data is not distinguished from that of the same person.

  Next, a person meta information providing apparatus according to the present invention will be described. The person meta information providing apparatus according to the present invention uses input time-series data as a moving image, an object as a performer, and a person who has time-series data (face image) using the above-described time-series data identification apparatus. The person meta information is added to the moving image.

In addition, in order to recursively obtain the LLR using the time-series data identification device described above, each of the input time-series data needs to be a face image of the same person. For example, a TV program currently being viewed, if two people have appeared as shown in FIG. 6, the time t _N and mapping to determine the same face between the next time t _{N + 1,} each face image Must be input to the time-series data identification device as two time-series data. Therefore, the person meta information providing apparatus described below includes a face image tracking unit and the like before the time-series data identification apparatus.

  FIG. 3 is a block diagram showing an embodiment of the person meta information providing apparatus according to the present invention. The person meta information providing apparatus according to the present embodiment includes a moving image storage unit 31 that stores moving images including face images, a face image tracking unit 32, a representative face determination unit 33, a face index construction unit 34, and a performer information extraction unit 35. A face registration unit 36, a face DB 37, a face identification unit 38, and a person meta information adding unit 39.

  The functions of the face image tracking unit 32, the representative face determination unit 33, and the face index construction unit 34 are the “Face index creation device for moving images and its face image tracking method” previously proposed by the present inventor (Japanese Patent Application No. 2007-88738). No.), so only the outline is explained.

  The face image tracking unit 32 sequentially reads the moving images from the moving image storage unit 31 in units of frames, tracks the face images of the same person that appears continuously over a plurality of frames, and detects the continuous display period. The representative face determination unit 33 determines a representative face for each continuous display period detected by the face image tracking unit 32. The face index construction unit 34 constructs a face index that associates the continuous display period of each face image with the representative face of the continuous display period. When a plurality of face images are included in one frame image, a face index is constructed for each face image.

  The performer information extraction unit 35 extracts performer information such as names of performers appearing in the moving images stored in the moving image storage unit 31. For example, if the moving image stored in the moving image storage unit 31 is a TV program and program information such as EPG (Electronic Program Guide) information is stored along with the TV program, the performer information is then added to the performer information. Extract as

  In the face registration unit 36, names and face images are registered in advance in association with many persons. The face DB 37 corresponds to the registration DB of FIG. 1 and is data about a person on the performer list extracted by the performer information extraction unit 35 from the data registered in the face registration unit 36. (The name of each person, the feature quantity of each person's face image, and the density function of the score distribution for each of the positive and negative examples) are extracted and registered.

  The face identification unit 38 receives time-series data of face images detected by tracking from the image by the face image tracking unit 32. When a plurality of face images are included in the image, time-series data for each face image is generated, and the plurality of time-series data are simultaneously input to the face identifying unit 38.

  The face identifying unit 38 is a part corresponding to the comparison score calculating unit 11, the state holding buffer 13, the LLR measuring unit 14, the LLR holding buffer 15 and the threshold identifying unit 16 of FIG. Data is read, a hypothesis as to which person the time-series data belongs to is sequentially subjected to a probability ratio test, and person identification information such as a person's name is output.

  The person meta information adding unit 39 describes the face index (continuous display period / representative face image) output from the face index constructing unit 34 and the person identification information output from the face identifying unit 38 as person meta information in units of frames. The image is assigned to the corresponding moving image in the moving image storage unit 31.

  FIG. 4 is a block diagram functionally expressing the face image tracking unit 32 and the face identification unit 38 of FIG. The face image tracking unit 32 is the same as that proposed in the above Japanese Patent Application No. 2007-88738, and the face identification unit 38 is the same as that shown in FIG.

  The face image tracking unit 32 includes a frame image acquisition unit 41, a current frame buffer 42, a previous frame buffer 43, a shot change identification unit 44, a face detection unit 45, a current detection result buffer 46, a previous detection result buffer 47, and an inter-face distance calculation. A unit 48, a distance dependency correspondence unit 49, and a similarity dependency correspondence unit 50.

  The face identifying unit 38 includes a comparison score calculating unit 51, a face DB 52, a state holding buffer 53, an LLR measuring unit 54, an LLR holding buffer 55, and a threshold identifying unit 56.

  The frame image acquisition unit 41 reads a still image for one frame from a moving image stored in the moving image storage unit 31 at an arbitrary time interval. The current frame buffer 42 stores the still image of the current frame read this time, and the previous frame buffer 53 stores the still image of the previous frame read last time.

  The shot change identifying unit 44 compares the still images stored in the current frame buffer 42 and the previous frame buffer 53, and identifies the presence or absence of a camera edit point between shots based on the similarity.

  The face detection unit 45 detects a face image from the still image of the current frame stored in the current frame buffer 42, and for each detected face image, a spatial position such as a position coordinate, a width, and a height of the display range. Ask for information.

  The detection result buffer 46 this time accumulates the position information of the face image obtained by the face detection unit 45. Note that the face image immediately after being detected by the face detection unit 45 is treated as an undetermined face candidate, and is treated as a confirmed face when a face image is detected at the same display position in the next frame. The previous detection result buffer 47 accumulates spatial position information such as the position coordinates, width, and height of the display range of the face candidate and the confirmed face detected in the previous frame.

  The face-to-face distance calculation unit 48 is detected by the face detection unit 45 and the display position of each face image of the current frame accumulated in the current detection result buffer 46 and each face of the previous frame accumulated in the previous detection result buffer 47 A distance Δd from the image display position (hereinafter referred to as a face-to-face distance) is calculated. This distance Δd can be calculated, for example, by obtaining the distance between the upper left coordinates of each face image.

  The distance dependence association unit 49 detects a combination of face images (face candidates and confirmed faces) in which the inter-face distance Δd calculated by the inter-face distance calculation unit 48 is less than a predetermined threshold Δdref in the current frame. The face candidate state is updated to a confirmed face in the current detection result buffer 46. At the same time, the state of the face candidate detected in the previous frame is updated to a confirmed face in the previous detection result buffer 47, and each face image is associated with each other as a face image series. Here, the predetermined threshold Δdref is preferably set to a value proportional to the size of the face image detected in the previous frame.

  The similarity dependence association unit 50 uses the confirmed face that is not associated with any face image of the current frame among the confirmed faces of the previous frame accumulated in the previous detection result buffer 47 as an image of the current frame. Template matching is performed, and an image of a region whose similarity exceeds a predetermined threshold is added to the current detection result buffer 46 as a new face image (determined face), and each face image is associated with each other as a face image series.

  The template matching application area corresponds to the display position in the pre-face frame of the confirmed face used as a template instead of the entire image of the current frame in order to reduce false detection in template matching and reduce the amount of calculation. It is preferable to limit the position and its vicinity.

  The previously detected face image and its region coordinates are held in the previous detection result buffer 47, and these are sent to the representative face determination unit 33 (FIG. 3) and the face feature amount creation unit 57.

  The face identification unit 38 has the same configuration as that in FIG. 1, and sequentially tests the hypothesis as to which of the registered persons the confirmed face image is, and outputs person identification information. However, here, a face feature amount creation unit 57 is interposed between the previous detection result buffer 47 and the face identification unit 38.

  The face feature quantity creation unit 57 creates face feature quantities suitable for face identification from the time series data of each confirmed face image output from the previous detection result buffer 47, and uses the face feature quantities as time series data to the comparison score calculation unit 51. input. The face feature amount can be created by the Eigen face method (proper face method), the Fisher face method, or other methods. The face feature amount is preferably a feature amount that is robust to variations in posture, lighting, and the like. It is also preferable to extract both organs such as pupils in the face and to normalize the position and size with reference to the extracted organs such as pupils in the face.

  As described above, in SPRT for determining a person based on a face image, the person who has registered time-series data is identified using each hypothesis that defines each person. Here, in the conventional SPRT, the identification is performed without considering the exclusivity between the plurality of time series data, whereas in the present invention, the identification is performed in consideration of the exclusivity between the plurality of time series data. The data is not mistakenly identified as that of the same person. Therefore, a plurality of time-series data can be identified with high accuracy, and the meta information of the characters can be accurately given to the moving image in units of frames.

It is a block diagram which shows one Embodiment of the identification apparatus of the time series data which concerns on this invention. It is operation | movement explanatory drawing of the sequential probability ratio test (SPRT) in FIG. It is a block diagram which shows one Embodiment of the person meta information provision apparatus which concerns on this invention. FIG. 4 is a block diagram functionally expressing a face image tracking unit and a face identification unit of FIG. 3. It is operation | movement explanatory drawing of the conventional SPRT. It is a figure which shows the example of the image of 1 frame in a certain scene of a TV program image | video. It is explanatory drawing of the problem in the conventional SPRT.

Explanation of symbols

10 ... Time-series data identification device, 11, 51 ... Comparison score calculator, 12 ... Registration database (DB), 13, 53 ... State holding buffer, 14, 54 ... LLR measurement , 15, 55 ... LLR holding buffer, 16,56 ... threshold identification unit, 31 ... moving image storage unit, 32 ... face image tracking unit, 33 ... representative face determination unit, 34 ... Face index construction part, 35 ... Performer information extraction part, 36 ... Face registration part, 37,52 ... Face DB, 38 ... Face identification part, 39 ... Person meta information Giving unit, 41 ... Frame image acquisition unit, 42 ... Current frame buffer, 43 ... Previous frame buffer, 44 ... Shot change identification unit, 45 ... Face detection unit, 46 ... This time Detection result buffer, 47 ... Previous detection result buffer, 48 ... Face distance calculation section, 49 ... Distance dependence correspondence section, 50 ... Similarity dependence correspondence section

Claims (10)

In the time-series data identification device for identifying which of the registered objects each of a plurality of input time-series data belongs,
A plurality of identification means for individually identifying which of the registered objects each of a plurality of input time-series data belongs;
When one of the plurality of identification means identifies that a certain time-series data belongs to a registered object, the other identification means includes an exclusion means for excluding the object from identification candidates. A characteristic time-series data identification device.   The identification means establishes a hypothesis defining that time series data belongs to an object for each object, obtains a likelihood ratio indicating a ratio of probabilistic probability to uncertainty of each hypothesis, and based on the magnitude 2. The time-series data identification device according to claim 1, wherein it identifies which of the registered objects the time-series data belongs.   The identification unit sequentially compares a feature database of each object, a feature amount of each object stored in the registration database, and a feature amount of each time-series data, and calculates a comparison score. Comparison score calculation means, LLR measurement means for obtaining a likelihood ratio using the comparison score calculated by the comparison score calculation means, and comparison means for comparing the likelihood ratio obtained by the LLR measurement means with a threshold value The time-series data identification device according to claim 2.   The exclusion means includes a state holding buffer that indicates whether each hypothesis is being adopted, rejected, or being tested, and the comparison score calculation means calculates a comparison score only for a hypothesis that is in the state being tested. The time-series data identification device according to claim 3.   5. The identification means, when a likelihood ratio for a certain object exceeds an upper limit value, identifies that time-series data belongs to the object, and changes the hypothesis state to adopted. The time-series data identification device described in 1.   5. The identification unit, when a likelihood ratio for a certain object falls below a lower threshold, identifies that time-series data does not belong to the object, and changes the hypothesis state to rejection. Or the time-series data identification device according to 5.   An LLR holding buffer that holds a log likelihood ratio of the previous time is provided, the likelihood ratio is a log likelihood ratio, and the identification unit uses the likelihood ratio of the previous time held in the LLR holding buffer. 7. The time-series data identification device according to claim 2, wherein the likelihood ratio at the current time is obtained recursively.   7. The time-series data identification apparatus according to claim 2, wherein the identification unit obtains a likelihood ratio using a Gaussian density function modeled in advance by learning. Time series data is a moving image of a face image, an object is a person,
A time-series data identification device according to any one of claims 1 to 8,
Face index construction means for sequentially loading moving images in units of frames and constructing a face index in which face images of the same person and their continuous display periods are associated with each other;
Provided with person meta information giving means,
The time-series data identification device identifies a face image and sends out the person identification information, and the person meta information giving means uses the person identification information and the face index as person meta information to correspond to each of moving images. An apparatus for assigning person meta information to a moving image, characterized by being attached to a frame.   The time-series data identifying device extracts a performer list from program information attached to a moving image, and identifies a moving image by focusing only on a hypothesis defining a person appearing in the performer list. The apparatus for adding person meta information to a moving image according to claim 9.

Images