JP3639520B2 - Moving image learning method and recording medium recording this program - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a learning technique for moving image recognition such as a human face moving image, and in particular, under real environment conditions, continuous video data over a long period of time without prior information on a category (person name) is provided. The present invention relates to a method for learning images in a self-organizing manner.
[0002]
[Prior art]
A face image recognition will be described as an example. Face image recognition is realized by pattern matching (calculation of distance value or similarity) between a face image model created in the learning stage and an unknown input face image.
[0003]
In the learning in the conventional face recognition method, a method is adopted in which learning is advanced from a small number of face images that are photographed under limited environmental conditions and are manually assigned a category such as a person's name.
[0004]
However, since this approach requires manual data preparation, it is difficult to use a large amount of face data for learning.
[0005]
When face recognition is performed in this environment using this approach, since only a limited number of face images can be used for learning, various variations that occur in the actual environment (for example, variations in lighting conditions, camera (viewpoint) It is difficult to accumulate sufficient learning data to simply cope with fluctuations in the angle and distance of the face with respect to), fluctuations in face shape due to facial expressions, glasses, hairstyles, and the like.
[0006]
Conventionally, in order to cope with this, pre-processing has been devised and features have been devised, but sufficient accuracy has not been obtained. In other words, the conventional face recognition has not been provided with the flexibility to cope with a real environment rich in complicated changes.
[0007]
On the other hand, in the case of human face recognition, when the learning system is operating at any time and new data is input, the internal state of the system can be updated and a new category can be added at any time without prior information.
[0008]
[Problems to be solved by the invention]
As described above, the recognition method based on pattern matching between a face image model and an unknown input face image requires manual data preparation work, and it is difficult to use a large amount of face data for learning. In addition, it is difficult to accumulate sufficient learning data and to obtain sufficient accuracy.
[0009]
In the present invention, without using information prepared in advance by humans, a computer autonomously learns long-term continuous data in a real environment in a self-organizing manner according to the purpose of its own task. We propose a way to add new categories naturally. In other words, an object of the present invention is to propose a method for additionally learning various variations included in long-term environmental video data without human intervention without a teacher.
[0010]
Another object of the present invention is to propose a learning method in which all the processes (data collection, storage, face area extraction, learning, recognition) are automatically performed and no manual work is required.
[0011]
[Means for Solving the Problems]
In order to achieve the above-mentioned object, the present invention automatically extracts moving objects appearing in the environment video data for a long time and uses the information of a plurality of extracted moving objects. By automatically associating, moving objects are automatically classified into categories without artificially assigning categories. In addition, when categorizing, it is possible to manually categorize only a part of video data in which the target information is included, and features the following learning method and recording medium. And
[0012]
(Invention of learning method)
In a moving image learning method that automatically learns a target object that moves in an image from a continuously captured environment image data without a teacher,
Shooting data for which the target object category information is not given in advance,
Cutting out a moving image of the target object in time and space to create a moving image array;
Calculating a distance between moving image sequences based on a distance between images included in the moving image sequence among all combinations of the moving image sequences;
Forming an initial sub-cluster using a distance between the video sequences;
combining all sub-clusters using consistent / inconsistent edges;
Creating a moving image dictionary for each of the combined categories;
It is comprised from these.
[0013]
Further, the method includes a step of providing prior information for categorization with respect to a part of the clipped moving image.
[0014]
In addition, in the step of combining all sag clusters using the consistent / inconsistent edge, the method includes a step of providing category information and a step of changing the consistent / inconsistent edge based on the given category information. And
[0015]
(Invention of recording medium)
A program for causing a computer to execute a moving image learning method for automatically learning an object moving in an image from a continuously captured environment image data without a teacher on a computer-readable recording medium A recorded recording medium,
Between the process of photographing data for which category information of the target object is not given in advance, the process of cutting out the moving image of the target object in time and space, and creating the moving image array, and between the moving image arrays of all combinations A process of calculating a distance between moving image arrays based on a distance between images included in the moving image array, a process of forming an initial sub-cluster using the distance between the moving image arrays, and consistent / inconsistent The process of combining all sub-clusters using edges and the process of creating a moving image dictionary for each of the combined categories are recorded by a program.
[0016]
DETAILED DESCRIPTION OF THE INVENTION
FIG. 1 is a block diagram showing an embodiment of the present invention. In the present embodiment, it is composed of three units, a face area extraction device (unit 1), a learning device (unit 2), and a recognition device (unit 3).
[0017]
The face area extraction apparatus 1 automatically extracts a face area of a person who appears in long-term continuous environmental video data, normalizes the size thereof, and a series of faces from when a person appears in the image until disappearance An image series (this is called a moving image array) is created, stored in a file, and input information is provided to the learning / recognition apparatuses 2 and 3 in the subsequent stage.
[0018]
The learning device 2 uses a learning algorithm described later to organize a large number of moving image arrays that do not include prior category information for each category. According to the result, the internal state of the system is also appropriately updated. Here, the internal state refers to an expression indicating which and which belong to the same category among moving image arrays.
[0019]
The recognizing device 3 recognizes a face based on a comparison between the latest moving image dictionary (registered moving image array) of the system and input data to be recognized (moving image array cut out from the input moving image by the face area extracting device). Recognize and output the recognition result.
[0020]
First, an example of a face area extracting apparatus as a preparation stage for learning and recognition will be described with reference to the flowchart of FIG.
[0021]
In step 201, environmental video data is continuously input from a fixed camera, but two adjacent images are subtracted and a threshold value is taken to create a binary difference image.
[0022]
In step 202, it is determined from the created difference image whether the observed environment has changed. If there is no change corresponding to the appearance of a person, steps 201 and 202 are repeated until there is a change, but if there is a change, the process proceeds to step 203.
[0023]
In step 203, a multi-resolution image pyramid is created while repeating subsampling from the binary difference image created in the previous step. Subsampling replaces each 2 × 2 pixel region of the original image with a “1” or “0” value pixel (based on the number of “1” s in the region). By using a multi-resolution image, the coordinates of the face area can be calculated more stably and accurately. That is, since a low-resolution image is hardly affected by noise, the face area can be determined relatively stably, but it is not accurate. Conversely, the face area can be accurately determined from a high-resolution image, but it is susceptible to noise. The face area extraction algorithm described below is performed in parallel on each resolution image and is integrated in step 207 as will be described later.
[0024]
In the next step 204, the following filter is applied to all images in order to fill “holes” of “0” value formed in the person area due to the influence of noise due to fluctuations in lighting conditions, clothing texture, and the like. Multiply.
[0025]
If a window of 3x3 pixels is moved through the image in one pixel step, and there are pixels of value "1" in at least two different rows and two different columns in the window at the same time, Replace the pixel at the center with “1”. This operation is performed several times (or until there is no change) in the left / right, right / left, up / down, and down / up directions.
[0026]
In the next step 205, the center of gravity of the image created in the previous stage and the x histogram (a function representing the number of “1” s included in each column of the image) are calculated, and 90% of the total energy of the x histogram is calculated from the center of gravity. An area including% is cut out as a person area. In other words, it is assumed that the area of the person is much larger than the area of other objects moving in the background, shadows, noise, and the like.
[0027]
In step 206, an x histogram and a y histogram are newly calculated on the person region extracted in step 205, and the contents are analyzed to determine the face coordinates.
[0028]
In step 207, the median of the face area coordinates calculated at each resolution is obtained to obtain the final face area coordinates.
[0029]
In step 208, the face image extracted as described above is normalized to a certain size. An example of a final moving image obtained from actual input data is shown in FIG.
[0030]
Thereafter, in step 210, the next image is read from the camera, and a binary difference image with the background image is created.
[0031]
In step 211, it is detected from the difference image created in step 210 whether or not the person is out of the field of view of the camera. If there is still a person due to this detection, the process returns to step 203 to process the acquired image.
[0032]
In step 212, when the person is no longer in the field of view of the camera, a series of face images up to that point is stored (stored) in the moving image file as a moving image array.
[0033]
Next, operation | movement of the learning apparatus 2 which is an important element of this invention is demonstrated below using the flowchart of FIG.
[0034]
First, in the first step 301, the moving image array (as a movie file) obtained in the previous stage (a series of face images and corresponding to one continuous moving image) is read into the memory of the computer. As described above, the category information regarding each moving image array is not given in advance. For example, N moving image arrays (N ≧ X) for X different persons are given, but information on which array corresponds to which person is not given in advance.
[0035]
In step 302, in order to calculate the distance between the two stored moving image arrays, the i-th facial image of one moving image array and the j-th facial image included in the other moving image array are calculated. Distance values are calculated between them and stored in the matrix D _ij . At that time, there are various methods for calculating the distance between the two face images. Here, as an example, a moving image is subtracted for each pixel, an absolute value is obtained, and a threshold value is obtained. If it is larger, “1” is assumed, and if it is smaller, “0” is assumed. The total number of “1” included in the binary image thus obtained is defined as a distance value between both face images.
[0036]
In step 303, the shortest distance between each moving image array and all other arrays is selected using the matrix _Dij and _stored in the shortest distance matrix M. For example, M _ij in the matrix M means the shortest distance between the array i and the array j. A large number of face images are included in both moving image arrays, and the closest face pair among them is selected as a representative, and the distance between them is M _ij .
[0037]
In step 304, a number of initial subcluster graphs are formed based on the values of the matrix M. Each array is displayed as a single node, and each node is combined with only the closest node (by an edge), and each graph thus created is named an initial subcluster graph. Two types of edges that connect two nodes are defined, one is called a consistent edge and used to connect nodes of the same category (moving image array), and the other is called an inconsistent edge and connected to nodes of different categories. Used for. However, all edges in the initial subcluster graph formed in step 304 are cosistive edges.
[0038]
In the next step 305, the initial subclusters are sorted by the number of nodes, starting with a small subcluster (small number of nodes included) and linking with the closest subcluster of the other subclusters at the edge. . The type of edge at that time is determined by a rule described later. Two subclusters are merged and merged simultaneously to create a new subcluster. This process is recursively repeated until all sub-clusters are merged into only one large cluster. When two subclusters are connected by the shortest distance edge (for example, the edge between node A and node B), the type of edge is determined by the following rule (consistency criterion). Since the edge of length L between node A and node B is consistent, two conditions that must be satisfied simultaneously are shown below.
[0039]
Condition 1 “The distance L1 between the node A and the node (FN) farthest from the node A among the nodes directly connected to the node A by the consistent edge is smaller than CxL, where C is a constant.”
Condition 2 “The distance L2 between the node (FN) farthest from the node B among the nodes directly connected to the node B at the consistent edge is smaller than CxL, where C is a constant.”
FIG. 5 is an example showing the relationship between the node A and the node B and the respective FN nodes. Nodes that do not satisfy condition 1 and condition 2 at the same time are connected by an inconsistent edge.
[0040]
In the last step 306, while traversing the graph formed by the above process, the moving image sequence displayed by the nodes connected by the consistent edge is output as the same face category. Here, the role of the inconsistent edge is to separate sub-clusters belonging to different categories. FIG. 6 shows an example of a graph formed in the above process when actual input data is used. In FIG. 6, each node corresponds to one moving image array, characters are initial letters of actual person names shown in the array, and numbers are array numbers. An edge displayed by a solid line is a consistent edge, a dotted line represents an inconsistent edge, and a number represents the length of the edge (the shortest distance between two nodes).
[0041]
This graph shows the current internal state of the system. In the recognition process, the internal state is temporarily fixed, and the node corresponding to the input is determined based on the consistency criterion described in step 305 and the internal state at the consistent / inconsistent edge. If it is combined with the graph and its edge is a cosient edge, then it is assumed that the category of the input is the same category as the node on the other side of the edge. On the other hand, an incoincid edge belongs to a new category (not yet registered).
[0042]
Additional learning is performed by the following algorithm. However, assuming that the current internal state of the system is expressed by a graph composed of N nodes, in the following, additional learning is performed on the N + 1th node input in the internal state shown in FIG. I will do it.
[0043]
Step 1: A new (N + 1) th column is calculated and added in the matrix M for the newly input node.
[0044]
Step 2: Find the closest node k of the new node and connect both nodes with a consistent / inconsistent edge based on the above consistency criteria. If it is an inconsistent edge, go to step 4.
[0045]
Step 3: For all nodes l belonging to the same cluster as node k, calculate the distance D ₁ between node l and the new node, and the distance D ₂ between node l and the nearest node n. . If D1 <D2, node 1 and the new node are connected by a consistent edge. Then, for all nodes m that are directly connected to node l by the cosistive edge (including node n), the distance D (l, m) between l and m and the distance between m and the new node Calculate the distance D (N + 1, m), and if D (l, m) <D (N + 1, m), delete the edge between m and l and a new edge between m and the new node (If there is no such edge yet).
[0046]
Step 4: For all nodes P _{i in the} cluster C _k (see FIG. 8) to which the new node belongs and the cluster C _i connected by the inconsistent edge E _ik (if such a cluster exists), the new node distance _{D (P i, N + 1} ) between the calculated, if _{D (P i, N + 1} ) <E ik, delete the E _ik, to insert a new edge E _ik between the new node and P _i (The edge type is determined based on the consistency criterion). Then, inconsistent Once the cluster C _i and C _j which is connected at the edge E _ij is present (However, although C _i are good to connect with C _k, C _j is only if you have not connected the C _k), cluster For all the nodes P _j belonging to C _j , the distance D (P _j , N + 1) to the new node is calculated, and if D (P _i , N + 1) <E _ij is satisfied, E _ij is deleted Then, a new edge E _jk is inserted between P _j and the new node (the edge type is determined based on the consistency criterion). However, if C _i is not connected to C _k , this operation is performed only when C _i or C _j is not disconnected from other clusters.
[0047]
The above-described additional learning method has a mechanism in which data is collected to some extent at the initial stage, and then additional learning is performed. However, sequential learning from the beginning (so-called online learning) is also considered. It is done. In the online version, initially only two nodes are given, no matter how far apart they are connected by a consistent edge, and then a newly entered node is inserted into the graph based on the above additional learning method. After a certain amount of data is collected, consistency check is periodically performed on all nodes, and edges between nodes that do not satisfy the consistency criterion are replaced with inconsistent edges.
[0048]
As a feature of the proposed learning method, in the case of additional learning, it is not necessary to re-learn the internal state of the system from the beginning, and a connection related to only a new part is updated and added, and an unrelated part remains as it is. As a result, not only natural additional learning that a living creature or the like performs can be realized, but also a huge amount of calculation can be saved. In particular, in an actual environment, since the internal state of the system is formed based on long-time data, it is advantageous to use this method.
[0049]
In addition, the learning apparatus 2 in the above embodiment, for example, when new target information is input in the state of FIG. 6, categorizing the target information in a graph of the target information by comparing with all nodes. shows the case of determining by automatically calculating the position, whether the information object information input is included in any category in advance by giving some manually, can be learned more accurate. For example, when the given target information is a category represented by F, the distance between the nodes F1 to F6 in FIG. 6 is directly calculated without automatic categorization, The exact graph position can be determined.
[0050]
In addition, when sub-clusters are combined using a consistent / inconsistent edge, a step of changing the consistent / inconsistent edge based on category information by automatic learning or category information given manually is shown to indicate the generated category. If the graph is connected by mistake, it can be corrected.
[0051]
Further, the above description shows a case where a face image is a target, but the present invention can perform the same processing for other target objects.
[0052]
Also, a part or all of the method shown in FIG. 2 and FIG. 3 or the apparatus shown in FIG. 1 is written in a computer program so that the computer program can be executed. It can be recorded and provided on a disk (registered trademark), MO, ROM, memory card, CD, DVD, removable disk, and distributed.
[0053]
【The invention's effect】
As described above, according to the present invention, the following effects can be obtained.
[0054]
(1) Under real-world conditions, the computer autonomously learns and recognizes long-term continuous data according to the purpose of its task without using information prepared in advance by humans. It becomes possible to do.
[0055]
(2) Since each face image of a moving image extracted continuously from input data is used as a template, it is less susceptible to variations in illumination conditions, size, and viewpoint angle, so the learning / recognition process Efficiency can be improved.
[0056]
(3) When new category data is given in the middle of learning, it is not necessary to re-learn the system from the beginning, and flexible additional natural learning and recognition are possible.
[0057]
(4) By using the learning method of the present invention, it is possible to recognize even when sample data in a class is continuously distributed and the distance between classes is larger than the distance in the class.
[0058]
(5) Since all the processes of data collection, storage, face area extraction, learning, and recognition are automatically performed, it is possible to save a lot of manual work.
[0059]
(6) As a feature of the learning method of the present invention, there is no problem even if the number of samples of each face category is extremely different. In other words, the method of the present invention has the advantage that it does not depend on the shape of the sample distribution.
[0060]
(7) Various features used in object recognition can be evaluated using the learning method of the present invention.
[Brief description of the drawings]
FIG. 1 is a flock diagram illustrating a configuration example of the present invention.
FIG. 2 is a flowchart for explaining a face area extraction method according to the present invention;
FIG. 3 is a flowchart for explaining the learning method of the present invention.
FIG. 4 is an example of a face image array extracted from an image array input from a camera. FIG. 5 is an auxiliary diagram for explaining consistency standards.
FIG. 6 shows an example of an internal state of a system formed by the learning method of the present invention.
FIG. 7 is an auxiliary diagram for explaining an additional learning method (steps 1 to 3) in the present invention.
FIG. 8 is an auxiliary diagram for explaining an additional learning method (step 4) in the present invention.
[Explanation of symbols]
1. Face area extraction device (unit 1)
2 ... Learning device (unit 2)
3. Recognition device (unit 3)

Claims (4)

In a moving image learning method that automatically learns a target object that moves in an image from a continuously captured environment image data without a teacher,
Shooting data for which the target object category information is not given in advance,
Cutting out a moving image of the target object in time and space to create a moving image array;
Calculating a distance between moving image sequences based on a distance between images included in the moving image sequence among all combinations of the moving image sequences;
Forming an initial sub-cluster using a distance between the video sequences;
combining all sub-clusters using consistent / inconsistent edges;
Creating a moving image dictionary for each of the combined categories;
A moving image learning method comprising:   The moving image learning method according to claim 1, further comprising a step of giving prior information for categorization with respect to a part of the cut out moving image.   The step of combining all sag clusters using the consistent / inconsistent edge comprises providing category information and changing the consistent / inconsistent edge based on the given category information. The moving image learning method according to claim 1 or 2. A program for causing a computer to execute a moving image learning method for automatically learning an object moving in an image from a continuously captured environment image data without a teacher on a computer-readable recording medium A recorded recording medium,
Between the process of photographing data for which category information of the target object is not given in advance, the process of cutting out the moving image of the target object in time and space, and creating the moving image array, and between the moving image arrays of all combinations A process of calculating a distance between moving image arrays based on a distance between images included in the moving image array, a process of forming an initial sub-cluster using the distance between the moving image arrays, and consistent / inconsistent Combining all sub-clusters using edges, creating a moving image dictionary for each of the combined categories,
A recording medium characterized by recording the program by a program.

Images