JPH04259075A - Model generator from time series image - Google Patents

Abstract

PURPOSE:To automatically generate an arbitrary object model expressed in a time series image by using another feature point that can be automatically extracted instead of a conventional joint hard to extract in a generator to generate the model from the time series image. CONSTITUTION:This generator is equipped with a silhouette image generating part 3 which inputs a time series label image 1 when labeling is performed on an area and generates a silhouette image in which the ones with the amount of travel of a label area at every time less than a threshold value are unified, and a model generating part 4 which performs line-thinning and extracts the feature point about a generated silhouette image and sets the appearance rate of feature point corresponding to between frames over the threshold value as a structural point. A time series model image formed by connecting those structural points with line segments is outputted.

Description

[Detailed description of the invention]

0001

[Field of Industrial Application] The present invention relates to a model creation device from time-series images, and more particularly, to a model creation device that automatically creates a simple model of an object depicted in the images from time-series images. be. The model created by this device is characterized by expressing the characteristic parts of the target structure with points and the connection relationships between the points with lines, and there is no need to assume the target in advance when creating the model. It is characterized by

[0002] An example of the field of application of this device is a device for recognizing objects in time-series images. In order to recognize an object in an image, it is essential to match the image with a predefined model, but it is extremely difficult to extract the information necessary for matching from the image. The model automatically generated by this device is a simple model that reflects the structure of the object, and matching the generated model with a predefined model facilitates processing. Also,
This device records the movement of the object in a time series as changes in position information in each frame of the points that make up the generated model, so by analyzing the generated model, it is possible to analyze the movement of the object in the image. It can be performed.

0003

2. Description of the Related Art Although there is no conventional method that directly corresponds to the present invention, there are the following similar methods. A typical method is to limit the object in the image to a non-rigid object in which a plurality of rigid rods are joined at joints, and estimate the connection relationship of the rods between the joints. The connection relationship between joints is determined from the condition that the length of the rod is constant. However, this method focuses on estimating the position in three-dimensional space from the position of each joint on the image and generating a model that includes three-dimensional position information. This method differs greatly from the present invention in that it assumes that the movement is already known.

0004

[Problem to be solved by the invention] According to the above method,
Since the target in the image is limited to a non-rigid object in which multiple rigid rods are connected by joints, it is necessary to accurately determine the joint positions and velocities on the image, but in reality, the joint positions are determined. It is difficult to do so. Therefore, there is a problem that it is difficult to apply it to automatic generation of models from time-series images. In addition, when the target is limited to a non-rigid body and automatic recognition of the target represented by an image is performed, it is not possible to determine in advance whether the target is composed of rigid or non-rigid body parts. There is a need for a method to deal with this situation.

The present invention aims to provide a method for automatically generating a model of any object represented by time-series images by using other feature points that can be automatically extracted in place of conventional joints that are difficult to extract. The purpose is

[0006]

[Means for Solving the Problems] FIG. 1 shows a diagram of the principle configuration of the present invention. In FIG. 1, the silhouette image generation unit 3 generates a silhouette image of time-series label images 1 in which the amount of movement of the label area at each time is less than or equal to a threshold value.

[0007] The model generation unit 4 performs line thinning and feature point extraction on the silhouette image, and generates as structural points the corresponding feature points between frames whose probability of appearance is equal to or higher than a threshold value.

[0008]

[Operation] As shown in FIG. 1, the present invention generates a silhouette image by collecting the time-series label images 1 in which the silhouette image generation unit 3 labels the regions, and the movement amount of the label region at each time is less than a threshold value. The model generation unit 4 performs line thinning and feature point extraction on the silhouette image, generates structural points when the appearance probability of corresponding feature points between frames is equal to or higher than a threshold, and connects these structural points with line segments. I am trying to output a time series model.

[0009] Therefore, by thinning the silhouette images, extracting feature points, and connecting these feature points to generate a time-series model, a silhouette image that is a collection of objects whose movement amount is less than a predetermined threshold from a time-series image is used to create a rigid body, It becomes possible to automatically generate a time series model from multiple time series images without relying on non-rigid objects.

0010

DESCRIPTION OF THE PREFERRED EMBODIMENTS Next, the structure and operation of the present invention will be explained in detail with reference to FIGS. 1 to 4. In Figure 1, time-series label image 1 is a time-series label in which the object to be modeled is separated from the background, divided into regions for each texture, and unique labels are added to corresponding regions in the time series. This is an image (see (a) in FIG. 3). Here, (a
), the image time t (t=1, 2...T)
The frame image of is written as F(t).

The silhouette image generation unit 3 extracts the center of gravity position and principal axis of inertia direction of the label region at each time of the inputted time-series label image 1, and calculates the amount of movement of the region between F(t) and F(t+1). A region movement amount extraction unit 31 that calculates the movement amount, an area movement amount storage unit 32 that stores the calculated movement amount, an area adjacency relationship extraction unit 33 that extracts the two-dimensional adjacency relationship at each time of the label area, and this extracted two-dimensional Area adjacency relationship storage unit 34 that stores adjacency relationships, area movement amount storage unit 32
And with reference to the area adjacency relationship storage unit 34, even if the label areas that have a small relative movement between areas at all times of 1≦t<T and that maintain a two-dimensional adjacency relationship with each other with a high probability are in three-dimensional space, It is composed of an image generation unit 35 that creates a time-series silhouette image O (Gothic font) by merging these images, assuming that they have an adjacency relationship.

The model generation section 4 generates a time series model image M (Gothic font) from the time series silhouette image O (Gothic font), and includes a thinning section 41, a distance conversion section 42, and a feature point extraction section. 43, a feature point correspondence processing section 44, a structure point extraction section 46, and the like. The thinning unit 41 thins each frame image O(t) of the time-series silhouette image O (Gothic font) (processing to remove the shape in the silhouette image from its surroundings and extract the center line).

The distance conversion unit 42 converts the distance conversion value of the point on the thinned image (the shortest distance value between the background of the silhouette image and the point)
This is to generate an image C(t) whose pixel values are (see (c) in FIG. 3). The feature point extraction unit 43 extracts the image C(t)
The feature point candidates are determined at each time by referring to the following. As feature point candidates, the number of intersections is 0 on image C(t).
(internal points, isolated points) and points other than 2 (connected points). Also, a point with the largest pixel value (distance conversion value) among the images C(t) sandwiched between these points is also added to the feature point candidates. The coordinate values of the feature point candidates and the connection relationship between them on the image C(t) are extracted, and the Euclidean distance (shortest distance) between the two feature points in the connection relationship and the pixel value (distance) of the two points are extracted. If two points exist close to each other, the feature point candidate with the smaller pixel value is merged with the feature point candidate with the larger pixel value. The points that exist after merging are defined as feature points. Through the above processing, coordinate values and connection relationships of feature points are extracted for all times.

The feature point correspondence processing unit 44 associates feature points between time t and time (t+1). To associate feature points, first, the feature point coordinates at time t are time (t+1).
Estimate where the feature point will move based on the amount of movement of the region in the time-series label image F(t) where the feature point exists, and compare the estimated correction value with the feature point coordinate value at time (t+1). , by associating items with close Euclidean distance.

After completing the correspondence between the feature points between the two frames, the structure point extraction unit 46 assigns a unique label to the feature points that have a corresponding relationship in the time series, and assigns a unique label to each feature point label. Calculate the probability of occurrence in the time series. A feature point whose appearance probability exceeds a certain threshold is considered to be a representative point representing the structure of the object in the image, and is registered as a structure point. Furthermore, the connection relationships between structural points on the thinned image are extracted. The time-series position coordinates of these structural points and their connection relationships are used as a model of the object in the desired time-series image, and at each time, the structure points are represented as points on the image, and lines are drawn between the points according to the connection relationships. By tying them together, a time-series model image M (Gothic font) is output, which is an image of the model ((d) in Figure 3).
reference).

As described above, the adjacency relationship between the label areas in the three-dimensional space in the image is assumed based on the movement and adjacency relationship of the label areas in the time-series label image F on the two-dimensional image. Then, the label areas in the assumed adjacency relationship are merged to generate a silhouette image that reflects the structure of the object in three-dimensional space, and based on this, an image C(t) is created by thinning etc. Feature point candidates are determined based on the pixel value (distance conversion value) of the point on C(t) and the number of intersections. This makes it possible to automatically obtain feature point candidates that reflect the three-dimensional structure of the object in the image from any time-series image.

Next, by merging the extracted feature point candidates using their distance transformation values, the spatial image object is simplified. Furthermore, the temporal object is simplified by removing feature points that exist with low probability in the time series and connection relationships between feature points. Therefore, it is possible to extract a description of a spatially and temporally simple object, which is shown in Figure 3 (
As shown in d), a time-series model image M (Gothic font) is generated and output by connecting the structural points with lines according to the connection relationship.

Next, the overall operation of the configuration shown in FIG. 1 will be explained according to the flowchart shown in FIG. In FIG. 2, S1
imports the image F in which closed regions are labeled. This involves separating the image to be modeled from the background, dividing it into regions, and importing a time-series label image F with a unique label added (see (a) in FIG. 3). S2 calculates the amount of movement of the label area between frames and extracts the adjacency relationship of the label areas. This calculates the amount of movement of the center of gravity of the label region between frames for the time-series label image F imported in S1.

In step S3, label areas with a high probability of adjacency whose movement amount is less than a threshold are combined into a silhouette image O(
Gothic). This generates an image O (Gothic font) in which adjacent regions whose movement amount is equal to or less than the threshold are grouped together, as shown in the time-series silhouette image O (Gothic font) in FIG. 3(b). In S4, a thin line image is generated by removing the contour of the silhouette image from its periphery.

In step S5, the shortest distance between each pixel on the thinned image and the outline of the silhouette image is assigned as a pixel value. As a result, (c) time-series image C (Gothic font) in Figure 3
generate. S6 selects intersections, end points, branch points, etc. as feature point candidates, compares the straight-line distance between candidate points with connected pixel values and each pixel value, and selects the candidate points with larger pixel values among the nearby candidate points. Determine things as feature points. This is determined independently for each frame.

[0021] In S7, correspondence between feature points between frames is determined by associating feature points that are close in distance. In S8, feature points for which the probability of appearance of corresponding feature points between frames is equal to or higher than a threshold are determined as structural points, and are combined to create a time-series model image M (Gothic font).

As described above, the time-series label image F (Gothic font) shown in FIG. 3(a) in which closed regions are labeled is input, and those whose movement amount is less than the threshold are collectively shown in FIG. 3(b). Generate a silhouette image O (Gothic font),
Furthermore, after thinning this silhouette image O (Gothic font) and assigning the shortest distance of the thinned pixels to the outline of the silhouette image as a pixel value ((c) time series image C (Gothic font) in FIG. 3), Figure 3 (d) is a time-series model image M (Gothic font) in which feature points are extracted and those whose appearance probability between frames is equal to or higher than a threshold are defined as structural points and these structural points are connected with lines.
By generating and outputting , it becomes possible to automatically generate a time-series model image M (Gothic font) from a time-series label image F.

FIG. 3 shows a diagram illustrating a specific example of the present invention. (a) of FIG. 3 shows a time-series label image F. Here, F
(0), F(1), F(2) are at time t=1, t=2,
This is a time-series label image F when t=3. These time-series label images F(0), F(1), and F(2) are generated at each time when the object to be modeled is separated from the background, divided into regions, and fixed labels are assigned, as described above. This is a schematic representation of an image.

FIG. 3(b) shows a time-series silhouette image O
(Gothic). As mentioned above, this is shown in Figure 3.
An image (=
silhouette image). FIG. 3C shows a time-series image C (Gothic font) (non-zero pixels are displayed in black). As described above, this is an image in which the time-series silhouette image O (Gothic font) shown in FIG.

FIG. 3(d) shows a time-series model image M (Gothic font). As mentioned above, this (
c) Extract feature points from time-series image C (Gothic font), extract those that appear frequently between frames as structural points, and connect these structural points with line segments to create an image (time-series label image F). (image modeled on Gothic font).

Next, a more detailed explanation will be given using FIG. 4. FIG. 4 shows a configuration diagram of one embodiment of the present invention. In FIG. 4, an image input unit 1-1 reads an animation original image A (Gothic font) using an image input device such as a camera. Each frame image A(t) is converted into a temporary label image L(t) in which a temporary label is added for each closed area, and its color information is extracted and sent to the inter-frame area correspondence section 2.

The inter-frame area correspondence unit 2 determines the area correspondence between temporary label images, and generates a time-series label image F (Gothic font) in which a unique label is added to the corresponding area at all times. It is composed of a temporary label image storage section 21, a region feature extraction section 22, a region feature storage section 23, a similarity generation section 24, a region correspondence processing section 25, a region correspondence storage section 26, a label image rewriting section 27, etc. It is something that

The temporary label image storage section 21 is connected to the image input section 1.
It stores the temporary label image L(t) input from -1. The area feature extraction unit 22 refers to the temporary label image storage unit 21 and extracts area features (area, perimeter, complexity, etc.) for each label area of each frame image of the temporary label image L(t).
6 types (chain code histogram, center of gravity position, direction of principal axis of inertia) are extracted and stored in the area feature storage unit 23. Here, the area of the area feature is the number of pixels in the temporary label area, the perimeter is the number of pixels on the outline of the temporary label area, the complexity is the square of the perimeter divided by the area, and the chain code histogram is the principal axis of inertia. The outline of the label area when the direction matches the vertical axis of the image is expressed as a chain code in 8 directions, and the frequency table is accumulated for each code from 0 to 7. The center of gravity is the coordinate on the image of the center of gravity of the temporary label area. The value and principal axis of inertia direction is the counterclockwise angle formed by the principal axis of inertia of the temporary label area and the horizontal axis of the image.

The region feature storage unit 23 stores the above region features,
It stores color information, etc. The similarity creation unit 24 calculates the similarity of area shapes for all combinations from the area features (the above area features and color information) extracted from the area feature storage unit 23. The area correspondence processing unit 25 selects and determines the association of the temporary label areas with the highest degree of similarity based on the degree of similarity between the two frames of the temporary label areas calculated by the similarity generation unit 24. be. The results of the association are stored in the area correspondence storage section 26.

The label image rewriting unit 27 rewrites the image into a time-series label image to which a unique label is added at all times based on the correspondence of the temporary label areas taken out from the area correspondence storage unit 26. Through the above processing, a time-series label image F (as shown in FIG. 3(a)) in which unique labels are added to corresponding regions at all times (
Gothic).

As described above, the silhouette image generation unit 3 generates the (b) time-series silhouette image O (Gothic font) in FIG. 3 from the (a) time-series label image F (Gothic font) in FIG. It is something to do. In the figure, area movement amount extraction unit 31
, the area movement amount storage section 32, the area adjacency relationship extraction section 33, the area adjacency relationship storage section 34, and the image generation section 35 are the same as those in FIG.

The area movement amount extraction unit 31 refers to the label image storage unit 30, extracts the barycenter position coordinates (x, y) of the label area at time t, and extracts the barycenter position coordinates (x, y) of the label area at time t. The amount of change in position in the image is calculated and determined as the amount of movement of the label area. Adjacent area relationship extraction unit 33
extracts the two-dimensional adjacency relationship between label regions on the image F(t) at time t. The image generation unit 35 assumes that label regions that have a relatively small amount of movement between regions at all times and maintain a two-dimensional adjacency relationship with a high probability have an adjacency relationship in three-dimensional space, and merge them. Create a time-series silhouette image O (Gothic font).

In addition, as described above, the model generation unit 4 generates the time-series silhouette image O (Gothic font) shown in FIG. 3(b).
From (c) time-series image C (Gothic) (non-0
3 (d) time-series model image M (Gothic font) is generated. Thinning unit 41, distance conversion unit 42, feature point extraction unit 43, and feature point correspondence processing unit 4 in the figure
4. The structure point extraction unit 46 is the same as that shown in FIG. Further, the silhouette image storage unit 40 stores time-series silhouette images, and the feature point correspondence storage unit 45 stores information on the correspondence of feature points determined by the feature point correspondence processing unit 44.

The thinning section 41 includes the silhouette image storage section 4
0, the (b) time-series silhouette image O(
Gothic font), remove the area shape from its surroundings, perform thinning processing to extract the center line of the width of the shape, set the pixel value of the thinned area shape (center line) to 1, and set the pixel values of other points to 1. 0
Create an image. The distance conversion unit 42 calculates the distance conversion value (between the background and the point with the coordinate value) of the point on the time-series silhouette image O (Gothic font) that has the same coordinate value as the point with the pixel value 1 on this thinned image. , and generates a thinned distance-converted image C(t) using this distance-converted value as the pixel value at the same coordinates. Here, the distance conversion value is 0 for the background and 1 for a point on the area outline. This results in
The image C(t) is an image in which the coordinates of a point with a non-zero pixel value represent the center line coordinates of the target silhouette shape, and the pixel value of that point represents the thickness of the shape. The feature point extraction unit 43 refers to the time-series image C (Gothic font) and determines feature points at each time. First, the state of connection of non-zero points of image C (Gothic font) (t) is determined by the number of intersections of that point (when 8 neighborhoods of that point are considered as one round), from the background (pixel value = 0) to the area ( Feature point candidates are selected by determining the number of times the pixel value changes to (pixel value≠0)).Feature point candidates are points with 1 intersection (end point) and points with 3 intersections (branch point) on image C(t). , 4 points (intersections).Next, based on the pixel values (distance conversion values) of non-zero points in image C(t), the points sandwiched by the feature point candidates that have already been selected based on the number of intersections are determined. Add the point with the maximum pixel value as a feature point candidate in the line segment section (however, if the point with the maximum pixel value is already a feature point candidate, do not add it).Determine feature point candidates. This is performed by extracting the coordinate values of the feature point candidates and the connection relationship between them on the image C(t), and merging the feature point candidates in the connection relationship.

After extracting the coordinate values and connection relationships of the feature points at all times through the above processing, the feature point correspondence processing unit 44 associates the feature points between the two frames of time t and time (t+1). conduct. Then, the structure point extraction unit 46 uniquely labels the feature points that have a corresponding relationship in the time series at all times. The feature point at t=1 adds all unique labels. When t≠1, the same corresponding label is added by referring to the correspondence of the feature points stored in the feature point correspondence storage section 45, and when it does not exist, a new unused label is added. As a result, a unique label is added to the feature point at all times. Then, the connection relationship is rewritten according to the added label. Next, the appearance probabilities in all frames are extracted for each label, and feature points with appearance probabilities higher than a certain threshold are regarded as representative points representing the target structure in the time series and are registered, as well as structural points at each time. Extract the coordinate values and connection relationships. The model image generation unit 5 connects these structure points with line segments based on the coordinate values and connection relationships at each time, and connects these structure points with line segments, as shown in (d) of FIG.
A time-series model image M as shown in is generated and output as the structure of the target model in the time-series image.

A more detailed explanation will be given below using mathematical expressions and according to FIG. In FIG. 4, the interframe area correspondence unit 2
The area correspondence between temporary label images is determined, and a time-series label image F (Gothic font) is created in which a unique label is added to the corresponding area at all times. Region feature extraction unit 2
2 refers to the temporary label image storage unit 21, extracts the area feature for each label area of each frame image of L (Gothic font), and stores it in the area feature storage unit 23. There are six types of region features to be extracted: area, perimeter, complexity, chain code histogram, center of gravity position, and principal axis of inertia direction. The area is the number of pixels in the temporary label area, the perimeter is the number of pixels on the outline of the temporary label area, the complexity is the square of the perimeter divided by the area, and the chain code histogram is based on the principal axis of inertia as the vertical axis of the image. The contour of the label area when matching is expressed as a chain code in 8 directions, and the frequency table is accumulated for each code from 0 to 7. The center of gravity position is the coordinate value on the image of the center of gravity of the temporary label area, and the direction of the principal axis of inertia is This is the counterclockwise angle between the principal axis of inertia of the temporary label area and the horizontal axis of the image. The region feature extraction unit 22 stores the extracted region features in the region feature storage unit 23.

The similarity creation unit 24 and the area correspondence processing unit 25 perform the following processing between the two images L(t) and L(t+1). First, L(t) stored in the area feature storage unit 23
The region shape similarity Rt(k,l) is calculated for all combinations of k and l from the region feature of the region with temporary label k in L(t+1) and the region feature of the region with temporary label l in L(t+1). In this embodiment, the degree of similarity between the temporary label k area and the temporary label l area is calculated using the above six types of area features and the color information of the area.

[0038]

[Math 1]

Here, both "sum" and "difference" are L(t)
This is an operation between the area features of the temporary label k area in L(t+1) and the temporary label l area in L(t+1). The value of Rt(k,l) is
It takes a real value of [0, 1], and takes a value of 0 between temporary label areas of different colors in the original image, and for other temporary label areas, the smaller the difference in area features between the two areas, the closer it is to 1. It is set to take a value. The calculated similarity value is sent to the area correspondence processing unit 25. In the area correspondence processing unit 25,
Based on Rt (k, l), the temporary label k area corresponding to the temporary label l area is set to 1 so as to maximize the next evaluation function Et.
Choose one against the other. The correspondence between the temporary label l area and the temporary label k area is expressed by an ordered pair (k, l), and the entire correspondence is expressed by a set Φt of ordered pairs.

[0040]

[Math 2]

[0041] Correspondence result Φt between two frames in the temporary label area
is stored in the area correspondence storage section 26. After Φt is extracted and stored for all times t below (T-1), the label image rewriting unit 27 rewrites the label of the time series temporary label image L (Gothic font) to label the time series Create image F (Gothic font).

[0042] The label image rewriting unit 27 first reads L(1)
A label image F(1) is created by adding unique labels to all the regions therein. For images with t≠1,
Referring to Φ(t-1) from the area correspondence storage unit 26, L(
t), the corresponding L(t
-1) If the area in the middle exists, add the same unique label that was added to the corresponding area, and if there is no corresponding area, add a new label that has not been used by time t. The image F(t) is created by adding the image F(t). F (Gothic font) created in this way is a time-series label image in which a unique label is added to the corresponding area at all times, and becomes an input image to the silhouette image generation unit 3 (Fig. 3 (see (a)).

The time-series image F (Gothic font) input to the silhouette image generation section 3 is stored in the label image storage section 30. The area movement amount extraction unit 31 refers to the label image storage unit 30, extracts the barycenter position coordinates (Xit, Yit) of the label area i at time t, and extracts the coordinates (Xit, Yit) of the center of gravity of the label area i at time t, and extracts the coordinates (Xit, Yit) of the center of gravity of the label area i at time t, and extracts the coordinates (Xit, Yit) of the center of gravity of the label area i at time t. The amount of change in the center of gravity position in the image between and F(t+1) is calculated and used as the amount of movement of the label area i.

Amount of movement in the X-axis direction: ΔXit=Xi(t+1)−XitY
Axial movement amount: ΔYit=Yi(t+1)−Yit However, when F(t) does not include the label area i in F(t+1), the movement amount is set to 0. Calculated ΔXit, Δ
Yit is stored in the area movement amount storage unit 32.

The area adjacency relationship extraction unit 33 extracts the two-dimensional adjacency relationship between the label i area and the label area j on the image F(t) at time t. This two-dimensional adjacency relationship Nt(i,j)
has a value defined as follows. Nt (i, j) = 1 (when i and j are adjacent with the contour line in between)
0 (if i and j are not adjacent across the outline)
The case where Nt(i,j) is 0 includes the case where i or j does not exist in F(t). Extract Nt (i, j) for all t, i, j, and store it in the area adjacency relationship storage unit 3.
Store in 4.

The image generation section 35 stores the area movement amount storage section 32.
With reference to the area adjacency relationship storage unit 34, it is determined that label areas that have a small relative movement between areas at all times and maintain a two-dimensional adjacency relationship with a high probability also have an adjacency relationship in the three-dimensional space. A time-series silhouette image O (Gothic font) is created by merging them. in particular,
First, the image generation unit 35 calculates from time t to time (
t+1), the normalized relative movement amount of label areas i and j is calculated.

[0047]

[Math 3]

Using this normalized relative movement amount ΔDt (i, j) and the two-dimensional adjacency relationship Nt (i, j) stored in the area adjacency relationship storage unit 34, the adjacency relationship of the label areas at time t is calculated. Redefine as follows. N't(i,j)=1 (ΔDt(i,j)<α
t and Nt(i,j)=1)
0 (When the above conditions are not satisfied) Here, αt is a threshold value for setting N't(i, j), and in this embodiment, it takes a value of 0.0. Using N't(i,j), the adjacency relationship Ξ(i,j) in the three-dimensional space of the label region is determined. Ξ(i,j) takes a value of 1 (connected) for i and j that satisfy the following formula, and otherwise takes a value of 0 (unconnected).

[0049]

[Math 4]

[0050] The left side is the area i and j in F (Gothic).
is the appearance probability of frames with an adjacency relationship, and β on the right side is the appearance probability threshold for setting Ξ (i, j),
In this embodiment, the value is assumed to be 0.9. Ξ(i,j
), the image generation unit 35 fills in the outline between label areas i and j with label i or j in each frame image in F (Gothic), and merges i and j. After that, the background and the contour parts that are not filled with labels are converted to pixel values of 0, and all label areas are converted to pixel values of 1, creating a time-series silhouette image O (Gothic font) (see (b) in Figure 3). ).

The model generation unit 4 generates a target model by inputting the time-series silhouette images O (Gothic font). O (Gothic font) is input to the silhouette image storage section 40 and stored. The thinning unit 41 refers to the silhouette image storage unit 40 and performs thinning processing on each frame image O(t) of O (Gothic font) to create a thinned image C0(t).
generate. The thinning process is a process of removing the shape of a region with a pixel value of 1 in O(t) from the periphery and extracting the center line of the shape with a width of 1. C0(t) is the thinned region shape (
The pixel value of the center line) is set to 1, and the other pixel values are set to 0. The distance converter 42 converts this thinned image C0
Find the distance conversion value of a point on O(t) that has the same coordinate value as the point with pixel value 1 on (t), and use this distance conversion value as the pixel value of the same coordinate in the thinned distance conversion image C(t ). The distance conversion value of any coordinate on O(t) is defined as the shortest distance value between the background (pixel value 0) and the point with that coordinate value,
The distance conversion value of the background takes a value of 0, the value of a point on the area outline takes a value of 1, and the value becomes larger for a point inside the area. Therefore, in C(t), the coordinates of a point with a non-zero pixel value represent the center line coordinates of the target silhouette shape (see (c) in Figure 3), and the pixel value of that point represents the thickness of the shape. It becomes an image.

The feature point extraction unit 43 refers to C(t) and determines feature points at each time. The feature point extraction unit 43
First, the state of connection of non-zero points of C(t) is determined by the number of intersections of the points, and feature point candidates are selected. The number of intersections of an arbitrary point is the number of times it transitions from the background (pixel value=0) to inside the area (pixel value≠0) when 8 neighborhoods of that point are considered as one round. As a feature point candidate, a point with the number of intersections of 1 on C(t) (
(end point), point 3 (branch point), and point 4 (intersection). Next, the feature point extraction unit 43 extracts a line between the feature point candidates selected by determining the number of intersections based on the pixel value (distance conversion value) of the non-zero point of C(t). Add the point with the maximum pixel value as a feature point candidate in the minute interval (however, if the point with the maximum pixel value is already a feature point candidate, it will not be added). The feature points are determined by extracting the coordinate values of the feature point candidates and the connection relationships between them on C(t), and merging the feature point candidates in the connection relationships. Two minutiae candidates are in a connected relationship if there is a path connecting the two minutiae candidates using non-zero points on C(t), and there are no other minutiae candidates on that path. means. When adjacent feature point candidates satisfy the following equation 5, the feature point candidate with a small distance conversion value is merged with the feature point candidate with a large distance conversion value.

[0053]

[Math 5]

However, b1 and b2 are the larger value and the smaller value of the distance conversion values of adjacent feature point candidates, respectively. When there are no more feature point candidates that can be merged, the remaining feature point candidates are determined as target feature points at time t, labels pt (pt=1,...Pt) are added, and the feature point coordinates (Xf [pt], Yf[pt]) and connection relation Jt
Extract (pt, p't).

Jt(pt, p't)=0 (pt=P't)
1
(When there is a connection between pt and p't)
0 (pt,
(When there is no connection relationship between p't) However, at the time of extracting this connection relationship, a feature point candidate that has been merged with another feature point is considered to have the same label as the feature point to which it has been merged. This makes it possible to avoid generating new connection relationships by merging feature point candidates.

After extracting the coordinate values and connection relationships of the feature points at all times, the feature point correspondence processing unit 44 extracts the coordinate values and connection relationships of the feature points at times t and (t
+1) The feature points are associated between the two frames. In order to associate feature points, first find out where the feature point at time t will move at time (t+1) using F where the feature point exists.
The coordinate values are estimated based on the amount of movement of the label area in (t) and corrected. The amount of movement of the label area in F(t) is stored in the area movement amount storage section 32. As a result, the coordinate values of the feature point pt after correction (Xm'[pt], Ym'[p
t]) is expressed as follows.

Xm'[pt]=Xf[pt]+ΔXitYm'[pt
]=Yf[pt]+ΔYit, where i is the label image F
(t) This is the value of the label of the label area including the point with the coordinate values (Xf[pt], Yf[pt]) on the top. Next, this correction value and the feature point p(t+1) at time (t+1) (p(t+1
)=1, . . . P(t+1)) with the shortest Euclidean distance are associated on a one-to-one basis. This correspondence result is the ordered pair (pt, p
(t+1)) is stored in the feature point correspondence storage unit 45 as a set Гt.

Гt={(pt, p(t+1)|pt is
The feature point at time t that corresponds one-to-one to the feature point p(t+1) at time (t+1)} The structure point extracting unit 46 extracts a unique feature point at all times for feature points that have a corresponding relationship in the time series. Label. t=
A unique label is added to each feature point. When t≠1, refer to Гt stored in the feature point correspondence storage unit 45, and if a corresponding feature point exists during time (t-1), use the same label as the corresponding feature point. If the label does not exist, a new unused label is added. As a result, a unique label ρ is added to the feature point at all times. Then, according to the added label ρ, the connection relation Jt(pt, p't) is changed to J't(ρ1, ρ
2) can be rewritten.

J't(ρ1, ρ2)=Jt(pt, p't) However,
ρ1 and ρ2 are labels newly added to pt and p't. Next, the structure point extraction unit 46 extracts the frequency of appearance in all frames for each label ρ, and determines the following condition. [Frequency of appearance of ρ in all time series]/T≧Φ・・・・
・・・・・・・・・■The feature point ρ that satisfies this conditional expression is regarded as a representative point representing the target structure in the time series, and the structural point ω
Register as. Here, the left side is the appearance probability of ρ, φ is a threshold value for setting ω, and the registration result is expressed as a set Ω of ω.

Ω={ω|ω=[Feature point ρ that satisfies ■]}
Also, the coordinate values of the structure point at each time (Xst[ω],
Yst[ω]) is extracted. The connection relationship G(ω1, ω2) of these structural points is the connection relationship J't(
ρ1, ρ2) based on Equation 6 below.

[0061]

[Math 6]

Here, ρ1 and ρ2 are feature point labels corresponding to the structural points ω1 and ω2, and ξ is a threshold value for determining G(ω1, ω2), and in this example, ξ=φ=0 .9
shall take the value of . The structure point extraction unit 46 extracts the coordinate values (Xst[ω], Yst
[ω]) and the connection relationship G(ω1, ω2) are output as the structure of the target model in the time-series images.

The model image generation unit 5 obtains (Xst[ω], Yst[ω]) from the structure point extraction unit 46 of the model generation unit 4.
and G (ω1, ω2) are input, and a time-series model image M (Gothic font) is created. Model image M(
t) is the coordinate value (Xst[ω], Yst[ω]
]), and the connection relationship between the structural points ω1 and ω2 is expressed as (
Xst[ω1], Yst[ω1]) and (Xst[ω2]
, Yst[ω2]) (see (d) in FIG. 3). In this embodiment, the process ends after outputting this time-series model image M (Gothic font).

[0064]

[Effects of the Invention] As explained above, according to the present invention,
Silhouette images that are a collection of time-series images whose movement amount is less than a predetermined threshold are thinned, feature points are extracted, and these feature points are connected to generate a time-series model. A time series model can be automatically generated from multiple time series images without relying on non-rigid objects. In addition, as the information extracted to generate the model uses feature points that can be automatically extracted from the image, there is no need for people to specify joint points on the image as in the past, and automatic model generation is possible. It becomes possible to do so.

[Brief explanation of the drawing]

FIG. 1 is a diagram showing the principle configuration of the present invention.

FIG. 2 is a flowchart explaining the operation of the present invention.

FIG. 3 is a diagram illustrating a specific example of the present invention.

FIG. 4 is a configuration diagram of one embodiment of the present invention.

[Explanation of symbols]

1: Time series label image 2: Inter-frame area correspondence part 3: Silhouette image generation section 31: Area movement amount extraction part 33: Region adjacency relationship extraction part 35: Image generation section 4: Model generation section 41: Thinning part 42: Distance conversion section 43: Feature point extraction section 44: Feature point correspondence processing unit 46: Structure point extraction part 5: Model image generation section

Claims (5)

[Claims] Claim 1: In a creation device that creates a model from time-series images, a time-series label image (1) in which areas are labeled is input, and a silhouette is created that summarizes the movement amount of the labeled area at each time is less than a threshold value. A silhouette image generation unit (3) that generates an image, and a model generation unit that thins the generated silhouette image, extracts feature points, and considers the appearance rate of the corresponding feature point between frames as a structural point to be a structural point. 1. A model creation device from time-series images, comprising: (4) and configured to output a time-series model image in which these structural points are connected by line segments. Claim 2: The silhouette image generation unit (3) comprises:
As input is a time-series label image (1) in which areas are labeled, and based on the amount of movement of the label area at each time and the adjacency relationship between the label areas, we summarize the items whose movement amount of the label area at each time is less than a threshold value. 2. The model creation device from time-series images according to claim 1, wherein the device is configured to generate a silhouette image reflecting the structure of the object in three-dimensional space. 3. Create a thin line image by thinning the silhouette image, generate an image in which the shortest distance between each pixel on the thin line image and the silhouette image is assigned as a pixel value, and Feature point candidates are extracted from the number of intersections and pixel values, and the feature point candidates are merged to form the feature point based on the distance between these feature point candidates and each pixel value. A model creation device from time-series images according to claim 1. 4. The coordinates of the feature point at time t corrected by the amount of movement of the label area on the time-series label where the feature point exists are the closest to the coordinates of the feature point at time (t+1). 2. The model creation device from time-series images according to claim 1, wherein the model creation device is configured to create corresponding feature points between two frames. 5. The correspondence between feature points in the entire time series is determined based on the feature points corresponding between two frames, and the probability of appearance of the corresponding feature point in the time series is greater than or equal to a threshold is structured. 2. The model creation device from time-series images according to claim 1, wherein the model creation device is configured to form a point.