JP2001101419A - Method and device for image feature tracking processing and three-dimensional data preparing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To suppress the occurrence of error in three-dimensional(3D) information by surely tracking an object while verifying the suitability of tracking in the case of tracking the object within a time sequential image group. SOLUTION: Plural images provided by picking up the image of a stationary object at different plural positions with a TV camera are inputted from an image input device 1, and a surface selected from the object by an area designating means 16 is set within the image as a tracking area expressed by boundary elements and image feature amounts. An area tracking means 17 generates the candidates of the tracking area on the basis of the boundary elements inside the image, generates a deformed area from the tracking area, compares the candidates of the tracking area with the deformed area and determines the tracking area. On the basis of the position relation of a reference area with the object restored by a form restoring means 19, it is verified by a tracking evaluating means 18 whether a non-picked-up image part is included in the tracking area or not. Besides, when there is a non-picked-up image part, the cause is estimated.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image feature tracking processing method, an image feature tracking processing apparatus and a three-dimensional image extracting method for extracting three-dimensional information on an object included in a time-series image group (moving image) as a two-dimensional image. It relates to the data creation method.

[0002]

2. Description of the Related Art In recent years, computer graphics (hereinafter abbreviated as 3DCG) technology has rapidly advanced. Such 3DCG technology is also used for CAD systems and virtual reality systems. However, 3DC
The modeling work to create G data takes a lot of time, which is the main reason that the application field of 3DCG cannot be expanded.

[0003] In order to facilitate the modeling work, a technique for automating the modeling work by performing measurement in a real space using a three-dimensional measuring device (such as motion capture) and converting the measured values into 3DCG data is practically used. However, the three-dimensional measurement device is very expensive because it performs stereoscopic vision, and the current three-dimensional measurement device has a relatively small space area that can be measured, so the application of this technology is limited. Have been. In particular, this type of three-dimensional measurement device
It is not suitable for measuring large spatial areas such as urban areas,
At present, in order to form a cityscape or the like by 3DCG, 3DCG data is manually created.

[0004] Therefore, as a technique that can be used for various objects in general while performing three-dimensional measurement using a relatively inexpensive device, a technique based on a two-dimensional moving image (time-series image group) by monocular vision is used. A technique for restoring a three-dimensional shape has been proposed. For example, "Takeo Kanade et al .: Reconstruction of Object Shape and Motion of Imaging Device by Factor Decomposition Method, Journal of the Institute of Electronics, Information and Communication Engineers, Vol. J76-D-II No. 8,
In the technique described in “1993/8” (hereinafter, referred to as factor decomposition method), feature points obtained by recognizing a feature on a two-dimensional moving image and tracking how the feature changes as a moving image. A matrix is created, and the three-dimensional shape of the object and the camera posture are restored by applying a factorization method to the feature point matrix. In this technique, if there is an error in the tracking processing for creating the feature point matrix, the error of the measurement result increases. Therefore, it is important to accurately track the feature on the moving image.

A technique for tracking a feature in a two-dimensional moving image for the purpose of three-dimensional measurement is disclosed in JP-A-10-11193.
There is a technique described in Japanese Patent Application Laid-Open No. 4 (Kokai) 4 In this publication, when applying the factorization method, by specifying a plurality of regions for one frame of image, applying a plurality of types of feature extraction methods to each region, and adopting an optimal result, It describes that accurate feature point tracking is performed.

Japanese Patent Application Laid-Open No. H10-255053 discloses that in applying the stereo method, after tracking a feature point, an appropriate point in an image is set as an image origin, and the rotation of the imaging device is controlled. The motion trajectory of the feature point is obtained by using the distance between the origin as a feature point that does not change and another feature point, and it is evaluated whether or not the motion trajectory has correlation with two different image groups, and tracking is performed. It describes that a three-dimensional shape is restored with high accuracy by detecting the presence or absence of a failure and using only correctly tracked feature points.

[0007]

However, according to the technique described in Japanese Patent Application Laid-Open No. H10-111934, it is necessary for an operator to determine and instruct whether the tracking of a feature point is correct or not. There is a problem that the operator cannot leave the apparatus.

In the technique described in Japanese Patent Application Laid-Open No. H10-255053, it is possible to detect a failure in feature tracking, but it is not possible to determine the reason for failure. When it disappears, it is not possible to determine whether it is a tracking error or another cause (such as when there are multiple objects and the object being tracked is hidden by other objects), and corrects the failure Therefore, there may be a portion in the image that cannot be measured.

In addition, since both of the above-described techniques track using local features extracted from an image, there is a problem that the tracking is likely to fail when the pattern has a repetitive pattern such as a lattice pattern. is there. In addition, the three-dimensional information obtained by any of the above-described techniques is the coordinates of each feature point, and the coordinates of each feature point are only discrete coordinate information in a three-dimensional space, and the relevance between the feature points is Unknown. Therefore, in order to use the information measured by the above-described technology as 3DCG data, another processing means for generating data having surface information such as polygons from discrete coordinate information is required.

The present invention has been made in view of the above circumstances, and an object thereof is to easily obtain three-dimensional information that can be used for 3DCG data from a time-series image group that is a two-dimensional image. And an image feature tracking processing method capable of verifying the validity of tracking when tracking an object in a time-series image group, reliably tracking the object, and suppressing the occurrence of errors in three-dimensional information ,
An object of the present invention is to provide an image feature tracking processing device and a three-dimensional data creation method.

[0011]

According to a first aspect of the present invention, a stationary object is imaged from a plurality of different positions by a TV camera to acquire a plurality of images, and a surface selected from the object is defined as a boundary element. Creates a deformed area that is set in the image as a tracking area represented by and an image feature quantity, and the tracking area set in one image is deformed to match the boundary element of the tracking area candidate in another image Then, by comparing at least one of a boundary element and an image feature amount with respect to a candidate for a tracking area and a deformation area in the other image, a tracking corresponding to the tracking area set in the one image from the candidates for the tracking area. A first process of selecting an area and a tracking area having a small number of unimaged areas such as concealment and frame-out
A reference area obtained by mapping in a three-dimensional space is set, and when at least a part of the tracking area cannot be tracked in an image, a TV camera is set based on a comparison of a boundary element between the tracking area and the reference area in a three-dimensional space. A second process of determining whether an area that cannot be imaged has occurred due to
The cause of the inability to image the tracking area with the TV camera is estimated based on the positional relationship of the reference area when the reference area is imaged at each position of the TV camera, and the tracking area can be tracked based on the estimation result. And a third process of continuing the tracking by making the above correction.

According to a second aspect of the present invention, in the first aspect, in the first step, a candidate corresponding to a boundary element of the tracking area set in one image on another image is extracted, and After generating a tracking candidate based on a combination of element candidates, a deformation area is generated by deforming the tracking area set in the one image so as to match the tracking candidate, and the shape of the boundary element is determined for the tracking candidate and the deformation area. The tracking area in the other image is determined from the tracking candidates by comparing an index selected from the pixel value and the image feature amount.

According to a third aspect of the present invention, in the second aspect, an edge is extracted from the another image,
Among the edges, an edge whose distance and direction to a line element constituting the tracking area in the one image are within a specified range is selected as a line element candidate.

According to a fourth aspect of the present invention, in the second aspect, an edge is extracted from the other image,
An edge having continuity is extracted by performing a Hough transform on each pixel on the edge, and an edge whose distance and direction with a line element constituting a tracking area in the one image are within a specified range is extracted. It is characterized in that it is selected as a line element candidate.

According to a fifth aspect of the present invention, in the second aspect of the present invention, a shape near a point element constituting a tracking area in the one image is used as a template, and a point element from a portion matching the template in the other image is used. Is extracted.

According to a sixth aspect of the present invention, in the second aspect, pixel values of the tracking candidate and the deformation area are used as the index.

According to a seventh aspect of the present invention, in the second aspect, an average luminance of the tracking area and the tracking candidate is used as the index.

According to an eighth aspect of the present invention, in the second aspect, the image is a color image, and the color of the tracking area and the tracking candidate is used as the index.

According to a ninth aspect of the present invention, in the second aspect, a spatial frequency distribution of the tracking area and the tracking candidate is used as the index.

According to a tenth aspect of the present invention, in the second aspect of the present invention, a portion where a change in pixel value exceeds a specified value in the tracking region and the tracking candidate region is extracted, and the position of this portion,
At least one element of a shape and an image feature is used as the index.

According to an eleventh aspect of the present invention, in the second aspect, the image is a color image, wherein the average luminance or color of the tracking area and the tracking candidate is used as the index, Is used as the index, and a case where a change in the pixel value within a region exceeds a specified value is extracted, and the position, shape, and at least one element of an image feature of this part are used as the index, The selection is made according to the luminance and color distribution patterns in the tracking area in the image.

According to a twelfth aspect of the present invention, in accordance with the second aspect of the present invention, the attitude of the TV camera when the average luminance of the plurality of images in the plurality of tracking areas is maximized is determined on the surface of the object corresponding to each tracking area. And the direction of the reflected light is estimated from the direction of the normal to the surface of the object corresponding to each tracking area and the direction of the reflected light.

According to a thirteenth aspect of the present invention, in the first aspect, in the second step, a tracking area corresponding to a plurality of images is mapped in a three-dimensional space, and a correspondence between boundary elements of the tracking area is mapped. The tracking area mapped in the three-dimensional space is used as the reference area when the degree of coincidence obtained from is equal to or greater than the threshold, and the boundary element between the tracking area corresponding to the reference area and the reference area is compared. It is characterized in that an area in which no image is captured is extracted.

According to a fourteenth aspect, in the thirteenth aspect, the number of boundary elements is limited to a range in which the number of boundary elements does not change between a plurality of images. , A tracking area having a contribution rate equal to or more than a threshold is used as a reference area candidate, and when a plurality of reference area candidates are obtained, a candidate having the largest area is adopted as the reference area.

According to a fifteenth aspect, in the fourteenth aspect, the change in the number of boundary elements focuses on one of a line element and a point element.

According to a sixteenth aspect, in the fourteenth aspect, the contribution ratio is obtained from a diagonal matrix component obtained when the tracking area is mapped into a three-dimensional space by a factorization method. .

According to a seventeenth aspect of the present invention, in the thirteenth aspect, the tracking area is mapped in a three-dimensional space and compared with a reference area, and an area defined by point elements which do not correspond to each other is imaged in the tracking area. It is characterized in that it is determined as an area that does not exist.

According to an eighteenth aspect of the present invention, in the thirteenth aspect, a tracking area is mapped in a three-dimensional space and compared with a reference area. It is characterized in that it is regarded as a part of a region in which no image is taken.

According to a nineteenth aspect of the present invention, in the first aspect of the invention, in the third step, all the reference areas in the three-dimensional space are projected on an image plane determined by the position of the TV camera. Is characterized by determining the cause of the occurrence of an area that is not imaged in the tracking area based on the positional relationship between the reference areas and the positional relationship of the reference area with respect to the image plane.

According to a twentieth aspect of the present invention, in the invention of the nineteenth aspect, when another reference area exists between the reference area of interest and the TV camera and both reference areas are not connected, the concealment of others is performed. Is determined.

According to a twenty-first aspect, in the nineteenth aspect, when there is another reference area between the reference area of interest and the TV camera and both reference areas are connected, self-concealment is performed. It is characterized by determining.

A twenty-second aspect of the present invention is characterized in that, in the nineteenth aspect, it is determined that the frame is out when the reference region of interest is located on the periphery of the screen.

According to a twenty-third aspect of the present invention, in the nineteenth aspect of the present invention, when an area that is not imaged in the tracking area exists, edge extension is performed, and the intersection of the extended edge is set as a new point element. The tracking processing is executed again, and if the contribution rate is improved when mapping to the three-dimensional space, it is determined to be concealment.

According to a twenty-fourth aspect of the present invention, there is provided an image input unit for providing an image input image picked up by a TV camera, and an image processing apparatus for processing the input image according to the image feature tracking processing method according to the first aspect. A storage device for storing an input image and an image processed by the image processing device, and a display unit for displaying a processed image in the image processing device;
And an area designating means for designating a tracking area for the image processing apparatus.

According to a twenty-fifth aspect of the present invention, after a plurality of images are obtained by picking up an object from a plurality of different positions using a TV camera, a plane selected from the object is set as a tracking area represented by a boundary element in the image. Set, and create a deformation area by deforming the tracking area set in one image so as to match the boundary element of the tracking area candidate in another image, and create a tracking area candidate and a deformation area in the other image. And selecting a tracking area corresponding to the tracking area set in the one image from the tracking area candidates by comparing the boundary elements with each other, and further, a plurality of surfaces of the object are imaged and continuously the same surface Is divided into one phase image group, and then the mapping of the plurality of phase images to the three-dimensional space of the surface selected from the object for each phase image group is performed. Determined the three-dimensional shape for each phase image group by performing grayed, by performing coordinate conversion so as to match each other coordinate system of the other phase images, characterized by creating a three-dimensional data.

According to a twenty-sixth aspect of the present invention, after a plurality of images are obtained by capturing an image of an object from a plurality of different positions with a TV camera, a plane selected from the object is set as a tracking area represented by a boundary element in the image. Set, and create a deformation area by deforming the tracking area set in one image so as to match the boundary element of the tracking area candidate in the other image, and create a tracking area candidate and a deformation area in the other image. And selecting a tracking area corresponding to the tracking area set in the one image from the candidates for the tracking area by comparing the boundary elements with each other, and further, a plurality of surfaces of the object are imaged and continuously the same surface Is divided into one phase image group, and then the mapping of the plurality of phase images to the three-dimensional space of the surface selected from the object for each phase image group is performed. The three-dimensional shape is obtained for each phase image group, and the coordinate transformation of the three-dimensional shape obtained for each phase image group is performed so that line elements common to other phase image groups are superimposed on each other. Is performed to create three-dimensional data.

According to a twenty-seventh aspect of the present invention, in the twenty-fifth or twenty-sixth aspect, a surface selected from the object in the plurality of images captured by the TV camera is defined as a point element and each point. Expressed as boundary elements consisting of directed line elements that sequentially connect elements, monitor the direction of tracing each point element by tracing the line element in each image obtained in time series,
It is characterized that one phase image group is taken until the traveling direction is reversed.

According to a twenty-eighth aspect of the present invention, in the twenty-fifth or twenty-sixth aspect of the present invention, in the coordinate conversion, after obtaining conversion parameters relating to rotation and translation for each boundary element, the conversion parameters obtained for each boundary element are obtained. The coordinate transformation is performed using the transformation parameters obtained by averaging the respective coordinates.

According to a twenty-ninth aspect of the present invention, in the invention of the twenty-fifth or twenty-sixth aspect, the position of each point element after performing the coordinate transformation on one of the three-dimensional shapes whose coordinate systems match each other and the other three-dimensional shape. It is characterized in that the midpoint between the position of each corresponding point element in the dimensional shape is the position of each point element.

[0040]

DESCRIPTION OF THE PREFERRED EMBODIMENTS (First Embodiment) FIG. 1 shows an apparatus used in the present invention. The two-dimensional image to be processed is
The image is input from the image input device 11 to the image feature tracking processing device 10. As the image input device 11, a playback device that plays back a video from a TV camera or a recording medium (video tape, CD, DVD, or the like) that records a moving image captured by the TV camera is used.
Is configured using a computer device (personal computer). As a TV camera (hereinafter abbreviated as a camera), a simple video camera for home use may be used, and an ITV camera for industrial use may be used. In the embodiments described below, a color image is targeted.

The image input to the image feature tracking processing device 10 is continuously moved from the viewpoint of the object 1 by moving the camera 2 with respect to the object 1 as shown by an arrow A as shown in FIG. Use an image that has been dynamically changed. That is,
A time-series image group (moving image) of a two-dimensional image in which the viewpoint with respect to the object 1 is continuously changed is used as the image feature tracking processing device 10.
Is input to Here, the image input to the image input unit 12 is imaged such that the moving amount of the target 1 between the frames of the moving image is sufficiently smaller than the size of the target 1. That is, the camera 2 that captures a moving image moves at a relatively low speed. It is also assumed that the object 1 has a relatively large number of flat portions, and that the object 1 does not move and does not change its shape with time. FIG. 3 shows an example of a moving image obtained by imaging the object 1 shown in FIG. In FIG. 3, one frame means one image V1 to V3,
A state is shown in which images V1 to V3 are arranged in chronological order from left to right.

The image feature tracking processor 10 is basically provided for storing image input means 12 serving as an interface with the image input apparatus 11 and for storing moving images and processing results input through the image input means 12. Storage device 13 comprising a hard disk and a memory, an image processing device 14 for extracting three-dimensional information from a time-series image group as an input two-dimensional image, and a display device for displaying input moving images and processing results And an area designating unit 16 including a keyboard and a mouse for setting a tracking area and the like corresponding to the object 1 with respect to the image processing apparatus 14.

The storage device 13 stores an image file F1 for storing an image input through the image input means 12,
A tracking data file F2 used as a work file when tracking the target 1 for obtaining three-dimensional information in a time-series image group that is a two-dimensional image, and three-dimensional information obtained by the image processing device 14 are stored. A three-dimensional shape data file F3 is provided.

The image processing apparatus 14 includes an area tracking means 17 for tracking the object 1 based on the tracking area set for the object 1 by the area specifying means 16 described above. Tracking evaluation means 18 for evaluating the validity of the tracking of the object 1 by the following. Also,
The image processing device 14 is provided with a shape restoring unit 19 that generates three-dimensional information from information in the two-dimensional image of the tracked object 1. The three-dimensional information generated by the shape restoring unit 19 is used for evaluating the validity of the tracking performed by the tracking evaluating unit 18.

Hereinafter, the operation of the image processing apparatus 14 will be described. First, when an image is input through the image input unit 12, the input image is stored in the image file F1 in the storage device 13.
And displayed on the display means 15. Here, the operator operating the area designating means 16 is
The initial tracking area is specified in association with the target object 1 by looking at the image displayed in. That is, one image is selected from the time-series image group, and an initial tracking area is set in this image.
Generally, the tracking area is set for each surface of the object 1, and all the surfaces of the object 1 displayed in one image are designated as the tracking area. By specifying the tracking area in this way, unnecessary information such as the background is removed. FIG. 4 shows a state where the tracking area D is designated. The tracking area D is a closed area composed of a set of boundary elements, and uses point elements and line elements connecting the point elements as the boundary elements. In FIG. 4, a white circle is a point element, and a line between two adjacent point elements represents a line element. The boundary element is a tracking area D represented by whether or not a line element is shared by the two tracking areas D.
The connection relationship between the two (the bold line in FIG. 4 indicates a line element straddling the two tracking areas D), and image characteristics (color, texture, etc.) of the area
May be included.

As described above, the format for expressing a three-dimensional shape by line elements and point elements as boundary elements is a boundary expression (B-
3DCG called REP (boundary representation)
Is commonly used. If the input image information is converted into the boundary representation, the data amount is significantly reduced as compared with the case where the image information is handled in pixel units, and the data conversion becomes easy when using three-dimensional data. FIG. 5 shows a setting example of the boundary area D by the boundary expression. In FIG. 5, the hatched portion indicates the boundary region D, and the line elements s1 to s8 and the point element p
An annular area is set by 1 to p8.

The area specifying means 16 can specify not only the tracking area D for the object 1 but also the accuracy with which the object 1 is measured. For example, drawer 2 as shown in FIG.
If three-dimensional measurement is performed on the furniture 20 having the furniture 1, the furniture 22 may be measured up to the handle 22 provided on the drawer 21, or the furniture 2 may be measured.
It is possible to specify the accuracy of whether or not to measure the entirety of 0 as a rectangular parallelepiped. When the object 1 in the image is not a complicated shape like furniture but a simple shape such as a combination of geometric shapes, each surface of the object 1 is automatically set as a tracking area D. You may make it set. That is, a region dividing unit that divides the internal region of the object 1 for each surface may be provided.

The tracking area D is stored in the tracking data file F2 in the storage device 13. The area tracking means 17
An area corresponding to the tracking area D set by the area specifying means 16 in each image of the time-series image group is detected. That is, the target surface of the object 1 is tracked in each image of the time-series image group. In order to track the tracking area D in each image of the time-series image group, the area tracking unit 17 uses one image (1) of the time-series image group stored in the image file F1.
A function of estimating the shape of the tracking area D of the next frame from the tracking area D set for the frame) and generating a deformation area. The obtained deformed area is compared with the image of the next frame, and the position of the deformed area in the next frame is detected. Then, the position of the deformation area in the next frame is detected by regarding the deformation area as a tracking area in the next frame. In this way, the deformation areas are successively generated for all the frames, and the positions of the deformation areas are stored as tracking data in the tracking data file F2.

The function of the area tracking means 17 will be described in more detail. FIG. 7 shows one image (1) of the time-series image group.
(Frame), only one object 1 is shown. The tracking area (hatched portion) D in FIG. 7 is set in the immediately preceding frame in the time-series image group, and does not match any surface of the object 1 in the frame shown in FIG. Therefore, the area tracking means 17 detects which part in the frame each line element sA, sB, sC of the tracking area D corresponds to. Here, Object 1
Is an image from which edges are extracted (referred to as an edge image). A well-known technique is used for edge extraction.

The area tracking means 17 generates a deformation area based on a line element and generates a deformation area based on a point element. When generating a deformation area based on a line element, first, an edge whose direction and distance are within a predetermined range with respect to each of the line elements sA, sB, and sC is extracted from the edges of the object 1. That is, by performing the Hough transform on the edge image of the object 1, the inclination of the straight line portion in the edge image can be known. Therefore, each straight line portion and each line element s in the edge image obtained by the Hough transform are obtained.
The angle difference between A, sB, and sC is determined, and when the angle difference is within a predetermined range, a straight line portion in the edge image is converted to a line element
Correspond to A, sB, and sC. In the illustrated example, the edges sa1 and sa2 correspond to the line element sA, the edges sb1 and sb2 correspond to the line element sB, and the edge sc1 corresponds to the line element sC.

At this stage, two edges sa1 and sa2 are associated with the line element sA, and two edges sb1 and sb2 are also associated with the line element sB.
To set three deformation areas, each line element sA, sB, s
It is necessary to associate the edges sa1, sa2, sb1, sb2, and sc1 with C in one-to-one correspondence. Here, four types of candidates are conceivable as combinations. That is, sa1-s
b1-sc1, sa1-sb2-sc1, sa2-sb
Any combination of 1-sc1 and sa2-sb2-sc1 (hereinafter, referred to as a tracking candidate) corresponds to the tracking area D. Here, since the edge is obtained by the Hough transform, information on the length of the edge is lost, and each combination corresponds to a figure shown by a solid line in FIG.
The two-dot chain line in FIG. 8 indicates an edge not of interest.

Next, the tracking area D is modified so as to match each of the tracking candidates described above. That is, the line element s
When A, sB, and sC are deformed so as to correspond to the tracking candidates shown in FIGS. 8A to 8D to generate a deformation area, the line elements sA1 to sA4 are obtained as shown in FIGS. , SB
Four types of deformation regions E1 to E4 having 1 to sB4 and sC1 to sC4 are generated. FIG. 9E shows the tracking area D before deformation. Here, the process of deforming the tracking area D is performed by changing each point element p in the original tracking area D as shown in FIG.
Each intersection point p of A, pB, pC and the edge in the deformation area E
a, pb, and pc, the vectors Ma, Mb, and Mc are obtained, and the vectors Ma, Mb, and Mc are interpolated according to the positions of the pixels pP included in the tracking area D, so that the deformation areas E corresponding to the pixels pP are obtained. The position of the pixel pp (vector Mp) within the. Further, the pixel value of each pixel in the deformation area E is also obtained by the interpolation processing. This type of processing is called warping deformation processing.

When the area tracking means 17 generates a deformation area based on a point element, attention is paid to the shape of the boundary element near the point elements pA, pB, and pC of the tracking area D. Now, assuming that an image as shown in FIG. 11 has been obtained, boundary elements in a predetermined range around each point element pA, pB, pC of the tracking area D are shown in FIGS. 12 (a) to 12 (c).
become that way. Therefore, a frame using the shape of FIG. 12 as a template (the next frame in which the tracking area D is set)
Is performed, and a part having a high degree of similarity is extracted. Since pattern matching only needs to be performed in the vicinity of a point element in the frame and there is no need to consider the inclination,
The processing amount is very small as compared with the case where pattern matching is performed for all regions in the image. In the example of FIG. 11, the point element sa1, sa2, sa3 is selected for the point element sA, the point element sb1 is selected for the point element sB, and the point element sc1 is selected for the point element sC. That is, as shown in FIG. 13, sa1-sb1-sc1, sa2-
Since three combinations of sb1-sc1 and sa3-sb1-sc1 are obtained as tracking candidates, three types of deformation regions E1 to E3 are generated as shown in FIG.

As described above, each pixel px in the deformation area E
Next, the pixel corresponding to the tracking area D is selected from the tracking candidates using the deformation area E.

First, the pixel value (density value) in the tracking area D
And when the differential value is within the specified range, it means that the pixel value is substantially constant or the change is smooth,
The pixel value of each pixel pp in the deformation area E is directly compared with the pixel value of each pixel of the tracking candidate. That is, since each deformation area E is set so as to match the shape of each tracking candidate, the difference in pixel value can be known by calculating the difference in pixel value of the corresponding part. Therefore, the sum of the absolute values of the pixel value differences is obtained, and the value obtained by dividing by the number of pixels included in the tracking area D is used as the evaluation value. Evaluation values are obtained for all the tracking candidates, and the tracking candidate with the smallest evaluation value is selected as the optimal tracking candidate. That is, the tracking candidate selected in this way is adopted as a new tracking area. In the example shown in FIG. 8, the tracking candidate in FIG. 8B is selected as a new tracking area, and in the example shown in FIG.
The tracking candidate of (a) is selected as a new tracking area.
In other words, a combination in which no other edge is included in the tracking candidate (for the line element, pa1-pb1-pc1,
In the point element, the evaluation value of pa1-pb1-pc1) becomes the minimum, and a new tracking area is obtained.

On the other hand, when the pixel value or the differential value has a portion exceeding the specified range in the tracking area D, it means that the pixel value has a large variation locally. For example, in a tracking area D including a fine pattern (texture) such as a stone surface, a pixel value or a differential value exceeds a specified range, and a local pixel value difference is large on such a surface. Therefore, the difference between the pixel values cannot be used as the evaluation value. Therefore, a difference between an image feature amount described later in the tracking area D and an image feature amount obtained from the deformation area E1 is used as an evaluation value, and a tracking candidate with the minimum evaluation value is adopted as a new tracking area.

Here, as the image feature amount, the tracking area D
The luminance information may be used when the luminance is almost uniform, and the color information may be used when there is little variation in color. That is, when the difference in the image feature amount between the tracking candidate and the tracking area D is within the specified value, the tracking candidate is adopted as a new tracking area. When color information is used, it is necessary to use a color image as an image. When the color information is used, for example, the tracking area of the RW line in the chromaticity diagram (R, G, B, W means red, green, blue, and white) as shown in FIG. The angle θ with the color Q may be used as the hue value.

Further, as shown in FIG. 16, when there is a pattern regarded as periodic in the tracking area D, Fourier transform or wavelet transform is performed to extract information relating to the spatial frequency and use it as an image feature amount. That is,
As shown in FIG. 17, the spatial frequency distributions d1 and d2 of the deformation area E and the tracking candidate are extracted, and the spatial frequency distribution d is extracted.
By comparing 1 and d2, a new tracking area can be selected from the tracking candidates.

As shown in FIG. 18, when characters or graphics are written on the surface of an object or a label is affixed thereto, a portion having a large contrast may occur in the tracking area D. In such a case, the tracking candidate and the deformation area E are compared with the notation of a character or figure or a label as the characteristic portion F. For example, if the deformation area E is set as shown in FIG. 18A and the tracking candidate E ′ (including the characteristic portion F ′) to be compared is set as shown in FIG. As shown in FIG. 19, the center of gravity G of the characteristic portion F
L1, L between each point of intersection pa, pb, pc of the edge
By using 2, 2, L3, the area of the characteristic portion F, the average luminance and the average hue in the characteristic portion F as the image characteristic amount, it is possible to select a new tracking area from the tracking candidates.

As described above, the evaluation method for selecting a new tracking area from the tracking candidates includes a case where the pixel value and the differential value in the tracking area D are within a specified range, and a case where the tracking area D
Are different from each other in the case where the pixel value or the derivative value is out of the specified range in the case where the notation or the label exists in the tracking area D. Therefore, it is necessary to select an evaluation method according to each condition. For this selection, a histogram of luminance and hue is created, and the selection is automated by selecting an evaluation method according to the pattern of the histogram.

That is, when no peak is generated in the histogram as shown in FIG. 20 when the histogram is created for the luminance and the hue, the image is first-order differentiated with respect to the deformation area E, and the sum of the differential values is expressed by the number of pixels. The value obtained by the division (that is, the average value of the differential values of the deformation area E) is obtained as the texture feature amount. Further, a threshold value for the texture feature amount is set, and when the texture feature amount is equal to or smaller than the threshold value, the differential value is within the specified range in the tracking area D.
Judgment that there is no fine pattern, adopts the sum of differences of pixel values as an evaluation method, and when the threshold value is exceeded, the differential value exceeds the specified range, so it is judged that there is a fine pattern, and the spatial frequency distribution is compared. Adopt a method to do.

On the other hand, as shown in FIG. 21, when a single peak occurs in at least one of the luminance and the hue, the average luminance and the average hue are used. In addition, as shown in FIG. 22, when a plurality of peaks occur in both the luminance and the hue, it is highly likely that characters or figures are written or labels are attached. To

As described above, it is possible to automatically set which processing is to be performed by using the distribution information of the luminance and the hue. FIG. 23 shows the overall flow.
That is, after setting the initial tracking area D (S1), a corresponding candidate is extracted from the boundary elements in the next frame (S2). Next, a tracking candidate is created by a combination of boundary elements (S3), and a histogram of luminance and hue is created for pixels in the tracking area D (S4). Here, the number of peaks occurring in the histogram is obtained (S
5) If there is no peak in both histograms (S6), the average value of the differential values is compared with a threshold (S7). If not, a new tracking area is selected from the tracking candidates by comparing only the pixel values (S8). . When the average of the differential values is larger than the threshold value, the tracking area is selected based on the spatial frequency distribution (S9).

On the other hand, if there is one peak in the luminance histogram (S10), the tracking area is selected using the average luminance (S11), and not one peak in the luminance histogram but one peak in the hue histogram.
If the number is (S12), the tracking area is selected using the average hue (S13). When there are peaks in the luminance and hue histograms, but when there are two or more peaks, it is considered that there is notation of a character or graphic or sticking of a label. (S14). The tracking candidate selected as described above is set as a new tracking area (S15), and the process moves to the next frame (S16).

Since a time-series image group is generated by moving the camera 2, a part of the initial tracking area D is focused on as shown in FIG. 24 according to the positional relationship between the object 1 and the camera 2. The object 3 (including other objects) existing between the object 1 and the camera 2 may be concealed (a concealed portion is indicated by a hatched portion). Further, the tracking area D of the target 1 may be concealed by being positioned on the opposite side of the target 1 with respect to the camera 2 as shown in FIG. 25 (a concealed portion is indicated by a hatched portion). As described above, the shape of the specific surface selected as the tracking area D may change, or the specific surface may be concealed and cannot be tracked. When the hidden surface is newly exposed, there is no problem because it is not designated as the tracking area D.

As described above, when the shape (the number of vertices and sides) of the tracking area D changes or when the tracking area D2 is completely hidden, the shapes (points) of the deformation area and the tracking candidate are changed. Since the number of elements and the number of line elements do not match, the degree of matching between the deformation area and the tracking candidate decreases. In such a case, the boundary element of the tracking area is increased or decreased according to the shape of the deformed tracking area, and the subsequent tracking is continued. For example,
When another object 3 exists between the object 1 and the camera 2 for which the tracking area has been set as shown in FIG.
While the tracking area D1 is rectangular as shown in FIG. 26A, a portion including a straight line indicated by a two-dot chain line and one vertex is hidden by the object 3 in FIG. In this case, the boundary area (point element, line element) is increased so that the tracking area D2 becomes a pentagon, and the subsequent tracking is performed. Such deformation of the tracking area D2 is automatically performed as much as possible. If the deformation is not performed automatically, a message requesting the deformation of the tracking area D2 is displayed on the display unit 15. When the message is displayed on the display means 15, the area specifying means 16 is operated to deform the tracking area D2.

In the above tracking process, the storage device 1
In 3, the positions of the tracking areas are sequentially stored, and image features such as the average luminance used when selecting a new tracking area from the tracking candidates are also stored in the storage device 13. These pieces of information are used by the tracking evaluation unit 18 and the shape restoration unit 19, and are used for estimating the cause of the abnormality when the tracking area in which the abnormality has occurred in the tracking is traced again.

For example, when a house is imaged outdoors during a sunny day, there is a strong single light source that irradiates the object 1, and as shown in FIG. One shadow OM is strongly generated. Such a shadow O
Since an edge is generated in M, there is a possibility that a deformation area is erroneously generated to hinder tracking, and that tracking may be abnormal. Therefore, the position of the single light source LM forming the shadow OM is detected from the image features stored in the storage device 13, and the occurrence of the shadow OM is predicted to remove the influence of the shadow OM.

This will be described more specifically. As shown in FIG. 28, when the tracking area D is illuminated by a single light source (including natural light) LM, the camera 2 is to be imaged from the position represented by the camera posture vectors v1 to vn. Illumination light vector r1 representing the direction of the light beam from single light source LM
And the reflected light vector r2 representing the direction of the specularly reflected light in the tracking area D have the same angle as the normal direction U of the surface on which the tracking area D is set. The closer the vectors v1 to vn are to the reflected light vector r2, the greater the value.

Therefore, the average brightness of the tracking area D set in each frame in the tracking process is stored in the storage device 13, and the shape restoring means 19 captures the frame having the maximum average brightness for each tracking area D. Then, the camera posture vectors v1 to vn are obtained, and the obtained camera posture vectors v1 to vn are regarded as an approximate value of the reflected light vector r2. When the reflected light vector r2 is obtained, the illumination light vector r is determined in relation to the normal direction U of the surface on which each tracking area D is set.
Since 1 can be estimated, the degree of variation in the estimated value of the illumination light vector r1 obtained for all the tracking areas D is evaluated. When the variation is small, it can be determined that a single light source exists. At this time, the average value of the estimated value of the illumination light vector r1 obtained for each tracking area D is used as the illumination light vector r1.

By the way, as described above, in the process of tracking the tracking area D by the camera 2, it may not be possible to capture an image of part or all of the tracking area D. The following three cases can be considered as causes that the tracking area D cannot be imaged.

That is, as shown in FIG. 29, when another object 3 exists between the tracking area D set in the object 1 and the camera 2 and the tracking area D is concealed (hereinafter, referred to as other person concealment). ), As shown in FIG. 30, the tracking area D set for the object 1
Is located on the back side of the object 1 with respect to the camera 2 (hereinafter referred to as self-concealment), as shown in FIG.
Are outside the field of view VF of the camera 2 (hereinafter referred to as frame-out).

As described above, an area that cannot be imaged by the camera 2 (hereinafter, referred to as an unimageable area) is extracted by the following procedure. That is, first, a tracking area in which no non-imaging area is generated in the tracking process is set as a reference area.
Such a reference region is set using a tracking region in which the number of boundary elements does not change in a plurality of images in the time-series image group. Here, any of a line element and a point element may be used as the boundary element. For example, FIG.
In the time-series image groups V11 to V13 shown in FIG.
D13 has five point elements and five line elements, and the numbers of point elements and line elements do not change.
Also, the time-series image groups V21 and V22 shown in FIG.
The tracking areas D21 and D22 have four point elements and four line elements. In this case, the numbers of the point elements and the line elements do not change.

When the reference area is set, a boundary area in which the number of boundary elements does not change in a plurality of images is selected as described above, and point elements are corresponded to each other, and a stereo method or a factorization method is used. The tracking area is mapped in the three-dimensional space by applying a simple method. For example, FIG.
It is assumed that, in two images as in (b), the coordinates of the point elements in the tracking areas D1 and D2 are set as shown in FIGS. Here, if the factorization method is adopted when each point element is mapped as shown in FIG. 33C, the coordinates of the point elements of the tracking areas D1 and D2 in each image and the coordinates in the three-dimensional space are obtained. The relationship with the coordinates of the point can be expressed in the form of Equation 1.

[0075]

(Equation 1) In Equation 1, the left determinant on the right side indicates the direction of the camera 2, and (CX1 CY1 CZ1) in the first row and (CX3 CY3 CZ3) in the third row are used when obtaining the image in FIG. The vectors in the x-axis and y-axis directions of the image plane of the camera 2 are shown. (CX2 CY2 CZ2) in the second row and (CX4 CY4 CZ4) in the fourth row are the cameras 2 when obtaining the image in FIG. 5 shows vectors in the x-axis and y-axis directions of the image plane of FIG. The matrix at the center on the right side is a diagonal matrix obtained by the factorization method, and from this component, the contribution rate K can be obtained by the following equation. K = (a + b + c) / (a + b + c + d) If no image-capable area does not occur, it is considered that there is no erroneous correspondence between the point elements between the tracking areas set in the time-series image group. An appropriate threshold value is set in advance, and when the contribution rate K obtained as described above exceeds the threshold value, it is determined that no image-capturing area does not occur, and the area mapped in the three-dimensional space is set as a reference area candidate. . Since reference region candidates are obtained for each tracking region in which the number of point elements in the boundary elements does not change, a plurality of reference region candidates corresponding to one surface in the three-dimensional space are generated. Therefore, a plurality of reference region candidates are obtained, and the reference region candidate having the largest area among them is adopted as a reference region for one surface in the three-dimensional space.

When the reference area is determined, the image pickup impossible area can be extracted. In other words, if the point elements do not correspond to each other while the tracking area in which the reference area has been determined is tracked by the time-series image group, it can be determined that the non-capturable area has occurred. For example, as shown in FIG. 34, if the reference area DS is set, the reference area DS includes point elements pS1 to pS4 as boundary elements. On the other hand, when the tracking area D is mapped to a three-dimensional space by the stereo method or the factorization method, it is assumed that the tracking area D includes five point elements p1 to p5 as shown in FIG. For example, although the point elements p1 to p3 of the tracking area D correspond to the point elements pS1 to pS3 of the reference area DS, the point elements p4 and p5 of the tracking area D have no point element corresponding to the reference area DS. . That is, the point element pS having no counterpart in the reference area DS and the tracking area D
An area surrounded by 4, 4, p5 can be regarded as an image-impossible area.

In the above description, a point element is used. However, when a line element is used, the line elements s2 and sS3 of the tracking area D are replaced with the line elements sS2 and sS3 of the reference area DS as shown in FIG.
3 and the line elements sS1, sS4 of the reference area DS.
Corresponds to the line elements s1 and s4 of the tracking area D, but there is no line element corresponding to the reference area DS for the line element s5 of the tracking area D. Therefore, this line element s5 is a part of the imaging impossible area. It can be determined that there is.

After the non-capturable area is extracted as described above, it is determined whether the cause of the non-capable area is hidden by others, self-concealed, or framed out. For this determination, the reference area is projected onto the image plane of each image in the time-series image group from a state where all the reference areas obtained as described above are mapped in the three-dimensional space. That is, since the position in the three-dimensional space of the image plane where the non-capturable area has occurred can be defined by the position of the camera 2, by projecting the reference area onto this image plane, the planes on which the reference area is set overlap. It is possible to know whether it is out of the image plane. In addition, it is possible to know which surface on which the reference region is set has an image-capturing-impossible region based on the distance relationship between the surface on which the reference region is set and the camera 2 (that is, the image surface). Hereinafter, the surface on which the reference region is set is referred to as a reference surface.

Here, there is a case where another reference plane exists between the camera 2 and the reference plane where the non-imaging area is generated in the three-dimensional space, and is continuous with the reference plane where the non-imaging area is generated. If the reference plane does not exist, it is determined that the concealment is by others. For example, in FIG.
Although a non-capturable area (shaded area) is generated in 1, since the reference plane SR2 adjacent to the reference plane SR1 in the image is not connected in the three-dimensional space, it is determined that other persons are hidden. .

On the other hand, when there is another reference plane between the camera 2 and the reference plane in which the non-capturable area occurs in the three-dimensional space, the reference is continuous with the reference plane in which the non-capturable area occurs. When the surface exists, it is determined that the image is self-concealing. For example, in FIG. 37, a non-capturable area (shaded portion) is generated on the reference plane SR3, and the reference planes SR4 and SR5 adjacent to the reference plane SR3 in the image are 3
In this case, since the connection is also made in the dimensional space, in this case, it is determined that the reference plane SR <b> 3 has become an imaging-impossible area due to self-concealment.

Further, as shown in FIG. 38, when a part of the reference plane SR6 in which the non-image-capturing area is generated straddles the periphery of the image plane, it is determined that the frame is out (of the reference plane SR6). The part that goes out of the screen is indicated by hatching.)

As described above, based on the distance between the reference plane and the image plane and the connection relationship between the reference plane having the non-capturable area and another reference plane, the cause of the non-capturable area may be caused. It is possible to determine whether it is concealed by others, self-concealed, or frame out.

After the cause of the occurrence of the non-capturable area is determined as described above, the result of the determination is verified. That is, as shown in FIG. 39, the edges E1 and E2 are extended in the direction in which the non-capturable area exists on the reference plane SR where the non-capturable area is generated. The tracking process is performed by regarding the point element as a simple point element. At this time, if the above-described contribution rate is improved, the non-capturable region is finally determined to be caused by concealment of others or self-concealment. Here, since the image capturing impossible area due to the concealment of others may be able to be imaged by moving the camera 2, the tracking process is continued, and the image capturing impossible area due to the self concealment can be imaged again by the movement of the camera 2. Then, the tracking process is terminated.

FIG. 40 shows a summary of the processing procedure for the image-capable area. That is, first, in order to set a reference area, a tracking area in which the number of boundary elements does not change in a plurality of images is extracted (S1). Here, the tracking area forming one surface of the object 1 is extracted from the plurality of images, and the tracking area is mapped in the three-dimensional space by performing coordinate transformation on the corresponding tracking area (S2). ,
The contribution rate at the time of mapping is obtained (S3). After the mapping of all extractable surfaces to the three-dimensional space and the calculation of the contribution ratio are completed (S4), the surfaces whose contribution ratios are equal to or larger than a threshold when mapped to the three-dimensional space are extracted as reference region candidates. (S5). Further, a candidate having the largest area is extracted from the reference region candidates and set as a reference region (S6). When the reference area is determined, an uncapturable area is extracted next (S7), and the positional relationship between another reference area and the camera in the same image as the reference area of interest in the three-dimensional space is determined (S8). Here, when the other reference area is separated from the reference area where the non-capturable area is generated (S9), it is determined that the other person is concealed (S10). On the other hand, if another reference area is connected (S11), it is determined to be self-concealment (S12). If neither concealment of others nor self-concealment has occurred, the position in the frame of the reference region where the non-capturable region has occurred is determined (S13). If the reference region exists near the periphery of the image, (S14), it is determined that the frame is out (S15). After performing the above processing for all the frames (S16), the edge corresponding to the non-imaging area is extended, and the tracking processing is executed again to determine whether or not concealment is performed (S17).

(Second Embodiment) The first embodiment is a method for detecting the presence or absence of concealment. However, in the present embodiment, a series of plural images in which no change occurs in a plane included in an image. A method in which an image is set as a phase image group, a tracking area is mapped in a three-dimensional space for each phase image group, and a three-dimensional shape obtained by the mapping is superimposed by coordinate transformation to obtain three-dimensional data of the entire target 1. explain.
Therefore, in the present embodiment, the processing for representing and tracking the target object 1 by the boundary element can be shared with the first embodiment. In the present embodiment, the object 1 which does not cause concealment of others can be processed independently.
After the process of detecting the presence / absence of concealment in the embodiment, the process of the present embodiment can be performed using the result of the tracking process used in the first embodiment.

In the following, for ease of explanation, it is assumed that the object 1 has a rectangular parallelepiped shape as shown in FIG. FIG.
, The arrow A indicates the movement of the camera 2. That is, the process of the present embodiment will be described assuming that no concealment of others occurs. Each surface of the object 1 is defined as shown in FIG.
That is, the upper surface of FIG.
The front left surface and the front right surface are denoted by f3 and f4, respectively, and the back right surface and the back left surface of FIG. 42 are denoted by f5 and f6, respectively. When such an object 1 is viewed from above, each surface f1, f3 to f3
The relationship of No. 6 is as shown in FIG. Now, an axis passing through the center of the upper surface f1 of the object 1 and orthogonal to the upper surface f1 (that is, FIG.
3), the camera 2 is rotated relative to the object 1 around an axis perpendicular to the plane of the drawing), and the object 1 is tilted upward at an angle of about 45 degrees with respect to the axis. It is assumed that images are taken from. Therefore, each image (frame) obtained by imaging always includes the upper surface f1. Also, when the camera 2 is located in a plane that includes the axis and is orthogonal to each of the planes f3 to f6 (when the camera 2 is located at the positions e3 to e6 in FIG. 43), each image (frame) includes a part other than the top surface f1. Is each surface f3
To f6. Camera 2
Are other positions (ranges d34, d45,
d56, d63), each image has an upper surface f
In addition to 1, two adjacent surfaces are included.

That is, when the position of the camera 2 is changed counterclockwise from the position e3 shown in FIG. 43, the contents of each image (each box in FIG. 44 indicates an image) changes as shown in FIG. Will do. However, an image group k3 including three surfaces of the upper surface f1 and the other two surfaces with one image h3 to h6 including only two surfaces of the upper surface f1 and the other one surface interposed therebetween.
The imaging conditions are set so as to obtain 4, k45, k56, and k63. That is, image groups k34 and k4 including three surfaces
5, k56, and k63 are set with imaging conditions so that a plurality of images are continuously obtained. Such a plurality of continuous images including the same plane are hereinafter referred to as a phase image group. Also, the phase image groups k34, k45, k
Images h3 to h6 including only two surfaces between 56 and k63 are 1
The imaging conditions are set so that images are obtained independently. In the images obtained under such imaging conditions, each of the phase image groups k34, k45, k56, and k63 is represented by an image h3 to h3.
h6 can be separated. In short, when the number of planes included in each image changes, the image is divided before and after the change, and when a plurality of images including the same plane are obtained continuously, they are collectively formed as a phase image group. The phase image groups k34, k45,
No change occurs in the planes included in the images k56 and k63.

In the present embodiment, the image of the object 1 acquired by the TV camera 2 is applied to the area designating means 16 shown in FIG.
, A boundary expression (B-R) consisting of a line element and a point element
A tracking area is set using EP (boundary representation). The area selected from the object 1 is tracked by the area tracking means 17. Such tracking processing is the same as that of the first embodiment. A tracking area is set for each surface of the object 1, and the shape of the tracking area of another screen is estimated from the tracking area of one screen. The transformation area is generated, the image of another screen is compared with the transformation area, and the position of the transformation area is tracked. If the surface of the object 1 is tracked in this manner, information on which surface is included in each image can be obtained. As described above, the phase image groups k34, k45,
k56 and k63 can be set.

As described above, the phase image group k34,
When k45, k56, and k63 are set, the tracking area is mapped to a three-dimensional space for each of the phase image groups k34, k45, k56, and k63. In order to set the phase image groups k34, k45, k56, and k63, FIG.
The phase image groups k34 and k34 are extracted from the original moving image stored in the image file F1 by using a means not shown in FIG.
45, k56 and k63. As a process of dividing the original moving image into a group of phase images k34, k45, k56, and k63, a method using an image processing unit that automatically tracks a surface by a contour line in a two-dimensional image, or a method in which a plurality of images are displayed on a screen Display and use one of the manual methods. Phase image group k34, k45, k5
Since the planes included in the images 6 and k63 do not change, the point elements do not disappear due to self-concealment in the images constituting the phase image groups k34, k45, k56 and k63. Group k34, k4
Within the range of 5, k56, and k63, a tracking area can be mapped in a three-dimensional space by applying a conventionally known method such as a factorization method or a stereo method.

Now, the phase image group k3 shown in FIG.
45, k56, and k63, if the tracking area can be mapped in a three-dimensional space as shown in FIGS.
The world coordinate systems of the three-dimensional shapes obtained from 34, k45, k56, and k63 are different from each other. 45A to 45D, the world coordinate system for each three-dimensional shape is represented by X1-Y1-Z1, X1, respectively.
2-Y2-Z2, X3-Y3-Z3, X4-Y4-Z4
Indicated by. Thus, the phase image group k34, k
If the world coordinate system is different for each of the three-dimensional shapes obtained from 45, k56, and k63, the three-dimensional data of the object 1 cannot be represented in one common coordinate system. A three-dimensional shape having each world coordinate system is superimposed.

When a plurality of three-dimensional shapes having different world coordinate systems are superimposed, the procedure differs depending on whether or not a common line element exists between the plurality of three-dimensional shapes to be superimposed. Occurs.

When a common line element exists, coordinate conversion is performed on a plurality of target three-dimensional shapes so that the common line element overlaps. For example, in FIG. 45, the line elements s13, s14, s15, and s16 surrounding the surface f1 are commonly present in FIGS. 45 (a) to (d), and the line element s45 as a boundary between the surface f4 and the surface f5 is FIG. 45 (a)
To (c). There are other line elements common to a plurality of three-dimensional shapes, but it is desirable to select line elements common to as many three-dimensional shapes as possible. In the present embodiment, line elements having different directions and intersecting at one point are selected. That is, a line element s14 which is a boundary line between the surface f1 and the surface f4, a line element f15 which is a boundary line between the surface f1 and the surface f5, and a line element s45 which is a boundary line between the surface f4 and the surface f5. Three are selected. By performing coordinate transformation such that these three line elements s14, s15, and s45 are superimposed, the world coordinate system of the four three-dimensional shapes is unified, and one world coordinate system (X −YZ) can represent three-dimensional data.

On the other hand, when there is no common line element, the tracking areas that are commonly present in each three-dimensional shape are overlapped. That is, in FIG. 45, since the surface f1 is common to all three-dimensional shapes, the surfaces f1 are overlapped. This means that the line elements s13, s14, s
This is equivalent to performing coordinate transformation so that the positions 15 and s16 overlap.

By the way, as described above, the phase image group k
When integrating the three-dimensional shape world coordinate system obtained for each of the 34, k45, k56, and k63 into one world coordinate system, each phase image group k34, k45, k56, k63
If there is an error in the three-dimensional shape obtained from, there is a shift in the two world coordinate systems to be superimposed, so that when one of the boundary elements is superimposed, the other boundary element does not overlap Inconvenience may occur.
Therefore, a transformation matrix for coordinate transformation is calculated for each boundary element (line element and point element) common to the two world coordinate systems, and each element of the plurality of obtained transformation matrices (that is, transformations related to rotation and translation). The coordinate transformation is performed by using a transformation matrix having an average value of the parameter) as an element. The conversion matrix can be uniquely determined by using the average value as described above.

Now, the three-dimensional shape shown in FIGS. 45A and 45B will be described. In the two three-dimensional shapes, the point element p134, which is the intersection of the three surfaces f1, f3, and f4, is:
At the position shown in FIGS. 47A and 47B, the rotation r shown in FIG.
47 (a) by performing coordinate transformation between
45 can be moved to the position shown in FIG.
It is considered that the world coordinate systems in (a) and (b) can be matched. Since such a coordinate transformation is generally represented in a matrix format, the coordinate transformation can be defined as a matrix element. Therefore, not only the point element p134 but also other point elements (more is better) are obtained as coordinate conversion elements, an average value is obtained for each same element, and coordinate conversion is performed using a conversion matrix having the average value as an element. Is applied.

As shown in FIG. 48, two three-dimensional shapes are superimposed on each other by using a transformation matrix having an average value for each element of the transformation matrix obtained for a plurality of boundary elements as described above. One three-dimensional shape can be obtained. In the illustrated example, the three-dimensional shape of FIG.
This is the result of performing coordinate transformation so as to overlap the three-dimensional shape of (a).
Is invisible on the back side. Here, in FIG. 48, a code with a dash (′) means a three-dimensional shape obtained by performing coordinate conversion on the three-dimensional shape in FIG. 47 (b). When the average value of each element of the transformation matrix is used as described above, the point element p134 ′ obtained by performing the coordinate transformation on the position of FIG. 47B becomes the point element p134 ′ at the position of FIG. It does not completely match with p134. Therefore,
Finally, the midpoint between the point element p134 and the point element p134 'is determined as the position of the point element.

As described above, a plurality of three-dimensional
It becomes possible to integrate into three three-dimensional data. In addition,
In this embodiment, since the image of the surface f2 is not captured by the camera 2, the surface f2 cannot be accurately defined. However, since the line elements and the point elements constituting the periphery of the surface f2 are captured by the camera 2, , And a plane represented by these line elements and point elements.

In the above example, a moving image (time-series image)
, And each phase image group k34, k45, k56, k
In order to set 63, the fact that the number of faces changes is used, but the phase image group may be divided according to the presence or absence of faces where self-concealment occurs as described below. In other words, in addition to representing boundary elements with point elements and line elements,
By determining the direction of each line element as a directed line element, it is determined that the surface that is the tracking area has become invisible using the change in the direction of the line element due to the self-concealment of each surface, and it is the tracking area. When the surface changes from a visible state to an invisible state, it is determined to be a break of the phase image group. In other words, since the line elements sequentially connect the point elements, the point elements are traversed by sequentially tracing the point elements, and when the circulating direction is monitored and the circulating direction is reversed in the image, It can be determined that the surface has become invisible.

More specifically, FIG.
As shown by an arrow in the figure, if it is defined that a boundary element constituting one surface can make a clockwise round when it is traced in the direction of the line element, the invisible surface is countered. Turn clockwise. For example, in FIG. 49, the surfaces f1, f3, f
4 is visible, and faces f2, f5, and f6 are invisible. Here, the camera 2 captures the image of the object 1 in the above-described positional relationship with respect to the object 1, so that the surface f1 is obtained.
Since the face f2 is always visible and the face f2 is always invisible, the faces f1 and f2 are excluded. Therefore, here, the target of the visibility determination is the surfaces f3 to f6. In this case, if the line elements can be traced clockwise with respect to the visible surfaces f3 and f4, the invisible surfaces f5 and f4
6 follows the line element counterclockwise. This is the same when the surfaces f3 and f4 become invisible. In short, if a direction is given to a line element so as to go around the boundary element of one surface by tracing the line element, it is determined whether the line element is traced clockwise depending on whether the surface is visible or invisible. Since whether or not the line element traces around changes, it can be used to determine that the surface that is the tracking area has become invisible due to self-concealment, and is set as the tracking area in the moving image (time-series image). When the surface becomes invisible from visible, it can be used as a partition of the group of phase images.

[0100]

According to the first aspect of the present invention, a stationary object is imaged from a plurality of different positions with a TV camera to acquire a plurality of images, and a surface selected from the object is defined as a boundary element, an image feature amount, and the like. Is set in the image as a tracking area represented by, a deformation area is created by deforming the tracking area set in one image so as to match a boundary element of a candidate for a tracking area in another image, and another image is created. Selecting a tracking area corresponding to the tracking area set in one image from the candidates for the tracking area by comparing at least one of the boundary element and the image feature amount with respect to the candidate for the tracking area and the deformation area in
And the tracking area with few non-imaged areas
A reference region obtained by mapping to a three-dimensional space is set, and when at least a part of the tracking region cannot be tracked in the image, a region that cannot be imaged by the TV camera based on the positional relationship of the plurality of reference regions in the three-dimensional space A second step of determining whether or not the TV has occurred,
Estimate the cause of the inability to image the tracking area with the TV camera based on the positional relationship of the reference area when imaging at each position of the camera, and correct so that the tracking area can be tracked based on the estimation result. A third step of continuing the tracking, wherein the tracking area is tracked between the moving images to easily obtain three-dimensional information from the two-dimensional image that can be used for 3DCG data. Is not only possible, but also detects the occurrence of a non-imaged area in the tracking area and estimates the cause of the non-imaging, so even if tracking area tracking fails once, take action according to the cause to track The area can be tracked reliably. As a result, the validity of the obtained three-dimensional information is verified, and the occurrence of errors is suppressed.

According to a second aspect of the present invention, in the first aspect, in the first step, a candidate corresponding to a boundary element of the tracking area set in one image on another image is extracted,
After generating a tracking candidate based on a combination of boundary element candidates, a deformation area is generated by deforming the tracking area set in one image so as to match the tracking candidate, and the shape of the boundary element is determined for the tracking candidate and the deformation area. The feature is to determine a tracking area in another image from among tracking candidates by comparing indices selected from pixel values and image feature amounts, instead of using local features as in the past, Since a line element (boundary), a point element (junction of the boundary), or a feature of an image that constitutes the tracking area is used, the tracking area can be tracked more reliably than before.

According to a third aspect of the present invention, in the second aspect of the present invention, an edge is extracted from another image, and a distance and a direction of one of the edges from a line element constituting a tracking area are within a specified range. Is selected as a candidate for a line element. According to a fourth aspect of the present invention, in the second aspect of the present invention, an edge is extracted from another image and a Hough transform is performed on each pixel on the edge. Extracting an edge having continuity by selecting an edge having a distance and a direction within a specified range with respect to a line element forming a tracking area in one of the edges as a line element candidate. According to a fifth aspect of the present invention, in the second aspect of the present invention, a shape near a point element constituting a tracking area in one image is used as a template, and a template is used in another image. The feature is that point element candidates are extracted from the parts that match the target, and in each case, a finite number of tracking candidates are generated by combining candidate boundary elements, so accurate tracking while limiting the processing amount Becomes possible.

According to the invention of claim 6, in the invention of claim 2, since the pixel values of the tracking candidate and the deformation area are used as the index, it is possible to compare the tracking candidate with the deformation area even when the image has no feature. It is.

According to a seventh aspect of the present invention, in the second aspect of the present invention, the average brightness of the tracking area and the tracking candidate is used as an index. Is a color image, wherein the color between the tracking area and the tracking candidate is used as an index. The invention according to claim 9 is the invention according to claim 2, wherein the spatial frequency distribution between the tracking area and the tracking candidate is used as an index. It is characterized by using
In any case, by using the features in the image, the tracking candidate and the deformation area can be easily compared by comparing the numerical values.

According to a tenth aspect of the present invention, in the second aspect of the present invention, a portion where a change in pixel value exceeds a specified value in the tracking region and the tracking candidate region is extracted, and the position, shape, and image characteristics of the portion are extracted. Since at least one element is used as the index, the comparison between the tracking candidate and the deformation area can be performed more reliably by using the feature in the image.

According to an eleventh aspect of the present invention, in the second aspect of the present invention, the image is a color image and the average luminance or color of the tracking area and the tracking candidate is used as an index, and the spatial frequency distribution is used as an index. Case, and a portion where the change of the pixel value exceeds the specified value in the region is extracted, and the position, shape,
Using at least one element of an image feature as an index,
It is characterized by selecting according to the distribution pattern of luminance and color in the tracking area in one image, and using the information of luminance and color, a method suitable for comparing the tracking candidate with the deformation area is Can be selected automatically.

According to a twelfth aspect of the present invention, in the second aspect of the present invention, the posture of the TV camera when the average luminance of the plurality of images in the plurality of tracking areas is maximized is determined by the plane of the object corresponding to each tracking area. The direction of the reflected light from the light source is estimated from the normal direction of the surface of the object corresponding to each tracking area and the direction of the reflected light, and the position of the light source is estimated. By doing so, it is possible to remove the influence of the shadow, and it is possible to prevent a tracking error due to the influence of the shadow when tracking the tracking area.

According to a thirteenth aspect of the present invention, in the first aspect, in the second step, a tracking area corresponding to a plurality of images is mapped in a three-dimensional space, and a correspondence between boundary elements of the tracking area is determined. The tracking area mapped to the three-dimensional space when the degree of coincidence is equal to or larger than the threshold value is set as a reference area, and a boundary element between the tracking area corresponding to the reference area and the reference area is compared. It is characterized by extracting a region that is not imaged in the above, and by simply reproducing the position of the object obtained from the image in the three-dimensional space by the reference region, the positional relationship of the object can be relatively easily. It can be verified, and it can be easily verified whether or not an unimaged area occurs.

According to a fourteenth aspect, in the thirteenth aspect, the number of boundary elements between a plurality of images is limited to a range in which the number does not change, and for each range, the contribution ratio obtained at the time of mapping is determined by the degree of coincidence. The tracking area whose contribution rate is equal to or larger than the threshold is used as a reference area candidate, and when a plurality of reference area candidates are obtained, the candidate having the largest area is adopted as the reference area. By obtaining the area, there is an advantage that a reference area that better corresponds to one surface of the object can be set.

According to a fifteenth aspect of the present invention, in the invention of the fourteenth aspect, since the change in the number of boundary elements focuses on one of the line element and the point element, the processing amount is small and high-speed processing is possible. Become.

According to a sixteenth aspect of the present invention, in the fourteenth aspect, the contribution factor is obtained from a diagonal matrix component obtained when the tracking area is mapped to a three-dimensional space by the factorization method. The solution can be obtained stably by the used linear operation.

According to a seventeenth aspect of the present invention, in the thirteenth aspect, a tracking area is mapped in a three-dimensional space, compared with a reference area, and an area defined by point elements that do not correspond to each other is imaged in the tracking area. The invention according to claim 18 is characterized in that the tracking area is mapped to a three-dimensional space and compared with the reference area, and when there is a line element that does not correspond to each other, It is characterized that this line element is regarded as a part of the area not imaged in the tracking area, and the range of the area not imaged can be easily obtained.

According to a nineteenth aspect of the present invention, in the first aspect of the present invention, in the third step, all the reference areas in the three-dimensional space are projected on an image plane determined by the position of the TV camera, and Based on the positional relationship between the reference areas and the positional relationship of the reference area with respect to the image plane, the cause of the occurrence of an area that has not been imaged in the tracking area is determined. By performing a simulation based on the positional relationship of the reference region reproduced in the above, it is possible to easily determine the cause of the occurrence of the region that is not imaged.

According to a twentieth aspect, in the nineteenth aspect, when there is another reference area between the reference area of interest and the TV camera and both reference areas are not connected, the concealment of others is performed. According to a twenty-first aspect of the present invention, in the nineteenth aspect, another reference area exists between the target reference area and the TV camera, and both reference areas are connected. The invention of claim 22 is characterized in that, when the reference region of interest is located at the periphery of the screen, it is determined that the frame is out, Based on the positional relationship between the reference areas and the image plane, it is possible to determine the cause of the occurrence of an area that has not been imaged.

According to a twenty-third aspect of the present invention, in the invention of the nineteenth aspect, when there is an area in the tracking area that is not imaged, edge extension is performed, and the intersection of the extended edge is set as a new point element. The tracking process is executed again, and if the contribution rate is improved when mapping to the three-dimensional space, the concealment is determined. This makes it possible to avoid a state in which the tracking becomes impossible due to the concealment of the tracking area. The region can be tracked more reliably.

According to a twenty-fourth aspect of the present invention, there is provided an image input section for providing an image input image picked up by a TV camera, and an image processing apparatus for performing processing on the input image by the image feature tracking processing method according to the first aspect. A storage device for storing an input image and an image processed by the image processing device, and a display unit for displaying a processed image in the image processing device;
The image processing apparatus further includes an area designating unit that designates a tracking area for the image processing apparatus. In addition to the same effect as the first aspect of the present invention, a moving image captured by a normal video camera or the like is used as a TV camera. Three-dimensional information can be easily obtained.

According to a twenty-fifth aspect of the present invention, after a plurality of images are obtained by capturing images of an object from a plurality of different positions with a TV camera, a plane selected from the objects is set as a tracking area represented by a boundary element in the image. Set, and create a deformation area by deforming the tracking area set in one image so as to match the boundary element of the tracking area candidate in another image, and create a tracking area candidate and a deformation area in the other image. And selecting a tracking area corresponding to the tracking area set in the one image from the tracking area candidates by comparing the boundary elements with each other, and further, a plurality of surfaces of the object are imaged and continuously the same surface Is divided into one phase image group, and then the mapping of the plurality of phase images to the three-dimensional space of the surface selected from the object for each phase image group is performed. A three-dimensional shape is obtained for each phase image group by performing the three-dimensional image processing, and coordinate conversion is performed so that the coordinate systems of the other phase image groups coincide with each other, thereby creating three-dimensional data. In the image (time-series image), a plurality of continuous images in which the same surface of the object is imaged are set as a phase image group, and a three-dimensional shape is obtained within the range of the phase image group. Therefore, the three-dimensional shape can be reliably and easily obtained without the surfaces being unable to be tracked due to self-concealment. That is, there is no inconvenience even if a three-dimensional shape is obtained by applying a conventionally known factorization method or stereo method. In this way, a three-dimensional shape can be easily obtained for each phase image group, and thereafter, one coordinate is obtained by performing coordinate transformation so that the coordinate system of the three-dimensional shape obtained from each phase image group matches. It can be summarized in three-dimensional data.

According to a twenty-sixth aspect of the present invention, after a plurality of images are obtained by picking up an image of an object from a plurality of different positions using a TV camera, a plane selected from the object is set as a tracking area represented by a boundary element in the image. Set, and create a deformation area by deforming the tracking area set in one image so as to match the boundary element of the tracking area candidate in another image, and create a tracking area candidate and a deformation area in the other image. And selecting a tracking area corresponding to the tracking area set in the one image from the tracking area candidates by comparing the boundary elements with each other, and further, a plurality of surfaces of the object are imaged and continuously the same surface Is divided into one phase image group, and then the mapping of the plurality of phase images to the three-dimensional space of the surface selected from the object for each phase image group is performed. The three-dimensional shape obtained for each phase image group is obtained by performing a grouping process to obtain a three-dimensional shape for each phase image group and overlapping line elements common to other phase image groups with each other. Is performed to generate three-dimensional data. A plurality of continuous images in which the same surface of the object is captured in a moving image (time-series image) are defined as a phase image group, and the range of the phase image group Since the three-dimensional shape is determined within the range, the respective surfaces cannot be traced due to self-concealment within the range of the phase image group, and the three-dimensional shape can be determined reliably and easily. That is, there is no inconvenience even if a three-dimensional shape is obtained by applying a conventionally known factorization method or stereo method. In this way, a three-dimensional shape can be easily obtained for each of the phase image groups, and thereafter, the coordinate transformation is performed so that the line elements of the three-dimensional shape obtained from each of the phase image groups are overlapped with each other. Three
It can be summarized in dimensional data.

According to a twenty-seventh aspect, in the twenty-fifth or twenty-sixth aspect, a plane selected from the object in the plurality of images captured by the TV camera is defined as a point element and each point. Expressed as boundary elements consisting of directed line elements that sequentially connect elements, monitor the direction of tracing each point element by tracing the line element in each image obtained in time series,
It is characterized in that one phase image group is set until the traveling direction is reversed, and the process of dividing the phase image group can be automated.

According to a twenty-eighth aspect of the present invention, in the twenty-fifth or twenty-sixth aspect of the present invention, in the coordinate transformation, after obtaining a conversion parameter relating to rotation and translation for each boundary element, the conversion parameter obtained for each boundary element is obtained. Is characterized in that coordinate transformation is performed using transformation parameters obtained by averaging the transformation parameters, and the transformation parameters of coordinate transformation are averaged to uniquely determine the transformation parameters even when the transformation parameters obtained from individual boundary elements do not match. be able to.

According to a twenty-ninth aspect of the present invention, in the twenty-fifth or twenty-sixth aspect of the present invention, the position of each point element after the coordinate transformation is performed on one of the three-dimensional shapes whose coordinate systems match each other and the other three-dimensional shape. The midpoint between the position of each corresponding point element in the dimensional shape is the position of each point element, and even if there is a deviation in the position of the point element when performing coordinate transformation, The position can be uniquely determined.

[Brief description of the drawings]

FIG. 1 is a block diagram showing an apparatus used in an embodiment of the present invention.

FIG. 2 is a perspective view showing a state where an image of an object is captured by a TV camera in the above.

FIG. 3 is a diagram showing a time-series image group obtained in the above.

FIG. 4 is a diagram showing an example of an image in the above.

FIG. 5 is a diagram showing an example of a tracking area in the embodiment.

FIG. 6 is a diagram showing an example of an object in the above.

FIG. 7 is an operation explanatory diagram of the above.

FIG. 8 is a diagram showing an example of a tracking candidate in the embodiment.

FIG. 9 is a diagram showing an example of a deformation area in the above.

FIG. 10 is a diagram showing a process of generating a deformation area in the above.

FIG. 11 is an operation explanatory diagram of the above.

FIG. 12 is a diagram showing an example of a template in the embodiment.

FIG. 13 is a diagram showing an example of a tracking candidate in the embodiment.

FIG. 14 is a diagram showing an example of a deformation area in the above.

FIG. 15 is an explanatory diagram of the operation of the above.

FIG. 16 is a diagram showing an example of a tracking area in the above.

FIG. 17 is an operation explanatory view of the above.

FIG. 18 is a diagram showing an example of a deformation area in the above.

FIG. 19 is a diagram illustrating the operation of the above.

FIG. 20 is an explanatory diagram of the operation of the above.

FIG. 21 is an explanatory diagram of the above operation.

FIG. 22 is an explanatory diagram of the above operation.

FIG. 23 is an explanatory diagram of the operation of the above.

FIG. 24 is an explanatory diagram of the operation of the above.

FIG. 25 is an explanatory diagram of the operation of the above.

FIG. 26 is an explanatory diagram of the above operation.

FIG. 27 is a diagram showing an example in which a shadow occurs in the above.

FIG. 28 is an explanatory diagram of the operation of the above.

FIG. 29 is an explanatory diagram of the above operation.

FIG. 30 is an explanatory diagram of the operation of the above.

FIG. 31 is an explanatory diagram of the above operation.

FIG. 32 is an explanatory diagram of the above operation.

FIG. 33 is an explanatory diagram of the operation of the above.

FIG. 34 is an explanatory diagram of the operation of the above.

FIG. 35 is an explanatory diagram of the operation of the above.

FIG. 36 is an explanatory diagram of the above operation.

FIG. 37 is an explanatory diagram of the operation of the above.

FIG. 38 is an explanatory diagram of the operation of the above.

FIG. 39 is an explanatory diagram of the above operation.

FIG. 40 is an explanatory diagram of the operation of the above.

FIG. 41 is a perspective view showing a state in which a TV camera captures an image of an object in the second embodiment of the present invention.

FIG. 42 is a perspective view showing the object used in the above.

FIG. 43 is a plan view showing an object used in the above.

FIG. 44 is a diagram illustrating the concept of a group of phase images in the above.

FIG. 45 is a diagram showing a state in which a three-dimensional shape is obtained from each phase image group in the above.

FIG. 46 is a diagram showing a state where three-dimensional shapes are integrated in the above.

FIG. 47 is an explanatory diagram of the operation of the above.

FIG. 48 is an explanatory diagram of the operation of the above.

FIG. 49 is an explanatory diagram of the operation of the above.

[Explanation of symbols]

 REFERENCE SIGNS LIST 1 object 2 TV camera 3 object 11 image input device 12 image input means 13 storage device 14 image processing device 15 display means 16 area designation means 17 area tracking means 18 tracking evaluation means 19 shape restoration means D tracking area E deformation area F1 image File F2 Tracking data file F3 3D shape data file

Continued on the front page (51) Int.Cl. ⁷ Identification symbol FI Theme coat ゛ (Reference) H04N 13/00 G06F 15/70 410 (72) Inventor ▲ Hamao Tadashio 1048 Kazuma Kajima, Kadoma, Osaka Prefecture Matsushita Electric Works, Ltd. F term in the formula company (reference) 2F065 AA53 BB05 CC14 DD03 DD08 FF04 FF28 GG10 GG18 JJ01 JJ02 JJ19 MM23 QQ13 QQ24 QQ25 QQ31 QQ32 QQ38 QQ42 QQ43 QQ44 UU05 5B057 BA02 BA24 CD02 DC03 DC03 DC03 DC05 DC03 DC03 DC03 DC03 DC03 DC03 DC03 DC03 DC03 DC03 DC03 DC03 DC03 DC03 DC03 DC03 DC03 DC03 DC03 DC03 DC03 DC03 DC03 DC03 DC02 AB03 AB08 AB21 5L096 CA04 CA24 FA03 FA06 FA10 FA23 FA25 FA37 FA59 FA66 FA67 FA81 GA08 GA28 HA05 JA09

Claims (29)

[Claims] 1. A stationary object is captured by a TV camera from a plurality of different positions to acquire a plurality of images, and a surface selected from the object is used as a tracking area represented by a boundary element and an image feature amount. Set in the image, create a deformed area by deforming the tracking area set in one image so as to match the boundary element of the candidate for the tracking area in another image, and create a candidate for the tracking area in the other image A first step of selecting a tracking area corresponding to the tracking area set in the one image from the candidates for the tracking area by comparing at least one of the boundary element and the image feature amount for the and the deformation area; By setting a reference area obtained by mapping a tracking area having a small area not imaged by the camera in a three-dimensional space, at least a part of the tracking area can be tracked in the image. A second step of judging whether or not an area that cannot be imaged by the TV camera has occurred based on the positional relationship of the plurality of reference areas in the three-dimensional space when there is no image; Estimating the cause of the inability to image the tracking area with the TV camera based on the positional relationship of the reference area at the time, correcting the tracking area so that the tracking area can be tracked based on the estimation result, and continuing the tracking. And an image feature tracking processing method. 2. In the first process, after extracting a candidate corresponding to a boundary element of a tracking area set in one image on another image and generating a tracking candidate by combining the boundary element candidates Generating a deformation area by deforming the tracking area set in the one image so as to match the tracking candidate, and selecting an index selected from the boundary element shape, pixel value, and image feature amount for the tracking candidate and the deformation area. 2. The image feature tracking processing method according to claim 1, wherein a tracking area in the other image is determined from tracking candidates by comparing the tracking candidates. 3. An edge is extracted from the other image, and an edge having a distance and a direction from a line element constituting a tracking area in the one image within a prescribed range is selected as a line element candidate. 3. The image feature tracking processing method according to claim 2, wherein: 4. An edge is extracted from the other image, a Hough transform is performed on each pixel on the edge to extract an edge having continuity, and a tracking area is formed in the one image among the edges. 3. The image feature tracking processing method according to claim 2, wherein an edge having a distance and a direction from the line element within a predetermined range is selected as a line element candidate. 5. The method according to claim 1, wherein a shape near a point element constituting a tracking area in the one image is used as a template, and a candidate for a point element is extracted from a part matching the template in the other image. 2. The image feature tracking processing method according to item 2. 6. The image feature tracking processing method according to claim 2, wherein pixel values of the tracking candidate and the deformation area are used as the index. 7. The image feature tracking processing method according to claim 2, wherein an average luminance of the tracking area and the tracking candidate is used as the index. 8. The image feature tracking processing method according to claim 2, wherein the image is a color image, and the colors of the tracking area and the tracking candidate are used as the index. 9. The method according to claim 2, wherein a spatial frequency distribution between the tracking area and the tracking candidate is used as the index.
The described image feature tracking processing method. 10. A method for extracting a portion where a change in pixel value exceeds a specified value in the tracking region and the tracking candidate region, and using at least one element of the position, shape, and image feature of the portion as the index. 3. An image feature tracking processing method according to claim 2, wherein: 11. The image is a color image, wherein the average brightness or color of the tracking area and the tracking candidate is used for the index, the spatial frequency distribution is used for the index, The case where a part whose value change exceeds a specified value is extracted and at least one element of the position, the shape, and the image feature of this part is used as the index, 3. The image feature tracking processing method according to claim 2, wherein the selection is performed according to a pattern. 12. Estimating that the attitude of the TV camera when the average luminance of the plurality of images in the plurality of tracking areas is maximized is close to the direction of the reflected light on the surface of the object corresponding to each tracking area. 3. The image feature tracking processing method according to claim 2, wherein an irradiation direction from a light source is estimated from a normal direction of a surface of the object corresponding to the tracking area and a direction of reflected light. 13. In the second step, a tracking area corresponding to a plurality of images is mapped to a three-dimensional space, and when a matching degree obtained from a correspondence between boundary elements of the tracking area is equal to or larger than a threshold value. A tracking area mapped in a three-dimensional space is used as a reference area, a boundary element between the tracking area corresponding to the reference area and the reference area is compared, and an area that is not imaged in the tracking area due to a change in the boundary element is extracted. The image feature tracking processing method according to claim 1, wherein: 14. A tracking area in which the number of boundary elements is not changed between a plurality of images and a contribution ratio equal to or more than a threshold value is used for each range by using a contribution ratio obtained at the time of mapping as a degree of coincidence. 14. The image feature tracking processing method according to claim 13, wherein is set as a reference region candidate, and when a plurality of reference region candidates are obtained, a candidate having the largest area is adopted as the reference region. 15. The image feature tracking processing method according to claim 14, wherein the change in the number of boundary elements focuses on one of a line element and a point element. 16. The image feature tracking processing method according to claim 14, wherein said contribution ratio is obtained from a diagonal matrix component obtained when mapping the tracking area into a three-dimensional space by a factorization method. 17. The tracking area is mapped in a three-dimensional space, compared with a reference area, and an area defined by point elements that do not correspond to each other is determined as an area not imaged in the tracking area. 14. An image feature tracking processing method according to claim 13. 18. A tracking area is mapped in a three-dimensional space and compared with a reference area. When there is a line element that does not correspond to each other, the line element is regarded as a part of an area not imaged in the tracking area. 2. The method according to claim 1, wherein
3. The image feature tracking processing method according to 3. 19. In the third step, all the reference regions in the three-dimensional space are projected on an image plane determined by the position of the TV camera, and the positional relationship between the reference regions in the image plane and the reference to the image surface 2. The image feature tracking processing method according to claim 1, wherein a cause of the occurrence of an area that has not been imaged in the tracking area is determined based on the positional relationship between the areas. 20. The method according to claim 19, wherein when there is another reference area between the reference area of interest and the TV camera, and the two reference areas are not connected, it is determined that another person is concealed. Image feature tracking processing method. 21. The self-concealment method according to claim 19, wherein when another reference area exists between the reference area of interest and the TV camera, and both reference areas are connected, self-concealment is determined. Image feature tracking processing method. 22. The image feature tracking processing method according to claim 19, wherein when the reference area of interest is located at the periphery of the screen, the frame is determined to be out. 23. When there is a region not imaged in the tracking region, edge extension is performed, and a tracking process is executed again with the intersection of the extended edge as a new point element.
20. The image feature tracking processing method according to claim 19, wherein when mapping is performed in a three-dimensional space, if the contribution rate increases, concealment is determined. 24. An image input unit for providing an image input image picked up by a TV camera, an image processing apparatus for performing processing by the image feature tracking processing method according to claim 1 on the input image, A storage device for storing the image and the image processed by the image processing device; a display unit for displaying the processed image in the image processing device; and an area designating unit for designating a tracking area for the image processing device. An image feature tracking processor that is a feature. 25. After capturing a plurality of images by capturing the target object from a plurality of different positions using a TV camera, a plane selected from the target object is set in the image as a tracking area represented by a boundary element. A deformation area is created by deforming the tracking area set in the image so as to match the boundary element of the tracking area candidate in the other image, and the boundary element is determined for the tracking area candidate and the deformation area in the other image. A tracking area corresponding to the tracking area set in the one image is selected from the tracking area candidates by comparing, and a plurality of surfaces of the object are imaged and the same surface is continuously imaged. By dividing a plurality of images into one group of phase images, and then mapping the plurality of phase images to a three-dimensional space of a surface selected from an object for each phase image group. Three-dimensional data is obtained for each phase image group, and three-dimensional data is generated by performing coordinate transformation so that the coordinate systems of the other phase image groups match each other. . 26. After capturing a plurality of images by imaging a target object from a plurality of different positions with a TV camera, a surface selected from the target object is set in the image as a tracking area represented by a boundary element. A deformation area is created by deforming the tracking area set in the image so as to match the boundary element of the tracking area candidate in the other image, and the boundary element is determined for the tracking area candidate and the deformation area in the other image. A tracking area corresponding to the tracking area set in the one image is selected from the tracking area candidates by comparing, and a plurality of surfaces of the object are imaged and the same surface is continuously imaged. By dividing a plurality of images into one group of phase images, and then mapping the plurality of phase images to a three-dimensional space of a surface selected from an object for each phase image group. By obtaining a three-dimensional shape for each phase image group, and performing coordinate transformation of the three-dimensional shape obtained for each phase image group so as to overlap line elements common to other phase image groups. A method for creating three-dimensional data, comprising creating three-dimensional data. 27. A surface selected from the object in the plurality of images captured and acquired by the TV camera is a boundary element composed of a point element and a directed line element sequentially connecting the point elements. 26. The method according to claim 25, wherein in each of the images obtained in a time series, the direction in which each point element is traversed by tracing the line element is monitored, and the phase until the circulating direction is reversed is regarded as one phase image group. 27. The three-dimensional data creation method according to claim 26. 28. In the coordinate conversion, after obtaining conversion parameters related to rotation and translation for each boundary element,
27. The three-dimensional data creation method according to claim 25, wherein coordinate transformation is performed using a transformation parameter obtained by averaging transformation parameters obtained for each boundary element. 29. A midpoint between the position of each point element after performing the coordinate transformation on one of the three-dimensional shapes whose coordinate systems match each other and the position of each corresponding point element on the other three-dimensional shape. 27. The method according to claim 25, wherein the position of each point element is set.