JP2003022442A - Method and device for object detection and position measurement, execution program for the same method, and its recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To perform object detection and its position measurement robustly and very fast with increased computation efficiency. SOLUTION: An object features quantity learning means 1 learns object features and features quantity distribution from a set of reference images having different distances, orientations, etc. A panning, tilting, and zoom-parameter computing means 2 computes parameter information needed for a search from the size of the features quantity distribution. A camera control and image input means 3 photographs an image which is not searched for with the parameter information to generate a features quantity image. An input image features quantity distribution computing means 4 computes the features quantity distribution in a window of interest set in the features quantity image. A features quantity collating means 5 computes the similarity value between the object features quantity distribution and the features quantity distribution in the window of interest to detect an object. A collation-omitted area computing means 6 computes many object features quantity similar to the object features and collation-omittable areas between many windows of interest nearby the window of interest. An object position computing means 7 computes the object position based on the detection direction of the detected object.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method and apparatus for detecting an object in a three-dimensional real environment in which a region similar to an object registered in advance is detected and measuring the position thereof. For example, a pet robot. It can be used as a non-contact sensor for eyes, information appliances, etc. That is, it relates to sign / target recognition by a mobile robot, an indoor monitoring system, a product management system, and the like.

[0002]

2. Description of the Related Art Conventionally, with respect to high-speed object detection, there is a method for searching at high speed using quantized colors at equal intervals, such as "Object detection device" (Japanese Patent Application No. 8-146857; prior application 1). A method for robustly searching for illumination fluctuations by a template matching method based on normalized cross-correlation, and a basic framework of camera control, "object detection method and apparatus and recording medium recording this method" (Japanese Patent Application No. 2000-35336; A method in which the request 2) is combined is known.

[0003]

However, in this conventional method, in the prior application 1, in the case of searching a three-dimensional real environment with a lot of fluctuations, the search cannot be performed with sufficient accuracy, and the template matching method by the normalized cross correlation is used. Since the matching cost is enormous, and it is weak against changes in orientation, a large number of reference images must be used to cope with changes and fluctuations, and the matching cost is proportional to the number of reference images. The number has increased, and the prior application 2 of “Object detection method and apparatus and recording medium recording this method” (Japanese Patent Application No. 2000-35336)
Even with this, there was the drawback that it would take an enormous amount of time.

The present invention provides a method and apparatus for performing object detection and position measurement which are more computationally efficient than known methods, and which are more robust and faster than known methods and at high speed in overwhelmingly less time. The challenge is to provide.

[0005]

[Means for Solving the Problems] In order to solve the above problems, one means of the present invention detects an object similar to an object registered in advance by using an active camera having a pan / tilt / zoom function, A method for detecting and measuring a three-dimensional position of an object, the object feature amount being learned from a set of reference images of a large number of objects photographed by changing imaging conditions such as perspective, direction, and illumination. The object feature amount learning process of learning the object feature and the feature amount distribution, and the pan required to search the entire field of view of the camera from the size of the feature amount distribution obtained in the object feature amount learning process on the image. Pan that calculates camera parameters for tilt and zoom
Of the camera parameter information obtained in the tilt / zoom parameter calculation process and the pan / tilt / zoom parameter calculation process, a camera is selected as a camera parameter that has not yet been searched, an image is captured, and the captured input image is displayed. For each pixel of the camera control / image input process for generating the feature amount image converted into the feature amount obtained in the object feature amount learning process, and the feature amount image obtained in the camera control / image input process, A target window corresponding to the size of the object feature amount distribution to be collated on the image is set, and the feature amount distribution calculation process for calculating the feature amount distribution in the target window and the object feature amount learning process are performed. A feature distribution for detecting an object by calculating a similarity value or a distance value between the object feature distribution and the feature distribution in the window of interest set in the input feature distribution calculation process and determining the presence or absence of the object Matching process, and matching between a large number of other object feature amounts similar to the object feature and a large number of attention windows around the attention window based on the similarity value or distance value calculated in the feature amount distribution comparison process. An object characterized by comprising a matching omission area calculation process for calculating an eliminable region and an object position calculation process for calculating an object position from the detection direction when an object is detected in the feature amount distribution matching process. detection/
It is a position measuring method.

Alternatively, the object detection / position measurement method is characterized in that an object feature quantity using a color code by vector quantization of color information is used in the object feature quantity learning process.

Alternatively, in the object feature amount learning process, an object detection / position measuring method is characterized in that a color code by vector quantization limited to colors included in the object is used for the object feature amount.

Alternatively, the object detecting / position measuring method is characterized in that a histogram of object characteristics is used for the object characteristic amount distribution in the object characteristic amount learning process.

Alternatively, in the feature amount distribution matching process, among the object feature amount distributions learned in the object feature amount learning process, an object feature amount distribution that cannot be treated as detecting an object due to erroneous detection or omission of detection is " It is characterized in that it is used as a "candidate selection" object feature amount distribution, and in the camera control / image input process, the camera is preferentially pointed to an area evaluated to have an object by the "candidate selection" object feature amount. This is an object detection / position measurement method.

Alternatively, the object detection / position measurement method is characterized in that the evaluation of the “feature selection” object feature quantity uses the evaluation by the similarity value or the distance value by the composite distribution.

Alternatively, in the feature amount distribution matching process, the upper limit value or the distance value of the similarity value of the local partial area around the local partial area used for the calculation of the similar value or the distance value is calculated from the calculation result of the similar value or the distance value. The object detection / position measurement method is characterized in that the lower limit value of is calculated, and the calculation of the similarity value in the region that does not reach the threshold value is omitted.

Alternatively, in the object feature amount learning process, a similarity value or a distance value between the object feature amount distributions is calculated in advance, and in the feature amount distribution matching process, the calculation result of the similarity value or the distance value and the object feature amount distribution are calculated. The upper limit value of the similarity value or the lower limit value of the distance value of the object feature amount similar to the object feature amount used for the calculation of the similarity value or the distance value is calculated from the similar value or the distance value between The object detection / position measurement method is characterized in that the calculation of the similarity value or the distance value of the object feature amount is omitted.

Alternatively, in the object feature amount learning process, a combined distribution obtained by combining a plurality of object feature amount distributions into one is calculated in advance, and in the feature amount distribution matching process, the combined distribution and a local partial region on the input image are calculated. An object characterized by calculating the upper limit value or the lower limit value of the distance value of the object feature amount distribution from the calculation result of the similarity value or the distance value and omitting the similarity calculation of the object feature amount that does not reach the threshold value This is a detection / position measurement method.

Alternatively, in the object position calculation process, the object position is calculated based on the direction of the object detected by the camera alone and the known size of the object.

Alternatively, in the object position calculating process, the object detecting / position measuring method is characterized in that the object position is calculated based on the directions of the objects detected by a plurality of cameras.

Alternatively, it is an apparatus for detecting an object similar to an object registered in advance by using an active camera having a pan / tilt / zoom function, and performing object detection and position measurement for obtaining a three-dimensional position of the object. , An object feature amount learning unit that learns an object feature amount from a reference image set of a large number of objects photographed by changing shooting conditions such as perspective, orientation, and illumination, and the object feature amount learning unit, The pan / tilt / zoom camera parameters required to search the entire field of view of the camera from the size of the feature distribution on the image obtained by the quantity learning means
Of the camera parameter information obtained by the tilt / zoom parameter calculation means and the pan / tilt / zoom parameter calculation means, a camera is selected as a camera parameter that has not yet been searched, an image is taken, and the input image is taken. A camera control / image input unit that generates a feature amount image converted into a feature amount obtained by the object feature amount learning unit for each pixel of, and a feature amount image obtained by the camera control / image input unit, It is obtained by the input feature amount distribution calculating means for setting the target window corresponding to the size of the object feature amount distribution to be collated on the image and calculating the feature amount distribution in the target window, and the object feature amount learning means. A feature distribution for detecting an object by calculating a similarity value or a distance value between the object feature distribution and the feature distribution in the window of interest set by the input feature distribution calculating means, determining the presence or absence of the object Matching means and matching between a large number of other object feature quantities similar to the object feature based on the similarity value or distance value calculated by the feature quantity distribution matching means and a large number of attention windows around the attention window. An object characterized by comprising collation omitted region calculation means for calculating an eliminable region, and object position calculation means for calculating an object position from the detection direction when the feature amount distribution collation means detects an object. detection/
It is a position measuring device.

Alternatively, there is provided an object detection / position measurement method execution program characterized in that a program for causing a computer to execute the procedure in the object detection / position measurement method.

Alternatively, the object detecting / position measuring method is characterized in that the procedure in the object detecting / position measuring method is a program for causing a computer to execute, and the execution program is recorded in a recording medium readable by the computer. It is a recording medium in which an execution program of the measuring method is recorded.

In the present invention, in particular, the "object detection device" (Japanese Patent Application No. 8-146857) and the template matching method are referred to as "object detection method and device and recording medium recording this method" (Japanese Patent Application No. 2000-35336). Compared with the method combined with, the features with high accuracy (fewer search omissions and less false detections) are used in the object feature amount learning process / means, and it is possible to change the lighting and camera parameters expected in a 3D real environment. It is robust against changes in the amount of object features that occur. In addition, a feature amount distribution such as a histogram can be calculated for these feature amounts, and the matching search using a reference image of a large number of objects can be significantly speeded up while ensuring accuracy in the feature amount distribution matching process / means. 5-9). In particular, in the three-dimensional environment search which is the subject of the present invention, the variation of the object feature distribution is large due to the difference in the direction, the difference in focus, etc. Therefore, the use of the synthetic feature used in claim 9 is effective. In addition, the number of camera controls can be reduced by increasing the number of image searches according to the fifth aspect. The image search time can be greatly reduced by the method according to claim 6 with almost no loss in accuracy, and the search time can be greatly reduced as a whole. The above is the main point of the present invention.

[0020]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described below with reference to the drawings.

FIG. 1 is a block diagram showing an embodiment of an object detecting and position measuring apparatus to which the present invention is applied.
Here, a color code obtained by vector quantization limited to colors included in an object that seems to have a relatively high effect is used for the object feature amount, and a histogram of the color code is used for distribution of the object feature amount. As the pan / tilt / zoom parameters for controlling the camera, “object detection method and apparatus and recording medium recording this method” (Japanese Patent Application No. 2000-3533).
The method shown in the example embodiment of 6) is used. The overlapping ratio of the feature amount distributions to be collated is used as the similarity value scale of the feature amount distribution collation. In the camera control method, in order to search for a “detection” object feature amount distribution that treats an object as detected, there is a matching omission method that uses similar values between object feature distributions and an “object detection device” (Japanese Patent Application No. No. 8-146857), it is used in combination with the collation omission method for the peripheral area of the local partial area on the input image, which is used for “candidate selection”.
For the search of the object feature amount distribution, the collation omission using the composite distribution and the “object detection device” (Japanese Patent Application No. 8-146857).
No.) is used in combination. The object position calculation uses a method of calculating the object position from the detection results from a plurality of cameras.

The object detecting and position measuring apparatus shown in FIG. 1 comprises an object feature quantity learning means 1, a pan / tilt / zoom parameter calculation means 2, a camera control / image input means 3.
, An input image feature amount distribution calculation unit 4, a feature amount distribution collation unit 5, a collation omission region calculation unit 6, and an object position calculation unit 7, which represent a reference image, that is, a sample object. Input the lighting condition, orientation, image taken with scale, initial position / direction / viewing angle of the camera, etc., and take the image with the threshold of the similarity of the object feature distribution of the reference image to the preset threshold value θ or more. Outputs the direction / three-dimensional position of the object from the captured camera.

The object feature amount learning means 1 learns an object feature amount from a reference image set of an object shot by changing shooting conditions such as perspective, direction, and illumination, and learns an object feature and a feature amount distribution.

The pan / tilt / zoom parameter calculating means 2 is necessary for searching the entire field of view of the camera from the size of the feature amount distribution obtained from the object feature amount learning means 1 on the image. -Calculate the zoom position.

The camera control / image input means 3 picks up an image by selecting a camera from among the camera parameter information obtained by the pan / tilt / zoom parameter calculation means 2 as a camera parameter that has not yet been searched, A feature amount image is generated by converting each pixel on the captured input image into the feature amount obtained by the object feature amount learning unit 1.

The input image feature amount distribution calculating means 4 sets a notice window corresponding to the size of the object amount distribution to be collated on the feature amount image obtained by the camera control / image input means 3, The feature distribution inside the window of interest is calculated.

The feature amount distribution collating means 5 is a similarity between the object feature amount distribution obtained by the object feature amount learning means 1 and the feature amount distribution within the target window set by the input image feature amount distribution calculating means 4. The value is calculated and the presence or absence of an object is determined.

The collation omission area calculation means 6 is based on the similarity value calculated by the characteristic amount distribution collation means 5 and is based on the similar values, a large number of other object feature quantities similar to the object feature and a large number of attention windows around the attention window. Calculate the optional area matching between and.

When the feature amount distribution matching unit 5 detects an object, the object position calculating unit 7 calculates the object position from the detection direction.

Next, with reference to the flow chart of FIG. 2, the processing in the object feature amount learning means 1 to the object position calculation means 7 of the above-mentioned configuration of FIG. 1 will be concretely described. S in FIG.
11 to S20 are steps representing the procedure of the processing.

In the object feature amount learning process of step S11, the object feature amount learning means 1 first takes an image of objects placed at various positions by changing the position and direction while changing the zoom value. As for the position of the object, in the case of a room with a size of about 30 m2, it is sufficient to take images at intervals of about 2 m.
In the case of the object shown in (3), it was sufficient to photograph in three directions: front, left and right. Further, even when more images are taken than this, the processing time hardly changes due to the method of omitting matching between the object feature amounts described later.

Next, the object area is cut out from the photographed image. Each image cut out is a reference image. Here, the area inside the object is cut out so that the background features are not included. Although the method described in the present invention can be easily applied to a region having an arbitrary shape, a rectangular case is used for the sake of simplicity. In addition, as a vector quantization method, it is possible to calculate an accurate representative vector by calculating a distance with the representative vector every time, but here, with emphasis on speed, a method using an encoding table described below is used. To adopt.

The encoding table shows which VQ code a small section of each color obtained by dividing each RGB axis into 32 (32 ³ sections) belongs, and associates the color with the VQ code in a constant time. Further, when it is intended to be used for object search, since VQ can be performed only on colors included in an object, by introducing a special code 0 representing a color not included in an object,
Codes can be efficiently assigned to colors included in an object. In this method, the number of codes is small, so that the histogram matching calculation becomes fast. Further, it is possible to quickly prun an area where no object is captured. The details of the procedure are described below.

Step 1. Calculate the average color and variance of all pixels. This average color is designated as the representative color C ₁ and all pixels are assigned to C ₁ .

Step 2. Select the representative color C _i with the maximum variance. If the variance α _i <σ of C _i , jump to step 4.

Step 3. C _i is divided into two, and the optimum representative color is determined by the well-known method LBG algorithm. The distances between all pixels and all representative colors are calculated and each pixel is assigned to the closest representative color. The average color and variance α of the colors of the pixels belonging to the representative color are calculated. Return to step 2.

Step 4. A representative color C _j closest to all the colors P _{i in the} color space is obtained. │P _i −C _j │ < _n α _j
If so, the code of C _j is assigned to P _i . Otherwise,
Assign code 0 to P _i .

The value n is empirically 2-3 so that the ratio of the pixels included in the code 0 among the pixels used for learning is less than a certain value.
Decide on the degree. When n is too small, the ratio of code 0 increases, and when it is too large, the ratio of codes other than 0 that include a color that is not similar to the color of the object increases. In either case, false positives increase. On the other hand, when σ is small, it is vulnerable to changes in illumination and orientation, and when it is large, erroneous detection increases. Therefore, it is determined according to the color distribution of the object.

The object feature amount learning means 1 color-codes each pixel in the reference image for each reference image, generates a histogram in which the number of each code is counted, and sends it to the feature amount distribution collating means 5. Also, the size of the reference image area that can be used for sufficient detection is empirically determined for each object (10
(0 pixels to 1000 pixels), and sends them to the pan / tilt / zoom parameter calculation means 2. Further, the obtained encoding table is sent to the camera control / image input means 3.

When the similarity of the reference image histogram or a histogram obtained by synthesizing the histograms is used for the calculation of the matching omission area, the similarity value and the combined histogram are calculated in advance by the object feature amount learning means 1 and the matching is performed. The data is sent to the omitted area calculation means 6 (the flow of the dotted line in FIG. 1 (claims 8 and 9)). The calculation method of the similarity value will be described in the feature amount distribution matching unit 5, and the calculation method of the composite histogram will be described in the matching omitted region calculation unit 6.

In the pan / tilt / zoom parameter calculation process of step S12, the pan / tilt / zoom parameter calculation means 2 determines the pan / tilt / zoom of the camera according to the size of the object accumulated in the object feature amount learning means 1. Set the zoom interval. Specifically, a parameter is used that guarantees accuracy and allows the entire field of view to be photographed as few times as possible. For this purpose, a known method, "object detection method and apparatus and recording medium recording this method" (Japanese Patent Application No. 2000-35336) is used. "Object detection method and apparatus and recording medium recording this method" (Japanese Patent Application No.
000-35336), the larger the ratio of the maximum object area size to the image size, the more the number of zooms can be reduced, but the number of overlapping areas needs to be increased. This is a method for obtaining the maximum object area size that minimizes the number of tilts and zooms from the minimum object area size.

First, as the output of the object feature amount learning means 1, the minimum size of the object feature amount distribution that can be guaranteed with sufficient accuracy for use in detection is obtained, and this is recorded as "object detection method and apparatus and this method. Recording medium "(Japanese Patent Application 20
00-35336) to find all parameters of pan / tilt / zoom over the entire field of view of the camera (including from near to far). The pan / tilt / zoom parameter calculation means 2 saves all the parameters calculated in this way as a list and sends them to the camera control / image input means 3.

In the camera control / image input process of step S13, the camera control / image input means 3 moves the camera to the parameter at the head of the camera parameter list obtained by the pan / tilt / zoom parameter calculation means 2. When the operation of the camera is completed, an image is taken and an input image is obtained. A feature amount image obtained by converting each pixel on the input image using the encoding table obtained by the object feature amount learning unit 1 is generated and sent to the input image feature amount distribution calculation unit 4.

Further, as a preparation for the input image characteristic amount distribution calculating means 4, a three-dimensional array of the bool values having the object characteristic amount distribution and the collation position on the input image as a subscript is created (hereinafter referred to as collation array). The size is the number of object feature distributions × the total number of windows of interest to be matched on the input image. All true are substituted as initial values. As for the search order of the object feature amount, it can be expected that the larger the number of pixels, the larger the overlap, and the greater the collation omission effect, so the search is performed in descending order of the number of pixels. On the other hand, when the similarity value between the object and the input image is large, the search order of the window of interest is faster when the pixels are shifted by one pixel, and when the similarity value is smaller, the large interval is faster.
This is because when the window of interest is shifted, the histogram of the shifted window of interest can be generated by adding the code of the newly entered pixel to the original histogram and reducing the code of the pixel that has left. It is faster than replacing all of them. On the other hand, when the similarity value with the input image is small, the collation can be omitted for the upper, lower, left, and right local regions as a result of one collation. In the case of shifting by one pixel, collation can be omitted only in the traveling direction, which is inefficient. Generally, most of the input images are expected not to be similar to the object, so first of all, the target region is collated at coarse intervals (about 0.5 times the size of the reference image), and then the remaining regions are Relatively good results can be obtained by comparing with each other.

In the step 14 of calculating the input image feature amount distribution, the input image feature amount distribution calculating means 4 selects one of the remaining attention windows on the feature amount image obtained by the camera control / image input means 3. The feature amount distribution inside the window of interest is calculated, and if there is a collation region, the obtained feature amount distribution is sent to the feature amount distribution collating means 5. If there is no collation region, the process returns to step 13 (step 15).

In the feature quantity distribution matching process of step 16, the feature quantity distribution matching means 5 first outputs the histogram of the reference image output from the object feature quantity learning means 1 and the input image feature quantity distribution calculation means 4. It receives a histogram of local regions on the input image. Then, for example, the similarity value (similit
Calculate y). The overlap rate S _AM of the histogram features H _M and H _A is defined as follows.

[0047]

[Equation 1]

Here, H _M and H _A are reference images M, respectively.
Is a histogram for the local area A in the input image, and H _Mi and H _Ai are the numbers of pixels each having the i-th code. Also, | M | is the number of pixels of the reference image (total number of features of the reference image), and I is the type of code.

If S _AM is greater than or equal to the search threshold value θ, it is detected as an object and the direction and size of the detected object are output to the object position calculation means 7 (step 17).

If the camera is dynamically controlled using the histogram for candidate selection (step 18 (see FIG.
If the value of the similarity value exceeds the preset threshold value for prediction, it can be known that there is an object at the current window position of interest. Returning to step 12, rearrangement is performed so that the camera parameter including the area after zooming in is at the top. The matching of the histogram for selecting the candidate with respect to the area that is to be zoomed in is terminated. When a plurality of images are detected, the camera parameter list is rearranged so that they are arranged in descending order of overlap.

When S _AM is equal to or smaller than the search threshold value θ, this similar value is output to the collation skipping area calculating means 6 (step 18).

In the matching omission area calculation process of step 19, a method of omitting the peripheral local area / similar object feature distribution matching using the matching result by the feature amount distribution matching means 5 performed by the matching omission area calculating means 6 will be described. After the processing in step 19, the process returns to step 14.

First, the local area pruning method will be described with reference to FIG. Let two reference images of different sizes be M and N (| M |> | N |). There is no change in the colors of M and N. That is, the histogram of N is a constant multiple of the histogram of M, and it is assumed that all N colors are included in M (generalization to the case where different colors are included will be described later). Where | A | = | M
Consider local areas A and B that satisfy |, | B | = | N | and have overlapping areas. Similarity value S between reference image M and local area A
The inequality (2) is established between _AM and the similar value S _BN of the reference image N and the local area B.

| N | · S _BN <| M | · S _AM + n (2) Here, | M | · S _AM indicates how many sets of pixels of the same color are present between A and M. means. This will be called the number of pixels of the same color. This formula is “the number of pixels of the same color between the object and B (| N |
・ S _BN ) has the same number of pixels (| M | · S _AM ) of the same color between the object and A localized in the common region of A and B, and the region of B not included in A (all pixels in FIG. 4 are There is no more than when the object and B have the same color pixel. ”That is, when S _AM becomes known, the upper limit of | N | · S _BN becomes the right side of equation (2). If the upper limit value is smaller than | N | .theta., The collation between N and B can be omitted, and the search can be speeded up.The range of n at which the collation can be omitted is given by the following equation.

| M | · S _AM + n <| N | · θ ∴n <| N | · θ− | M | · S _AM (3) Next, in the reference image N, a pixel whose color is different from that of the reference image M is used. Expression (2) is extended to handle the case where is included. Among N pixels, the number of pixels different in color from M (referred to as different color pixels with respect to M of N) is represented by | N | (1-S _MN ) (FIG. 5). S
_{Since AM} does not provide information as to whether a different color pixel for N M exists in A, a different color pixel for N M is A
It is necessary to estimate the upper limit of | N | · S _BN , assuming that it appears inside, and obtain a region that does not reach | N | · θ. That is, the following inequality (4) is derived. OR this method
Let's call it search.

| N | · S _BN <| M | · S _AM + | N | (1-S _MN ) + n n <| N | · (θ−1 + S _MN ) − | M | · S _AM (4) On the other hand, when the difference in the histograms between the reference images to be searched is large, the addition part | N | (1
-S _MN ) · | M | becomes large, and the effect of omitting collation is small. Therefore, it is considered that unnecessary matching is omitted by reflecting the difference in the histogram in the similar value unlike in the OR search, but by reflecting the difference in the histogram itself (FIG. 6). All histograms of the reference image

[0057]

[Equation 2]

It is assumed that At this time, the composite histogram H _U

[0059]

[Equation 3]

It is defined as H_UiIs the number of each code in U
It H_UiEach element of consists of the maximum value of each reference image
Therefore, the similarity value between certain search windows A and U belongs to A and U.
M⁰, M¹, ..., M^kIs always greater than the similarity value of. Therefore
｜ U ｜ ・ S_AUIs | M | ・ S^J _AM(∀j ∈ 1, ..., k)
It becomes the upper limit. By searching U⁰, M¹，
…, M ^kCan be omitted. However, the similar value itself is calculated
The area detected by U, if necessary.
And M⁰, M¹, ..., M^kNeed to search using
It Of course, selecting candidates to zoom in later and evaluate again
It is not necessary to search repeatedly in the selection stage.

This method will be called a UNION search.

All the collating sequences corresponding to the window of interest having the overlapping area in the range of n are set to false.

When an object is detected, in the object position calculation process of step 20, in the object position calculation means 7, when the object direction is detected by the feature amount distribution matching means 5, the coordinates of the object on the image are detected.

[0064]

[Equation 4]

Is obtained.

From now on, the camera coordinate system (the center of the lens of the camera is the origin, on the image plane, the pan axis [rightward right] on the left and right, the tilt axis [upward positive] up and down, the Z axis [forward positive] on the optical axis).
The direction of the object at θ (θ _x , θ _y [rad])) is obtained.

Three-dimensional external parameters of camera (position

[0068]

[Equation 5]

, Direction

[0070]

[Equation 6]

))). i is the camera number, and the three-dimensional external parameters (position

[0072]

[Equation 7]

)).

[0074]

[Equation 8]

However, the field of view is

[0076]

[Equation 9]

The input image size is

[0078]

[Equation 10]

It is

The size of the object

[0081]

[Equation 11]

When is known, the distance L (l _x ,
l _y ) is

[0083]

[Equation 12]

It becomes At this time, the coordinates of the object are

[0085]

[Equation 13]

Can be obtained as Where Y is Y
Rotation around the axis, and Z indicates rotation around the Z axis.

However, in the case of histogram matching, the accuracy of the size of the detected object is not very accurate. Therefore, it is also possible to measure the object position by stereoscopic viewing with a large number of cameras. When an object is detected by a certain camera and the position is obtained, the other camera is selected from the camera parameters capable of detecting the object at the position and the camera parameter capable of detecting the object at the position considering the error.
The camera parameter with the stepwise wide angle is placed at the top of the camera parameter list. When an object is detected by two or more cameras, the search ends. When the object is not detected by another camera, the first detection result can be set as the correct answer, but here, the search is continued until the object is detected as an incorrect answer by two or more cameras.

A position measuring method when an object is detected by two or more cameras will be described below.

The coordinates of the object are

[0090]

[Equation 14]

(K: unknown number representing distance, α = d ⁱ _x + θ ⁱ _x , β = d ⁱ _y + θ ⁱ _y ).

When k is eliminated, q _x cos (α) + q _y sin (α) = p _x cos (α) + _py sin (α) (10) q _y sin (β) −q _z cos (α) sin (Β) = p ⁱ _y sin (β) + p ⁱ _z cos (α) sin (β) (11) Initially, three of the coordinates of Q are unknowns, and each time the result of one camera increases, the number of unknowns and the number of formulas increase by one.
That is, if there are two or more cameras, the position of the object can be obtained. Therefore, when two or more detections are performed, a well-known method such as singular value decomposition can be used to obtain an inverse matrix that minimizes the residual error, and an accurate object position can be obtained.

Next, the results of experiments conducted to show the feasibility and specificity of the present invention will be described. In this experiment, an object detection experiment was performed in the environment obtained in FIG. 8 at the widest angle with the specifications described in FIG. 7. The object used is the object of FIG. The size of the object is about 5 cm to 10 cm in height.

First, at the learning stage, about 100 target objects were appropriately photographed and used as reference images. More specifically, in order to change the illumination condition, they were arranged at five points A to E in FIG. 8, and the front, right and left three directions in total of the object were photographed by changing the camera zoom by 1.2 times.

In the search stage, the object was placed at 5 points a to e in FIG. 8 from near to far within 5 m from the camera. Furthermore, the orientation of the object was changed to the left and right. At this time, the contour shape of the object may be different. Although the active search method can be applied to this case as well, in order to perform the search at a higher speed, a rectangle inside the object with similar contour shapes was used as the reference image. As for σ used for the histogram partition generation, a large σ is selected for each object having a large color variation due to a change in orientation, and a small σ is selected for each object having a small color variation (σ = 1.5-3). σ did not significantly affect the calculation speed. It is presumed that the reason is that the effect of improving the calculation speed due to the decrease in the number of codes and the effect of decreasing the pruning efficiency due to the increase in the background similarity value are offset. The threshold was set for each object by preliminary experiments.

In order to evaluate the search accuracy by the code feature (VQ method) generated by the vector quantization limited to the colors included in the object, the difference in the search accuracy from the conventional method using the equally divided histogram partitions was evaluated. . As the input image, 15 images taken in the wide-angle stage were used.

Precision is a precision ratio (Precision rat)
It evaluated by the average value of e) and a recall rate. Here, the precision ratio is a ratio of correct ones output as the search result, and the recall ratio is a ratio of those output as the search result among those to be searched. If both precision and recall are 100%,
This means that there was no missed detection or extra detection.

The experimental results are shown in FIG. In this experiment, the accuracy was evaluated using the size of the search window as a parameter.

According to the conventional method, in the case where the length of one side is 10 to 30 pixels, 60
We could only achieve accuracy of about%. On the other hand, the VQ method
Even for an object with 10 to 30 pixels on a side, an accuracy of nearly 90% was achieved.

Next, in order to evaluate the effect of speeding up the OR method and the UNION method, the search speed was measured under the same conditions as in the previous experiment (FIG. 10).

The methods used for comparison are "object detection device" (Japanese Patent Application No. 8-146857), VQ method + "object detection device".
(Japanese Patent Application No. 8-146857) (VQ method), an OR method in which an OR search is performed in addition to the VQ method, and a UNION method in which a UNION search is performed similarly.

First, it can be seen that the VQ method improves the search time about twice. By using the OR method, the speed is further improved by about 10% while ensuring the accuracy. On the other hand, in the UNION method, the speed is further improved by about 80%. The reason why the speed is improved by the UNION method is
This is because the collation of 5 to 10 multiple reference images is collectively searched by one. In the merge search, there is no omission in object detection, but there is a risk of erroneously detecting another area. However, in this case, since the histograms for a plurality of reference images are similar, the accuracy is not completely guaranteed, but sufficient accuracy is obtained for use in camera control.

Further, the above method was evaluated for the object search requiring camera control (FIG. 11). In this experiment,
A to c that requires at least one zoom to detect
The object in is the search target.

Methods for comparison include a method without prediction (conventional method), a VQ prediction method in which VQ is introduced into a conventional active search to detect an object and dynamically control a camera, and an object detection and a camera in a VQ + OR search. VQ + OR for dynamic control of
Method, and the VQ + UNION method in which the object is detected by the VQ + OR search and the camera is dynamically controlled by the VQ + UNION search.

The search precision was 100% in all cases. It can be seen that the time until object detection was reduced by about 50% by the dynamic control. 1 in parallel search
The search time could be reduced by 0% and 50% by the merged search. However, in the merged search, when the size of the search window is a small area of 10 pixels or less on one side, useless camera control is increased and search time is slightly increased due to increase in false detection of search candidates for some objects. Sometimes I did.

Finally, the results of object detection by cameras installed in the four corners of a 7.5 m × 3.5 m room will be described. Using the same learning data, an experiment was conducted on objects placed at positions α to f shown in FIG. FIG. 13 shows how detection is performed. 14 (1) to (3) are obtained by obtaining the object position from the obtained line-of-sight direction. The straight line represents the line of sight, and the square represents the coordinates of the intersection determined by singular value decomposition. It can be seen that robust detection can be performed by using a large number of line-of-sight information. FIGS. 15 (1) and 15 (2) show the measurement results examined at all points. The circle in FIG. 15 (1) represents the number of detected cameras. It has been detected by almost four cameras. The number of cameras detected may not be less than 2 in this experiment, although the number may be 3 or less when it is hidden or when it is very far away and invisible. Fig. 15 (2)
Indicates the error between the true value and the measured value. It can be seen that the more cameras, the closer to the center the accuracy is. Here, the object position is obtained with an average error of 28 cm and a standard deviation of 6.7 cm, which is a good result for detecting the position of an object having a size of about 30 cm.

In the above embodiment, the case where the similar value is used has been described, but it is possible to use the distance value instead of the similar value. In this case, the upper limit of the similarity value becomes the lower limit of the distance value.

Further, a part or all of the functions of each unit in the apparatus shown in FIG. 1 can be configured by a computer program, and the program can be executed by the computer to implement the present invention, or It goes without saying that the processing procedure shown in 2 can be configured by a computer program and the computer can be caused to execute the program, or the computer executes the processing procedure. The program for executing the program is recorded on a computer-readable recording medium, for example, F
D (floppy disk (registered trademark)), MO, RO
It is possible to record, save, or distribute to M, a memory card, a CD, a DVD, a removable disk, or the like. It is also possible to provide the above program through a network such as the Internet or electronic mail.

[0109]

As is apparent from the above, according to the present invention, in particular, "object detecting device" (Japanese Patent Application No. 8-146857).
No.) or template matching method, "Object detection method and apparatus and recording medium recording this method" (Japanese Patent Application No. 2000-
35336), a feature with high accuracy (less search omission, less false detection) is used in the object feature amount learning process / means, and the illumination and camera parameters that are assumed in a three-dimensional real environment are used. It is robust against changes in the object feature amount caused by changes. Further, for these feature amounts, a feature amount distribution such as a histogram can be calculated, and the collation search with reference images of many objects can be significantly speeded up while the accuracy is guaranteed in the feature amount distribution collation process / means. In particular, in the three-dimensional environment search which is a subject of the present invention, the variation of the object feature distribution is large depending on the direction difference, the focus difference, and the like, and thus the use of the synthetic feature is effective. Further, the number of camera controls can be reduced by increasing the image search by using the “candidate selection” object feature amount distribution. The image search time can be greatly reduced by evaluating the “candidate selection” object feature amount with the similarity value or distance value of the composite distribution, and it is possible to significantly reduce the accuracy, and the search time as a whole. Can be greatly reduced.

[Brief description of drawings]

FIG. 1 is a block diagram illustrating an exemplary embodiment of the present invention.

FIG. 2 is a flowchart showing a flow of processing according to an embodiment of the present invention.

3 (1), (2), and (3) are diagrams showing an example of an object used in an embodiment of the present invention.

FIG. 4 is a diagram for explaining collation omission in a peripheral area according to the present invention.

FIG. 5 is a diagram illustrating the omission of matching between similar reference images according to the present invention.

FIG. 6 is a diagram illustrating the omission of matching of multiple reference images according to the present invention.

FIG. 7 is a diagram showing experimental specifications of an embodiment example according to the present invention.

FIG. 8 is a diagram showing an input image in the widest angle stage of an exemplary embodiment according to the present invention.

FIG. 9 is a diagram showing accuracy results of an example embodiment according to the present invention.

FIG. 10 is a diagram showing a result of image search speed according to an embodiment of the present invention.

FIG. 11 is a diagram showing a result of an object search speed including a camera control time according to an embodiment of the present invention.

12 (1) and 12 (2) are diagrams showing a state of an object position measurement experiment of a plurality of cameras according to an embodiment of the present invention.

FIG. 13 is a diagram showing how an object is detected by a plurality of cameras according to an embodiment of the present invention.

14 (1), (2), and (3) are diagrams showing an example of line-of-sight information used for object position measurement by a plurality of cameras according to an embodiment of the present invention.

15 (1) and 15 (2) are diagrams showing results of object position measurement by a plurality of cameras according to an embodiment of the present invention.

[Explanation of symbols]

1 ... Object feature amount learning means 2. Pan / tilt / zoom parameter calculation means 3 ... Camera control / image input means 4 ... Input image feature distribution calculating means 5 ... Feature amount distribution matching means 6 ... Collation omitted area calculation means 7 ... Object position calculation means S11 ... Object feature amount learning process S12 ... Pan / tilt / zoom parameter calculation process S13 ... Camera control / image input process S14 ... Input image feature amount distribution calculation process S16 ... Feature value distribution matching process S19 ... Collation omitted region calculation process S20 ... Object position calculation process

Front page continuation (51) Int.Cl. ⁷ identification code FI theme code (reference) G06T 7/20 100 G06T 7/20 100 H04N 7/18 H04N 7/18 C K (72) Inventor Shigeru Takagi Chiyoda, Tokyo 2-3-1, Otemachi, Okumachi, Nihon Telegraph and Telephone Corporation F-term (reference) 2F065 AA04 AA32 AA34 AA37 AA51 DD06 FF04 FF05 FF61 FF65 FF67 HH14 JJ03 JJ26 LL06 QQ21 QQ29 QQ36 QQ37 QQ39 QQ42 Q0543 FC15 A15A16 A15 FE00 HA05 5L096 AA02 AA06 AA09 BA05 CA04 CA05 DA02 EA35 FA18 FA37 GA40 GA41 GA51 HA08 JA03 JA11 JA18 JA25

Claims (14)

[Claims] 1. A method of detecting an object similar to an object registered in advance and detecting a three-dimensional position of the object using an active camera having a pan / tilt / zoom function and performing position measurement. , An object feature amount learning process of learning an object feature amount from a set of reference images of a large number of objects photographed under different shooting conditions, and learning the object feature amount and the feature amount distribution, and the object feature amount learning process Pan / tilt / zoom parameter calculation process for calculating camera parameters of pan / tilt / zoom necessary for searching the entire field of view of the camera from the size of the feature distribution on the image, and the pan / tilt / zoom parameter Of the camera parameter information obtained in the calculation process, the camera is selected as a camera parameter that has not been searched yet, an image is captured, and the captured input image A camera control / image input process for generating a feature amount image converted into the feature amount obtained in the object feature amount learning process for each pixel above, and a feature amount image obtained in the camera control / image input process. , An input feature amount calculation process of setting a target window corresponding to the size of the object feature amount distribution to be collated on the image, and calculating the feature amount distribution in the target window, and the object feature amount learning process. A feature distribution for detecting an object by calculating the similarity value or the distance value between the obtained object feature distribution and the feature distribution in the window of interest set in the input feature distribution calculation process and determining the presence or absence of the object Matching process, and matching between many other object feature amounts similar to the object feature based on the similarity value or distance value calculated in the feature amount distribution matching process and many attention windows around the attention window. Collation omitting area calculation to calculate the optional area Extent and, when detecting an object in the feature distribution matching process, the object position calculation process of calculating the object position from the detection direction, the object detection / position measuring method characterized by comprising. 2. The object detection / position measurement method according to claim 1, wherein an object feature quantity using a color code by vector quantization of color information is used in the object feature quantity learning process. 3. The object detection / position measurement according to claim 1, wherein a color code by vector quantization limited to colors included in the object is used for the object feature in the object feature learning process. Method. 4. The object detecting / position measuring method according to claim 1, wherein a histogram of object features is used for the object feature amount distribution in the object feature amount learning process. 5. In the feature amount distribution collation process, among the object feature amount distributions learned in the object feature amount learning process, an object feature amount distribution that cannot be treated as having detected an object due to erroneous detection or omission of detection is “ It is characterized in that it is used as a "candidate selection" object feature amount distribution, and in the camera control / image input process, the camera is preferentially pointed to an area evaluated to have an object by the "candidate selection" object feature amount. The object detection / position measurement method according to any one of claims 1 to 4. 6. The evaluation of the “for candidate selection” object feature quantity includes:
The object detection / position measurement method according to claim 5, wherein evaluation based on a similarity value or a distance value based on a composite distribution is used. 7. The upper limit value or the distance value of the similarity value of the local subregions around the local subregion used for the calculation of the similarity value or the distance value from the calculation result of the similarity value or the distance value in the feature amount distribution matching process. 7. The object detection / position measurement method according to claim 1, wherein a lower limit value is calculated, and calculation of a similarity value or a distance value of a region that does not reach a threshold value is omitted. 8. The similarity value or distance value between the object feature amount distributions is calculated in advance in the object feature amount learning process, and the calculation result of the similarity value or distance value and the object feature amount distribution in the feature amount distribution matching process. The upper limit value of the similarity value or the lower limit value of the distance value of the object feature amount similar to the object feature amount used for the calculation of the similarity value or the distance value is calculated from the similar value or the distance value between 8. The object detection / position measurement method according to claim 1, wherein the calculation of the similarity value or the distance value of the object feature amount that is not performed is omitted. 9. In the object feature amount learning process, a combined distribution obtained by combining a plurality of object feature amount distributions into one is calculated in advance, and in the feature amount distribution matching process, the combined distribution and a local partial region on the input image are calculated. And calculating the upper limit value or the lower limit value of the distance value of the object feature amount distribution from the calculation result of the similarity value or the distance value with, and omitting the similarity calculation of the object feature amount that does not reach the threshold value. Item 9. The object detection / position measurement method according to any one of items 1 to 8. 10. The object position calculation process according to claim 1, wherein the object position is calculated based on the direction of the object detected by the camera alone and the size of the known object. The object detection / position measurement method described in 1. 11. The object detection / detection method according to claim 1, wherein, in the object position calculation process, the object position is calculated based on the directions of the objects detected by a plurality of cameras. Position measurement method. 12. A device for detecting an object similar to an object registered in advance and detecting a three-dimensional position of the object using an active camera having a pan / tilt / zoom function and performing position measurement. , An object feature amount learning unit that learns an object feature amount from a reference image set of a large number of objects that have been shot under different shooting conditions, and learns the object feature and the feature amount distribution; and the object feature amount learning unit. Pan / tilt / zoom parameter calculation means for calculating the pan / tilt / zoom camera parameters necessary to search the entire field of view of the camera from the size of the feature distribution on the image, and the pan / tilt / zoom parameter Of the camera parameter information obtained by the calculation means, a camera is selected for a camera parameter that has not yet been searched, an image is captured, and the captured input image is captured. A camera control / image input unit for generating a feature amount image converted into a feature amount obtained by the object feature amount learning unit for each pixel on the image, and a feature amount image obtained by the camera control / image input unit In the input feature distribution calculating means for setting a target window corresponding to the size of the object feature distribution to be collated on the image, and calculating the feature distribution in the target window, the object feature learning means A feature amount for detecting the object by calculating the similarity value or the distance value between the obtained object feature amount distribution and the feature amount distribution in the window of interest set by the input feature amount distribution calculating means, determining the presence or absence of the object Between the distribution matching means and a large number of other object feature quantities similar to the object feature based on the similarity value or distance value calculated by the feature quantity distribution matching means and a large number of attention windows around the attention window. Cross-reference omission area total that calculates the area that can be skipped And means, when detecting an object in the feature distribution verification means, object detection / position measuring device, characterized in that the object position calculating means for calculating an object position from the detection direction, provided. 13. Execution of an object detection / position measurement method, characterized in that a program for causing a computer to execute the procedure in the object detection / position measurement method according to any one of claims 1 to 11 is executed. program. 14. A program for causing a computer to execute the procedure in the object detection / position measurement method according to claim 1, wherein the execution program is a computer-readable recording medium. A recording medium on which an execution program of the object detection / position measurement method is recorded.