JPH08315136A - Recognition device - Google Patents

Abstract

PURPOSE: To obtain an object recognition device which is strong to changes in environment, a shift in the view point of a robot, etc., by using plural feature images and preparing a template image for a feature area of an object to be recognized, performing correlating operation between them and an acquired image and extracting feature candidate areas, and recognizing the object from how much the areas overlap each other. CONSTITUTION: Template images 1a... of featured parts of the object to be recognized are prepared for plural different kind of feature image, the template images 1a... and the acquired image are correlated with each other, and the correlation results are regarded as feature candidate areas, and the target recognition object in the acquired image is extracted according to the arrangement relation of the applied template images. Further, a template image which is different in size and resolution is prepared as a template used for correlation with the acquired image in each image. A background image other than the object is used as a template to extract a part other than the object, and the part wherein the object is detected is narrowed down.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION The present invention relates to a manipulator, etc.
The present invention relates to a recognition device for recognizing an instructed recognition target in a recognition image acquired by a visual device such as a camera, and more particularly to a recognition device for recognizing a recognition target using a plurality of templates.

For example, when a meal transport robot or the like holds a food tray on a table by using a robot manipulator, the robot needs to recognize the tray and measure the position of the tray. Therefore, the vision of the robot is
Accurate recognition of the tray is required.

[0003]

2. Description of the Related Art FIG. 11 is a diagram for explaining a conventional object recognition apparatus. Conventionally, when a robot recognizes an object, the model shape of the object and the image captured by the camera are
The mainstream method has been matching with an outline image (differential image) of an object obtained by using a predetermined window or the like.

Further, the characteristics of the object to be recognized by using the Japanese Patent Application No. 06-225419 "Local Area Image Tracking Device" etc. which the applicant of the present application has shown in FIGS. 11 (a) and 11 (b). When a portion, for example, a recognition target object is a desk, the correlation processing between the acquired image and the reference images of the characteristic portions at the four corners is performed at high speed,
There is a method to recognize the (desk) at high speed.

FIG. 11A shows an example of the configuration of a local area image tracking device disclosed in Japanese Patent Application No. 06-225419 "Local Area Image Tracking Device".
The analog image obtained by imaging in 1. is converted into digital image data by the analog / digital converter (A / D converter) installed in the image input device 2, and the correlation tracking processing shown in FIG. 11 (b) is performed. The fixed reference image data, which is stored in the search image memory 30 in the device 3 and is stored in the reference image memory 31 in advance and is related to the predetermined object to be tracked, is subjected to the correlation processing by the correlation calculation unit 32, and the correlation value is Ask.

At this time, as shown in FIG. 11A, in this local area image tracking device, the tracking process for a plurality of search images is performed, for example, by a plurality of correlation tracking processing devices 3.
By performing distributed processing, the recognition and tracking of a high-speed object can be performed at high speed. In the local region image tracking device shown in FIG. 11B, each correlation tracking processing device 3
A search image memory 30 for storing at least three or more search image data, a reference image memory 31 for storing the reference image data, and a reference image data and a search image from the search image memory 30 and the reference image memory 31. The reference image data and the search image data are selectively provided with address generating means for generating an address for reading out data, a correlation calculator 32 for correlating the reference image data and the search image data. By switching to, the recognition and tracking are performed at high speed.

In addition to the above correlation processing, there is also a method of recognizing an object by obtaining a range image using a range finder or the like and matching it with a three-dimensional shape model of the object to be recognized.

[0008]

In the method using the model shape of the object and the contour image or the method using the range image, it takes time to recognize the object, and the shift of the viewpoint of the robot (change in the view direction). Or, due to the interruption of the contour,
There was a problem that there were many cases of erroneous recognition.

Further, even in the method of obtaining the correlation shown in FIGS. 11 (a) and 11 (b), the time required for the correlation processing is the above-mentioned Japanese Patent Application No. 06-22541.
Although it can be shortened by using the local area image tracking device disclosed in No. 9 “Local area image tracking device”, a change in the environment in which the recognition target object is placed, for example, a change in the projection light on the target object (that is, However, there was a problem that it was vulnerable to a lot of misrecognition.

In view of the above-mentioned conventional drawbacks, it is an object of the present invention to provide a recognition device which is robust against environmental changes and changes in appearance, that is, which is capable of robust object recognition. .

[0011]

1 to 10 are views showing an embodiment of the present invention. The above problems can be solved by a recognition device using a plurality of templates configured as follows.

(1) Each of a plurality of different types of characteristic images in the recognition object recognition space is provided with a plurality of template images a, ... , ~, And a plurality of template images applied by the plurality of feature image local correlation calculation units 40a, ~, which correlate with the plurality of different types of acquired images, and the plurality of template images applied by the feature image local correlation calculation unit 40a, ... Template memory for each characteristic image that stores a, ...
41a, ~, and the feature image local correlation calculation unit 40a, each feature candidate region is a correlation result obtained in ~, and each of the feature image template memory 41a, obtained from ~,
The feature image local correlation calculation unit 40a, comprising a feature candidate area evaluation unit 42 for extracting a target recognition target object in the acquired image from the layout relationship of the plurality of template images applied by Image local correlation calculator 40
a, it is configured to extract a target recognition target object in the acquired image from the feature candidate image area which is the correlation result and the arrangement relationship of the plurality of applied template images a ,. . {Refer to FIGS. 1 to 3} (2) The recognition device 4 according to item (1), wherein the recognition target object is provided to each of a plurality of different types of characteristic images in the recognition space of the recognition target object. As a plurality of template images of the characteristic part, in each of a plurality of different images,
Each of the plurality of different types of characteristic images is provided with each of the plurality of different types of characteristic images, each of which has a plurality of template images aa, ... The plurality of template images aa, ... Is applied to the image, and the region where the correlation with the plurality of template images aa ,. It is configured to include a candidate area determining unit 43a, which is a feature candidate area. {Refer to FIG. 4 to FIG. 6} (3) The recognition device according to the item (2), wherein the feature image local correlation calculation unit is used for a plurality of different types of feature images in the recognition space of the recognition object. 40a, template memory for each feature image that stores result information of 4
1a, ..., from the plurality of template images aa, ..., each of which has a different size and resolution of each template image a, ... The feature candidate regions extracted by the correlation calculation units 40a, ... Are combined and configured to include a feature candidate region evaluation unit 42a for determining feature candidate regions. {Refer to FIG. 7} (4) The recognition device according to item (1), wherein each of a plurality of different types of characteristic images in the recognition space of the recognition target object is different from a part that becomes a feature of the recognition target object. Is provided with the template image d of
At 0d, the template image d of the portion other than the characteristic portion of the recognition target is taken into correlation with the acquired image, the portion other than the recognition target is extracted, and the image range for detecting the recognition target is set. Configure to narrow down. {Refer to FIG. 8 and FIG. 9} (5) The recognition device according to (1), (2), (3), or (4), which is accurate for the acquired image. The template adding / deleting unit 4 sets the region contributing to the recognition of the target object as the template image e for the next recognition of the same target object 4
Configure to have 4. {Refer to Fig. 10}

[0013]

That is, in the recognition apparatus of the present invention, by using a plurality of different types (characteristics) of feature images such as a grayscale image, an edge image, and a color image for one recognition target object, The advantage of the feature image can be utilized, and the disadvantage of the correlation processing by a single feature image can be compensated.

For example, in the correlation processing using the grayscale image based on the density template image a, even if the change in density is the same and a high correlation value is obtained, it is not possible to distinguish objects having different colors. However, in the recognition apparatus according to the present invention, even if a color image is used, since the correlation value can be obtained from the color template image c, the recognition target object can be accurately recognized. {Corresponding to claim 1} In the recognition apparatus of the present invention, the same density template images or color template images a and c are different in size and resolution from each other.
By having ac, it is possible to remove noise in the acquired image and perform strong correlation processing against changes in the environment and changes in the appearance. {Corresponding to claims 2 and 3} By having a template image other than the recognition target, for example, the template image d of the background image, the target is cut out using the background, that is, the part other than the recognition target is cut out. It becomes possible to narrow down the range for extracting and detecting the recognition target object.
(Corresponding to claim 4) Also, by setting the region that contributed to the recognition of the object accurately in the acquired image as the template image e for the next recognition of the same object, the template image of various environments is obtained. Can be obtained, and as a result, a more robust recognition device can be constructed. {Corresponding to claim 5}

[0015]

Embodiments of the present invention will be described in detail below with reference to the drawings. FIG. 1 to FIG. 10 described above are diagrams showing an embodiment of the present invention, and FIG. 1 shows a part of a characteristic object of a recognition object for a plurality of different types (characteristics) of characteristic images. FIG. 2 shows a configuration example in which the template image a, ... Is held and the correlation with the acquired image is obtained.
FIG. 3 shows the basic processing flow of the feature candidate area evaluation unit, and FIG. 4 shows the features of the recognition target object for a plurality of different types (characteristics) of images. FIG. 5 shows a configuration example in which template images aa, ab, ... having different sizes and resolutions of the template image of the portion to be obtained are correlated with the acquired image.
6 shows the relationship between the template images aa, ab, ... having different resolutions and the acquired image, FIG. 6 shows an example of the template images aa, ab, ... having different resolutions, and FIG. 7 shows a plurality of different images shown in FIG. Template images aa having different sizes and resolutions of the template image of the characteristic part of the recognition target with respect to the type (property) characteristic image,
FIG. 8 shows another example of the configuration example in the case of having ab, ..., And a correlation with the acquired image, FIG. 8 shows the configuration example in the case of using the template image d other than the recognition target object, and FIG. FIG. 10 shows the relationship between the template image d other than the recognition target object and the correlation value image. FIG. 10 shows the template image for recognizing the next same target object in the region that contributed to the recognition of the target object accurately in the acquired image. By setting e, a configuration example in the case of acquiring template images of various environments is shown.

In the present invention, each of a plurality of different types of characteristic images in the recognition space of the recognition object is
A plurality of template images a to d of a characteristic portion of the recognition object are provided, and a plurality of local correlations for feature images for correlating the template images a, to the plurality of different types of acquired images. Each of the feature image template memories 41a, for storing the plurality of template images a, ... Applied by the operation unit 40a, ... And the feature image local correlation operation unit 40a ,. Section 40a, each feature candidate area which is the correlation result obtained in, and each of the feature image template memory 41a,
Obtained from the above-mentioned feature image local correlation calculation unit 40a,
A feature candidate area evaluation unit 42 or the like that extracts a target recognition target object in the acquired image based on the layout relationship of the plurality of template images a applied in, and the means necessary for implementing the present invention. Is. Note that the same reference numerals indicate the same object throughout the drawings.

The structure and operation of the recognition apparatus of the present invention will be described below with reference to FIGS. In the following description, for convenience of description, for example, a grayscale image will be mainly described. However, in the present invention, processing performed on the grayscale image, that is, correlation calculation, determination of feature candidates by majority logic, The recognition of the recognition target object based on the layout relationship information of the template images a, ... Is similarly performed for other characteristic images such as edge images and color images.

(1) Embodiment of Claim 1 (Embodiment 1): First, FIG. 1 shows a structural example of a recognition device according to claim 1. In FIG. 1, the gray-scale image local correlation calculation unit 40a and the color image local correlation calculation unit 40c are, for example, Japanese Patent Application No. 06-225419 “Local Area Image Tracking Device” previously filed by the applicant of the present application. The correlation tracking processor in. The gradation image template memory 41a, the edge image template memory 41b, and the color image template memory 41c include an acquired image obtained from the camera 1, for example,
For the grayscale image, edge image, and color image, the grayscale image template image, etc., a, and their arrangement information (in practice, it is necessary to change the search range each time a correlation is taken,
The search position (window position) changes is stored.

In the feature candidate area evaluation unit 42, the area information having a high correlation value obtained from each of the grayscale image local correlation calculation units 40a, ... And the grayscale images from the grayscale image template memory 41a ,. Image template image a,
Based on the arrangement information of ~, the area where the recognition target object exists in the acquired image is evaluated, and the correct position of the recognition area is output.

FIG. 2 shows an example of an image obtained from the camera 1 and grayscale image template images a, ... According to the present invention. In the recognition apparatus of the present invention, as shown in FIGS. 2 (a1) and (a2), a plurality of characteristic parts A, B, ... of the recognition object shown in FIG. Template images a, ..., For example, grayscale template images aA, a
B, the edge template images bA and bB, and the color template images cA and cB are stored in the above-mentioned respective grayscale image template memories 41a ,.

Then, as described above, each template image aA, aB, ...
The target object is recognized based on the arrangement information of each of the template images aA, aB, ... Corresponding to B, ... FIG. 3 shows a basic processing flow in the feature candidate area evaluation section 42 shown in FIG. First, the template image number = A is set, and the grayscale image template image a corresponding to the characteristic region A is set.
A, aB, edge image templates bA, b
B, the color image templates cA, cB extract the most overlapping region in the acquired image. The above extraction processing is performed for all template image numbers B, C ,. The feature candidate area in FIG. 2B indicates the area extracted as described above. {Refer to processing steps 100 to 102 in FIG. 3} Subsequently, it is verified whether or not the correspondence between the extracted regions and the correspondence between the template images aA, aB, ... match. As shown in FIG. 2B, in the template number B (corresponding to the characteristic portion B), when one of the feature candidates having a high correlation value is deviated, for example, by a majority vote, Evaluate the matching,
When the association is "NG", it is assumed that the target object does not exist in the acquired image, and when the association is "OK", the object exists in the acquired image. If there is no object in the acquired image, the camera 1
Is moved, a new acquired image is obtained, and the same recognition process is repeated. {Refer to the processing steps 103, 104, 105 of FIG. 3} (2) Embodiment of claim 2 (embodiment 2): FIG.
The structural example of the recognition apparatus shown in FIG. In FIG. 4, the grayscale image local correlation calculation unit 40a, ..., The grayscale image template memory 41a ,.
However, in the present embodiment, for each characteristic part of the recognition object, for example, as the grayscale image template image, the size and resolution shown in FIG. Different template images aa, ab, a
c, ad, ... Are prepared and stored in advance in the grayscale image template memory 41a ,.

Then, the local correlation calculation unit 40a for grayscale image,
, Etc., the search area of the acquired image and the grayscale template memory 41a, which are the grayscale template images stored in the, etc., but the template images aa, ab, ac,
Correlate with ad, ~.

A plurality of correlation values obtained by the gray image local correlation calculating section 40a, ... And the gray image template memory 41a, ... Stored template images of different sizes and resolutions. aa, ab, ac,
From the placement information of ad, ..., For example, the majority area is used to determine the candidate area in the correlation processing.

FIG. 5 (a) shows an example of template images aa, ab, ac, ad having different sizes and resolutions in the grayscale image template image. FIG. 5 (b) is a diagram showing a region matched with the template images aa, ab, ac, and ad having different sizes and resolutions.

In the candidate area determining units 43a, ... Of the present invention, among the matching areas, the portion where the matching areas are most overlapped is set as the candidate area in the acquired image. After that, for other edge images and color images of the grayscale image, similar correlation processing is performed, feature candidate areas are determined, and sent to the feature candidate area evaluation unit 42, as described with reference to FIGS. The same evaluation process is performed to evaluate whether or not the object is present in the acquired image.

FIG. 6 shows an example of the difference in resolution between the grayscale image template images aa and ab. With normal resolution, all pixels in a certain area can be identified, and it is configured so that precise evaluation can be performed.
With the resolution of every other pixel, rough evaluation can be performed. Template images a having such different resolutions
By using a and ab, it is possible to flexibly respond to changes in the environment and appearance of the acquired pixels, and robust recognition processing can be performed.

(3) Embodiment of Claim 3 (Embodiment 3): FIG.
Shows a configuration example of the recognition device shown in claim 3.
In this recognition device, all candidate regions obtained by the local correlation calculation unit 40a for grayscale image, that is, grayscale, edge,
The feature candidate region evaluation unit 42a evaluates the feature candidate regions having high correlation values obtained by the respective local correlation calculation units 40a, ... Corresponding to the acquired images of all colors. That is, a feature candidate is determined by, for example, majority voting from the feature region region that matches each template image region of each feature portion of the recognition target obtained in all the shaded, edge, and color acquired images. That is, the grayscale image template images aa to
The part where the regions matching with ad, ~ overlap is the feature candidate.

Then, the grayscale image template images aa to ad, ..., etc. obtained from the grayscale image template memory 41a, the edge image template memory 41b, ... of the obtained feature candidate area are obtained. The recognition target is recognized using the arrangement information.

This embodiment is essentially the same as the embodiment described with reference to FIG. 4, and in FIG. 4, for example, a partial region having a high correlation value obtained from the grayscale image local calculation unit 40a. In the grayscale image candidate region determination unit 43a, for example, after determining the feature candidate region by majority logic, in the feature candidate region evaluation unit 42, the grayscale image template memory 41a.
The grayscale image template images aa, ...
Although the characteristic portion of the recognition target object is recognized using the arrangement relation information, the present embodiment determines the characteristic candidate area by the majority logic and arranges the grayscale image template images aa, ... Ad. The difference is that the recognition of the object based on the information is performed by one feature candidate area evaluation unit 42a.

(4) Embodiment of Claim 4 (Embodiment 4): FIG.
Shows a configuration example of the recognition device shown in claim 4.
The portion for extracting the characteristic region of the object from each grayscale image, edge image, and color image may have any of the configurations described in the first to third embodiments.

In the present embodiment, the background template image d is prepared in any of the configurations described in the first to third embodiments, and the background template image d is acquired by the local correlation calculation unit 40d.
Color image search area and background template memory 41d
Correlation processing is performed with the background template image d obtained from the above, and the target object candidate area narrowing unit 43 excludes the background area obtained as an area having a high correlation value to obtain the target object candidate area.

The background template image d is shown in FIG. 9 (a).
As shown in, the texture is formed in the background area where the recognition object does not exist. By manipulating the search area of the acquired image and correlating with the background template image d, it is possible to divide into a highly correlated area and a lowly correlated area shown in FIG. 9B.

At this time, there is a high possibility that the object to be recognized is present in the area where the correlation value is low. From this result and the positional relationship in the acquired image plane with the characteristic candidate area of the recognition target object, the characteristic candidate area that is erroneously associated is deleted, and the positional relationship of the remaining feature candidates is used to accurately identify the target object. Can be recognized. In the correlation value image obtained by using the background template image d (see FIG. 9B), if there is a feature candidate in the highly correlated portion, that is, in the background image, it is determined that the correspondence is incorrect. It is possible to recognize the object more accurately. In addition, if there is no region with a low correlation value that allows the target to exist in the correlation image obtained using the background template image d, it is determined that the recognition target does not exist in the acquired image. You can

(5) Embodiment of Claim 5 (Embodiment 5): FIG.
0 is a diagram showing a configuration example corresponding to claim 5. The portion for extracting the characteristic region of the object from each characteristic image, that is, the grayscale image, the edge image, and the color image is the same as in the first to third embodiments.
Any of the configurations of 3 may be adopted.

Template addition and deletion section 4 in this embodiment
In 4, in the acquired image, a portion that contributed to the recognition of the object accurately is added as a template image e,
It is used to recognize the next same recognition target object. In addition, by leaving a history when the used template images a, ~ do not contribute to the recognition of the object, the template images a, ~ that do not contribute to the recognition of the object are deleted over a certain period. To do.

Usually, in the first recognition process, the object is recognized by using as many template images a, ... as possible, and thereafter, it is possible to accurately recognize the object in the acquired image. By grasping the target using the added template image e, it is possible to construct a recognition device capable of efficiently recognizing the target and grasping the recognized target. .

As described above, the recognition apparatus according to the present invention uses, for each of a plurality of different types of characteristic images (for example, a grayscale image, a color image, an edge image, etc.), a template image of a portion that is a characteristic of the recognition target object. a, ..., Correlation of the template images a, ... With the acquired images (shades, colors, edge images, etc.), the correlation result is used as a feature candidate area, and this feature candidate area and a plurality of applied A target recognition target object in the acquired image is extracted from the layout relationship of the template images. In addition, a plurality of template images aa, ab, ... having different sizes and resolutions are prepared as templates used for correlating with the acquired image in each of the images (for example, in the grayscale image). Further, as the template, a template image (background image) d of a part other than the target is used to extract a part other than the recognition target and narrow down a part for detecting the recognition target. Further, it is characterized in that the region of the acquired image that accurately contributes to the recognition of the target object is used as the template image e for the next recognition of the same target object.

[0038]

As described above in detail, according to the recognition apparatus of the present invention, a plurality of characteristic images such as grayscale images and color images are used, and a template image a for a plurality of characteristic regions of the recognition object is used. , ~, And perform a correlation operation between the template images a, ~ and the search range of the acquired image to extract a feature candidate region, the degree of overlap of the regions, the template image a, ~ placement information. Therefore, by recognizing the object,
Strong against shifts in the robot's viewpoint (that is, changes in perspective),
The remarkable effect that a so-called robust object recognition device can be constructed is obtained.

[Brief description of drawings]

FIG. 1 is a diagram showing an embodiment of the present invention (No. 1).

FIG. 2 is a diagram showing an embodiment of the present invention (Part 2).

FIG. 3 is a diagram showing an embodiment of the present invention (part 3).

FIG. 4 is a diagram showing an embodiment of the present invention (No. 4).

FIG. 5 is a diagram showing an embodiment of the present invention (No. 5).

FIG. 6 is a diagram showing an embodiment of the present invention (No. 6).

FIG. 7 is a diagram showing an embodiment of the present invention (No. 7).

FIG. 8 is a diagram showing an embodiment of the present invention (No. 8).

FIG. 9 is a diagram showing an embodiment of the present invention (No. 9).

FIG. 10 is a view showing an embodiment of the present invention (No. 10).

FIG. 11 is a diagram illustrating a conventional object recognition device.

[Explanation of symbols]

1 Image capturing device, camera 2 Image input device 3 Correlation tracking processing device 4 Object recognition device, recognition device 40a, 40b, ~ feature image local correlation calculator 41a, 41b, ~ feature image template memory 42, 42a Feature candidate Area evaluation unit 43a, 43b, ~ Candidate area determination unit 44 Target object candidate area narrowing unit a, b, ~ Feature image template image aa, ab, ~ Template image with different size and resolution d Background template image e Additional template Image correlation value, feature candidate area Template image layout information, layout relationship

Claims (5)

[Claims] 1. A plurality of different types of feature images in a recognition space of a recognition target object are provided with a plurality of template images of portions that are features of the recognition target object, and the template image and the plurality of template images are provided. A plurality of local correlation calculators for feature images that correlate with different types of acquired images; a template memory for each feature image that stores a plurality of template images applied by the local correlation calculator for feature images; Each feature candidate image area that is a correlation result obtained by the feature image local correlation calculation unit, and a plurality of template images obtained from the feature image template memory and applied by the feature image local correlation calculation unit And a feature candidate area evaluation unit that extracts a target recognition target object in the acquired image from the arrangement relationship of A recognition apparatus, which extracts a target recognition target object in the acquired image from the arrangement relationship between the feature candidate image obtained as the correlation result and the plurality of applied template images. 2. The recognition device according to claim 1, wherein a plurality of templates of a portion that is a feature of the recognition target object are provided for each of a plurality of different types of feature images in the recognition space of the recognition target object. In each of the plurality of different images, a plurality of template images having different sizes and resolutions of regions to be correlated are provided in each of the plurality of different images. In the correlation calculation unit, the plurality of template images are applied to each of the acquired images, and the region in which the regions in which the correlations with the plurality of template images match each other are most overlapped with each other is the recognition target. A recognition device, comprising: a candidate area determination unit for defining existing feature candidate areas. 3. The recognition device according to claim 2, wherein result information of the feature image local correlation calculation unit is stored for a plurality of different types of feature images in the recognition space of the recognition object. The candidate images extracted by the above-described feature image local correlation calculation unit are summarized based on the arrangement relationship information of each template image obtained from the feature image template memory and the size and resolution of each template image for the same feature image. And a feature candidate area evaluation unit for determining a feature candidate area. 4. The recognition apparatus according to claim 1, wherein each of a plurality of different types of characteristic images in a recognition space of a recognition target object has a template of a part other than a part that becomes a feature of the recognition target object. An image is provided, and in the feature image local correlation calculation unit, the template image of a portion other than the characteristic portion of the recognition target is correlated with the acquired image, and the portion other than the recognition target is extracted to perform recognition. A recognition device characterized by narrowing an image range for detecting an object. 5. The recognition apparatus according to claim 1, or 2, or 3, or 4, wherein a region that contributes to the recognition of an object accurately in an acquired image is defined by the following same object. A recognition device, comprising: a template addition / deletion unit that serves as a template image for recognition.