JP2002342758A - Visual recognition system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a visual recognition system capable of satisfactorily functioning even under a situation that the distance between a camera and an object to be recognized or the direction of the object arbitrarily fluctuates. SOLUTION: This visual recognition system is provided with a picture fetching means 1, a normalizing means 2 for generating a normalized picture by matching the picture with at least one reference distance, a storage means 6, and a recognizing means 7. In a learning process, an object to be learnt and the distance data are fetched by using the picture fetching means, and the normalized picture of the fetched object to be learnt is generated by the normalizing means, and the normalized picture to which a concept such as designation and meaning is correlated is stored in the storage part. In a recognition process, the object to be recognized and the distance data are fetched by using the picture fetching means, and the normalized picture of the object to be recognized is generated by the normalizing means, and the normalized picture of the object to be recognized is contrasted with the normalized picture of the object to be learnt stored in the storage part by using the recognizing means so that the concept of the object to be recognized can be recognized.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a system for causing a robot or a computer to visually recognize an object.

[0002]

2. Description of the Related Art Conventionally, there has been a system in which a computer visually recognizes an object using a camera or the like. Such a visual recognition system is used for recognizing components on a transport line when it is necessary to perform a process according to the type of a component to be transported one after another on a production line of a factory, for example. The principle of recognition as described above will be briefly described. First, the appearance of a part conveyed to the production line is registered in advance as image information, and a part code is associated with the pre-registered image information.

[0003] The production line is provided with a device such as a CCD camera that captures the appearance of the parts conveyed there, and the conveyed parts photographed by this camera are matched with the previously registered image information. At this matching stage, when the closest component, that is, the closest image information is found, the component code corresponding to the image information is recognized as the transport component.

[0004] As described above, in the case of parts conveyed on a fixed conveying line, the same parts can always be read in the same size. In other words, as long as the position of the camera is specified, the image of the conveyed component and the image of the registered component can always be associated with the same size. As described above, the two can be made to correspond with the same size, so that the type of the component can be recognized.

[0005]

In such a conventional system, the size of a component read at the time of recognition is determined.
If the size of the image of the part registered in advance does not match, the type of the part cannot be accurately recognized. Therefore, there is a problem that this system does not function well in an environment where the distance between the part to be recognized and the camera is not constant.

Further, in the case of a part having a pattern drawn on the surface, the pattern on the surface may not be clearly seen if the part is far away from the part. For example, a black-and-white striped pattern may appear gray at a distance. This is the same even through a camera, and objects far from the camera may have difficulty seeing the surface pattern. In other words, the captured image differs depending on the distance between the target and the camera.

Therefore, if the distance between the image registered in advance and the image captured at the time of recognition is different, it becomes difficult to recognize the object. Furthermore, if the orientation of the part with respect to the camera is different, the part and the registered image will not match. Therefore, even if the components are the same, if the orientation changes, the components cannot be recognized.

In order to solve the above-mentioned problem, a method of reading and registering images at all distances and all directions for each component in advance is conceivable.
However, in such a method, the number of images to be registered in advance is an astronomical number.

That is, in the prior art, it is extremely difficult to accurately recognize a large number of objects in a situation where the distance between the camera and the recognition target and the direction of the target fluctuate arbitrarily. An object of the present invention is to provide a visual recognition system that functions well under such circumstances.

[0010]

According to a first aspect of the present invention, there is provided an image capturing means, a normalizing means for generating a normalized image in which an image is adjusted to one or a plurality of reference distances, a storage means, and a recognizing means. In the learning process, the learning target is captured together with the distance data by the image capturing means, and a normalized image of the learning target captured by the normalizing means is generated. In the recognition process, the recognition target is captured together with the distance data by the image capturing means, a normalized image of the recognition target captured by the normalizing means is generated, and the recognition means performs the recognition. The concept of the recognition target is recognized by comparing the normalized image of the target and the normalized image of the learning target stored in the storage unit. That has a feature to the point.

According to a second aspect of the present invention, there is provided image analysis means for locally developing a feature of an image into a basis function, and feature vector generation means for generating a feature vector by adding coefficients of a local basis function for each basis function. In the learning process, a feature vector is generated from the normalized image to be learned by the feature vector generation unit, and the concept of the learning target is stored in the storage unit in association with the feature vector. In the recognition process, the feature vector is generated. A generating unit generates a feature vector from the normalized image to be recognized, and the recognition unit recognizes the concept of the recognition target by comparing the feature vector of the recognition target with the feature vector stored in the storage unit. It is characterized by points.

According to a third aspect of the present invention, there is provided a discriminant vector generating means for projecting a feature vector into a discriminant space in which a feature of a plurality of feature vectors can be more easily discriminated as a discriminant vector. Generating a discrimination vector of the learning target from the feature vector of the learning target, the storage means stores the discrimination vector of the learning target and the concept of the learning target in association with each other, and in the recognition process, the discrimination vector generation means A discrimination vector is generated from the feature vector of the target, and the recognition unit compares the discrimination vector of the recognition target with the discrimination vector stored in the storage unit, thereby recognizing the concept of the recognition target. Have.

According to a fourth aspect of the present invention, there is provided an image capturing means, a storing means, and a recognizing means. In the learning process, the learning object is captured by the image capturing means together with the distance data, and the learning object image is learned. In the recognition process, the concept of the target is stored in the storage unit, and in the recognition process, the image capturing unit captures the target together with the distance data, and specifies the recognition target based on the distance data from the captured image, It is characterized in that the recognition means recognizes the concept of the recognition target by comparing the specified image of the recognition target with the image stored in the storage means.

According to a fifth aspect of the present invention, in the learning process, the image capturing means captures the target together with the distance data, specifies a learning target from the captured image based on the distance data, and specifies the specified learning target image. The feature is that the concept is stored in the storage unit in association with the concept. In addition,
The identification of the recognition target in the fourth and fifth inventions refers to cutting out an image of a target to be recognized from a captured image from a background or deleting an image other than the cut out image.

In a sixth aspect of the present invention, a normalizing means is provided, and in the learning process, a normalized image is generated by normalizing the image to be learned by the normalizing means with one or a plurality of reference distances. In the recognition process, a concept is associated with the image and stored in the storage unit. In the recognition process, the normalization unit generates a normalized image in which the image to be recognized is normalized by one or a plurality of reference distances. The feature is that the normalized image to be recognized is compared with the normalized image stored in the storage means, and the concept of the recognition target is recognized.

According to a seventh aspect of the present invention, there is provided an image analyzing means for locally expanding a feature of an image to a basis function, and a feature vector generating means for generating a feature vector by adding coefficients of a local basis function for each basis function. In the learning process, a feature vector is generated from the learning target image by the feature vector generating unit, and the learning target concept is stored in the storage unit in association with the feature vector. In the recognition process,
A feature vector is generated from the image to be recognized by the feature vector generation unit, and the recognition unit compares the feature vector of the recognition target with the feature vector stored in the storage unit and recognizes the concept of the recognition target. It is characterized by points.

An eighth aspect of the present invention is characterized in that the normalizing means has a function of adjusting the orientation of a recognition target in an image to a reference angle by image processing. A ninth aspect of the present invention is characterized in that the image analysis unit analyzes an image by dividing the image into a plurality of parts, and the feature vector generation unit creates a plurality of partial feature vectors and performs recognition based on the partial feature vectors.

In the learning process, the image reading means, the normalizing means, the image analyzing means, the feature vector generating means, the discriminating vector generating means and the storing means may be the same as the respective means in the recognition process. Alternatively, they may be provided individually. That is, it is also possible to use the learning means and the recognition means as separate systems, and transplant (copy) the contents learned in advance to a plurality of recognition systems.

[0019]

1 to 10 show a first embodiment of the present invention.
An example will be described. In order to recognize an object, a learning process and a recognition process are required. FIG. 1 shows a system for executing both processes at the same time.
As shown in FIG. 1, the visual recognition system of the first embodiment
Image capturing means 1 for capturing an image to be learned or recognized, normalizing means 2 for normalizing the captured image, image analyzing means 3, feature vector generating means 4, discriminating vector generating means 5, storage means 6 and recognition means 7. An input unit 8 for inputting a concept of an object from outside;
And an output unit 9 for outputting a result recognized by the visual recognition system to the outside.

The image capturing means 1 can capture texture image data of an object and distance data to the object. In the first embodiment,
An image capturing unit 1 includes a stereo camera and a data processing unit that processes data input from the camera.
Is configured. Since the image capturing means 1 captures an image using a stereo camera, distance data can be obtained simultaneously with the texture (pattern on the surface of the object). The image capturing means 1 calculates a representative distance from the camera to the object from the distance data.

The representative distance is a distance from the stereo camera to a representative point of an object to be captured. Since the object has a size, a center point or a point closest to the camera is determined in advance as a representative point, and a distance from the stereo camera to the representative point is set as a representative distance. Alternatively, distances from the camera to a plurality of points on the object surface may be obtained, and the average distance between them may be used as the representative distance.

The normalizing means 2 converts the captured image into
It is about to be normalized. The above normalization is actually
This is a process performed when reading images of objects at various distances to adjust those distances to the reference distance or to adjust the inclination. This will be described in more detail with reference to FIGS. . In FIG. 2, the stereo camera 1
The representative distance from 0 to the object 11 is S. In the first embodiment, two reference distances S1 and S2 are set.

First, the object 1 is
Capture the first image. The representative distance S is calculated by the image capturing means 1, and the representative distance S is input to the normalizing means 2 together with the image data. The normalizing means 2 normalizes the input image data with reference distances S1 and S2. That is, the object 11 is more stereo camera 1 than it actually is.
As an image when placed at the reference distance S1, which is a distance close to 0, the actually captured image is enlarged by the ratio of the reference distance S1 / representative distance S, and is enlarged to the first normalized image 11a.
And Further, the image data 11b reduced at the ratio of the reference distance S2 / representative distance S is used as the second normalized image 11b.

This normalization procedure is the same in both the learning system and the recognition system. In the first embodiment, two reference distances S1 and S2 are set to create the first and second normalized images, but the number is not limited to two. As will be described later, the recognition rate of the object can be increased by appropriately setting the reference distance. The set number of the reference distances is set according to the size and number of the objects to be recognized.

The normalizing means 2 also normalizes the orientation of the object to be recognized. When importing images,
If the direction of the object with respect to the camera 10 is different, the same object may look different. Therefore, in order to make the appearance uniform, it is necessary to perform a normalization process for making the orientations uniform.
In the first embodiment, an image captured by image processing such as affine transformation is rotated so that the object surface is orthogonal to the optical axis of the camera 10. That is, a plane parallel to the imaging plane of the camera is set as the reference plane in the first embodiment.

As a specific processing method, the direction (inclination) of the surface is adjusted while measuring the distances of the plurality of points so that the distances at the plurality of points on the surface of the object are constant. The orientation of the image of the object is adjusted while rotating the image of the object so that the sum of the normal vectors of the surfaces becomes parallel to the optical axis, and a normalized image is generated. For example, an image in which a wall is viewed obliquely is converted into an image in which a wall is viewed directly in front. For an object having a plurality of surfaces such as a cube, a reference surface is determined for each surface. A cylinder like a juice can is transformed into an image of the can viewed from the side. For an object having a more complicated shape, a plurality of reference planes are generally determined based on the above-described criteria. However, rotation around the optical axis of the camera is not performed. For the sake of simplicity, the following description will be given with reference to the flat objects 12 and 13 shown in FIGS. 3A and 3B as examples. First, a method of processing when the object 12 in the captured image is inclined with respect to the imaging surface of the camera 10 will be described.

3A and 3B, the camera 10
A plane parallel to the imaging plane is defined as an xy plane. FIG. 3 (a)
Is a case where the object 12 in the captured image is inclined by an angle α around the x axis with respect to the xy plane. When such an image 12 is captured, the normalization means 2 rotates the object 12 by the angle α in the direction of the arrow to generate a normalized image 12a. FIG. 3B shows a case where the object 13 in the captured image is inclined by an angle β around the y-axis with respect to the xy plane. When such an object 13 is captured, the normalizing means 2 uses the object 1
3 is rotated by an angle β in the direction of the arrow to obtain a normalized image 1
3a is generated.

If the captured image is tilted with respect to both the x-axis and the y-axis, rotation around the x-axis and
By performing both rotations about the y-axis, it is possible to generate a normalized image parallel to the xy plane, which is the imaging plane. As described above, the normalizing means 2 normalizes the distance and the inclination of the object to generate a normalized image. Either the processing for the distance or the processing for the inclination may be performed first.

However, it is not always necessary for the normalizing means 2 to perform both the process of adjusting the distance and the process of adjusting the inclination. Even if only the normalization of the distance is performed, the accuracy of recognition is greatly improved as compared with the related art. When only the processing related to the distance is performed as the normalization processing, the distance to one point on the object may be used as the representative distance. However, in order to adjust the inclination, the distance at a plurality of points is used. Data is needed.
That is, it is necessary for the image capturing means 1 to capture distance data of a plurality of points.

The image analyzing means 3 in FIG. 1 is means for analyzing the normalized image generated as described above. Here, a method of analyzing a normalized image will be described with reference to FIG. FIG. 4 shows a normalized image 14 of the bag. The normalized image 14 is displayed on a display.
Here, for the sake of simplicity, a line drawing is used. The image is analyzed by superimposing the mask pattern on this display while scanning.

The mask pattern is set to extract local features of the normalized image, and is, for example, a pattern as shown in FIG. FIG.
Are shown, but the number of individual mask patterns is not limited to this, and the number of types of mask patterns in one set is not limited to 25. What is necessary is just a set of mask patterns for developing a digital image into basis functions.

The individual mask patterns m1 to m2 in FIG.
5 is a square, each having a different pattern. While moving on the image 14, 25 mask patterns m1 to m25 are overlapped at each position. In particular,
At a certain position, the mask patterns m1 to m25 are sequentially overlapped, and a value indicating which element of the mask pattern m1 to m25 the image located below has and how much is calculated as an element amount of the mask pattern. For example, as for the mask pattern m24, a value obtained by multiplying three pieces of gray value data of an image portion corresponding to three gray squares forming an L-shape is set as an element amount of the mask pattern m24 at that position. In this way, the element amounts of all the mask patterns m1 to m25 are calculated. This process is performed at each position while moving the mask pattern. The movement pitch at this time is usually one pixel. As the pitch width is increased, the recognition accuracy is gradually reduced, but the processing speed is improved.

Although the mask pattern shown in FIG. 5 is shown large for the sake of explanation, this embodiment uses a set of mask patterns of 3 × 3 pixels. However,
The size of the mask pattern can be set arbitrarily according to the type of feature to be taken. For example, when a set of small mask patterns is used, fine (high-frequency) features can be extracted, and when a large mask pattern is used, large (low-frequency) features can be extracted. For example, a smaller mask pattern is used when trying to recognize that the bag is made of crocodile skin than when trying to recognize whether the bag is a bag.

If a plurality of sets of mask patterns having different sizes are stored in the image analysis means 3 and the analysis is performed by using all the sets of mask patterns, a large feature to a small feature can be obtained. You will be able to take on any feature. If the number of features increases, it becomes easier to determine the target. The size of the mask pattern set and the number of mask patterns are determined according to the features and the number of the objects to be recognized. For example, a human evaluates a result of the recognition analysis performed by the image analysis unit 3 using a specific set of mask patterns, and the image analysis unit 3 automatically selects an optimal set according to the evaluation. Approach.

The feature vector generating means 4 in FIG.
The analysis result of the normalized image 11a performed by the image analysis means 3 is totaled for each mask pattern. That is,
The normalized image 11a totals how many features of each mask pattern are provided. The result of the aggregation is shown in FIG.
In this figure, the horizontal axis indicates the type of mask pattern, and the vertical axis indicates the element amount of each pattern. A 25-dimensional feature vector A generated by using each mask pattern as each dimension is generated. In the first embodiment, each mask pattern corresponds to a basis function of the present invention. The feature vector A1 and the features of the normalized image 11a are quantified.

The discriminant vector generation means 5 generates the feature vector A generated by the feature vector generation means 4.
1. Conversion into a discrimination vector. The discrimination vector is a vector obtained by weighting a dimension effective for object determination among the dimensions of the feature vector, setting a new axis as a discrimination axis, and reducing the number of dimensions.

For example, consider a case where it is necessary to distinguish between normalized images of two different objects. First, n images of an object are captured according to subtle changes in the light source, etc.
A is a set of feature vectors of the second object, and B is a set of feature vectors of the second object. The feature vector groups A and B are 25-dimensional vectors generated by the feature vector generating means 4 described above. In order to simplify the explanation of the concept of the discrimination vector, the two are two-dimensional vectors.
It is assumed that the intra-class variance of is sufficiently small. At this time, as shown in FIG. 7, the feature vectors A and B are two-dimensional (x−
y) Represented on a plane. Here, a discriminant axis L for satisfactorily separating the end point groups a and b of the A and B vector groups is calculated by a discriminant analysis method or the like. In this way, an axis (space) that better separates A and B is obtained.

The space defined by the above-mentioned discrimination axis is called a discrimination space. The space for such discrimination can also be constructed by unsupervised data analysis methods such as principal component analysis and kernel principal component analysis, and supervised methods such as kernel discriminant analysis, support vector machines, and neural networks. it can. The discrimination vector generation means 5 stores a calculation formula for converting the feature vector into a discrimination vector in the discrimination space, for example, a conversion matrix.

The storage means 6 in FIG. 1 stores the concept of the object in association with the discrimination vector generated by the discrimination vector means 5 in the learning process. The concept of an object is
Input externally by keyboard, mouse, or voice.
The storage means 6 also stores a group of labeled feature vectors for learning corresponding to the concept of the object. At the time of recognition, the recognition means 7 compares the discrimination vector of the recognition target object with the discrimination vector stored in the storage means 6 to recognize the object. here,
Information on the recognized object is output to the outside via the output unit 9. For example, output to an external control means (not shown),
Other devices can be controlled according to the recognition result.

The procedure for recognizing an object using the visual recognition system shown in FIG. 1 will be described below. First, the learning process will be described with reference to the flowchart of FIG. This learning process is a process that makes the system memorize various objects for the first time. First, in step 1, the image capturing means 1 shown in FIG. 1 captures a texture image of a learning target object and distance data. In this step 1, a plurality of images of a plurality of objects to be memorized are read while changing conditions such as a light source little by little.

In step 2, each of the textures is normalized by the reference distance and the reference angle by the normalizing means 2 to create a normalized image. If a plurality of reference distances for normalization are set, this step is repeated to generate a plurality of normalized images for the image group of each object. In step 3, each normalized image is developed by the image analysis means 3 using a mask pattern. In step 4, the feature vector generation means 4 generates a feature vector based on the image analysis result. The above steps 1 to 4 each perform processing on a plurality of objects.
Steps 1 to 4 may be repeated as many times as the number of learning targets by performing only the processing on one object.

In step 5, the discrimination vector generation means 5
Generates a discrimination space based on the plurality of generated feature vectors by step 4. Then, a transformation matrix, which is a calculation formula for projecting the feature vector onto the discrimination space, is calculated and stored. When a plurality of normalized images are generated for one object in step 2 described above, the discrimination space generated in step 5 is also generated for each reference distance for normalization.
Further, in step 6, the discrimination vector generation means 5
Generates a discrimination vector by multiplying all the feature vectors by the above formula. In step 7, the concept of the object input from the outside is associated with each discrimination vector and stored in the storage means 6. Thus, the initial learning process ends.

Next, an additional learning process for learning a new object will be described with reference to the flowchart of FIG. Steps 11 to 14 are almost the same as steps 1 to 4 in FIG. 8, and thus detailed description is omitted. In short, in steps 11 to 14, a feature vector of the learning target image is generated from the texture image of the learning target object and the distance data captured by the image capturing unit 1.

In step 15, the discrimination vector generation means 5 calls the learned feature vector stored in the storage means 6 and adds the feature vector generated in step 14 to all of these feature vectors. From the vectors, a new discriminant space is identified and a new transformation matrix is generated. In step 16, the above-mentioned transformation matrix is stored in the discriminant space vector generation means 5. This formula is updated every time learning is performed.

In step 17, the discrimination vector generation means 5 converts the feature vector into a discrimination vector by using the generated conversion matrix. In step 18, the concept of the learning target object input from the outside, the discrimination vector generated in step 16, and the feature vector generated in step 14 are stored in the storage unit 6 in association with each other. As described above, steps 11 to 17 are performed each time additional learning is performed.

The feature vectors stored in the storage means 6 in step 7 of FIG. 8 and step 18 of FIG. 9 are as follows.
This is because the learned vector needs to be learned when the discrimination vector generating means 5 generates a new calculation formula. Thus, the learning process ends. In addition, this first
In the embodiment, the image capturing unit 1, the normalizing unit 2, the image analyzing unit 3, the feature vector generating unit 4, the discriminant vector generating unit 5, the storage unit 6, and the input unit 8 in FIG. 1 constitute a learning system. .

Next, the recognition process will be described with reference to the flowchart of FIG. In step 21, the texture to be recognized and the distance data are read by the image capturing means 1. In step 22, the normalizing means 2
Normalizes the texture. In step 2, even if a plurality of reference distances are set, a normalized image at one reference distance is first generated. In step 23, the image analysis unit 3 develops the normalized image. In step 24, the feature vector generation means 4 generates a feature vector.

At step 25, the discriminant vector generation means 5
Generates a discrimination vector. Here, the discrimination vector is generated by converting the feature vector generated in step 24 using the calculation formula generated in step 16 in FIG. In step 26, the recognition means 7
The discrimination vector of the recognition target generated in step 25 is compared with the learned discrimination vector stored in the storage means 6 to calculate the distance between each learned discrimination vector and the discrimination vector of the recognition target. The distance calculated here is the distance between the end points of the respective vectors.

In step 27, the discrimination vector generation means 5 calculates the distance between the discrimination vectors based on the calculation result.
The learned discrimination vector closest to the recognition target discrimination vector is specified. Then, in step 28, it is determined whether or not the distance is equal to or less than a set value. The set value is a reference value that indicates when the difference between the discrimination vector of the recognition target and the discrimination vector of the learning target is the same, that is, how close the difference is. Here, if the distance is equal to or less than the set value, the process proceeds to step 29, where the specified discrimination vector and the distance are stored. On the other hand, if the distance exceeds the set value in step 28, the process proceeds to step 30.

At step 30, it is determined whether or not there is another reference distance, that is, whether or not there is a reference distance that has not been used for normalization among a plurality of set reference distances. If there is another reference distance in step 30, the process proceeds to step 22, where a normalized image is created using the reference distance. Then, the following steps are repeated from step 22. However, step 27
When the learned discrimination vector closest to the recognition target discrimination vector is specified, the closest discrimination vector including the distance to the discrimination vector stored in step 29 is selected. Thus, by step 29, only the discrimination vector closest to the recognition target discrimination vector and whose distance is equal to or less than the set value is selected.

If there is no other reference distance in step 30, the process proceeds to step 31. In step 31, it is determined whether or not the specified discrimination vector exists. If not, the process proceeds to step 33, in which the recognition is not possible, and the result is output (step 34). If there is a specific vector in step 31, the process proceeds to step 32. In step 32, the storage unit 6 outputs the concept associated with the specified discrimination vector as the concept of the recognition target object (step 34). Thus, the recognition process ends.

In the first embodiment, the image capturing means 1 of FIG. 1, the normalizing means 2, the image analyzing means 3, the feature vector generating means 4, the discriminant vector generating means 5, the recognizing means 7, and the output unit 9 are provided. Constitute the recognition system of the present invention. In FIG. 1, the flow of data in the learning system is indicated by solid arrows, and the flow of data in the recognition system is indicated by broken arrows.

According to the system of the first embodiment, since the learning target image is normalized in the learning process, the captured image is also normalized in the recognition process, thereby enabling accurate comparison of the discrimination vectors. Can be. That is, even if the actual position of the object is different from the position of the image read during learning, accurate recognition can be performed. Therefore, it is not necessary to take in images at all distances during learning as in the related art.

When the image to be captured is a color image,
The image analysis means 3 performs color separation of the image, and then masks each image to perform separation. The color separation referred to here is, for example, to separate a color image into three images of blue, green and red, which are three primary colors of light. However, the elements of color separation are not limited to the above three primary colors. For example, it may be decomposed into elements in various colorimetric methods such as hue, saturation, and brightness (luminance). For example, since the saturation is hardly affected by the brightness of the image capturing environment, it is conceivable to utilize such characteristics.

In the first embodiment, the image capturing means 1 may be of any type as long as it can capture image data and distance data. In the first embodiment, the texture image and the distance data can be captured at the same time by using a stereo camera. However, if the distance data is captured using another distance measurement method, the image capture can be performed. Need not use a stereo camera. Methods for measuring the distance to another object include, for example, a method using three or more cameras, a method using a range finder, and a method using an ultrasonic sonar.

In the first embodiment, the process of learning and recognizing a texture image has been described.
If the distance image of the object can be captured by the image capturing means 1, learning and recognition can be performed on the distance image in exactly the same manner as the texture. The distance image expresses a change in unevenness / depth of an object or a scene as a change in color or a change in luminance. The mask pattern used for analyzing the texture can be used as it is. As an example of the distance image, an image 13 in which the distance to an object in the image is represented by shading is shown in FIG.
It is shown in FIG. Here, objects that are closer to the camera are lighter (whiter), and farther away are darker (blacker).

If both the discrimination vector for the texture and the discrimination vector for the distance image are learned, the object can be recognized from either one of the image data at the time of recognition. For example, if the object to be recognized has become old and the surface pattern has become invisible at the time of capturing the image, the feature may not appear in the texture discrimination vector, and recognition may not be possible. As described above, even if the object cannot be recognized from the texture determination vector, the object can be recognized from the distance image determination vector by storing the distance image determination vector during learning. That is, by performing recognition using both discrimination vectors, it is possible to improve the reliability of recognition.

Further, in the first embodiment, normalization is performed for each of a plurality of reference distances in both the learning process and the recognition process.
However, a plurality of reference distances may not be provided, and even if a plurality of reference distances are provided, it is not always necessary to normalize all the reference distances. For example, at the time of learning, normalization is performed with a plurality of reference distances, a discriminant space is created for each of the reference distances, and at the time of recognition, normalization is performed only with a reference distance close to the actual distance, and the reference distance is calculated. Alternatively, only the determination in the determination space may be performed.

On the other hand, in the learning process, an appropriate reference distance is selected from a plurality of reference distances according to the size of the object, normalization is performed only with the reference distance, and a discrimination space for the reference distance is created. At the time of recognition, a method of performing normalization with a plurality of reference distances and searching for the closest discrimination vector may be used. In short, if the distance of the object taken in at the time of learning and the position of the object at the time of recognition are aligned and compared, accurate recognition can be performed.

In the learning process, when an image to be learned is captured, an object is placed at a position where the characteristics of the object are easily seen, unnecessary surrounding objects are removed, and then only the learning target can be captured. If it is impossible to actually move an object other than the learning target, for example, a mountain in the background, the operator can perform processing such as erasing the background from the captured image. That is,
In the learning process, it is possible to intentionally create an environment in which objects placed around the learning target object are excluded in order to correctly remember the learning target.

On the other hand, in the recognition process, there are fewer cases where the object to be recognized is independently placed in a black background. In such a situation, if both are analyzed integrally from an image in which the recognition target and the background are captured at the same time, the recognition target object cannot often be accurately recognized. For example, even if the same object has completely different backgrounds, the objects may be recognized as different objects due to differences in the characteristics of the background. For this purpose, there is a method in which a recognition target object is cut out from a background, and an image of the cut out object is analyzed and recognized.

In the second embodiment described with reference to FIG. 12, the image capturing means 1 of the first embodiment shown in FIG. 1 has a function of deleting a background portion from a captured image and specifying a recognition target. It has been added. The other configuration and the operation of each component are the same as those of the first embodiment. Therefore,
FIG. 1 is used for the description of the second embodiment. Since the learning process in the visual recognition system of the second embodiment is the same as that of the first embodiment, the description is omitted here.
In the following, a recognition process for recognizing an object after necessary learning is completed will be described. However, in the above learning process, it is assumed that the bag to be learned is learned in front of a black wall without a pattern.

On the other hand, in the recognition process, it is assumed that the bag to be recognized is placed on the floor. First, the image of the bag is captured by the image capturing means 1. As shown in FIG. 12, the image 15 captured here also includes the pattern of the floor 15b on the background of the bag 15a. If the analysis is performed with the entire screen 15 as a recognition target, an object in which the bag and the floor are integrated will be analyzed. It is necessary to cut out only the bag from the screen 15.

Therefore, the image capturing means 1 cuts out the object to be recognized based on the distance data captured together with the image data. That is, the image capturing means 1 includes a stereo camera and a data processing unit. The data processing unit determines a difference between the distance between the bag and the floor surface based on the distance data captured by the stereo camera. Can be detected. Floor 1 above
The distance to 5b is farther from the camera than the distance to the bag 15a. Moreover, the difference from the distance to the floor surface 14b at the boundary of the bag 15a is substantially constant. Accordingly, the floor and the bag can be recognized as different objects, and the bag can be cut out from the floor as a recognition target.

In practice, the image of the floor surface portion as the background is excluded and the image is processed to be completely black. If only the target object is cut out in this way, FIG. 12 becomes the same as the image 11a having no background as shown in FIG. The normalizing means 2 normalizes the image from which the background has been removed in this way, and the step of analyzing and recognizing the normalized image is the step after step 22 in FIG. Same as the example. That is, as in the first embodiment, analysis using the mask pattern m, generation of a feature vector, and generation of a discrimination vector are performed, and the recognition unit 7 recognizes the bag.

Note that if the background can be removed from the captured image as in the recognition process of the second embodiment, the object to be recognized may actually exist in any background. Since it is possible to perform accurate recognition without depending on the background pattern or the like, it is not always necessary to perform the normalization processing in the normalization means 2. Even if the normalization process is not performed, the recognition accuracy is significantly higher than in the conventional case where the background and the target object are integrally recognized. However, if a normalized image is created after performing the process of removing the background as in the second embodiment, it is natural that more accurate recognition can be performed.

As described above, the object capturing from the background can be automatically performed by the image capturing means 1. As a result, it is possible to analyze only necessary parts and perform accurate recognition. However, when the depth at the boundary with different objects is smoothly continuous, it is difficult to cut out only the target object from the different objects. In such a case, the operator may manually perform image processing to delete the background.

If the distance of the object to be recognized is known in advance, it can be programmed so that only objects within the range are analyzed as objects to be recognized. By doing so, the object can be cut out efficiently and accurately. Further, the process of removing the background described here can be performed in exactly the same manner even in the learning process. That is, even in the learning process, the learning target object can be automatically specified from the images captured by the image capturing unit 1 based on the difference in distance from the background, and the background can be deleted.

The third embodiment shown in FIGS. 13 and 14 is an example in which correct recognition can be performed even when there is another object in front of the object to be recognized. In the third embodiment, the image capturing means 1 has a function of specifying and cutting out a recognition target object, and the image analyzing means 3 has a function of partially performing image analysis. Others are the same as the above embodiment. Also in the third embodiment,
Since the overall configuration of the system is the same as in the first embodiment,
This will be described with reference to FIG.

FIG. 13 shows a normalized image 16 in the learning process. The normalized image 16 is obtained by deleting the background from the captured image in the same manner as in the second embodiment, and further by the normalizing means 2 in FIG. When such a normalized image 16 is created, the image analysis means 3 performs image analysis. In the third embodiment, the image 16 is divided into a plurality of windows W for processing. Here, the windows W1 to Wn are divided, but these windows W1 to Wn are arranged so as to be slightly shifted from each other in the vertical and horizontal directions. In other words, one window is moved in the horizontal and vertical directions in the figure.

The image analysis means 3 is provided with the window W
The analysis using the mask pattern m is performed every 1 to Wn.
That is, within each of the windows W1 to Wn, the mask patterns m1 to m25 shown in FIG. The element amounts are totaled for each window, and a feature vector is generated for each window. The feature vectors generated for each window are referred to as partial feature vectors a1 to an.

In the third embodiment, n windows W
Are overlapped on the screen 17,
This means that the same part is analyzed a plurality of times to generate a partial feature vector.

Then, the discrimination vector generation means 5 generates partial discrimination vectors c1 to cn for each of the partial feature vectors a1 to an. A discrimination vector C for the entire image is also generated and stored in the storage means 6. When storing in the storage means 6, each of the partial determination vectors c1
To cn are respectively associated with the concept of “bag”. However, as a method of attaching the concept, it is not necessary to attach the same concept to all the partial determination vectors c1 to c12. Each part determination vector may have a different concept, such as "bag part" or "bag clasp". This enables more detailed recognition.

As described above, if the learning method of generating the partial feature vector and storing the partial discrimination vector and the concept in association with each other is adopted, in the recognition process, the recognition target is only partially visible. In such a case, the object can be recognized. For example, as in the case of an image 17 shown in FIG. 14, even when there is something else in front of the bag, the bag can be recognized, and the method will be described below. Note that this image 17 is an image captured from the image capturing means 1 in the recognition process. The bag 17a in this image 17 is
It is the same as the bag of the image 16 of No. 3. That is, in this system, the bag has already been learned. However, in the recognition process, the tape cutter 17b and the book 17c are placed in front of the bag 17a.

Here, the data processing section of the image capturing means 1 converts the bag 17a to be recognized and the tape cutter 17b and book 17c as other objects from the camera from the distance data captured together with the image. It can be separated from the different distances. And here,
The bag 17a is specified as a recognition target, and other portions are deleted together with the background. In order for the image capturing means 1 to leave only the portion of the bag 17a from the captured image 17, it must be set in advance that only objects located near a specific distance are to be recognized. It can also be realized by Alternatively, at the stage where the image 17 has been captured, the operator can designate the portion of the bag 17a, thereby instructing the data processing unit of the image capturing means 1 that it is the target.

As described above, the bag 1 is obtained from the image 17.
When 7a is specified as a recognition target and other parts are deleted,
As shown in FIG. 15, the image 18 includes the target object 18a in which a part of the bag 17a in FIG. 14 is missing. And
Such an image 18 is normalized by the normalizing means 2.
Next, the image analysis means 3 performs image analysis of the normalized image. Here, the image analysis will be described using the image 18 shown in FIG. 15 as a normalized image. This image analysis means 3
Divides the image 18 into a plurality of windows W1 to Wn, as in the learning process. And each window W
In 1 to Wn, the analysis is performed by the mask patterns m1 to m25, and the partial feature vectors d1 to dn are generated by the feature vector generating unit 4 in FIG. Further, the discrimination vector generation means 5 sets the
Generate partial discrimination vectors e1 to en.

The recognizing means 7 includes the partial determination vectors e1 to en and the partial determination vectors c1 to c of the storage means 7.
If n and n are the same or similar vectors, the recognition target is recognized as a “bag”. For example, in FIG. 15, the windows W2 to W4 are portions where images are missing. Therefore, the features of the bag cannot be found from the partial determination vectors e2 to e4 in the windows W2 to W4.

However, the partial determination vectors corresponding to the other windows W include the partial determination vectors c1 to c of the bag.
A vector close to any one of n should be included. Therefore, it is possible to recognize that the recognition target is a “bag” by comparing some partial determination vectors. That is, as in the third embodiment, if the analysis target image is divided into a plurality of windows to generate a partial discrimination vector, the image can be seen even in a situation where the entire object cannot be captured as an image. From the part, the object can be recognized. Even when the whole is visible, if both the partial vector and the whole vector are matched, the recognition can be performed with higher certainty.

As described above, the image capturing means 1 of the third embodiment discriminates between a plurality of objects in the same image 17 as shown in FIG. 14, based on the distance data. be able to. For example, when specifying the tape cutter 17b as the target object, the bag 17a or the book 17c different in distance from the tape cutter 17b can be deleted together with the background, and the same applies to the case where the book 17c is the target object. Therefore, not only the bag 17a but also the tape cutter 17b and the book 17c can be specified as different objects to be recognized. If the tape cutter 17b and the book 17c are at the same distance, the bag 17a is deleted and both the tape cutter 17b and the book 17c remain. Even in such a case, the individual object can be recognized from the partial feature vector and the discrimination vector in the window corresponding to the portion.

FIG. 16 shows a fourth embodiment. In the fourth embodiment, the same learning target is repeatedly read, a plurality of discrimination vectors are generated, and the object is recognized based on these vectors. In the fourth embodiment as well, the overall structure of the system is the same as that of the first embodiment, so that FIG.

As described in the first to third embodiments, the recognizing means 7 shown in FIG. 1 compares the discrimination vector generated from the recognition target image with the discrimination vector generated in the learning process. The closeness of the vector is calculated as the distance between the end points of each vector. Then, when the distance is the shortest and within a certain set value, it is recognized as a learned concept. However, actually, even for the same object, depending on the angle at the time of capturing the image and the brightness,
Exactly the same discrimination vector may not be generated. For this reason, in the fourth embodiment, even during learning, the same object is fetched an image a plurality of times to generate a plurality of discrimination vectors.

In the learning process of the fourth embodiment, the learning process of the first embodiment as described with reference to FIGS. 8 and 9 is repeated for the same object to generate a plurality of discrimination vectors for the same concept. For example, a plurality of discrimination vectors P1, P2,..., Pn for the first object P and a discrimination vector Q for a second object Q different from the first object P
, Qn,..., Qn are generated. In FIG. 15, the end points of the discrimination vectors P1, P2,..., Pn are indicated by a plurality of □ marks, and the end points of the discrimination vectors Q1, Q2,. R1 is a discrimination vector of the object R to be recognized.

Further, in the figure, the point po is the end point of the representative vector Po of the discrimination vectors P1, P2,..., Pn for the object P, and the radius of the circle Pc represented by a broken line centered on this point po. Is the set distance. Similarly, the point qo
Are discrimination vectors Q1, Q2,
..., the end point of the representative vector Qo of Qn, and this point Qo
Is the set distance.
Then, in the recognition process of the fourth embodiment, the discrimination vector of the recognition target object generated by the discrimination vector generation means 5 of FIG. 1 is compared with the representative vector in the discrimination space, and the closest representative vector is obtained. Is within the set distance, the concept associated with the representative vector is set as the concept of the recognition target object.

For example, in the recognition process for recognizing the object R, when the representative vector closest to the generated discrimination vector R1 is the above-mentioned representative vector Po, and when the above-mentioned discrimination vector R1 is within the above-mentioned circle Pc, The recognition means 7 recognizes the object R as P. As in the fourth embodiment, if the same object is repeatedly read and a plurality of discrimination vectors are generated, the characteristics of the learning target object can be grasped more accurately. As a result, the discrimination and recognition of the object can be made less affected by various conditions at the time of capturing the image.

If the characteristics of the object P and the object Q are very similar, or if the characteristics of both are ambiguous, the discrimination vectors P1, P2,...
Q2,..., Qn may be mixed. In such a case, the object can be recognized by the following method.

First, the discrimination vector R1 of the object R to be recognized
, Pn, Q1,.
The distances to Q2,..., Qn are calculated. Then, a predetermined number, for example, k discrimination vectors are specified from the closest distance among all the distances. Then, the concept of the object R is specified based on which of the k discrimination vectors contains the discrimination vector of the object P and the discrimination vector of the object Q. When the k discrimination vectors include more discrimination vectors of the object P, it is determined that “the object R is P”,
If the number of discrimination vectors of the object Q is larger than the number of the k objects, it is determined that “the object R is Q”.

Also, the k discrimination vectors specified as described above may be weighted by their respective distances to determine whether the object is an object P or Q. For example, even if the k pieces include the same number of both the discrimination vector for the object P and the discrimination vector for the object Q, the discrimination vector closer to the discrimination vector R1 is If there are more discrimination vectors, the object R is determined to be Q. It should be noted that such a criterion is not limited to the above method, and various methods can be considered. The above P1 ...
A partial feature vector can be set for each of Pn and Q1... Qn. In this case, the same processing is performed only by increasing the data amount.

[0088]

According to the first to third aspects of the present invention, it is possible to more accurately recognize a recognition target object as compared with the prior art regardless of the distance at the time of capturing an image. it can. According to the second invention, the feature of the image can be abstracted by using a feature vector. Thereby, the robustness regarding the representation of the object can be increased. According to the third aspect, since a space more suitable for discrimination is configured, the recognition rate can be improved.

According to the fourth to sixth aspects, in the recognition process, the recognition target can be specified from the captured image based on the distance data. For example, it is also possible to distinguish the target object from the background, delete the background, and leave only the recognition target. Further, a plurality of objects in the same image can be sequentially cut out based on distance information as a clue, and each object can be recognized.

According to the fifth aspect, in the learning process, the learning target can be automatically specified from the captured image. According to the sixth aspect, at the time of capturing an image, it is possible to satisfactorily recognize the recognition target object irrespective of the distance. Seventh
According to the invention, in addition to the effects of the fourth to sixth inventions, further abstraction can be achieved by converting the feature of the image into a feature vector. Thereby, the robustness of the representation of the object is improved, and the recognition rate can be improved.

According to the eighth aspect, when the image is taken in,
Robustness / adaptability to the difference in orientation between the recognition target objects with respect to the camera is increased, and recognition can be performed better. According to the ninth aspect, since the determination can be made based on the partial vector, the object can be recognized even when the recognition target is partially hidden. In addition, even when a plurality of recognition targets exist in one image, objects can be individually recognized from different partial feature vectors.

[Brief description of the drawings]

FIG. 1 is a diagram showing a configuration of a first embodiment.

FIG. 2 is a diagram illustrating normalization by a reference distance.

FIG. 3 is a diagram illustrating normalization by an angle,
(A) is a figure when rotating about an x-axis, (b) is a figure in the case of rotating about a y-axis.

FIG. 4 is a diagram of a screen for performing image analysis according to the first embodiment.

FIG. 5 is a diagram showing a mask pattern of the first embodiment.

FIG. 6 is a graph showing an element amount of a feature vector according to the first embodiment.

FIG. 7 is a diagram for explaining the principle of a discrimination vector.

FIG. 8 is a flowchart illustrating an initial learning procedure according to the first embodiment.

FIG. 9 is a flowchart illustrating a procedure of additional learning according to the first embodiment.

FIG. 10 is a flowchart illustrating a recognition procedure according to the first embodiment.

FIG. 11 is an example of a distance image.

FIG. 12 is a diagram of a screen for performing image analysis according to the second embodiment.

FIG. 13 is a diagram of a screen in a learning process according to the third embodiment.

FIG. 14 is a diagram of an image captured by an image capturing unit in the recognition process of the third embodiment.

FIG. 15 is a diagram of a screen in a recognition process of the third embodiment.

FIG. 16 is a diagram illustrating a discrimination vector according to a fourth embodiment.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Image taking-in means 2 Normalization means 3 Image analysis means 4 Feature vector generation means 5 Discrimination vector generation means 6 Storage means 7 Recognition means S1, S2 Reference distance L Discrimination axis

 ──────────────────────────────────────────────────続 き Continued on the front page F term (reference) 5B057 AA02 AA05 BA02 CA13 CA16 CD03 DA11 DB03 DC08 DC34 DC36 DC40 5L096 AA09 BA05 CA05 EA03 EA16 EA23 FA02 FA25 FA34 FA59 FA66 GA12 GA19 HA09 JA11 KA04

Claims (9)

[Claims] 1. A learning process, comprising: an image capturing unit; a normalizing unit configured to generate a normalized image in which an image is adjusted to one or more reference distances; a storage unit; and a recognizing unit. Means for capturing the learning target together with the distance data, generating a normalized image of the learning target captured by the normalizing means, storing the normalized image in the storage unit in association with a concept such as a name and a meaning, In the recognition process, the recognition target is captured together with the distance data by the image capturing means, a normalized image of the recognition target captured by the normalization means is generated, and the recognition means stores the normalized image of the recognition target and the storage unit. A visual recognition system that recognizes the concept of the recognition target by comparing the normalized image of the learning target stored in the storage device with the learning target. 2. A learning process, comprising: an image analysis means for locally expanding a feature of an image into a basis function; and a feature vector generation means for adding a coefficient of a local basis function for each basis function to generate a feature vector. Then, a feature vector is generated from the normalized image to be learned by the feature vector generating means, the concept of the learning target is stored in the storage means in association with the feature vector, and the recognition process is performed by the feature vector generating means. 2. The concept of claim 1, wherein a feature vector is generated from the normalized image of the target, and the recognition unit recognizes the concept of the recognition target by comparing the feature vector of the recognition target with the feature vector stored in the storage unit. Described visual recognition system. 3. A discriminant vector generating means for projecting a feature of a plurality of feature vectors to a discriminant space in which discrimination is easier to make a discriminant vector. The discrimination vector of the learning target is generated, and the storage unit stores the discrimination vector of the learning target and the concept of the learning target in association with each other. In the recognition process, the discrimination vector generation unit determines the discrimination vector from the feature vector of the recognition target. The visual recognition system according to claim 2, wherein a vector is generated, and the recognition unit recognizes the concept of the recognition target by comparing the determination vector of the recognition target with the determination vector stored in the storage unit. 4. An image capturing means, a storage means, and a recognizing means. In the learning process, a learning target is captured together with the distance data by the image capturing means, and a concept of the learning target is stored in the image of the learning target object. In the recognition process, the image capturing means captures an object together with the distance data, specifies a recognition target from the captured image based on the distance data, and the recognition means A visual recognition system that recognizes the concept of the recognition target by comparing the specified image of the recognition target with the image stored in the storage unit. 5. In the learning process, the image capturing means captures an object together with the distance data, specifies a learning target from the captured images based on the distance data, and associates the concept with the specified image of the learning target. The visual recognition system according to claim 4, wherein the visual recognition is performed and stored in the storage unit. 6. A learning process comprising a normalization means,
A normalization image is generated by normalizing the image to be learned by the normalization means by one or a plurality of reference distances, and the concept is stored in the storage means in association with the normalized image. Generating a normalized image obtained by normalizing the image to be recognized by one or a plurality of reference distances by the converting means, and recognizing the normalized image to be recognized and the normalized image stored in the storage means. The visual recognition system according to claim 4, wherein the concept of the recognition target is recognized by comparison. 7. A learning process, comprising: image analysis means for locally developing a basis function of an image, and feature vector generation means for generating a feature vector by adding coefficients of a local basis function for each basis function. Then, a feature vector is generated from the image to be learned by the feature vector generating means, the concept of the learning target is stored in the storage means in association with the feature vector, and in the recognition process, the feature vector is generated by the feature vector generating means. A feature vector is generated from an image, and the recognition unit recognizes the concept of the recognition target by comparing the feature vector of the recognition target with the feature vector stored in the storage unit.
A visual recognition system according to any one of the preceding claims. 8. The visual recognition system according to claim 1, wherein the normalizing means has a function of adjusting the orientation of the image to be recognized to a reference angle by image processing. 9. The method according to claim 2, wherein the image analysis means analyzes the image by dividing the image into a plurality of parts, and the feature vector generation means creates a plurality of partial feature vectors.
The visual recognition system according to any one of claims 7 and 8.