JPH1031742A - Image processor and object transfer device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To recognize a three-dimensional position of an object in a short time by an image memory of a small capacity and simple device constitution by generating a distance image of loaded a plurality of objects and extracting a top stage surface area from the distance image. SOLUTION: The image of a recognition object is inputted to a distance image generation means 12 by an image input means 11 and the distance image is generated. Then, by a top stage surface area extraction means 13, the top stage surface area provided with a height equivalent to the object of a top stage is extracted. Since the object of the top stage surface area expressed by binary values looks like a rectangular pattern on the image, by detecting side edge parts crossed at the right angle, a candidate is extracted. An object candidate is positioned by using a template matching method by an object position detection means 17. By coarsely generating the distance image, extracting the candidate of the object from there and executing detailed position detection in the form of extremely limiting the range of pattern retrieval, the distance image generator is made small in the device scale.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus for recognizing a position of a loaded luggage of a robot or an industrial machine for automatically transferring a luggage.

[0002]

2. Description of the Related Art An example of an image processing apparatus for automatically transferring a load is disclosed in Japanese Patent Application Laid-Open No. Hei 6-249631. FIG. 27 is a schematic explanatory diagram of the image measurement processing procedure of the invention, FIG. 28 is a flowchart showing the operation,
FIG. 29 is a diagram showing the principle of spatial coding using pattern light.

This image processing apparatus measures a cardboard box W stacked on a pallet 1 as shown in FIG.
As shown in FIG. 9, the projector 2 comprises a projector 2 disposed diagonally above the pallet 1 and a camera 3 disposed above the center of the pallet 1.
The light projector 2 emits various pattern lights in chronological order, so that the measurement space is overlapped with a wedge-shaped measurement area r0.
To rn. The slit pattern is, for example, A, B, C in FIG.
The black and white binary pattern is projected, and white is a portion that is exposed to light, and black is a portion that is not exposed to light. Such a pattern can be created, for example, using a dot matrix electric shutter such as a liquid crystal shutter.

When a state where light is illuminated is expressed as “1” and a state where light is not illuminated is expressed as “0”, FIG.
When the pattern A is projected, the values are half and 1, 0 from the front to the rear. In pattern B, the values are 1, 0, 1, and 0 in quarters from the front. In pattern C,
It becomes 1, 0, 1, 0, 1, 0, 1, 0 in 1/8 increments.

When an image is captured by projecting these three patterns, each pixel can be made to correspond to any of data encoded into a 3-bit code of “000” to “111”. Such a code is called a space code. A wedge-shaped area in a three-dimensional space corresponds to a certain space code. If the object W exists there, a code is assigned to an image area on the surface of the object. On the other hand, from the image captured by the camera, if the spatial code of a certain pixel on the image is known, it is possible to know which wedge-shaped region corresponds to the height of the object surface, that is, the distance from the camera, based on the principle of triangulation. Will be. In this description, since the pattern types are A, B, and C, the entire image is divided into eight wedge-shaped regions.
If the number of patterns is eight, the number of wedge-shaped regions is 25.
6, the accuracy of the distance information can be improved more precisely. That is, by generating a space code image, the distance of each pixel on the image from the camera can be measured.

FIG. 27 will be described with reference to a flowchart. When the system is started in step ST01, a space code image is generated in step ST02, and a space code image as shown in FIG. 27D is obtained on the upper surface of the cargo in the state of FIG. By horizontally scanning from left to right and extracting an area having data having the largest space code, an image corresponding to the upper surface partial load of the uppermost object can be obtained as shown in FIG. Step ST0
In 4, the uppermost object area is grouped. When the space code is close to a predetermined range between adjacent pixels in the space code image, the regions are determined to be the same group, or the space code of a groove-shaped portion or a gap generated between the upper edges of each cargo. Are grouped on the basis of changes.
The three-dimensional position of each of the grouped objects is measured in the subsequent step ST05. Some three-dimensional coordinates of a portion corresponding to the peripheral portion of the grouped objects are sampled, and in step ST06, the position and orientation of the center of gravity and the height of the object are obtained based on them. Step ST
The process ends at 07. Using the position data of the cargo thus obtained, the cargo can be transferred one by one by a robot or the like.

In this image processing apparatus, it is necessary to obtain all information necessary for cargo recognition from a spatial code image and to generate a high-resolution distance image such as a resolution of 256 × 256 or 512 × 512 pixels. Since a high-precision pattern projector having a mechanism to change the projection pattern in a sequential manner is used, and an image is input in correspondence with each of a plurality of projection patterns generated in a time series,
Requires a large image memory.

[0008] As another conventional example, see IEICE Transactions Vol. J71-D. No. 7p1240-1257
Sato and Iguchi's paper "Liquid crystal range finder-High speed range image measurement system using liquid crystal shutter-
・ Introduce the contents described in "." This technique relates to a range image acquisition system based on the spatial coding method used in the techniques shown in FIGS.
FIG. 30 shows the principle. A slit pattern of an alternating binary code is projected onto an object in time series using the liquid crystal shutter 41, and images corresponding to the respective patterns are captured by the camera 3 to generate a space code image. As the patterns, seven patterns as shown in FIG. 42 are used. The principle of the measurement is the same as that used in the prior art shown in FIGS.

Another conventional technique is disclosed in Japanese Patent Application Laid-Open No. 7-10 / 1995.
"Stereo-compatible search device" described in Japanese Patent No. 3734
Will be described. FIG. 31 is an explanatory diagram of the hierarchical block matching (sparse / dense search method) of the present invention, and FIG. 32 is a flowchart of the operation of the present invention.

[0010] Stereo vision refers to measuring a parallax corresponding to the same point on an object between two images input from image input means arranged at a plurality of different positions, and using a camera based on the principle of triangulation. This is a method for calculating the distance to the target point. In ST11 of the flowchart of FIG. 32, an image is input, the two images input in ST12 are respectively stored in an image memory, and in ST13, processing for converting the stored image to a hierarchically lower resolution is performed. An image is generated. FIG. 31 schematically shows the correspondence between the image of the D-th hierarchy and the image of the D + 1-th hierarchy. In this example, a quarter area of the image on the D-th layer is an image on the D + 1-th layer. In step ST14, the image is divided into small blocks, and in step ST90
In step 5, the initial search position of each block is calculated by the initial search position calculator. In step ST16, the corresponding point of the block in the other image is calculated by pattern matching. In step ST17, the position information of the corresponding point between the two images is calculated. Is used to calculate the parallax by the parallax calculator. Based on the parallax image obtained in this way,
The disparity is calculated again using the one-step high-resolution image.
By repeating this processing, a corresponding point search is finally performed on the image having the same highest resolution as the input image.

[0011]

In the image processing apparatus shown in FIGS. 27 to 29, all the information necessary for recognizing the cargo is obtained from the spatial code image. For example, the resolution is 256 × 256 or 512 ×. It is necessary to generate a high-resolution range image such as 512 pixels, a high-precision pattern projector is required, and it is necessary to have a mechanism for changing the projection pattern in a time-series manner.
It becomes a large-scale expensive device, and furthermore, it is necessary to input an image corresponding to each of a plurality of light emitting patterns generated in time series, a large-capacity image memory is required, and a large-scale expensive device is required. There was a problem. In addition, there is also a problem that the total image input time required for this takes much time.

In the technique shown in FIG. 30 as well, since the cargo is recognized based only on the spatial code image, a large image memory is required as in the case of FIGS. There was a problem.

Further, in the image processing apparatus shown in FIGS. 27 to 29 and 30, in order to obtain all the information necessary for the cargo from the spatial code image, the object to be grasped in the cargo and the adjacent object come into close contact with each other. In other words, in the case of a cardboard box with a small ridge chamfer, the boundary with the adjacent object does not clearly appear as a three-dimensional step, or a bag such as a cement bag containing soft contents. In the case of (3), since the object is in close contact with the adjacent object, the boundary is unclear three-dimensionally, and there is a problem that it is difficult to separate the individual objects and hold them by the transfer means.

The image processing apparatus shown in FIGS. 31 and 32 aims at searching for a stereo correspondence point at high speed and acquiring a distance image in a short time. The search is always performed from the image with the coarser resolution to the image with the denser resolution in small block units that are set uniformly on the image, so if a part that needs high-resolution information is known in advance However, there is a problem that the processing time is long.

SUMMARY OF THE INVENTION The present invention has been made to solve the above problems, and does not require a large-capacity image memory. It is an object to provide a processing device and an object transfer device equipped with the image processing device.

[0016]

According to a first aspect of the present invention, there is provided an image processing apparatus, comprising: a distance image generating means for generating a distance image of a plurality of loaded objects; An uppermost-level surface area extracting means for extracting a surface area; an uppermost-level object candidate extracting means for individually extracting and extracting objects from the uppermost-level surface area; an object size database storing dimension data of a recognition target object; A two-dimensional reference pattern generating means for generating a standard pattern image on a two-dimensional image of the object based on the size data of the object, and a gray-scale original image storage means for storing a gray-scale original image input from the camera. Object position detecting means for detecting the position of the object using the two-dimensional reference pattern and the information of the original gray-scale image for the extracted top-level object candidate, the detection result of the object position obtained by each means and generation of a distance image It is obtained and an information integration unit for integrating the distance information of each object obtained by stage.

According to a second aspect of the present invention, there is provided an object transfer device for generating a distance image of a plurality of stacked objects, and extracting an uppermost surface area of the stacked object from the distance images. A top-level surface area extracting unit, a top-level object candidate extracting unit that individually separates and extracts objects from the top-level surface region, an object size database storing dimension data of the recognition target object, and an object size data. A two-dimensional reference pattern generating means for generating a standard pattern image on a two-dimensional image of the object; a gray-scale original image storing means for storing a gray-scale original image input from a camera; Object position detecting means for detecting the position of the object using the information of the two-dimensional reference pattern and the grayscale original image for the object candidate; the object position detection result obtained by each means and the distance image generating means; And those in which a object transfer means for gripping and transferring an image processing apparatus and the object with an information integration unit for integrating the distance information of each object.

According to a third aspect of the present invention, in the image processing apparatus, the distance image generating means emits a random texture pattern, the first texture input means inputs a stereo image, and the second image input means inputs a stereo image. And a stereo image block associating means for associating the two images captured by the first and second image input means.

According to a fourth aspect of the present invention, there is provided an object transfer device, wherein the distance image generating means projects a random texture pattern, the first image inputting means for inputting a stereo image, and the second image inputting means. An image processing apparatus including two image input means, a stereo image block associating means for associating two images captured by the first and second image input means, and holding and transferring an object Object transfer means.

According to a fifth aspect of the present invention, in the image processing apparatus, the distance image generation means includes a low resolution distance image generation means for generating a coarse resolution distance image, and extracts a step region portion of the object from the low resolution distance image. Step area extracting means, high-resolution distance image generating means for generating a high-resolution distance image for the step area, and a distance image combining two distance images having different resolutions of the low-resolution distance image and the high-resolution distance image And a combining means.

According to a sixth aspect of the present invention, in the object transfer apparatus, the distance image generating means includes a low-resolution distance image generating means for generating a coarse-resolution distance image; A stepped region extracting means for extracting, a high-resolution distance image generating means for generating a high-resolution distance image for the stepped region, and a distance for combining two distance images having different resolutions of the low-resolution distance image and the high-resolution distance image The image processing apparatus includes an image processing apparatus configured by an image synthesizing unit, and an object transfer unit that grips and transfers an object.

According to a seventh aspect of the present invention, in the image processing apparatus, the block associating means includes a high-speed similarity calculating means for quickly searching for the left and right corresponding image blocks, and a pair of blocks as a search result with high reliability. It comprises high-reliability similarity calculating means for calculating similarity, and corresponding-block reliability determining means for determining that there is no corresponding block if the high-reliability similarity is equal to or less than a certain threshold value.

In the object transfer apparatus according to an eighth aspect of the present invention, the block associating means includes a high-speed similarity calculating means for quickly searching for the left and right corresponding image blocks, and a highly reliable pair of the search result blocks. An image processing apparatus comprising: a high-reliability similarity calculating means for calculating the similarity to the object; and a corresponding block reliability determining means for determining that there is no corresponding block if the high-reliability similarity is equal to or less than a predetermined threshold value; And an object transfer means for holding and transferring the object.

According to a ninth aspect of the present invention, there is provided an image processing apparatus having three or more image input means, wherein the block associating means comprises two of the respective images obtained from the three or more image input means. Two or more types of image pairs from which images have been selected are selected, block association is performed, and a search result of each association is integrated to generate a distance image.

According to a tenth aspect of the present invention, there is provided an object transfer device having three or more image input means, and the block association means is one of two of the images obtained from the three or more image input means. An image processing device that selects two or more types of image pairs in which one image is selected, performs block association, and integrates each association search result to generate a distance image;
Object transfer means for holding and transferring an object.

[0026] An image processing apparatus according to an eleventh aspect of the present invention is a distance image generating means for generating a distance image of a plurality of stacked objects, and extracting an uppermost surface area of an uppermost object from the distance image. A top-level surface area extracting unit, an object candidate combination hypothesis generating unit that enumerates and generates a plurality of combinations of object candidates from the output of the top-level surface region extracting unit, and an object size database that stores size data of the recognition target object. A hypothesis validity verification unit that verifies the validity of the object candidate combination hypothesis generated by the hypothesis generation unit using information included in the two-dimensional reference pattern and the grayscale original image; Recognize based on hypotheses,
An information integrating means for integrating the recognized image and the distance information of each object obtained by the distance image generating means.

According to a twelfth aspect of the present invention, there is provided an object transfer device, comprising: a distance image generating means for generating a distance image of a plurality of stacked objects; An uppermost plane area extracting means for extracting, an object candidate combination hypothesis generating means for generating a plurality of combinations of object candidates as hypotheses from the output of the uppermost plane area extracting means, and an object size database for storing dimension data of the recognition target object A hypothesis validity verification unit that verifies the validity of the object candidate combination hypothesis generated by the hypothesis generation unit using the information included in the two-dimensional reference pattern and the grayscale original image; Recognize based on high hypotheses,
An image processing device that includes information integration means for integrating the recognized image and distance information of each object obtained by the distance image generation means, and outputs three-dimensional position information of each target object; It is provided with transfer means for transferring.

According to a thirteenth aspect of the present invention, there is provided an image processing apparatus, comprising: an object array reference database storing data serving as an object array reference; and a hypothesis validity verifying validity of the object candidate combination hypothesis generated by the hypothesis generation means. An object arrangement judging means for comparing the image recognized by the sex verification means with the reference data to determine whether or not the arrangement is correct, and generating a warning to warn an operator of the judgment result when the recognized image does not match the reference data Means.

An object transfer device according to a fourteenth aspect of the present invention provides an object array reference database storing data serving as an object array reference, and a hypothesis for verifying the validity of the object candidate combination hypothesis generated by the hypothesis generation means. An object arrangement determining unit that compares the image recognized by the validity verification unit with the reference data to determine whether the arrangement is correct, and warns an operator of a determination result when the recognized image does not match the reference data; The image processing apparatus includes an image processing apparatus having a warning generating unit, and a transfer unit that holds and transfers an object.

An image processing apparatus according to a fifteenth aspect of the present invention detects the position of a target object by using an artificial reference pattern automatically generated from an object size database storing data serving as a reference of an object arrangement. And a real image reference pattern cutout unit that cuts out an image area corresponding to the target object at the detection position, and a real image reference pattern storage unit that stores the cut out real image reference pattern. Is configured to operate so as to use the actual image reference pattern.

The object transfer device according to claim 16 of the present invention uses an artificial reference pattern automatically generated from an object size database that stores data serving as a reference of an object arrangement, and determines the position of the target object. Detected and has a real image reference pattern cutout means for cutting out an image area corresponding to the target object at the detection position, and a real image reference pattern storage means for storing the cut out real image reference pattern,
The subsequent recognition operation includes an image processing apparatus configured to operate so as to use the real image reference pattern, and a transfer unit that grips and transfers an object.

According to a seventeenth aspect of the present invention, there is provided an image processing apparatus comprising: a distance image generating means for measuring a distance distribution of a plurality of stacked target objects from an image input means; One distance image storage means, an initial grasping position calculating means for calculating the grasping position of the object based on the information of the distance image stored in the first distance image generating means, After being evacuated to a predetermined position, a second distance image storage unit that stores a distance image of the remaining cargo, a distance image stored in the first distance image storage unit,
A distance image comparing means for comparing the distance image stored in the second distance image storing means, and a gripping position of the evacuated object is determined based on a comparison result of the distance image comparing means; Position correction amount detecting means for detecting a difference from the position as a position correction value.

An object transfer device according to an eighteenth aspect of the present invention stores a distance image generating means for measuring a distance distribution of a plurality of stacked target objects from an image input means, and stores the generated distance image. First distance image storage means, object moving means for gripping and moving an object, initial holding position calculating means for calculating the holding position of the object based on the information of the distance image, and object transferring means Grips the object at the gripping position,
Second distance image storage means for temporarily storing a distance image of the remaining cargo after being compared with a predetermined position, a distance image stored in the first distance image storage means, and a second distance image storage A distance image comparison means for comparing the distance images stored in the means, and a position correction amount for determining a gripping position of the retracted object using the comparison result of the distance images and detecting a difference from the initial gripping position as a position correction value The image processing apparatus includes an image processing apparatus having a detecting unit, and a transfer device for gripping and transferring an object.

[0034]

BEST MODE FOR CARRYING OUT THE INVENTION

Embodiment 1 FIG. Embodiment 1 FIG. FIG. 1 is a block diagram showing the configuration of FIG. 1, FIG. 2 is a flowchart showing the operation flow, and FIG. 3 shows the generated images at each stage. Here, it is assumed that the object to be recognized so that the operation description is easy to understand is assumed to be a polyhedral shape such as a cardboard box, and that the objects are stacked in the same shape and the same size.

In FIG. 1, reference numeral 11 denotes an image input means composed of two cameras arranged at a predetermined interval;
2 is a distance image generating means for generating a distance image from the image captured by the image input means 11, 13 is an uppermost plane area extracting means for extracting the uppermost plane from the distance image, 14 is an uppermost object candidate extracting means, 15 is Density image storage means for storing an original image; 16 an object size database; 17 a two-dimensional reference pattern generation means for automatically generating a reference pattern as a two-dimensional pattern based on information stored in the object size database 16; Reference numeral 18 denotes an object position detecting unit that detects a two-dimensional object position, and 19 denotes an information integrating unit that integrates the object position detected by the object position detecting unit 18 and the distance information of the distance image.

3A shows an original image captured by the image input means 11, FIG. 3B shows an uppermost object area extracted by the uppermost surface area extracting means 13, and FIG. 3C shows a two-dimensional reference pattern generating means. A template of a reference pattern generated using object size data, (d) is a schematic diagram showing one of the candidates extracted from the uppermost object region, and (e) shows a positioning situation by a method using template matching.

In step ST 101, when the apparatus is started, the image to be recognized shown in FIG. In step ST102, a distance image is generated by the distance image generating means 12. Means for generating a distance image include a section coding method or a stereo correspondence search method. In step ST103, the uppermost object region extracting means 13 performs
An uppermost object region having a height corresponding to the uppermost object shown in (b) is extracted. In step ST104, the original image captured by the image input unit 11 is stored in the grayscale image storage unit 15, and in step ST105, the two-dimensional reference pattern generation unit 17 uses the two-dimensional pattern based on the information stored in the object size database 16. Is automatically generated. This reference pattern is called a template, and as shown in FIG. 3C, the template is an outline plate expressing the outline of the object.

In step ST106, since the object in the uppermost object region represented by the binary value looks like a rectangular pattern on the image, candidates are extracted by detecting edges that intersect at right angles. As described above, since the resolution of the range image is coarse, the position of the candidate is inaccurate. Step S
At T107, the object candidate is determined by the object position detecting means 1
The positioning is performed by the template matching method using the two-dimensional reference pattern. However, since the rough position has already been detected at the stage of candidate extraction, matching is sufficient with a process of searching for only its nearest neighbor from the candidate detected position. Therefore, template matching can be executed at high speed. The state of the positioning is shown in FIG. When the object position is detected, the two-dimensional object position information is transmitted to the distance image generation unit 1 by the information integration unit 19.
2 and is output as the final result.

In step ST108, it is determined whether or not all the objects have been detected. If not, the process is executed again from step ST106 in the processing flow, and a series of processing steps ST106 and S106 are performed for another object candidate.
T107 is performed. If all the objects have been detected, the recognition process ends in step ST109. As described above, according to the present invention, a distance image is roughly generated, object candidates are extracted based on the result, and detailed position detection is performed by using template matching, which is a general two-dimensional processing method, in the range of pattern search. By performing the processing in a very limited form, the device scale of the range image generation device can be reduced.

As the distance image generating means, a method such as using a spatial coding method with a coarse resolution, or a method such as a stereoscopic vision or a light cutting method using slit light scanning may be used. Any device can be used as long as the device can generate the distance image of.

Although the example of the positioning by the template matching based on the contour information has been described, a method may be used in which the contour of the object is regarded as a combination of straight lines and the position is aligned for each side as a straight line. Further, normal two-dimensional template matching may be used using the contents of the object size database. Although the data storage format used inside the template matching process is not specifically described, other than the method of storing the data as a two-dimensional image, the position of the edge of the contour portion is stored one-dimensionally inside. A method may be used.

Furthermore, when extracting the candidate for the top object, the method of using the corner portion from the top surface area information obtained from the distance image has been described. Similar effects can be obtained by applying a method of coarsely detecting a position by a binary template matching method using a separately prepared template as a binary distance image using a distance image.

The operation flowchart of FIG. 2 is an example, and another processing flow may be used as long as the input / output relationship of each unit shown in FIG. 1 is appropriate. For example, either the step of generating the distance image or the step of inputting the original gray-scale image may be performed first.

The above description has been made on the assumption that the object to be recognized is a single-type object of a polyhedron, that is, a box-shaped object. Image recognition can be realized with the same configuration without changing the configuration of the present invention. Next, the situation in the case of recognizing a bag-shaped object will be described in comparison with the case of the box-shaped object.

In step ST105 of FIG. 2, a two-dimensional reference pattern is automatically generated by the two-dimensional reference pattern generation means 16. This corresponds to template generation in template matching for positioning an object. In the case of a bag-shaped object, since the shape is unstable unlike a box-shaped object such as a deformed edge, it is not sufficient to take out a rectangular outline as shown in FIG. 3 as the reference pattern. In this case, a partial template representing only the corners of the object by the contour or a straight-line contour partial template representing the sides of the object is automatically generated. In either case, the contents are generated with reference to the contents of the object size database 15 storing the size of the bag-shaped object.

In the object candidate extraction in step ST106, in the case of a bag-shaped object, it is difficult to extract a candidate using information on the edges that intersect at right angles as described above. In this case, the image in which the uppermost object region is expressed is regarded as a binary image, and a binary object template is automatically generated from the information on the bag-like object model stored in the object size database 15, and these are roughly processed. By performing binary template matching, rough candidate extraction becomes possible.

In step ST107, the position of the object is detected. Here, the contour template of the corner portion or the straight edge portion of the object is matched with the edge image obtained from the input image as a partial template. You. The position of the object is the result of the matching
Comprehensively determined and calculated from matching positions of a plurality of partial templates. As described above, a bag-shaped object can be operated to the same degree as recognition of a box-shaped object.

In the first embodiment described above. Then
Although it is assumed that objects of a single type are stacked, dimension data of a plurality of objects is stored in the object dimension database 16, and a plurality of two-dimensional reference patterns are automatically generated to detect an object position. In the means 17, the plurality of reference patterns are all matched, and an operation is performed such that a template having the highest similarity is present, so that a plurality of different types of objects are stacked. However, it can be operated properly.

By incorporating the image processing apparatus having the above-described configuration into the object transfer apparatus, it is possible to eliminate the need for a high-precision pattern projector and a large-capacity image memory, and to operate the object transfer apparatus which can be operated in a short time with a short image input time. Mounting device.

Embodiment 2 Embodiment 2 FIG. FIG. 4 is a block diagram showing the configuration of FIG. 5, and FIG. 5 is a flowchart showing the operation flow. FIG. 6 is an explanatory diagram of random dot pattern irradiation, and FIG. 7 is an explanatory diagram illustrating a method of block association processing from a stereo image. Embodiment 2 Is the first embodiment. The distance image generating means is configured to generate a distance image by associating the distance image with stereo by random texture pattern projection. This point is different, and the other configurations are the same as those of the first embodiment. Is the same as

In FIG. 4, reference numeral 20 denotes a random texture pattern projecting means for projecting a random dot pattern;
Reference numeral 21 denotes an image input unit, which includes a first image input unit 21a and a second image input unit 21b. Reference numeral 22 denotes a stereo image block associating unit. Other uppermost image area extracting means 13, uppermost object candidate extracting means 1
4. Original density image storage means 15, object size database 1
6. The two-dimensional reference pattern generating means 17, the object position detecting means 18, and the information integrating means 19 are the same as those in the first embodiment. This is the same configuration as.

Hereinafter, the second embodiment will be described. Will be described with reference to the flowchart of FIG. Step S
When the power is turned on in T201, the power of the random texture pattern projection unit 20 is turned on, and a random dot pattern is projected on the recognition target. As shown in FIG. 6A, the random texture pattern projecting means projects a random dot pattern on an object to be recognized by the pattern projector 30. The pattern floodlight is installed downward above the object as shown in FIG.
A random dot as shown in (b) is projected on the recognition object. Embodiment 2 FIG. In, the target object to be recognized is a cardboard box, but a bag-like object such as a cement bag can be recognized. Random dots are projected on the upper surface of the object, and in step ST203, a stereo image of a pair of images captured by the first image input unit 21a and the second image input unit 21b is input. This stereo image is captured by two cameras having the same resolution and the same focal length at a predetermined distance from the optical axis.

Next, the random dot pattern will be described. The random dots shown in FIG. Is configured with a resolution of 128 × 128,
Each pixel is a square, and a computer assigns a value of white or black to each pixel in a uniform random manner, and the ratio of the number of white and black pixels is approximately 1: 1. Pattern floodlight 3
0 is provided with a mechanism for adjusting the focus of the projection pattern. When the approximate distance between the projector and the target object is known, the focus is adjusted so as to focus on the object surface based on the data. You. If no distance is known in advance, the camera operates so as to focus on the distance adjusted in advance.

When the input of the stereo image pair is completed, in step ST204, the random texture
FF is performed. Next, in step ST205, the stereo image is processed by the block-corresponding stage 22. FIGS. 7A and 7B are a pair of stereo images obtained by photographing a target object. 7A is called a left image, and FIG. 7B is called a right image. Block matching is performed by dividing the right image into small blocks in a grid as shown in the figure. Now, of the plurality of small blocks generated by division, the block of interest bR is called a block of interest. Actually, the random dot pattern is projected on the left and right images. Here, the pattern is not drawn for the sake of conversion. A position on the left image corresponding to the block of interest bR is searched. A pattern having the same pattern as bR is searched for in a separately set search range. As a search method, a search based on minimizing the accumulated difference absolute value is employed. In this method, the blocks of interest bR are sequentially superimposed in the search range on the left image while being moved, and a position at which the similarity becomes the best is set as a corresponding point. At this time, the cumulative difference absolute value is used as the similarity evaluation. For example, if the block of interest is bR (i,
j), the left image is bL (i, j), and the block size is N ×
Assuming that N, the cumulative difference absolute value S _SAD (dx, dy) at the position (dx, dy) on the left image is represented by (Equation 1). Embodiment 2 Then, N = 16.

[0055]

(Equation 1)

S _SAD obtained by (Equation 1) is minimized (d
By detecting (x, dy), the block bL on the left image corresponding to the target block bR is found.
The stereo method is a method of obtaining the distance from a camera to an object based on the principle of triangulation by using the difference between the blocks obtained in this way as parallax. The series of block association processing described above is repeatedly performed on all the small blocks on the right image, and the corresponding position on the left image is searched for each block to generate a distance image.

Hereinafter, the first embodiment will be described. Is the same as
In step ST206, the uppermost step surface area extracting means 1
3, an area corresponding to the uppermost object plane is extracted from the distance image, and in the subsequent step ST207, the original gray-scale image captured by the first image input unit 21a or the second image input unit 21b is stored. It is stored in the means 15. At this time, of course, the random dot pattern is not projected. In step ST208,
The two-dimensional reference pattern is automatically generated by the two-dimensional reference pattern generation means 17 using the size data of the recognition target object stored in the object size database 16.

In step ST 209, the image from which the uppermost surface area has been extracted is divided into two, that is, the uppermost surface and the other surface, and is binarized.
With this, one of the top-level object candidates is extracted. In step ST210, the two-dimensional reference pattern and the object position detecting means 1 are used for the vicinity of the position of the extracted one object candidate.
The accurate position of the candidate object is measured and detected by the object position detecting means 18 using the gray-scale original image information stored in 5. In step 211, the determined two-dimensional position of the candidate object is integrated with the distance image information by the information integrating means 19, and the processes in steps ST209 to ST211 are repeated until it is determined in step ST212 that all objects have been recognized. If it is determined in step ST212 that all objects have been recognized, finally,
Output as dimension information.

If it is determined that all the objects have been recognized by the processing in steps ST209 to ST211, the processing is terminated in step ST213. The determination as to whether or not all the objects have been recognized is based on the binarized distance image from which the uppermost surface area has been extracted. This is performed by checking whether or not the uppermost surface area exists.

As described above, the second embodiment.
Since a random dot pattern is projected on the object to be recognized, it is possible to obtain distance data at each position on the surface of, for example, a plain cardboard box having no design on the surface. It is possible to obtain a stereo image which is not affected by an indeterminate texture such as a destination label. Also, a simple block matching can be applied to the matching process for stereo correspondence.

The second embodiment. In the above description, a random dot pattern of 128 × 128 pixels was projected to the camera field of view as the resolution of the camera, but a finer or coarser dot pattern may be projected depending on the size of the recognition target. The block size is 16 × 16 pixels, but may be 8 × 8 pixels, 32 × 32 pixels, or the like. Also, the shape of the block need not necessarily be square. Furthermore, the search range at the time of matching is set to a rectangle in FIG. 7, but it can be improved in various ways according to the situation to improve the reliability of the matching and to shorten the processing time. It is.

Further, in the second embodiment. In the above, a black-and-white binary square dot is used as a random dot pattern. However, a dot pattern with shading or a colored pattern may be used. In addition, the shape may be used in combination with marks of various sizes and various shapes. Although the example in which the pattern is projected toward is shown, the distance image can be similarly detected even when the pattern is projected obliquely.

Although an example using a dot pattern generated at a random position as a pattern to be projected has been described, it is considered that a search in the horizontal direction is almost performed when searching for a corresponding point in the block association processing. It goes without saying that the same effect can be obtained by projecting a vertical slit group having a random period or a vertical slit group having a random width as a pattern.

The method of forming the pattern projector has not been described in detail, but the projection position accuracy of random dots is not required to be high, and it is not necessary to change the projection pattern during processing. For example, as shown in the related art, a complicated projection device such as a liquid crystal projector is not required, and a device having a simple configuration such as an ordinary home slide screen is sufficient. This is one of the advantages of the device configuration of the present invention.

Embodiment 2 As described above, the operation has been described assuming a corrugated cardboard box as a recognition target object, that is, a hand-faced object. Also, a distance image can be easily generated by the configuration of the present invention, and the three-dimensional position of the object can be measured by the recognition procedure in the present embodiment. Note that the processing subsequent to the generation of the distance image is the same as that of the first embodiment of the present invention as described above. In other words, a final object position can be detected by having a corner portion and a linear contour portion of the object as a two-dimensional reference pattern and integrating the results of positioning with each recognition target. As described above, the present invention is applicable not only to the case where the shape of the object is a polyhedron, but also to the case where the shape of the object is
Dimension information can be recognized.

Embodiment 3 Embodiment 3 FIG. FIG. 8 is a block diagram showing the configuration of FIG. 8, and FIG. 9 is a flowchart showing the operation flow. FIG. 10 is an explanatory diagram of a situation where a step region portion is extracted from a low-resolution distance image, and high-resolution stereo correspondence is performed only in the step region portion of the high-resolution distance image. In FIG. It has the same function as, and the description is omitted. 31 is an image input means 2
1 is a low-resolution block correspondence unit that generates a low-resolution distance image from a stereo image captured at a low resolution, 32 is a step region extracting unit that extracts a step region with a low resolution, 33
Is a high-resolution block associating means for generating a high-resolution distance image for the step region portion extracted from the low-resolution image, 34 is a low-resolution distance image, and a high-resolution distance image of the step region portion extracted from the distance image. Is a distance image synthesizing means for synthesizing.

In step ST301, when the power is turned on and started, in step ST302, the lamp power of the random texture pattern projection means 20 is turned on, and a random dot pattern is projected on the recognition target. In step ST303, a stereo image of a pair of images is input to the low-resolution block association unit 31 and the high-resolution block association unit 32 by the image input unit 21 including a pair of cameras. When the input of the stereo image is completed, the random texture pattern projection unit 20 is turned off in step ST304. In step ST305, the gray image of the target object is stored in the gray image storage means 15 from the first image input means 21a or the second image input means 21b.

In step ST306, the stereo image is processed by the low resolution block association means 31. FIG. 10A is a diagram schematically showing a low-resolution distance image portion A obtained by the low-resolution block association unit 31. Although a CCD camera having a resolution of 512 × 512 pixels is used as an image input means, a distance image of 32 × 32 pixels is generated as a low-resolution distance image, and a method of block matching is described in Embodiment 1. The method is performed in the same manner as the method shown in FIG.
It is obtained by making it a pixel. In step ST307, the step region extracting means 32 extracts a step region portion B of the object based on the low resolution distance image. FIG. 10B is a schematic diagram of the step region portion B. In step ST308, the high-resolution block associating unit 33 generates a higher-resolution distance image only for the step region portion (the hatched portion of B-A). In this case, high resolution is 1
It is 28 × 128 pixels. This processing is obtained by setting the block size to 8 × 8 in the block matching processing, and detecting the corresponding points while shifting the blocks by 4 pixels. FIG. Show. In FIG. 10C, a hatched portion (in a frame of B) is a step region of the object obtained from the low-resolution distance image,
The gray part (within the frame of C) is a part corresponding to the step region part of the distance image newly obtained as a result of the generation of the high-resolution distance image. In step ST309, the low-resolution distance image and the high-resolution distance image are synthesized by the distance image synthesizing unit 34, and a distance image having a resolution of 128 × 128 as shown in FIG. 10D is obtained.

Step ST310 and subsequent steps are the same as those in the second embodiment. Is processed in the same way as In step 310, the uppermost plane area extracting means 13 extracts the uppermost area corresponding to the uppermost object plane from the distance image, and then proceeds to step ST311.
Then, the two-dimensional reference pattern is automatically generated by the two-dimensional reference pattern generation means 17 using the size data of the recognition target object stored in the object size database 16. In step ST312, the image from which the above-mentioned uppermost surface area is extracted is divided into two, that is, the uppermost surface and the other surfaces, and is binarized. One is extracted. Step ST
In 313, the precise position of the candidate object is measured and detected by the object position detecting means 18 using the two-dimensional reference pattern and the grayscale original image information in the vicinity of the position of the extracted one object candidate. The two-dimensional position of the candidate object determined in this way is integrated with the high-resolution range image information synthesized by the information integration means 19, and finally output as three-dimensional information of the object. Step ST3
In step 14, if it is determined in the steps ST312 and ST313 that all objects have been recognized or not, it is determined in step ST315 that the process ends, and it is determined that there are still unrecognized objects. If so, the process returns to steps ST312 and ST313 and repeats the process of extracting object candidates. The determination as to whether or not all the objects have been recognized is based on the binarized distance image from which the uppermost surface area has been extracted. This is performed by checking whether or not the uppermost surface area exists.

As described above, in the third embodiment. In order to finally obtain a high-resolution range image, a low-resolution range image is obtained in a short time in advance, and high-resolution stereo correspondence processing is performed only in an area corresponding to a step portion of an object. In particular, it is possible to generate a high-resolution range image in a short calculation time, and it is also possible to improve the extraction accuracy of an object candidate and the like.

The low-resolution distance image was used as 32 × 32 pixels, and the high-resolution distance image was used as 128 × 128 pixels.
A different resolution may be set in consideration of the permissible processing time, the size of the target object to be recognized, and the like. The benefits are not lost.

Further, the stereo image at the time of generating the low-resolution distance image and the stereo image at the time of generating the high-resolution distance image are configured as the same image. Block matching may be performed based on a reduced image obtained by reducing the image of or な ど. further,
When a high-resolution distance image is generated, the lenses of the CCD cameras of the first and second image input means 21 are zoomed up,
The step portion of the object in the low-resolution distance image may be captured in more detail.

Further, the random dot pattern projected on the target object is a fixed pattern, but two types of dot pattern projectors having different sizes are prepared, and a large dot pattern is projected when a low-resolution distance image is generated, and a high-resolution dot pattern is projected. When the distance image is generated and modified so as to project a smaller dot pattern, a more accurate distance image can be obtained.

Embodiment 4 Embodiment 4 FIG. FIG. 11 is a block diagram showing the configuration of FIG. 11, and FIG. 12 is a flowchart showing the operation flow. In FIG. It has the same function as, and the description is omitted. 41a is a first image storage means, 41b is a second image storage means, 42 is a high-speed similarity calculation means, 43 is a high reliability similarity calculation means, 44 is a corresponding block reliability determination means,
Reference numeral 5 denotes a distance image storage unit.

When the power is turned on and the apparatus is started in step ST401, the power of the random texture pattern projecting means 20 is turned on in step ST402, and a random dot pattern is projected on the recognition target. In step ST403, the first image input unit 21a and the second image input unit 21b capture a stereo image of the target object, and input the stereo images to the first image storage unit 41a and the second image storage unit 41b, respectively. . The two cameras of the first and second image input means 21a and 21b use CCDs of the same resolution, are equipped with lenses of the same focal length, and are arranged at a predetermined distance from the optical axis. After the image is input, the power of the random texture projecting means 20 is turned off in step ST404, and the random dot pattern projected on the object disappears. In step ST405,
An image on which random dots are not projected is picked up by the first image input means 41a, and the original dark and light image storage means 15 is taken.
Is stored in This image is used later for object position detection.

The first and second image input means 21a,
The first image input unit 21a is called a right camera, and the second image input unit 21b is called a left camera. Accordingly, the image stored in the first image storage unit 41a is called a right image, and the image stored in the second image storage unit 41b is called a left image.

In step ST406, the high-speed similarity calculating means 42 performs high-speed block matching based on the stereo images stored in the first and second image storing means 41a and 41b. The block matching is to associate the left and right images for each block in a stereo vision, in which the unit of association is a small area (block) divided into a lattice shape. For block matching,
Embodiment 1 FIG. It is as described in. At this time, the cumulative difference absolute value is used as the evaluation of the similarity between the two image blocks. For example, if the target block is bR (i, j), the left image is L (i, j), and the block size is N × N,
The cumulative difference absolute value S _SAD (dx, dy) at the position (dx, dy) on the left image is represented by (Equation 2).

[0078]

(Equation 2)

An L (left) image is searched for a certain bR, and
The position (dx, dy) where S _SAD is minimum is detected. Thus, the block bL on the left image corresponding to the block of interest bR is detected. When a pair of corresponding blocks is found, the high-reliability similarity calculating means 43 calculates the high-reliability similarity of the pair by the above (Equation 2) in step ST407. At this time, the left image is not scanned. (Dx, dy) obtained by evaluating (Expression 2) is evaluated for similarity by the following (Expression 3). S _CORR is generally called a normalized cross-correlation function.

[0080]

(Equation 3)

(Equation 3) calculates a more accurate and reliable similarity than (Equation 2). The obtained similarity S _CORR is stored in the distance image storage unit 45 together with the parallax obtained from the above association result, that is, the distance information.

In step ST408, the corresponding block reliability determination means 44 compares the highly reliable similarity with a similarity threshold previously set and stored in the computer. It is determined that the association has been made, and if the similarity is smaller than the threshold value, it is determined that the association has been made erroneously. The determination result is stored in the distance image storage unit 45. Subsequent step ST40
In step 9, when all blocks in the right image are associated with each other using the high-speed similarity and the highly-reliable similarity, and the reliability of the association has not been determined, the processing step ST4 is performed again.
The same processing is repeated from 06.

Step ST410 and thereafter are the same as those in the second embodiment. Is processed in the same way as In step 410, the uppermost plane area extracting means 13 extracts the uppermost area corresponding to the uppermost object plane from the distance image, and then proceeds to step ST411.
Then, the two-dimensional reference pattern is automatically generated by the two-dimensional reference pattern generation means 17 using the size data of the recognition target object stored in the object size database 16. In step ST 412, the image from which the uppermost surface area is extracted is divided into two, that is, the uppermost surface and the other surface, and is binarized. One is extracted. Step ST
In 413, the precise position of the candidate object is measured and detected by the object position detecting means 18 using the two-dimensional reference pattern and the grayscale original image information in the vicinity of the position of the extracted one object candidate. The two-dimensional position of the candidate object determined in this way is integrated with the high-resolution range image information synthesized by the information integration means 19, and finally output as three-dimensional information of the object. Step ST4
In step 14, if it is determined in the steps ST412 and ST413 that all objects have been recognized or not, it is determined in step ST415 that the process ends, and it is determined that there are still unrecognized objects. If so, the process returns to steps ST412 and ST413, and the process of extracting object candidates is repeated.

As described above, in the fourth embodiment. Then, when associating a stereo image in units of blocks, first, an image search is performed using a high-speed similarity evaluation unit, and a highly-reliable similarity is evaluated only once for the obtained corresponding block pair. By precisely calculating the similarity of the corresponding block, time-consuming search processing can be performed at high speed, and finally, whether the corresponding block is correct can be determined with high reliability.

Note that this Embodiment 4. In the above, the accumulated difference absolute value was used as a high-speed similarity calculation formula, and the normalized cross-correlation was used as a highly-reliable similarity calculation formula. The maximum value may be used, or the sum of the squares of the differences may be used as the highly reliable similarity. Embodiment 4 FIG. The essence is obtained by a combination of a similarity calculation means that emphasizes high speed rather than reliability and a similarity calculation means having the opposite characteristic.

Further, in the fourth embodiment. Used the pixel values of the image blocks in the same manner in the high-speed similarity calculation and the high-reliability similarity calculation.
/ 2 or 1/4. In this case, even if the similarity evaluation formulas used in the two types of similarity calculation are the same, the high-speed similarity Since the calculation is performed at high speed, the same effect is obtained.

Further, in the fourth embodiment. Is based on the assumption that all processing is performed by a computer. At the time of high reliability similarity calculation using the hardware, the similarity may be calculated by another method that ensures high reliability.

Embodiment 5 Embodiment 5 FIG. FIG. 13 is a block diagram showing the configuration of FIG. 13, and FIG. 14 is a flowchart showing the operation flow. In FIG. It has the same function as, and the description is omitted. Reference numeral 51 denotes an image input unit, and the first image input unit 5
1a, second image input means 51b, and third image input means 51c. Reference numeral 52 denotes a stereo image block associating unit.

When the power is turned on and the apparatus is started in step ST501, the random texture pattern projection means is turned on in step ST502, and a random dot pattern is projected on the recognition target.
In steps 3 to ST505, the first image input unit 51a, the second image input unit 51b, and the third image input unit 51
By c, three images are sequentially input. The three image input means 51a, 51b, 51c are CCD cameras,
The three cameras have the same resolution and the same optical specifications with lenses of the same focal length.
The cameras are arranged such that the optical axes of the cameras are on the same plane in space and are parallel so that no optical axes intersect. Here, the three cameras are referred to as a left camera, a center camera, and a right camera, respectively, for convenience. The distance between the optical axes is set such that the distance between the left camera and the center camera is equal to the distance between the center camera and the right camera. This camera arrangement is the same as in the second embodiment. ~ 4. In this case, this corresponds to the arrangement of a third camera at the center of the two stereo arrangement cameras.

At the same time as the image input is completed, the images are stored in image storage means, each being different image storage means.
In step ST507, the lamp power of the random texture pattern projector 20 is turned off, and the random dot pattern on the object surface disappears. In step 407, a grayscale image of the object is captured by the central second image input unit 51 b and stored in the grayscale original image storage unit 15.

In step ST508, the stereo image block associating means 52 performs stereo image associating processing. FIG. 15 shows a detailed operation flowchart of this processing flow. Stereo associating processing starts in ST508-1 and first step ST50
In step 8-2, a corresponding point is searched for between the image input by the first image input unit 51a (left camera) and the image input by the second image input unit 51b (center camera). This process is a process of associating two images with blocks, and is performed in the second embodiment.
The processing is performed in the same manner as the processing shown in FIG. Embodiment 5 Then
The captured image of the central camera is divided into small blocks, and a corresponding point is searched for on the left camera for each block. FIG. 16 schematically shows how the two images are associated with blocks. FIG. 16A shows an image showing a cardboard box as a recognition target object. In the case of packed cardboard boxes, a transparent film is often attached to the surface. In such a case, light from the pattern projector 20 or another lighting device in the room, particularly, irregularly reflected light may cause a block to be determined as information loss as shown in FIG. 16B. The information missing portion means that, for example, diffuse reflection does not occur in the tape portion in the center camera 51b, but occurs in the left camera 51a, and an image pattern corresponding to the block is found in the other image. The fact that you can't.
The process in this state is executed by determining whether or not the block is an information missing block in step ST508-3.

In step ST508-4, block association is performed between the second image picked up by the central camera and the third image picked up by the right camera. The image obtained by the central camera is divided into blocks, and a point corresponding to each block is searched on the right camera image. At this time, an information missing block is generated as in the above-described corresponding point search. further,
In step ST508-5, information on the information missing point obtained by the combination of the center camera 51b and the right camera 51c is stored in the left camera 51a and the center camera 51b.
Information missing data is combined. At this time, there is no direct relationship between the position of the information missing block in the left-center pair and the position of the information missing block in the center-right pair. In either pair, since the search is performed on the left or right camera image with reference to the image of the central camera 51b, the information missing block information of each pair is
It is given in block units on the central camera image and can be easily compared.

Now, if it is determined that a block in which information is missing in the left-center pair is determined not to be missing information in the center-right pair, the block is determined not to be missing information and the parallax information is determined. Is calculated using a pair that is not missing information. If both pairs are determined to be missing information, the block is determined to be missing information and no parallax information is stored. The same processing is performed on all the blocks of the central camera, and a step ST50 is performed.
At step 8-6, the stereo association processing ends. That is, left-
As long as there is no information missing block in both the center pair and the center-right pair, it is possible to obtain disparity information, that is, distance information.

As the similarity calculation means used in the similarity point search in the stereo correspondence search, the following cumulative difference absolute value S _SAD (dx, dy) of (Expression 7) is used.

[0095]

(Equation 4)

Next, in step ST509 of FIG.
Using the distance image generated in the stereo block association processing, the uppermost plane area extracting unit 13 extracts the area of the uppermost plane of the object.

[0097] Step ST510 and thereafter are the same as those in the second embodiment. Is processed in the same way as In step ST510, the two-dimensional reference pattern generation unit 1 uses the dimension data of the recognition target object stored in the object dimension database 16.
7, a two-dimensional reference pattern is automatically generated.
In step ST 511, the image from which the uppermost surface area has been extracted is divided into two, that is, the uppermost surface and the other surfaces, and is binarized. One is extracted. In step ST512, regarding the vicinity of the position of the extracted one object candidate, using the two-dimensional reference pattern and the grayscale original image information,
The precise position of the candidate object is measured and detected by the object position detecting means 18. The two-dimensional position of the candidate object determined in this way is integrated with the high-resolution range image information synthesized by the information integration means 19, and finally output as three-dimensional information of the object. In step ST513, by the processing of steps ST511 and ST512,
It is determined whether or not all the objects have been recognized. If it is determined that all the objects have been recognized, the process ends in step ST514,
If it is determined that there is still an unrecognized object, the process returns to steps ST511 and ST512 to repeat the process of extracting an object candidate.

As described above, the fifth embodiment.
Then, two sets of stereo images are generated by using three cameras of the image input means 51, and information missing blocks generated in one set of stereo images are generated by disparity information based on another set of stereo image images. Complementing the information prevents loss of the information.

Note that, in the fifth embodiment. In the above, three image input means are arranged so as to have optical axes parallel to each other at equal intervals on the same plane. , The camera spacing is not equal, or the three optical axes are not necessarily on the same plane, the effect of complementing the missing block can be obtained. .

Embodiment 5 In the above, the accumulated difference absolute value was used as a high-speed similarity calculation formula, and the normalized cross-correlation was used as a highly-reliable similarity calculation expression. Or the sum of the squares of the differences may be used as the highly reliable similarity.

Embodiment 6 FIG. Embodiment 6 FIG. FIG. 17 is a block diagram showing the configuration of FIG. 17, and FIG. 18 is a flowchart showing the operation flow. 17, 11 to 13, 15
17 to 19, and 21 correspond to the first embodiment. It has the same function as, and the description is omitted. In the figure, 64 is an object candidate combination hypothesis generation unit, 65 is a hypothesis validity verification unit,
69 is an object position detecting means.

In step ST601, when the apparatus is started, the images of the target object are input to the distance image generation means 12 by the two cameras of the image input means 11 and stored individually. The two cameras having the same specifications without sharing the optical axis are used. In step ST602, the two stereo images are stereo-matched by the distance image generation means 12, and a distance image is generated from the parallax.

At step ST603, the image input means 11
Is stored in the grayscale image storage means. Sixth Embodiment FIG. 19A shows an original gray-scale image of the object to be recognized. The object is a box-like object such as a cardboard box, and they are stacked in several stages. In step ST604, the distance image is extracted as an area from the image of the upper surface of the uppermost object by the uppermost surface area extracting means 13. FIG. 19B shows the extracted uppermost object region. As shown in the figure, areas other than the top row are also shown.

In step ST605, the reference pattern expressing the two-dimensional image pattern of the object by the two-dimensional reference pattern generation means 17 using the data stored in the object size database 16 storing the size data of the box-shaped object. Is automatically generated. FIG. 19C shows a schematic diagram of the reference pattern.

The following operation is described in the sixth embodiment.
Is the essential part of In step ST607, the object candidate combination hypothesis generating means 64 generates one combination hypothesis of the object candidate from the extracted uppermost surface area and the two-dimensional reference pattern. What is a hypothesis?
This is the same as that shown in FIG. Then, this hypothesis is called “hypothesis 1”. The hypothesis is Figure 19
19B, a reference pattern of a recognition target is processed by a pattern matching technique, for example, a template matching technique, and an uppermost surface area as shown in FIG. Is obtained by estimating the combination of the two-dimensional reference patterns forming. At the same time, the object position
As shown in FIG. 9D, the two-dimensional position of the object is detected. The object position detection is performed by comparing the stored grayscale original image with the two-dimensional reference pattern. Using the information of the contour part of the two-dimensional reference pattern, template matching is performed with the edge image detected from the grayscale image. At this time, since the coarse position of the object is known at the hypothesis generation stage, the pattern search area in the template matching process need only be near the above position.

At step ST608, it is verified whether the above hypothesis is valid. The hypothesis validity verification means 65 generates a hypothesis image as shown in FIG. 19D, and generates a hypothesis image as shown in FIG.
It is calculated whether the uppermost surface region such as can be generated reasonably. That is, the hypothesis image generated by the six object candidates is compared with the top-level surface image, and the area of the difference region is calculated. The larger the area is, the more erroneous the hypothesis is determined. Works. Step ST60
In step 8, one validity index is given to one hypothesis generated in step ST606. The reciprocal of the above area is the validity index.

In step ST609, step ST6
The processes from 06 to ST608 are repeated until all possible hypotheses are generated. In step ST610, all the hypotheses are generated, and the most appropriate hypothesis is selected from them. That is, the validity index for each hypothesis is compared, and the hypothesis having the highest index is selected. FIG.
(E) and (f) are different hypotheses 2 and 3, respectively. As is apparent from the figure, in this case, hypothesis 2 has the highest validity index, and this is selected as the best hypothesis in step ST610.

At step ST611, the information integrating means 1
In step ST612, the two-dimensional position information corresponding to the above-mentioned best hypothesis and the distance image are integrated.

Embodiment 6 Used stereo vision with two cameras as the distance image generation means, but described as stereo vision with three or more cameras, stereo vision with combined projection of characteristic projection patterns such as random dots, or as a conventional technique. A distance image generating means using the space coding method described above may be used. Since the sixth embodiment is characterized in that a hypothesis of a recognition result obtained from a distance image is verified to obtain a plausible result, any method of generating a distance image may be used.

Embodiment 6 of the present invention. Used a method of calculating the validity index based on the difference between the hypothesis image and the actual image as a method of verifying the validity of the hypothesis.For example, the template matching calculated when obtaining the above hypothesis was used. The same effect can be obtained by using an index such that the larger the sum of all the similarities of the similarity of the candidate objects is, the higher the validity is.

Embodiment 6 of the present invention. In the method of detecting the position of the object, the contour information of the object obtained from the reference pattern and the edge image generated from the original gray-scale image are processed by the template matching method. The same effect can be obtained by using the technique.

Further, in the sixth embodiment. In the above, we explained a process flow in which the generation and verification of combination hypotheses of object candidates are performed sequentially.However, a large number of hypotheses are generated and stored at once, and their validity is verified at one time. May be selected.

Embodiment 6 In the above description, all the hypotheses are generated. However, since the hypotheses are obtained as a combination of object candidates, the same effect can be obtained even if the hypothesis is treated as an object combination optimization problem and an approximate solution is obtained.

Embodiment 7 FIG. Embodiment 7 FIG. FIG. 20 is a block diagram showing the configuration of FIG. 20, and FIG. 21 is a flowchart showing the operation flow. 20, 11 to 13, 15
19 to 19 are the sixth embodiment. It has the same function as, and the description is omitted. 71 is an object array reference database, 7
2 is an object arrangement determining means, and 73 is a warning generating means.

Embodiment 7 FIG. In the sixth embodiment. Is configured to generate a warning when the object arrangement is incorrectly arranged. Therefore, FIG.
Steps ST701 to ST711 of the flow of FIG.
8 is the same as Steps 601 to 611.

When the apparatus is started in step ST701, the image of the target object is input to the distance image generating means by the image input means 11 and stored individually. Step S
At T702, the two images are stereo-matched by the distance image generation unit 12, and a distance image is generated from the parallax. This processing does not necessarily need to be simple stereo matching. The stereo vision using a pattern projector as shown in FIG.

At step ST703, the image input means 1
The gray scale original image of the recognition target is input by 1 and stored in the gray scale original image storage means 15. In step ST704, the uppermost surface area extraction unit 13 extracts the uppermost surface area. In step ST705, the two-dimensional reference pattern generation unit 17 automatically generates a reference pattern expressing a two-dimensional image pattern of the object using the object size database 16 storing the size data of the box-shaped object. In step ST706, the object candidate combination hypothesis generation unit 6
In step 4, one combination hypothesis of the object candidate is generated from the extracted uppermost surface area and the reference pattern. Embodiment 6 is a hypothesis. FIG. 19D similar to that described in FIG.
It is as shown in. Embodiment 7 In FIG. 19B, the hypothesis is obtained by processing the reference pattern of the recognition target object by a pattern matching technique, for example, a template matching technique with respect to an image in which the uppermost object area as shown in FIG. Is obtained by estimating a combination of two-dimensional reference patterns forming the uppermost surface area as described above. In step ST707, the two-dimensional position of each of the candidate objects included in the above hypothesis is detected by the object position detecting means 69 as shown in FIG. The object position detection is performed by using the stored grayscale original image,
Using the information of the contour part of the two-dimensional reference pattern, it is detected by template matching with the edge image detected from the grayscale image.

In step ST708, the hypothesis image generated from the plurality of object candidates is compared with the top-level image, and the area of the difference area is calculated. Judge as high. Step 7
In step 09, the processing of steps ST706 to ST708 is repeated until all possible hypotheses are generated.
At 0, all hypotheses are generated and the most relevant hypothesis is selected. That is, the validity index for each hypothesis is compared, and the hypothesis having the highest index is selected.

At step ST711, the information integrating means 1
9 integrates the two-dimensional position information corresponding to the best hypothesis and the distance image. At this point, this embodiment 7. In the image processing apparatus in the above, the arrangement of the objects and the three-dimensional position of each object are known, and are stored inside the apparatus. In step 712, the object arrangement determination means 72 determines whether the stored recognition result matches the correct arrangement data prepared in advance or not. The correct array data is an array pattern determined by the recognized object stored in the object array reference database 71. For example, in the seventh embodiment. In a logistics factory in which such a device is used, the method of stacking the boxes on a pallet called a correct “stacking pattern” is determined in advance for the shape and dimensions of the cardboard box as the object to be recognized.

The object arrangement reference database describes the arrangement of objects to be used as such references. If the result recognized in step ST712 does not match the reference stacking pattern, the process proceeds to step ST713.
In the above, a warning is generated by the warning generating means 73 to the user of the apparatus or the manager of the host system to notify that the object is not correctly loaded. Embodiment 7
, A warning is issued by the sound of a buzzer and the display of characters on a display device connected to the device.
If it is determined in step ST712 that the recognition result is a correct pattern, the final object recognition result is output by the information integration means 19, and for example, data is transmitted to the robot to perform the object gripping operation by the robot. Will be. As the recognition operation, the operation ends in step ST715.

Embodiment 7 FIG. Then, in addition to recognizing the object in this way, by comparing the array pattern with the reference data, it is determined whether or not it is an expected array, and if the above recognition result is different from the reference data, a warning is generated It has the function to do. As a result, there is an advantage that the operator can notice a mistake in his / her loading operation or detect a severe collapse of the cargo during transportation.

A method of verifying stereoscopic vision by two cameras as a range image generating means and a method of verifying the validity of a hypothesis based on a difference between a hypothetical image and an actual image is described below.
As a method of detecting an object position, a template matching method using contour information of an object is used. However, it is needless to say that the same effect can be obtained even if these partial processing means are realized by another method. For example, Embodiment 2 is used as a distance image generation unit. The stereo vision using the pattern described in Section 4 is used as a method for verifying the validity of the hypothesis. Alternatively, a template matching method using a normalized cross-correlation and a grayscale image pattern prepared in advance for each object included in the hypothesis may be adopted.

Also, an example has been described in which a warning is generated when it is determined that the recognition result does not match the object sequence reference database stored in advance, but the content of the warning is determined according to the degree of similarity with the reference data. Alternatively, the device user or the operator may be notified. For example, a message such as "The correctness of the stacking method determined as a result of the automatic recognition is level 8" is displayed on the control display. In addition, as a result of the recognition, the collapse state of the cargo is very remarkable, and if it is judged that it is not appropriate to continue the work as it is, not only will the above warning be issued, but also the higher-level management computer will stop the system. The request may be made, or the apparatus may be directly shut down.

Further, although a character display by a buzzer and a display device is employed as the warning means, a method of generating a synthesized human voice may be used as a means for sending a warning to a human.

Embodiment 8 FIG. Embodiment 8 FIG. FIG. 22 is a block diagram showing the configuration of FIG. 22, and FIG. In FIG. 22, 11 to 15, 1
7 and 19 are the first embodiment. It has the same function as, and the description is omitted. In the figure, reference numeral 81 denotes an artificial reference pattern generating means for generating an artificial two-dimensional reference pattern based on the dimension data of an object, and reference numeral 82 denotes an actual reference pattern for extracting a reference pattern of a real image corresponding to the size of a recognition target object from a grayscale original image. Image reference pattern cutout means, 83 is an actual image reference pattern storage means, and 88 is an object position detection means.

When the apparatus is started in step ST801, the image of the object to be recognized is input to the distance image generating means 12 by the image input means 11. Step ST802
Then, using the input image, the distance image generation unit 12
Generates a distance image. Although any means may be used as the distance image generating means, the present embodiment will be described in the eighth embodiment. Then, it is assumed that stereo vision by two cameras is used. Therefore, the image input means uses two television cameras. These two cameras have the same optical specifications and are arranged so as not to share the optical axis. Normally, they are arranged such that the optical axes of the cameras are almost parallel like human eyes. In step ST803, one of the cameras of the image input unit 11 is used to capture an original gray-scale image and store it in the gray-scale original image storage unit 15. As the original image, a grayscale image captured at the time of generating the distance image may be used as it is.

In step ST804, the uppermost surface area extracting unit 13 extracts the uppermost surface area of the object using the distance image. The region of the extracted top-level object is shown in FIG.
This is the same as the one schematically depicted in FIG. That is, the uppermost area is expressed as a binary image such as “1”, and the other areas are expressed as “0”. In addition, this Embodiment 8. In, the recognition target is a box-like object such as a cardboard box, and the same object piled up on the ground is imaged by image input means installed downward on the ceiling, that is, a television camera. In step ST805, the artificial reference pattern generation means 81 generates an artificial two-dimensional reference pattern based on the dimension data of the box-shaped object to be recognized, which is stored in the object size database 17 in advance.
This is the first embodiment. Is the same as the two-dimensional reference pattern in the description of the eighth embodiment. In particular, we will call it this way to emphasize that it is an artificial, automatically generated pattern. This reference pattern expresses only the contour portion of the object as shown in FIG.

In step ST806, the uppermost object candidate extracting means 14 extracts one object candidate considered to exist at the uppermost stage from the binary image representing the extracted uppermost surface region. Here, since the object looks like a rectangular pattern on the image with respect to the uppermost object region expressed in binary, candidates are extracted by detecting edges that intersect at right angles. The schematic diagram of the extracted candidates is shown in the first embodiment. 3D in the description of FIG. In step ST807, the object candidate is positioned by the object position detecting means 14 by using the template matching method. The artificial reference pattern is used as a template. In the template matching, as an image to be searched for a template, an edge image obtained as a result of performing an edge detection process on the original image stored in the density original image storage unit 15 in the object position detection unit 88 is used. . In this template matching, since a rough position has already been detected at the stage of the candidate extraction, a process of searching for only a very near position from the candidate detection position is sufficient for the matching here. The positioning is as shown in FIG.

As a result of the positioning process, a similarity to the template is obtained together with the position information. This is data expressing the similarity to the artificial reference pattern generated in advance. The higher the degree of similarity, the more the detected object resembles the correct box assumed in advance. If the degree of similarity is low, it can be said that there is a high possibility that positioning has failed. Therefore, in step ST808, the obtained similarity data is regarded as the reliability of the object detection, and it is determined whether or not this data is larger than a reliability threshold value stored in the present apparatus in advance. If the reliability of the object obtained as a result of the positioning is smaller than the reliability threshold value, it is determined that an erroneous object has been detected, and the process is repeated from step ST806 to again extract another top-level object candidate and extract the artificial pattern. Perform position detection using. If the obtained reliability is larger than the threshold value, the process proceeds to the next step ST809.

In step ST809, the real image reference pattern extracting means 82 extracts the reference pattern of the real image corresponding to the size of the object to be recognized at the finally detected object position, and stores it in the real image reference pattern storage means 83. Is done. At this time, the actual image means the grayscale original image stored in the grayscale original image storage means 15, and the cutout position is the position of the detected object. The cutout area is rectangular, and the size is obtained by converting the actual size data of the object into a size on the image based on the distance between the camera and the object obtained from the distance image. That is, a rectangular image area as a grayscale image corresponding to one object to be recognized is cut out.

In step ST810, the object position detecting means 88 detects the second and subsequent objects from the image. Here, a method different from that used when the first object is detected is used. A pattern corresponding to one object as a gray-scale real image stored in the real image reference pattern storage unit 83 is used as a template, and a pattern position most similar to the template in the original image stored in the gray-scale original image storage unit 83 is determined. Apply the template matching method of detecting.

In step ST811, it is determined whether or not the detected object corresponds to all the objects requested to be recognized in advance, and if all the objects have been recognized, the information integrating means 19 determines whether the detected object has been detected. The three-dimensional position of each object is calculated by integrating the position and the distance image information, and the result is transmitted to a host computer or a robot, and the process is terminated in step ST812. The process is repeated from step ST810 until object recognition is completed. That is, the first object is recognized using a distance image or the like, but the second and subsequent objects are detected by template matching based on the actual image pattern as a grayscale image obtained at the time of the first recognition.

Embodiment 8 In the above, an example in which general stereo vision is adopted as the range image generating means has been described. Similar effects can be obtained by stereoscopic viewing using random dot pattern projection as described in, or distance image generation by the spatial coding method described in the related art.

Embodiment 9 FIG. Embodiment 9 FIG. FIG. 24 is a block diagram showing the configuration of FIG. 24, and FIG. 25 is a flowchart showing the operation flow. 24, reference numeral 11 denotes an image input unit, 91 denotes a distance image generation unit, 92a denotes a first distance image storage unit, 92b denotes a second distance image storage unit, 93 denotes a distance image comparison unit, and 94 denotes an initial grip position calculation. Means, 95 is a position correction amount calculating means, and 96 is an object moving means.

The shape of the object is assumed to be a polyhedron, for example, a cardboard box, having the same shape and the same size, so that the description can be easily understood. Embodiment 9 FIG. Is
The shape of the recognition target object does not necessarily have to be a polyhedron shape, and even if the target is a bag-like object such as a cement bag, etc. Can be used.

When the apparatus is started in step ST 901, the image of the object to be recognized is input to the distance image generating means 91 by the image input means 11. Step ST90
In step 2, the distance image generating means 9 is used by using the input image.
1, a distance image is generated, and in step ST903, a first distance image is generated and stored in the first distance image storage unit 92a. Although any means may be used for generating the distance image, there are, for example, a space coding method and a method of generating the distance image from stereo vision.

The distance image will be described with reference to FIG. FIG. 26A shows the initial state, that is, step ST90.
The original image captured by the image input means 11 in a state where the box-shaped objects are stacked when Step 1 is executed is schematically represented. FIG. 26B schematically shows a distance image stored in the first distance image storage means 92a. The distance image is a visual representation of the distance from the camera of the image input means to the object, and is expressed in a brighter color as the distance from the camera is shorter, and is expressed in a darker color as the distance from the camera is larger. .

In the next step ST904, the initial grip position is calculated from the distance image by the initial grip position calculating means 94. The initial gripping position is a position where the object is gripped by the object moving means 96 such as a robot in order to move the object. In order to determine the initial gripping position, the first distance image is processed, and a corner portion is recognized from the object region located at the uppermost stage, that is, the region displayed in bright colors in FIG. The position where the offset is applied is set as the initial gripping position. In this case, it is set on the object A. The direction of the arrow in FIG.
The intersection of the arrow and the straight line indicates the grip position. However, here, the gripping position is not determined intentionally to grip the object A, and even if the gripping position obtained by a simple method from the distance image exists on an object other than the object A in this case,
As a result, if any one of the initial gripping positions is set on the object, there is no problem in transferring the cargo.

In step 905, the robot as the object transfer means holds the initial holding position, and the object is temporarily retracted out of the visual field of the image input means. FIG. 26C shows a state after the object is evacuated. The evacuation operation of the object is performed by the robot sucking the gripping position on the object A with the suction hand and moving the hand to a predetermined evacuation position. However, here, the robot retracts by holding the hand still while holding the object A. The predetermined evacuation position is a position where the position where the hand rests the object is clearly understood to be outside the field of view of the image input means 11, and is set as such. Position.

In step ST906, the distance image generating means 91 generates a distance image of the cargo again after the object is evacuated, and in step 907, the second distance image storing means 92.
b. FIG. 26D shows the ninth embodiment. Is a schematic representation of the second distance image. In this figure, an area corresponding to the evacuated object A extends from the uppermost object area.

In step ST908, the distance image comparing means 93 compares the first distance image with the second distance image, and further based on the result, the position correction amount calculating means 95.
Is used to calculate the position correction amount. Embodiment 9 FIG. Then, a simple difference is performed as a comparison of the distance images. FIG. 26E is detected by subtracting the second distance image from the first distance image. FIG. 26 (c) shows the temporarily evacuated object A
Is the only area B corresponding to the difference between the two distance images. There is only one object that is held and evacuated by the robot, and it is natural that before and after the evacuation operation, only a portion corresponding to the evacuated object exists as a difference in the distance image. Since it is known that the region B corresponds to one object as a result of the difference between the distance images, the shape and the position of the center of gravity can be easily calculated. As a method of calculating the center of gravity, shape, and orientation of a region clearly separated from the background on the image, for example, binarizing the difference image, performing labeling for each region, and then calculating the center of gravity for each labeled region. Conventional and generally used methods for calculating the moment maximum direction can be applied.

FIG. 26 (f) is an enlarged view of the retracted object. In addition to the initial gripping position, a newly calculated gripping position obtained from the distance image difference result has been obtained. It is the calculated gripping position. The position correction amount is calculated as the difference between these two gripping positions.

At this point, the robot still holds the initial holding position, but the difference from the correct holding position, that is, the position of the center of gravity of the object has already been calculated. Therefore, by sending the position correction amount data to the robot, in step ST909, the robot knows which position of the retracted object A the hand is holding in the object. Therefore, in the subsequent movement operation by the robot, the robot can perform the movement operation in consideration of the correction amount. This is equivalent to assuming that the robot already knew exactly the center of gravity of the object, that is, the position to be gripped when the object was gripped for the first time in the cargo transfer operation. Thus, the same operation as that of performing the moving operation while grasping is realized.

At step ST910, it is determined whether or not all objects have been recognized.
Go to 1 to complete the process. If the processing has not been completed, the processing is repeated from step ST902 until the processing is completed.

As described above, the ninth embodiment is applied to this embodiment. Based on the initial gripping position recognized from the first distance image, retracts only one object out of the image, and then generates the shape and position information of the object that appears as a difference from the second distance image generated. The accurate grasping position of the currently retreating object is calculated based on the original, and the difference between the initial grasping position and the accurate grasping position is corrected when the robot moves the object to the target position.

In the ninth embodiment. Then, as a method of obtaining the initial gripping position, the gripping position is determined as a position offset from the corner point by a certain offset based on the corner portion of the uppermost object region from the first distance image. The grip position may be determined using the binary template matching method with the upper region and other regions regarded as binary images.

In the ninth embodiment. In the above, the operation of holding the initial holding position with the robot hand and moving the object while holding the object with the hand outside the visual field was performed as the temporary evacuation of the object, but the object may be temporarily lowered to a predetermined place. If the initial gripping position can be gripped when gripping the object again with the hand after the lowering, this embodiment 9. Can be considered as the same operation as.

Further, in the ninth embodiment. In the evacuation operation, the robot hand holding the object is stopped at a predetermined position.
By executing the processing from 906 to ST908 at high speed, the retreat operation can also serve as the operation of continuously moving the hand. In other words, the robot moves out of the image field of view of the object, completes the position correction amount calculation while moving the robot hand to the vicinity of the transfer location as the target of the moving work, and the robot moves the object to a predetermined place for transfer. The same effect can be obtained even if an operation taking into account the position correction amount is performed at this time.

[0149]

According to the first aspect of the present invention, there is provided an image processing apparatus comprising: a distance image generating means for generating a distance image of a plurality of objects; and an uppermost surface region extraction for extracting an uppermost surface region of the object from the distance images. Means, an uppermost object candidate extracting means for individually extracting and extracting an object from the uppermost surface area, and generating a two-dimensional reference pattern on a two-dimensional image of the object based on the size data of the object to be recognized Two-dimensional reference pattern generation means;
Original gray-scale image storage means for storing the gray-scale original image input from the camera, and object position detection for detecting the position of the object using the information of the two-dimensional reference pattern and the gray-scale original image for the individually extracted top-level object candidate Means, and an information integrating means for integrating the detection result of the object position obtained by each means and the distance information of each object obtained by the distance image generating means, by Since the image input means and the distance image generating means only need to be of coarse accuracy, the apparatus may be small and inexpensive, and the light projecting apparatus may be simple, and the reliability is improved.

The object transfer device according to a second aspect of the present invention is a distance image generating means for generating a distance image of a plurality of objects, and an uppermost surface area extracting means for extracting an uppermost surface area of the object from the distance images. And an uppermost-stage object candidate extracting means for individually separating and extracting objects from the uppermost-plane region, and generating a two-dimensional reference pattern on a two-dimensional image of the object based on the size data of the recognition target object. Dimensional reference pattern generation means;
Original gray-scale image storage means for storing the gray-scale original image input from the camera, and object position detection for detecting the position of the object using the information of the two-dimensional reference pattern and the gray-scale original image for the individually extracted top-level object candidate Combining an object processing device with an image processing apparatus comprising means and information integration means for integrating distance detection information of an individual object obtained by a distance image generation means and a detection result of an object position obtained by each means. In this way, the object transfer device needs only coarse and accurate image input means and range image generation means for the object to be recognized. The optical device may be simple, and a highly reliable object transfer device can be obtained.

An image processing apparatus according to a third aspect of the present invention is the image processing apparatus, wherein the distance image generating means projects a random texture pattern, the first image inputting means for inputting a stereo image, and the second image inputting means. And a stereo image block associating means for associating the two images captured by the first and second image input means, so that a random dot pattern is projected on the recognition target object. Because it is illuminated, even if it is a solid object with no pattern on the surface or a label that reflects light on the surface, distance data at each value of the surface can be accurately obtained, and the configuration of the projector is simple and compact Inexpensive ones and good reliability.

According to a fourth aspect of the present invention, in the object transfer device, the distance image generating means projects random texture patterns, the first image inputting means for inputting a stereo image, and the second image inputting means. Combination of an object processing means with an image processing device composed of two image input means and a stereo image block associating means for associating two images captured by the first and second image input means. By doing so, since a random dot pattern is projected on the recognition target object, even if a solid object without a pattern on the surface, or even if a label reflecting light is affixed to the surface, Each position on the surface is obtained accurately, the object detection time as an object transfer device is short, the position of the object is accurately detected, the transfer of the object can be performed quickly and accurately, Configuration of the optical device is also small, it is inexpensive, the reliability is improved.

In the image processing apparatus according to a fifth aspect of the present invention, the distance image generating means extracts a stepped area portion of the object from the low resolution distance image, the low resolution distance image generating means generating a coarse distance image. Step area extracting means, high-resolution distance image generating means for generating a high-resolution distance image for the step area, and a distance image combining two distance images having different resolutions of the low-resolution distance image and the high-resolution distance image Since it is configured with the synthesizing means, there is an effect that a high-resolution distance image can be generated with a small amount of image memory in a short calculation time, and the extraction accuracy of an object candidate can be increased.

According to a sixth aspect of the present invention, in the object transfer device, the distance image generating means includes a low-resolution distance image generating means for generating a coarse-resolution distance image, and a step area portion of the object from the low-resolution distance image. A stepped region extracting means for extracting, a high-resolution distance image generating means for generating a high-resolution distance image for the stepped region, and a distance for combining two distance images having different resolutions of the low-resolution distance image and the high-resolution distance image An image processing apparatus configured with an image synthesizing unit is combined with an object transferring unit. In a short calculation time, a high-resolution range image is obtained, and an object candidate extraction time is short.
The extraction accuracy is high, the position of the object is accurately detected, and the transfer of the object can be performed quickly and accurately.

According to a seventh aspect of the present invention, in the image processing apparatus, the block associating means includes a high-speed similarity calculating means for quickly searching the left and right corresponding image blocks, and a block of the search result block with high reliability. A distance image is generated by a high-reliability similarity calculation means for calculating the similarity, and a corresponding-block reliability determination means for determining that there is no corresponding block if the high-reliability similarity is equal to or less than a certain threshold value. As a result, a high-accuracy distance image is obtained at a high speed, the reliability of image association is increased, and the reliability of the entire apparatus is increased.

According to an eighth aspect of the present invention, in the object transfer apparatus, the block associating means includes a high-speed similarity calculating means for quickly searching the left and right corresponding image blocks, and a highly reliable pair of the search result blocks. An image processing apparatus comprising: a high-reliability similarity calculating means for calculating a similarity, and a corresponding-block reliability determining means for determining that there is no corresponding block if the high-reliability similarity is equal to or less than a predetermined threshold value, Combined with the transfer means, it is possible to obtain high-accuracy distance images at high speed, so the extraction time of object candidates is short, the extraction accuracy is high, the reliability of image association is high, and Is accurately detected, and the transfer of the object can be performed quickly and accurately.

An image processing apparatus according to a ninth aspect of the present invention has three or more image input means, and the block associating means has two of the respective images obtained from the three or more image input means. Since two or more types of image pairs from which images have been selected are selected, block correspondence is performed, and a search result of each correspondence is integrated to generate a distance image. Therefore, even if a missing block of a pair of stereo images occurs. Since complementation is performed by changing the combination of stereo images, loss of information is prevented, and there is an effect that a highly accurate range image can be obtained.

An object transfer device according to a tenth aspect of the present invention has three or more image input means, and the block associating means is two out of each of the images obtained from the three or more image input means. Since two or more types of image pairs in which one image is selected are selected, block association is performed, and an image processing apparatus that generates a distance image by integrating search results of each association is combined with the object transfer means. In addition, the position of the object is accurately detected, and the transfer can be performed accurately.

An image processing apparatus according to an eleventh aspect of the present invention provides a distance image generating means for generating a distance image of a plurality of loaded objects, and extracts an uppermost surface area of the uppermost object from the distance image. A top-level surface region extracting unit, a plurality of object candidate combination hypothesis generating units that enumerate and generate a plurality of combinations of object candidates from the output of the top-level surface region extracting unit, and an object size database storing size data of the recognition target object A hypothesis validity verification unit that verifies the validity of the object candidate combination hypothesis generated by the hypothesis generation unit using the information included in the two-dimensional reference pattern and the grayscale original image; Recognize based on hypotheses,
Information integrating means for integrating the recognized image with the distance information of each object obtained by the distance image generating means, and the uppermost surface area extracted from the distance image, Since the positions of a plurality of candidate objects are detected from the arrangement hypothesis, the information need only be transmitted once to the transfer means.
If the communication time is short and the loading method is determined in advance, it is possible to detect that the loading is incorrect,
It is also possible to determine from which of the topmost objects to grasp.

According to a twelfth aspect of the present invention, there is provided an object transfer device, comprising: a distance image generating means for generating a distance image of a plurality of stacked objects; An uppermost plane area extracting means for extracting, an object candidate combination hypothesis generating means for generating a plurality of combinations of object candidates as hypotheses from the output of the uppermost plane area extracting means, and an object size database for storing dimension data of the recognition target object A hypothesis validity verification unit that verifies the validity of the object candidate combination hypothesis generated by the hypothesis generation unit using information included in the two-dimensional reference pattern and the grayscale original image; Recognize based on high hypotheses,
An image processing device for outputting three-dimensional position information of an individual target object, comprising: an information integration unit for integrating the recognized image and distance information of each object obtained by the distance image generation unit; and an object transfer unit And the position of a plurality of candidate objects is detected from the arrangement hypothesis of the target object in the uppermost surface area extracted from the distance image. In the case of a short and predetermined loading method, it is possible to detect that a wrong loading is being performed, and it is also possible to determine which one of the uppermost objects is to be gripped.

An image processing apparatus according to a thirteenth aspect of the present invention provides an object array reference database for storing data serving as a reference for an object array, and a hypothesis validity for verifying the validity of the object candidate combination hypothesis generated by the hypothesis generation means. An object arrangement determining unit that compares the image recognized by the sex verification unit with the reference data to determine whether the arrangement is correct, and a warning that warns the operator of a determination result when the recognized image does not match the reference data Since the means is provided, if the stacking pattern of the recognized object is not the standard stacking pattern, a warning is issued to the user of the device or the manager of the host system to prevent the stacking from being performed correctly, thereby avoiding a dangerous state. be able to.

An object transfer device according to a fourteenth aspect of the present invention provides an object array reference database for storing data serving as an object array reference, and a hypothesis for verifying the validity of the object candidate combination hypothesis generated by the hypothesis generation means. An object arrangement determining unit for comparing the image recognized by the validity verification unit with the reference data to determine whether or not the arrangement is correct; and warning an operator of a determination result when the recognized image does not match the reference data. Since the image processing apparatus having the warning generating means and the transfer means are combined, if the stacking pattern of the recognition object is not the standard stacking pattern, the stacking pattern can be correctly stacked with the apparatus user or the manager of the host system. It warns that it is not rare and avoids dangerous situations.

An image processing apparatus according to a fifteenth aspect of the present invention detects the position of a target object by using an artificial reference pattern automatically generated from an object size database storing data serving as a reference of an object arrangement. And a real image reference pattern cutout unit that cuts out an image area corresponding to the target object at the detection position, and a real image reference pattern storage unit that stores the cut out real image reference pattern. Is configured to operate to use the actual image reference pattern,
The image memory may have a small capacity, the image input time is short, and the second and subsequent objects can be quickly and accurately detected.

The object transfer device according to claim 16 of the present invention uses an artificial reference pattern automatically generated from an object size database storing data serving as a reference of an object arrangement, and determines the position of the target object. Detected and has a real image reference pattern cutout means for cutting out an image area corresponding to the target object at the detection position, and a real image reference pattern storage means for storing the cut out real image reference pattern,
In the subsequent recognition operation, the image processing apparatus configured to operate so as to use the real image reference pattern and the transfer means are combined, so that the image input time is short and the object is accurately detected and transferred. The object transfer device can be used.

According to a seventeenth aspect of the present invention, there is provided an image processing apparatus, comprising: a distance image generating means for generating distance images of a plurality of stacked target objects; and a first distance image storage for storing the generated distance images. Means, initial grasping position calculating means for calculating the grasping position of the object based on information of the distance image stored in the first distance image generating means, and temporarily retracting the object to a predetermined position. After that, the second distance image storage means for storing the distance image of the remaining cargo, the distance image stored in the first distance image storage means, and the distance image stored in the second distance image storage means And a position correction unit that determines a gripping position of the evacuated object based on a comparison result of the distance image comparing unit, and detects a difference between the gripping position and an initial gripping position as a position correction value. And quantity detecting means. In the distance image can only be the position recognition of the object, even shorter be applied to process time for difficult objects to use the shading or edge image.

An object transfer device according to claim 18 of the present invention stores a distance image generating means for measuring a distance distribution of a plurality of stacked target objects from an image input means, and stores the generated distance image. First distance image storage means, object moving means for gripping and moving an object, initial holding position calculating means for calculating the holding position of the object based on the information of the distance image, object transferring means Grips the object at the gripping position,
A second distance image storage unit for temporarily storing a distance image of the remaining cargo after being compared with a predetermined position, a distance image stored in the first distance image storage unit, and a second distance image storage A distance image comparison means for comparing the distance images stored in the means, and a position correction amount for determining a gripping position of the retracted object using the comparison result of the distance images and detecting a difference from the initial gripping position as a position correction value It is a combination of an image processing device having a detecting means and an object transferring means for gripping and transferring an object. The object transfer device can also accurately recognize the object and safely transfer it to an appropriate position.

[Brief description of the drawings]

FIG. 1 shows a first embodiment of the present invention. FIG. 3 is a block diagram showing the configuration of FIG.

FIG. 2 is a first embodiment. 5 is a flowchart showing a flow of the operation of FIG.

FIG. 3 is a first embodiment. Is an image generated at each stage in.

FIG. 4 is a second embodiment of the present invention. FIG. 3 is a block diagram showing the configuration of FIG.

FIG. 5 shows a second embodiment. 5 is a flowchart showing a flow of the operation of FIG.

FIG. 6 is an explanatory diagram of irradiation of a random dot pattern.

FIG. 7 is an explanatory diagram of block matching of a stereo image.

FIG. 8 shows a third embodiment of the present invention. FIG. 3 is a block diagram showing the configuration of FIG.

FIG. 9 shows a third embodiment. 5 is a flowchart showing a flow of the operation of FIG.

FIG. 10 is an explanatory diagram of a situation where a step region portion is extracted from a low-resolution distance image, and high-resolution stereo correspondence is performed only in the step region portion of a high-resolution distance image.

FIG. 11 shows a fourth embodiment of the present invention. FIG. 3 is a block diagram showing the configuration of FIG.

FIG. 12 shows a fourth embodiment. 5 is a flowchart showing a flow of the operation of FIG.

FIG. 13 is a fifth embodiment of the present invention. FIG. 3 is a block diagram showing the configuration of FIG.

FIG. 5 is a flowchart showing a flow of the operation of FIG.

FIG. 15 is a flowchart of a stereo block association operation.

FIG. 16 is a diagram schematically showing how stereo images are associated with stereo.

FIG. 17 is a view showing a sixth embodiment of the present invention; FIG. 3 is a block diagram showing the configuration of FIG.

FIG. 5 is a flowchart showing a flow of the operation of FIG.

FIG. 19 is a diagram showing each stage of an operation state of extracting an object candidate using the object candidate hypothesis.

FIG. 20 shows a seventh embodiment of the present invention. FIG. 3 is a block diagram showing the configuration of FIG.

21. Embodiment 7 FIG. 5 is a flowchart showing a flow of the operation of FIG.

FIG. 22. Embodiment 8 of the present invention. FIG. 3 is a block diagram showing the configuration of FIG.

23. Embodiment 8 FIG. 5 is a flowchart showing a flow of the operation of FIG.

FIG. 24 is a ninth embodiment of the present invention. FIG. 3 is a block diagram showing the configuration of FIG.

25. Embodiment 9 FIG. 5 is a flowchart showing a flow of the operation of FIG.

FIG. 26 is a diagram showing each stage of an operation of correcting an initial grip position.

FIG. 27 is an explanatory diagram showing a processing procedure of a conventional image processing apparatus.

FIG. 28 is a flowchart of the operation of the process shown in FIG. 27.

FIG. 29 is a diagram illustrating the principle of spatial coding using pattern light.

FIG. 30 is an explanatory diagram of another conventional distance image measurement system.

FIG. 31 is an explanatory diagram of another conventional stereo correspondence search device for image processing.

FIG. 32 is a flowchart of another conventional image processing shown in FIG. 30;

[Explanation of symbols]

11 image input means, 12 distance image generation means, 13
Top-level surface area extraction means, 14 Top-level object candidate extraction means, 15 density original image storage means, 16 object size database, 17 two-dimensional reference pattern generation means, 18 object position detection means, 19 information integration means, 20 random texture pattern Light emitting means, 21 Image input means, 2
2 stereo image block correspondence means, 30 pattern projectors, 31 low resolution block correspondence means, 32 step area extraction means, 33 high resolution block correspondence means,
34 distance image synthesis means, 41a first image storage means, 41b second image storage means, 42 high-speed similarity calculation means, 43 high reliability similarity calculation means, 44 corresponding block reliability determination means, 45 distance image storage means , 51a
First image input means, 51b second image input means,
51c third image input means, 52 stereo image block associating means, 64 object candidate combination hypothesis generating means,
65 hypothesis validity verification means, 69 object position detection means,
71 object arrangement reference database, 72 object arrangement determination means, 73 warning generation means, 81 artificial reference pattern generation means, 82 real image reference pattern cutout means, 83
Actual image reference pattern storage means, 88 object position detection means, 91 distance image generation means, 92a first distance image storage means, 92b second distance image storage means, 93 distance image comparison means, 94 initial grip position calculation means, 95
Position correction amount calculating means, 96 Object moving means.

Claims (18)

[Claims] 1. A distance image generating means for generating a distance image of a plurality of stacked objects, an uppermost surface area extracting means for extracting an uppermost surface area of an uppermost object from the distance image, An uppermost-stage object candidate extracting means for individually separating and extracting an object from the surface region, an object size database for storing size data of the recognition target object, and a two-dimensional image of the object based on the size data of the object. A two-dimensional reference pattern generating means for generating a standard pattern image, a gray-scale original image storage means for storing a gray-scale original image input from a camera, and the two-dimensional reference pattern for the individually extracted top-level object candidate And object position detecting means for detecting the position of the object using the information of the grayscale original image, and the detection result of the object position obtained by each of the above means and the individual object obtained by the distance image generating means. An image processing apparatus comprising: information integration means for integrating distance information; and outputting three-dimensional position information of each target object. 2. A distance image generating means for generating a distance image of a plurality of stacked objects, an uppermost surface area extracting means for extracting an uppermost surface area of an uppermost object from the distance image, An uppermost-stage object candidate extracting means for individually separating and extracting an object from the surface region, an object size database for storing size data of the recognition target object, and a two-dimensional image of the object based on the size data of the object. A two-dimensional reference pattern generation means for generating a standard pattern image, a gray-scale original image storage means for storing a gray-scale original image input from a camera, and a two-dimensional reference pattern and gray-scale An object position detecting means for detecting the position of the object by using information of the original image; and an information integration means for integrating the detection result of the object and the distance information of each object obtained by the distance image generating means. And an image processing apparatus for outputting three-dimensional position information of each target object, and an object transfer means for gripping and transferring the object, and transferring the object based on an output signal of the image processing apparatus. Object transfer device. 3. The distance image generating unit includes: a random texture projecting unit that projects a random texture pattern; a first image input unit and a second image input unit that input a stereo image; 2. The image processing apparatus according to claim 1, further comprising stereo image block associating means for associating the two images captured by the second image input means. 4. A distance image generating means, comprising: a random texture light projecting means for projecting a random texture pattern; a first image input means and a second image input means for inputting a stereo image; An image processing apparatus comprising stereo image block association means for associating the two images captured by the second image input means, and an object transfer means for gripping and transferring an object; 3. The object transfer device according to claim 2, wherein the object is transferred based on an output signal of the device. 5. A distance image generating means, comprising: a low resolution distance image generating means for generating a coarse resolution distance image; a step area extracting means for extracting a step area portion of an object from the low resolution distance image; And a distance image synthesizing unit for synthesizing two distance images having different resolutions of the low-resolution distance image and the high-resolution distance image. The image processing apparatus according to claim 1, wherein: 6. A distance image generating means, comprising: a low-resolution distance image generating means for generating a coarse-resolution distance image; a step area extracting means for extracting a step area portion of an object from the low-resolution distance image; A high-resolution distance image generating means for generating a high-resolution distance image, and an image processing apparatus comprising distance image synthesizing means for synthesizing two distance images having different resolutions of the low-resolution distance image and the high-resolution distance image. 3. The object transfer device according to claim 2, further comprising an object transfer means for holding and transferring the object, and transferring the object based on an output signal of the image processing device. 7. A high-speed similarity calculating means for quickly searching left and right corresponding image blocks, and a high-reliability similarity calculating means for calculating a similarity with respect to a pair of blocks as search results with high reliability. 3. The image processing apparatus according to claim 2, comprising: a corresponding block reliability determining unit that determines that there is no corresponding block if the high reliability similarity is equal to or smaller than a predetermined threshold. 8. A high-speed similarity calculating means for quickly searching for left and right corresponding image blocks, and a high-reliability similarity calculating means for calculating a similarity with respect to a pair of blocks as a search result. And an image processing apparatus configured with a corresponding block reliability determination unit that determines that there is no corresponding block if the high reliability similarity is equal to or less than a certain threshold, and an object transfer unit that holds and transfers an object. 5. The object transfer device according to claim 4, further comprising: transferring an object based on an output signal of the image processing device. 9. An image processing apparatus comprising: three or more image input means, wherein the block association means selects two or more image pairs in which two images are selected from the respective images obtained from the three or more image input means. The image processing apparatus according to claim 1, wherein block association is performed, and a distance image is generated by integrating respective association search results. 10. An image processing apparatus comprising three or more image input means, wherein the block association means selects two or more image pairs in which two images are selected from the respective images obtained from the three or more image input means. An image processing apparatus that performs block association, integrates each association search result, and generates a distance image, and an object transfer unit that grips and transfers an object, and includes an output signal of the image processing apparatus. The object transfer device according to claim 2, wherein the object is transferred based on the information. 11. A distance image generating means for generating a distance image of a plurality of loaded objects, an uppermost surface area extracting means for extracting an uppermost surface area of an uppermost object from the distance image, Object candidate combination hypothesis generation means for generating a plurality of combinations of object candidates as hypotheses from the output of the plane area extraction means, an object size database storing dimension data of the recognition target object, and included in the two-dimensional reference pattern and the grayscale original image Hypothesis validity verification means for verifying the validity of the object candidate combination hypothesis generated by the above-mentioned hypothesis generation means using the hypothesis generation means, and performing recognition based on the hypothesis having the highest evaluation value among a plurality of hypotheses. Processing apparatus for outputting three-dimensional position information of an individual target object, comprising information integrating means for integrating the obtained image and the distance information of the individual object obtained by the distance image generating means . 12. A distance image generating means for generating a distance image of a plurality of loaded objects, an uppermost surface area extracting means for extracting an uppermost surface area of an object located at an uppermost position from the distance image, Object candidate combination hypothesis generation means for generating a plurality of combinations of object candidates as hypotheses from the output of the plane area extraction means, an object size database storing dimension data of the recognition target object, and included in the two-dimensional reference pattern and the grayscale original image Hypothesis validity verification means for verifying the validity of the object candidate combination hypothesis generated by the above-mentioned hypothesis generation means using the hypothesis generation means, and performing recognition based on the hypothesis having the highest evaluation value among a plurality of hypotheses. An image processing device comprising information integration means for integrating the obtained image and the distance information of each object obtained by the distance image generation means, and outputting three-dimensional position information of each target object; and An object transfer device comprising a transfer means for holding and transferring an object, and transferring the object based on an output signal of the image processing device. 13. An object array reference database storing data serving as a reference of an object array, an image recognized by a hypothesis validity verification unit for verifying the validity of the object candidate combination hypothesis generated by the hypothesis generation unit, and An object arrangement judging means for judging whether or not the arrangement is correct by comparing with data, and a warning generating means for warning an operator of a result of the judgment when the recognized image does not match the reference data, wherein 12. The image processing device according to item 11. 14. An object array reference database storing data serving as a reference of the object array, an image recognized by a hypothesis validity verification unit for verifying the validity of the object candidate combination hypothesis generated by the hypothesis generation unit, and An image processing apparatus having an object arrangement determining means for comparing the data with the data to determine whether or not the arrangement is correct, and a warning generating means for warning a worker of a result of the determination when the recognized image does not match the reference data; 13. The object transfer device according to claim 12, further comprising a transfer unit that transfers the object based on an output signal of the image processing device. 15. An actual image reference pattern cutout that detects a position of a target object and cuts out an image region corresponding to the target object at the detected position using an artificial reference pattern automatically generated from an object size database. And means for storing a cut-out real image reference pattern, wherein the real image reference pattern is stored in a subsequent recognition operation. The image processing apparatus according to any one of the preceding claims. 16. An actual image reference pattern cutout for detecting a position of a target object using an artificial reference pattern automatically generated from an object size database and cutting out an image area corresponding to the target object at the detected position. Means, and an image processing apparatus having real image reference pattern storage means for storing the cut-out real image reference pattern, and in the subsequent recognition operation, operating to use the real image reference pattern, and holding the object. 3. The object transfer device according to claim 2, further comprising a transfer unit that transfers the object, based on an output signal of the image processing device. 17. A distance image generating means for measuring a distance distribution of a plurality of stacked target objects from an image input means,
First distance image storage means for storing the generated distance image, and an initial gripping position for calculating the gripping position of the object based on information of the distance image stored in the first distance image generating means Calculating means, second distance image storing means for temporarily storing the distance image of the remaining cargo after the object is temporarily evacuated to a predetermined position, and distance image stored in the first distance image storing means. And a distance image comparing means for comparing the distance image stored in the second distance image storing means with the distance image comparing means. A gripping position of the retracted object is determined based on a comparison result of the distance image comparing means. An image processing apparatus comprising: a position correction amount detection unit that detects a difference between a position and the initial grip position as a position correction value. 18. A distance image generating means for measuring a distance distribution of a plurality of stacked target objects from an image input means,
First distance image storage means for storing the generated distance image, object moving means for gripping and moving the object, and initial holding position for calculating the holding position of the object based on the information of the distance image Calculating means, a second distance image storage means for holding an object at the holding position by the object moving means, temporarily storing the distance image of the remaining cargo after being compared with a predetermined position, and A distance image stored in the distance image storage means, a distance image comparison means for comparing the distance image stored in the second distance image storage means, and the retracted object using a comparison result of the distance images. Based on the output of the image processing apparatus having the position correction amount detecting means for determining the grip position of the object and detecting the difference from the initial grip position as the position correction value, and the image processing apparatus having the object transfer device. Depending on the transfer method Object transfer device for transferring an object.