JP2921496B2 - Image processing device and object transfer device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus and an object transfer apparatus for recognizing a position of a loaded load of a robot or an industrial machine for automatically transferring a load.

[0002]

2. Description of the Related Art An example of an image processing apparatus for automatically transferring a load is disclosed in Japanese Patent Application Laid-Open No. 6-249631. FIG. 16 is a schematic explanatory diagram of the image measurement processing procedure of the invention, FIG. 17 is a flowchart showing the operation,
FIG. 18 is a diagram showing the principle of space coding using pattern light.

[0003] The image processing apparatus is for the cardboard box W stacked on the pallet 1 and the measurement object, Figure 1
As shown in FIG. 8 , the pallet 1 comprises a light projector 2 obliquely above and a camera 3 disposed above the center of the pallet 1.
The light projector 2 emits various pattern lights in chronological order, so that the measurement space is overlapped with a wedge-shaped measurement area r0.
To rn. A slit pattern, for example, FIG. 18, B, C
The black and white binary patterns of the three patterns are projected. White is a light-irradiated part, and black is a light-irradiated part. Such a pattern can be created, for example, using a dot matrix electric shutter such as a liquid crystal shutter.

[0004] The state light is striking "1", when expressed as "0" when no hit, in FIG. 16,
When the pattern A is projected, the values are half and 1, 0 from the front to the rear. In pattern B, the values are 1, 0, 1, and 0 in quarters from the front. In pattern C,
It becomes 1, 0, 1, 0, 1, 0, 1, 0 in 1/8 increments.

When an image is picked up by projecting these three patterns, each pixel can be made to correspond to any of data encoded into a 3-bit code of “000” to “111”. Such a code is called a space code. A wedge-shaped area in a three-dimensional space corresponds to a certain space code. If the object W exists there, a code is assigned to an image area on the surface of the object. On the other hand, from the image captured by the camera, if the spatial code of a certain pixel on the image is known, it is possible to determine which wedge-shaped area corresponds to it. Therefore, the height of the object surface, that is, the distance from the camera can be determined by the principle of triangulation. Will be. In this description, since the types of patterns are three types of A, B, and C, the entire image is divided into two wedge-shaped regions. However, if the number of types of patterns is eight, the number of wedge-shaped regions is 256.
And the accuracy of the distance information can be more precisely improved. That is, by generating a space code image, the distance of each pixel on the image from the camera can be measured.

FIG. 16 will be described with reference to the flowchart shown in FIG . When the system is started in step ST01, the generated spatial code image in step ST02, the spatial code image as shown in FIG. (D) on the upper surface of the load in the state shown in FIG. 16 (A) is obtained, image By horizontally scanning from left to right and extracting an area having data having the largest space code, an image corresponding to the upper surface partial load of the uppermost object can be obtained as shown in FIG. In step ST04, the uppermost object regions are grouped. When spatial codes are close to each other within a predetermined range between adjacent pixels in a spatial code image, those areas are determined to be the same group, or a spatial code of a groove-shaped portion or a gap generated between upper edges of each cargo. Are grouped on the basis of changes. Each grouped object is processed in the following step S
At T05, the three-dimensional position of each object is measured. Some three-dimensional coordinates of a portion corresponding to a peripheral portion of the grouped objects are sampled, and in step ST06, the center of gravity position, posture, and height information of the object are obtained based on the three-dimensional coordinates. The process ends in step ST07. Using the position data of the cargo thus obtained, the cargo can be transferred one by one by a robot or the like.

In this image processing apparatus, it is necessary to obtain all information necessary for cargo recognition from a spatial code image and generate a high-resolution distance image such as a resolution of 256 × 256 or 512 × 512 pixels. A high-precision pattern projector with a mechanism to change the light projection pattern is used, and an image is input corresponding to each of multiple light emission patterns generated in time series, so a large capacity image memory is required And

[0008] As another conventional example, see IEICE Transactions Vol. J71-D. No. 7p1240-1257
Sato and Iguchi's paper, "Liquid Crystal Range Finder -High-speed Range Image Measurement System Using Liquid Crystal Shutter-"
Introduce the contents described in “−”. This technique relates to a distance image acquiring system according to the spatial encoding method used in the above 16 to 18 to the indicated techniques,
FIG. 19 shows the principle. A slit pattern based on an alternating binary code is projected onto an object in time series using the liquid crystal shutter 4, and an image corresponding to each pattern is captured by the camera 3 to generate a spatial code image. As the patterns, seven patterns as shown in FIG. 16 are used. The principle of measurement is the same as that used in the prior art of FIG. 16 to 18.

Another conventional technique is disclosed in Japanese Patent Application Laid-Open No. 7-10 / 1995.
"Stereo-compatible search device" described in Japanese Patent No. 3734
Will be described. FIG. 20 is an explanatory diagram of the hierarchical block matching (sparse / dense search method) of the present invention, and FIG. 21 is a flowchart of the operation of the present invention.

[0010] Stereo vision refers to measuring the parallax corresponding to the same point on an object between two images input from image input means arranged at a plurality of different positions, and using a camera based on the principle of triangulation. This is a method for calculating the distance to the target point. In ST11 of the flowchart of FIG. 21 , an image is input, the two images input in ST12 are respectively stored in an image memory, and in ST13, processing for converting the stored image to a hierarchically lower resolution is performed. An image is generated. FIG. 20 schematically shows the correspondence between the image of the D-th layer and the image of the D + 1-th layer. In this example, a quarter area of the image on the D-th layer is an image on the D + 1-th layer.

In step ST14, the image is divided into small blocks. In step ST15, an initial search position of each block is calculated by an initial search position calculator. In step ST16, a corresponding point of a block in another image is calculated by pattern matching. And step S
At T17, the parallax is calculated by the parallax calculator using the position information of the corresponding point between the two images. Based on the parallax image obtained in this way, the parallax is calculated again using a one-step high-resolution image. By repeating this process, a corresponding point search is finally performed on an image having the same highest resolution as the input image.

[0012]

In the image processing apparatus shown in FIGS. 16 to 18 , all the information necessary for recognizing the cargo is obtained from the spatial code image. For example, the resolution is 256 × 256 or 512 ×. It is necessary to generate a distance image with a high resolution of 512 pixels, a high-precision pattern projector is required, and it is necessary to have a mechanism for changing the projection pattern in a time-series manner.
It becomes a large-scale expensive device, and furthermore, it is necessary to input an image corresponding to each of a plurality of light-emitting patterns generated in time series, a large-capacity image memory is required, and a large-scale expensive device is required. There was a problem. In addition, there is also a problem that the total image input time required for this takes much time.

[0013] Also in the technique shown in FIG. 19, since recognizing the load on the basis of only the spatial code image, FIGS. 16 to
As in the case of No. 18 , there is a problem that a large image memory is required and a large-scale expensive device is required.

Further, in the image processing apparatus shown in FIGS. 16 to 18 and 19 , in order to obtain all the information necessary for the cargo from the space code image, the object to be grasped in the cargo and the adjacent object are brought into close contact. In other words, in the case of a cardboard box with a small ridge chamfer, the boundary with the adjacent object does not clearly appear as a three-dimensional step, or a bag such as a cement bag containing soft contents. In the case of (3), since the object is in close contact with the adjacent object, the boundary is unclear three-dimensionally, and there is a problem that it is difficult to separate the individual objects and hold them by the transfer means.

The image processing apparatus shown in FIGS. 20 and 21 aims at searching for a stereo corresponding point at high speed and acquiring a distance image in a short time. The search always starts from the image with the coarser resolution to the image with the higher resolution in small block units that are set uniformly on the image, so if a part that needs high-resolution information is known in advance However, there is a problem that the processing time is long.

The present invention has been made to solve the above-mentioned problem, and does not require a large-capacity image memory, and can grasp the three-dimensional position of an object in a short time with a simple device configuration. It is an object of the present invention to provide an image processing device and an object transfer device equipped with the image processing device.

[0017]

According to a first aspect of the present invention, there is provided an image processing apparatus, comprising: a distance image generating means for generating a distance image of a plurality of loaded objects; A top-level surface area extracting unit for extracting a surface region, a top-level object candidate extracting unit for individually separating and extracting objects from the top-level surface region, an object size database storing size data of a recognition target object, A two-dimensional reference pattern generating means for generating a standard pattern image on a two-dimensional image of the object based on the size data of the object, and a gray-scale original image storing means for storing a gray-scale original image input from the camera. Object position detecting means for detecting the position of the object using the two-dimensional reference pattern and the information of the grayscale original image with respect to the extracted top-level object candidate, and detecting the object position obtained by each means and generating a distance image It is obtained and an information integration unit for integrating the distance information of each object obtained by the stage.

According to a second aspect of the present invention, there is provided an object transfer apparatus for generating a distance image of a plurality of stacked objects, and extracting an uppermost surface area of the stacked object from the distance images. A top-level surface area extracting unit, a top-level object candidate extracting unit that individually separates and extracts objects from the top-level surface region, an object size database that stores size data of the recognition target object, and a size data of the object. A two-dimensional reference pattern generating means for generating a standard pattern image on a two-dimensional image of the object; a gray-scale original image storing means for storing a gray-scale original image input from a camera; Object position detecting means for detecting the position of the object using the information of the two-dimensional reference pattern and the grayscale original image for the object candidate, and the object position detection result obtained by each means and the distance image generating means. The is that a object transfer means for gripping and transferring an image processing apparatus and the object with an information integration unit for integrating the distance information of each object.

According to a third aspect of the present invention, in the image processing apparatus, the distance image generating means emits a random texture pattern, the first image input means inputs a stereo image, and the second image input means inputs a stereo image. And stereo image block associating means for associating the two images taken by the first and second image input means.

According to a fourth aspect of the present invention, in the object transfer device, the distance image generating means projects a random texture pattern, the first image inputting means for inputting a stereo image and the second image inputting means. An image processing apparatus comprising two image input means, a stereo image block associating means for associating two images taken by the first and second image input means, and holding and transferring an object Object transfer means.

According to a fifth aspect of the present invention, in the image processing apparatus, the distance image generating means includes a low-resolution distance image generating means for generating a coarse-resolution distance image, and extracts a step region portion of the object from the low-resolution distance image. Step area extracting means, high-resolution distance image generating means for generating a high-resolution distance image for the step area, and a distance image combining two distance images having different resolutions of the low-resolution distance image and the high-resolution distance image And a combining means.

According to a sixth aspect of the present invention, there is provided an object transfer apparatus, wherein the distance image generating means generates a low-resolution distance image having a coarse resolution, and extracts a step region portion of the object from the low-resolution distance image. Step area extracting means, high-resolution distance image generating means for generating a high-resolution distance image for the step area, and a distance image combining two distance images having different resolutions of the low-resolution distance image and the high-resolution distance image The image processing apparatus includes an image processing device configured by a combining unit and an object transfer unit that grips and transfers an object.

According to a seventh aspect of the present invention, there is provided an image processing apparatus, comprising: a distance image generating means for generating a distance image of a plurality of stacked objects; and extracting an uppermost surface area of an uppermost object from the distance image. A top-level surface region extracting unit, a plurality of object candidate combination hypothesis generating units that enumerate and generate a plurality of combinations of object candidates from the output of the top-level surface region extracting unit, A hypothesis validity verification unit that verifies the validity of the object candidate combination hypothesis generated by the hypothesis generation unit using the information included in the two-dimensional reference pattern and the grayscale original image; Recognize based on hypotheses,
An information integrating means for integrating the recognized image and the distance information of each object obtained by the distance image generating means.

According to an eighth aspect of the present invention, there is provided an object transfer apparatus, comprising: a distance image generating means for generating a distance image of a plurality of stacked objects; An uppermost plane area extracting means for extracting, an object candidate combination hypothesis generating means for generating a plurality of combinations of object candidates as hypotheses from the output of the uppermost plane area extracting means, and an object size database for storing dimension data of the recognition target object A hypothesis validity verification unit that verifies the validity of the object candidate combination hypothesis generated by the hypothesis generation unit using information included in the two-dimensional reference pattern and the grayscale original image; Information recognition means for performing recognition based on a high hypothesis, and integrating the recognized image with the distance information of each object obtained by the distance image generation means, and An image processing apparatus for outputting dimension position information, in which with a transfer means for transferring the object is gripped.

An image processing apparatus according to a ninth aspect of the present invention provides an object array reference database for storing data serving as a reference for an object array, and a hypothesis validity for verifying the validity of the object candidate combination hypothesis generated by the hypothesis generation means. An object arrangement determining unit that compares the image recognized by the sex verification unit with the reference data to determine whether the arrangement is correct, and a warning that warns the operator of a determination result when the recognized image does not match the reference data Means.

According to a tenth aspect of the present invention, an object array reference database for storing data serving as an object array reference, and a hypothesis for verifying the validity of the object candidate combination hypothesis generated by the hypothesis generation means. An object arrangement determining unit for comparing the image recognized by the validity verification unit with the reference data to determine whether or not the arrangement is correct; and warning an operator of a determination result when the recognized image does not match the reference data. The image processing apparatus includes an image processing apparatus having a warning generating unit, and a transfer unit that holds and transfers an object.

[0027]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiment 1 FIG. 1 is a block diagram showing the configuration of the first embodiment, FIG. 2 is a flowchart showing the flow of operation, and FIG. 3 shows images generated at each stage. Here, the object to be recognized is assumed to have a polyhedral shape such as a cardboard box, for example, so that the description of the operation is easy to understand.

In FIG. 1, reference numeral 11 denotes an image input means comprising two cameras arranged at a predetermined interval;
2 is a distance image generating means for generating a distance image from the image captured by the image input means 11, 13 is an uppermost plane area extracting means for extracting the uppermost plane from the distance image, 14 is an uppermost object candidate extracting means, 15 is Density image storage means for storing an original image; 16 an object size database; 17 a two-dimensional reference pattern generation means for automatically generating a reference pattern as a two-dimensional pattern based on information stored in the object size database 16; Reference numeral 18 denotes an object position detecting unit that detects a two-dimensional object position, and 19 denotes an information integrating unit that integrates the object position detected by the object position detecting unit 18 and the distance information of the distance image.

FIG. 3A shows an original image captured by the image input means 11, FIG. 3B shows an uppermost object area extracted by the uppermost surface area extracting means 13, and FIG. 3C shows a two-dimensional reference pattern generating means. A template of the reference pattern generated using the object size data, (d) is a schematic diagram showing one of the candidates extracted from the uppermost object region, and (e) shows a positioning situation by the template matching method.

In step ST 101, when the apparatus is started, the image to be recognized shown in FIG. In step ST102, a distance image is generated by the distance image generating means 12. Means for generating a distance image include a section coding method or a stereo correspondence search method. In step ST103, the uppermost object region extracting means 13 performs
An uppermost object region having a height corresponding to the uppermost object shown in (b) is extracted. In step ST104, the original image photographed by the image input unit 11 is stored in the grayscale image storage unit 15, and in step ST105, the two-dimensional reference pattern generation unit 17 uses the two-dimensional pattern based on the information stored in the object size database 16. Is automatically generated. This reference pattern is called a template, and as shown in FIG. 3C , the template is a contour plate expressing the contour of the object.

In step ST106, since the object in the uppermost object region represented by the binary image looks like a rectangular pattern on the image, candidates are extracted by detecting edges that intersect at right angles. As described above, since the resolution of the range image is coarse, the position of the candidate is inaccurate. Step S
At T107, the object candidate is determined by the object position detecting means 1
The positioning is performed by the template matching method using the two-dimensional reference pattern. However, since the rough position has already been detected at the stage of candidate extraction, matching is sufficient with a process of searching for only the nearest neighbor from the candidate detection position. Therefore, template matching can be executed at high speed. The state of the positioning is shown in FIG. When the object position is detected, the two-dimensional object position information is transmitted to the distance image generation unit 1 by the information integration unit 19.
2 and is output as the final result.

In step ST108, it is determined whether or not all the objects have been detected. If no object has been detected, the process is repeated from step ST106 in the processing flow, and a series of processing steps ST106 and S106 are performed for another object candidate.
T107 is performed. If all the objects have been detected, the recognition process ends in step ST109. As described above, according to the present invention, a range image is roughly generated, an object candidate is extracted based on the result, and detailed position detection is performed using template matching, which is a two-dimensional general-purpose processing method, in the range of pattern search. By performing the processing in a very limited form, the device scale of the range image generation device can be reduced.

It should be noted, as the distance image generating means, how Ru used to roughen the resolution spatial encoding method, or may be a method such as a light-section method according to the stereo vision or slit light scanning, coarse resolution Any device can be used as long as the device can generate the distance image of.

Although the example of the positioning by the template matching based on the contour information has been described, a method may be used in which the contour of the object is regarded as a combination of straight lines and the position is aligned for each side as a straight line. Further, normal two-dimensional template matching may be used using the contents of the object size database. Although the data storage format used inside the template matching process is not particularly specified, in addition to the method of storing the data as a two-dimensional image, the positions of the edges of the outline part are arranged one-dimensionally and stored inside. A method may be used.

Further, when extracting the candidates for the top object, the method of using the corner portion from the top surface area information obtained from the distance image has been described. The same effect can be obtained by applying a method of coarsely detecting a position by a binary template matching method using a separately prepared template as a binary distance image using a distance image.

The operation flowchart of FIG. 2 is an example, and another processing flow may be used as long as the input / output relationship of each unit shown in FIG. 1 is appropriate. For example, either the step of generating the distance image or the step of inputting the original grayscale image may be performed first.

The above description has been made on the assumption that a single-type object of a polyhedron, that is, a box-shaped object is assumed as an object to be recognized. However, even if the object is a bag-shaped object such as a cement bag or a rice bag, Image recognition can be realized with the same configuration without changing the configuration of the present invention. Next, the situation in the case of recognizing a bag-shaped object will be described in comparison with the case of the box-shaped object.

In step ST105 of FIG. 2, a two-dimensional reference pattern is automatically generated by the two-dimensional reference pattern generation means 16. This corresponds to template generation in template matching for positioning an object. In the case of a bag-shaped object, since the shape is unstable unlike a box-shaped object such as a deformed edge, it is not sufficient to take out a rectangular contour portion as shown in FIG. 3 as the reference pattern. In this case, a partial template representing only the corners of the object by the contour or a straight-line contour partial template representing the sides of the object is automatically generated. In either case, the contents are generated with reference to the contents of the object size database 15 storing the size of the bag-shaped object.

In the object candidate extraction in step ST106, in the case of a bag-shaped object, it is difficult to extract a candidate using the information of the edges that intersect at right angles as described above. In this case, the image expressing the uppermost object region is regarded as a binary image, and a binary object template is automatically generated from the information on the bag-shaped object model stored in the object size database 15 and these are roughly processed. By performing binary template matching, rough candidate extraction becomes possible.

In step ST107, the position of the object is detected. Here, the contour template of the corner portion or the straight edge portion of the object is matched with the edge image obtained from the input image as a partial template. You. The position of the object is the result of the matching
Comprehensively determined and calculated from matching positions of a plurality of partial templates. As described above, a bag-shaped object can be operated to the same degree as recognition of a box-shaped object.

In the first embodiment described above, a situation in which objects of a single type are stacked is assumed. However, the dimension data of a plurality of objects is stored in the By automatically generating a two-dimensional reference pattern and causing the object position detecting means 17 to perform an operation of matching all of the plurality of reference patterns and determining that the object has a template with the highest similarity. It can be operated properly even in a situation where a plurality of different types of objects are stacked.

By incorporating the image processing apparatus having the above-described configuration into the object transfer apparatus, it is possible to eliminate the need for a high-precision pattern projector and a large-capacity image memory and to operate the object transfer apparatus which can operate in a short time with a short image input time. Mounting device.

Embodiment 2 FIG. 4 is a block diagram showing the configuration of the second embodiment, and FIG. 5 is a flowchart showing the operation flow. FIG. 6 is an explanatory diagram of random dot pattern irradiation, and FIG. 7 is an explanatory diagram illustrating a method of block association processing from a stereo image. Embodiment 2 Is to generate a distance image by associating a part of the distance image generating means of Embodiment 1 with stereo by random texture pattern projection. This point is different, and the other configuration is different from that of Embodiment 1. Same as 1.

In FIG. 4, a random texture pattern projecting means 20 projects a random dot pattern.
Reference numeral 21 denotes an image input unit, which includes a first image input unit 21a and a second image input unit 21b. Reference numeral 22 denotes a stereo image block associating unit. Other uppermost image area extracting means 13, uppermost object candidate extracting means 1
4. Original density image storage means 15, object size database 1
6. The two-dimensional reference pattern generating means 17, the object position detecting means 18, and the information integrating means 19 are the same as those in the first embodiment. This is the same configuration as.

The operation of the second embodiment will be described below with reference to the flowchart of FIG. Step ST
When the power is turned on at 201, the power of the random texture pattern projection unit 20 is turned on, and a random dot pattern is projected on the recognition target. As shown in FIG. 6A, the random texture pattern projecting means projects a random dot pattern on an object to be recognized by the pattern projector 30. The pattern floodlight is installed downward above the object as shown in FIG.
A random dot as shown in (b) is projected on the recognition object. Embodiment 2 FIG. In, the target object to be recognized is a cardboard box, but a bag-like object such as a cement bag can be recognized. Random dot is projected on the upper surface of the object, in step ST 203, the stereo image of the pair of images captured by the first image input unit 2 1 a and the second image input unit 21b is input. This stereo image is captured by two cameras having the same resolution and the same focal length at a predetermined distance from the optical axis.

Next, the random dot pattern will be described. In the second embodiment, the random dots shown in FIG. 6B are configured with a resolution of 128 × 128, each pixel is a square, and a computer randomly determines whether each pixel is white or black. Is assigned, and the ratio of the number of white and black pixels is approximately 1: 1. Pattern floodlight 30
Is equipped with a mechanism to adjust the focus of the projection pattern, and if the approximate distance between the projector and the target object is known, the focus is adjusted to focus on the object surface based on that data . If no distance is known in advance, the camera operates so as to focus on the distance adjusted in advance.

When the input of the stereo image pair is completed, in step ST204, the random texture pattern
0 is turned off. Next, in step ST205, the stereo image is processed by the block-corresponding stage 22. FIGS. 7A and 7B are a pair of stereo images obtained by photographing a target object. 7A is called a left image, and FIG. 7B is called a right image. Block matching is performed by dividing the right image into small blocks in a grid as shown in the figure. Now, of the plurality of divided small blocks generated, the block of interest bR is called a block of interest. Actually, the random dot pattern is projected on the left and right images. Here, the pattern is not drawn for the sake of conversion. A position on the left image corresponding to the block of interest bR is searched.
A pattern having the same pattern as bR is searched for in a separately set search range. As a search method, a search based on minimizing the accumulated difference absolute value is employed. In this method, the blocks of interest bR are sequentially superimposed in the search range on the left image while being moved, and a position at which the similarity becomes the best is set as a corresponding point. At this time, the similarity evaluation
The cumulative difference absolute value is used. For example, if the target block is bR
Assuming that (i, j), the left image is bL (i, j), and the block size is N × N, the cumulative difference absolute value S _SAD (dx, dy) at the position (dx, dy) on the left image is ( Equation 1)
It is represented by Embodiment 2 Then, N = 16.

[0048]

(Equation 1)

The S _SAD obtained by (Equation 1) is minimized (d
By detecting (x, dy), a block bL on the left image corresponding to the target block bR is found.
The stereo method is a method of obtaining the distance from a camera to an object based on the principle of triangulation by using the difference between the blocks obtained in this way as parallax. The series of block association processing described above is repeatedly performed on all the small blocks on the right image, and the corresponding position on the left image is searched for each block to generate a distance image.

The subsequent steps are the same as in the first embodiment. In step ST206, the uppermost surface area extracting means 13
The area corresponding to the uppermost object plane is extracted from the distance image by the following step ST207. In the subsequent step ST207, the grayscale original image captured by the first image input unit 20a or the second image input unit 20b is stored in the grayscale original image storage unit. 15 is stored. At this time, of course, the random dot pattern is not projected. In step ST208, using the dimension data of the recognition target object stored in the object dimension database 16, the two-dimensional reference pattern generating means 1 is used.
7, a two-dimensional reference pattern is automatically generated.

In step ST 209, the image from which the uppermost surface area is extracted is divided into two, that is, the uppermost surface and the other surfaces, and is binarized.
With this, one of the top-level object candidates is extracted. In step ST210, the two-dimensional reference pattern and the object position detecting means 1 are used for the vicinity of the position of the extracted one object candidate.
The accurate position of the candidate object is measured and detected by the object position detecting means 18 using the gray-scale original image information stored in 5. In step 211, the determined two-dimensional position of the candidate object is integrated with the distance image information by the information integration means 19, and the processing in steps ST209 to ST211 is repeated until it is determined in step ST212 that all objects have been recognized. If it is determined in step ST212 that all objects have been recognized, finally,
Output as dimension information.

If it is determined that all the objects have been recognized by the processing from steps ST209 to ST211, the processing ends in step ST213. The determination as to whether or not all the objects have been recognized is performed by sequentially erasing the objects whose positions have been determined based on the binarized distance image from which the uppermost surface area has been extracted, and remaining on the image. This is performed by checking whether or not the uppermost surface area exists.

As described above, the second embodiment.
Since a random dot pattern is projected on the recognition target object, for example, even in the case of a plain cardboard box having no pattern on the surface, it is possible to obtain distance data at each position on the surface, and It is possible to obtain a stereo image which is not affected by an indeterminate texture such as a destination label. Also, a simple block matching can be applied to the matching process for stereo correspondence.

In the second embodiment, as the resolution of the camera, a random dot pattern of 128 × 128 pixels is projected on the camera field of view. However, depending on the size of the recognition target, a finer or coarser dot pattern is projected. The size of 16 × 16 pixels was adopted as the block size for block matching.
× 8 pixels, 32 × 32 pixels, or the like may be used. Also, the shape of the block need not necessarily be square. Further, the search range at the time of matching is set to a rectangle in FIG. 7, but it is possible to improve the reliability of the matching and to shorten the processing time by variously changing according to the situation. It is.

Further, in the second embodiment, black and white binary square dots are used as random dot patterns, but dot patterns with shading or colored patterns may be used, for example. Regarding the shape, it is also possible to use a combination of marks of various sizes and various shapes, and the geometrical positional relationship between the pattern projector and the target object can be viewed from vertically above the object to directly below it. Although the example of projecting a pattern has been described, a range image can be similarly detected by projecting obliquely.

Also, an example in which a dot pattern generated at a random position is used as a pattern to be projected has been described.
Considering that the search for a corresponding point in the block matching process is almost a search in the horizontal direction, for example, even if a vertical slit group with a random period or a vertical slit group with a random width is projected as a pattern, the same applies. Needless to say, it works.

Although the method of forming the pattern projector has not been described in detail, the accuracy of the projection position of random dots is not required to be high, and it is not necessary to change the projection pattern during the processing. For example, as shown in the prior art, a complicated projection device such as a liquid crystal projector is not required, and a device having a simple configuration such as an ordinary home slide screen is sufficient. This is one of the advantages of the device configuration of the present invention.

In the second embodiment, as described above, the operation has been described on the assumption that the object to be recognized is a cardboard box, that is, a hand-shaped object, but an object having another shape, for example, a cement bag. Even with a bag-shaped object, a range image can be easily generated by the configuration of the present invention, and the three-dimensional position of the target object can be measured by the recognition procedure in the present embodiment. The processes after the generation of the distance image are the same as those in the first embodiment . In other words, a final object position can be detected by having a corner portion and a linear contour portion of the object as a two-dimensional reference pattern and integrating the results of positioning with each recognition target. As described above, in the second embodiment, the three-dimensional information of the object can be recognized not only in the case where the shape of the object is a polyhedron but also in the case of a shape such as a bag.

Embodiment 3 FIG. 8 is a block diagram showing the configuration of the third embodiment, and FIG. 9 is a flowchart showing the operation flow. FIG. 10 is an explanatory diagram of a situation in which a step region portion is extracted from a low-resolution distance image, and high-resolution stereo correspondence is performed only in the step region portion of the high-resolution distance image. In FIG. It has the same function as, and the description is omitted.
Reference numeral 31 denotes a low-resolution block associating unit that generates a low-resolution distance image from a stereo image captured by the image input unit 21 at a low resolution; 32, a step-region extracting unit that extracts a step region at a low resolution; A high-resolution block association unit for generating a high-resolution distance image for the step region extracted from the image; a distance for combining the low-resolution distance image and the high-resolution distance image of the step region extracted from the distance image; Image combining means.

In step ST301, when the power is turned on and started, in step ST302, the lamp power of the random texture pattern projecting means 20 is turned on, and a random dot pattern is projected on the recognition target. In step ST303, a stereo image of a pair of images is input to the low-resolution block association unit 31 and the high-resolution block association unit 32 by the image input unit 21 including a pair of cameras. When the input of the stereo image is completed, the random texture pattern projection unit 20 is turned off in step ST304. In step ST305, the gray image of the target object is stored in the gray image storage means 15 from the first image input means 21a or the second image input means 21b.

In step ST306, the stereo image is processed by the low resolution block association means 31. FIG. 10A is a diagram schematically showing a low-resolution distance image portion A obtained by the low-resolution block association unit 31. Although a CCD camera with a resolution of 512 × 512 pixels is used as the image input means, a distance image of 32 × 32 pixels is generated as the low-resolution distance image, and the method of block matching is the method described in the first embodiment. The method is performed in the same manner as described above, for example, by setting the block size to 16 × 16 pixels. In step ST307, the step region extracting means 32 extracts a step region portion B of the object based on the low resolution distance image. FIG. 10B is a schematic diagram of the step region portion B. In step ST308,
The high-resolution block association unit 33 generates a higher-resolution distance image only for the step area (the hatched area B-A). In this case, the high resolution is 128
× 128 pixels. This process is obtained by setting the block size to 8 × 8 in the block matching process, and detecting the corresponding point while shifting the block by 4 pixels. FIG. 10C shows the distance image C of the step region portion. Show. In FIG. 10C, a hatched portion (in the frame B) is a step region of the object obtained from the low-resolution distance image, and a gray portion (in the frame C) is newly obtained as a result of generating the high-resolution distance image. This is a portion corresponding to the step region portion of the obtained distance image. In step ST309, the low-resolution distance image and the high-resolution distance image are synthesized by the distance image synthesizing means 34, and a distance image having a resolution of 128 × 128 as shown in FIG. 10D is obtained.

Step ST310 and subsequent steps are the same as those in the second embodiment. Is processed in the same way as In step 310, the uppermost plane area extracting means 13 extracts the uppermost area corresponding to the uppermost object plane from the distance image, and then proceeds to step ST311.
Then, the two-dimensional reference pattern is automatically generated by the two-dimensional reference pattern generation means 17 using the size data of the recognition target object stored in the object size database 16. In step ST312, the image from which the above-mentioned uppermost plane area is extracted is divided into two, that is, the uppermost plane and the other planes, and is binarized. One is extracted. Step ST
In 313, the precise position of the candidate object is measured and detected by the object position detecting means 18 using the two-dimensional reference pattern and the grayscale original image information in the vicinity of the position of the extracted one object candidate. The two-dimensional position of the candidate object determined in this way is integrated with the synthesized high-resolution range image information by the information integration means 19, and finally output as three-dimensional information of the object. Step ST3
At 14, it is determined whether or not all the objects have been recognized by the processing of steps ST312 and ST313, and the recognition is performed.
If it is determined that the identification will end the processing at step ST 315, if the object does not recognize it is determined that still exists, repeat the processing of extracting the object candidate returns to the step ST 312, ST 313. It is determined whether or not all the objects have been recognized.
The determination is performed by erasing the objects whose positions of the object are sequentially determined based on the digitized distance image and checking whether or not the uppermost surface area remaining on the image exists.

As described above, in the third embodiment, in order to finally obtain a high-resolution distance image, a low-resolution distance image is obtained in a short time in advance, and an area corresponding to a step portion of an object is obtained. By performing high-resolution stereo correspondence processing limited to the above, it is possible to generate a high-resolution distance image in a short calculation time, and it is possible to improve the extraction accuracy of object candidates and the like. .

The low-resolution distance image was used as 32 × 32 pixels, and the high-resolution distance image was used as 128 × 128 pixels.
A different resolution may be set in consideration of the permissible processing time, the size of the target object to be recognized, etc., depending on the combination of the high-speed processing of the low-resolution distance image generation and the high-resolution stereo correspondence processing. The benefits are not lost.

Further, the stereo image at the time of generating the low-resolution distance image and the stereo image at the time of generating the high-resolution distance image are configured as the same image.
Block matching may be performed based on a reduced image obtained by reducing an image of a full resolution (the same resolution as the CCD camera resolution) to に or １ ／. Further, at the time of generating a high-resolution distance image, the lenses of the CCD cameras of the first and second image input means 21 may be zoomed up so as to capture a step portion of the object in the low-resolution distance image in more detail.

Further, the random dot pattern projected on the target object is a fixed pattern. However, two types of dot pattern projectors having different sizes are prepared, and a large dot pattern is projected when a low-resolution distance image is generated. When the distance image is generated and modified so as to project a smaller dot pattern, a highly accurate distance image can be obtained.

Embodiment 4 FIG. FIG. 11 is a block diagram showing the configuration of the fourth embodiment, and FIG. 12 is a flowchart showing the operation flow. In FIG. 11, 11 to 13, 15 to 17, 19, and 21 have the same functions as in the first embodiment, and a description thereof will be omitted. In the figure, reference numeral 64 denotes an object candidate combination hypothesis generation unit, 65 denotes a hypothesis validity verification unit, and 69 denotes an object position detection unit.

When the apparatus is started in step ST601, the images of the target object are input to the distance image generation means 12 by the two cameras of the image input means 11 and stored individually. The two cameras having the same specifications without sharing the optical axis are used. In step ST602, the two stereo images are stereo-matched by the distance image generation means 12, and a distance image is generated from the parallax.

At step ST603, the image input means 11
Is stored in the grayscale image storage means. Sixth Embodiment FIG. 13A shows an original grayscale image of an object to be recognized. The object is a box-like object such as a cardboard box, and they are stacked in several stages. In step ST604, the distance image is extracted as an area from the image of the upper surface of the uppermost object by the uppermost surface area extracting means 13. FIG. 13B shows the extracted uppermost object region. As shown in the figure, areas other than the top row are also shown.

In step ST605, using the data stored in the object dimension database 16 storing the dimension data of the box-shaped object, the reference pattern expressing the two-dimensional image pattern of the object by the two-dimensional reference pattern generating means 17 is used. Is automatically generated. FIG. 13C shows a schematic diagram of the reference pattern.

The following description of the operation is the essential part of the fourth embodiment. In step ST607,
The object candidate combination hypothesis generating means 64 generates one combination hypothesis of the object candidate from the extracted uppermost surface area and the two-dimensional reference pattern. Fig. 13 (c)
In the fourth embodiment,
This hypothesis is called “hypothesis 1”. The hypothesis is that a reference pattern of a recognition target is processed by pattern matching, for example, a template matching technique with respect to an image expressing the uppermost object region as shown in FIG.
It is obtained by inferring a combination of two-dimensional reference patterns forming the uppermost surface area as shown in FIG. Also,
At the same time, the two-dimensional position of the object is detected by the object position detecting means 69 as shown in FIG. The object position detection is performed by comparing the stored grayscale original image with the two-dimensional reference pattern. Using the information of the contour portion of the two-dimensional reference pattern, template matching is performed with the edge image detected from the grayscale image. At this time, since the coarse position of the object is known at the hypothesis generation stage, the pattern search area in the template matching process need only be near the above position.

In step ST608, it is verified whether the above hypothesis is valid. The hypothesis validity verification unit 65 generates a hypothesis image as shown in FIG. 13D, and generates a hypothesis image by combining six objects from object candidates 1 to 6 as shown in FIG.
It is calculated whether the uppermost surface region such as can be generated reasonably. That is, the hypothesis image generated by the six object candidates is compared with the top-level surface image, and the area of the difference region is calculated. The larger the area is, the more erroneous the hypothesis is determined. Works. Step ST60
In step 8, one validity index is given to one hypothesis generated in step ST606. The reciprocal of the above area is the validity index.

In step ST609, step ST6
The processes from 06 to ST608 are repeated until all possible hypotheses are generated, and in step ST610, all the hypotheses are generated, and the hypothesis with the highest validity is selected from them. That is, the validity index for each hypothesis is compared, and the hypothesis having the highest index is selected. FIG.
(E) and (f) are different hypotheses 2 and 3, respectively. As is clear from the figure, in this case, hypothesis 2 has the highest validity index, and this is selected as the best hypothesis in step ST610.

In step ST611, the information integrating means 1
In step ST612, the two-dimensional position information corresponding to the above-described best hypothesis and the distance image are integrated. In step ST612, the distance image is output from the apparatus as a recognition result, and the process ends.

In the fourth embodiment, stereo vision by two cameras is used as the distance image generation means, but stereo vision by three or more cameras and projection of a characteristic projection pattern such as random dots are also used. It may be a stereo image or a range image generating means using a spatial coding method described as a conventional technique. Embodiment 4 is characterized in that a hypothesis of a recognition result obtained from a distance image is verified to obtain a plausible result, so that any method of generating a distance image may be used.

In the fourth embodiment, as a method of verifying the validity of a hypothesis, a method of calculating a validity index based on a difference between a hypothesis image and an actual image is used.
A similar effect can be obtained by using an index such that the larger the sum of the similarities of the template matching calculated in obtaining the above hypothesis for all the candidate objects, the higher the validity is.

In the fourth embodiment, as a method of detecting an object position, contour information of an object obtained from a reference pattern and an edge image generated from an original gray-scale image are processed by a template matching method. The same effect can be obtained by using a positioning method such as template matching of a binary image.

Further, in the fourth embodiment, a processing flow in which generation and verification of combination hypotheses of object candidates are sequentially performed has been described. However, a large number of hypotheses are generated and stored at once, and their validity is stored. It is also possible to perform sex verification at one time and select the most appropriate one.

Further, in the fourth embodiment, all hypotheses have been described as being generated. However, since hypotheses are obtained as combinations of object candidates, they can be treated as an object combination optimization problem and an approximate solution can be obtained. Similar effects can be obtained.

Embodiment 5 FIG. 14 is a block diagram showing the configuration of the fifth embodiment, and FIG. 15 is a flowchart showing the operation flow. 14, 11 to 13 and 15 to 19 correspond to the sixth embodiment. It has the same function as, and the description is omitted. Reference numeral 71 denotes an object array reference database, 72 denotes an object array determining unit, and 73 denotes a warning generating unit.

The fifth embodiment is different from the fourth embodiment in that a warning is issued when an object array is incorrectly arranged. Therefore, steps ST701 to ST711 of the flow in FIG. 15 are the same as steps 601 to 611 in FIG.

When the apparatus is started in step ST701, the image of the target object is input to the distance image generating means by the image input means 11 and stored individually. Step S
At T702, the two images are stereo-matched by the distance image generation unit 12, and a distance image is generated from the parallax. This processing does not necessarily need to be simple stereo matching. The stereo vision using a pattern projector as shown in FIG.

In step ST703, the image input means 1
The gray scale original image of the recognition target is input by 1 and stored in the gray scale original image storage means 15. In step ST704, the uppermost surface area extraction unit 13 extracts the uppermost surface area. In step ST705, a reference pattern expressing a two-dimensional image pattern of the object is automatically generated by the two-dimensional reference pattern generation means 17 using the object size database 16 storing the size data of the box-shaped object. In step ST706, the object candidate combination hypothesis generation unit 6
In step 4, one combination hypothesis of the object candidate is generated from the extracted uppermost surface area and the reference pattern. Embodiment 6 is a hypothesis. FIG. 13D similar to that described in FIG.
It is as shown in. Embodiment 7 In FIG. 13B, the hypothesis is obtained by processing the reference pattern of the recognition target object by pattern matching, for example, a template matching technique, with respect to an image expressing the uppermost object region as shown in FIG. Is obtained by estimating a combination of two-dimensional reference patterns forming the uppermost surface area as described above. In step ST707, the two-dimensional position of each of the candidate objects included in the above hypothesis is detected by the object position detecting means 69 as shown in FIG. The object position detection is performed by using the stored grayscale original image,
Using the information of the contour part of the two-dimensional reference pattern, it is detected by template matching with the edge image detected from the grayscale image.

In step ST708, the hypothesis image generated by the plurality of object candidates is compared with the top image, and the area of the difference area is calculated. Judge as high. Step 7
In step 09, the processing of steps ST706 to ST708 is repeated until all possible hypotheses are generated.
At 0, all hypotheses are generated and the most relevant hypothesis is selected. That is, the validity index for each hypothesis is compared, and the hypothesis having the highest index is selected.

In step ST711, the information integrating means 1
9 integrates the two-dimensional position information corresponding to the best hypothesis and the distance image. At this point, the image processing apparatus according to the fifth embodiment knows the arrangement of the objects and the three-dimensional position of each object, and is stored inside the apparatus. In step 712, the object arrangement determination means 72 determines whether the stored recognition result matches the correct arrangement data prepared in advance or not. The correct array data is an array pattern determined by the recognized object stored in the object array reference database 71. For example, in a logistics factory in which the apparatus according to the fifth embodiment is used, a cardboard box as a recognition target object needs to be stacked in advance on a pallet called a “stacking pattern” that is correct according to its shape and dimensions. It is decided.

The object array reference database describes the array of objects to be used as such a reference. If the result recognized in step ST712 does not match the reference stacking pattern, the process proceeds to step ST713.
In the above, a warning is generated by the warning generating means 73 to the user of the apparatus or the manager of the host system to notify that the object is not correctly loaded. This Embodiment 5
A warning is generated by the sound of the buzzer and the display of characters on the display device connected to the device. If it is determined in step ST712 that the recognition result is a correct pattern, the final object recognition result is output by the information integration means 19, and the data is transmitted to the robot, for example, so that the robot performs the object gripping operation. Will be. Step ST for the recognition operation
The operation ends at 715.

In the fifth embodiment, in addition to recognizing an object as described above, it is determined whether or not the array is an expected array by comparing the array pattern with reference data. If it is different from the above, it has a function to generate a warning. As a result, there is an advantage that the operator can notice a mistake in his / her loading operation or detect a severe collapse of the cargo during transportation.

As a method of verifying the validity of a hypothesis, a method of verifying stereoscopic vision by two cameras as a distance image generating means based on a difference between a hypothesis image and an actual image is described.
As a method of detecting an object position, a template matching method using contour information of an object is used. However, it is needless to say that the same effect can be obtained even if these partial processing means are realized by another method. For example, Embodiment 2 is used as a distance image generation unit. The stereo vision using the pattern described in the above is used as a method for verifying the validity of the hypothesis.A method that determines that the higher the sum of similarities comparing each object with the gray level reference pattern, the higher the validity is, is the object position detection method. Alternatively, a template matching method using a normalized cross-correlation and a grayscale image pattern prepared in advance for each object included in the hypothesis may be adopted.

Also, an example has been described in which a warning is issued when it is determined that the recognition result does not match the object array reference database stored in advance, but the content of the warning is determined according to the degree of similarity with the reference data. Alternatively, the device user or the operator may be notified. For example, a message such as "The correctness of the stacking method determined as a result of the automatic recognition is level 8" is displayed on the control display. Furthermore, as a result of the recognition, the collapse state of the cargo is very remarkable, and if it is judged that it is not appropriate to continue the work as it is, not only will the above warning be issued, but also the upper management computer will stop the system. May be requested, or the system may directly stop the system.

Further, although a character display by a buzzer and a display is employed as the warning means, a method of generating a synthesized human voice may be used as a means for transmitting a warning to a human.

[0091]

According to the first aspect of the present invention, there is provided an image processing apparatus comprising: a distance image generating means for generating a distance image of a plurality of objects; and an uppermost surface region extraction for extracting an uppermost surface region of the object from the distance images. Means, an uppermost object candidate extracting means for individually extracting and extracting an object from the uppermost surface area, and generating a two-dimensional reference pattern on a two-dimensional image of the object based on the size data of the object to be recognized Two-dimensional reference pattern generation means;
Original gray-scale image storage means for storing the gray-scale original image input from the camera, and object position detection for detecting the position of the object using the information of the two-dimensional reference pattern and the gray-scale original image for the individually extracted top-level object candidate Means, and information integration means for integrating the distance information of each object obtained by the distance image generation means and the detection result of the object position obtained by each means, and Since the object candidates can be detected by matching the coarse distance image with the two-dimensional reference pattern, the image memory can be of a small capacity, the image input time is shortened, and individual objects can be clearly detected three-dimensionally.

The object transfer device according to a second aspect of the present invention is a distance image generating means for generating a distance image of a plurality of objects, and an uppermost surface area extracting means for extracting an uppermost surface area of the object from the distance images. And an uppermost-stage object candidate extracting means for individually separating and extracting objects from the uppermost-plane region, and generating a two-dimensional reference pattern on a two-dimensional image of the object based on the size data of the recognition target object. Dimensional reference pattern generation means;
Original gray-scale image storage means for storing the gray-scale original image input from the camera, and object position detection for detecting the position of the object using the information of the two-dimensional reference pattern and the gray-scale original image for the individually extracted top-level object candidate Means, and an image processing apparatus having information integrating means for integrating the distance information of each object obtained by the distance image generating means and the detection result of the object position obtained by each means. In this way, as an object transfer device, the object detection time is short, the position is accurately detected,
The transfer of objects can be performed quickly and accurately.

According to a third aspect of the present invention, in the image processing apparatus, the distance image generating means projects a random texture pattern, the first image inputting means for inputting a stereo image, and the second image inputting means. And a stereo image block associating means for associating the two images taken by the first and second image input means, so that a random dot pattern is projected on the recognition target object. Since light is emitted, even when a solid object having no design on the surface or a label reflecting light is affixed to the surface, there is an effect that distance data at each value of the surface can be accurately obtained.

According to a fourth aspect of the present invention, in the object transfer device, the distance image generation means projects a random texture pattern, the first image input means for inputting a stereo image, and the second image input means. Combination of an object processing means with an image processing apparatus comprising two image input means and a stereo image block associating means for associating two images captured by the first and second image input means. By doing so, since a random dot pattern is projected on the recognition target object, even if a solid object without a pattern on the surface, or even if a label reflecting light is affixed to the surface, Each surface value is accurately obtained, and as an object transfer device, the object detection time is short,
The position of the object is accurately detected, and the transfer of the object can be performed quickly and accurately.

According to a fifth aspect of the present invention, in the image processing apparatus, the distance image generating means extracts a stepped area of the object from the low resolution distance image, the low resolution distance image generating means generating a coarse resolution distance image. Step area extracting means, high-resolution distance image generating means for generating a high-resolution distance image for the step area, and a distance image combining two distance images having different resolutions of the low-resolution distance image and the high-resolution distance image Since it is configured with the synthesizing means, there is an effect that a high-resolution distance image can be generated with a small amount of image memory in a short calculation time, and the extraction accuracy of an object candidate can be increased.

According to a sixth aspect of the present invention, in the object transfer device, the distance image generating means includes a low-resolution distance image generating means for generating a coarse-resolution distance image, and a step area portion of the object from the low-resolution distance image. A stepped region extracting means for extracting, a high-resolution distance image generating means for generating a high-resolution distance image for the stepped region, and a distance for combining two distance images having different resolutions of the low-resolution distance image and the high-resolution distance image An image processing apparatus configured with an image synthesizing unit is combined with an object transferring unit. In a short calculation time, a high-resolution range image is obtained, and an object candidate extraction time is short.
The extraction accuracy is high, the position of the object is accurately detected, and the transfer of the object can be performed quickly and accurately.

According to a seventh aspect of the present invention, there is provided an image processing apparatus, comprising: a distance image generating means for generating a distance image of a plurality of loaded objects; and extracting an uppermost surface area of the uppermost object from the distance image. A top-level surface region extracting unit, a plurality of object candidate combination hypothesis generating units that enumerate and generate a plurality of combinations of object candidates from the output of the top-level surface region extracting unit, and an object size database storing size data of the recognition target object. A hypothesis validity verification unit that verifies the validity of the object candidate combination hypothesis generated by the hypothesis generation unit using the information included in the two-dimensional reference pattern and the grayscale original image; Recognize based on hypotheses,
Information integrating means for integrating the recognized image with the distance information of each object obtained by the distance image generating means, and the arrangement of the target object is performed with respect to the top cross-sectional area extracted from the distance image. Since the position of the candidate object is detected from the hypothesis, it is possible to accurately detect the position of the randomly stacked target object.

An object transfer device according to an eighth aspect of the present invention is a distance transfer device for generating a distance image of a plurality of stacked objects, wherein the distance image generation unit generates a distance image of the plurality of stacked objects. An uppermost plane area extracting means for extracting, an object candidate combination hypothesis generating means for generating a plurality of combinations of object candidates as hypotheses from the output of the uppermost plane area extracting means, and an object size database for storing dimension data of the recognition target object A hypothesis validity verification unit that verifies the validity of the object candidate combination hypothesis generated by the hypothesis generation unit using information included in the two-dimensional reference pattern and the grayscale original image; Recognize based on high hypotheses,
An image processing device for outputting three-dimensional position information of an individual target object, comprising: an information integration unit for integrating the recognized image and distance information of each object obtained by the distance image generation unit; and an object transfer unit Because it was combined with
Even irregularly stacked target objects can be accurately grasped and transferred.

An image processing apparatus according to a ninth aspect of the present invention provides an object array reference database for storing data serving as a reference for an object array, and a hypothesis validity test for verifying the validity of the object candidate combination hypothesis generated by the hypothesis generation means. An object arrangement determining unit that compares the image recognized by the sex verification unit with the reference data to determine whether the arrangement is correct, and a warning that warns the operator of a determination result when the recognized image does not match the reference data Since the means is provided, if the stacking pattern of the recognized object is not the standard stacking pattern, a warning is issued to the user of the device or the manager of the host system to prevent the stacking from being performed correctly, thereby avoiding a dangerous state. be able to.

An object transfer device according to a tenth aspect of the present invention provides an object array reference database storing data serving as an object array reference, and a hypothesis for verifying the validity of the object candidate combination hypothesis generated by the hypothesis generation means. An object arrangement determining unit for comparing the image recognized by the validity verification unit with the reference data to determine whether or not the arrangement is correct; and warning an operator of a determination result when the recognized image does not match the reference data. Since the image processing apparatus having the warning generating means and the transfer means are combined, if the stacking pattern of the recognition object is not the standard stacking pattern, the stacking pattern can be correctly stacked with the apparatus user or the manager of the host system. It warns that it is not rare and avoids dangerous situations.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a configuration of a first embodiment of the present invention.

FIG. 2 is a flowchart showing a flow of an operation according to the first embodiment.

FIG. 3 is an image generated at each stage according to the first embodiment.

FIG. 4 is a block diagram showing a configuration of a second embodiment of the present invention.

FIG. 5 is a flowchart showing a flow of an operation according to the second embodiment.

FIG. 6 is an explanatory diagram of irradiation of a random dot pattern.

FIG. 7 is an explanatory diagram of block matching of a stereo image.

FIG. 8 is a block diagram showing a configuration of a third embodiment of the present invention.

FIG. 9 is a flowchart showing a flow of an operation according to the third embodiment.

FIG. 10 is an explanatory diagram of a situation in which a step region portion is extracted from a low-resolution distance image, and high-resolution stereo correspondence is performed only in a step region portion of a high-resolution distance image.

FIG. 11 is a block diagram showing a configuration of a fourth embodiment of the present invention.

FIG. 12 is a flowchart showing a flow of an operation according to the fourth embodiment.

FIG. 13 is a diagram showing each stage of an operation state of extracting an object candidate using the object candidate hypothesis.

FIG. 14 is a block diagram showing a configuration of a fifth embodiment of the present invention.

FIG. 15 is a flowchart showing a flow of an operation according to the fifth embodiment.

FIG. 16 is a diagram illustrating a processing procedure of a conventional image processing apparatus.
FIG.

FIG. 17 is a flowchart of the processing operation shown in FIG . 16;
is there.

FIG. 18 illustrates the principle of spatial coding using pattern light .
FIG.

FIG. 19 is a description of another conventional range image measurement system.
FIG.

FIG. 20 shows another conventional stereo-compatible search device for image processing.
FIG.

21 is a flowchart of another conventional image processing shown in FIG. 20;
It is.

[Explanation of symbols]

11 image input means, 12 distance image generation means, 13
Top-level surface area extraction means, 14 Top-level object candidate extraction means, 15 density original image storage means, 16 object size database, 17 two-dimensional reference pattern generation means, 18 object position detection means, 19 information integration means, 20 random texture pattern Light emitting means, 21 Image input means, 2
2 stereo image block correspondence means, 30 pattern projectors, 31 low resolution block correspondence means, 32 step area extraction means, 33 high resolution block correspondence means,
34 range image synthesis means, 64 object candidate combination hypothesis generation means, 65 hypothesis validity verification means, 69 object position detection means, 71 object array reference database, 72 object array determination means, 73 warning generation means.

Claims (10)

(57) [Claims] 1. A distance image generating means for generating a distance image of a plurality of stacked objects, an uppermost surface area extracting means for extracting an uppermost surface area of an uppermost object from the distance image, An uppermost-stage object candidate extracting means for individually separating and extracting an object from the surface region, an object size database for storing size data of the recognition target object, and a two-dimensional image of the object based on the size data of the object. A two-dimensional reference pattern generating means for generating a standard pattern image, a gray-scale original image storage means for storing a gray-scale original image input from a camera, and the two-dimensional reference pattern for the individually extracted top-level object candidate And object position detecting means for detecting the position of the object using the information of the grayscale original image, and the detection result of the object position obtained by each of the above means and the individual object obtained by the distance image generating means. An image processing apparatus comprising: information integration means for integrating distance information; and outputting three-dimensional position information of each target object. 2. A distance image generating means for generating a distance image of a plurality of stacked objects, an uppermost surface area extracting means for extracting an uppermost surface area of an uppermost object from the distance image, An uppermost-stage object candidate extracting means for individually separating and extracting an object from the surface region, an object size database for storing size data of the recognition target object, and a two-dimensional image of the object based on the size data of the object. A two-dimensional reference pattern generating means for generating a standard pattern image; a gray-scale original image storage means for storing a gray-scale original image input from a camera; And object position detecting means for detecting the position of the object using the information of the grayscale original image, and the detection result of the object position obtained by each of the above means and the individual object obtained by the distance image generating means. Information integration means for integrating the distance information, and an image processing apparatus for outputting three-dimensional position information of each target object, and an object transfer means for gripping and transferring the object, the output of the image processing apparatus object transfer device for transferring the object have groups Dzu the signal. 3. The distance image generating unit includes: a random texture projecting unit that projects a random texture pattern; a first image input unit and a second image input unit that input a stereo image; 2. The image processing apparatus according to claim 1, further comprising stereo image block associating means for associating the two images captured by the second image input means. 4. A distance image generating means, comprising: a random texture light projecting means for projecting a random texture pattern; a first image input means and a second image input means for inputting a stereo image; An image processing apparatus comprising stereo image block association means for associating two images captured by the second image input means and an object transfer means for gripping and transferring an object; 3. The object transfer device according to claim 2, wherein the object is transferred based on an output signal of the device. 5. A distance image generating means, comprising: a low resolution distance image generating means for generating a coarse resolution distance image; a step area extracting means for extracting a step area portion of an object from the low resolution distance image; And a distance image synthesizing unit for synthesizing two distance images having different resolutions of the low-resolution distance image and the high-resolution distance image. The image processing apparatus according to claim 1, wherein: 6. A distance image generating means, comprising: a low-resolution distance image generating means for generating a coarse-resolution distance image; a step area extracting means for extracting a step area portion of an object from the low-resolution distance image; A high-resolution distance image generating means for generating a high-resolution distance image, and an image processing apparatus comprising distance image synthesizing means for synthesizing two distance images having different resolutions of the low-resolution distance image and the high-resolution distance image. 3. The object transfer device according to claim 2, further comprising an object transfer means for holding and transferring the object, and transferring the object based on an output signal of the image processing device. 7. A distance image generating means for generating a distance image of a plurality of stacked objects, an uppermost surface area extracting means for extracting an uppermost surface area of an uppermost object from the distance image, Object candidate combination hypothesis generation means for generating a plurality of combinations of object candidates as hypotheses from the output of the surface region extraction means, an object size database for storing the size data of the recognition target object, included in the two-dimensional reference pattern and the grayscale original image Hypothesis validity verification means for verifying the validity of the object candidate combination hypothesis generated by the above-mentioned hypothesis generation means using the information generated by the hypothesis generation means, and performing recognition based on the hypothesis having the highest evaluation value among a plurality of hypotheses, and An image processing apparatus comprising: information integrating means for integrating the obtained image and distance information of each object obtained by the distance image generating means, and outputting three-dimensional position information of each target object. 8. A distance image generating means for generating a distance image of a plurality of loaded objects, an uppermost surface area extracting means for extracting an uppermost surface area of an object located at an uppermost position from the distance image, Object candidate combination hypothesis generation means for generating a plurality of combinations of object candidates as hypotheses from the output of the surface region extraction means, an object size database for storing the size data of the recognition target object, included in the two-dimensional reference pattern and the grayscale original image Hypothesis validity verification means for verifying the validity of the object candidate combination hypothesis generated by the above-mentioned hypothesis generation means using the information generated by the hypothesis generation means, and performing recognition based on the hypothesis having the highest evaluation value among a plurality of hypotheses, and An image processing device, comprising information integration means for integrating the obtained image and the distance information of each object obtained by the distance image generation means, and outputting three-dimensional position information of each target object; An object transfer device comprising a transfer means for holding and transferring an object, and transferring the object based on an output signal of the image processing device. 9. An object array reference database storing data serving as a reference of the object array, an image recognized by a hypothesis validity verification unit for verifying the validity of the object candidate combination hypothesis generated by the hypothesis generation unit, and An object arrangement determining means for comparing the data with the data to determine whether the arrangement is correct, and warning generating means for warning an operator of a result of the determination when the recognized image does not match the reference data. 8. The image processing device according to 7. 10. An object array reference database for storing data serving as a reference for the object array, an image recognized by a hypothesis validity verification unit for verifying the validity of the object candidate combination hypothesis generated by the hypothesis generation unit, An image processing apparatus having an object arrangement determining means for comparing the data with the data to determine whether or not the arrangement is correct, and a warning generating means for warning an operator of a result of the determination when the recognized image does not match the reference data, and grasping the object 9. The object transfer device according to claim 8, further comprising a transfer unit that transfers the object based on an output signal of the image processing device.