JP2021105862A - Three-dimensional data generation device for interpolating three-dimensional data and robot system - Google Patents

Abstract

To interpolate a deficient area of three-dimensional data with high accuracy with few man-hours.SOLUTION: A three-dimensional data generation device 10 includes: an object area identification unit 12 for identifying object areas of three-dimensional data and two-dimensional data acquired from a sensor 11; a deficient area identification unit 13 for identifying a deficient area 41 within an object area 40 of the three-dimensional data; a similar area identification unit 14 for identifying one or more similar areas 42 within the object area 40 of the two-dimensional data; a similar area identification unit 15 for identifying a similar corresponding area 43 of the three-dimensional data corresponding to the similar area 42 of the two-dimensional data; and a three-dimensional data interpolation unit 16 for interpolating the three-dimensional data of the deficient area 41 on the basis of the similar corresponding area 43 of the three-dimensional data.SELECTED DRAWING: Figure 1

Description

The present invention relates to a three-dimensional data generator and a robot system, and more particularly to a three-dimensional data generator and a robot system that interpolate three-dimensional data.

Various methods such as a stereo method and a TOF (time of flight) method are known as methods for generating three-dimensional data in the target space. The stereo method is a passive stereo method in which matching processing is performed between images acquired by a plurality of cameras and three-dimensional data is generated based on the parallax between the images, or slit light or pattern light is projected onto the subject to perform slit light or There is an active stereo method that generates three-dimensional data based on pattern light. In the TOF method, the direct method of irradiating the subject with the reference light and directly measuring the time until the reference light returns to generate three-dimensional data of the subject, or the reference light and the reflected light from the target space are used. There is an indirect method that detects the phase difference between the two and generates three-dimensional data of the subject. However, when the target space includes a strong reflection area or a dark color area, the stereo method cannot acquire features such as the shape and pattern of an object, slit light, and pattern light, and the TOF method requires so-called saturation, halation, and underexposure. Etc. may occur, resulting in a missing area of three-dimensional data. As a technique for interpolating the missing region of such three-dimensional data, the documents described later are known.

Patent Document 1 is a method of interpolating the distance data of a pixel whose distance measurement has not been performed in a distance image with reference to the distance data of a pixel whose distance measurement has been performed located in the vicinity of the pixel. A color image is also acquired for the image capture target of the distance image, and the weighting coefficient is calculated using the pixel color of the color image corresponding to the pixel of interest to be interpolated and the pixel color of the color image corresponding to the reference pixel. A method of calculating and interpolating the distance data by applying the calculated weighting coefficient to the distance data of the reference pixel is disclosed.

In Patent Document 2, model constituent points on an object are generated by measuring three-dimensional position data of model measurement points by stereo measurement processing, and three-dimensional shape data is generated as a set of the model constituent points, and then the data. It is inspected whether there are model constituent points in the left-right direction of the missing points, and if there are model constituent points in both the left-right directions, the data missing points are set as the interpolation target points, and the height data of the interpolation target points is used. Data is obtained by averaging the height data of the left and right model constituent points detected earlier, and by calculating the three-dimensional position of the interpolation target point from this height data and the position of the model measurement point and setting the model constituent points. It is disclosed that three-dimensional shape data without a defect is generated to construct a three-dimensional shape model.

In Patent Document 3, three-dimensional shape data obtained from multiple directions are synthesized in real time by using a plurality of cameras at the same time, and data loss portions caused by blind spots of the cameras are interpolated with each other to obtain an object to be measured. A three-dimensional shape measuring device for obtaining the overall three-dimensional shape data of the above is disclosed.

In Patent Document 4, pixels are assigned as data developed on a two-dimensional plane for a three-dimensional object in computer graphics, and it is determined whether or not the data is a missing pixel to which a pixel is not assigned, and the missing pixel is determined. The angle between the reference line in the normal direction with respect to each plane or each tangent plane including each of the plurality of peripheral pixels existing around the pixel and the angle with respect to the reference line in the normal direction with respect to the plane including the missing pixels or the tangent plane. It is disclosed that at least two peripheral pixels of a plurality of peripheral images are selected based on the difference, and data interpolation of missing pixels is performed based on the selected at least two peripheral pixels.

Japanese Unexamined Patent Publication No. 2000-230809 Japanese Unexamined Patent Publication No. 2001-227926 Japanese Unexamined Patent Publication No. 2001-324313 Japanese Unexamined Patent Publication No. 2016-119041

As another solution to the missing region of the three-dimensional data, a method of adjusting the exposure time, the aperture, the imaging mode, etc., and a method of adjusting the intensity of illumination or the posture of the object can be considered. However, since trial and error is required for optimization, a large amount of man-hours are required. There is also a method in which a human teaches the range of an object and the background to remove the background and estimates the three-dimensional structure of the object from a two-dimensional image, but when supervised learning is not used, the object and the background are judged. It is difficult to judge whether it is convex or concave, and human teaching is required. On the other hand, when supervised learning is used, a large amount of training data is required to generate a highly accurate learning model, and a large amount of computational resources are also required for inference.

Therefore, there is a demand for a technique for accurately interpolating a missing region of three-dimensional data with a small number of man-hours.

One aspect of the present disclosure is a three-dimensional data generator that acquires three-dimensional data and two-dimensional data of an object space including an object from a sensor, and specifies an object area of the acquired three-dimensional data and two-dimensional data. Target area identification part to specify, defect area identification part to specify the missing area in the object area of the three-dimensional data, and similar area identification to specify one or more similar areas in the object area of the two-dimensional data. 3D that interpolates the 3D data of the missing area based on the part, the similarity correspondence area identification part that specifies the similarity correspondence area of the 3D data corresponding to the similarity area of the 2D data, and the similarity correspondence area of the 3D data. Provided is a three-dimensional data generation device including a data interpolation unit.
Another aspect of the present disclosure is a robot system including the above-mentioned three-dimensional data generation device, which includes a robot control device that corrects a robot operation command based on the three-dimensional data interpolated by the three-dimensional data generation device. , Provides robot systems.

According to one aspect of the present disclosure, since the missing area in the object area of the three-dimensional data is interpolated in consideration of the object area of the two-dimensional data, the three-dimensional data is interpolated accurately with less man-hours as compared with the conventional case. can.

It is a block diagram which shows the schematic structure of the 3D data generation apparatus in one Embodiment. It is a figure which shows an example of 3D data and 2D data detected by a sensor. It is a figure which shows an example of 3D data and 2D data which specified the object area. It is a figure which shows an example of three-dimensional data which specified the missing area. It is a figure which shows an example of the 2D data which specified the similar region. It is a figure which shows an example of 3D data which specified the similarity correspondence area. It is a flowchart which shows the operation of the robot system equipped with the 3D data generation apparatus. It is a block diagram which shows an example of the robot system which incorporated the 3D data generation apparatus into a robot control apparatus.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In each drawing, the same or similar components are given the same or similar reference numerals. In addition, the embodiments described below do not limit the technical scope of the invention and the meaning of the terms described in the claims.

FIG. 1 shows a schematic configuration of the three-dimensional data generation device 10 in the present embodiment. The three-dimensional data generation device 10 may be a computer device (for example, a personal computer) provided with a processor such as a CPU (central processing unit), but an FPGA (field-programmable gate array) or an ASIC (application specific integrated circuit). It may be a personal computer device provided with other processors such as. The three-dimensional data generation device 10 acquires three-dimensional data and two-dimensional data of the object space including the object from the sensor 11, and considers the object area of the two-dimensional data and deletes the three-dimensional data in the object area. Interpolate the area. The object may be a work to be worked on by the robot. The three-dimensional data generation device 10 includes an object area identification unit 12, a missing area identification unit 13, a similar area identification unit 14, a similarity correspondence area identification unit 15, and a three-dimensional data interpolation unit 16. ..

FIG. 2 shows an example of three-dimensional data and two-dimensional data detected by the sensor 11. The sensor 11 may be, for example, a sensor such as a stereo camera or a TOF sensor. The sensor 11 detects both three-dimensional data and two-dimensional data, but may be a three-dimensional sensor that detects only three-dimensional data. In this case, a two-dimensional sensor such as a camera may be provided separately from the sensor 11, and the two-dimensional data may be detected from the two-dimensional sensor. However, in order to facilitate the subsequent processing, it is preferable that the angles of view of the target space match between the detected three-dimensional data and the detected two-dimensional data. The three-dimensional data may be, for example, point cloud data representing a three-dimensional position (for example, XYZ coordinates), or may be a distance image (depth image) representing a distance (or depth) from the sensor 11. In the case of point cloud data, it is preferable to convert it into a distance image in order to facilitate subsequent processing. Further, the two-dimensional data may be, for example, a grayscale image showing the light intensity from white to black, but may also be a color image showing the light intensity of the three primary colors (RGB) of light.

FIG. 3 shows an example of three-dimensional data and two-dimensional data in which the object region 40 is specified. The object area specifying unit 12 identifies the object area 40 of the three-dimensional data and the two-dimensional data acquired from the sensor 11. For example, the object area 40 and the background area other than the object area 40 may be specified according to the instruction of the user, but the object area 40 may be performed by performing matching processing, blob analysis, and processing combining these. Should be specified automatically.

In the matching process, for example, the data of the object (which may be two-dimensional or three-dimensional) is prepared in advance, and the acquired three-dimensional data and the acquired two are changed while changing the position and orientation of the data of the object prepared in advance. The object region 40 may be specified by performing a matching process with at least one of the dimensional data. When there are many missing areas in the three-dimensional data, the similarity between the data of the object prepared in advance and the acquired data becomes low. Therefore, the matching process is performed only with the two-dimensional data to identify the object area 40. The object corresponding area of the three-dimensional data corresponding to the specified object area 40 may be specified. As the evaluation function for calculating the similarity, for example, SAD (sum of absolute difference), SDD (sum of squared difference), NCC (normalized cross-correlation), ZNCC (zero-means normalized cross-correlation) and the like can be used.

In the blob analysis, for example, the object region 40 may be specified by applying various image processing filters to the acquired two-dimensional data and performing edge detection, morphology processing, and the like. Blob analysis is useful when the background area has few features, as edges in the background area may be detected erroneously. After the object area 40 is specified as described above, it is preferable to perform the subsequent processing only on the object area 40. This makes it possible to avoid secondary errors such as accidentally interpolating the missing area based on the three-dimensional data of the background area.

FIG. 4 shows an example of three-dimensional data in which the missing region 41 is specified. The defective region identification unit 13 identifies the defective region 41 within the object region 40 of the three-dimensional data. For example, the data in the background region (including the defective region) other than the object region 40 may be removed, or may be replaced with other data.

FIG. 5 shows an example of two-dimensional data in which the similar region 42 is specified. The similar area identification unit 14 identifies one or more similar areas 42 in the object area 40 of the two-dimensional data. The similar region 42 is defined as a region in which the distance (or depth) from the sensor 11 is the same or similar. For example, the similar region 42 may be specified based on the light intensity of the acquired two-dimensional data, or when the acquired two-dimensional data is a color image, the similar region 42 may be specified based on each light intensity of RGB. You may. However, even if the colors are the same or similar, the distance (or depth) from the sensor 11 may be different. Therefore, in addition to the acquired two-dimensional data (light intensity), the acquired three-dimensional data (distance) is added and similar. The region 42 may be specified. As a result, it is possible to avoid a secondary error such as interpolating the missing area based on the three-dimensional data of the similar area 42 specified by mistake.

As a method for identifying the similar region 42, for example, clustering, matching processing, blob analysis, and a combination thereof can be used. That is, the cluster area obtained by clustering the object area 40 may be specified as the similar area 42, or the matching area obtained by matching the object area 40 may be specified as the similar area 42. Alternatively, the analysis region obtained by blob-analyzing the object region 40 may be specified as the similar region 42.

In clustering, as the number of similar regions 42 increases (that is, as the number of clusters increases), the interpolation accuracy of the missing region 41 improves, so that even if the number of clusters is large, it can be executed with a relatively small number of computational resources. Hierarchical clustering is preferred. However, when the object is plain and has a simple shape (for example, a rectangular parallelepiped shape), the features of the object are small and the number of clusters is small. May be used.

As a non-hierarchical clustering method, for example, a k-means method, an x-means method, or the like can be used. In the k-means method, the number of clusters k is determined in advance, and k initial average values of data constituting the cluster area (multidimensional data such as light intensity and distance; the same applies hereinafter) are prepared. The cluster area to which each data belongs is determined by selecting the closest initial average value for each acquired data. Next, the average value is updated based on the data constituting each cluster area, and the cluster area to which each data belongs is updated by selecting the closest average value for each data. This process is repeated until the average value is not updated or the maximum number of loops is reached to determine the final cluster area. FIG. 5 shows an example in which the number of clusters k is set to 7 in one object region 40. Further, the number of clusters may be estimated automatically by using the x-means method, or the number of clusters may be evaluated by the elbow method, silhouette analysis, or the like to determine the final cluster area.

In the matching process, for example, a plurality of sample data (multidimensional data such as light intensity and distance may be used; the same applies hereinafter) of data constituting the similar region 42 are prepared, and the sample data prepared in advance and each acquired data are prepared. It is advisable to specify the matching area by performing a matching process with and. As the evaluation function for calculating the similarity, the above-mentioned SAD, SDD, NCC, ZNCC and the like can be used. Further, the final similar region 42 may be specified by adding the matching region to the cluster region.

In the blob analysis, for example, various image processing filters are applied to both the object area 40 of the acquired two-dimensional data and the object area 40 of the acquired three-dimensional data, and edge detection, morphology processing, and the like are performed. The analysis area may be specified by adding the processing results of both. Further, the final similar region 42 may be specified by adding the analysis region to at least one of the cluster region and the matching region.

FIG. 6 shows an example of three-dimensional data in which the similarity corresponding region 43 is specified. The similarity correspondence area identification unit 15 identifies the similarity correspondence area 43 of the three-dimensional data corresponding to the similarity area 42 of the two-dimensional data. Then, the three-dimensional data interpolation unit 16 interpolates the three-dimensional data of the missing region 41 based on the similarity corresponding region 43 of the three-dimensional data. For example, the three-dimensional data of the pixel of interest may be interpolated with the three-dimensional data of the similar corresponding region 43 to which the pixel of interest of the missing region 41 belongs. At this time, the three-dimensional data of the reference pixel of the reference pixel of the similarity correspondence region 43 closest to the attention pixel may be interpolated with the three-dimensional data of the attention pixel, or a plurality of similarity correspondence regions 43 to which the attention pixel belongs may be interpolated. The three-dimensional data may be averaged and the three-dimensional data of the pixel of interest may be interpolated.

However, there is a case where the similar corresponding region 43 to which the attention pixel of the missing region 41 belongs does not exist, and in this case, the three-dimensional data of the attention pixel cannot be interpolated. However, when a plurality of objects exist in the target space, the three-dimensional data interpolation unit 16 uses the three-dimensional data of the similarity corresponding region 43 in another target region 40 similar to the target region 40 to provide three-dimensional data of the pixel of interest. Data may be interpolated. According to the above processing, the missing area 41 of the three-dimensional data in the object area 40 is interpolated in consideration of the two-dimensional data (light intensity, distance, etc.) in the object area 40. It is possible to perform interpolation with high accuracy with a small number of man-hours.

With reference to FIG. 1 again, the present invention can also be applied to a robot system 1 provided with a three-dimensional data generation device 10. The robot system 1 may include a robot control device 20 that corrects an operation command of at least one of the robot 30 and the tool 31 based on the three-dimensional data interpolated by the three-dimensional data generation device. There may be a plurality of robot control devices 20 instead of one, and in the case of a plurality of robot control devices 20, they are connected to the three-dimensional data generation device 10 in a many-to-one manner via wireless or wired. That is, the robot system 1 may be a server system or a cloud system. The robot control device 20 is a computer device including a processor such as a CPU (central processing unit), and generates an operation command of at least one of the robot 30 and the tool 31 according to the operation program and the storage unit 21 that stores the operation program. The operation control unit 22 is provided. The motion control unit 22 corrects at least one motion command of the robot 30 and the tool 31 based on the interpolated three-dimensional data. The robot control device 20 includes a robot drive unit 23 that drives the robot drive motor 32 and a tool drive unit 24 that drives the tool drive motor 33 based on the corrected operation command. Similar to the robot control device 20, there may be a plurality of robots 30 and tools 31 instead of one.

FIG. 7 shows the operation of the robot system 1 provided with the three-dimensional data generation device 10. In step S10, the object region 40 of the three-dimensional data and the two-dimensional data acquired from the sensor 11 is specified by performing matching processing, blob analysis, processing combining these, and the like. In step S11, the missing area 41 in the object area 40 of the three-dimensional data is specified by removing the background area other than the object area 40 or replacing it with another data.

In step S12, by performing clustering, matching processing, blob analysis, and processing combining these based on multidimensional data such as light intensity and distance, one or one of the two-dimensional data is performed in the object region 40. Identify a plurality of similar regions 42. In step S13, the similarity corresponding region 43 of the three-dimensional data corresponding to the similarity region 42 of the two-dimensional data is specified.

In step S14, the three-dimensional data of the missing region 41 is interpolated based on the similarity corresponding region 43. At this time, the three-dimensional data of the reference pixel of the similarity corresponding region 43 closest to the attention pixel of the missing region 41 may be interpolated with the three-dimensional data of the attention pixel, or the attention pixel of the missing region 41 belongs to the three-dimensional data. The three-dimensional data of the similar corresponding region 43 may be averaged and the three-dimensional data of the pixel of interest may be interpolated. Further, when a plurality of objects exist in the target space, the three-dimensional data of the pixel of interest may be interpolated with the three-dimensional data of the similarity corresponding region 43 in another object region 40 similar to the target region 40. good. In step S15, the operation command of at least one of the robot 30 and the tool 31 is corrected based on the interpolated three-dimensional data.

FIG. 8 shows an example of a robot system 1 in which the three-dimensional data generation device 10 is incorporated in the robot control device 20. In the robot system 1 of this example, at least one of the components of the three-dimensional data generation device 10 is incorporated in the robot control device 20. That is, the robot system 1 may be a stand-alone system.

According to the above embodiment, since the missing area 41 of the three-dimensional data in the object area 40 is interpolated in consideration of the two-dimensional data in the object area 40, the three-dimensional area can be accurately three-dimensionalized with less man-hours as compared with the conventional case. The data can be interpolated.

The program executed by the processor described above may be recorded and provided on a computer-readable non-temporary recording medium such as a CD-ROM, or may be provided via a wired or wireless LAN (wide area network) or. It may be distributed and provided from a server device on a LAN (local area network).

Although various embodiments have been described herein, it is recognized that the present invention is not limited to the above-described embodiments, and various modifications can be made within the scope of the claims below. I want to be.

1 Robot system 10 Three-dimensional data generator 11 Sensor 12 Object area identification unit 13 Missing area identification unit 14 Similar area identification unit 15 Similar correspondence area identification unit 16 Three-dimensional data interpolation unit 20 Robot control device 21 Storage unit 22 Motion control unit 23 Robot drive unit 24 Tool drive unit 30 Robot 31 Tool 32 Robot drive motor 33 Tool drive motor 40 Object area 41 Missing area 42 Similar area 43 Similar correspondence area

Claims (10)

A three-dimensional data generator that acquires three-dimensional data and two-dimensional data of the target space including an object from a sensor.
An object area specifying unit that specifies the object area of the acquired three-dimensional data and the two-dimensional data, and
A defect region identification portion that identifies a defect region in the object region of the three-dimensional data, and a defect region identification portion.
A similar area identification unit that specifies one or more similar areas in the object area of the two-dimensional data,
A similarity corresponding area specifying unit that specifies a similar corresponding area of the three-dimensional data corresponding to the similar area of the two-dimensional data, and a similar corresponding area specifying unit.
A three-dimensional data interpolation unit that interpolates the three-dimensional data of the missing region based on the similarity corresponding region of the three-dimensional data, and a three-dimensional data interpolation unit.
A three-dimensional data generator equipped with. The three-dimensional data according to claim 1, wherein the two-dimensional data includes light intensities of the three primary colors of light, and the similar region specifying portion identifies the similar region based on the light intensities of the three primary colors in the object region. Generator. The three-dimensional data generation device according to claim 1 or 2, wherein the similar region specifying unit identifies the similar region by adding the three-dimensional data in the object region in addition to the two-dimensional data. The three-dimensional data generation device according to any one of claims 1 to 3, wherein the similar region specifying unit specifies a cluster region obtained by clustering the object region as the similar region. The three-dimensional data generation device according to any one of claims 1 to 4, wherein the similar region specifying unit specifies a matching region obtained by matching the object region as the similar region. The three-dimensional data generation device according to any one of claims 1 to 5, wherein the similar region specifying unit identifies an analysis region obtained by blob-analyzing the object region as the similar region. From claim 1, the three-dimensional data interpolation unit averages a plurality of the three-dimensional data of the similar corresponding region to which the attention pixel of the missing region belongs and interpolates the three-dimensional data of the attention pixel. The three-dimensional data generator according to any one of 6. When a plurality of the objects exist in the target space, the three-dimensional data interpolation unit uses the missing area based on the three-dimensional data of the similar corresponding area in another object area similar to the object area. The three-dimensional data generation device according to any one of claims 1 to 7, which interpolates the three-dimensional data of the above. A robot system including the three-dimensional data generation device according to any one of claims 1 to 8, and corrects a robot operation command based on the three-dimensional data interpolated by the three-dimensional data generation device. A robot system equipped with a robot control device. The robot system according to claim 9, wherein the three-dimensional data generation device is incorporated in the robot control device.

Images