WO2021117117A1

WO2021117117A1 - Model generation device and picking robot

Info

Publication number: WO2021117117A1
Application number: PCT/JP2019/048190
Authority: WO
Inventors: 繁伸浅田; 亮輔川西
Original assignee: 三菱電機株式会社
Priority date: 2019-12-10
Filing date: 2019-12-10
Publication date: 2021-06-17
Also published as: JP7162760B2; JPWO2021117117A1; CN114766037A; DE112019007961T5

Abstract

This invention comprises: a sensor (101) that acquires scene information including information about a subject workpiece for which a model is to be generated; a workpiece information acquisition unit (102) that acquires workpiece information, i.e. the information about the subject workpiece, from the scene information; a standard model determination unit (103) that determines a standard model, i.e. a model serving as a standard for expressing an irregular shape, on the basis of the scene information and the workpiece information; a deformation constraint condition determination unit (104) that determines a deformation constraint condition, i.e. a condition for deformation of the standard model, on the basis of the standard model and results obtained by analyzing a workpiece shape using the scene information and the workpiece information; and an amorphous model generation unit (106) that, on the basis of the deformation constraint condition, determines a deformation parameter for the standard model within a range of the deformation constraint condition, and deforms the standard model in accordance with the deformation parameter.

Description

Model generator and picking robot

The present invention relates to a model generator and a picking robot that generate a model of a target work.

Conventionally, there is a device that can be put in a returnable box or the like and can grip a work whose orientation is not aligned in a certain direction. In Patent Document 1, a model of the target work is input, image data is generated while simulating the scene of the target work, the gripping position is estimated for the image data, and the target work is taken out based on the estimation result. The technology of a robot simulation device that simulates a process is disclosed. When it is difficult to take out the target work, the robot simulation device described in Patent Document 1 adjusts the parameters of the simulation related to the robot movement, and picking is the ratio of the number of successful grips to the number of trials of the gripping movement of the target work. It is improving the success rate.

JP-A-2018-144158

In a situation where a robot picks a work in a bulk pile, so-called bulk picking, it is first necessary to recognize the work to be picked and know the gripping position. In order to achieve a high picking success rate in bulk picking, it is necessary to adjust the recognition parameters of the object recognition method.

However, in the robot simulation device described in Patent Document 1, the work to which the simulation can be applied is only a standard work in which a CAD (Computer Aided Design) model of an industrial part exists. Therefore, the robot simulation apparatus described in Patent Document 1 has a problem that the simulation cannot be applied to a work such as food in which the shape is irregular depending on the individual and the CAD model does not exist.

The present invention has been made in view of the above, and an object of the present invention is to obtain a model generator capable of generating a model of a target work that has an irregular shape and is applicable to simulation.

In order to solve the above-mentioned problems and achieve the object, the model generator of the present invention is information on the target work from the sensor that acquires the scene information including the information on the target work for generating the model and the scene information. A work information acquisition unit that acquires work information, a reference model determination unit that determines a reference model that is a reference model for expressing irregular shapes based on scene information and work information, and scene information and work information. Based on the result of analyzing the work shape using and the reference model, the deformation constraint condition determination unit that determines the deformation constraint condition that is the condition for deforming the reference model, and the deformation of the reference model based on the deformation constraint condition. It is characterized by including an indefinite model generation unit that determines parameters within a range of deformation constraint conditions and deforms a reference model according to deformation parameters.

According to the present invention, the model generator has an effect that it has an irregular shape and can generate a model of a target work applicable to simulation.

Block diagram showing a configuration example of the model generator according to the first embodiment A first flowchart showing the operation of the model generator according to the first embodiment. A second flowchart showing the operation of the model generator according to the first embodiment. The figure which shows the example of the case where the processing circuit included in the model generator which concerns on Embodiment 1 is configured by a processor and a memory The figure which shows the example in the case of configuring the processing circuit provided in the model generation apparatus which concerns on Embodiment 1 with dedicated hardware. Block diagram showing a configuration example of the reference model determination unit according to the second embodiment A flowchart showing the operation of the reference model determination unit according to the second embodiment. Block diagram showing a configuration example of the picking robot according to the third embodiment Block diagram showing a configuration example of the simulation condition setting unit according to the third embodiment Block diagram showing a configuration example of the data set generation unit according to the third embodiment Block diagram showing a configuration example of the parameter adjustment unit according to the third embodiment Block diagram showing a configuration example of the gripping parameter adjusting unit according to the third embodiment A flowchart showing the operation of the gripping parameter adjusting unit according to the third embodiment. Block diagram showing a configuration example of the recognition parameter adjustment unit according to the third embodiment A flowchart showing the operation of the recognition parameter adjusting unit according to the third embodiment.

Hereinafter, the model generator and the picking robot according to the embodiment of the present invention will be described in detail with reference to the drawings. The present invention is not limited to this embodiment.

Embodiment 1.
FIG. 1 is a block diagram showing a configuration example of the model generation device 10 according to the first embodiment of the present invention. The model generation device 10 generates a model of a target work having an irregular shape and applicable to simulation. The model of the target work generated by the model generation device 10 may be a three-dimensional target work model or a two-dimensional target work model. Hereinafter, a method in which the model generation device 10 generates a three-dimensional model of the target work will be specifically described. In addition, the model of the target work may be simply referred to as a model. When generating a two-dimensional model, the model generation device 10 can be realized by dimensional compression such as projecting a three-dimensional model method onto a plane. The target work that can generate a model with the model generation device 10 is not limited to the amorphous work that is a target work having irregular shapes, but also includes a fixed shape work that is a target work having individual shapes.

As shown in FIG. 1, the model generation device 10 includes a sensor 101, a work information acquisition unit 102, a reference model determination unit 103, a deformation constraint condition determination unit 104, a storage unit 105, and an amorphous model generation unit 106. And. Details of each configuration will be described together with the operation of the model generator 10. FIG. 2 is a first flowchart showing the operation of the model generation device 10 according to the first embodiment. The flowchart of FIG. 2 shows the operation from the sensor 101 to the deformation constraint condition determination unit 104. FIG. 3 is a second flowchart showing the operation of the model generation device 10 according to the first embodiment. The flowchart of FIG. 3 shows the operation of the amorphous model generation unit 106.

The sensor 101 measures the target work for which the model is to be generated in the present embodiment (step S101), and acquires the scene information of the target work. The scene information is information including information such as the background of the target work as well as the work information which is the information of the target work for which the model is to be generated. The work information is information representing the shape, color, etc. of the target work. The sensor 101 includes, for example, a 2D (Dimensions) camera, a 3D sensor, and the like. The sensor 101 may acquire 2D data as a grayscale image or a color image using a 2D camera, or may acquire 3D data as a 3D point cloud or a distance image using a 3D sensor. The scene information data may be measurement data or image data. In the following description, 2D may be referred to as two-dimensional, and 3D may be referred to as three-dimensional.

The work information acquisition unit 102 acquires work information, which is information on the target work to be model-generated, from the scene information acquired by the sensor 101 (step S102). The work information acquisition unit 102 may acquire information such as a size range and a color indicated by RGB (Red Green Blue) from the data acquired by the sensor 101. Regarding the method of acquiring the work information, for example, one target work is arranged on the rotation stage of uniaxial rotation, and the sensor 101 measures while rotating the target work on the rotation stage. The data does not have to be the entire circumference of the target work, and the measurement by the sensor 101 may be performed only once. The work information acquisition unit 102 deletes the background and the like unnecessary for model generation from the obtained data, and acquires the data of only the target work. The work information acquisition unit 102 calculates the size range of the target work, the average color information, and the like based on the obtained data of the target work. The size range of the target work is, for example, a range specified by each of the X coordinate, the Y coordinate, and the Z coordinate of the three-dimensional point cloud.

The reference model determination unit 103 determines a reference model, which is a reference model for expressing an irregular shape, based on the scene information obtained by the sensor 101 and the work information obtained by the work information acquisition unit 102. (Step S103). The reference model determined by the reference model determination unit 103 may be a primitive shape, for example, a geometric shape such as a rectangular parallelepiped, a sphere, or a triangular pyramid, a model created by a user, geometric shapes, or a user-created model. It may be a combination of the models that have been created. The method of expressing the model may be a set of points having information on three-dimensional coordinates, that is, a three-dimensional point cloud, a mesh model, a numerical value capable of expressing a geometric shape, or a combination thereof. The expression method of the reference model is not limited to the description content of the present embodiment, and other expression methods may be used. Here, there are two possible methods for determining the reference model: automatic determination and manual determination.

In the method of determining the reference model by automatic determination, for example, the reference model determination unit 103 selects one scene information from a plurality of scene information obtained by the sensor 101, and is an object represented by the work information included in the scene information. Calculate the position of the center of gravity of the work. The reference model determination unit 103 uses the calculated center of gravity position as the center point of the reference model, and generates several different types of reference models based on the center point. The reference model determination unit 103 determines the size of the reference model to be generated based on the work information obtained by the work information acquisition unit 102. The reference model determination unit 103 calculates the distribution of the normals of the measurement data and the reference model, and automatically uses the reference model closest to the normal distribution of the measurement data as the reference model for expressing the irregular shape of the target work. You may decide with. The method of determining the reference model by such automatic determination can be expressed by the following equation (1). That is, the reference model determination unit 103 executes the following equation (1) in determining the reference model by automatic determination. In the equation (1), M is the measurement data, P is the reference model, g (M) is the normal distribution of the measurement data, and g (P) is the normal distribution of the reference model.

In the method of determining the reference model by manual determination, the user may determine the geometric shape most similar to the target work.

The deformation constraint condition determination unit 104 determines the deformation constraint condition, which is a condition for deforming the reference model, based on the result of analyzing the work shape using the scene information and the work information and the reference model (step S104). The deformation constraint condition determination unit 104 stores the determined deformation constraint condition in the storage unit 105. Here, in the present embodiment, the deformation constraint condition determination unit 104 defines how to deform the reference model as parameters. The deformation constraint condition determination unit 104 sets the deformation parameters to different values for each model in order to make the shapes of the individual models uneven. In order to deform the reference model and reproduce the characteristics of the shape close to the target work, it is necessary to set constraints on the degree of deformation by appropriately determining the range of values of each deformation parameter. Therefore, the deformation constraint condition determination unit 104 determines the range of possible values of the deformation parameter.

As the deformation parameters representing the shape of the target work, for example, the size of the target work, the amount of deformation of the reference model when the reference model is deformed, the curvature, the frequency of appearance of irregularities, the appearance range of irregularities, etc. can be considered. The amount of deformation is, for example, the amount of movement of mesh vertices in the case of a mesh model. That is, the deformation constraint condition includes at least one of the frequency of appearance of unevenness of the target work, the appearance range of unevenness of the target work, and the curvature as numerical conditions. The deformation constraint condition determination unit 104 can improve the reproducibility of the model by using numerical conditions such as the appearance frequency of irregularities.

The deformation constraint condition determination unit 104 determines the deformation constraint condition based on the scene information obtained by the sensor 101, the work information obtained by the work information acquisition unit 102, and the reference model determined by the reference model determination unit 103. To do. The deformation constraint condition determination unit 104 constrains, for example, the movement amount of the mesh vertices of the reference model by assuming that the vector consisting of the mesh vertices of the reference model and the center point of the reference model is the movement range of the mesh vertices of the reference model. Add conditions. As a result, the deformation constraint condition determination unit 104 can prevent the failure of model generation such as the intersection of other meshes or the surface of the mesh being turned inside out when the mesh vertices are moved. The deformation constraint condition determination unit 104 may determine the size range of the target work based on the work information obtained by the work information acquisition unit 102. The deformation constraint condition determination unit 104 estimates the appearance frequency of irregularities based on the scene information obtained by the sensor 101, and if the distribution of the normal estimation is within a predetermined threshold value, the appearance of irregularities appears. The frequency may be set as one of the deformation parameters, and the range of the appearance frequency of the unevenness may be determined.

The deformation constraint condition determination unit 104 may extract and use the contour of the work for analyzing the shape of the work. That is, the work shape of the target work analyzed by the deformation constraint condition determination unit 104 includes the contour of the target work. As a method of extracting the contour, for example, the deformation constraint condition determination unit 104 sets an arbitrary cross section for the measurement data of the work, and searches for the measurement data included within a certain distance from the set cross section. Can be considered. The contour is not limited to one with respect to the measurement data of the work. For example, in the method described above, a contour can be obtained for each position and posture in which the cross section is set. As a method of analyzing the work contour, it is conceivable that the deformation constraint condition determining unit 104, for example, performs curve fitting on the work contour and determines the range of the appearance frequency of unevenness based on the result of the curve fitting. The deformation constraint condition determination unit 104 can easily determine the deformation constraint condition according to the irregularity of the target work by analyzing the work contour information of the target work. As a result, the model generator 10 can improve the reproducibility of the model.

The deformation constraint condition determination unit 104 may automatically acquire the shape feature by a learning method using a neural network instead of using a shape feature designed manually such as a curve by curvature or polynomial approximation. The deformation constraint condition determination unit 104 learns, for example, a neural network that classifies the target work and the work other than the target work by using the data obtained by imaging the normal distribution obtained from the measurement data of the target work as input information. , It is conceivable to use the feature map of the intermediate layer of the neural network after learning as a shape feature. As described above, the deformation constraint condition determination unit 104 may use the feature map of the intermediate layer of the neural network that has learned the shape of the target work as the deformation parameter. Since the model generator 10 can acquire deformation parameters suitable for the target work by machine learning, it is possible to accurately reproduce a complicated work shape. As a result, it can be expected that the deformation constraint condition determination unit 104 can obtain deformation constraint conditions that are more versatile and have high expressive ability than manually designing the shape features. The configuration of the neural network is not limited, and the learning target is not limited to the classification problem. However, it is desirable to have a network configuration and learning target in which the feature map of the intermediate layer well represents the features of the work shape.

The storage unit 105 stores the deformation constraint condition which is the range of the deformation parameter determined by the deformation constraint condition determination unit 104.

The irregular model generation unit 106 reads out the deformation constraint condition from the storage unit 105, determines the deformation parameter of the reference model within the range of the deformation constraint condition based on the read deformation constraint condition (step S105), and follows the deformation parameter. The reference model is transformed (step S106). As described above, the model generation device 10 is supposed to generate a plurality of amorphous models. Therefore, the amorphous model generation unit 106 accepts the user to specify the number of models to be generated. When the amorphous model generation unit 106 has not generated the number of models specified by the user (step S107: No), it returns to step S105 and repeats the above-described operation. When the amorphous model generation unit 106 generates the number of models specified by the user (step S107: Yes), the operation ends.

Specifically, the amorphous model generation unit 106 determines the deformation parameters based on the deformation constraint conditions determined by the deformation constraint condition determination unit 104. For example, when the output model is a mesh model, the irregular model generation unit 106 searches for the vector with the shortest distance in each measurement data in the amount of movement of the mesh of the reference model, and lowers the measurement data to the vector. The intersection of the line and the vector may be determined as the destination of the mesh vertices of the reference model. Since the amorphous model generation unit 106 is represented by the number of meshes of the reference model to be used in terms of curvature, the number of meshes of the reference model is increased in the measurement data where the density of the point cloud is high. The number of meshes may be determined. The irregular model generation unit 106 may estimate the normal line with respect to the measurement data and determine the appearance frequency of the unevenness based on the distribution of the normal line in the appearance frequency of the unevenness. This is because it is considered that the wider the range of the normal distribution, the higher the frequency of appearance of unevenness. In the irregular appearance range, the irregular model generation unit 106 may estimate the uneven portion based on the normal distribution estimated for the measurement data, and calculate the uneven range at the estimated portion.

When calculating the value of each deformation parameter, the amorphous model generation unit 106 randomly changes the value so that each model has a different shape. However, it is necessary to make the parameter value after the change within the range of the deformation constraint condition. After determining the deformation parameters, the amorphous model generation unit 106 deforms the reference model based on the determined deformation parameters.

The amorphous model generation unit 106 may use the contour information of the target work extracted from the scene information when moving the mesh vertices. For example, the amorphous model generation unit 106 replaces all or part of the contour of the reference model with the contour information of the target work, and sets the movement destination of the mesh vertices to the three-dimensional point on the contour having the shortest distance from the mesh vertices. To do. The contour information of the target work is, for example, a group of three-dimensional points constituting the contour. As the contour information to be replaced, for example, a work contour extracted from a different work is randomly selected, and the selected work contour is not used as it is, but is enlarged in the length direction of the unevenness or the contour within the range of the deformation constraint condition. It is conceivable to use the contour information generated by synthesizing different work contours, or to generate and use different contour information by performing scale conversion to reduce the scale.

That is, the amorphous model generation unit 106 may use the work contour generated by synthesizing two or more different work contours as the work contour information. As a result, the amorphous model generation unit 106 can increase the variation in the shape of the generated model by synthesizing the contour information of the target work, that is, the actual work, and generating a new contour. Further, the amorphous model generation unit 106 may use the work contour generated by enlarging or reducing the scale of the work contour as the information of the work contour. As a result, the amorphous model generation unit 106 can increase the variation of the shape of the generated model by further increasing the variation of the contour to be used. Further, the amorphous model generation unit 106 can be expected to generate a more natural work model by utilizing the work contour information actually acquired from the work when determining the deformation amount of the reference model. As a result, the amorphous model generation unit 106 can generate a model more like the target work by using the information of the work contour of the target work, that is, the actual work.

In the model generation device 10, the amorphous model generation unit 106 may be replaced by a model generation method using a neural network. The neural network is configured to input at least one of scene information, work information, deformation constraint conditions, and deformation parameters, and output a model of the target work in a defined expression method. For example, in the case of a 2D model such as an image, the neural network may be configured to output a contour image by inputting a random initial value. The value to be input may be one or a vector having a plurality of numerical values. In such a case, the correct answer data of the input numerical group and the contour model of the work associated with each numerical group are prepared in advance as the training data set of the neural network, and their transformations are learned. This gives a trained neural network.

Regarding the neural network, for example, when it is desired to output a three-dimensional model, a method such as configuring the network so that the output part of the network corresponds to the voxel space can be considered. As a framework for realizing such a function, GAN (Generative Adversarial Networks) and the like have been proposed. The above-mentioned network configuration, contents of the learning data set, and the like are examples, and the present invention is not limited to these, and other network configurations, the contents of the learning data set, and the like may be used. The model generation device 10 can be expected to be more versatile than a manually constructed model generation algorithm by generating a model with high expressive power by a neural network, and the applicable range of the work can be expanded.

Next, the hardware configuration of the model generator 10 will be described. In the model generator 10, the sensor 101 is a measuring instrument such as a camera or a laser. The storage unit 105 is a memory. The work information acquisition unit 102, the reference model determination unit 103, the deformation constraint condition determination unit 104, and the amorphous model generation unit 106 are realized by a processing circuit. The processing circuit may be a processor and memory for executing a program stored in the memory, or may be dedicated hardware.

FIG. 4 is a diagram showing an example in which the processing circuit included in the model generation device 10 according to the first embodiment is configured by a processor and a memory. When the processing circuit is composed of the processor 91 and the memory 92, each function of the processing circuit of the model generator 10 is realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in the memory 92. In the processing circuit, each function is realized by the processor 91 reading and executing the program stored in the memory 92. That is, the processing circuit includes a memory 92 for storing a program in which the processing of the model generation device 10 is to be executed as a result. It can also be said that these programs cause a computer to execute the procedures and methods of the model generator 10.

Here, the processor 91 may be a CPU (Central Processing Unit), a processing device, an arithmetic unit, a microprocessor, a microcomputer, a DSP (Digital Signal Processor), or the like. Further, the memory 92 includes, for example, non-volatile or volatile such as RAM (Random Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Program ROM), and EPROM (registered trademark) (Electrical EPROM). This includes semiconductor memories, magnetic disks, flexible disks, optical disks, compact disks, mini disks, DVDs (Digital Versailles Disc), and the like.

FIG. 5 is a diagram showing an example in which the processing circuit included in the model generation device 10 according to the first embodiment is configured by dedicated hardware. When the processing circuit is composed of dedicated hardware, the processing circuit 93 shown in FIG. 5 includes, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), and the like. FPGA (Field Processor Gate Array) or a combination of these is applicable. Each function of the model generator 10 may be realized by the processing circuit 93 for each function, or each function may be collectively realized by the processing circuit 93.

Note that some of the functions of the model generator 10 may be realized by dedicated hardware, and some may be realized by software or firmware. As described above, the processing circuit can realize each of the above-mentioned functions by the dedicated hardware, software, firmware, or a combination thereof.

As described above, according to the present embodiment, the model generation device 10 measures the target work with the sensor 101, and determines the reference model from the information obtained by the measurement with the sensor 101. The model generation device 10 determines the deformation constraint conditions for the reference model, determines the deformation parameters of the reference model within the range of the deformation constraint conditions, and deforms the reference model according to the deformation parameters. As a result, the model generation device 10 can generate a model even if the target work is an amorphous work. Therefore, the model generator 10 can be used in a food factory or the like that frequently handles amorphous workpieces. By utilizing the model generated by the model generator 10 by simulation, it is possible to reduce the period and cost required for on-site introduction and verification of the operation of the industrial robot for the purpose of labor saving.

According to the present embodiment, the model generation device 10 acquires some measurement data of the target work and deforms the reference model based on the deformation constraint condition of the target work determined based on the analysis result of the measurement data. Therefore, a model can be generated even if the target work is an indefinite work. The model generation device 10 can improve the reproducibility of the model by analyzing numerical conditions such as the appearance frequency of irregularities and contour information of the target work. Further, the model generation device 10 can generate a model more like the target work by using the contour information of the target work, that is, the actual work. The model generation device 10 can increase the variation in the shape of the model to be generated by synthesizing the contour information of the target work, that is, the actual work and generating a new contour. The model generation device 10 can increase the variation of the shape of the model to be generated by further increasing the variation of the contour to be used. Further, since the model generation device 10 can acquire deformation parameters suitable for the target work by machine learning, it is possible to accurately reproduce a complicated work shape. Since the model generation device 10 generates a model with high expressive ability in the model generation by the neural network, it can be expected to be more versatile than the manually constructed model generation algorithm, and the applicable range of the work can be expanded.

Embodiment 2.
In the second embodiment, a specific configuration and operation of the reference model determination unit 103 included in the model generation device 10 will be described.

FIG. 6 is a block diagram showing a configuration example of the reference model determination unit 103 according to the second embodiment. As shown in FIG. 6, the reference model determination unit 103 includes an individual model determination unit 201, a registration unit 202, an average shape generation unit 203, a shape element division unit 204, an individual difference category determination unit 205, and a work. It includes a spindle determination unit 206, a primitive collation unit 207, and a fitting model determination unit 208. Details of each configuration will be described together with the operation of the reference model determination unit 103. FIG. 7 is a flowchart showing the operation of the reference model determination unit 103 according to the second embodiment. The flowchart of FIG. 7 shows the details of the operation of the reference model determination unit 103, that is, the operation of step S103 in the flowchart of FIG.

The individual model determination unit 201 models each work information acquired by the work information acquisition unit 102, and determines an individual model in which each work information is modeled (step S201). The individual model is a model of each of the plurality of work information acquired by the work information acquisition unit 102. Specifically, the individual model determination unit 201 deletes the background unnecessary for model generation from the data obtained by the sensor 101, and extracts only the data of the target work. The individual model determination unit 201 may generate a model by deforming the geometric shape with respect to the obtained target work data, or may determine the obtained target work data as an individual model as it is. Good. When generating the model, the individual model determination unit 201 automatically determines the geometric shape most similar to the measurement data of the target work based on the normal distribution, as in the case of determining the reference model described in the first embodiment. Alternatively, the user may manually determine the geometric shape that most closely resembles the target work.

The registration unit 202 automatically integrates a plurality of individual models determined by the individual model determination unit 201 (step S202). Integration is, for example, the alignment of multiple individual models. The alignment method to be used may be ICP (Iterative Closest Points) or a feature point-based alignment method.

The average shape generation unit 203 generates an average shape model of the target work based on the integration result by the registration unit 202 (step S203). The average shape model is a model in which the features of a plurality of individual models integrated by the registration unit 202 are averaged. For example, when the output model is a mesh model, the average shape generation unit 203 calculates the average value of the mesh vertices and the average value of the normals from each individual model after alignment, and uses the model composed of them as the average shape model. It may be generated. For example, when the output model is a three-dimensional point cloud, the average shape generation unit 203 calculates the average value of the three-dimensional point cloud of each individual model after alignment, and generates a model composed of them as an average shape model. May be good. The reference model determination unit 103 can easily determine the reference model by using the average shape model of the target work, and can improve the reproducibility of the model.

The shape element dividing unit 204 divides the average shape model generated by the average shape generating unit 203 or the individual model determined by the individual model determining unit 201 into at least one or more shape elements constituting each model (step). S204). The shape element dividing unit 204 divides each shape element constituting the average shape model or the individual model by, for example, performing a clustering process on the average shape model or the individual model. In the shape element dividing unit 204, if the input data is a three-dimensional point cloud, it is conceivable to cluster the three-dimensional point cloud for each shape element by the k-means method.

The individual difference category determination unit 205 is an individual that represents the irregularity of the work by rank with respect to the average shape model generated by the average shape generation unit 203 and divided into one or more shape elements by the shape element division unit 204. The difference category is determined (step S205). In detail, the individual difference category determination unit 205 determines the individual difference category for the average shape model generated by the average shape generation unit 203 and divided into one or more shape elements by the shape element division unit 204. To do.

Among objects whose shape varies from individual to individual, the degree of variation and the tendency of variation differ from object to object. For example, unprocessed fruits have the same general tendency of shape, but the size and shape differ depending on the individual. Therefore, it is difficult to apply the conventionally used object detection method by template matching on a two-dimensional image. In addition, paste products such as chikuwa and processed foods are heat-treated after being molded. Therefore, the color of the surface may be slightly different or the fine unevenness of the surface may be different due to the variation in the degree of burning, but the size itself can be regarded as almost the same. Further, in fried foods such as fried chicken with batter, the shape of the batter varies from individual to individual, and the ingredients inside, specifically, in the case of fried chicken, the shape of chicken may be irregular. Therefore, there are some that cannot find regularity in the shape. In order to generate a model that is more like a target work, it is necessary to classify the variation in the shape of the work into several categories based on the nature of the work, that is, the individual difference category, and apply the model generation method suitable for each classification. It is valid.

For example, the individual difference category determination unit 205 calculates the normal distribution of the average shape model for determining the individual difference category, and calculates the normal distribution of the average shape model and the normal distribution of each work information extracted from the scene information. Match. The individual difference category determination unit 205 classifies the work having a large difference into a category having no regularity in shape, and the work having a small difference into a category of work having regularity in shape.

The work spindle determination unit 206 is based on the results obtained by the individual difference category determination unit 205 with respect to the average shape model generated by the average shape generation unit 203 or the individual model determined by the individual model determination unit 201. The work spindle, which is the inertial spindle, is determined (step S206). The work spindle determination unit 206 is specifically generated by the average shape generation unit 203 and determined by the average shape model divided into one or more shape elements by the shape element division unit 204, or the individual model determination unit 201. The work spindle is determined for the individual model divided into one or more shape elements by the shape element dividing portion 204. In the case of a category in which the shape is not regular, the work spindle determination unit 206 may determine that there is no work spindle because the work spindle is not obtained.

The primitive collation unit 207 performs primitive collation to collate the average shape model generated by the average shape generation unit 203 with a primitive shape, for example, a geometric shape such as a rectangular parallelepiped, a sphere, or a triangular pyramid (step S207). In detail, the primitive collation unit 207 performs primitive collation on the average shape model generated by the average shape generation unit 203 and divided into one or more shape elements by the shape element division unit 204. The primitive collating unit 207 may calculate the similarity based on the normal distribution of the average shape model and the normal distribution of the geometric shape. As a method of calculating the similarity, the primitive collation unit 207 obtains, for example, the difference between the normal distribution of the average shape model and the normal distribution of the geometric shape, and looks up by associating the similarity with the difference of the normal distribution. A table may be prepared in advance, and the similarity may be obtained based on the calculated difference in normal distribution and the lookup table.

The fitting model determination unit 208 determines the fitting model, which is the reference model, based on the matching result of the primitive matching by the primitive matching unit 207 (step S208). The fit model determination unit 208 may consider that the fit model cannot be determined when the calculated similarity is equal to or less than a predetermined threshold value, and may determine the average shape model as the fit model. When the similarity is equal to or greater than the threshold value, the fitting model determining unit 208 may determine the geometric shape having the highest similarity as the fitting model. Since the reference model determination unit 103 can determine a reference model close to the target work, the reproducibility of the generated model can be improved. The fitting model determining unit 208 may determine a reference model, that is, a fitting model for each shape element divided by the shape element dividing unit 204. The model generation device 10 can generate a model of a target work connected by a plurality of objects such as skewered foodstuffs by determining a fitting model for each element constituting the divided target work.

In the model generation device 10, the deformation constraint condition determination unit 104 determines the deformation constraint condition of the contour based on the numerical conditions and the shape analysis result of the reference model (step S104). The model generation device 10 can generate a model based on the acquired contour information even when the contour shape of the target work is complicated. The deformation constraint condition determination unit 104 may include the contour information of the average shape model in the acquisition of the contour information. Further, the deformation constraint condition determination unit 104 may use the contour information of the average shape model as a reference when determining the deformation constraint condition. For example, the height of the unevenness may be based on the contour of the average shape model instead of the center of the model, or the height of the unevenness may be normalized by the height of the contour of the average shape model. When the work shape is complicated, it is difficult for the deformation constraint condition determination unit 104 to fit the work shape with a simple curve, but by normalizing with an average shape model, the work shape is approximated by a simple sine wave, a polynomial, or the like. It is expected that it will be possible.

In the model generator 10, the operations of other configurations are the same as those in the first embodiment.

As described above, according to the present embodiment, the model generator 10 determines the reference model based on the result of the individual difference category and the average shape model, so that the model can be changed according to the individual difference of the target work. Reproducibility can be improved. As a result, by using the model generated by the model generator 10, the reproducibility is improved even in the simulation process before the introduction of the industrial robot, and it becomes easy to make the robot arrangement or operation check for picking work accurate. It has the effect of becoming. Further, since the model generation device 10 can generate a model of a target work in which a plurality of objects are connected, for example, it is possible to generate a model of dumplings, skewered foodstuffs, etc. handled at a food site, which is a target of model generation. The applicable range of the work is expanded.

According to the present embodiment, the model generation device 10 uses the average shape model generated by the average shape generation unit 203 as the reference model. This makes it easier for the model generation device 10 to determine the reference model by the average shape model, and even in the case of a workpiece that is difficult to represent with a primitive shape such as a rectangular parallelepiped, a sphere, or a triangular pyramid, the average shape. A model of the target work can be generated by using the model as a reference model. Further, the model generation device 10 can generate a model based on the acquired contour information even when the contour shape of the target work is complicated. Further, the model generation device 10 determines the reference model based on the work spindle and the average shape model, so that it becomes easy to accurately determine the individual difference category of the target work based on the work spindle. As a result, the model generation device 10 can improve the reproducibility of the model by being able to determine a reference model close to the target work according to the individual difference of each target work. Further, the model generation device 10 can also generate a model of the target work connected by a plurality of objects such as skewered foodstuffs by determining a fitting model for each element constituting the divided target work.

Embodiment 3.
In the third embodiment, the picking robot including the model generation device 10 described in the first embodiment or the second embodiment will be described.

FIG. 8 is a block diagram showing a configuration example of the picking robot 20 according to the third embodiment. The picking robot 20 includes a model generation device 10, a simulation condition setting unit 107, a scene generation unit 108, a data set generation unit 109, a parameter adjustment unit 110, a recognition processing unit 111, and a picking unit 112. .. The model generator 10 may have either the configuration of the first embodiment or the second embodiment.

The simulation condition setting unit 107 reads out the deformation constraint condition from the storage unit 105, generates a model of the target work by the amorphous model generation unit 106 based on the deformation constraint condition, and uses the generated model to generate sensor information in the simulation and the sensor information in the simulation. Set simulation conditions that include at least one of the work information. The deformation constraint condition stored in the storage unit 105 is determined by the model generation device 10 as described above. The amorphous model generation unit 106 can also generate a model of the target work by controlling other configurations. FIG. 9 is a block diagram showing a configuration example of the simulation condition setting unit 107 according to the third embodiment. As shown in FIG. 9, the simulation condition setting unit 107 includes a sensor information setting unit 301, a work information setting unit 302, and an environment information setting unit 303.

The sensor information setting unit 301 sets at least one of the specifications and installation information of the sensor 101 in the simulation. The sensor information setting unit 301 may set parameters such as the angle of view of the sensor 101, the working distance, the installation angle of the sensor 101, the resolution, and other specifications of the sensor 101, or the sensor installation information in the simulation. The sensor information setting unit 301 generates at least one model of the target work by the amorphous model generation unit 106 based on the deformation constraint condition stored in the storage unit 105, and uses the generated model to perform a sensor on the simulation. It may be confirmed in advance whether the target work fits within the field of view of 101.

The work information setting unit 302 sets the work information of the target work. The work information setting unit 302 generates at least one model of the target work by the amorphous model generation unit 106 based on the deformation constraint condition stored in the storage unit 105, and the generated model is used to generate the color of the target work. Information, size, the number of models of the target work used by the scene generation unit 108, and the like may be set.

The environment information setting unit 303 sets the environment information in the simulation. The environment information setting unit 303 may set the light source environment such as diffuse reflection or ambient light, the box size for making the pieces in a bulk state, the work supply method, and the like. As a work supply method, a bulk state in which the position and orientation of the target work are completely irregular, an alignment state in which the arrangement position and orientation of the target work are determined, and the like can be considered. The environment information setting unit 303 generates a model of the target work by the amorphous model generation unit 106 based on the deformation constraint condition stored in the storage unit 105, and confirms the supply state on the simulation using the generated model. You may.

The simulation condition setting unit 107 uses the sensor information setting unit 301, the work information setting unit 302, and the environment information setting unit 303 to set the simulation conditions for the sensor information, the work information, and the environment information. The picking robot 20 can improve the picking success rate by simulating the user's actual environment on the simulator and making it easier to adjust the parameters according to the user's actual environment.

The scene generation unit 108 generates a model of the target work by the amorphous model generation unit 106 based on the deformation constraint condition stored in the storage unit 105, and the generated model and the simulation set by the simulation condition setting unit 107 Simulate the scene of the target work using the conditions. The scene information obtained by the measurement of the sensor 101 includes the background information of the target work, whereas the scene of the target work simulated by the scene generation unit 108 targets only the target work. When simulating a scene, the scene generation unit 108 does not have to be different for each generated model, and the same generation model may be reused. For example, when a scene is generated using 50 models, the scene generation unit 108 duplicates 5 models of each type with 10 variations of the model of the target work to be generated, and the total number of models used for scene generation. May be 50 pieces. When the calculation cost of model generation is large, it is considered to be effective in reducing the calculation cost.

The data set generation unit 109 generates a data set for adjusting parameters that control the picking operation of the target work, based on the scene generated by the scene generation unit 108. Specifically, the data set is for adjusting parameters that control the operation of the picking unit 112 that picks the target work. FIG. 10 is a block diagram showing a configuration example of the data set generation unit 109 according to the third embodiment. As shown in FIG. 10, the data set generation unit 109 includes a 2D data generation unit 401, a 3D data generation unit 402, and an annotation data generation unit 403.

The 2D data generation unit 401 is a two-dimensional data generation unit that generates 2D data for the scene generated by the scene generation unit 108. The 2D data generation unit 401 may generate a grayscale image or an RGB image.

The 3D data generation unit 402 is a three-dimensional data generation unit that generates 3D data for the scene generated by the scene generation unit 108. The 3D data generation unit 402 may generate a three-dimensional point cloud or a distance image.

The annotation data generation unit 403 generates annotation data for the scene generated by the scene generation unit 108. The annotation data generation unit 403 may generate labeling data colored for each type of object, position / orientation information of each work in the data, or data to which a type name or the like is added.

Since the data generated by the data set generation unit 109 is simulation data, it is conceivable that there may be discrepancies due to the degree of noise added compared to the actual image. Therefore, the data set generation unit 109 may have a function of reproducing the measurement noise. The data generated by the data set generation unit 109, that is, the data generated by the 2D data generation unit 401, the 3D data generation unit 402, and the annotation data generation unit 403 are collectively referred to as a data set. The picking robot 20 can automatically generate data necessary for parameter adjustment on a simulation by the data set generation unit 109, so that the load of data collection work by the user can be reduced.

The parameter adjustment unit 110 generates a model of the target work by the amorphous model generation unit 106 based on the deformation constraint condition stored in the storage unit 105, and the generated model and the data generated by the data set generation unit 109. Automatically adjust the picking parameters using the set. FIG. 11 is a block diagram showing a configuration example of the parameter adjusting unit 110 according to the third embodiment. As shown in FIG. 11, the parameter adjusting unit 110 includes a gripping parameter adjusting unit 501 and a recognition parameter adjusting unit 502.

The gripping parameter adjusting unit 501 generates a model of the target work by the amorphous model generation unit 106 based on the deformation constraint conditions stored in the storage unit 105, and the robot hand provided in the picking unit 112 using the generated model. Adjust the parameters for. The gripping parameter adjusting unit 501 may, for example, target a robot hand such as a tweezers hand, a parallel hand, or a suction pad, and adjust the opening width or the claw width of the robot hand that makes it easy to grip the target work. The type of robot hand may be determined by the user. The gripping parameter adjusting unit 501 may adjust all kinds or a plurality of gripping parameters at the same time in adjusting the gripping parameters, or may individually adjust the gripping parameters according to a predetermined order. Hereinafter, a case where the gripping parameters are adjusted individually will be mainly described.

The detailed configuration and operation of the gripping parameter adjusting unit 501 will be described. FIG. 12 is a block diagram showing a configuration example of the gripping parameter adjusting unit 501 according to the third embodiment. As shown in FIG. 12, the gripping parameter adjusting unit 501 includes a gripping parameter adjusting range determining unit 601, a gripping parameter changing unit 602, a model rotating unit 603, a gripping evaluation unit 604, and a gripping parameter value determining unit 605. A gripping parameter adjustment end determination unit 606 is provided. Details of each configuration will be described together with the operation of the gripping parameter adjusting unit 501. FIG. 13 is a flowchart showing the operation of the gripping parameter adjusting unit 501 according to the third embodiment.

The gripping parameter adjustment range determining unit 601 generates a model of the target work by the amorphous model generation unit 106 based on the deformation constraint conditions stored in the storage unit 105, and sets the gripping parameter adjustment range based on the generated model. Determine (step S301). The gripping parameter adjustment range determination unit 601 generates at least one model of the target work by the amorphous model generation unit 106 based on the deformation constraint conditions stored in the storage unit 105, and based on the size of the generated model and the like. , The adjustment range of the gripping parameter, the initial value of the gripping parameter, and the like may be determined.

The gripping parameter changing unit 602 changes the type of gripping parameter to be adjusted (step S302).

The model rotation unit 603 generates a model of the target work by the amorphous model generation unit 106 based on the deformation constraint condition stored in the storage unit 105, and rotates the generated model (step S303). The model rotation unit 603 generates at least one model of the target work by the amorphous model generation unit 106 based on the deformation constraint condition stored in the storage unit 105, and randomly or in advance determines the generated model. It may be rotated in the order given.

The grip evaluation unit 604 rotates the model on the model rotation unit 603 to perform grip evaluation (step S304). The grip evaluation unit 604 fits the model of the robot hand when the model of the target work is rotated by the model rotation unit 603, and the frequency of occurrence of deviation between the spindle of the target work and the grip direction F _Ang and the frequency of interference with the target work. The evaluation function of the gripping parameter composed of the two evaluation values of F _{Col may be calculated.}

When the rotation of the model by the model rotation unit 603 is not completed (step S305: No), the grip parameter adjusting unit 501 returns to step S303 to rotate the model by the model rotation unit 603 and grip the model by the grip evaluation unit 604. Continue the evaluation. When the rotation of the model by the model rotating unit 603 is completed (step S305: Yes), the gripping parameter adjusting unit 501 proceeds to the operation of step S306.

The gripping parameter value determining unit 605 determines the value of the gripping parameter based on the evaluation result of the gripping evaluation unit 604 (step S306). Specifically, the gripping parameter value determining unit 605 searches for the gripping parameter value that minimizes the evaluation function calculated by the gripping evaluation unit 604, and determines the gripping parameter value found by the search as the adjustment parameter value. To do.

The gripping parameter adjustment end determination unit 606 determines whether or not the adjustment of all the gripping parameters to be adjusted has been completed (step S307). When the adjustment of all the gripping parameters to be adjusted is not completed (step S307: No), the gripping parameter adjustment end determination unit 606 instructs each unit to return to the operation of step S302 and perform the same operation as described above. To do. When the adjustment of all the gripping parameters to be adjusted is completed (step S307: Yes), the gripping parameter adjustment end determination unit 606 ends the operation of the gripping parameter adjusting unit 501.

In this way, in the picking robot 20, the gripping parameter adjusting unit 501 automatically adjusts the gripping parameter. The picking robot 20 can automatically adjust the parameters of the robot hand that easily grips the target work by using the generated model, and can reduce the load of the user's parameter adjustment work.

Return to the explanation of FIG. The recognition parameter adjusting unit 502 generates a model of the target work by the irregular model generation unit 106 based on the deformation constraint condition stored in the storage unit 105, and the generated model and the data set generation unit 109 generate the model. The recognition parameters of the object recognition method are adjusted using the data set and the grip parameters adjusted by the grip parameter adjusting unit 501. The recognition parameter adjusting unit 502 may adjust the recognition parameter of the method of detecting the candidate gripping position from the 2D data or the 3D data, or may adjust the parameter of the learning model of the gripping position detection by deep learning. However, the parameters of the learning model of deep learning used as auxiliary processing for improving the recognition rate, for example, segmentation, may be adjusted. Hereinafter, the recognition parameter adjustment of the method of detecting the candidate gripping position from the 2D data will be mainly described. Further, the recognition parameter adjusting unit 502 may adjust all kinds or a plurality of recognition parameters at the same time in adjusting the recognition parameters, or may individually adjust the recognition parameters according to a predetermined order. In the following, the case where the recognition parameters are adjusted individually will be mainly described.

The detailed configuration and operation of the recognition parameter adjusting unit 502 will be described. FIG. 14 is a block diagram showing a configuration example of the recognition parameter adjusting unit 502 according to the third embodiment. As shown in FIG. 14, the recognition parameter adjustment unit 502 includes a recognition parameter adjustment range determination unit 701, a recognition parameter change unit 702, a recognition trial unit 703, a recognition evaluation unit 704, and a recognition parameter value determination unit 705. A recognition parameter adjustment end determination unit 706 is provided. Details of each configuration will be described together with the operation of the recognition parameter adjusting unit 502. FIG. 15 is a flowchart showing the operation of the recognition parameter adjusting unit 502 according to the third embodiment.

The recognition parameter adjustment range determination unit 701 generates a model of the target work by the amorphous model generation unit 106 based on the deformation constraint condition stored in the storage unit 105, and uses the generated model to adjust the recognition parameter adjustment range. Is determined (step S401). The recognition parameter adjustment range determination unit 701 generates at least one model of the target work by the amorphous model generation unit 106 based on the deformation constraint condition stored in the storage unit 105, and based on the size of the generated model and the like. , The adjustment range of the recognition parameter, the initial value of the recognition parameter, and the like may be determined.

The recognition parameter changing unit 702 changes the type of recognition parameter to be adjusted (step S402).

The recognition trial unit 703 performs a recognition process on the data generated by the data set generation unit 109 using the grip parameters adjusted by the grip parameter adjusting unit 501 (step S403). The recognition trial unit 703 returns to step S403 when the recognition process is not completed (step S404: No), continues the recognition process, and recognizes the recognition result when the recognition process is completed (step S404: Yes). Output to the evaluation unit 704.

The recognition evaluation unit 704 evaluates the recognition result by the recognition trial unit 703 based on the recognition result acquired from the recognition trial unit 703 (step S405). _{The recognition evaluation unit 704 has an amount of deviation E Pos} between the gripping position and the optimum gripping position when recognized by the recognition trial unit 703, an _{amount of deviation E Ang} between the main axis of the target work and the gripping direction, and _{an evaluation value E Num of the} number of recognized pieces. You may calculate the evaluation function of the recognition parameter consisting of three. The optimum gripping position may be the position of the center of gravity of the target work or a position determined in advance by the user.

The recognition parameter value determination unit 705 determines the value of the recognition parameter based on the evaluation result of the recognition evaluation unit 704 (step S406). Specifically, the recognition parameter value determination unit 705 searches for the value of the recognition parameter that minimizes the evaluation function calculated by the recognition evaluation unit 704, and determines the value of the recognition parameter found by the search as the value of the adjustment parameter. To do.

The recognition parameter adjustment end determination unit 706 determines whether or not the adjustment of all the recognition parameters to be adjusted has been completed (step S407). When the recognition parameter adjustment end determination unit 706 has not completed the adjustment of all the recognition parameters to be adjusted (step S407: No), the recognition parameter adjustment end determination unit 706 instructs each unit to return to the operation of step S402 and perform the same operation as described above. To do. When the adjustment of all the recognition parameters to be adjusted is completed (step S407: Yes), the recognition parameter adjustment end determination unit 706 ends the operation of the recognition parameter adjustment unit 502.

In this way, in the picking robot 20, the recognition parameter adjusting unit 502 automatically adjusts the recognition parameter. By using the generated model and data set, the picking robot 20 can automatically adjust the recognition parameters that tend to have a high recognition rate, and can reduce the load of the user's parameter adjustment work. The parameter adjusting unit 110 automatically adjusts the gripping parameter and the recognition parameter. By automatically adjusting the parameters of the picking robot 20, the parameter adjustment work by the user becomes unnecessary, and the load of the user's parameter adjustment work can be reduced.

Return to the explanation in Fig. 8. The storage unit 105 may store the parameters for picking adjusted by the parameter adjustment unit 110.

The recognition processing unit 111 performs object recognition processing on the data acquired by the sensor 101 based on the parameters adjusted by the parameter adjustment unit 110. The recognition processing unit 111 may perform object recognition processing on the data acquired by the sensor 101 based on the adjusted parameters stored in the storage unit 105, and detect a candidate gripping position. .. The recognition processing unit 111 may directly acquire the parameters adjusted by the parameter adjustment unit 110 from the parameter adjustment unit 110.

The picking unit 112 includes a robot hand. The picking unit 112 performs a picking operation of the target work by the robot hand based on the recognition processing result by the recognition processing unit 111. Based on the recognition processing result, the picking unit 112 may repeatedly perform operations such as approaching the robot hand to the gripping position having the highest gripping likelihood, picking the target work, and moving it to the specified position. Good.

The picking robot 20 may be equipped with sensors other than the sensor 101, such as a force sensor, a proximity sensor, and a tactile sensor for controlling force. Further, the model generation device 10 may target a plurality of picking robots 20, or may target a picking robot 20 including a plurality of picking units 112. The picking robot 20 can improve the picking success rate of the target work by simulating the actual environment of the user on the simulator and adjusting the parameters using the generated data set and the model of the generated target work.

The hardware configuration of the picking robot 20 will be described. In the picking robot 20, the picking unit 112 is realized by the robot hand and the robot hand control unit. The robot hand control unit of the simulation condition setting unit 107, the scene generation unit 108, the data set generation unit 109, the parameter adjustment unit 110, the recognition processing unit 111, and the picking unit 112 is realized by a processing circuit. As in the case of the model generator 10, the processing circuit may be a processor and a memory for executing a program stored in the memory, or may be dedicated hardware.

As described above, according to the present embodiment, the picking robot 20 automatically adjusts the parameters for picking using the data set generated on the simulator and the model of the generated target work. As a result, the picking robot 20 can improve the picking success rate, and can reduce the period and cost of trial and error related to the introduction and operation of the industrial robot.

According to the present embodiment, the picking robot 20 can simulate the actual environment of the user on the simulator and automatically generate the data necessary for parameter adjustment using the model of the work generated without using the actual machine. The load of data collection work can be reduced. Further, the picking robot 20 has a high recognition rate and can automatically adjust parameters that make it easy to grip the target work, so that the load of the parameter adjustment work by the user can be reduced and the picking success rate of the target work can be improved. ..

The configuration shown in the above-described embodiment shows an example of the content of the present invention, can be combined with another known technique, and is one of the configurations without departing from the gist of the present invention. It is also possible to omit or change the part.

10 model generator, 20 picking robot, 101 sensor, 102 work information acquisition unit, 103 reference model determination unit, 104 deformation constraint condition determination unit, 105 storage unit, 106 irregular model generation unit, 107 simulation condition setting unit, 108 scenes Generation unit, 109 data set generation unit, 110 parameter adjustment unit, 111 recognition processing unit, 112 picking unit, 201 individual model determination unit, 202 registration unit, 203 average shape generation unit, 204 shape element division unit, 205 individual difference category Determination unit, 206 work spindle determination unit, 207 primitive collation unit, 208 fitting model determination unit, 301 sensor information setting unit, 302 work information setting unit, 303 environment information setting unit, 401 2D data generation unit, 402 3D data generation unit, 403 annotation data generation unit, 501 grip parameter adjustment unit, 502 recognition parameter adjustment unit, 601 grip parameter adjustment range determination unit, 602 grip parameter change unit, 603 model rotation unit, 604 grip evaluation unit, 605 grip parameter value determination unit, 606 Gripping parameter adjustment end determination unit, 701 recognition parameter adjustment range determination unit, 702 recognition parameter change unit, 703 recognition trial unit, 704 recognition evaluation unit, 705 recognition parameter value determination unit, 706 recognition parameter adjustment end determination unit.

Claims

A sensor that acquires scene information including information on the target work that generates a model,
A work information acquisition unit that acquires work information that is information on the target work from the scene information,
A reference model determination unit that determines a reference model, which is a reference model for expressing an irregular shape, based on the scene information and the work information.
A deformation constraint condition determining unit that determines a deformation constraint condition that is a condition for deforming the reference model based on the result of analyzing the work shape using the scene information and the work information and the reference model.
An irregular model generation unit that determines the deformation parameters of the reference model based on the deformation constraint conditions within the range of the deformation constraint conditions and deforms the reference model according to the deformation parameters.
A model generator characterized by comprising.
The deformation constraint condition includes at least one of the frequency of appearance of unevenness of the target work, the appearance range of unevenness of the target work, and the curvature as numerical conditions.
The model generator according to claim 1.
The work shape analyzed by the deformation constraint condition determination unit includes the work contour of the target work.
The model generator according to claim 1 or 2.
The amorphous model generation unit uses the information of the work contour when determining the amount of deformation of the reference model.
The model generator according to claim 3.
The amorphous model generation unit uses a work contour generated by synthesizing two or more different work contours as the work contour information.
The model generator according to claim 4.
The amorphous model generation unit uses the work contour generated by enlarging or reducing the scale of the work contour as the information of the work contour.
The model generator according to claim 4 or 5.
The reference model determination unit
An individual model determination unit that models each work information acquired by the work information acquisition unit and determines an individual model that models each work information.
The registration unit that integrates the individual models and
Based on the integration result by the registration unit, the average shape generation unit that generates the average shape model of the target work, and the average shape generation unit.
The model generator according to any one of claims 1 to 6, wherein the model generator is provided.
As the reference model, the average shape model generated by the average shape generation unit is used.
The model generator according to claim 7.
The deformation constraint condition determination unit determines the deformation constraint condition of the contour based on the numerical conditions and the shape analysis result of the reference model.
The model generator according to claim 7 or 8.
The reference model determination unit
An individual difference category determination unit that determines an individual difference category that expresses the unevenness of the work by rank with respect to the average shape model.
The model generation device according to any one of claims 7 to 9, wherein the reference model is determined based on the result of the individual difference category and the average shape model.
The reference model determination unit
A work spindle determination unit that determines the work spindle, which is the inertial spindle, with respect to the average shape model or the individual model.
The model generation device according to any one of claims 7 to 10, wherein the reference model is determined based on the determined work spindle and the average shape model.
The reference model determination unit
A primitive collation unit that collates the average shape model with the geometric shape,
A fitting model determination unit that determines a reference model based on the matching result of the primitive matching unit, and
The model generation device according to any one of claims 7 to 11, wherein the fitting model to be the reference model is determined.
The reference model determination unit
A shape element dividing portion that divides the average shape model or the individual model into at least one shape element constituting each model.
The model generation device according to any one of claims 7 to 12, wherein the reference model is determined for each of the divided shape elements.
The deformation constraint condition determination unit uses a feature map of the intermediate layer of the neural network that has learned the shape of the target work as the deformation parameter.
The model generator according to any one of claims 1 to 13.
The amorphous model generation unit is a neural network that outputs a model of the target work, and the input of the neural network is at least one of the scene information, the work information, the deformation constraint condition, and the deformation parameter. Including one
The model generator according to any one of claims 1 to 14.
The model generator according to any one of claims 1 to 15.
A simulation condition setting that generates a model of the target work from the deformation constraint conditions determined by the model generator and sets a simulation condition including at least one of the sensor information and the work information in the simulation using the generated model. Department and
A scene generation unit that generates a model of the target work from the deformation constraint condition and simulates the scene of the target work by using the generated model and the simulation condition.
Based on the scene, a data set generator that generates a data set for adjusting parameters that control the picking operation of the target work, and a data set generator.
A parameter adjustment unit that generates a model of the target work from the deformation constraint conditions and automatically adjusts parameters for picking using the generated model and the data set.
A recognition processing unit that performs object recognition processing on the data acquired by the sensor based on the parameters adjusted by the parameter adjustment unit.
Based on the recognition processing result by the recognition processing unit, the picking unit that performs the picking operation of the target work by the robot hand, and
A picking robot characterized by being equipped with.
The simulation condition setting unit
A sensor information setting unit that sets at least one of the sensor specifications and installation information in the simulation,
A work information setting unit that sets the work information of the target work, and
The environment information setting unit that sets the environment information in the simulation,
The picking robot according to claim 16, further comprising:, and setting simulation conditions for sensor information, work information, and environmental information.
The data set generator
A two-dimensional data generation unit that generates two-dimensional data for the scene generated by the scene generation unit, and a two-dimensional data generation unit.
A 3D data generator that generates 3D data,
Annotation data generator that generates annotation data and
The picking robot according to claim 16 or 17, wherein a data set is generated based on the scene.
The parameter adjustment unit
A grip parameter adjusting unit that generates a model of the target work from the deformation constraint conditions and adjusts the grip parameters for the robot hand using the generated model.
A recognition parameter that generates a model of the target work from the deformation constraint condition and adjusts the recognition parameter of the object recognition method by using the generated model, the data set, and the grip parameter adjusted by the grip parameter adjusting unit. Adjustment part and
The picking robot according to any one of claims 16 to 18, wherein the picking robot is characterized by automatically adjusting a gripping parameter and a recognition parameter.
The gripping parameter adjusting unit is
A grip parameter adjustment range determination unit that generates a model of the target work from the deformation constraint conditions and determines the grip parameter adjustment range based on the generated model.
A grip parameter change unit that changes the type of grip parameter to be adjusted, and a grip parameter change unit.
A model rotating unit that generates a model of the target work from the deformation constraint conditions and rotates the generated model,
A grip evaluation unit that rotates the model with the model rotation unit to evaluate the grip,
A gripping parameter value determining unit that determines the value of the gripping parameter based on the evaluation result of the gripping evaluation unit,
A grip parameter adjustment end determination unit that determines whether or not all the grip parameters to be adjusted have been adjusted, and a grip parameter adjustment end determination unit.
19. The picking robot according to claim 19, wherein the gripping parameter is automatically adjusted.
The recognition parameter adjustment unit
A recognition parameter adjustment range determination unit that generates a model of the target work from the deformation constraint conditions and determines the adjustment range of the recognition parameter using the generated model.
A recognition parameter change section that changes the type of recognition parameter to be adjusted,
A recognition trial unit that performs recognition processing on the data generated by the data set generation unit using the grip parameters adjusted by the grip parameter adjustment unit.
A recognition evaluation unit that evaluates the recognition result by the recognition trial unit,
A recognition parameter value determination unit that determines the value of the recognition parameter based on the result of the recognition evaluation unit,
A recognition parameter adjustment end determination unit that determines whether or not all the recognition parameters to be adjusted have been adjusted, and a recognition parameter adjustment end determination unit.
The picking robot according to claim 19 or 20, wherein the recognition parameter is automatically adjusted.