JP2015178138A - Information processing device and method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technique for determining success and failure of assembly even when there is play on a fitting part of a component.SOLUTION: An information processing device acquires plural relative positions and/or postures between one component and the other component, corresponding to at least one of a success state and failure state of assembly of one component to the other component. The distribution of the acquired plural relative positions and/or posture is used for creating information for determining whether assembly succeeded or not.

Description

  The present invention relates to a technique for determining whether component assembly is successful.

  With the recent development of robot technology, robots are starting to perform complicated tasks that have been performed by humans, such as assembly of industrial products. In such a robot, parts are picked and assembled by an end effector such as a hand. In parts picking by a robot and subsequent assembling work, parts assembling may fail due to causes such as deviations in the position and orientation of parts when gripping a part by a hand mechanism and mixing of abnormal parts. Therefore, after assembling the parts, it is necessary to perform an inspection after assembling the parts to confirm whether or not the assembling has been normally performed. As an inspection after the assembly, a visual inspection by an assembly operator has been common until now, but in recent years, an effort for automating the inspection has begun.

  For example, in the technique disclosed in Patent Document 1, a region for inspecting a difference at the time of assembly is designated on a screen using two-dimensional images of three cameras that observe the time of component assembly. Since the designated part can be acquired as a three-dimensional point using triangulation from at least two cameras, assembly inspection can be performed by looking at the difference of the three-dimensional point in the part assembly part.

Special table 2010-528318 gazette

  However, with respect to a part having a gap called play at the fitting part, there is a degree of freedom in the relative position and orientation between a plurality of parts to be assembled even if the assembly is successful. If only a single position / posture is set as successful, even if the relative position / posture is within the range of freedom (play), it may be determined that the assembly has failed.

  The present invention has been made in view of such a problem, and provides a technique for making it possible to determine the success or failure of assembly even for a part where play is present at a fitting part.

  According to one aspect of the present invention, a plurality of the one component and the other component corresponding to at least one of a state in which the assembly of the other component to the one component has succeeded and a state in which the other component has failed have occurred. In order to determine whether the assembling has succeeded by using an acquisition means for acquiring a relative position and / or orientation between them and a distribution of a plurality of relative positions and / or orientations acquired by the acquisition means And generating means for generating the information.

  According to the configuration of the present invention, it is possible to provide a technique for making it possible to determine the success or failure of assembly even for a part where play is present at the fitting part.

FIG. 2 is a block diagram illustrating an example of a functional configuration of the information processing apparatus 1. The figure explaining a virtual object. The figure explaining a position and attitude | position. The figure which shows distribution of relative position orientation, and probability density distribution. The flowchart of the process which information processing apparatus performs. FIG. 3 is a block diagram showing an example of a functional configuration of the information processing apparatus 2. The figure explaining a virtual object. FIG. 3 is a block diagram illustrating a functional configuration example of the information processing apparatus 3; FIG. 3 is a block diagram illustrating a functional configuration example of the information processing apparatus 4. FIG. 3 is a block diagram showing an example of a functional configuration of the information processing apparatus 5. The figure which shows the example of a display of the screen which the assembly | attachment success confirmation part 41 displays. FIG. 3 is a block diagram illustrating a functional configuration example of the information processing apparatus 6; The figure which shows the example of an external appearance of a system.

  Preferred embodiments of the present invention will be described below with reference to the accompanying drawings. The embodiment described below shows an example when the present invention is specifically implemented, and is one of the specific examples of the configurations described in the claims.

[First Embodiment]
In the present embodiment, in order to generate information (component assembly success / failure determination condition) for determining whether or not the assembly of the other component to one component is successful, the user displays “one side” displayed on the screen. "Virtual object representing one part" and / or "virtual object representing the other part" to change its position and orientation, and "virtual object representing the other part" to "virtual object representing one part" To the assembled state. Since the assembly pattern of the other component with respect to one component is various, the user assembles the “virtual object representing one component” into the “virtual object representing one component” in various assembly patterns, and the embodiment relates to this embodiment For each assembly pattern, the information processing apparatus acquires a relative position and orientation between the “virtual object representing one component” and the “virtual object representing the other component” in the assembly pattern. The information processing apparatus according to the present embodiment uses the acquired relative position and orientation distribution as the “part assembly success / failure determination condition”, that is, as a continuous region in the 6-degree-of-freedom space of the relative position and orientation. , Generate a successful assembly range.

  First, a functional configuration example of the information processing apparatus 1 according to the present embodiment will be described with reference to the block diagram of FIG. As illustrated in FIG. 1, the information processing apparatus 1 according to the present embodiment includes a constraint condition generation unit 12 and an assembly success range determination unit 13, and the information processing apparatus 1 holds a three-dimensional shape model. The unit 11 is connected. In FIG. 1, the three-dimensional shape model holding unit 11 is illustrated as an external storage device of the information processing apparatus 1, but may be a memory provided in the information processing apparatus 1.

  First, the three-dimensional shape model holding unit 11 will be described. The three-dimensional shape model holding unit 11 stores data of a virtual object (three-dimensional shape model) of a part to be assembled and data of a virtual object (three-dimensional shape model) of a part to be assembled. In the present embodiment, it is assumed that the virtual object is composed of polygons. Therefore, the data defining each polygon constituting the virtual object is stored in the three-dimensional shape model holding unit 11.

  There are various representation methods for representing the three-dimensional geometric information (FIG. 7C) of the virtual object, and various representation methods may be applied to the present embodiment. For example, a set of simple three-dimensional points, a set of three-dimensional lines representing edges, a set of three-dimensional points and a simple three-dimensional point represented by sets of lines and lines It may be expressed in terms of expression.

  Using the data stored in the three-dimensional shape model holding unit 11, the constraint condition generation unit 12 generates a virtual object of the assembly source component and a virtual object of the assembly destination component, and generates the generated virtual objects. The object is displayed on a display device (not shown) connected to the information processing apparatus 1. While viewing each virtual object displayed on the display device, the user can change the position and orientation of one or both virtual objects using an operation unit (not shown) connected to the information processing device 1 and assemble The virtual object of the assembly destination part is assembled to the virtual object of the original part (assembled successfully state). Here, “position” is represented by translational components (x, y, z) as shown in FIG. 3, and “posture” is rotational components (α, β shown by Euler angles as shown in FIG. , Γ), but the expression method is not limited to this.

  Note that the method for setting the virtual object of the assembly destination part to the virtual object of the assembly source part (the assembly success state) is not limited to the above method. For example, when the user inputs a translation component and a rotation component as numerical values, the constraint condition generation unit 12 may adopt a method of changing the position and orientation of the virtual object according to the input translation component and rotation component. I do not care.

  When the user determines that the assembling is completed, the fact is input using the operation unit, so the constraint condition generation unit 12 acquires the relative position and orientation between the virtual objects at this time. The user uses this operation (using the operation unit to change the position and orientation of one or both virtual objects so that the virtual object of the assembly destination part is assembled to the virtual object of the assembly source part, and the assembly is completed. If it is determined, the fact is input multiple times using the operation unit), the constraint condition generation unit 12 acquires the relative position and orientation between the virtual objects at that time each time.

  It should be noted that a file group in which relative positions and orientations between virtual objects that have been successfully assembled may be registered in advance and registered in the three-dimensional shape model holding unit 11. In this case, the constraint condition generation unit 12 displays a list of the file group on the display device in accordance with the operation of the operation unit by the user, and the “between virtual objects” registered in the file selected by the user using the operation unit. "Relative position / posture".

  The assembly success range determination unit 13 inputs an instruction to the user to end the above operation using the operation unit, and upon detecting this input, a plurality of “relative position and orientation” acquired by the constraint condition generation unit 12. Therefore, a probability density distribution of “relative position and orientation” regarded as assembly success is obtained as a condition for determining the success of component assembly. The distribution referred to here is a distribution expressed in a 6-degree-of-freedom parameter space. In this embodiment, a Gaussian distribution is assumed as the probability density distribution of successful assembly, and the relative position and orientation distribution that results in successful assembly is obtained. FIG. 4A shows an example of the position and orientation distribution obtained in this way. In this example, for ease of explanation, the position and orientation are represented by two parameters of x and y, but the same can be executed for the six-parameter position and orientation in actual fitting. In this case, the assembly success range determination unit 13 calculates the average vector μ of the relative position and orientation and the covariance matrix Σ using the plurality of “relative position and orientation” acquired by the constraint condition generation unit 12. FIG. 4B shows a contour line in which probability density is constant in the two-dimensional Gaussian distribution superimposed on FIG.

  Then, the successful assembly range determination unit 13 registers the distribution thus obtained in a memory such as the three-dimensional shape model holding unit 11. If such a distribution is obtained, when the relative position and orientation between the assembly source component and the assembly destination component actually become a certain position and orientation X, this position and orientation X is successfully assembled. This distribution can be used to determine whether the position / posture falls within the range. Specifically, if the distribution value (probability density) corresponding to the position and orientation X in the distribution is greater than or equal to a threshold value, the position and orientation X is a position and orientation corresponding to successful assembly, that is, the assembly source component and the assembly destination It can be determined that the assembly with the part is successful.

  In this embodiment, the Gaussian distribution is applied as the probability density distribution of successful assembly. However, other distributions may be used as long as the distribution approximates the relative position and orientation as a continuous spatial region. A rectangular distribution including a relative position and orientation may be applied. In addition, the distribution value may be a continuous real value, or success or failure may be represented by a binary value. Next, a process performed by the information processing apparatus 1 to generate a component assembly success / failure determination condition will be described with reference to FIG. 5 showing a flowchart of the process.

(Step S1000)
Using the data stored in the three-dimensional shape model holding unit 11, the constraint condition generation unit 12 generates a virtual object of the assembly source component and a virtual object of the assembly destination component, and generates the generated virtual objects. The object is displayed on a display device (not shown) connected to the information processing apparatus 1. When the user determines that the assembly has been completed, the fact is input using the operation unit, so that the constraint condition generation unit 12 detects the relative relationship between the virtual objects at this time each time this input is detected. To obtain a specific position and orientation.

(Step S1100)
The assembly success range determination unit 13 inputs an instruction to the user to end the above operation using the operation unit, and upon detecting this input, a plurality of “relative position and orientation” acquired by the constraint condition generation unit 12. Therefore, a probability density distribution of “relative position and orientation” regarded as assembly success is obtained as a condition for determining the success of component assembly.

  As described above, according to the present embodiment, the user designates the relative position and orientation between the virtual objects in a state where the assembly is successful without performing a difficult task of designating the distribution, so that there is a part with play. Even so, it is possible to generate information for enabling assembly success / failure determination.

<Modification 1>
In the first embodiment, the virtual object is assembled by changing the position and orientation without a constraint condition. For example, the user designates the assembly direction and gives a constraint to the relative position and orientation with six degrees of freedom. The calculation may be performed by reducing the number of parameters when creating the distribution. For example, when assembling from the translation y direction, parameters other than the translation component y may be fixed, and only the translation component y may be varied to assemble the virtual object. That is, some of the relative position and / or posture components are fixed, and the user operation changes components other than the partial components of the relative position and / or posture components. To do. As described above, by adding the constraint condition to the fluctuation of the virtual object, it is possible to reduce the time and effort of the operation, and more accurately obtain the relative position and orientation between the virtual objects that are successfully assembled.

<Modification 2>
For example, when a position / posture change in one direction (rotation around the axis of the cylinder in this example) is not related to the success / failure of the assembly, such as when a cylindrical part is inserted into a cylindrical hole. The constraint condition generation unit 12 designates this direction and that the position and orientation can be freely changed in this direction, and the assembly success range determination unit 13 specifies a position and orientation parameter space corresponding to this direction. Are determined as regions that are successfully assembled. As a result, it is possible to determine whether assembly is successful or not even for components whose position and orientation can be freely changed during assembly.

<Modification 3>
In the first embodiment, a plurality of “relative positions and orientations between one part and the other part” corresponding to a state where the assembly of the other part to the one part has been successfully designated is designated by a user operation. It was. However, conversely, a plurality of “relative positions and orientations between one part and the other part” corresponding to a state in which the assembly of the other part to the one part has failed may be designated by a user operation. . In this case, a Gaussian distribution is assumed as the probability density distribution generated by the assembly success range determination unit 13, and a relative position and orientation distribution that results in assembly failure is obtained.

  If such a distribution is obtained, when the relative position and orientation between the assembly source component and the assembly destination component actually become a certain position and orientation X, this position and orientation X is successfully assembled. This distribution can be used to determine whether the position / posture falls within the range. Specifically, if the distribution value corresponding to the position / orientation X in the distribution is less than the threshold value, the position / orientation X is a position / orientation corresponding to successful assembly, that is, a combination of the assembly source component and the assembly destination component. It can be judged that the date is a success.

  In addition, a relative position and orientation corresponding to a state in which the assembly of the other part to one part is successful and a plurality of relative position and orientations corresponding to the state in which the assembly has failed are acquired, respectively, for determination of the success or failure of the assembly. You can use it.

[Second Embodiment]
In the first embodiment and the modifications thereof, the user specifies the relative position and / or orientation between components that have been successfully assembled using the operation unit. In the present embodiment, the relative position and / or orientation between components is acquired using captured images of the components that have been successfully assembled.

  A functional configuration example of the information processing apparatus 2 according to the present embodiment will be described with reference to the block diagram of FIG. In the configuration shown in FIG. 6, the constraint condition generation unit 12 has an image input unit 21 and a relative position and orientation calculation unit 22, and a two-dimensional image pickup device 26 and a distance image pickup device 27 are connected to the image input unit 21. 1 is different from the configuration of FIG. Hereinafter, differences from the first embodiment will be mainly described, and the following description will be made assuming that they are the same as those of the first embodiment unless otherwise specified.

  This is the same as in the first embodiment, but the virtual object data includes local data on the object surface composed of a three-dimensional position and a three-dimensional normal direction as shown in FIG. 3D plane information (hereinafter referred to as a local surface feature) and a local 3D line on an object contour composed of a 3D position and a 3D line segment direction as shown in FIG. What is constituted by partial information (hereinafter referred to as local line features and simply referred to as geometric features refers to both local surface features and local line features) is used. In addition, as shown in FIG. 2, in the virtual object 299 representing the assembly source part, in the virtual object 298 representing the assembly destination part, the geometric feature of the location (shaded part) 100 that contacts the virtual object 298 of the assembly destination part The geometric feature of the part (shaded part) 200 that contacts the virtual object 299 of the part to be assembled may be registered in the three-dimensional shape model holding unit 11 so that it can be referred to.

  The two-dimensional image pickup device 26 is a camera including an image pickup lens and an image pickup device. The imaging lens is an optical system configured to form an image of an imaging target on an imaging device. The image sensor is various photoelectric conversion elements such as a CMOS sensor and a CCD sensor. The captured image may be a gray image or a color image, but in the present embodiment, it is assumed to be a gray image. For internal parameters such as the focal length, principal point position, and lens distortion parameter of the imaging lens, refer to the specifications of the device used or [R. Y. Tsai, "A versatile camera calibration technique for high-accuracy 3D machine vision metrology of R & E and the selenium TV and ens. RA-3, no. 4, 1987. ] Is calibrated in advance by the method disclosed in the above.

  The distance image capturing device 27 measures three-dimensional information of points on the surface of a component that is a measurement target. A distance sensor that outputs a distance image is used as the distance image capturing device 27. A distance image is an image in which each pixel has depth information. In the present embodiment, the distance sensor is a one-shot active type sensor that irradiates a multi-slit line with a color ID having a different wavelength, images the reflected light with a camera, and performs distance measurement by triangulation. Use. However, the distance sensor is not limited to this, and may be a Time-of-flight method that uses the flight time of light. Moreover, the passive type which calculates the depth of each pixel by the triangulation from the image which a stereo camera image | photographs may be used. In addition, any device that measures a distance image may be applied to the distance image capturing device 27.

  Note that the optical axis of the distance image pickup device 27 and the optical axis of the two-dimensional image pickup device 26 coincide with each other, and each pixel of the grayscale image output from the two-dimensional image pickup device 26 and the distance image pickup device 27 output. It is assumed that the correspondence relationship with each pixel of the distance image is known. However, depending on conditions, it is not essential that the grayscale image and the distance image have the same viewpoint. For example, the imaging device that captures a grayscale image and the imaging device that captures a range image may be in different positions and orientations, and the grayscale image and the range image may be captured from different viewpoints. In this case, the relative position and orientation between the imaging devices need to be known, and by projecting the three-dimensional point group in the distance image onto the grayscale image using this relative position and orientation, Correspondence between each pixel on the grayscale image and each pixel on the distance image is taken. As long as the relative position and orientation between the imaging devices that capture the same object are known and the correspondence between the images can be calculated, there is no particular limitation on the positional orientation relationship between the imaging devices.

  The image input unit 21 transfers the captured image captured by the two-dimensional image capturing device 26 and the distance image captured by the distance image capturing device 27 to the relative position and orientation calculation unit 22 at the subsequent stage. If each of the 2D image capturing device 26 and the distance image capturing device 27 captures a moving image, the image input unit 21 captures the captured image and the distance image capturing device 27 of each frame captured by the 2D image capturing device 26. The distance image of each frame captured by is transferred to the subsequent relative position and orientation calculation unit 22.

  The relative position / orientation calculation unit 22 includes the captured image and the distance image transferred from the image input unit 21, the virtual object data of the assembly source component acquired from the 3D shape model holding unit 11, and the virtual object of the assembly destination component. And the relative position and orientation between virtual objects are obtained. A process performed by the relative position / orientation calculation unit 22 to obtain a relative position / orientation between virtual objects will be described in detail.

  First, the approximate values of the positions and orientations of the assembling source component and the assembling destination component for the two-dimensional image capturing device 26 and the distance image capturing device 27 are acquired. This approximate value is used as an approximate value assuming that the approximate position and orientation of the part are known in advance. The approximate value may be input by the user using the operation unit. When the approximate value is registered in the 3D shape model holding unit 11 in advance, the approximate value is input to the 3D shape model holding unit 11. Get it from. That is, the approximate value acquisition method is not limited to a specific method.

  For example, assuming that the information processing device 2 continuously measures the position and orientation of the component in the time axis direction, the previous measurement value (previous time) may be used as the approximate position and orientation. In addition, the speed and angular velocity of a part may be estimated using a time series filter based on the measurement of the past position and orientation, and the current position and orientation may be predicted from the past position and orientation and the estimated speed / acceleration. . In addition, it is possible to estimate the approximate position and orientation of a component by holding an image of the component captured in various orientations as a template and performing template matching on the input image. Alternatively, when the position and orientation of a part can be measured by another sensor, the output value from the sensor may be used as the approximate value of the position and orientation. The sensor may be, for example, a magnetic sensor that measures the position and orientation by detecting a magnetic field generated by the transmitter with a receiver mounted on a component. Moreover, the optical sensor which measures a position and attitude | position by image | photographing the marker arrange | positioned on components with the camera fixed to the scene may be sufficient. In addition, any sensor may be used as long as it measures a position and orientation with six degrees of freedom.

  Next, based on the approximate position and orientation of the assembly source component and the approximate location and orientation of the assembly destination component, the three-dimensional point group in the distance image, the virtual object of the assembly source component, and the virtual of the assembly destination component Associate with an object. Each local surface feature constituting the virtual object is projected onto the distance image using the approximate position and orientation of each part and the internal parameters of the calibrated distance image capturing device 27. And the distance point on the distance image corresponding to each projected local surface feature is held as a three-dimensional point corresponding to each surface. At this time, when virtual objects overlap on the image and an occlusion occurs, the occlusion area may be estimated to suppress the association in the occlusion area. Specifically, based on the approximate position and orientation of the parts corresponding to each virtual object, a relative position and orientation between the virtual objects is obtained, and a front-rear determination between the virtual objects with respect to the imaging apparatus is performed. As a result, the occlusion area is roughly estimated, and processing is performed by suppressing the association between the virtual object and the image in the area where the occlusion occurs.

  Next, the edge on the grayscale image is associated with the virtual object. Using the approximate position and orientation of each virtual object and the internal parameters of the calibrated two-dimensional image capturing device 26, local line features constituting the virtual object are projected onto the image, and the detected edges and the virtual object in the virtual object are projected on the image. The local line feature is associated. When a plurality of edges are detected corresponding to each local line feature, among the detected edges, the closest edge on the image is associated with the projected local line feature.

  Next, based on the correspondence data between the edge on the grayscale image corresponding to each line segment in the detected virtual object and the calculated three-dimensional point in the distance image corresponding to each surface in the virtual object, the assembly destination The position and orientation of the part and the part from which it is assembled are calculated. In this step, the position and orientation are updated by solving the linear simultaneous equations so that the error between the measurement data and the virtual object is minimized based on the calculated correspondence data.

  Here, since the distance between the image and the distance in the three-dimensional space are different in scale, there is a possibility that the contribution rate is biased to either one of the measurement data simply by solving the simultaneous equations. Therefore, in this embodiment, [Tateno, Kotake, Uchiyama, “Highly accurate and stable model fitting method that best integrates distance and grayscale images for bin picking,” IEICE Transactions D, Information System J94-D (8), 1410-1422, 2011. By performing the optimization based on the maximum likelihood estimation as shown in FIG. Since the position and orientation estimation method based on the maximum likelihood estimation is well known, detailed description thereof is omitted. Refer to the above-mentioned literature for details. Note that the method of calculating the position and orientation of the measurement target object is not limited to the above-described method based on the maximum likelihood estimation, and may be a simpler method, for example, by performing repetitive calculations using the Levenberg-Marquardt method. It may be performed by the steepest descent method. Further, other nonlinear optimization calculation methods such as a conjugate gradient method and an ICCG method may be used. In addition to the optimization calculation based position and orientation calculation, a large number of positions and orientations are generated so that the values of 6 degrees of freedom can be comprehensively covered within a certain range around the approximate position and orientation. In addition, the position and orientation may be estimated by evaluating the degree of matching between the geometric feature that can be observed in the orientation, the grayscale image, and the distance image.

  Next, it is determined whether or not the updated position and orientation has converged, that is, whether or not iterative calculation is required. Specifically, it is determined that convergence has occurred when the correction value is approximately 0, or when the difference before and after correction of the square sum of the error vectors is approximately 0. If it has not converged, the position and orientation calculation process is performed again using the updated position and orientation. When it is determined that the image has converged, a final estimated value of the relative position and orientation between the imaging device and the measurement target object is determined.

  In this way, the constraint condition generation unit 12 can determine the position and orientation of the component at the assembly source and the position and orientation of the component at the assembly destination. Find the relative position and orientation. The obtained position / orientation is a relative position / orientation between components that have been successfully assembled.

  In the process performed by the information processing apparatus 2 according to the present embodiment to generate a component assembly success / failure determination condition, the constraint condition generation unit 12 performs the above-described process in step S1000, so that the relative position between the virtual objects is determined. Ask for posture. In step S1100, the same processing as in the first embodiment is performed.

  In the present embodiment, the relative position and orientation between the virtual objects are obtained using both the captured image by the two-dimensional image capturing device 26 and the distance image by the distance image capturing device 27. You may obtain | require the relative position and orientation between virtual objects using only one side.

  As described above, according to the present embodiment, it is possible to generate information for enabling assembly success / failure determination even for a part with play without performing the above-described work by the operator.

[Third Embodiment]
In the present embodiment, the constraint condition generation unit 12 acquires a relative position and orientation that does not cause interference between virtual objects among the relative positions and orientations that can be taken between virtual objects. A functional configuration example of the information processing apparatus 3 according to the present embodiment will be described with reference to the block diagram of FIG. The configuration shown in FIG. 8 is different from the configuration of FIG. 1 in that the constraint condition generation unit 12 includes a relative position and orientation setting unit 31 and a component interference determination unit 32. Hereinafter, differences from the first embodiment will be mainly described, and the following description will be made assuming that they are the same as those of the first embodiment unless otherwise specified.

  The relative position / orientation setting unit 31 acquires a plurality of relative positions / orientations between virtual objects designated by the user in the same manner as in the first embodiment.

  The component interference determination unit 32 determines, for each relative position / posture acquired by the relative position / posture setting unit 31, whether interference has occurred between virtual objects having the relative position / posture. Since the process of determining whether or not there is interference between virtual objects is generally known since the determination of the interference between characters is also performed in the field of video games, description of this process is omitted. Then, the component interference determination unit 32 selects a relative position and orientation that does not cause interference between virtual objects from among the relative position and orientation acquired by the relative position and orientation setting unit 31, and outputs the selected relative position and orientation to the successful assembly range determination unit 13. To do. Thereby, the assembly success range determination unit 13 generates a probability density distribution based on a relative position and orientation in which no interference occurs between the virtual objects among the relative positions and orientations between the virtual objects designated by the user.

  As described above, according to the present embodiment, among the relative positions and orientations designated by the user, the relative positions and orientations in which no interference occurs between the virtual objects is sent to the assembly success range determination unit 13. Information for enabling determination can be generated.

<Modification 1>
In the third embodiment, the interference determination is performed on the relative position and orientation specified by the user. However, the interference determination may be performed on the relative position and orientation acquired by another method. For example, various relative positions and orientations may be generated by simulation, and the above-described interference determination may be performed for each of the generated relative positions and orientations. Thus, information for enabling success / failure determination can be generated without performing designation by the user.

[Fourth Embodiment]
In the present embodiment, for each relative position and orientation in the probability density distribution obtained by the assembly success range determination unit 13 in the first to third embodiments and their modifications, the position of the virtual object in the relative position and orientation Let the user see the relationship. A functional configuration example of the information processing apparatus 4 according to the present embodiment will be described with reference to the block diagram of FIG. The configuration shown in FIG. 9 is obtained by adding an assembly success confirmation unit 41 to the configuration of FIG. Hereinafter, differences from the first embodiment will be mainly described, and the following description will be made assuming that they are the same as those of the first embodiment unless otherwise specified.

  As shown in FIG. 11, the assembly success confirmation unit 41 plots the relative position and orientation acquired by the constraint condition generation unit 12 for creating the probability density distribution (in FIG. 11, the position and orientation are x, (represented by two parameters of y) is displayed on the display device. The display range is a range including the relative position and orientation acquired by the constraint condition generation unit 12 for creating the probability density distribution in the probability density distribution generated by the assembly success range determination unit 13, and this range is divided into a plurality of rectangular regions. It is divided and displayed. Note that the division pattern is not limited to the pattern that is equally divided as shown in FIG. 11, and the user may check the number of set positions and orientations at random and allow the user to confirm.

  When the user designates one of a plurality of rectangular regions using the operation unit, the assembly success confirmation unit 41 sets a relative position and orientation corresponding to the center position (for example, a triangle in the figure) of the designated rectangular region. The two virtual objects having the relative position and orientation are identified and displayed on the display device. Thereby, the user can visually recognize the relative position and orientation relationship between the virtual objects corresponding to the respective rectangular regions in the probability density distribution.

  Note that the designation target is not limited to the rectangular area, and a plotted point (relative position and orientation) may be designated. In that case, two virtual objects having relative positions and orientations corresponding to the designated points are displayed on the display device.

[Fifth Embodiment]
In this embodiment, the user designates each plot point in the screen displayed by the assembly success confirmation unit 41 on the display device, and the relative position and orientation corresponding to the plot point corresponds to the assembly success. Check if it is a relative position and posture. The confirmation method is not limited to this. When the user determines that the relative position and orientation corresponding to the plot point does not correspond to the assembly success, the user designates that the plot point is not a target using the operation unit. The designated plot points may be changed in display form or hidden. By this designation, the probability density distribution is updated by setting the probability density corresponding to the relative position and orientation corresponding to the designated plot point to 0 in the current probability density distribution.

  A functional configuration example of the information processing apparatus 5 according to the present embodiment will be described with reference to the block diagram of FIG. The configuration shown in FIG. 10 is obtained by adding an assembling determination unit 51 and an assembling success range changing unit 52 to the information processing apparatus 4 of FIG. Hereinafter, differences from the fourth embodiment will be mainly described, and the description will be made assuming that the fourth embodiment is the same as the fourth embodiment unless otherwise specified.

  The assembly determination unit 51 receives a non-target designation for the plot points via the operation unit. The assembly success range changing unit 52 updates the probability density distribution by setting the probability density corresponding to the relative position and orientation corresponding to the plot point designated as out of target in the current probability density distribution to 0.

  As described above, according to the present embodiment, the user determines whether the relative position and orientation generated by the constraint condition generation unit 12 is a relative position and orientation corresponding to successful assembly, and the result of the determination is based on whether the component assembly is successful. It can be reflected in the judgment conditions. However, the range of successful assembly can be determined more accurately.

[Sixth Embodiment]
In this embodiment, after obtaining the assembly success range, a process involving the assembly success determination is executed, and the assembly success range is updated according to the determination result. In the following, a part assembly operation is performed by the robot arm, and the assembly success range is updated according to the result of the assembly success determination by the assembly operation. An example of the functional configuration of the information processing apparatus 6 according to the present embodiment is shown in FIG. An example of the appearance of the system according to this embodiment is shown in FIG. In FIGS. 12 and 13, the reference numerals already shown are the same as described above unless otherwise specified.

  The robot 90 is controlled by the robot controller 80, moves the arm tip to the commanded position, and performs object grasping / assembling processing and the like. The assembled component 70 includes an assembly source component and an assembly destination component. The two-dimensional image pickup device 26 and the distance image pickup device 27 are installed at positions where an object to be measured (an assembled component 70 in the present embodiment) such as a hand of an arm of the robot 90 can be imaged.

  Similarly to the relative position / orientation calculation unit 22, the runtime relative position / orientation calculation unit 61 includes the captured image and the distance image transferred from the image input unit 21, and the assembly source component acquired from the three-dimensional shape model holding unit 11. Using the virtual object data and the virtual object data of the part to be assembled, the relative position and orientation between the virtual objects are obtained.

  More specifically, the runtime relative position / orientation calculation unit 61 is the nearest neighbor between the local surface feature belonging to the location (shaded portion) 100 and the local surface feature belonging to the location (shaded portion) 200 as shown in FIG. The pair that becomes is calculated. For example, [Oshitake Takeshi, Nakazawa Atsushi, Ikeuchi Katsushi, “High-speed simultaneous registration of multiple range images using index images,” IEICE paper Magazine, Vol. J89-D No. 3, pp. 513-521, Mar. 2006. ] Or the like in the distance image alignment may be used. Then, the runtime relative position and orientation calculation unit 61 calculates the relative position and orientation between the paired local surface features. Here, the relative position / orientation is obtained by paying attention to a part of the assembly, but the relative position / orientation may be estimated by performing model fitting on the entire component, and the value may be used.

  After the robot 90 finishes assembling the parts, the runtime assembly determination unit 62 displays a screen for inquiring the user as to whether or not the assembly is successful on the display device. The user visually recognizes the assembled component 70 and inputs a determination result as to whether or not the assembly in the assembled component 70 has been successful using the operation unit. Therefore, the runtime assembly determination unit 62 acquires this determination result. . Note that the method for acquiring information indicating whether or not the assembly in the assembled component 70 has been successfully performed by the runtime assembly determination unit 62 is not limited to this. For example, an abnormal pressure value may be obtained using a pressure sensor of a robot. When indicated, it may be determined as a failure. The runtime assembly determination unit 62 includes information indicating whether the assembly of the assembled component 70 has failed (input by the user via the operation unit), and the relative position / posture determined by the runtime relative position / posture calculation unit 61 at this time. Are sent to the assembly success range changing unit 52.

  When the “information indicating whether or not the assembly of the assembled component 70 has failed” received from the runtime assembly determination unit 62 indicates “failed”, the assembly success range changing unit 52 indicates the current probability density distribution. The probability density distribution is updated by setting the probability density corresponding to the relative position and orientation received from the runtime assembly determination unit 62 together with the information to zero. If the “information indicating whether or not the assembly of the assembled component 70 has failed” received from the runtime assembly determination unit 62 indicates “successful”, the assembly success range changing unit 52 indicates the current probability. In the density distribution, the probability density corresponding to the relative position and orientation received from the runtime assembly determination unit 62 together with the information may be increased by a specified amount.

  Thus, according to the present embodiment, the assembly success range can be determined more accurately by feeding back the relative position and orientation at the time of assembly and the success / failure determination. In addition, you may use combining embodiment or the modification demonstrated above suitably for one part or all part.

[Seventh Embodiment]
Each functional unit in the information processing apparatus shown in FIGS. 1, 6, 8, 9, 10, and 12 may be configured by hardware, but may be configured by software (computer program). In this case, any computer having a memory that holds the computer program and an execution unit that executes the computer program can be applied to the information processing apparatuses described in the above embodiments and modifications. In addition, what was demonstrated as an operation part in said description can be comprised using user interfaces, such as a mouse | mouth and a keyboard, and what was demonstrated as a display apparatus should be comprised using CRT, a liquid crystal screen, etc. Can do.

(Other examples)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (10)

A plurality of relative positions between the one part and the other part corresponding to at least one of a state where the assembly of the other part to the one part is successful and a state where the other part is unsuccessful; and / or An acquisition means for acquiring a posture;
An information processing apparatus comprising: generating means for generating information for determining whether or not the assembling has been successful using distribution of a plurality of relative positions and / or postures acquired by the acquiring means . The acquisition means includes
When a virtual object representing the one part and / or a virtual object representing the other part is moved by a user operation and the virtual object representing the other part is assembled to the virtual object representing the one part The relative position and / or orientation between the virtual object representing the one part and the virtual object representing the other part corresponds to a state in which the assembly of the other part to the one part is successful. The information processing apparatus according to claim 1, wherein the information processing apparatus acquires the relative position and / or posture between the one component and the other component.   Some of the components of the relative position and / or orientation are fixed, and the user operation allows components other than the some of the components of the relative position and / or orientation to be fixed. The information processing apparatus according to claim 2, wherein the information processing apparatus is changed. The acquisition means includes
The relative position and / or posture between the one part and the other part is determined from the captured images of the one part and the other part in a state where the other part is assembled to the one part. And obtaining the relative position and / or orientation between the one part and the other part corresponding to a state in which the assembly of the other part to the one part is successful. The information processing apparatus according to 1. The acquisition means includes
Using a relative position and / or posture between the one part and the other part, interference occurs between a virtual object representing the one part and a virtual object representing the other part. If the interference does not occur, the relative position and / or posture is set to the one component corresponding to the state where the assembly of the other component to the one component is successful. The information processing apparatus according to claim 1, wherein the information processing apparatus is acquired as a relative position and / or orientation with respect to the other component. The generating means includes
Generating a probability density distribution of a plurality of relative positions and / or orientations acquired by the acquisition means as information for determining whether or not the assembly of the other part to the one part is successful; The information processing apparatus according to any one of claims 1 to 5, wherein the information processing apparatus is characterized in that: The generating means includes
The probability density distribution is updated by setting the probability density corresponding to a relative position and / or posture specified by a user operation to 0 in the probability density distribution. Information processing device. The generating means includes
The probability density distribution is updated by increasing the probability density corresponding to the relative position and / or orientation specified by the user operation by a specified amount in the probability density distribution. The information processing apparatus described. An information processing method performed by an information processing apparatus,
The acquisition unit of the information processing apparatus includes a plurality of the one component and the other component corresponding to at least one of a state where the assembly of the other component to the one component is successful and a state where the other component is unsuccessful. An acquisition step of acquiring a relative position and / or posture between,
A generating step of generating information for determining whether the assembly is successful, using a plurality of relative position and / or orientation distributions acquired in the acquiring step; An information processing method characterized by comprising:   The computer program for functioning a computer as each means of the information processing apparatus of any one of Claims 1 thru | or 8.

Images