JP2020016446A - Information processing device, control method of information processing device, program, measurement device, and article production method - Google Patents

Abstract

To provide a technique enabling acquisition of a position and orientation of an object with good accuracy.SOLUTION: An information processing device comprises: image acquisition means to acquire a captured image captured by an imaging section 104, the image including distance information of an object 106; position and orientation acquisition means to acquire a position and orientation of the object from the distance information of the object; estimation means to estimate a presence range of the object from the distance information of the object when an object different from the object whose position and orientation is acquired by the position and orientation acquisition means is included in the captured image; and determination means to determine a position and orientation of the imaging section on the basis of the presence range of the object.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an information processing device, a control method for the information processing device, a program, a measuring device, and an article manufacturing method.

  With the development of robot technology in recent years, complicated tasks that have been performed by humans, such as assembly of industrial products, are being replaced by robots. Such a robot picks and assembles a part by an end effector such as a hand. Conventionally, parts for picking are supplied by using a device called a parts feeder for arranging and supplying parts one by one, or by stacking parts in various postures on a pallet (container). Have been done.

  In the case of using a parts feeder, since the position and orientation of each component are supplied in a predetermined state, picking by the robot is relatively easily performed. However, the cost for preparing the parts feeder device is extra. Also, different parts feeders must be prepared according to the shape of the parts. On the other hand, when parts are supplied in piles, it is only necessary to place the parts on a pallet, so that an increase in cost can be avoided. Further, in response to the recent trend of small-quantity multi-product production, attention has been paid to a piled supply capable of quickly responding to various parts.

  Under such a trend, a picking operation system using a combination of a two-dimensional camera or a three-dimensional measuring device and component recognition has been developed. As information processing apparatuses for recognizing the position and orientation of an object, there are techniques described in Patent Literature 1 and Patent Literature 2. Patent Literature 1 describes a robot that performs measurement by moving a distance measurement sensor to a region where a work is taken out. Patent Document 2 discloses a work takeout apparatus that measures the position and posture of a work with a visual sensor and moves a work takeout means (robot) based on the measured position and posture.

JP 2017-217726 A JP-A-2003-34430

  However, in Patent Documents 1 and 2, if the position and orientation of the object cannot be measured correctly in the first measurement, the position and orientation of the visual sensor in the second measurement cannot be determined, and the position of the object cannot be determined in the second measurement. There is a problem that the posture cannot be measured correctly.

  An object of the present invention is, for example, to provide a technology that enables accurate acquisition of the position and orientation of an object.

  In order to solve the above-described problems, the present invention provides an image acquisition unit that acquires an image captured by an imaging unit and includes distance information of an object, and a position and orientation acquisition unit that acquires a position and orientation of the object from the distance information of the object. And, when an object different from the object whose position and orientation has been acquired by the position and orientation acquisition unit is included in the captured image, based on estimating means for estimating the existence range of the object from the distance information of the object, based on the existence range of the object, Determining means for determining the position and orientation of the imaging unit.

  According to the present invention, for example, it is possible to provide a technology that enables the position and orientation of an object to be acquired with high accuracy.

FIG. 1 is a schematic diagram illustrating an example of a robot system including an information processing device according to a first embodiment. FIG. 3 is a block diagram illustrating a hardware configuration example of an information processing device. FIG. 2 is a diagram illustrating a functional configuration of a robot system equipped with the information processing device according to the first embodiment. FIG. 4 is a diagram illustrating an example of a use case of the information processing apparatus according to the first embodiment. 5 is a flowchart illustrating a processing flow of the information processing apparatus according to the first embodiment. It is a flowchart which shows the process of the existence range estimation of an object in step S508. FIG. 7 is a schematic diagram illustrating a process in step S510 according to the first embodiment. It is a flowchart which shows the process of step S510 concerning 1st Embodiment. It is a flow chart which shows processing of Step S510 concerning a 2nd embodiment. It is a flow chart which shows processing of Step S510 concerning a 3rd embodiment.

  Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings.

(1st Embodiment)
[Hardware configuration]
FIG. 1 is a schematic diagram illustrating an example of a robot system 1 including the information processing device 105 according to the first embodiment. The robot system 1 is a picking system that takes out the objects 106 piled up in a pallet 107. The robot system 1 includes a robot arm 101, a robot control device 103, a sensor unit 104, and an information processing device 105. The robot arm 101 is, for example, an articulated robot, and operates upon receiving a control command from the robot control device 103. A hand 102 is mounted on the tip of the robot arm 101. The hand 102 grasps the object 106 and takes it out of the pallet 107. As the hand 102, for example, a hand having a chuck mechanism may be used, or a hand that can be driven by a motor may be used. Further, a suction pad that suctions the object 106 by air pressure may be used.

  The robot control device 103 controls the robot arm 101 to move the hand 102 to a position where the objects 106 piled up on the pallet 107 are in the field of view of the sensor unit 104. The robot control device 103 instructs the information processing device 105 to measure the position and orientation of the object 106. At this time, the robot control device 103 transmits the position and orientation of the hand 102 to the information processing device 105. Note that the robot control device 103 may be integrated with the robot arm 101.

  When receiving the instruction to measure the position and orientation of the object 106 from the robot control device 103, the information processing device 105 controls the sensor unit 104 to acquire a captured image including the object 106. The information processing device 105 acquires the position and orientation of the object 106 based on the captured image and the position and orientation of the hand 102, and transmits the acquired position and orientation to the robot control device 103.

  The sensor unit 104 is an imaging unit that captures a two-dimensional image. The sensor unit 104 is fixedly attached to the hand 102 at the tip of the robot arm 101. The sensor unit 104 projects a pattern light on an imaging target to acquire a pattern image. The pattern image is an image obtained by observing the projected pattern light. Further, the sensor unit 104 acquires a grayscale image of the imaging target. The grayscale image is an image in which the brightness of each pixel is represented by white to black gradation. Note that the information processing device 105 may be incorporated in the sensor unit 104 and be integrated, or may be incorporated in the robot control device 103 and form a part of the robot control device.

  FIG. 2 is a block diagram illustrating a hardware configuration example of the information processing apparatus 105. The information processing apparatus 105 according to the present embodiment is, for example, a computer, and includes a CPU (microprocessor) 201 and a memory 202 such as a random access memory. Further, an input unit 203 such as a keyboard and an external storage device 204 such as a hard disk drive are provided. Further, a communication interface (hereinafter, referred to as an imaging I / F) 205 with the sensor unit 104 and a communication interface (hereinafter, referred to as an RCI / F) 206 with the robot control device 103 are provided. The CPU 201 executes various processes in accordance with programs stored in the memory 202, and particularly executes a position and orientation acquisition process described later. These programs can be stored in the external storage device 204 or supplied from an external device (not shown).

  The information processing device 105 transmits imaging conditions such as the amount of pattern light in the sensor unit 104 and the exposure time to the sensor unit 104 via the imaging I / F 205. In addition, the imaging unit 205 instructs the sensor unit 104 to execute imaging via the imaging I / F 205, and receives an image captured by the sensor unit 104. Further, the information processing device 105 receives a measurement instruction from the robot control device 103 via the RCI / F 206, and transmits the calculated position and orientation of the measurement target to the robot control device 103. Further, the information processing apparatus 105 outputs various information to the monitor 208 via the video I / F 207 and inputs various information via the input unit 203.

[Software Configuration]
FIG. 3 is a diagram illustrating a functional configuration of a robot system equipped with the information processing device according to the first embodiment. The information processing device 105 includes an image acquisition unit 304, a pallet measurement unit 305, a position and orientation acquisition unit 306, a determination unit 307, and an estimation unit 308.

  The image obtaining unit 304 obtains a pattern image captured by the sensor unit 104, and obtains a distance point group representing distance information of an object to be measured from the obtained image. The acquisition of the distance point group can be realized by a known method such as a gray code pattern method or a phase shift method. In the method using the gray code pattern method, several hundred slits are coded by a binary number, and a space is coded by projecting a binary pattern in time series. In the method using the phase shift method, a distance point group is obtained by projecting a stripe pattern in which the projection intensity is modulated into a sine wave while shifting the phase.

  The pallet measuring unit 305 detects the pallet 107 and measures its position and orientation. The input at the time of measuring the position and orientation of the pallet 107 in the pallet measurement unit 305 is a grayscale image, a pattern image, and a distance point group. The position and orientation of 107 are measured. The pallet measurement unit 305 outputs the measured position and orientation of the pallet 107.

  The position and orientation acquisition unit 306 detects an object to be measured using the grayscale image, the pattern image, and the distance point group, and acquires the position and orientation. The position and orientation acquisition unit 306 can be realized by a known method such as a method using template matching, a method using a neural network, and a method using an ICP algorithm. In the method using template matching, the template image is scanned on the grayscale image, and the position of the object is acquired based on the position of the template image at the time of the highest similarity. In the method using a neural network, a network that acquires the position of an object from a grayscale image is learned, and the position of the object is acquired using the network. In the method using the ICP algorithm, the position and orientation of the object are acquired by repeating the alignment using a point group obtained from the 3D model.

  The estimating unit 308 estimates the existence range of the object from the distance information of the object when the captured image includes an object different from the object whose position and orientation has been acquired by the position and orientation acquisition unit 306. That is, when an object whose position and orientation cannot be acquired by the position and orientation acquisition unit 306 is included in the captured image, the estimation unit 308 estimates the existence range of the object whose position and orientation cannot be acquired. The estimating unit 308 estimates the existence range of the object whose position and orientation cannot be obtained using the grayscale image, the pattern image, and the distance point group.

  The determination unit 307 determines the position and orientation of the sensor unit 104 based on the existence range of the object estimated by the estimation unit 308. Details of the process of determining the position and orientation of the sensor unit 104 will be described later with reference to FIGS.

  The robot control device 103 includes a sensor position control unit 302 and a hand control unit 303. The sensor position control unit 302 controls the robot arm 101 to move the sensor unit 104 to the position and orientation determined by the determination unit 307. The hand control unit 303 controls the hand 102 based on the position and orientation of the object 106 acquired by the position and orientation acquisition unit 306 to grasp and move the object.

[Overview of Embodiment]
FIG. 4 is a diagram illustrating an example of a use case of the information processing apparatus 105 according to the first embodiment. FIG. 4A is a diagram showing a state in the first measurement. The object 106 is an object to be measured in the first measurement. The pallet 107 is a container in which the object 106 is stored. The visual field range 405 is the visual field range of the sensor unit 104 in the first measurement. In FIG. 4A, the sensor unit 104 exists right above the pallet 107. The object 106 leans on the inner side surface of the pallet 107 and is in a vertically placed state. At this time, only a small surface of the object 106 can be imaged in the visual field range 405 due to the relative positional relationship between the sensor unit 104 and the object 106. In this case, since sufficient information for acquiring the position and orientation of the object 106 is not obtained, it is difficult to measure the correct position and orientation of the object 106. That is, at this time, it can be said that the object 106 is an object whose position and orientation cannot be acquired by the position and orientation acquisition unit 306.

  FIG. 4B is a diagram illustrating a state in the second measurement. Here, the second measurement is a measurement performed after the first measurement. The visual field range 410 is the visual field range of the sensor unit 104 in the second measurement. In FIG. 4B, the sensor unit 104 captures an image of the object 106 from a direction obliquely above the pallet 107. At this time, many surfaces of the object 106 can be observed in the visual field range 410. In this case, since sufficient information for acquiring the position and orientation of the object 106 is obtained, the measurement of the position and orientation becomes easy, and the correct position and orientation of the object 106 can be measured. In the present embodiment, the position and orientation of the sensor unit 104 in the second measurement is determined based on the existence range of the object 106 whose position and orientation could not be obtained in the first measurement.

[Processing flow]
FIG. 5 is a flowchart illustrating a processing flow of the information processing apparatus according to the first embodiment. Hereinafter, the processing of each step will be described. First, in step S501, the sensor position control unit 302 moves the sensor unit 104 to the first sensor position and orientation. Since the sensor unit 104 is attached to the hand 102, the sensor position control unit 302 controls the position and orientation of the sensor unit 104 by driving the robot arm 101 and the hand 102. The first sensor position and orientation is the position and orientation of the sensor unit 104 in the first measurement. As the first sensor position and orientation, a position and orientation registered by the user in advance may be used, or a position and orientation set by initial setting may be used.

  Next, in step S502, the sensor unit 104 captures the first pattern image and the first grayscale image, and the image acquisition unit 304 obtains the first pattern image and the first grayscale image from the sensor unit 104. . The first pattern image and the first grayscale image are used for the first measurement. Then, in step S503, the image acquisition unit 304 calculates a distance point group (first distance point group) from the first pattern image. The calculation of the distance point group may be performed in the sensor unit 104. The first distance point group is used for the first measurement. In step S504, the pallet measurement unit 305 measures the position and orientation of the pallet 107. At this time, the inputs to the pallet measurement unit 305 are the first grayscale image, the first pattern image, and the first distance point group. The pallet measurement unit 305 measures the position and orientation of the pallet 107 from the input first grayscale image, first pattern image, and first distance point group, and the position and orientation of the pallet 107 expressed in six degrees of freedom. Is output. Step S504 can be realized by a known method such as a method using template matching, a method using a neural network, and a method using an ICP algorithm which is one of the positioning methods.

In step S505, the position and orientation acquisition unit 306 detects the object 106 and acquires the approximate position and orientation. In step S506, it is determined whether the position and orientation acquisition unit 306 has been able to acquire the approximate position and orientation of the object. A recognition score obtained by obtaining the approximate position and orientation is used as the evaluation value of the determination. The recognition score is a score indicating the degree of similarity between the observed object and the object registered in the recognition dictionary. If the recognition score is _{equal to} or _larger than the threshold th _recog, it is determined that the approximate position and orientation has been acquired. If the recognition score is less than the threshold th _recog, it is determined that the approximate position and orientation has not been acquired. If the approximate position and orientation have been acquired (Yes), the process proceeds to step S507 described below, and in step S507, the position and orientation acquisition unit 306 acquires the detailed position and orientation of the object 106 to be measured. Thereafter, the process proceeds to step S516.

If the approximate position and orientation cannot be acquired (No), the process proceeds to step S508 described below, and in step S508, the estimation unit 308 estimates the existence range of the object 106. The existence range of the object 106 is indicated by a point cloud. Details of step S508 will be described later with reference to FIG. In step S509, based on the existence range of the object 106 estimated in step S508, the determination unit 307 determines whether or not the object 106 exists within the field of view of the sensor unit 104. At this time, the inputs to the determination unit 307 are the estimated existence range of the object and the first grayscale image. Here, a method of determining whether or not the object 106 exists within the visual field range of the sensor unit 104 will be described. First, the existence range of the object 106 is projected onto a grayscale image, and the number of pixels occupied by the existence range of the object 106 is calculated. If the number of pixels occupying the existence range of the object 106 is equal to or _larger than the threshold th _exit, it is determined that the object 106 exists. If the number of pixels occupying the existence range of the object 106 is less than the threshold th _exit, it is determined that the object 106 does not exist. If it is determined that an object exists (Yes), the process proceeds to step S510 described below. When it is determined that the object does not exist (No), it is determined that “the pallet 107 is empty”, and the processing of the information processing apparatus ends.

  In step S510, the determination unit 307 determines the second sensor position and orientation based on the existence range of the object 106. The second sensor position and orientation is the position and orientation of the sensor unit 104 in the second measurement. Details of step S510 will be described later with reference to FIG. In step S511, the sensor position control unit 302 moves the sensor unit 104 to the second sensor position and orientation. In step S512, the sensor unit 104 captures the second pattern image and the second grayscale image, and the image acquisition unit 304 obtains the second pattern image and the second grayscale image from the sensor unit 104. The second pattern image and the second grayscale image are used for the second measurement. In step S513, the image acquisition unit 304 calculates a distance point group (second distance point group) from the second pattern image. The second distance point group is used for the second measurement.

  In step S514, the position and orientation acquisition unit 306 detects the object 106 as in step S505, and estimates the approximate position and orientation. In step S515, the position and orientation acquisition unit 306 acquires the detailed position and orientation of the object 106 to be measured as in step S507. In step S516, the hand control unit 303 drives the hand 102 based on the position and orientation of the object 106 acquired in step S507 or S515, causes the hand 102 to grip the object 106, and takes out the object 106 from the pallet 107. After the completion of step S516, the information processing apparatus 105 proceeds to step S501 again, and repeats the processing until there is no more object 106 on the pallet 107. When the object 106 whose position and orientation can be acquired in the second measurement is exhausted, the existence range of the object may be estimated from the second distance point group, and the sensor position and orientation in the next measurement may be determined.

  FIG. 6 is a flowchart illustrating the process of estimating the existence range of the object 106 in step S508. Each step in this flow can be executed by the estimation unit 308. Between the range where the object 106 exists and the range where the object 106 does not exist, a great difference occurs in the z value (value in the z direction) representing the depth in the distance point group. Therefore, in step S508, the estimation unit 308 estimates the existence range of the object 106 based on the difference between the z values. The inputs to the estimation unit 308 in step S508 are the first distance point group, the position and orientation of the pallet 107 measured in step S504, and the distance point group of the pallet 107 when the pallet 107 is empty. The distance point group (reference distance information) of the pallet 107 when the pallet is empty is measured in advance and stored in the empty pallet. Further, the distance point group of the pallet 107 when it is empty may be calculated by plane estimation using the first distance point group. The estimating unit 308 estimates and outputs the existence range of the object 106 based on the first distance point group, the position and orientation of the pallet 107, and the distance point group of the pallet 107 when the pallet 107 is empty. Hereinafter, each process of step S508 will be described.

First, in step S601, a difference point group in the z direction is acquired between the first distance point group and the distance point group of the pallet 107 when the pallet 107 is empty. The first distance point group _{^{P i measure (x i m,}} y i m, z i m) and the pallet 107 the bottom surface of the case is empty, i.e., the distance point group that has been projected on the surface of the object 106 is placed the _{^{P i empty (x i e,}} y i e, z i e) to. Also it defines the difference between point group in the z direction _{^{P i diff (x i d,}} y i d, z i d) and. At this time, i is the id of a point included in the distance point group. Also, the difference point group in the z direction _{^{is, x i d = xi m,}} y i d = y i m, is calculated as _{^{z i d = z i m -z}} i e.

Next, in step S602, the background point is removed from the difference point group P ⁱ _diff in the z direction. ^{Specifically,} when ^z _{i d} is smaller than the threshold value _{th object,} it determines that the background point. If determined to be a background point, the point P ^k _measure is not included in the existence range of the object 106. If not determined as a background point, the point P ^k _measure is included in the existence range of the object 106. At this time, P ^k _measure represents a certain point included in P ⁱ _measure . In step S603, the existence range of the object 106 is clustered. Step S603 can be realized by a known method such as the shortest distance method. In the shortest distance method, the smallest inter-sample distance between samples of two clusters is defined as the distance between clusters. In step S603, a clustering result is output.

In step S604, a noise cluster is removed from the cluster group based on the clustering result output in step S603. Specifically, first, the number of points constituting each cluster is added up. If the sum is less than the threshold th _class, it is determined to be a noise cluster. If it is determined that the cluster is a noise cluster, the point cloud forming the cluster is excluded from the range of the object. If it is not determined that the cluster is a noise cluster, the point cloud forming the cluster is not excluded from the range in which the object 106 exists.

  In step S605, point group interpolation is performed on the existence range of the object 106. Each point constituting the cluster group corresponds to only a portion that has not become a blind spot in the first pattern image. However, since the object 106 may also exist at a position where the sensor unit 104 has become a blind spot, a point group corresponding to the object 106 is interpolated. However, points that are inside the existence range of the object 106 and become blind spots from any sensor position and orientation do not contribute to the estimation of the second sensor position and orientation. Therefore, only points that are outside the existence range of the object 106 and fall within the field of view from any of the sensor positions and postures are complemented. Hereinafter, details of the flow of the process of step S605 will be described. First, in the cluster group, points forming boundaries between the inside and outside of the cluster are selected and set as boundary points. Next, a point projected on the bottom surface of the pallet 107 with respect to the selected boundary point is set as a ground point. Subsequently, points are complemented at regular intervals on a line segment connecting the boundary point and the ground point, and are set as ridge points. The existence range of the object 106 is complemented with the ground point and the ridge point, and the estimation of the existence range of the object 106 ends.

  In the present embodiment, the existence range of the object 106 is a point group as an example. However, the object 106 may be a rectangular parallelepiped or a polygonal closed space in a three-dimensional space in which a distance point group having a large z-value background difference exists. Further, a set of pixels obtained by the background difference of the grayscale image or a rectangle including the set of pixels may be used.

  FIG. 7 is a schematic diagram illustrating the process of step S510 according to the first embodiment. FIG. 7A is a diagram illustrating an example of a pallet in which an object to be measured is stored. This figure is a view of the pallet 107 viewed from the + Z direction. The pallet spaces 704a to 704e are partial spaces obtained by dividing a three-dimensional space in the pallet 107, which is a space where the object 106 is arranged. The pallet spaces 704a to 704e may be mechanically divided by the determination unit 307 such that the area of each pallet space is equal to or smaller than a predetermined value, or may include a space adjacent to each side surface inside the pallet 107. May be mechanically divided into Since the pallet spaces 704a to 704e include spaces adjacent to the respective inner side surfaces of the pallet 107, the object 106 leans on the inner side surface of the pallet 107, and the second sensor position and orientation can be set even when the pallet 107 is placed vertically. Can be determined correctly. Further, the pallet spaces 704a to 704e may be divided according to a designation from a user.

  FIG. 7B is a cross-sectional view of the pallet 107 in a state where the first measurement is being performed. FIG. 7C is a list showing the second sensor position and orientation corresponding to each of the pallet spaces 704a to 704e. The second sensor positions / postures 708a to 708e correspond to the pallet spaces 704a to 704e, respectively. The values of the second sensor positions / postures 708a to 708e may be set by the determination unit 307, for example. The determination unit 307 determines a second sensor position and orientation corresponding to a sensor position and orientation that maximizes the surface area of the object 106 that falls within the visual field range of the sensor unit 104 when the object 106 to be measured exists in the pallet space. You may set as. Further, when the object 106 to be measured is present in the pallet space, the determination unit 307 determines a second orientation corresponding to the position and orientation that maximizes the area of the object 106 in the image captured by the sensor unit 104. It may be set as a sensor position and orientation. Further, the value of the corresponding second sensor position and orientation may be set by the user.

  Here, the processing of the second sensor position and orientation will be described. Here, as an example, a case where the existence range 702 of the object 106 exists in the pallet space 704c will be described. In the second sensor position and orientation determination in step S510, the determination unit 307 selects a sensor position and orientation suitable for measuring the existence range 702 of the object 106 and its vicinity. The deciding unit 307 selects a pallet space 704c which is a pallet space including many of the existence range 702 of the object 106 from the pallet spaces 704a to 704e. Then, the second sensor position / posture 708c corresponding to the pallet space 704c is selected from the second sensor position / postures 708a to 708e, and the second sensor position / posture is determined. The determination unit 307 outputs information on the determined second sensor position and orientation to the robot control device 103.

  FIG. 8 is a flowchart illustrating the process of step S510 according to the first embodiment. Each step in this flow can be executed by the determination unit 307. First, in step S801, the three-dimensional space in the pallet 107 is divided into a plurality of pallet spaces. As described above, the second sensor position / posture corresponding to each of the divided pallet spaces is set. Next, in step S802, a pallet space to be measured in the second measurement is determined. Specifically, in each pallet space, the pallet space including the largest number of points constituting the existence range of the object 106 is set as the pallet space to be measured.

  Finally, in step S803, the second sensor position and orientation corresponding to the determined pallet space to be measured are acquired, and the second sensor position and orientation are determined. In the present embodiment, the second sensor position and orientation are determined based on the pallet space including a large range of the object 106 as an example, but may be determined based on the pallet space including the center of the range of the object 106. .

  As described above, according to the present embodiment, in order to determine the second sensor position and orientation based on the existence range of the object to be measured, for example, the object leans against the inner side surface of a container such as a pallet and is placed vertically. However, the position and orientation of the second sensor can be correctly determined. Therefore, it is possible to acquire the position and orientation of the object in the second measurement. In addition, for example, even when measuring the position and orientation of a vertically placed object, erroneous determination of the second sensor position and orientation can be reduced, and the measurement time can be shortened.

<Second embodiment>
In the first embodiment, step S510 determines the second sensor position and orientation based on the existence range of the object 106. However, for example, when the object is placed vertically at the center of the pallet 107, the object 106 may be blocked by a container such as the pallet 107 in the field of view of the sensor unit 104. In such a case, the appearance of the object 106 is greatly different depending on each of the second sensor positions and orientations, so that it is difficult to correctly determine the second sensor position and orientation only from the existing range of the object. In the second embodiment, the second sensor position and orientation are determined in consideration of the shape of the range in which the object 106 exists. Note that the configuration and processing flow of the present embodiment are the same as those of the first embodiment except for step S510, and thus description thereof is omitted.

  FIG. 9 is a flowchart illustrating the process of step S510 according to the second embodiment. Each step in this flow can be executed by the determination unit 307. In the second embodiment, a second sensor position and orientation that can measure more surfaces of the object 106 is determined in consideration of the shape of the range in which the object 106 exists.

  First, in step S901, a mesh surface is generated from a group of points in the existing range of the object 106. The determination unit 307 generates, for example, a mesh model including vertex information and information on a surface connecting the vertices. Step S901 can be realized by a known method such as an octree-based division method. The octree-based partitioning method is a method in which a boundary of a region is reproduced with voxels of an octree, each voxel is divided into tetrahedrons, and finally, tetrahedrons outside the boundaries are removed. In step S902, the generated mesh surface is simplified. Step S902 can be realized by a known method such as QSlim. The method using QSlim is a method in which local geometric / topological operations are performed using priority queues and the number of meshes is reduced while approximating the original shape.

In steps S903 to S905, it is determined whether or not each mesh surface can be measured at each second sensor position and orientation. In step S903, it is determined whether the simplified mesh surface to be determined falls within the field of view of the sensor unit 104 in the second sensor position and orientation to be determined. The visual field range of the sensor unit 104 is determined by the sensor position and orientation parameters having six degrees of freedom. In the case of the second sensor position and orientation to be determined, if the threshold th _S % or more of the mesh surface to be determined falls within the visual field range of the sensor unit 104, it is determined that the mesh surface is within the visual field and can be measured. If the threshold th _S % or more of the mesh surface to be determined does not fall within the visual field range of the sensor unit 104, it is determined that the mesh surface is out of the visual field and cannot be measured.

In step S904, it is determined whether the inclination of the sensor unit 104 between the determination target mesh surface and the determination target second sensor position and orientation is sufficiently small. If the _angle formed between the determination target mesh surface and the sensor unit 104 in the determination target second sensor position and orientation is equal to or smaller than the threshold _thangle , it is determined that the mesh surface can be measured. If the _angle formed by the determination target mesh surface and the sensor unit 104 in the second sensor position and orientation to be determined is not smaller than or equal to the threshold th _angle , it is determined that the mesh surface cannot be measured.

In step S905, it is determined whether or not the pallet 107 has sufficiently small shielding between the sensor unit 104 in the second sensor position and orientation to be determined and the mesh surface to be determined. When the ratio of the surface hidden by the pallet 107 among the mesh surfaces to be determined is equal to or smaller than the threshold _thoclusion , it is determined that the mesh surface can be measured. When the ratio of the surface hidden by the pallet among the mesh surfaces to be determined exceeds the threshold _thoclusion , it is determined that the mesh surface cannot be measured. In step S906, it is determined whether the determination of all the mesh surfaces in the second sensor position and orientation to be determined has been completed. If it is determined that the determination of all the mesh surfaces has not been completed (No), steps S903 to S905 are repeated until the determination of all the mesh surfaces is completed. If it is determined that all the mesh surfaces have been determined (Yes), the process proceeds to step S907 described below.

  In step S907, the sum of the measurable mesh surface areas in the second sensor position and orientation to be measured is calculated. The area sum of the measurable mesh surfaces includes only the areas of the mesh surfaces determined to be measurable in all of steps S903 to S905. After that, in step S908, it is determined whether the determination of all the second sensor positions is completed. If it is determined that the determination of all the second sensor positions is not completed (No), steps S903 to S908 are repeated until the determination of all the second sensor positions is completed. If it is determined that the determination of all the second sensor positions has been completed (Yes), the process proceeds to step S909 described below.

  In step S909, the best parameter is selected from the parameters of each second sensor position and orientation. The parameter of each second sensor position and orientation may be a sensor position and orientation parameter automatically generated by the determination unit 307 based on the range and interval of each parameter value of 6 degrees of freedom. Also, the second sensor position and orientation parameters listed by the user may be used. The best second sensor position / posture parameter is the second sensor position / posture parameter in which the sum of the measurable mesh surface areas is the largest and the distance between the sensor unit 104 and the object 106 is the smallest. Then, the second sensor position and orientation is determined based on the best sensor position and orientation parameters. By performing measurement with the second sensor position and orientation that can measure more surfaces of the object to be measured, the position and orientation of the object can be correctly measured.

  As described above, according to the second embodiment, for example, even if the object is placed vertically at the center of the pallet 107 and the object 106 is blocked by a container such as the pallet 107, It is possible to determine the second sensor position and orientation from which the position and orientation can be measured. Therefore, erroneous determination of the second sensor position and orientation can be reduced, and the measurement time can be reduced.

<Third embodiment>
In the second embodiment, step S510 determines the second sensor position and orientation based only on the existence range and the shape of the object. However, when a plurality of objects that are difficult to be measured by the first measurement remain on the pallet 107, there is a possibility that the second sensor position and orientation that determines only a part of each object in the field of view may be determined. In the second sensor position and orientation in which only a part of each object is included in the field of view, all the objects can be seen from the field of view, and sufficient information for obtaining the position and orientation of the object cannot be obtained. It is difficult to measure a correct position and orientation. Therefore, in the third embodiment, the second sensor position and orientation are determined using the distribution of the object existence range in addition to the object existence range and its shape. Note that the configuration and processing flow of the present embodiment are the same as those of the first and second embodiments except for step S510, and thus description thereof will be omitted.

  FIG. 10 is a flowchart illustrating the process of step S510 according to the third embodiment. Each step in this flow can be executed by the determination unit 307. In the present embodiment, the second sensor position and orientation are determined in consideration of the distribution of the existence range of the object. That is, the position and orientation of the second sensor are determined so that the existence range of the object in which the group of points is more than a certain value is always included in the field of view.

Steps S901 to S908 are the same as those in FIG. 9 in the second embodiment, and a description thereof will not be repeated. In step S1001, the number of clusters that can be measured with the second sensor position and orientation to be determined is calculated. The measurable cluster is a cluster in which the sum of the measurable mesh surfaces is _{equal to} or _larger than the threshold th _clSter among the mesh surfaces included in the cluster. In step S909, the best sensor position and orientation parameter is selected from each sensor position and orientation parameter. The best sensor position / posture parameter is the sensor position / posture parameter in which the number of the measurable clusters is the largest and the distance between the visual sensor and the object existence range is the smallest. Then, a second sensor position and orientation is determined based on the best sensor position and orientation parameters.

  As described above, according to the third embodiment of the present invention, for a plurality of objects that are difficult to measure by the first measurement, the second sensor position and orientation are determined using the distribution of the existence range of the objects. . Thereby, it is not necessary to determine an erroneous second sensor position and orientation such that only a part of the object is included in the field of view. Therefore, any object can be seen from the visual field, and the measurement does not fail. Therefore, there is an effect that the position and orientation of the object can be measured in a short measurement time.

(Embodiment related to article manufacturing method)
The information processing device 105 described above can be used for a method of manufacturing an article. In such an article manufacturing method, an object whose position and orientation has been acquired by the information processing device 105 is moved by a robot, and the moved object is processed to manufacture an article. The processing may include, for example, at least one of processing, cutting, transporting, assembling (assembly), inspection, and sorting. The article manufacturing method of the present embodiment is advantageous in at least one of the performance, quality, productivity, and production cost of the article, as compared with the conventional method.

(Other embodiments)
The embodiments of the present invention have been described above, but the present invention is not limited to these embodiments, and various changes can be made within the scope of the gist.

104 sensor unit 105 information processing device 106 object 304 image acquisition unit 306 position and orientation acquisition unit 307 determination unit 308 estimation unit

Claims (13)

Image acquisition means which is imaged by the imaging unit and acquires a captured image including distance information of the object,
Position and orientation acquisition means for acquiring the position and orientation of the object from the distance information of the object,
When an object different from the object whose position and orientation has been acquired by the position and orientation acquisition unit is included in the captured image, an estimation unit that estimates the existence range of the object from the distance information of the object,
Determining means for determining the position and orientation of the imaging unit based on the existence range of the object.   2. The information processing apparatus according to claim 1, wherein the estimating unit estimates an existence range of the object when an object whose position and orientation cannot be acquired by the acquiring unit is included in the captured image. 3.   The said estimation means estimates the existence range of the said object based on the difference of the reference distance information which is the distance information of the surface on which the said object is mounted, and the distance information of the said object. 3. The information processing device according to 1 or 2.   The information processing apparatus according to claim 1, wherein the determination unit stores a position and orientation of the imaging unit corresponding to a range in which the object exists.   The method according to claim 4, wherein the determining unit divides a space in which the object is arranged into a plurality of subspaces, and stores a position and orientation of the imaging unit corresponding to each of the plurality of subspaces. An information processing apparatus according to claim 1.   The information processing apparatus according to claim 5, wherein the partial space includes a space adjacent to an inner side surface of a container that stores the object.   The method according to claim 1, wherein the determining unit determines a position and orientation of the image capturing unit that maximizes a surface area of the object that falls within a visual field range of the image capturing unit as the position and orientation of the image capturing unit. An information processing apparatus according to claim 1.   7. The information processing apparatus according to claim 1, wherein the determination unit determines a position and orientation of the captured image at which an area of the object is maximized as a position and orientation of the imaging unit. 8. apparatus.   9. The apparatus according to claim 1, wherein the determining unit determines a position and orientation of the image capturing unit when the area of the object shielded by the container is equal to or smaller than a threshold value. 10. Information processing device. An image acquisition step of acquiring a captured image that is captured by the imaging unit and includes distance information of the object,
A position and orientation acquisition step of acquiring the position and orientation of the object from the distance information of the object,
When an object different from the object whose position and orientation is obtained in the obtaining step is included in the captured image, an estimation step of estimating the existence range of the object from the distance information of the object,
A determining step of determining the position and orientation of the imaging unit based on the existence range of the object. A program for causing a computer to function as the information processing device according to any one of claims 1 to 9. An imaging unit that captures a captured image including distance information of an object,
A position and orientation acquisition unit that acquires the position and orientation of the object from the distance information of the object,
When an object different from the object whose position and orientation has been acquired by the position and orientation acquisition unit is included in the captured image, an estimation unit that estimates the existence range of the object from the distance information of the object,
A determination unit that determines the position and orientation of the imaging unit based on the existence range of the object. A method of manufacturing an article, comprising:
An object whose position and orientation have been obtained by the information processing apparatus according to any one of claims 1 to 9 is moved by a robot,
Manufacturing the article by performing the processing of the moved object;
A method for manufacturing an article, comprising:

Images