JP2015206654A - Information processing apparatus, information processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To make it possible to determine a pattern direction suitable for accurately performing position and attitude measurement of a target object by taking the characteristics of the target object into consideration.SOLUTION: An information processing apparatus includes: image acquisition means for acquiring an image including a target object photographed by imaging means; calculation means for calculating information indicating a change in the shape of the target object; and determination means for determining the direction of a pattern to be projected on the target object by projection means on the basis of the information indicating a change in the shape.

Description

  The present invention relates to a technique for measuring a position and orientation of an object by projecting a pattern.

  In the measurement of the position and orientation of an object, a three-dimensional measurement technique using an image is widely used. In the stereo method, which is one of the representative methods of 3D measurement technology using images, the principle of triangulation is based on images taken by two cameras (stereo cameras) whose relative positions and orientations are known. Based on the 3D measurement. When using a stereo camera, it is difficult to search for a corresponding point between the cameras in an area where the luminance change on the image is small. Therefore, it is widely performed to easily search for corresponding points by projecting a pattern onto a measurement target using an illumination device such as a projector in addition to a stereo camera. In addition, three-dimensional measurement is widely performed in which one of the stereo cameras is replaced with an illumination device such as a projector, and the camera and the illumination device are used as a stereo pair.

  In stereo measurement using such an illuminating device, a pattern composed of a plurality of lines is often used as a projection pattern. In this case, the distribution and accuracy of three-dimensional measurement points obtained by measurement vary depending on the direction of the pattern. In Patent Document 1, two projectors are arranged at different positions, and a pattern is first projected from one projector and captured by a camera. It is disclosed that when a captured image includes many influences of shadows caused by a projected pattern and is determined to be unsuitable for three-dimensional measurement, the pattern is projected by the other projector.

JP-A-7-234113

  However, since the method disclosed in Patent Document 1 does not consider the three-dimensional shape of the target object, the set line pattern may not be suitable. For example, when unevenness is repeated in the direction of the surface of the target object, the surface unevenness can be measured by setting the longitudinal direction of the line pattern in a direction perpendicular to the direction in which the unevenness is repeated. Can not.

  The present invention has been made in view of the above problems, and an object of the present invention is to determine a pattern direction for suitably measuring the position and orientation of a target object.

  The information processing apparatus according to the present invention includes, for example, an image acquisition unit that acquires an image including a target object captured by an imaging unit, and a calculation that calculates information indicating a change in the shape of the target object based on the image And a determining unit that determines the orientation of the pattern projected onto the target object by the projecting unit based on the information indicating the change in shape.

  According to the present invention, it is possible to determine the optimum pattern direction for accurately measuring the position and orientation of the target object by considering the characteristics of the target object.

1 shows a configuration of a position and orientation measurement system using an information processing apparatus of the present invention. 1 shows a configuration diagram of an apparatus according to a first embodiment of the present invention. FIG. 1 shows a flowchart according to a first embodiment of the present invention. 3 shows a flowchart according to a method for calculating a distribution of shape change according to the first embodiment of the present invention. The figure which shows the process based on the 1st Embodiment of this invention is shown. The screen which shows the processing result concerning a 1st embodiment of the present invention is shown. The block diagram of the information processing apparatus of the 2nd Embodiment of this invention is shown. The flowchart which concerns on the calculation method of distribution of the shape change of the 2nd Embodiment of this invention is shown. The flowchart which concerns on the calculation method of distribution of the shape change of the 3rd Embodiment of this invention is shown. The structure of the information processing apparatus which concerns on the 5th Embodiment of this invention is shown. 10 shows an example of a screen for assisting in setting a pattern on a projector according to a fifth embodiment of the present invention. The structure of the apparatus which concerns on the 6th Embodiment of this invention is shown. 9 shows a flowchart according to a sixth embodiment of the present invention. The example of the screen which shows a user the selected screen or camera which concerns on the 6th Embodiment of this invention is shown. 1 shows a hardware configuration of an information processing apparatus of the present invention.

  Prior to describing each embodiment according to the present invention, a hardware configuration in which the information processing apparatus shown in each embodiment is mounted will be described with reference to FIG.

  FIG. 15 is a hardware configuration diagram of the information device according to the present embodiment. In the figure, a CPU 1510 comprehensively controls each device connected via a bus 1500. The CPU 1510 reads and executes processing steps and programs stored in a read-only memory (ROM) 1520. In addition to the operating system (OS), each processing program, device driver, and the like according to the present embodiment are stored in the ROM 1520, temporarily stored in a random access memory (RAM) 1530, and appropriately executed by the CPU 1510. The input I / F 1540 is input as an input signal in a format that can be processed by the information processing apparatus 1 from an external device (display device, operation device, or the like). The output I / F 1550 outputs an output signal to an external device (display device) in a format that can be processed by the display device.

  Each of these functional units is realized by the CPU 1510 developing a program stored in the ROM 1520 in the RAM 1530 and executing processing according to each flowchart described later. Further, for example, when hardware is configured as an alternative to software processing using the CPU 1510, arithmetic units and circuits corresponding to the processing of each functional unit described here may be configured.

(First embodiment)
In this embodiment, the distribution of shape changes, specifically the distribution of changes in the normal direction of the object surface, is taken into consideration as the object characteristics, and projection is performed so that distance points in the direction in which the normal direction changes rapidly can be measured closely. Determine the orientation of the pattern. In this embodiment, the direction of the pattern is determined by regarding the direction in which the normal direction changes drastically as a direction in which the two-dimensional image obtained by photographing the target object is Fourier transformed and the frequency has a large Fourier spectrum. This is because the Fourier spectrum of the frequency in the direction in which the change in the normal direction is drastically increased. In general, in active stereo 3D measurement using parallel line patterns, distance points can be measured most closely in the direction of parallel lines, so that the direction of parallel lines matches the direction in which the normal direction of the object changes drastically. Determine the orientation of the projection pattern.

  Details of this embodiment will be described below.

  FIG. 1 is a diagram illustrating an example of a configuration of a position and orientation measurement system according to the present embodiment.

  The information processing apparatus 10 according to the present embodiment is connected to a camera 110, a projection device 120, and a display device. The information processing apparatus 10 associates the pixels in the image with the pixels in the projection image of the liquid crystal projector based on the acquired pattern detection result, and calculates a three-dimensional shape by triangulation. The present embodiment is an information processing apparatus that determines the optimal pattern light projection direction for measuring the position and orientation in such a three-dimensional measurement system.

  The object 100 is a target object for measuring the position and orientation.

  The camera 110 captures an image of the target object 100. The camera 110 may be a monochrome camera or a color camera. It is assumed that internal parameters such as the focal length, image center, and lens distortion parameter of the camera 110 are calibrated in advance. It is assumed that the relative position and orientation (external parameters) of the camera 110 with respect to the projector 120 are also calibrated in advance.

  The projector 120 projects a pattern on the target object 100. For example, a liquid crystal projector is used as the projector 120, but the projector 120 is not limited to this. A pattern 121 is a pattern projected from the projected projector 120. The pattern to be projected is composed of parallel lines and codes assigned on the parallel lines so that each parallel line can be uniquely identified. Pattern projection is performed by outputting an image with a liquid crystal projector, and the direction of the pattern can be controlled by changing the output image. The relative position and orientation of the camera 110 and the projector 120 are calibrated in advance and are known.

  FIG. 2 is a diagram illustrating a configuration of the information processing apparatus 10 according to the present embodiment.

  The image acquisition unit 130 acquires an image captured by the camera 110.

  The pattern output unit 140 controls the projector 120 to project a pattern.

  The shape change calculation unit 150 calculates the distribution of the shape change of the target object. Specifically, the distribution of the shape change of the measurement target used for determining the pattern direction by processing the image of the target object is calculated.

  The pattern direction determination unit 160 determines the direction of the pattern projected by the projector 120. The pattern direction is determined based on the distribution of the shape change.

  The three-dimensional shape calculation unit 170 calculates the three-dimensional shape of the target object. Specifically, the image of the target object on which the pattern is projected in the determined direction is processed, and the three-dimensional coordinates of each position on the target object surface are calculated.

  The model holding unit 180 holds a shape model representing the three-dimensional shape of the target object. In the present embodiment, description will be made assuming that the information is stored in a storage area inside the information processing apparatus 10, but the model may be stored in an external storage medium and acquired from the storage medium.

  The position / orientation deriving unit 190 derives the position / orientation of the target object. The position and orientation of the target object are calculated by fitting the shape model of the target object to the three-dimensional coordinates of the target object calculated by the three-dimensional shape calculation unit 240.

  FIG. 3 is a flowchart showing a processing procedure according to this embodiment.

(Step S301)
In step S301, the shape change calculation unit 150 calculates the distribution of the shape change of the target object. This process will be further described in detail with reference to the flowchart of FIG.

(Step S401)
In step S401, the camera 110 captures an image of the target object in a state where no pattern is projected by the projector. FIG. 5A is a diagram illustrating an image of the target object acquired in this way.

(Step S402)
In step S402, the shape change calculation unit 150 obtains a discrete Fourier transform F (u, v) of the acquired image f (x, y). (X and y are two-dimensional coordinates of a pixel on the acquired image, f is a luminance value of the pixel, u and v are spatial frequencies in the x direction and y direction, respectively, and F is a complex number of the frequency. .)
(Step S403)
In step S403, the shape change calculation unit 150 creates a bin representing the pattern direction. Here, eight bins are created by dividing 180 degrees into 22.5 degrees. Each bin represents the pattern direction based on 0 degree, 22.5 degrees, 45 degrees, 67.5 degrees, 90 degrees, 112.5 degrees, 135 degrees, and 157.5 degrees, and the bin score is It is a numerical value indicating how well the direction is suitable for three-dimensional measurement. The score of each bin is initialized with 0. Although the number of bins is 8 in this embodiment, the number of divisions and the direction thereof may be determined according to the resolution of the pattern direction that can be switched by the projector.

(Step S404)
In step S404, the shape change calculation unit 150 calculates a Fourier spectrum at each u and v obtained by the discrete Fourier transform, and adds it to the bin score corresponding to u and v. Here, the Fourier spectrum is the absolute value of the complex number F in u and v.

であり

で定義される値である。ただし、Ｒｅ｛Ｆ（ｕ、ｖ）｝、Ｉｍ｛Ｆ（ｕ、ｖ）｝はそれぞれＦ（ｕ、ｖ）の実部と虚部を表す。こうして全てのデータにおいて、対応するｂｉｎのスコアにフーリエスペクトルの値を加算したものを形状変化の分布とする。図５（ｂ）は離散的フーリエ変換の結果をｕ、ｖ平面上にフーリエスペクトルを輝度として画像にしたものである。この図上で線分で区切られた領域４００〜４７０がそれぞれ上記で説明したｂｉｎの０度、２２．５度、４５度、６７．５度、９０度、１１２．５度、１３５度、１５７．５度に対応している。このようにして各領域において輝度（フーリエスペクトル）を加算したものを形状変化の分布とする。図５（ｃ）は形状変化の分布をヒストグラムにして図示したものである。

And

It is a value defined by. However, Re {F (u, v)} and Im {F (u, v)} represent a real part and an imaginary part of F (u, v), respectively. In this way, in all the data, the distribution of the shape change is obtained by adding the Fourier spectrum value to the corresponding bin score. FIG. 5B shows the result of the discrete Fourier transform as an image on the u and v planes with the Fourier spectrum as the luminance. In this figure, regions 400 to 470 divided by line segments are 0 degrees, 22.5 degrees, 45 degrees, 67.5 degrees, 90 degrees, 112.5 degrees, 135 degrees, and 157 of bin described above. Corresponds to 5 degrees. A shape change distribution is obtained by adding luminance (Fourier spectrum) in each region in this way. FIG. 5C shows the distribution of the shape change as a histogram.

  The description will be returned to the flowchart of FIG.

(Step S302)
In step S302, the pattern direction determination unit 160 determines the direction represented by bin having the highest score in the distribution of shape changes as the pattern direction. In the example of FIG. 5C, the 0 degree direction is acquired. In this embodiment, the pattern direction is determined based only on the bin score, but an index other than the bin score may be used in combination. For example, in the case of a liquid crystal projector, anti-aliasing is applied depending on the pattern direction, and the pattern image is blurred. Therefore, a direction in which anti-aliasing is not applied may be preferentially selected from directions having a good bin score.

(Step S303)
In step S303, the pattern output unit 140 projects the pattern in the direction determined in step S302. In the liquid crystal projector, since the image output to the built-in liquid crystal panel becomes the projection pattern as it is, an image in the determined pattern direction is generated and output to the liquid crystal projector to project the pattern in the determined direction.

(Step S304)
In step S304, the image acquisition unit 130 acquires an image of the target object on which the pattern is projected.

(Step S305)
In step S305, the three-dimensional shape calculation unit 170 processes the acquired image to calculate the three-dimensional shape of the target object, specifically, the distance point group on the surface of the target object.

(Step S306)
In step S <b> 306, the position / orientation deriving unit 190 acquires the shape model of the target object from the model holding unit 180.

(Step S307)
In step S307, the position and orientation deriving unit 190 calculates the position and orientation of the target object by fitting the shape model of the target object to the distance point group of the target object. Specifically, the model fitting uses a known ICP algorithm to calculate a position and orientation that minimizes the three-dimensional distance between the distance point group of the target object and the model. At this time, if the distance point group of the target object is not flat and has a flat surface, the position in the plane is difficult to be determined. In the present embodiment, as described above, the position is easily determined by densely measuring the distance point group in the direction in which the unevenness of the target object is intense, and the position and orientation measurement accuracy can be improved.

  In the present embodiment, the pattern is a pattern of a plurality of parallel lines. However, the present invention can be implemented with other geometric patterns as long as the measurement density varies depending on the projection direction. For example, a pattern in which the color of each line of the parallel lines is changed may be used instead of adding a code to identify the parallel lines. Furthermore, the three-dimensional measurement is not limited to a method in which a single pattern is irradiated and photographed, but a plurality of patterns may be projected and photographed. For example, a plurality of binary patterns having different line widths used in the spatial encoding method may be used, or a plurality of sine wave patterns used in the phase shift method may be used.

  In this embodiment, the bin is divided exclusively without overlapping each other. However, the bin may be set so as to overlap with a nearby bin. For example, when the projector pattern direction switching resolution is 1 degree, 180 bins may be created for the reference direction one degree at a time, and the width of each bin may be 30 degrees, for example. By doing so, it is possible to exclude a case where a bin whose score is accidentally high due to the influence of noise or the like is inappropriately selected when the number of bin divisions increases.

  In this embodiment, the direction represented by the bin with the highest score is set as the pattern direction. However, actually, there are cases where candidates for directions that can be projected are narrowed down in advance due to restrictions on the hardware of the projector and restrictions on the position of the camera. . For example, if the direction of the epipolar line for searching for the corresponding point in the three-dimensional measurement and the direction of the parallel line of the pattern are close to parallel, the accuracy of detecting the corresponding pixel between the camera image and the projection image of the liquid crystal projector is lowered. The pattern direction is not appropriate. In the first place, there may be a case where the pattern direction cannot be set in any direction due to hardware restrictions. In consideration of such a situation, a range in which the pattern direction can be set is determined in advance, and the direction represented by the bin having the highest score in the range may be determined as the pattern direction. In addition to the direction represented by the bin with the highest score, the direction corresponding to the bin with the second and third highest scores is also stored as a pattern direction candidate and presented to the user, and the final judgment is made based on the user instruction. May be determined.

  In the present embodiment, a method has been described in which the projector is a liquid crystal projector, and the pattern projection direction is switched by switching the output to the liquid crystal panel. However, the present invention is not limited to this. That is, a means for mechanically switching the direction of the pattern to be projected onto the target object may be provided. For example, a pattern projection direction may be controlled by mechanically rotating a housing of a liquid crystal projector or a liquid crystal panel. If the projector is not a liquid crystal projector but a method that projects the pattern by passing the irradiation light through the slit plate with the pattern engraved, the projection direction of the pattern can be changed by mechanically rotating the slit plate. Also good. Furthermore, the pattern projection direction on the target object may be switched by rotating the platform on which the target object is installed without changing the pattern projection direction.

  As described above, the position and orientation measurement can be performed with high accuracy by determining the direction of the pattern to be projected so that the direction in which the unevenness of the object is intense can be densely three-dimensionally measured.

[Modification 1-1]
In the first embodiment, in step S301, the shape change calculation unit 150 regards the distribution of the shape change as the distribution of the change in the normal direction, and uses the distribution of the change in the normal direction as a discrete Fourier transform of the image of the target object. The example of calculating by the above has been described. However, the calculation method is not limited to this method as long as the distribution of changes in the normal direction is obtained. For example, the direction in which the normal direction of the target object changes drastically may be determined on the basis of the luminance gradient of the image, assuming that the luminance change of the observed image also increases. In this case, the luminance gradient vector at each pixel of the image obtained by photographing the target object is calculated, and the shape change distribution is obtained by adding the magnitude of the luminance gradient vector to the bin score corresponding to the direction of the luminance gradient vector. .

[Modification 1-2]
In the first embodiment, in step S301, the shape change calculation unit 150 calculates the distribution of shape change from the entire image of the target object. However, if something other than the target object appears in the image, the position of the target object in the image is narrowed down first by template matching, etc., and the pattern direction is determined by calculating the distribution of shape change only for that region. May be. By doing this, it is possible to eliminate the influence of objects other than the target object and the background. Further, when there are a plurality of target objects in the image and the regions can be separated, the pattern direction may be determined by dividing the region and calculating the distribution of the shape change for each region. In this case, if the projector is a device that can switch the pattern direction for each area, such as a liquid crystal projector, it is possible to measure with the optimum pattern direction for each area in one measurement.

[Modification 1-3]
In the first embodiment, the method of measuring the three-dimensional shape of the target object in the determined pattern direction has been described. However, a means for presenting the determined pattern direction to the user may be included. In this case, the pattern output unit 140 outputs the determined pattern direction to an external display device, and the display device displays a screen that presents the pattern direction.

  FIG. 6 is an example of a display screen that presents the determined pattern to the user. An area 500 is an area indicating a pattern candidate projection direction, and displays an image of a target object captured by a camera. A line 510 represents a pattern projected in the determined pattern direction. As shown in this figure, a pattern may be superimposed on the image of the target object, or if the position and orientation of the target object and the shape model are known, the target object and pattern viewed from the projector or camera will be simulated The image rendered with may be displayed. The area 520 is an area for displaying a numerical value indicating the projection direction of the pattern selected in the area 500.

  It is possible to present the pattern direction to the user by illustrating how the pattern thus determined is projected onto the target object.

  The screen shown here is an example, and the components and layout of the screen are not limited to this example as long as the screen displays the pattern direction. In addition, the second and third candidate directions obtained from the distribution of shape change when determining the pattern direction may be switched and displayed by pressing a button provided on the screen, or a list may be displayed. You may display.

(Second Embodiment)
In the first embodiment, the distribution of the shape change is calculated based on the two-dimensional image obtained by photographing the target object. In the present embodiment, a method for calculating the distribution of the shape change based on the three-dimensional shape of the target object will be described. . In this embodiment as well, in the same way as in the first embodiment, the distribution of the shape change, specifically the distribution of the change in the normal direction of the object surface, is taken into consideration as the object characteristics, Determine the orientation of the projection pattern so that it can be measured closely. In this case, the direction of the pattern is determined by subjecting the distance image generated from the position and orientation of the target object and the shape model to the direction in which the normal direction is drastically changed, and considering the direction having a large frequency Fourier spectrum as the direction. Thus, by calculating the distribution of the shape change based on the three-dimensional shape instead of the two-dimensional image of the target object, the direction in which the unevenness of the object is intense can be determined more accurately as the pattern direction.

  The configuration of the three-dimensional measurement system according to the present embodiment is shown in FIG. 1, and the configuration diagram of the information processing apparatus 20 is shown in FIG. This embodiment is different from the first embodiment in the process of step S301 by the shape change calculation unit 220. Hereinafter, this difference will be described.

  FIG. 7 is a flowchart showing a detailed procedure when processing step S301 is performed by the method according to the present embodiment.

(Step S801)
In step S <b> 801, the shape change calculation unit 750 acquires the shape model of the target object from the model holding unit 780.

(Step S802)
In step S <b> 802, the image acquisition unit 730 acquires an image of the target object from the camera 110.

(Step S803)
In step S803, the shape change calculation unit 750 calculates (estimates) the position and orientation of the target object by template matching. Specifically, images of various postures of the target object are generated in advance based on the shape model of the target object, and set as a template image. The template image is slid on the image of the target object acquired in step S802 and collated, and the posture of the target object that generated the best matching template image and its detected position are set as the position and posture of the target object.

(Step S804)
In step S804, the shape change calculation unit 750 generates a distance image of the object viewed from the camera by simulation. Specifically, using the position and orientation of the target object calculated up to step S703 and the shape model information, the distance of the object surface that collides in the light ray direction of each pixel when the image is captured by the camera is calculated and used as the value of each pixel. The distance image.

(Step S805)
In step S805, the shape change calculation unit 750 obtains a discrete Fourier transform F (u, v) of the distance image f (x, y). (X and y are two-dimensional coordinates of a pixel on a distance image, f is a distance value of the pixel, u and v are spatial frequencies in the x and y directions, respectively, and F is a complex number of the frequency. .)
In steps S806 and S807, processing similar to that in steps S403 and S404 is performed on the discrete Fourier transform result of the distance image, respectively.

  In addition to the above processing, the pattern orientation can be determined and the position and orientation of the target object can be measured by performing the same processing as in the first embodiment.

  In the present embodiment, the position and orientation of the target object in the image is calculated by template matching. However, the position and orientation calculation method is not limited to this method. For example, a more accurate position and orientation is obtained by fitting the shape model of the target object to the three-dimensional measurement result of the target object measured in the temporary pattern direction. Then, a distance image may be generated by simulation based on the obtained position and orientation, and a Fourier transform may be performed to calculate a shape change distribution. Furthermore, the position and orientation of the target object may be obtained directly from an external supply device (such as a robot).

  As described above, in order to calculate the pattern direction more accurately so that three-dimensional measurement can be performed densely in the direction where the unevenness of the object is intense, the method of determining based on the three-dimensional shape instead of the two-dimensional image of the target object is explained. did. By closely measuring the distance point group in a direction in which the unevenness of the target object is intense, the position can be easily determined, and the position and orientation measurement can be made highly accurate.

(Third embodiment)
In this embodiment, in order to accurately measure the position and orientation, it is desirable to measure the three-dimensional shape so that the accuracy of the three-dimensional distance point is increased. In general, in active stereo 3D measurement using a pattern of parallel lines, if the surface on which the pattern hits tilts in a direction perpendicular to the direction of the parallel lines of the pattern, the light line becomes thicker, and the line is drawn from the image. The accuracy of detection decreases. On the other hand, when the pattern is inclined in the direction of the parallel lines, the thickness of the light line does not change, and the detection accuracy does not decrease. Therefore, in the present embodiment, the distribution of the shape change, specifically, the distribution in the normal direction of the surface of the target object hit by the pattern is considered as the object characteristic. Then, a method for performing three-dimensional shape measurement with high accuracy by determining the projection direction of the pattern so that the direction of inclination of the target object and the direction of the parallel lines are the same will be described.

  Since the configuration of the three-dimensional measurement system according to this embodiment and the configuration diagram of the information processing apparatus included in the system configuration are the same as those in FIGS. 1 and 7 shown in the second embodiment, the numbers of the functional units are In the following description, the same numbers as those in the first embodiment are given. In the present embodiment, the processing in steps S803 to S805 by the shape change calculation unit 750 of the second embodiment is different. Hereinafter, this difference will be described.

  FIG. 9 is a flowchart showing a detailed procedure when step S301 is performed by the method according to the present embodiment.

(Step S901)
In step S901, the shape change calculation unit 750 acquires a shape model of the target object.

(Step S902)
In step S902, the shape change calculation unit 750 calculates the position and orientation of the target object. Since this calculation method is the same as the processing in step S803 described in the second embodiment, description thereof is omitted here.

(Step S903)
In step S903, the shape change calculation unit 750 calculates the direction of the surface normal when the surface of the target object is observed from the projector. Specifically, using the position and orientation of the target object calculated up to step S902 and the shape model information, the normal direction of the object surface that collides with the light ray direction of each pixel when the image is captured by the camera is calculated.

(Step S904)
In step S904, the shape change calculation unit 750 creates a bin representing the pattern direction. Here, eight bins are created by dividing 180 degrees into 22.5 degrees. The score of each bin is initialized with 0.

(Step S905)
In step S905, the shape change calculation unit 750 calculates the tilt direction with respect to the camera viewing direction of the normal direction of the target object in each pixel calculated in step S803. Then, 1 is added to the score of bin corresponding to the tilt direction. In this way, the sum of the bin scores in all the pixels is used as the distribution of shape change.

  Other than the above processing, the same processing as in the first embodiment is performed. In step S302, as in the first embodiment, the pattern direction determining unit 160 determines the direction represented by bin having the highest score in the distribution of shape change as the pattern direction. Thereby, the inclination direction of the target object and the direction of the parallel line become the same direction, and the position and orientation of the target object can be accurately measured.

  In this embodiment, 1 is added to the score of bin corresponding to the tilt direction in each pixel. However, the greater the tilt angle with respect to the projector, the thicker the pattern and the lower the measurement accuracy. Therefore, instead of adding the same value uniformly, for example, the weight may be added by adding the tilt angle in the tilt direction.

  In this embodiment, the distribution in the normal direction is calculated based on the shape model and the position and orientation of the target object. However, the method for calculating the distribution in the normal direction is not limited to this method. For example, a normal may be calculated by applying a mesh to a three-dimensional point group obtained by separately performing three-dimensional measurement.

  As described above, the method of increasing the accuracy of three-dimensional measurement and increasing the accuracy of position and orientation measurement by determining the pattern direction so that the direction of the parallel line is the same as the direction of inclination of the surface of the target object hit by the pattern Explained.

(Fourth embodiment)
The configuration of the information processing apparatus of this embodiment is the same as that of the information processing apparatus 10 according to the first embodiment of FIG. In the model fitting performed by the target object position / orientation calculation unit 260 in step S307 of the first embodiment, the method of minimizing the three-dimensional distance between the distance point group of the target object and the model has been described. , Fitting in different ways.

  In this embodiment, when there is a texture on the surface of the target object, the direction of the pattern to be projected when the model fitting for the two-dimensional image and the distance point group as described above is performed hybridly on the target object is determined. How to do will be described.

This embodiment is different in the processing of step S301, step S302, and step S307 of the first embodiment.
(Step S301)
In step S301, the shape change calculation unit 150 calculates a gradient change image, that is, an edge image, based on the acquired image of the target object.

(Step S302)
In step S302, the pattern direction determination unit 160 determines the direction of the pattern to be projected based on the change in the luminance gradient of the two-dimensional image obtained in step S301. In the present embodiment, as described above, since the fitting method performed in step S307 is different from that in the first embodiment, a different process is performed for determining the pattern direction.

  That is, in this embodiment, the fitting between the edge image and the model and the fitting between the distance point group and the model are performed in a hybrid manner. Specifically, the three-dimensional shape of the target object, that is, the three-dimensional distance between the distance point cloud and the model, and the two-dimensional distance between the edge on the edge image of the target object and the model projected on the edge image are minimized simultaneously. To calculate the position and orientation.

  A method of determining the pattern direction in such hybrid model fitting will be described below.

  First, in the model fitting based on the luminance gradient for the two-dimensional image, the position and orientation in the direction in which the change in the luminance gradient is large can be accurately determined, but on the other hand, the position and orientation in the direction in which the change in the luminance gradient is small is difficult to be determined. Therefore, in the hybrid model fitting method of this embodiment, the three-dimensional shape (distance point group) is calculated densely by aligning the pattern direction of the projection light with the direction in which the change in the luminance gradient of the two-dimensional image is small. By doing so, it is possible to compensate for the insufficient accuracy of the position and orientation when fitting only to a two-dimensional image, and to improve the accuracy of the finally obtained position and orientation.

  For the calculation of the direction in which the luminance gradient change of the two-dimensional image is small, a bin corresponding to the pattern direction is created as in step S403 of the first embodiment. Next, calculate the vector of the luminance gradient change at each pixel of the image of the target object and add the magnitude of the luminance gradient change vector to the bin score corresponding to the direction of the luminance gradient change vector. The distribution of shape change is selected, and the direction with the smallest score in this distribution is selected. Although the pattern direction obtained in this way is different from the pattern direction obtained by the method described in the first to third embodiments, it is possible to compensate for the insufficient accuracy of the position and orientation calculated by fitting to the two-dimensional image. In the hybrid fitting method according to this embodiment, the pattern direction is optimum.

  In this embodiment, the change in the luminance gradient is calculated and the pattern direction is determined based on the change in the luminance gradient. However, the luminance gradient is calculated and the pattern direction is calculated based on the luminance gradient. It doesn't matter.

(Step S307)
In step S307, the position / orientation deriving unit 190 derives the position / orientation of the target object using the hybrid fitting method described above.

  As described above, in the present embodiment, in the position and orientation measurement system that performs model fitting for two-dimensional images and distance images in a hybrid manner, the pattern direction is determined so that directions in which the position is difficult to be determined by fitting two-dimensional images can be densely measured in three dimensions. To do. Thereby, the position and orientation measurement can be made highly accurate.

(Fifth embodiment)
As shown in FIG. 10, the information processing apparatus 90 of the present embodiment has a display device 91 connected to the outside. In a system in which external parameters of the projector are not determined, the pattern direction determined in step S303 of the first embodiment cannot be automatically set in the projector. Such a system may have a function of assisting the user in setting the pattern direction determined by the method according to any of the first to fourth embodiments to the projector. In the present embodiment, an implementation method including means for assisting the user to set the determined pattern direction for the projector will be described.

  FIG. 11 is an example of a screen that assists in setting the projection direction of the actual pattern to the pattern direction determined while confirming the state of the target object displayed on the display screen of the display device 91. The display device 91 receives the pattern output from the pattern output unit 940 and displays it on the display screen. In this respect, the pattern output unit functions as a display control unit.

  An area 600 is an area for displaying a target object photographed by the camera. The pattern 610 is a pattern that is displayed as a sample to be set and determined by the method according to any of the first to fourth embodiments. The pattern 620 is a pattern that is actually projected from the projector at the current set angle. The slider bar 630 is a slider bar that adjusts the pattern direction of the projector, and can change the pattern direction of the projector in conjunction with the value of the slider bar. The user can adjust the direction of the pattern by adjusting the slider bar 630 so that the pattern 620 overlaps the pattern 610 while checking on the screen. A numerical value 640 is a numerical value indicating the direction of the pattern projected by the current projector.

  In a system that does not have a mechanism for switching the pattern direction, the user may manually set the pattern direction of the projector while checking the screen without using the slider bar 630.

  This screen is only an example, and if the screen has a configuration that can assist the user by displaying information for setting the projector so that projection can be performed in the determined pattern direction, the components and layout of the screen are this example Not limited to.

  As described above, the present embodiment has explained that the screen for assisting the user to set the pattern direction determined by the method according to any one of the first to fourth embodiments to the projector is displayed. Thus, the user can efficiently set the pattern direction of the projector even when using a device that does not have a mechanism for switching the pattern direction or in a system in which external parameters of the projector are not determined.

(Sixth embodiment)
In the first to fifth embodiments, the case where there is one camera in the configuration of the three-dimensional measurement system has been described, but in this embodiment, the case where there are a plurality of cameras will be described. In the configuration using a plurality of cameras, when measuring the three-dimensional shape of the target object by projecting the pattern in the direction determined by the method according to any of the first to fifth embodiments, the three-dimensional shape measurement is performed. It is necessary to provide means for determining an image to be performed. The image is determined on the basis of suitability in calculating the three-dimensional shape of the image. In active stereo 3D measurement using a parallel line pattern, the correspondence can be obtained more accurately as the direction of the epipolar line for searching the correspondence of stereo in the image and the pattern direction of the parallel line are closer to 90 degrees. Therefore, in the present embodiment, it is assumed that the aptitude in the calculation of the three-dimensional shape is higher as the direction of the epipolar line in the image and the pattern direction of the parallel line are closer to 90 degrees. For each image that can be acquired from a plurality of cameras, the suitability for calculating the three-dimensional shape is calculated, and the image having the highest suitability is determined as the image to be used for the three-dimensional shape calculation. Hereinafter, a method for calculating the position and orientation of the target object according to the present embodiment will be described. In this embodiment as well, in the same way as in the first embodiment, the shape change distribution, specifically the distribution change in the normal direction of the object surface, is taken into consideration as the object characteristics, and the direction in which the normal direction change is significant. The direction of the projection pattern is determined so that can be measured closely.

  FIG. 12 shows the configuration of the information processing apparatus 60 according to the present embodiment. In the present embodiment, a plurality of cameras 110 are connected to the image acquisition unit 1230. The image determination unit 1291 determines an image used for calculation of a three-dimensional shape based on the pattern direction currently projected from images acquired by a plurality of cameras. The target object three-dimensional shape calculation unit 1270 calculates a three-dimensional shape using the selected image. Other device configurations are the same as those in the first embodiment.

  FIG. 13 is a flowchart showing a processing procedure according to this embodiment. Hereinafter, a specific processing procedure according to the present embodiment will be described with reference to this flowchart.

(Step S1301)
In step S1301, the shape change calculation unit 1250 calculates the distribution of shape changes. The sum of the shape change distributions of the target object calculated by any of the methods described in the above embodiments for each image acquired by a plurality of cameras is used as the shape change distribution in this embodiment.

  Steps S1302 and 1303 are the same processes as steps S302 and S303, respectively, and are realized by any of the methods of the embodiments described above.

(Step S1304)
In step S1304, the image acquisition unit 1230 acquires images with a plurality of cameras.

(Step S1305)
In step S1305, the image determination unit 1291 selects an image to be used for 3D shape calculation. First, the direction of the pattern in the image acquired by each camera is obtained. This direction is calculated based on the pattern direction determined in step S1302 and the relative position and orientation between the projector and the camera, which are obtained in advance by calibration. Next, the direction of the epipolar line used for stereo correspondence is obtained. This direction is calculated from the relative position and orientation of the camera and projector. This relative position and orientation is also obtained in advance by calibration. In this way, the angle formed by the direction of the pattern of parallel lines and the direction of the epipolar line in the image is obtained for each camera image, and the angle of the camera closest to 90 degrees is determined.

  The processing after step S1306 is the same processing as the processing after step S305, and is realized by any method of the embodiments described so far.

  In step S1301 of this embodiment, the distribution of the shape change is calculated by acquiring images with a plurality of cameras, but it is not always necessary to use a plurality of cameras, and is suitable for acquiring an image for determining the pattern direction. A single camera may be selected. For example, the orientation of the pattern may be determined using a distribution of shape changes obtained using an image acquired by a camera closest to the projector.

  In the present embodiment, the method of determining the image of one camera on the assumption that stereo correspondence between the projection image of the projector and the image of the camera has been described. However, the present invention can be similarly implemented even when a combination of cameras is selected without using a projector for stereo correspondence. In that case, the angle formed between the direction of the pattern of parallel lines and the direction of the epipolar line may be calculated in the same way for all combinations of cameras, and the combination of cameras with the calculated angle closest to 90 degrees may be selected.

  In this embodiment, the direction of the pattern in the image is obtained by geometric calculation based on the relative position and orientation of the projector and the camera. However, the image is actually acquired by a plurality of cameras, and the pattern is detected more accurately from each image. Alternatively, the direction of the pattern in the image may be calculated.

  When the image used in the present embodiment is statically obtained from the pattern direction and the relative position and orientation of the projector and the camera, the image captured by the other camera is captured by capturing the image only with the camera capturing the image. This process may be omitted to improve efficiency.

  In the present embodiment, the method of determining an image in which the angle between the direction of the epipolar line for performing the stereo correspondence search and the direction of the pattern of the parallel lines is close to 90 degrees has been described. However, it is not always necessary to select images. The weight is determined based on the angle between the epipolar line direction and the pattern direction, and the three-dimensional shape calculation result using the images of a plurality of cameras is used. You may integrate.

  As described above, by selecting an image in which the angle between the direction of the epipolar line for performing the stereo correspondence search and the direction of the pattern of the parallel lines is close to 90 degrees, the accuracy of the three-dimensional shape calculation is improved, and the position and orientation measurement is performed. High accuracy can be achieved.

[Modification 6-1]
In active stereo 3D measurement, 3D points cannot be measured in areas where specular reflection is difficult to detect from an image or areas where the pattern does not reach. Therefore, in this modification, an example in which a measurable area is considered in addition to the angle formed by the pattern direction and the epipolar line direction in the image described above will be described as suitability for calculation of a three-dimensional shape.

  Specifically, in step S1304, a plurality of images in which the angle formed by the pattern direction and the epipolar line direction is larger than a predetermined angle are secured, and the image having the smallest non-measurable region due to specular reflection or shadow is selected from these images. To do. The non-measurable area is obtained by photographing the target object with the camera with all pixels of the projector turned on, and the area where the luminance in the image is larger than the predetermined value is set as the area that cannot be measured by specular reflection, and the area that is smaller than the predetermined value is shadowed. Is determined as an unmeasurable area.

  In the above, the non-measurable area was calculated based on the brightness of the image. However, if the shape and surface properties of the target object and the position and orientation of the projector and camera can be obtained separately, geometrically calculated based on these information. You may do it. In this case, if the light reflected from the projector as viewed from the camera is reflected from the surface of the target object and then the reflected area on the target object enters the camera from the front, the specular reflection occurs there. An area that has occurred and cannot be measured. An area that is not reflected by the surface of the target object and does not reach the light from the projector and that can be observed from the camera is determined as an area that cannot be measured by a shadow.

  In the above description, a camera image in which the angle formed by the pattern direction and the epipolar line direction is close to 90 degrees and the non-measurable area is the smallest is selected. However, it is not always necessary to select one image, and the image to be used may be switched for each region. For example, first, an image of a camera whose angle between the pattern direction and the epipolar line direction is closest to 90 degrees is selected. Then, the measurable area in the image is calculated by using the image, and the three-dimensional shape is calculated. The non-measurable area is another camera whose angle is as close to 90 degrees as possible, and the area where the area can be measured. What is necessary is just to calculate a three-dimensional shape using an image. Furthermore, not only the image used for the three-dimensional shape may be switched for each region, but the pattern direction of the projection light may be switched.

  As described above, by calculating the measurable area based on the brightness of the image, even if there is a specular reflection or shadow area where it is difficult to measure the target object, the three-dimensional shape of the target object is measured and positioned with high accuracy. Explained how to measure posture.

[Modification 6-2]
The image determination unit 1291 described in this embodiment may present the camera determined in step S1305 and the image thereof to the user.

  FIG. 14 shows an example of a screen that presents the camera determined in step S1305 and its image to the user. An area 700 is an area for displaying an image of a camera currently selected by a button 760 for selecting a candidate camera. A pattern 710 is a projected pattern, and an area 720 indicates the direction of an epipolar line for searching for stereo correspondence between the camera and the projector. An area 730 is an area for displaying the determined attribute of the image, and displays the number of the camera that captures the image, the pattern direction in the image, the epipolar search direction, the angle formed by the epipolar search direction and the pattern direction, and the like. An area 740 displays an arrangement of projectors and cameras that are components of the system. If the shape and position / posture of the target object can be obtained separately, the target object is also displayed. In the figure, the camera is highlighted and displayed so that the camera taking the image determined in step S1305 can be seen (750). Reference numeral 760 denotes a button for selecting a candidate camera. The candidates are arranged in the order of suitability in the calculation of the three-dimensional shape of the image of the camera calculated in S1305. By selecting with this button, an arbitrary camera and image can be displayed in the area 700 and the area 740.

  As described above, the method of presenting the camera and the image determined by the image determination unit 260 in step S1305 so that the user can confirm has been described.

(Other embodiments)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

<Definition>
The object characteristic in the present invention may be any object characteristic that characterizes a result obtained by three-dimensional measurement. An example is the distribution of shape change in the first to third embodiments.

  The distribution of the shape change of the object in the present invention may be any as long as it characterizes the result obtained by the three-dimensional measurement due to the shape of the object. One example is the distribution of changes in the normal direction of the first and second embodiments and the distribution of the normal direction of the third embodiment.

  The distribution of the change in the normal direction in the present invention includes a value correlated with the change in the normal direction obtained from the characteristics of the two-dimensional image if it is difficult to directly calculate. One example is the intensity distribution of the Fourier spectrum of the Fourier transform of the two-dimensional image of the target object of the first embodiment and the intensity distribution of the gradient of the two-dimensional image. In addition, the intensity distribution of the Fourier spectrum of the Fourier transform of the distance image of the target object according to the second embodiment is an example.

  An image suitable for calculation of a three-dimensional shape in the present invention may be any scale as long as the conditions for calculating the three-dimensional shape with the image are suitable. An example is based on the angle formed by the direction of the epipolar line for performing the stereo correspondence search in the image described in the sixth embodiment and the direction of the pattern of the parallel lines, and the accuracy of calculation of the three-dimensional shape is taken as a scale. Another example of the scale is that the size of a specular reflection or shadow region in which it is difficult to measure the target object in the image described in [Modification 6-1] is small.

DESCRIPTION OF SYMBOLS 110 Camera 120 Projector 130 Image acquisition part 140 Pattern output part 150 Shape change calculation part 160 Pattern direction determination part 170 Three-dimensional shape calculation part 180 Model holding part 190 Position and orientation derivation part

Claims (19)

Image acquisition means for acquiring an image including a target object imaged by the imaging means;
Calculation means for calculating information indicating a change in the shape of the target object based on the image;
An information processing apparatus comprising: a determining unit that determines a direction of a pattern to be projected onto the target object by a projecting unit based on information indicating the change in shape.   The information according to claim 1, wherein the calculation unit estimates a position and orientation of the target object based on the image, and calculates information indicating a change in the shape based on the position and orientation. Processing equipment.   3. The information according to claim 1, wherein the information indicating the change in shape is a distribution in a normal direction of a surface of the target object or a distribution of a change in a normal direction of the surface of the target object. Processing equipment.   4. The determination unit according to claim 1, wherein the determining unit determines a direction in which a change in shape appears more as a direction of the pattern based on information indicating the change in the shape. 5. Information processing device.   The said determination means determines the direction of the said pattern based on the information which shows the change of the said shape, and the direction of the pattern which the said projection means can project, The one of the Claims 1 thru | or 4 characterized by the above-mentioned. The information processing apparatus described. The distance point used in the position and orientation measurement system for acquiring the position and orientation of the target object by associating the two-dimensional image including the target object with each of the distance point groups indicating the target object and the model of the target object An information processing apparatus for determining a pattern direction for acquiring a group,
Image acquisition means for acquiring an image including a target object imaged by the imaging means;
Derivation means for deriving a change in luminance gradient of the image based on the image;
An information processing apparatus comprising: a determining unit that determines a direction of a pattern to be projected onto the target object by a projecting unit based on the derived change in luminance gradient.   The information processing apparatus according to claim 6, wherein the determination unit determines a direction in which the luminance gradient appears less as a direction of the pattern.   The information processing apparatus according to claim 1, further comprising a display control unit that causes a display unit to display a direction of the pattern determined by the determination unit.   The information processing apparatus according to claim 1, further comprising a projection unit that projects a pattern onto the target object in a pattern direction determined by the determination unit. A projection unit that projects a pattern having a pattern orientation determined by the determination unit; and a plurality of imaging units that capture the target object on which the pattern is projected,
The image acquisition unit acquires a plurality of images captured by the plurality of imaging units,
10. The apparatus according to claim 1, further comprising an image determination unit that determines an image to be used for three-dimensional measurement of the target object from the plurality of acquired images. Information processing device.   The image determining means determines an image to be used for three-dimensional measurement of the target object from the plurality of acquired images based on the arrangement of each of the plurality of imaging devices and the projection device. The information processing apparatus according to claim 10.   The image determining means derives a region in the image where the three-dimensional measurement of the target object is impossible in each of the plurality of acquired images, and based on the size of the derived region, the target object The information processing apparatus according to claim 10, wherein an image to be used for three-dimensional measurement is determined.   The information processing apparatus according to claim 9, further comprising: a display control unit that causes a display unit to display a photographing unit that captures an image determined by the image determination unit.   The information processing apparatus according to claim 10, further comprising a unit that derives a three-dimensional shape of the target object based on the image determined by the image determination unit. In an information processing apparatus that measures a three-dimensional shape of a target object based on an image obtained by projecting a pattern onto a target object by a projecting unit and imaging the target object on which the pattern is projected.
Means for calculating characteristics of the target object;
An information processing apparatus comprising: a determination unit that determines a direction of a pattern projected by the projection unit based on the characteristic. An image acquisition step of acquiring an image including the target object imaged by the imaging means;
A calculation step of calculating information indicating a change in the shape of the target object based on the image;
An information processing method comprising: a determining step of determining a direction of a pattern projected onto the target object by a projecting unit based on information indicating the change in shape. The distance point used in the position and orientation measurement system for acquiring the position and orientation of the target object by associating the two-dimensional image including the target object with each of the distance point groups indicating the target object and the model of the target object An information processing method for determining a direction of a pattern for acquiring a group,
An image acquisition step of acquiring an image including the target object imaged by the imaging means;
A derivation step for deriving a luminance gradient of the image based on the image;
And a determining step of determining a direction of a pattern to be projected onto the target object by a projecting unit based on the derived luminance gradient. In an information processing method for measuring a three-dimensional shape of a target object based on an image obtained by projecting a pattern onto a target object by a projecting unit and imaging the target object on which the pattern is projected,
Calculating the characteristics of the target object;
A determination step of determining a direction of a pattern projected by the projection unit based on the characteristic.   A program for causing a computer apparatus to function as each unit of the information processing apparatus according to any one of claims 1 to 15 when executed by a computer.

Images