JP2019012070A - Information processing device, information processing method, and program - Google Patents

Abstract

To increase measurement accuracy of a three-dimensional measurement device configured of one projector and a plurality of cameras.SOLUTION: An information processing device connected to projection means and a plurality of imaging parts 1 to 4 comprises: projection control means for causing the projection means to project bright/dark stripe patterns several number of times; acquisition means for acquiring an image obtained by imaging a measurement object on which the stripe patterns have been projected by the projection means by at least one of the plurality of imaging parts 1 to 4; and measurement means for performing three-dimensional measurement of the measurement object on the basis of the image acquired by the acquisition means.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a three-dimensional measurement technique using an image.

  The three-dimensional measurement technique using an image can be used for various purposes such as generating a three-dimensional model from an actual object and measuring the position and orientation of an object. In the stereo method, which is one of the representative methods, three-dimensional measurement is performed based on the principle of triangulation from images taken by two cameras (stereo cameras) whose relative positions and orientations are known. When using a stereo camera, it is difficult to search for a corresponding point between the cameras in an area where the luminance change on the image is small. For this reason, it is widely performed to facilitate the search for corresponding points by projecting slit light or pattern light onto a measurement target using an illumination device such as a projector. In addition, three-dimensional measurement is widely performed in which one of the stereo cameras is replaced with an illumination device such as a projector, and the camera and the illumination device are used as a stereo pair.

  When performing three-dimensional measurement of an object having a large amount of specular reflection components such as metal, it is difficult to photograph reflected light on the object surface of the projection pattern of the projector with a small number of cameras. On the other hand, by increasing the number of cameras and observing from various directions, it is possible to stabilize the three-dimensional measurement of an object having many specular reflection components. However, depending on the combination of the projection pattern and the arrangement of the camera or projector, there are cases where measurement cannot be performed with high accuracy. For example, if the projection pattern is a pattern composed of a plurality of parallel straight lines, and the direction of the epipolar line of the camera and projector (or camera and camera) that make up the stereo pair is close to the direction of the projection pattern, the measurement accuracy is high. descend. For this reason, Patent Document 1 discloses a method of switching the direction of a stripe-shaped projection pattern in accordance with the deviation of the optical axis of the camera and the optical axis of the illumination device when one illumination device and three cameras are used. Has been.

Japanese Patent No. 45525240

  However, like the method disclosed in Patent Document 1, for each camera, setting the direction of the epipolar line so as not to be close to the direction of the projection pattern in consideration of the direction of the camera and the direction of the projector It was not a simple task.

  The present invention has been made in view of the above problems, and in a three-dimensional measurement apparatus constituted by a projector that projects a pattern and an arbitrary number of cameras, it is possible to more easily set a pattern that can be measured accurately. The purpose is to do.

  The information processing apparatus of the present invention is connected to a projection unit and a plurality of imaging units, and has a projection unit, a plurality of imaging units, a projection control unit that causes the projection unit to project a bright and dark stripe pattern a plurality of times, and the projection An acquisition unit that acquires an image obtained by capturing the measurement target on which the fringe pattern is projected by at least one of the plurality of imaging units, and a three-dimensional measurement target based on the image acquired by the acquisition unit Measuring means for measuring.

  An object of the present invention is to more easily set a pattern that can be measured with high accuracy in a three-dimensional measurement apparatus that includes a projector that projects a pattern and an arbitrary number of cameras.

It is a figure which shows the structure of the information processing apparatus 1 in 1st Embodiment. It is a figure which shows arrangement | positioning in the actual utilization scene of the projection part and imaging part which comprise a three-dimensional measuring device. It is a figure explaining GUI which concerns on 1st Embodiment. It is a flowchart which shows the process sequence of the projection pattern determination in 1st Embodiment. It is a figure explaining GUI which concerns on 2nd Embodiment. It is a flowchart which shows the process sequence of the projection pattern determination in 3rd Embodiment. It is a figure explaining GUI which concerns on the modification 1. FIG. It is a figure explaining GUI which confirms arrangement | positioning of a camera projector. It is a figure explaining GUI which concerns on the modification 4. FIG. It is a figure explaining GUI which concerns on the modification 5. FIG. It is a figure explaining GUI which concerns on 4th Embodiment. It is a figure explaining GUI which concerns on the modification 6. FIG. It is a figure which shows the example of the hardware constitutions of the information processing apparatus of this invention.

  Hereinafter, embodiments (embodiments) for carrying out the present invention will be described with reference to the drawings.

  Prior to describing each embodiment according to the present invention, a hardware configuration in which the information processing apparatus shown in each embodiment is mounted will be described with reference to FIG.

  FIG. 13 is a hardware configuration diagram of the information device 1 in the present embodiment. In the figure, a CPU 1010 comprehensively controls devices connected via a bus 1000. The CPU 1010 reads and executes processing steps and programs stored in a read only memory (ROM) 1020. In addition to the operating system (OS), each processing program, device driver, and the like according to the present embodiment are stored in the ROM 1020, temporarily stored in a random access memory (RAM) 1030, and appropriately executed by the CPU 1010. The input I / F 1040 is input as an input signal in a format that can be processed by the information processing apparatus 1 from an external device (display device, operation device, or the like). The output I / F 1050 is output as an output signal in a format that can be processed by the display device to an external device (display device).

  Each of these functional units is realized by the CPU 1010 expanding a program stored in the ROM 1020 in the RAM 1030 and executing processing according to each flowchart described later. Further, for example, when hardware is configured as an alternative to software processing using the CPU 1010, arithmetic units and circuits corresponding to the processing of each functional unit described here may be configured.

(First embodiment)
In the first embodiment, a case will be described in which the present invention is applied to a three-dimensional measurement apparatus that performs three-dimensional measurement using a camera and a projector as a stereo pair, among three-dimensional measurement apparatuses including a projector and a plurality of cameras. . As a three-dimensional measurement method, a projector sequentially projects a pattern set for each of a plurality of cameras and performs three-dimensional measurement based on an image captured by the selected camera. Is assumed. In the present embodiment, a method for setting a pattern for each camera before performing such a three-dimensional measurement method will be described. Specifically, the user selects the most suitable pattern direction for each camera.

  FIG. 1 is a diagram illustrating a configuration of an information processing apparatus 1 according to the present embodiment. The information processing apparatus 1 according to the present embodiment includes a projection unit 10, a pattern information setting unit 30, a pattern information holding unit 40, an imaging unit 50, an image acquisition unit 70, a three-dimensional measurement unit 80, a camera selection unit 90, and a user operation acquisition unit. 110. The information processing apparatus 1 is connected to an external display unit 100 and an operation unit 120. Note that the display unit 100 and the operation unit 120 may be integrated into the functions of the information processing apparatus 1 and configured as a single unit.

  FIG. 2 is a diagram showing the arrangement of the projection unit and the imaging unit that constitute the three-dimensional measurement apparatus in an actual usage scene, and shows a case where four cameras are configured. In FIG. 2, the cameras 1 to 4 correspond to an imaging unit, and the projector corresponds to a projection unit. Hereafter, each part which comprises the information processing apparatus 1 is demonstrated, referring FIG.

The projection unit 10 is a projector that projects pattern light for three-dimensional measurement onto a measurement object, and can project an arbitrary two-dimensional image as pattern light. As shown in FIG. 2, in this embodiment, the projection unit 10 (projector) projects a multiline pattern (line pattern) in which a plurality of parallel straight lines are arranged at equal intervals. As the multiline pattern, for example, the Durdle pattern described in the following non-patent document is used.
N. G. Durdle, J.M. Thayyoor, and V.A. J. et al. Raso, “An improved structured light technical for surface restructuring of the human truncation,” IEEE Canadian Conference on Electric. Electr. 2, pp. 874-877, 1998.

  The Durdle et al. Pattern is a monochrome pattern configured so that each straight line has one of a plurality of brightness values in order to identify the straight line on the captured image. 3D measurement is possible. In order for the projection unit 10 to set a pattern corresponding to the camera selected by the camera selection unit 90, the user uses the graphical user interface (hereinafter abbreviated as GUI) set in the display unit 100 to obtain information regarding the pattern. Enter. Based on the input pattern light information, the projection unit 10 determines a two-dimensional pattern image to be projected, and projects the determined two-dimensional pattern image. The display of the projection unit 10 is controlled (projection control) by the CPU 1010.

  The pattern information setting unit 30 sets information related to the pattern in the projection unit 10 via the GUI set in the display unit 100 during setup. In the present embodiment, it is assumed that the information regarding the pattern is the direction of the pattern projected by the projector. The pattern information setting unit 30 acquires a two-dimensional pattern image from the pattern information holding unit 40, and sends the pattern image rotated according to the set pattern direction to the projection unit 10, thereby projecting the pattern image. Projection control of the unit 10 is performed.

  The pattern information holding unit 40 holds information related to the pattern projected by the projector. In the present embodiment, a two-dimensional pattern image projected by the projector is held for each of a plurality of cameras constituting the imaging unit 50.

  The imaging unit 50 includes a plurality of cameras. The camera may be a monochrome camera or a color camera. In the present embodiment, since the projection pattern is a monochrome pattern, the camera is assumed to be monochrome. As shown in FIG. 2, in this embodiment, the imaging part 50 shall be comprised by four cameras. It is assumed that internal parameters such as focal lengths, image centers, and lens distortion parameters of the cameras constituting the imaging unit 50 and the projector that is the projection unit 10 are calibrated in advance. It is also assumed that the relative position and orientation (external parameters) of each camera with respect to the projector have been calibrated in advance.

  The image acquisition unit 70 acquires an image captured by the camera selected by the camera selection unit 90 and inputs it to the three-dimensional measurement unit 80.

  The three-dimensional measurement unit 80 is based on the image captured by the camera selected by the camera selection unit 90 among the images input from the image acquisition unit 70 and the pattern information held in the pattern information holding unit 40. 3D measurement of the measurement object is performed. Three-dimensional measurement is performed using internal parameters and external parameters of a camera and a projector held in a storage device (not shown). The three-dimensional measuring unit 80 sends the measurement result to the display unit in a format that can be displayed by the external display unit 100 via an interface (not shown). In addition to the measurement result, arrangement information of the imaging unit and the projection unit, an image captured by the imaging unit, and the like may be sent together.

  The camera selection unit 90 receives a signal from the user operation acquisition unit 110 and inputs identification information of a target camera for setting a pattern to the imaging unit 50. The identification information is a number unique to each camera.

  The display unit 100 displays a three-dimensional coordinate group on the surface of the measurement object obtained by the three-dimensional measurement unit 80. For example, the display unit 100 is a liquid crystal display or a CRT display. The display of the display unit 100 is controlled (display control) by the CPU 1010.

  The user operation acquisition unit 110 functions as an interface for receiving an operation signal from the external operation unit 120. The acquired operation signal is sent to the pattern information setting unit 30 and the camera selection unit 90 in a receivable format.

  The operation unit 120 is a mouse or a keyboard for the user to operate the UI. Further, when the display unit 100 includes a button or a touch panel, the button or the touch panel also corresponds to the operation unit 120. The operation unit 120 sends a signal operated by the user to the user operation acquisition unit 110 in a format that the user operation acquisition unit 110 can receive.

  Next, the GUI according to the present embodiment will be described with reference to FIG. It is assumed that the GUI is displayed on a monitor (not shown).

  The camera selection UI (1100) corresponds to the camera selection unit 90. Information for identifying the camera is displayed in the camera selection UI (1100). The user selects a camera using the camera selection UI (1100). The user selects the camera by clicking the radio button with the mouse which is the operation unit 120. With this UI, the user can set an appropriate pattern for each camera.

  An information display UI (1110) corresponds to the display unit 100. The information display UI (1110) presents the three-dimensional coordinate group on the surface of the measurement object obtained in the three-dimensional measurement unit 80 to the user as an image of a state in which each measurement point is distributed in the three-dimensional space. The user can check whether the measurement performance of the 3D measurement result when using the selected camera and the selected pattern is sufficient from the visualized distribution, accuracy, number of measurement points, etc. A suitable pattern can be determined. In this UI, the user can freely set the viewpoint position / direction, the angle of view, etc. for observing the three-dimensional point group with the mouse or the like as the operation unit 120.

  For example, there may be a function in which the observation direction changes when the mouse is left-clicked on the information display UI (1110) and dragged while the mouse is dragged while the mouse is right-clicked. Further, zooming may be possible by rotating a mouse wheel.

  In FIG. 3, radio buttons for switching between display of a two-dimensional image (camera view) and display of a three-dimensional point group (virtual view) are arranged. If the direction of the epipolar line and the direction of the projection pattern are close to parallel, highly accurate three-dimensional measurement is not performed.Therefore, the epipolar line determined from the camera and projector arrangement is superimposed on the two-dimensional image, and the pattern It can also be used to determine the direction. When displaying a three-dimensional point group, all the obtained three-dimensional point groups are presented in the same color, or the color is changed for each point according to measurement accuracy, measurement stability, or the like. The measurement accuracy is obtained by performing imaging and three-dimensional measurement of an object whose shape is known, and comparing the three-dimensional measurement result with an ideal value. For example, from the plane of each point when plane fitting is performed on the obtained 3D point cloud so that only the workbench (plane) is photographed without placing any measurement object on the workbench The color is changed according to the distance of the display.

  The pattern information input UI (1120) corresponds to the pattern information setting unit 30. The pattern information input UI (1120) displays information for identifying the pattern direction. The user uses the pattern information input UI (1120) to select a pattern. In the GUI shown in FIG. 3, the user selects one pattern from a plurality of predetermined patterns. The user selects a pattern by clicking a radio button with the mouse of the operation unit 120. However, the pattern selection method is not limited to this, and any method may be used as long as one pattern is selected from a plurality of selectable patterns. For example, it may be selected by a check box instead of a radio button, or may be selected from a list box or a drop-down list. Moreover, as shown in FIG. 3, each pattern may be presented simultaneously.

  If preview button (1130) is clicked on with a mouse, the projector projects a pattern selected using pattern information input UI (1120), and the camera selected by camera selection UI (1100) captures an image. Based on the photographed image, the three-dimensional measuring unit 80 performs three-dimensional measurement and presents it on the information display UI (1110). By repeating pattern setting and preview for the selected camera, the user can visually confirm the three-dimensional measurement result when the pattern is changed and determine the pattern to be selected. If the application button (1140) is clicked with the mouse when a desired pattern is set, it is registered in the pattern information holding unit 40 as a pattern to be projected when shooting with the selected camera.

  Next, a processing procedure for determining the projection pattern of each camera to be performed before performing the three-dimensional measurement will be described. FIG. 4 is a flowchart illustrating a processing procedure for determining a projection pattern in the first embodiment.

(Step S1000)
In step S1000, the camera selection unit 90 performs initialization. In the initialization, the projection patterns corresponding to all the cameras are in an undecided state.

(Step S1010)
In step S1010, the camera selection unit 90 selects a camera. Specifically, first, the user operation acquisition unit 110 receives an operation signal from the operation unit 120, and transmits the received signal in a format that can be received by the camera selection unit 90. And the camera selection part 90 selects a camera by sending the identification information of the camera which sets the pattern to the imaging part 50 based on the received signal. The operation of the operation unit 120 by the user (selection of the camera) is performed by operating the camera selection UI (1100) of FIG. The user selects a camera whose pattern to be projected is not determined during runtime measurement. In this embodiment, a camera is selected by a user instruction to the camera selection UI. However, a camera selection order may be determined in advance, and the selection may be automatically performed in the determined order.

(Step S1020)
In step S1020, the pattern information setting unit 30 sets a pattern for the camera selected in step S1010. That is, the pattern projected from the projection unit when the selected camera captures an image is set for each camera. First, the user operation acquisition unit 110 receives an operation signal from the operation unit 120, and transmits the received signal in a format that can be received by the pattern information setting unit 30. The operation of the operation unit 120 by the user (pattern setting) is performed by operating the pattern information setting UI (1120) of FIG. The pattern information setting unit 30 sends the set pattern information to the projection unit 10.

(Step S1030)
In step S1030, the three-dimensional measurement unit 80 performs three-dimensional measurement. Specifically, first, the projection unit 10 projects the pattern set in step S1020. Then, the image acquisition unit 70 inputs an image obtained by capturing the scene on which the pattern is projected with the camera selected in step S1010. Then, three-dimensional measurement is performed based on the photographed image. Three-dimensional measurement is performed as follows.

  First, line detection is performed from the input image, and association processing for specifying a line on the pattern image corresponding to the line on the input image is performed using the luminance value of the adjacent line. Next, using the above-mentioned internal and external parameters, each pixel on the input image that constitutes the associated line is a straight line in the three-dimensional space, and the corresponding line on the pattern image is in the three-dimensional space. The three-dimensional coordinates of the target pixel are calculated by converting to a plane and obtaining the intersection of the straight line and the plane. This process is repeated for the line detected on the image and each pixel constituting the line to obtain a three-dimensional point group on the surface of the measurement object.

(Step S1040)
In step S1040, the display unit 100 displays the result of the three-dimensional measurement in step S1030 in association with each camera. The user checks the three-dimensional measurement result displayed on the three-dimensional point cloud display UI (1110) of FIG. 3 in the display unit 100, and determines whether or not the selected pattern is appropriate. As described above, this determination is performed by the user determining how much the three-dimensional measurement result deviates from the actual value (actual shape). If the user determines that the selected pattern is valid, the operation device 110 inputs a signal indicating that the selected pattern is valid to the user operation acquisition unit 110. The user may observe the three-dimensional point group by changing the viewpoint position and direction. In addition, the user may confirm whether or not the three-dimensional measurement is accurately performed with the color indicating the accuracy. Further, the camera selection unit 90 may automatically determine a deviation from a predetermined value.

(Step S1050)
In step S1050, the camera selection unit 90 determines whether or not there is an instruction to adopt the currently selected pattern as a pattern to be projected during runtime three-dimensional measurement when the user uses the currently selected camera. Specifically, the determination is made based on a signal from the user operation acquisition unit 110. If the currently set pattern is determined (the process is terminated), the process proceeds to step S1060. If not, the process returns to step S1020 to select a different pattern. For example, if the shape of the measurement object for three-dimensional measurement is known, the deviation between the measured value and the actual value is evaluated, and a pattern with a high evaluation value is automatically set without waiting for a user instruction. You may make it do.

(Step S1060)
In step S1060, the user operation acquisition unit 110 determines whether there is an end instruction from the user. If all the cameras have been determined, the process proceeds to step S1070. If not, the process returns to step S1010 to select a camera that has not yet been determined. Note that the process also proceeds to step S1070 when projection patterns have been determined for all the cameras constituting the imaging unit 50.

(Step S1070)
In step S1070, the pattern information holding unit 40 stores the information on the pattern selected for each camera constituting the imaging unit 50 in association with each camera. This information is referenced when performing runtime 3D measurements.

  At the time of three-dimensional measurement at runtime, the cameras are sequentially switched, the projection pattern set by the above-described method is projected by the projector for each camera, and the three-dimensional measurement is performed based on the photographed image. The three-dimensional measurement method is as described in the description of step S1030.

  As described above, in the first embodiment, in the three-dimensional measurement apparatus including a projector and a plurality of cameras, a method of selecting a pattern suitable for each camera when each camera and the projector are used as a stereo pair. Explained.

(Second Embodiment)
In the second embodiment, a case will be described in which the present invention is applied to a three-dimensional measurement apparatus that performs three-dimensional measurement using a camera and a camera as a stereo pair, among three-dimensional measurement apparatuses including a projector and a plurality of cameras. . In the present embodiment, two cameras (camera pairs) are selected from among a plurality of cameras as a three-dimensional measurement, and a projector projects a pattern suitable for the selected camera pair, and is picked up by the selected camera pair. A method of performing three-dimensional measurement based on a plurality of images is assumed.

  Since the configuration of the information processing apparatus 2 in the present embodiment is the same as the configuration of the information processing apparatus 1 in the first embodiment, a diagram thereof is omitted. Hereinafter, each part which comprises the information processing apparatus 2 is demonstrated.

  Since the projection unit 10, the pattern information setting unit 30, the imaging unit 50, the display unit 100, the user operation acquisition unit 110, and the operation unit 120 are the same as those in the first embodiment, description thereof is omitted. In the second embodiment, the internal parameters of the projector and the external parameters between the projector and the camera are not necessary.

  The pattern information holding unit 40 holds information related to the pattern displayed by the projector. In the present embodiment, a plurality of camera pairs are created by combining two cameras among a plurality of cameras constituting the imaging unit 50. In the present embodiment, the camera 1 and camera 3 and the camera 2 and camera 4 in FIG. In the present embodiment, the arrangement (external parameters) between the cameras (between the imaging means) is known, but the arrangement (external parameters) between the cameras and the projectors need not be known. The pattern information holding unit 40 holds information about which two-dimensional pattern the projector projects for each camera pair. The information holding method may be pattern direction (angle) information or a two-dimensional pattern image rotated in accordance with the pattern direction. In the case of pattern direction (angle) information, a pattern image corresponding to the direction is generated and irradiated when performing runtime three-dimensional measurement.

  The image acquisition unit 70 inputs an image captured by two cameras constituting the camera pair selected by the camera selection unit 90 to the three-dimensional measurement unit 80.

  The three-dimensional measurement unit 80 includes the image input from the image acquisition unit 70 and the pattern information held in the pattern information holding unit 40, the internal parameters of the camera held in a storage device (not shown), and the external between the cameras. Perform 3D measurement of the measurement object based on the parameters.

  The camera selection unit 90 inputs identification information of cameras constituting the target camera pair for setting a pattern to the system.

  Next, the GUI according to the present embodiment will be described with reference to FIG. It is assumed that the GUI is displayed on a monitor (not shown).

  The camera selection UI (2100) corresponds to the camera selection unit 90. Using this UI, the user selects a camera pair. The user selects a camera by clicking a radio button with a mouse (not shown). With this UI, the user can set an appropriate pattern for each camera pair.

  The information display UI (2110) corresponds to the display unit 100. This UI is basically the same as the information display UI (1110) of the first embodiment. However, in the first embodiment, it is possible to switch between the display of the two-dimensional image and the three-dimensional point group of one camera, whereas in the present embodiment, the two cameras constituting the selected camera pair The display of the 2D image and the display of the 3D point cloud can be switched. Furthermore, when a two-dimensional image of the camera is displayed, the epipolar line determined from the arrangement of the cameras constituting the camera pair is displayed in a superimposed manner, and the direction of the pattern projected by the projector can be compared with the direction of the epipolar line. Like that.

  The pattern information input UI (2120) corresponds to the pattern information setting unit 30. This UI is basically the same as the pattern information input UI (1120) of the first embodiment. If preview button (2130) is clicked on with a mouse, it projects the pattern selected using pattern information input UI (1120), and the two cameras selected by camera selection UI (2100) capture images. . Based on the two captured images, the three-dimensional measurement unit 80 performs three-dimensional measurement and presents it on the data display UI (2110). If the application button (1140) is clicked with the mouse when a desired pattern is selected, it is registered in the pattern information holding unit 40 as a pattern to be projected when shooting with the selected camera pair.

  Next, a processing procedure for determining the projection pattern of each camera pair performed before performing three-dimensional measurement will be described. Since this processing procedure is almost the same as the processing procedure in the first embodiment, it will be described with reference to the flowchart of FIG. Steps S2000 to S2090 correspond to steps S1000 to S1090 in FIG.

(Step S2000)
In step S2000, the camera selection unit 90 performs initialization. In the initialization, the projection patterns corresponding to all the camera pairs are in an undecided state.

(Step S2010)
In step S2010, the camera selection unit 90 selects a camera pair. The selection of the camera is performed according to a user instruction input via the camera selection UI (2100) in FIG. The user selects a camera pair whose pattern to be projected at the time of measurement at runtime is not determined through the camera selection UI (2100).

(Step S2020)
In step S2020, the pattern information setting unit 30 selects a projection pattern for the camera pair selected in step S2010. The selection of the projection pattern is performed by a user instruction input via the pattern information input UI (2120) in FIG.

(Step S2030)
In step S2030, the three-dimensional measurement unit 80 performs three-dimensional measurement. Specifically, first, the projection unit 10 projects the pattern selected in step S2020. Then, the scene on which the pattern is projected is captured by the two cameras constituting the camera pair selected in step S2010. Then, three-dimensional measurement is performed based on the photographed image. Three-dimensional measurement is performed as follows.

  First, line detection is performed from an image input to one camera (hereinafter referred to as camera A) of cameras constituting a camera pair, and the other side corresponding to the line on the input image using the luminance value of the adjacent line. The association process for specifying the line on the captured image of the camera (hereinafter referred to as camera B) is performed. Next, using the above-described internal / external parameters, each pixel on the captured image of the camera A constituting the associated line is converted into a straight line in the three-dimensional space. Then, the line on the captured image of the corresponding camera B is converted into a plane in the three-dimensional space. Then, the three-dimensional coordinates of the target pixel are calculated by obtaining the intersection points between the converted straight lines and the plane. This process is repeated for the line detected on the image and each pixel constituting the line to obtain a three-dimensional point group on the surface of the measurement object. Then, the information of the obtained three-dimensional point group is sent to the display unit 100.

(Step S2040)
In step S2040, the display unit 100 displays the result of the three-dimensional measurement in step S2030. The user confirms the three-dimensional measurement result displayed on the information display UI (2110) (FIG. 5) in the display unit, and determines whether or not the selected pattern is appropriate. This determination is the same as the processing in S1040.

(Step S2050)
In step S2050, the camera selection unit 90 determines whether to perform three-dimensional measurement by projecting the currently selected pattern when the user uses the currently selected camera pair. If the selected pattern is determined, the process proceeds to step S2060; otherwise, the process returns to step S2020 to select a different pattern.

(Step S2060)
In step S2060, the camera selection unit 90 determines whether projection patterns have been determined for all predetermined camera pairs. If all camera pairs have been determined, the process proceeds to step S2070. If not, the process returns to step S2010 to select an undecided camera pair.

(Step S2070)
In step S2070, the pattern information holding unit 40 stores information on the patterns selected for all the predetermined camera pairs by repeating steps S2010 to S2060. This information is referenced when performing runtime 3D measurements.

  At the time of three-dimensional measurement at runtime, the camera pairs are sequentially switched, the projection pattern set by the above-described method is projected by the projector for each camera pair, and three-dimensional measurement is performed based on the photographed image. The three-dimensional measurement method is as described in the description of step S2030.

  As described above, in the second embodiment, in the three-dimensional measurement apparatus including a projector and a plurality of cameras, a method for selecting a pattern suitable for each pair when the camera and the camera are a stereo pair. explained.

  In the present embodiment, the pattern projected by the projector is determined on the assumption that three-dimensional measurement is performed using images captured by two cameras. However, the number of cameras is not limited to two, and two or more cameras may be used. In the present embodiment, it is assumed that camera pairs are determined in advance. However, the camera pair may be determined by the user at the time of pattern selection.

(Third embodiment)
In the first and second embodiments, a pattern to be projected by the user is selected for the selected camera or camera pair, and a determination is made by looking at the result of three-dimensional measurement. In the third embodiment, a suitable pattern to be projected is automatically calculated based on the arrangement of the camera and the projector and presented to the user. In the third embodiment, a case where three-dimensional measurement is performed from a pair of a camera and a projector will be described as in the first embodiment.

  Since the configuration and GUI of the information processing apparatus 3 in the third embodiment are the same as those in the first embodiment, a diagram thereof is omitted.

  Next, a processing procedure for determining the projection pattern of each camera to be performed before performing the three-dimensional measurement will be described. FIG. 6 is a flowchart showing a processing procedure for determining a projection pattern in the third embodiment.

(Step S3010)
In step S3010, the camera selection unit 90 selects a camera. The user selects a camera via the camera selection UI (1100) in FIG. The user selects a camera whose pattern to be projected is not determined during runtime measurement.

(Step S3025)
In step S3025, the pattern information setting unit 90 calculates an optimal projection pattern for the camera selected in step S3010. The optimal projection pattern is calculated based on the internal parameters of the camera and the projector and the external parameters between the camera and the projector. Here, it is assumed that the candidate patterns to be projected include a multiline pattern parallel to the vertical direction of the projector image and a multiline pattern parallel to the horizontal direction of the projector image. Specifically, the direction on the image when a vector from the lens center of the projector to the lens center of the camera is projected on the projector image in the three-dimensional space is calculated. When the angle between the calculated direction and the horizontal direction of the projector image is 45 degrees or less, a multiline pattern parallel to the vertical direction of the projector image is selected. Select a multi-line pattern parallel to the horizontal direction.

(Step S3030)
In step S3030, the three-dimensional measurement unit 80 performs three-dimensional measurement. Specifically, first, the projector (projection unit) projects the pattern selected in step S3025 or S3020. Then, the scene (measurement target) on which the pattern is projected by the camera (imaging unit) selected in step S3010 is photographed, and three-dimensional measurement is performed based on the photographed image. The three-dimensional measurement unit 80 sends the three-dimensional measurement result to the display unit 100.

(Step S3040)
In step S3040, the display unit 100 displays the result of the three-dimensional measurement in step S3030. A data display UI (1110) in FIG. 3 is an example of a display result by the display unit 100. The user confirms the three-dimensional measurement result displayed on the UI shown in FIG. 3, determines whether the selected pattern is valid, and sends the determination result to the information processing apparatus 3 using the operation unit 120. As described above, the user may observe the three-dimensional point group by changing the viewpoint position and direction. In addition, the user may check whether the three-dimensional measurement is normally performed with a color indicating accuracy.

(Step S3050)
In step S3050, the pattern information setting unit 30 determines whether to perform three-dimensional measurement by projecting the currently selected pattern when the user uses the currently selected camera. If it is determined to be the selected pattern, the process proceeds to step S3060. If not, the process proceeds to step S3020 to select a different pattern.

(Step S3020)
In step S3020, the pattern information setting unit 30 sets a projection pattern for the camera selected in step S3010. The projection pattern is selected by the user via the pattern information input UI (1120) in FIG.

(Step S3060)
In step S3060, the camera selection unit 90 determines whether an end instruction has been input by the user. If it is input, the process proceeds to step S3070. If not, the process returns to step S3010 to select a camera that has not yet been determined. Even when projection patterns have been determined for all cameras constituting the imaging unit 50, the process proceeds to step S3070.

(Step S3070)
In step S3070, the pattern information holding unit 40 stores information on the pattern selected for each camera constituting the imaging unit 50 in the pattern information holding unit 40. This information is referenced when performing runtime 3D measurements.

  Since the processing procedure of runtime three-dimensional measurement in this embodiment is the same as that of the first embodiment, the description thereof is omitted.

  In the third embodiment, the method for determining the optimum pattern based on the known arrangement information of the camera and the projector has been described. However, the stereo pair is not limited to this, and a three-dimensional measurement may be performed by determining an optimum pattern when the camera and the camera are a stereo pair, and the three-dimensional measurement result may be presented to the user.

(Fourth embodiment)
In the fourth embodiment, a case will be described in which the color of the projection pattern is changed for each camera or camera pair. In the present embodiment, as in the second embodiment, a camera pair is selected and an optimum pattern is set for each camera pair.

  FIG. 11 is a diagram illustrating a GUI according to the present embodiment. A color designation unit 4150 is added to the GUI in FIG. 11 in addition to the GUI in FIG. Since the processing procedure for determining the projection pattern in the present embodiment is basically the same as that in the second embodiment, only step S2020 in which the processing is different will be described. In the second embodiment, only the pattern is selected in step S2020. However, in this embodiment, the color of the pattern to be projected for the camera pair selected using the color designation unit 4150 is also determined. For example, in FIG. 2, the camera pair of the camera 1 and the camera 3 can be selected as a monochrome multiline pattern, and the camera 2 and the camera 4 can be selected as a color pattern (the aforementioned Boyer multiline pattern). The monochrome pattern and the color pattern need not have the same shape. Further, it may be displayed so that the camera to be selected on the GUI can be seen as a color camera or a monochrome camera so that an optimum pattern can be selected. For example, in the camera selection UI, the display of the color camera and the monochrome camera may be changed so that the user can easily determine the color of the pattern. Furthermore, a pattern that cannot be used according to the type of camera (monochrome / color) may not be selected.

  In the present embodiment, the color of the projection pattern is changed for each camera pair, but the present invention is not limited to this. Similar to the first embodiment, when a projection pattern is set for each camera, the color of the projection pattern may be changed.

  In the fourth embodiment, the case where the camera and the camera are a stereo pair has been described. However, the stereo pair is not limited to this, and the camera and projector may be a stereo pair.

<Other examples>
(Modification 1)
In the embodiment described above, the optimum pattern for the three-dimensional measurement is selected without being limited to a specific area among the areas captured in the image. However, the pattern selection method is not limited to this, and the use of the pattern selected by the user may be determined by looking at the three-dimensional measurement result in a specific area. A GUI according to this modification will be described with reference to FIG. FIG. 7 shows a GUI data display UI (1110) according to the first embodiment. When the size of the three-dimensional measurement target object 1160 in the image is smaller than the image size, the user operates the mouse cursor 1170 with the mouse as the operation unit 120 to specify the rectangular area 1160 in the image. The three-dimensional measuring unit 80 calculates only the three-dimensional point group in the area designated by the rectangular area 1160. The user determines whether or not the pattern is accepted by looking at the three-dimensional measurement result in the designated area. As a result, a pattern most suitable for the measurement target can be selected without being affected by the three-dimensional measurement result of the region other than the measurement target.

  In the present modification, the case where the camera and the projector are paired as a stereo pair has been described as in the first embodiment. However, the stereo pair is not limited to this, and may be a camera-camera stereo pair as in the second embodiment. In this case, an area may be specified in an image captured by either camera, or an area may be specified by both cameras. Since the observation areas of the two cameras do not match, the result of 3D measurement in the area specified by one camera is projected onto the other image using known internal and external parameters, and the specified area Only the three-dimensional point group inside may be displayed on the data display UI (1110).

  Also, the area specified in the image does not necessarily have to be a rectangular area. For example, it may be a polygonal area having an arbitrary shape, or may be an area surrounded by a curve such as an ellipse or a circle. That is, any region may be used as long as it is a closed region on an image designated by a user interface such as a mouse.

(Modification 2)
In the embodiments described above, different patterns are switched, projected, and imaged sequentially for each camera or camera pair. However, when a common pattern is projected by different cameras or camera pairs, shooting may be performed without switching the pattern. For example, when a common pattern is set for the camera pair of camera 1 and camera 3 and the camera pair of camera 2 and camera 4 in FIG. That is, after the pattern for the camera 1 and the camera 3 is projected and shooting is performed with the camera 1 and the camera 3, the pattern is switched and shooting is performed with the camera 2 and the camera 4. That is, it is possible to reduce the time required for shooting by not performing switching when the patterns are common.

  Further, for example, when the absolute value of the difference between the orientations of the patterns optimal for two cameras is below a predetermined threshold, only one of the patterns is used instead of selecting the optimal pattern for each camera. In some cases, the time required for shooting may be shortened.

  In this modification, the case where the camera and the camera are paired in stereo is described as in the second embodiment. However, the stereo pair is not limited to this, and may be a stereo pair of a camera and a projector as in the first embodiment.

(Modification 3)
The GUI shown in FIGS. 3 and 5 described in the above embodiment displays a UI for selecting a camera, a UI for inputting information related to a pattern, and a three-dimensional measurement result measured with the selected camera and pattern. It consisted of a data display UI.

  However, the components of the GUI are not limited to this. For example, as shown in FIG. 8, the camera and the projector are arranged in a world coordinate system defined in the virtual space based on the external parameters of the camera and the projector calibrated in advance. Then, an image at a virtual viewpoint set separately may be generated by computer graphics and placed on the GUI. On this screen, the selected camera is displayed separately from other cameras by changing the color or the like, so that the user can easily grasp which camera pattern is selected. Further, the display method may be changed between a camera for which a pattern has been selected and an unselected camera so that the user can easily select an unselected camera. Furthermore, not only the camera and the projector but also a state in which the pattern set for the selected camera is projected in the virtual space may be displayed.

  This modification can be applied to any case where the camera and projector are in a stereo pair as in the first embodiment, and where the camera and camera are in a stereo pair as in the first embodiment. Is possible.

(Modification 4)
In each of the embodiments described above, the user selects the direction of the pattern to be projected set discretely. However, the information regarding the pattern set by the user is not limited to this. The direction of the pattern is not discrete and may be set as a continuous value using a slider or the like. Further, for example, the user may determine the interval and width of lines of a multi-line pattern (line pattern) via a GUI as shown in FIG. The distance between the lines needs to be determined according to the shape of the object to be measured. If it is desired to measure a fine shape, it is necessary to reduce the line interval. If a rough shape is to be measured, the line interval may be increased. Also, when the line spacing is narrow, when measuring the influence of blur or dynamic environment, the lines spread due to motion blur and different lines may overlap on the image, resulting in a decrease in measurement accuracy. By observing the three-dimensional measurement result via the GUI, the user can determine the optimum line spacing while taking into consideration the influence of blurring and motion blur. Also, the line width needs to be determined based on the resolution ratio between the camera and the projector. If the line width is narrower than one pixel on the camera side, the line cannot be detected with sufficient accuracy, so the line width needs to be increased. Similar to the line interval, the user can determine the optimum line width while referring to the three-dimensional measurement result via the GUI.

  This modification can be applied to any case where the camera and the projector are in a stereo pair as in the first embodiment, and where the camera and the camera are in a stereo pair as in the second embodiment. Is possible.

(Modification 5)
In each of the embodiments described above, the user selects a pattern presented on the pattern selection GUI without discrimination. However, for example, when the epipolar line direction of the camera and the projector (or the camera and the camera) and the direction on the image of the projection pattern photographed by the camera are close to each other, it is difficult to measure with high accuracy. You should not choose a pattern. Therefore, the epipolar line direction and the direction on the image of the projection pattern photographed by the camera are calculated, and as shown in FIG. Alternatively, the user may not make an incorrect selection. In FIG. 10, a pattern that should not be selected is grayed out (indicated by a broken line in the figure) so that it cannot be selected.

  This modification can be applied to any case where the camera and the projector are in a stereo pair as in the first embodiment, and where the camera and the camera are in a stereo pair as in the second embodiment. Is possible.

(Modification 6)
In the embodiment described above, the three-dimensional measurement result is presented to the user by changing the color depending on the accuracy. However, the method of presenting the three-dimensional measurement result is not limited to this. For example, as shown in FIG. 12, accuracy and the number of measurement points may be presented as numerical values. Since the three-dimensional measurement may fail, the number of measurement points is used as an index indicating the stability of the three-dimensional measurement. The accuracy at this time uses, for example, an RMS (Root Mean Square) error when a plane is fitted to a measured three-dimensional point group when a plane is a measurement target. Further, an RMS error when an object having a known shape other than a plane is measured and a three-dimensional model of the object is fitted to a three-dimensional point group may be displayed. Furthermore, a field in which the user inputs the target accuracy may be provided in the GUI, and a pattern that achieves the target accuracy may be automatically selected and presented to the user.

  This modification can be applied to any case where the camera and the projector are in a stereo pair as in the first embodiment, and where the camera and the camera are in a stereo pair as in the second embodiment. Is possible.

(Modification 7)
In the embodiment described above, the display of the pattern selected by the user is basically the same. However, it is not always necessary to display the patterns in the same manner, and the patterns to be selected by the user may be displayed according to the arrangement of the camera and the projector. That is, the outer frame of the pattern image having the largest angle between the direction of the epipolar line determined from the arrangement of the camera and the projector and the direction of the line of the projection pattern on the image is displayed thickly and selected by the user. It is possible to make the pattern to be easy to understand.

  This modification can be applied to any case where the camera and the projector are in a stereo pair as in the first embodiment, and where the camera and the camera are in a stereo pair as in the second embodiment. Is possible.

(Other embodiments)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, etc.) of the system or apparatus reads the program. It is a process to be executed.

<Effect of each embodiment>
According to the first embodiment, the result of actual three-dimensional measurement is presented to the user, so that the projector can project an appropriate projection for each camera in the three-dimensional measurement apparatus that performs the three-dimensional measurement using the camera and projector as a stereo pair. The pattern can be determined. In addition, by storing the projection pattern information for each camera set in advance, the same work is not required during runtime measurement. In addition to displaying only the measurement results of the measured 3D point cloud, it is possible to easily determine the validity of the pattern selected by the user by giving information related to measurement performance such as accuracy. it can. Since the user can freely change the position and direction of observing the three-dimensional point group, it becomes possible to determine the accuracy that is difficult to determine from a fixed viewpoint. Furthermore, an appropriate pattern can be determined by displaying a two-dimensional image of the selected camera and superimposing an epipolar line so that the direction relative to the projection pattern can be observed.

  According to the second embodiment, it is possible to determine an appropriate pattern projected by the projector for each camera in a three-dimensional measurement apparatus that performs three-dimensional measurement using the camera and projector as a stereo pair. Also, the user selects a camera pair and a pattern via the GUI, and presents to the user a result of actual three-dimensional measurement using a combination of the selected camera pair and pattern. Thereby, even if the number of camera pairs is arbitrary, the user can select an appropriate pattern for each camera pair.

  According to the third embodiment, pattern candidates that are determined to be optimal by the apparatus side are presented, so that the user does not need to select all patterns and check the three-dimensional measurement result, thereby improving user convenience.

<Definition>
A monochrome camera was used as the camera constituting the imaging unit in the first embodiment. However, the camera is not limited to this, and may be a color camera or a camera that captures light having a wavelength other than visible light, such as an infrared camera. When a color camera is used, the pattern to be projected is not limited to a monochrome pattern, and may be a color pattern. As the color pattern, for example, a multiline pattern indexed with the color of Boyer et al. Disclosed in the following non-patent document may be used. When an infrared camera is used, a monochrome multiline pattern may be irradiated using an infrared light source.
K. L. Boyer and A.M. C. Kak, “Color-encoded structured for rapid active ranging,” IEEE Transactions on Pattern Analysis, vol. 9, no. 1, pp. 14-28, 1987.

The Durdle multi-line pattern in which the brightness of each line is not the same was used as the three-dimensional measurement pattern irradiated by the projection unit in the first embodiment. However, the three-dimensional measurement pattern is not limited to this, and may be another multi-line pattern as long as the line detected on the image can be associated with the line on the projection pattern. For example, a pattern in which a line of Maruyama et al. Disclosed in the following non-patent document is divided into a plurality of line segments may be used.
M.M. Maruyama and S.M. Abe, “Range sensing by projecting multiple slits with random cuts,” IEEE Transactions on Pattern Analysis, vol. 15, no. 6, pp. 647-651, 1993.

  Furthermore, the three-dimensional measurement is not limited to a method in which a single pattern is irradiated and photographed, but a plurality of patterns may be irradiated and photographed. For example, it may be performed by a spatial coding method that performs three-dimensional measurement based on images obtained by irradiating and capturing a plurality of binary patterns having different line widths. Further, it may be performed by a phase shift method in which a plurality of sine wave patterns are irradiated and imaged, or a spatial encoding method and a phase shift method may be combined. The pattern described above is a pattern constituted by a plurality of parallel straight lines. In addition to this, other patterns may be used as long as they have a directivity other than a straight line, such as a pattern constituted by a plurality of parallel wavy lines.

  In the first embodiment, when the user selects a camera or a pattern, it is performed using a radio button on the GUI. However, the method of selecting a camera or pattern is not limited to this, and any method may be used as long as one method is selected from a plurality of candidates. For example, it may be selected by a check box instead of a radio button, or may be selected from a list box or a drop-down list. Further, as shown in FIG. 3, an image captured by each camera may be presented simultaneously with a UI for selecting the camera. Furthermore, it may be displayed so that a camera for which a pattern has already been set can be distinguished from a camera for which a pattern has not been set. For example, the captured image of the camera may be translucent, or may not be selectable with a radio button.

  In the first embodiment, the information held by the pattern information holding unit 40 is a two-dimensional pattern image projected by the projector. However, the information holding method is not limited to this, and any information may be used as long as it can be finally converted into a two-dimensional pattern image. For example, it may be information on the direction (angle) of the pattern. In the case of pattern direction (angle) information, a pattern image corresponding to the direction is generated and projected when performing three-dimensional measurement at runtime.

  In the first embodiment, the relative position and orientation of each camera with respect to the projector is calibrated in advance. However, the calibration information may be held by another method as long as the relative position and orientation of each camera with respect to the projector can be determined instead of the relative position and orientation of each camera with respect to the projector. For example, information on the relative position and orientation of one camera and a projector and the relative position and orientation between the cameras may be stored.

  In the first embodiment, the image acquisition unit 70 inputs an image captured by the camera selected by the camera selection unit 90 to the three-dimensional measurement unit 80. However, an image captured by a camera other than the camera selected by the camera selection unit 90 may be input. In that case, only the image captured by the camera selected in the three-dimensional measuring unit 80 is used.

  In the first embodiment, the camera identification information is a number unique to each camera. However, any information may be used as long as the information uniquely identifies the camera, such as a character string representing the name of each camera.

DESCRIPTION OF SYMBOLS 10 Projection part 30 Pattern information setting part 40 Pattern information holding part 50 Imaging part 70 Image acquisition part 80 Three-dimensional measurement part 90 Camera selection part 100 Display part 110 User operation acquisition part 120 Operation part

Claims (11)

An information processing apparatus connected to the projection means and the plurality of imaging units,
Projection control means for projecting a light and dark stripe pattern a plurality of times on the projection means;
An acquisition unit that acquires an image obtained by imaging the measurement target on which the fringe pattern is projected by the projection unit by at least one of the plurality of imaging units;
An information processing apparatus comprising: a measurement unit that performs three-dimensional measurement of the measurement target based on an image acquired by the acquisition unit.   The information processing apparatus according to claim 1, wherein the projection control unit causes the projection unit to project a light and dark fringe pattern including stripe patterns having different fringe directions from each other.   The information processing apparatus according to claim 2, wherein the stripe patterns having different stripe directions include stripe patterns that are substantially orthogonal to each other.   The information processing apparatus according to claim 1, wherein the imaging unit is switched in correspondence with a stripe direction of a stripe pattern projected by the projection unit.   The information processing apparatus according to claim 1, wherein the acquisition unit acquires an image captured by at least one of the plurality of imaging units each time the fringe pattern is projected by the projection unit.   The information processing apparatus according to claim 1, wherein an image used for measurement by the measurement unit is a stereo pair image.   When a vector heading from the projection center of the projection unit to the imaging center of the imaging unit that images the image on which the stripe pattern is projected is projected onto the stripe pattern image, the vector and the direction of the stripe of the stripe pattern The information processing apparatus according to claim 1, wherein an angle formed by is greater than 45 degrees.   The information processing apparatus according to claim 1, wherein relative positions and orientations of the plurality of imaging units are calibrated in advance.   The information processing apparatus according to claim 1, wherein the plurality of imaging units are at least four imaging units. An information processing method executed by an information processing apparatus connected to a projection unit and a plurality of imaging units,
A projection step of projecting a light and dark stripe pattern a plurality of times on the projection means;
An acquisition step of acquiring an image obtained by imaging the measurement target on which the fringe pattern is projected by the projection means by at least one of the plurality of imaging units;
An information processing method comprising: a measurement step of performing three-dimensional measurement of the measurement target based on an image acquired in the acquisition step. The program for functioning a computer as each means of the information processing apparatus of any one of Claims 1 thru | or 9.

Images