JP6299150B2 - Control device, robot, control system, control method, and control program - Google Patents

Description

  The present invention relates to a control device, a robot, a control system, a control method, and a control program.

  When the work is performed by the industrial robot, the position of the object to be handled by the industrial robot needs to be grasped by the robot controller. In order to grasp the position of an object, for example, the target object is irradiated with light or laser light emitted from a projector, and the reflection information is processed, so that three-dimensional information representing the position and orientation of the object is obtained immediately. Research and development of ways to do this is being conducted. As the three-dimensional information of an object, it is known to use discrete three-dimensional point cloud information or the like, and various methods for obtaining the three-dimensional point cloud information have been proposed.

  In this connection, a plurality of images whose brightness or color periodically changes in a predetermined direction and whose phases of the periodic changes are different from each other are projected by the projector, and the projected images are photographed by the camera. A calibration method for obtaining a relative relationship between the position and orientation of a projector and the position and orientation of a camera based on the obtained image is known (see Non-Patent Document 1). This calibration method is a method for accurately performing calibration performed between the camera and the projector in order to use the phase shift method which is one of the three-dimensional point group acquisition methods.

Song Zhang; Peisen S. Huang, “Novel method for structured light system calibration”, Optical Engineering 45_8_, 083601_August 2006_

  However, the conventional method is sufficient for use in a special environment realized in the laboratory, but in the actual environment the influence of disturbance etc. could not be ignored and sufficient accuracy was maintained. Calibration may not be possible.

  Therefore, the present invention has been made in view of the above-described problems of the prior art, and is capable of performing calibration between a camera and a projector with sufficient accuracy even in a real environment, a robot, a control system, and a control. A method and a control program are provided.

[1] The present invention has been made in order to solve the above-described problem, and is an image captured by the image capturing unit and an image projected by the projection unit, and changes in luminance or color periodically. A plurality of first captured images that are repeated and have different wave phases representing the luminance or color change period, and a second captured image in which the reference object projected by the projection unit is captured by the imaging unit. Based on each of the first captured images, a calculation unit that calculates a phase of each pixel on the captured image based on a position on the captured image plane where the reference object is captured in the second captured image; Based on the phase of each pixel on the captured image calculated by the calculation unit, the position on the captured image plane of the imaging unit is associated with the position on the projection image plane of the projection unit. And association unit, a control device comprising a.

  With this configuration, the control device calculates the phase of each pixel on the captured image based on the position on the captured image plane where the reference object is captured in the second captured image for each of the first captured images. Based on the phase of each pixel on the captured image, the position on the captured image plane of the image capturing unit and the position on the projected image plane of the projection unit are associated with each other. And the projection unit can be calibrated.

[2] Further, the present invention is the control device according to [1], wherein the projection light in the image obtained by imaging the projection light by the imaging unit and the luminance of the projection light projected by the projection unit. It is a control apparatus provided with the determination part which determines the brightness | luminance output of the said projection part which projects a said 1st captured image and a said 2nd captured image based on the brightness | luminance of.

  With this configuration, the control device displays the first captured image and the second captured image based on the brightness of the projection light projected by the projection unit and the brightness of the projection light in the image obtained by capturing the projection light by the imaging unit. Since the luminance output of the projection unit to be projected is determined, it is possible to prevent a decrease in calibration accuracy due to insufficient imaging resolution of the imaging unit.

[3] The present invention is the control device according to [1] or [2], wherein the image projected by the projection unit is a captured image captured by the imaging unit, and the projection unit Between the projection unit and the imaging unit based on a third captured image for deriving information representing a positional relationship between the projection unit and the imaging unit, and the phase of each pixel calculated by the calculation unit It is a control apparatus provided with the calibration process part which performs the calibration process which derives | leads-out the information showing the positional relationship of these.

  With this configuration, the control device can generate a projection unit based on the third captured image for deriving information representing the positional relationship between the projection unit and the imaging unit, and the phase of each pixel calculated by the calculation unit. 3D point cloud information for deriving the position and orientation of an object to be worked by a robot or the like with higher accuracy in order to perform a calibration process for deriving information representing the positional relationship between the camera and the imaging unit Can be generated.

[4] The present invention is the control device according to [3], wherein the third captured image is an image in which red and blue lattice patterns are captured.

  With this configuration, the control device represents the positional relationship between the projection unit and the imaging unit based on the image obtained by capturing the red and blue grid patterns and the phase of each pixel calculated by the calculation unit. Since the calibration process for deriving information is performed, it is possible to prevent the change in luminance or color of the first captured image from becoming indistinguishable due to the lattice pattern.

[5] The present invention is the control device according to [3] or [4], wherein the information representing the positional relationship represents a three-dimensional positional relationship between the projection unit and the imaging unit. 3D points for generating 3D point cloud information of the object based on information representing the positional relationship derived by the calibration processing unit and an image of the object imaged by the imaging unit It is a control apparatus provided with a group production | generation part.

  With this configuration, the control device generates the three-dimensional point cloud information of the object based on the information representing the three-dimensional positional relationship derived by the calibration processing unit and the image of the object imaged by the imaging unit. In addition, the position and orientation of an object to be worked by a robot or the like can be derived with higher accuracy.

[6] The present invention is the control device according to [5], wherein the three-dimensional position and orientation of the object are determined based on the three-dimensional point group information generated by the three-dimensional point group generation unit. A control apparatus including a robot control unit that derives and controls a robot based on the derived three-dimensional position and orientation of the object.

  With this configuration, the control device derives the three-dimensional position and orientation of the object based on the three-dimensional point cloud information, and controls the robot based on the derived three-dimensional position and orientation of the object. Work on an object can be performed with higher accuracy.

[7] Furthermore, the present invention is a robot including the control device according to [6] and an actuator that operates a movable part of the robot.

  With this configuration, the robot performs calibration between the imaging unit and the projection unit with sufficient accuracy even in an actual environment, so that the work on the target object can be performed with higher accuracy.

[8] The present invention is a control system including a projection unit that projects an image, an imaging unit that captures an image projected by the projection unit, and a control device, wherein the control device includes: A plurality of images in which the image projected by the projection unit is a captured image captured by the imaging unit, the change in luminance or color is periodically repeated, and the phases of the waves representing the cycle of the change in luminance or color are different from each other Based on the first captured image and the second captured image obtained by capturing the reference object projected by the projecting unit with the image capturing unit, the reference image in the second captured image for each of the first captured images. A calculation unit that calculates the phase of each pixel with a position on the captured image plane where the object is captured as a base point, and imaging of the imaging unit based on the phase of each pixel calculated by the calculation unit Comprising a position on the image plane, and a correspondence unit that performs correspondence between positions on the projection image plane of the projection portion is a control system.

  With this configuration, the control system calculates the phase of each pixel on the captured image based on the position on the captured image plane where the reference object is captured in the second captured image for each of the first captured images. Based on the phase of each pixel on the captured image, the position on the captured image plane of the image capturing unit and the position on the projected image plane of the projection unit are associated with each other. And the projection unit can be calibrated.

[9] Further, according to the present invention, the control device is a captured image in which an image projected by the projection unit is captured by the imaging unit, and changes in luminance or color are periodically repeated. The first captured image based on a plurality of first captured images having different wave phases representing a change period and a second captured image in which the reference object projected by the projection unit is captured by the imaging unit. For each of the above, the phase of each pixel is calculated based on the position on the captured image plane where the reference object is captured in the second captured image, and the imaging unit captures the image based on the calculated phase of each pixel. In this control method, the position on the image plane is associated with the position on the projection image plane of the projection unit.

  With this control method, the control device calculates the phase of each pixel on the captured image based on the position on the captured image plane where the reference object is captured in the second captured image for each of the first captured images. Based on the phase of each pixel on the captured image, the position on the captured image plane of the imaging unit and the position on the projected image plane of the projection unit are associated with each other, so that the image can be captured with sufficient accuracy even in a real environment. Calibration between the projection unit and the projection unit can be performed.

[10] Further, the present invention is a captured image obtained by capturing an image projected by the projection unit on the computer of the control device by the imaging unit, and changes in luminance or color are periodically repeated. The first imaging based on a plurality of first captured images having different wave phases representing a color change period and a second captured image in which the reference object projected by the projection unit is captured by the imaging unit. For each of the images, the phase of each pixel is calculated based on the position on the captured image plane where the reference object is captured in the second captured image, and the imaging unit is based on the calculated phase of each pixel. It is a control program which performs matching with the position on the projection image plane of the said projection part, and the position on the projection image plane of the said projection part.

  With this control program, the control device calculates, for each first captured image, the phase of each pixel on the captured image based on the position on the captured image plane where the reference object is captured in the second captured image, and calculates Based on the phase of each pixel on the captured image, the position on the captured image plane of the imaging unit and the position on the projected image plane of the projection unit are associated with each other, so that the image can be captured with sufficient accuracy even in a real environment. Calibration between the projection unit and the projection unit can be performed.

Further, the present invention is a captured image in which an image projected by the projection unit is captured by the imaging unit, and a change in luminance or color is periodically repeated, and a wave representing a cycle of the change in luminance or color Based on a plurality of first captured images having different phases and a second captured image in which a reference object projected by the projection unit is captured by the imaging unit, the second captured image is obtained for each of the first captured images. A calculation unit that calculates a phase of each pixel on the captured image based on a position on the captured image plane at which the reference object is captured in the captured image, and each pixel on the captured image calculated by the calculation unit Based on the phase, an association unit that associates a position on the captured image plane of the imaging unit with a position on the projection image plane of the projection unit, and projection light projected by the projection unit Based on the projected light brightness that is the brightness and the captured image brightness that is the brightness of the projected light in the image in which the projected light is captured by the imaging unit, the change in the projected light brightness among the projected light brightness Then, the projection light luminance at which the captured image luminance starts to change linearly is derived as reference projection light luminance, and the first captured image and the second captured image are projected based on the derived reference projection light luminance. And a determination unit that determines a luminance output of the projection unit.
Moreover, this invention is a control system which comprises the projection part which projects an image, the imaging part which images the image projected by the said projection part, and a control apparatus, Comprising: The said control apparatus is based on the said projection part. The projected image is a picked-up image picked up by the image pickup unit, and a change in luminance or color is periodically repeated, and a plurality of first phases in which the phases of waves representing the change in the luminance or color change are different from each other. Based on the captured image and the second captured image in which the reference object projected by the projection unit is captured by the imaging unit, the reference object is captured in the second captured image for each of the first captured images. A calculation unit that calculates a phase of each pixel with a position on the captured image plane as a base point, and a captured image plane of the imaging unit based on the phase of each pixel calculated by the calculation unit. An association unit that associates the upper position with a position on the projection image plane of the projection unit, a projection light luminance that is a luminance of the projection light projected by the projection unit, and the projection by the imaging unit The projected light whose luminance starts to change linearly with respect to the change of the projected light luminance of the projected light luminance based on the captured image luminance that is the luminance of the projected light in the image where light is imaged. A determination unit that derives luminance as reference projection light luminance and determines a luminance output of the projection unit that projects the first captured image and the second captured image based on the derived reference projection light luminance; The control system.
In the present invention, the control device is a captured image in which an image projected by the projection unit is captured by the imaging unit, and a change in luminance or color is periodically repeated, and a cycle of the change in luminance or color For each of the first captured images based on a plurality of first captured images having different wave phases representing a second captured image in which the reference object projected by the projection unit is captured by the imaging unit The phase of each pixel is calculated based on the position on the captured image plane where the reference object is captured in the second captured image, and on the captured image plane of the imaging unit based on the calculated phase of each pixel And the position on the projection image plane of the projection unit, the projection light luminance that is the luminance of the projection light projected by the projection unit, and the projection light by the imaging unit Based on the captured image brightness that is the brightness of the projected light in the imaged image, the projected light brightness where the captured image brightness starts to change linearly with respect to the change in the projected light brightness of the projected light brightness. It is a control method which derives | leads-out as reference | standard projection light brightness | luminance, and determines the brightness | luminance output of the said projection part which projects a said 1st captured image and a said 2nd captured image based on the derived | led-out reference | standard projection light brightness | luminance.
Further, the present invention is a captured image obtained by capturing an image projected by the projection unit on the computer of the control device by the imaging unit, and the luminance or color change is periodically repeated, and the luminance or color change is performed. Each of the first captured images is based on a plurality of first captured images having different phases of waves representing the period of the first image and a second captured image in which the reference object projected by the projection unit is captured by the imaging unit. The phase of each pixel is calculated based on the position on the captured image plane where the reference object is captured in the second captured image, and the captured image of the imaging unit is calculated based on the calculated phase of each pixel. A projection light luminance that is a luminance of the projection light projected by the projection unit, which associates a position on the plane with a position on the projection image plane of the projection unit, and the imaging unit The captured image brightness starts to change linearly with respect to the change in the projected light brightness of the projected light brightness based on the captured image brightness that is the brightness of the projected light in the image in which the projected light is captured. Control that causes the projection light luminance to be derived as a reference projection light luminance, and determines a luminance output of the projection unit that projects the first captured image and the second captured image based on the derived reference projection light luminance. It is a program.
As described above, the control device calculates, for each of the first captured images, the phase of each pixel on the captured image using the position on the captured image plane where the reference object is captured in the second captured image as a base point. Based on the calculated phase of each pixel on the captured image, the position on the captured image plane of the imaging unit and the position on the projected image plane of the projection unit are associated with each other. Thus, calibration between the imaging unit and the projection unit can be performed.

It is a figure which shows an example of the utilization condition of the control system 1 which concerns on 1st Embodiment. 3 is a diagram illustrating an example of a hardware configuration of a control device 40. FIG. 3 is a diagram illustrating an example of a functional configuration of a control device 40. FIG. It is a figure which shows the example of the projection image which the image generation part 52 produces | generates. 4 is a flowchart illustrating an example of a flow of processing executed by each functional unit of the initial processing unit 60. 4 is a graph showing an example of the relationship between the luminance output of the projection unit 20 and the luminance on the captured image plane captured by the imaging unit 10; It is a figure which shows an example of the grid pattern formed with red and blue. It is a figure which shows the comparative example of the captured image at the time of projecting white light and red light, respectively on a calibration board. It is a figure which shows an example of the relationship between the position and phase on a projection image plane, the position and phase on the captured image plane which imaged the projection image, and a projection area | region and an imaging area. 5 is a flowchart illustrating an example of a flow of a 3D point cloud image generation process executed by a 3D point cloud generation unit 70; It is a figure which shows the example of an area | region when the image for switching determination is divided | segmented into the several area | region. It is an example of the graph which plotted the projection image projected at the time of the last subroutine execution regarding the projection image projected in step S200, the pixel on the projection image plane projected at the time of this subroutine execution, and the brightness | luminance of the pixel. The difference calculated by the imaging time determination unit 72 is an example of a graph plotted for each pixel. It is a figure which shows an example of a function structure of the control system 1 in 2nd Embodiment.

<First Embodiment>
A first embodiment of the present invention will be described below with reference to the drawings. FIG. 1 is a diagram illustrating an example of a usage situation of the control system 1 according to the first embodiment. The control system 1 includes, for example, an imaging unit 10, a projection unit 20, a robot 30, and a control device 40. The imaging unit 10 is a camera including, for example, a CCD (Charge Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor), or the like, which is an imaging element that converts collected light into an electrical signal. Hereinafter, in order to simplify the description, the imaging unit 10 captures a still image. Note that the imaging unit 10 may capture a moving image instead of capturing a still image.

  The imaging unit 10 is connected to the control device 40 through a cable, for example, so as to be communicable. Wired communication via a cable is performed according to standards such as Ethernet (registered trademark) and USB (Universal Serial Bus), for example. The imaging unit 10 and the control device 40 may be connected by wireless communication performed according to a communication standard such as Wi-Fi (registered trademark). The imaging unit 10 captures an area CA surrounded by a broken line. The area CA is an area set in advance on a table or the like. The imaging unit 10 is installed at a position where the area CA can be imaged. The imaging unit 10 acquires a request for imaging from the control device 40, and images the area CA at the timing when the request is acquired. Then, the imaging unit 10 captures an image of the object OBJ when the object OBJ is installed inside the area CA by the user. Then, the imaging unit 10 outputs the captured image to the control device 40 by communication.

  The projection unit 20 is, for example, a projector including a liquid crystal light valve and a projection lens for projecting a projection image, a liquid crystal driving unit, and an ultrahigh pressure mercury lamp, a metal halide lamp, and the like as a light source. The projection unit 20 is connected so as to be communicable with the control device 40 by a cable, for example. Wired communication via a cable is performed according to standards such as Ethernet (registered trademark) and USB, for example. In addition, the projection part 20 and the control apparatus 40 may be connected by the wireless communication performed by communication standards, such as Wi-Fi. The projection unit 20 acquires various images from the control device 40 by communication, and projects the acquired various images as a projection image onto a region PA surrounded by a dotted line. Details of these projected images will be described later. In FIG. 1, the projection unit 20 is installed so that the area PA includes the area CA inside, but the size of the area PA and the area CA may be reversed, and the area PA and the area CA are It may overlap partially. Note that the object OBJ is large enough to be included inside the area where the area PA and the area CA overlap.

  The robot 30 is, for example, a 6-axis vertical articulated robot, and can perform an operation with 6 degrees of freedom by an operation in which a support base, an arm unit, and a gripping unit cooperate. The robot 30 may operate with 5 degrees of freedom or less. The robot 30 includes a grip portion. The grip portion of the robot 30 includes a claw portion that can grip or pinch an object. The robot 30 is communicably connected to the control device 40 by a cable, for example. Wired communication via a cable is performed according to standards such as Ethernet (registered trademark) and USB, for example. The robot 30 and the control device 40 may be connected by wireless communication performed according to a communication standard such as Wi-Fi. The robot 30 acquires a control signal based on the three-dimensional position and orientation of the object OBJ from the control device 40, and performs a predetermined operation on the object OBJ based on the acquired control signal. The predetermined work is, for example, that the object OBJ is grasped by the grasping unit of the robot 30 and the grasped object OBJ is moved from the currently installed position to another position, or after being moved, For example, assembling the device.

  The control device 40 controls the robot 30 to perform a predetermined work. More specifically, first, the control device 40 performs a calibration process between the imaging unit 10 and the projection unit 20 based on a captured image of the object OBJ captured by the imaging unit 10. The control device 40 can derive a more accurate three-dimensional position and orientation of the object OBJ by this calibration process. Details of the calibration process will be described later. Then, after the calibration process between the imaging unit 10 and the projection unit 20 by the control device 40 is completed, the control device 40 performs the tertiary processing of the object OBJ based on the captured image of the object OBJ captured by the imaging unit 10. The original position and orientation are derived. The control device 40 controls the robot 30 by generating a control signal based on the derived three-dimensional position and orientation and outputting the generated control signal to the robot 30. Moreover, the control apparatus 40 outputs the various images produced | generated by the image generation part 52 mentioned later with which an own apparatus is provided to the projection part 20, and controls the projection part 20 so that the output various images may be projected. Further, the control device 40 controls the imaging unit 10 so as to capture a captured image.

  Next, the hardware configuration of the control device 40 will be described with reference to FIG. FIG. 2 is a diagram illustrating an example of a hardware configuration of the control device 40. The control device 40 includes, for example, a CPU (Central Processing Unit) 41, a storage unit 42, an input reception unit 43, and a communication unit 44, and the other imaging unit 10, the projection unit 20, and the like via the communication unit 44. Communication with the robot 30 is performed. The CPU 41 executes various programs stored in the storage unit 42. The storage unit 42 includes, for example, an HDD (Hard Disk Drive), an SSD (Solid State Drive), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a ROM (Read-Only Memory), a RAM (Random Access Memory), and the like. Various information, images, and programs processed by the control device 40 are stored. Note that the storage unit 42 may be an external storage device instead of the one built in the control device 40.

  The input receiving unit 43 is, for example, a keyboard, mouse, touch pad, or other input device. The input receiving unit 43 may function as a display unit, and may be configured as a touch panel. The communication unit 44 includes, for example, an Ethernet port as well as a digital input / output port such as a USB.

  Next, the functional configuration of the control device 40 will be described with reference to FIG. FIG. 3 is a diagram illustrating an example of a functional configuration of the control device 40. The control device 40 includes, for example, a storage unit 42, an imaging control unit 46, an image acquisition unit 48, a projection control unit 50, an initial processing unit 60, a three-dimensional point group generation unit 70, and a robot control unit 80. Is provided. Among these functional units, some or all of the imaging control unit 46, the image acquisition unit 48, the projection control unit 50, the initial processing unit 60, the three-dimensional point group generation unit 70, and the robot control unit 80. For example, the CPU 41 is implemented by executing various programs stored in the storage unit 42. Some or all of these functional units may be hardware functional units such as LSI (Large Scale Integration) and ASIC (Application Specific Integrated Circuit). In accordance with a request from the initial processing unit 60 or the three-dimensional point group generation unit 70, the imaging control unit 46 controls the imaging unit 10 to perform imaging at a requested timing. The image acquisition unit 48 acquires the captured image captured by the imaging unit 10 and causes the storage unit 42 to store the acquired captured image.

  The projection control unit 50 includes an image generation unit 52, for example. The image generation unit 52 reads various information for generating a projection image from the storage unit 42 according to a request from the initial processing unit 60 or the three-dimensional point group generation unit 70, and generates a projection image based on the read information. . The projection control unit 50 outputs various images generated by the image generation unit 52 to the projection unit 20 as projection images, and controls the projection unit 20 to project the output projection images. In addition, the projection control unit 50 acquires information indicating a suitable luminance output from the initial processing unit 60, and controls the projection unit 20 to adjust the luminance output of the projection unit 20 to the acquired suitable luminance output. The details of the suitable luminance output will be described later. For example, the luminance output is sufficiently higher than the sensitivity of the imaging element of the imaging unit 10.

  When the luminance output of the projection unit 20 exceeds the sensitivity of the image sensor of the imaging unit 10, when the imaging unit 10 captures the projection image projected by the projection unit 20, all of the projection image is clearly displayed on the captured image plane. Imaged. The projection control unit 50 reads various images from the storage unit 42 instead of the image generation unit 52 generating various images, and outputs the read various images to the projection unit 20 as projection images. It is good also as what controls the projection part 20 so that an image may be projected.

  Here, the projection image which the image generation part 52 produces | generates is demonstrated with reference to FIG. FIG. 4 is a diagram illustrating an example of a projection image generated by the image generation unit 52. The image generation unit 52 generates a red image RIMG. The red image RIMG is an image used when performing a calibration process described later. The image generation unit 52 further includes a linear image VIMG extending in the vertical direction on the projection image plane through the optical center of the projection unit 20, and a linear image extending in the horizontal direction on the projection image plane through the optical center of the projection unit 20. Generate a HIMG. The straight line image VIMG and the straight line image HIMG are images used when performing phase connection processing described later.

  The image generation unit 52 also includes four images HFI-0 to HFI-0, in which the luminance periodically changes on the projection image plane, and a wave representing the periodic change in luminance proceeds in the horizontal direction on the projection image plane. Generate HFI-3. Images HFI-0 to HFI-3 are images in which the phases of waves representing periodic changes in luminance are different from each other by π / 2. In the images HFI-0 to HFI-3, a point with high brightness is a point with a high luminance value, and a point with low brightness is a point with a low luminance value. Similar to the images HFI-0 to HFI-3, the image generation unit 52 periodically changes the luminance on the projection image plane, and a wave representing the periodic change in luminance proceeds in the vertical direction on the projection image plane. Four images VFI-0 to VFI-3 are generated. Images VFI-0 to VFI-3 are images in which the phases of waves representing periodic changes in luminance are different from each other by π / 2. As shown in FIG. 4, the waves representing the periodic changes in luminance in the images HFI-0 to HFI-3 and the images VFI-0 to VFI-3 are plane waves that travel in the horizontal and vertical directions on the projection image plane. Although it is a (sine wave), it may be a plane wave traveling diagonally or a spherical wave or the like. Note that the images HFI-0 to HFI-3 and the images VFI-0 to VFI-3 may be images whose colors change periodically instead of images whose luminance changes periodically. The images HFI-0 to HFI-3 and the images VFI-0 to VFI-3 are images used when performing a phase connection process described later, and are further used to generate a three-dimensional point cloud image of the object OBJ described later. Also use images.

  Returning to FIG. The initial processing unit 60 includes, for example, a luminance output determination unit 62, a phase calculation unit 64, a coordinate association unit 66, and a calibration processing unit 68. The initial processing unit 60 controls the projection control unit 50 to sequentially project the various projection images shown in FIG. The initial processing unit 60 controls the imaging control unit 46 to cause the imaging unit 10 to sequentially capture the projection images projected by the projection unit 20. In addition, the initial processing unit 60 outputs a suitable luminance output determined by the luminance output determining unit 62 to the projection control unit 50, and adjusts the luminance output of the projection unit 20 to a suitable luminance output.

  Further, the initial processing unit 60 controls the imaging control unit 46 to adjust the white balance, gain, aperture, and exposure time of the imaging unit 10. The white balance adjustment will be described later. The initial processing unit 60 adjusts the gain and the aperture to the lowest value that can reproduce the color at each point of the image projected with a suitable luminance output. In addition, the initial processing unit 60 adjusts the exposure time to about three times the reciprocal of the refresh rate of the projection unit 20 so that the captured image obtained by capturing the projection image does not flicker due to the flicker phenomenon. The refresh rate of the projection unit 20 is, for example, 60 [Hz].

  Further, the initial processing unit 60 outputs information indicating the positional relationship between the imaging unit 10 and the projection unit 20 derived by the calibration processing unit 68 to the three-dimensional point group generation unit 70. The information indicating the positional relationship between the imaging unit 10 and the projection unit 20 is, for example, coordinate conversion between coordinates on the captured image plane of the imaging unit 10 and coordinates on the projection image plane of the projection unit 20 ( It is expressed by a translation and rotation transformation matrix for realizing (affine transformation).

  The luminance output determination unit 62 determines a suitable luminance output of the projection unit 20. Details of the process for determining a suitable luminance output will be described later. The phase calculation unit 64 calculates a phase necessary for a coordinate association process for associating coordinates on the captured image plane of the imaging unit 10 with coordinates on the projection image plane of the projection unit 20. Here, the phase is a wave phase representing a periodic change in luminance at each point on the image HFI-0 to HFI-3 plane or the image VFI-0 to VFI-3 plane shown in FIG. The coordinate association processing is processing for associating coordinates on the captured image plane of the imaging unit 10 with coordinates on the projection image plane of the projection unit 20. Details of the phase calculation process and the coordinate association process will be described later.

  The coordinate association unit 66 performs a coordinate association process based on the phase at each point on the captured image plane calculated by the phase calculation unit 64. The calibration processing unit 68 performs calibration processing based on the coordinates associated with the coordinate association unit 66. The calibration processing unit 68 derives information indicating the positional relationship between the imaging unit 10 and the projection unit 20 through calibration processing.

  The three-dimensional point group generation unit 70 includes an imaging time determination unit 72, for example. The three-dimensional point group generation unit 70 acquires information indicating the positional relationship between the imaging unit 10 and the projection unit 20 from the initial processing unit 60. Further, the three-dimensional point group generation unit 70 controls the projection control unit 50 to sequentially project the images HFI-0 to HFI-3 and / or the images VFI-0 to VFI-3 on the projection unit 20. Further, the three-dimensional point group generation unit 70 controls the imaging control unit 46 to cause the imaging unit 10 to capture the image projected by the projection unit 20 at the timing determined by the imaging time determination unit 72. The three-dimensional point cloud generation unit 70 is a captured image stored in the storage unit 42, and the images HFI-0 to HFI-3 and / or the images VFI-0 to VFI-3 are respectively projected onto the object OBJ. On the basis of the plurality of captured images and information indicating the positional relationship between the imaging unit 10 and the projection unit 20 acquired from the initial processing unit 60, for example, a three-dimensional point cloud image of the object OBJ by a phase shift method or the like Is generated. The 3D point cloud generation unit 70 outputs the generated 3D point cloud image to the robot control unit 80. Here, the 3D point cloud image is an example of 3D point cloud information.

  The imaging time determination unit 72 determines the timing at which the imaging unit 10 should capture the projection image projected by the projection unit 20, and causes the three-dimensional point group generation unit 70 to control the imaging control unit 46 at the determined timing. Thus, the imaging unit 10 is caused to capture a projection image. Details of the timing determination process will be described later.

  The robot control unit 80 includes, for example, a three-dimensional position / orientation deriving unit 82. The robot control unit 80 controls the robot 30 to perform a predetermined operation on the object OBJ based on the three-dimensional position and posture of the object OBJ derived by the three-dimensional position / orientation deriving unit 82. The three-dimensional position / orientation deriving unit 82 derives the three-dimensional position and orientation of the object OBJ based on the three-dimensional point cloud image of the object OBJ acquired from the three-dimensional point group generation unit 70.

  Hereinafter, the processing performed by each functional unit of the initial processing unit 60 will be described with reference to FIG. FIG. 5 is a flowchart illustrating an example of the flow of processing executed by the initial processing unit 60. First, the luminance output determination unit 62 determines a suitable luminance output of the projection unit 20. Then, the luminance output determination unit 62 outputs the determined preferable luminance output to the projection control unit 50, and causes the projection control unit 50 to adjust the luminance output of the projection unit 20 (step S100).

  Here, a process for determining a suitable luminance output will be described. First, the luminance output determination unit 62 controls the projection control unit 50 to cause the projection unit 20 to project a white image. At this time, the luminance output determination unit 62 changes the luminance output of the white image step by step, and controls the imaging control unit 46 to perform imaging every time the luminance output is changed. The imaging unit 10 is caused to capture an image. The luminance output of the projection unit 20 can be changed by, for example, 255 gradations from 0 to 254. The luminance output determination unit 62 determines the relationship between the luminance output of the projection unit 20 and the luminance on the captured image plane captured by the imaging unit 10 based on the white image captured for each gradation, as RGB (Red Green Derived every Blue). FIG. 6 is a graph illustrating an example of the relationship between the luminance output of the projection unit 20 and the luminance on the captured image plane captured by the imaging unit 10.

  As shown in FIG. 6, the luminance on the captured image plane imaged by the imaging unit 10 has a relatively small change rate up to the reference line RL when the luminance output of the projection unit 20 is increased. However, the luminance on the plane of the captured image captured by the imaging unit 10 is approximately directly proportional to the luminance output of the projection unit 20 with the reference line RL as a boundary. The luminance output determination unit 62 sets the luminance output of the projection unit 20 at the reference line RL (approximate direct proportional start point) as the minimum value, and sets any value within the range from the minimum value to the maximum value as a suitable luminance. Determine as output. In the following description, this minimum value is determined as a suitable luminance output.

  Returning to FIG. In the following description, it is assumed that the user has installed the calibration board in an area where the area PA and the area CA overlap after the luminance output of the projection unit 20 is adjusted to a suitable luminance output in step S100. The calibration board is a grid pattern formed of, for example, red and blue. FIG. 7 is a diagram illustrating an example of a grid pattern formed in red and blue. As shown in FIG. 7, in the grid pattern, a blue grid pattern BGD is drawn on a red background RBG. After the user installs the calibration board in the area where the area PA and the area CA overlap, the initial processing unit 60 controls the imaging control unit 46 to thereby adjust the white balance, gain, aperture, and exposure of the imaging unit 10. The time (referred to as parameters in FIG. 5) is adjusted (step S110).

  Here, since the adjustment of the gain, aperture, and exposure time is as described above, the description thereof is omitted. The white balance adjustment will be described with reference to FIG. FIG. 8 is a diagram illustrating a comparative example of captured images when white light and red light are respectively projected onto the calibration board. The leftmost diagram in FIG. 8 is a captured image when white light is projected onto the calibration board. When the initial processing unit 60 controls the projection control unit 50 to project white light from the projection unit 20 onto the calibration board illustrated in FIG. 7, a captured image as illustrated in the leftmost diagram of FIG. 8. GDB can be imaged. Then, the initial processing unit 60 converts the leftmost diagram in FIG. 8 into a gray scale. At this time, the initial processing unit 60 is based on the image converted to grayscale, so that the reflected light from the blue grid BP and the reflected light from the red grid RP have the same luminance. The white balance of 10 is adjusted by the imaging control unit 46.

  An image obtained by capturing the calibration board converted to the gray scale after this adjustment is shown in the middle of FIG. As shown in the middle diagram of FIG. 8, the image obtained by imaging the calibration board converted to the gray scale has a substantially uniform luminance as a whole, and distinguishes the blue grid BP from the red grid RP. It becomes impossible. By doing in this way, when the control device 40 projects the images HFI-0 to HFI-3 and / or the images VFI-0 to VFI-3 on the calibration board, the blue grid BP and the red grid RP It is possible to prevent light and dark due to the difference in reflection luminance from occurring on the captured image. In other words, the control device 40 can perform processing using white light as if there is no calibration board. Further, the captured image when the red image RIMG shown in FIG. 4 is projected onto the calibration board is the rightmost diagram of FIG. The reflection luminance of the blue grid BP and the red grid RP is the same for white light, but is different for red light, so that the captured image is as shown in the right-hand side of FIG.

  Returning to FIG. After the initial processing unit 60 adjusts the white balance, gain, aperture, and exposure time of the image capturing unit 10 in step S110, the phase calculation unit 64 controls the projection control unit 50, whereby the image VIMG, the image HIMG, and the image HFI. −0 to HFI-3 and images VFI-0 to VFI-3 are sequentially projected onto the projection unit 20. And the phase calculation part 64 controls the imaging control part 46, and makes the imaging part 10 image these images sequentially (step S120).

  Next, the phase calculation unit 64 reads the captured image captured in step S120 from the storage unit 42, and calculates the phase at each point on the captured image plane based on the read captured image (step S130). Here, the details of the process of calculating the phase at each point on the captured image plane will be described with reference to FIG. FIG. 9 is a diagram illustrating an example of the relationship between the projected image, the captured image, and the phase at each point. In the example illustrated in FIG. 9, the initial processing unit 60 illustrates a situation in which the imaging unit 10 captures a portion of the image HFI-0 projected onto the area PA as a captured image CAI. . The captured image CAI captured at this time is stored in the storage unit 42. The phase calculation unit 64 reads the captured image CAI stored in the storage unit 42 and detects the luminance value at each point on the read captured image CAI plane. Here, the luminance value at each point on the captured image plane is expressed by the following equation (1).

Here, n = 0, 1, 2, and 3. _{n = I n (x, y} ) corresponding to each of 0, 1, 2, 3, coordinates representing each point on the captured image plane of the captured each image HFI-0,1,2,3 (x , Y) represents the luminance value. Therefore, I ₀ (x, y) represents a luminance value at coordinates (x, y) representing each point on the captured image CAI plane. A (x, y) represents the amplitude of a wave representing a change in luminance on each captured image plane at a point represented by coordinates (x, y). B (x, y) represents the luminance value when the amplitude at the point represented by the coordinates (x, y) of the wave representing the luminance change on each captured image plane is zero. θ (x, y) represents a phase at coordinates (x, y) representing each point on the captured image plane. By combining the four equations for I ₀ (x, y) to I ₃ (x, y) shown in equation (1), the phase at coordinates (x, y) representing each point on each captured image plane θ (x, y) can be expressed as the following equation (2).

  The phase calculation unit 64 is based on Expression (2) and the luminance value at the coordinates (x, y) representing each point on each captured image plane detected by the phase calculation unit 64 described above. The phase θ (x, y) at the coordinates (x, y) representing each point is calculated. Returning to FIG. After the phase calculation unit 64 calculates the phase θ (x, y) at each point on each captured image plane in step S130, the coordinate association unit 66 calculates the phase θ (x, y) calculated by the phase calculation unit 64. Based on the above, the coordinate association processing for associating the coordinates on the captured image plane with the coordinates on the projection image plane is performed (step S140).

Here, referring to FIG. 9 again, the coordinate association processing by the coordinate association unit 66 will be described. Based on the equation (2) and the luminance value at coordinates (x, y) representing each point on each captured image plane detected by the phase calculation unit 64 described above, the phase θ (x , Y) can take a range of −π / 2 ≦ θ (x, y) ≦ π / 2. However, since changes in luminance on the projected image plane and the captured image plane are repeated periodically, there are a plurality of in-phase lines. In the example shown in FIG. 9, the phases of the points existing on the same phase lines SPL1, SPL2, and SPL3 on the projection image plane all have the same value. For this reason, the coordinate association unit 66 keeps the phase θ (x ₁ , y ₁ ) of the point P (x ₁ , y ₁ ) existing on the same phase line SPL3 on the captured image CAI plane as it is in the area PA. It is impossible to determine which phase is on the same phase line SPL1, SPL2, or SPL3 on the image HFI-0 plane projected onto the image HFI-0.

  Therefore, the coordinate association unit 66 in the present embodiment performs the phase connection process on the basis of a predetermined position common to the projection image and the captured image. Phase connection processing means that each time the phase changes by π / 2 from a predetermined reference position, π / 2 is added to or subtracted from the phase at that point on the projected image plane and the captured image plane. This is a process that makes it possible to make one-to-one correspondence between the same phase lines of a wave representing a change in luminance. The coordinate association unit 66 uses, for example, a straight line extending in the vertical direction on the image VIMG plane with respect to the images HFI-0 to HFI-3 and a captured image obtained by capturing them as the predetermined position serving as a reference. . The coordinate association unit 66 uses, for example, a straight line extending in the horizontal direction on the image HIMG plane with respect to the images VFI-0 to VFI-3 as the reference predetermined position.

In the example shown in FIG. 9, since the projection image is the image HFI-0, the coordinate association unit 66 sets the straight line extending in the vertical direction on the image VIMG plane as a straight line VL representing a predetermined position as a reference. Then, phase connection processing is performed with reference to the straight line VL. Since the coordinate association unit 66 includes the straight line VL in both the area PA and the area CA, the phase θ at the point P (x ₁ , y ₁ ) on the captured image plane is obtained by performing the phase connection process. (X ₁ , y ₁ ) can be associated with the phase of each point on the corresponding in-phase line SPL3 on the projection image plane. By this association, the coordinate association unit 66 has a point P ′ on the projection image plane corresponding to the point P (x ₁ , y ₁ ) on the captured image plane anywhere on the same phase line SPL3. Confirm that. The predetermined reference position may be a reference point, a reference pattern, or the like instead of a straight line such as the reference line VL. In either case, the predetermined position is always on the projected image plane and the captured image plane. Must be included in both.

The coordinate association unit 66 performs the phases performed on the images HFI-0 to HFI-3 with respect to the images VFI-0 to VFI-3 in which a wave representing a change in luminance proceeds in the vertical direction on the projection image plane. Processing similar to the connection processing is performed. By these processes, the phase calculation unit 64 extends in the longitudinal direction on the projection image plane and has the same phase as the phase θ (x ₁ , y ₁ ) at the point P (x ₁ , y ₁ ) on the captured image plane. The phase line extending in the horizontal direction and the phase line extending in the horizontal direction are obtained, so that the coordinate correspondence unit 66 has a point P where the phase line extending in the vertical direction and the phase line extending in the horizontal direction intersect each other. '(X' ₁ , y ' ₁ ) can be associated with a point P (x ₁ , y ₁ ) on the captured image plane. By this association, the coordinate association unit 66 can more accurately associate coordinates representing points on the captured image plane with coordinates representing points on the projection image plane.

  The coordinate association unit 66 represents a point on the captured image plane when the total number of pixels on the captured image plane of the imaging unit 10 is larger than the total number of pixels on the projected image plane of the projection unit 20. You may perform the process which matches a coordinate and the coordinate showing the point on a projection image plane with subpixel accuracy. For example, the coordinate association unit 66 performs the process of associating with subpixel accuracy by dividing the coordinates on the projection image plane and generating and complementing the coordinate values corresponding to the coordinates on the captured image plane. In this way, by associating the coordinates representing the points on the captured image plane with the coordinates representing the points on the projected image plane with sub-pixel accuracy, the three-dimensional point group of the object OBJ by the phase shift method is obtained. When generating, it is possible to realize a point group number equal to or higher than the imaging resolution of the imaging unit 10.

  Returning to FIG. In step S140, the coordinate association unit 66 associates the coordinates representing the points on the captured image plane with the coordinates representing the points on the projected image plane, and then the calibration processing unit 68 performs the calibration shown in FIG. The image RIMG shown in FIG. 4 is projected onto the projection board, and a captured image obtained by capturing the projected calibration board with the imaging unit 10 is read from the storage unit 42. And the calibration process part 68 produces | generates a calibration image based on the read captured image (step S150). The calibration image is a three-dimensional model of the captured image X1 of the calibration board imaged by the imaging unit 10.

  Next, for example, the calibration processing unit 68 generates a projection image in which the imaging unit 10 can capture the same captured image as the captured image X1 by performing affine transformation on the calibration image. Then, the calibration processing unit 68 derives the translation and rotation transformation matrix used for the affine transformation at this time as information indicating the positional relationship between the imaging unit 10 and the projection unit 20 (step S160).

  Hereinafter, the 3D point cloud image generation process performed by the 3D point cloud generation unit 70 will be described with reference to FIG. FIG. 10 is a flowchart illustrating an example of the flow of a 3D point cloud image generation process executed by the 3D point cloud generation unit 70. In the following description, it is assumed that, after the calibration process by the initial processing unit 60 is completed, the user installs the object OBJ in the area where the area PA and the area CA overlap. First, the imaging time determination unit 72 selects one of the images HFI-0 to HFI-3 and / or one unselected image from the images VFI-0 to VFI-3 shown in FIG. Select and read from the storage unit 42 (step S200). In the following description, it is assumed that the imaging time determination unit 72 has selected one unselected image among the images HFI-0 to HFI-3.

  Next, the imaging time determination unit 72 determines whether or not there is an unselected image in step S200 (step S210). When it is determined that there is an unselected image (step S210-Yes), the imaging time determination unit 72 controls the projection control unit 50 to control the projection control unit 50 to input the image selected and read in step S200 to the object OBJ. (Step S220). Next, the imaging time determination unit 72 determines whether or not the projection image projected during the previous subroutine execution related to the projection image selected in step S200 has completely switched to the projection image projected during the current subroutine execution. Therefore, by controlling the imaging control unit 46, the imaging unit 10 is caused to capture the area CA in which the object OBJ is installed as a switching determination image (step S230). Note that the switching determination image is an example of a time-series image. Next, the imaging time determination unit 72 divides the switching determination image captured in step S230 into a plurality of regions. And the imaging time determination part 72 detects the brightness | luminance or color for every pixel in the divided | segmented area | region for every one part or all the divided | segmented area | region (step S240). The divided areas may overlap, may be adjacent to each other, or may not be adjacent to each other, but the area includes a part or all of a periodic change in luminance or color. You must be out.

  Here, the process of dividing the switching determination image into a plurality of regions will be described with reference to FIG. FIG. 11 is a diagram illustrating an example of a region when the switching determination image is divided into a plurality of regions. The image PIC is an example of a switching determination image captured by the imaging unit 10 in step S230. Since the image PIC was captured in the middle of switching from the image HFI-0 to another image, different projected images were captured in the upper and lower stages of the image PIC. The area DA is one of the areas obtained by dividing the image PIC into a plurality of parts in step S240. As shown in FIG. 11, the area DA includes a periodic change in luminance or color in the area. The imaging time determination unit 72 detects the luminance or color at each pixel in the area DA.

  Returning to FIG. The imaging time determination unit 72 determines the luminance or color in each pixel for each of a plurality of areas into which the switching determination image captured during the previous subroutine execution related to imaging in step S230 and the switching captured during the current subroutine execution. The difference between the luminance or the color in each pixel for each region into which the determination image is divided is calculated (step S250). Here, referring to FIG. 12 and FIG. 13, the luminance of each pixel for each divided area at the time of the previous subroutine execution and the luminance of each pixel for each divided area at the time of the current subroutine execution are calculated. Processing for calculating the difference will be described. FIG. 12 is an example of a graph plotting the projected image projected during the previous subroutine execution regarding the projection image projected in step S200, the pixels on the projected image plane projected during the current subroutine execution, and the luminance of the pixels. It is.

  The solid line W1 is, for example, a sine wave graph in which the luminance value for each pixel in a plurality of regions obtained by dividing the image HFI-0 projected during the previous subroutine execution is plotted. The dotted line W2 is, for example, a sine wave graph in which the luminance value for each pixel in a plurality of regions obtained by dividing the image HFI-1 projected at the previous subroutine execution is plotted. The periodic changes in luminance between the image HFI-0 and the image HFI-1 are out of phase with each other by π / 2. Reflecting this, the solid line W1 and the dotted line W2 are out of phase with each other by π / 2. The imaging time determination unit 72 calculates a luminance difference DL between the solid line W1 and the dotted line W2 for each pixel.

  When the difference DL calculated by the imaging time determination unit 72 is plotted for each pixel, a graph shown in FIG. 13 is obtained. FIG. 13 is an example of a graph in which the difference calculated by the imaging time determination unit 72 is plotted for each pixel. Here, when switching from the projection image projected at the previous subroutine execution related to the projection image projected at step S200 to the projection image projected at the current subroutine execution has been completed, the pixel of FIG. In many cases, the calculated difference exceeds a predetermined threshold value Z1. Therefore, when the ratio of the pixels whose difference exceeds the predetermined threshold value Z1 exceeds the predetermined threshold value Z2, the imaging time determination unit 72 calculates the current subroutine from the projection image projected at the previous subroutine execution time. It is determined that the switching to the projected image projected at the time of execution has been completed.

  Returning to FIG. After calculating the difference in step S250, the imaging time determination unit 72 determines whether or not the ratio of the pixels whose calculated difference exceeds the predetermined threshold Z1 exceeds the predetermined threshold Z2 (step S260). . When it is determined that the predetermined threshold value Z2 has not been exceeded (No at Step S260), the imaging time determination unit 72 returns to Step S230 and captures the switching determination image again. On the other hand, when it is determined that the predetermined threshold value Z2 has been exceeded (step S260-Yes), the imaging time determination unit 72 controls the imaging control unit 46 so that the three-dimensional point cloud generation unit 70 can perform the phase shift method. The imaging unit 10 is controlled to capture a projection image as a processing image to be used for (step S270). Then, the imaging time determination unit 72 returns to step S200 and selects the next projection image. The processing image is an example of a processing target image.

  On the other hand, when it is determined in step S210 that there is no unselected image (step S210-No), the 3D point cloud generation unit 70 is an image necessary for generating a 3D point cloud image by the phase shift method. A three-dimensional point cloud image is generated based on a certain processing image. The three-dimensional point cloud generation unit 70 ends the three-dimensional point cloud image generation process. Note that the imaging time determination unit 72 may detect the luminance for each pixel in at least one of the divided regions. In that case, the imaging time determination unit 72 uses the region near the lower right of the image as the divided region because the projection unit 20 sequentially switches the projection image from the upper stage of the image.

  As described above, the control device 40 according to the first embodiment includes the image VIMG and the image in which the straight line as the reference object is captured for each of the images HFI-0 to HFI-3 and the images VFI-0 to VFI-3. The phase of each pixel on the captured image is calculated using the position on the HIMG plane as a base point, and imaging is performed based on the calculated phase of each pixel on the images HFI-0 to HFI-3 and the images VFI-0 to VFI-3. Since the coordinates indicating the position on the captured image plane of the unit 10 and the coordinates indicating the position on the projected image plane of the projection unit 20 are associated with each other, the imaging unit 10 and the projection unit 20 are sufficiently accurate even in an actual environment. Can be calibrated.

  Moreover, the control apparatus 40 is based on the brightness | luminance of the projection light projected by the projection part 20, and the brightness | luminance of the projection light in the image by which the projection light was imaged by the imaging part 10, and image HFI-0-HFI-3 and Since the luminance output of the projection unit 20 that projects the images VFI-0 to VFI-3 and the image VIMG and the image HIMG is determined, it is possible to prevent a reduction in calibration accuracy due to insufficient imaging resolution of the imaging unit 10. .

  The control device 40 also includes a calibration image for deriving a translation and rotation transformation matrix that is information representing the positional relationship between the projection unit 20 and the imaging unit 10, and an image HFI-0 calculated by the calculation unit. Based on the phase of each pixel on the HFI-3 and image VFI-0 to VFI-3 planes, a translation and rotation transformation matrix that is information representing the positional relationship between the projection unit 20 and the imaging unit 10 is derived. Since the calibration process is performed, a three-dimensional point cloud image for deriving the position and orientation of an object to be worked by a robot or the like can be generated with higher accuracy.

  Further, the control device 40 has an image obtained by capturing a calibration board having a red and blue grid pattern, and the images HFI-0 to HFI-3 and the images VFI-0 to VFI-3 calculated by the calculation unit. In order to perform a calibration process for deriving a translation and rotation transformation matrix that is information representing the positional relationship between the projection unit 20 and the imaging unit 10 based on the phase of each pixel, the images HFI-0 to HFI-3 are used. In addition, it is possible to prevent changes in luminance or color of the images VFI-0 to VFI-3 from being made indistinguishable by the grid pattern.

  Further, the control device 40 determines the object OBJ based on the translation and rotation transformation matrix, which is information representing the three-dimensional positional relationship derived by the calibration processing unit, and the image of the object OBJ captured by the imaging unit 10. Since the three-dimensional point cloud image is generated, the three-dimensional position and posture of the object OBJ that is a work target by the robot or the like can be derived with higher accuracy.

  The control device 40 derives the three-dimensional position and orientation of the object OBJ based on the three-dimensional point cloud image, and controls the robot based on the derived three-dimensional position and orientation of the object OBJ. The work on the object OBJ can be performed with higher accuracy.

  In addition, since the robot 30 performs calibration between the imaging unit 10 and the projection unit 20 with sufficient accuracy even in an actual environment, the robot 30 can perform the work on the target object OBJ with higher accuracy.

  In addition, the control device 40 controls the imaging unit 10 to capture the projection image projected from the projection unit 20 as a processing image for use in predetermined image processing, and captures the processing image. Before the three-dimensional point cloud generation unit 70 determines the timing at which the imaging unit 10 captures the processing image based on the comparison between the plurality of switching determination images in which the projection images are captured in time series. It is possible to realize synchronization control with sufficient accuracy at a low cost and without requiring a projection area for synchronization.

  Moreover, since the control apparatus 40 produces | generates a three-dimensional point cloud image based on the image for a process, it can calculate a three-dimensional position and attitude | position more stably.

  Further, the control device 40 detects the luminance or color in each pixel in at least one detection area among the plurality of areas into which the switching determination image is divided, and determines whether to switch the luminance or color detected in the detection area. Since the three-dimensional point cloud generation unit 70 determines the timing at which the imaging unit 10 captures the processing image based on the comparison between the processing images, the timing at which the processing image is captured can be determined at a higher speed. .

  In addition, the control device 40 calculates a luminance or color difference in each pixel between detection areas of the switching determination images that are temporally continuous. Based on the calculated difference, the three-dimensional point cloud generation unit 70 Compared with the synchronization between the projection by the projection unit 20 using the time and the imaging by the imaging unit 10 to determine the timing at which the imaging unit 10 captures the processing image, the synchronization failure due to the processing delay is prevented. can do.

  Further, the control device 40 determines the timing at which the ratio of the number of pixels in which the difference exceeds the predetermined threshold Z1 with respect to the total number of pixels constituting the detection area exceeds the predetermined threshold Z2 as a three-dimensional point. Since the group generation unit 70 determines the timing for causing the imaging unit 10 to capture the processing image, it is possible to prevent the imaging timing from being delayed due to an erroneous determination.

  Further, since the control device 40 derives the three-dimensional position and orientation and controls the robot based on the derived three-dimensional position and orientation, it can be realized at a lower cost and requires a projection area for synchronization. Therefore, more accurate robot control can be realized.

  Further, the robot 30 derives the three-dimensional position and orientation, and controls the actuator based on the derived three-dimensional position and orientation, so that the robot 30 performs a predetermined operation based on synchronous control with sufficient accuracy at a lower cost. be able to.

<Second Embodiment>
Hereinafter, a second embodiment of the present invention will be described with reference to the drawings. In the control device 40 according to the second embodiment, the imaging time determination unit 72 included in the control device 40 according to the first embodiment is omitted. The configuration will be described with reference to FIGS. 1, 2, and 3, with the same reference numerals assigned to the same functional units.

  FIG. 14 is a diagram illustrating an example of a functional configuration of the control system 1 according to the second embodiment. The control device 40 includes, for example, a storage unit 42, an imaging control unit 46, an image acquisition unit 48, a projection control unit 50, an initial processing unit 60, a three-dimensional point group generation unit 70a, and a robot control unit 80. Is provided. The three-dimensional point group generation unit 70 a acquires information indicating the positional relationship between the imaging unit 10 and the projection unit 20 from the initial processing unit 60. The three-dimensional point group generation unit 70a controls the projection control unit 50 to cause the projection unit 20 to sequentially project the images HFI-0 to HFI-3 and / or the images VFI-0 to VFI-3. Further, the three-dimensional point group generation unit 70a controls the imaging control unit 46 to cause the imaging unit 10 to sequentially capture the images projected by the projection unit 20. The three-dimensional point group generation unit 70 is a captured image stored in the storage unit 42, and a plurality of images HFI-0 to HFI-3 and images VFI-0 to VFI-3 projected onto the object OBJ. Based on the captured image and information indicating the positional relationship between the imaging unit 10 and the projection unit 20 acquired from the initial processing unit 60, for example, a three-dimensional point cloud image of the object OBJ is generated by the phase shift method. . The 3D point cloud generation unit 70 outputs the generated 3D point cloud image to the robot control unit 80.

  As described above, the control device 40 according to the second embodiment includes the image VIMG and the image in which the straight line as the reference object is captured for each of the images HFI-0 to HFI-3 and the images VFI-0 to VFI-3. The phase of each pixel on the captured image is calculated using the position on the HIMG plane as a base point, and imaging is performed based on the calculated phase of each pixel on the images HFI-0 to HFI-3 and the images VFI-0 to VFI-3. Since the coordinates indicating the position on the captured image plane of the unit 10 and the coordinates indicating the position on the projected image plane of the projection unit 20 are associated with each other, the imaging unit 10 and the projection unit 20 are sufficiently accurate even in an actual environment. Can be calibrated.

  Moreover, the control apparatus 40 is based on the brightness | luminance of the projection light projected by the projection part 20, and the brightness | luminance of the projection light in the image by which the projection light was imaged by the imaging part 10, and image HFI-0-HFI-3 and Since the luminance output of the projection unit 20 that projects the images VFI-0 to VFI-3 and the image VIMG and the image HIMG is determined, it is possible to prevent a reduction in calibration accuracy due to insufficient imaging resolution of the imaging unit 10. .

  The control device 40 also includes a calibration image for deriving a translation and rotation transformation matrix that is information representing the positional relationship between the projection unit 20 and the imaging unit 10, and an image HFI-0 calculated by the calculation unit. Based on the phase of each pixel on the HFI-3 and image VFI-0 to VFI-3 planes, a translation and rotation transformation matrix that is information representing the positional relationship between the projection unit 20 and the imaging unit 10 is derived. Since the calibration process is performed, a three-dimensional point cloud image for deriving the position and orientation of an object to be worked by a robot or the like can be generated with higher accuracy.

  Further, the control device 40 has an image obtained by capturing a calibration board having a red and blue grid pattern, and the images HFI-0 to HFI-3 and the images VFI-0 to VFI-3 calculated by the calculation unit. In order to perform a calibration process for deriving a translation and rotation transformation matrix that is information representing the positional relationship between the projection unit 20 and the imaging unit 10 based on the phase of each pixel, the images HFI-0 to HFI-3 are used. In addition, it is possible to prevent changes in luminance or color of the images VFI-0 to VFI-3 from being made indistinguishable by the grid pattern.

  Further, the control device 40 determines the object OBJ based on the translation and rotation transformation matrix, which is information representing the three-dimensional positional relationship derived by the calibration processing unit, and the image of the object OBJ captured by the imaging unit 10. Since the three-dimensional point cloud image is generated, the three-dimensional position and posture of the object OBJ that is a work target by the robot or the like can be derived with higher accuracy.

  The control device 40 derives the three-dimensional position and orientation of the object OBJ based on the three-dimensional point cloud image, and controls the robot based on the derived three-dimensional position and orientation of the object OBJ. The work on the object OBJ can be performed with higher accuracy.

  In addition, since the robot 30 performs calibration between the imaging unit 10 and the projection unit 20 with sufficient accuracy even in an actual environment, the robot 30 can perform the work on the target object OBJ with higher accuracy.

DESCRIPTION OF SYMBOLS 1 Control system, 10 Imaging part, 20 Projection part, 30 Robot, 40 Control apparatus, 41 CPU, 42 Storage part, 43 Input reception part, 44 Communication part, 46 Imaging control part, 48 Image acquisition part, 50 Projection control part, 52 Image generation unit, 60 Initial processing unit, 62 Luminance output determination unit, 64 Phase calculation unit, 66 Coordinate correspondence unit, 68 Calibration processing unit, 70, 70a Three-dimensional point group generation unit, 72 Imaging time determination unit, 80 Robot control unit, 82 3D position and orientation derivation unit

Claims (9)

The image projected by the projection unit is a captured image captured by the imaging unit, and a change in luminance or color is repeated periodically, and a plurality of waves having different phases indicating the cycle of the change in luminance or color are mutually different Based on the first captured image and the second captured image in which the reference object projected by the projection unit is captured by the imaging unit, the reference object in the second captured image for each of the first captured images. A calculation unit that calculates a phase of each pixel on the captured image with a position on the captured image plane where the image is captured as a base point;
Correspondence for associating the position of the imaging unit on the captured image plane with the position of the projection unit on the projected image plane based on the phase of each pixel on the captured image calculated by the calculation unit And
The projection light luminance based on the projection light luminance that is the luminance of the projection light projected by the projection unit and the captured image luminance that is the luminance of the projection light in the image obtained by imaging the projection light by the imaging unit. The first captured image is derived on the basis of the derived reference projection light brightness, the projection light brightness at which the captured image brightness starts to change linearly with respect to the change of the projection light brightness. And a determination unit that determines a luminance output of the projection unit that projects the second captured image;
A control device comprising: The control device according to claim 1 ,
A third captured image for deriving information representing a positional relationship between the projection unit and the imaging unit, wherein the image projected by the projection unit is a captured image captured by the imaging unit; A calibration processing unit that performs a calibration process for deriving information representing a positional relationship between the projection unit and the imaging unit based on the phase of each pixel calculated by the calculation unit;
Control device. The control device according to claim 2 ,
The third captured image is an image in which red and blue lattice patterns are captured.
Control device. The control device according to claim 2 or 3 ,
The information representing the positional relationship is information representing a three-dimensional positional relationship between the projection unit and the imaging unit,
A three-dimensional point group generation unit that generates three-dimensional point group information of the object based on information representing the positional relationship derived by the calibration processing unit and an image of the object imaged by the imaging unit; Prepare
Control device. The control device according to claim 4 ,
Based on the 3D point cloud information generated by the 3D point cloud generation unit, the 3D position and orientation of the object are derived, and the robot is controlled based on the derived 3D position and orientation of the object. A robot controller that
Control device. A control device according to claim 5 ;
An actuator for operating the movable part of the robot;
Robot equipped with. A control system comprising a projection unit that projects an image, an imaging unit that captures an image projected by the projection unit, and a control device,
The controller is
The image projected by the projection unit is a captured image captured by the imaging unit, and changes in luminance or color are periodically repeated, and the phases of waves representing the period of change in luminance or color are different from each other. Based on a plurality of first captured images and a second captured image in which the reference object projected by the projecting unit is captured by the image capturing unit, for each of the first captured images, in the second captured image, A calculation unit that calculates a phase of each pixel with a position on a captured image plane where the reference object is captured as a base point;
An association unit that associates a position on the captured image plane of the imaging unit with a position on the projection image plane of the projection unit based on the phase of each pixel calculated by the calculation unit;
The projection light luminance based on the projection light luminance that is the luminance of the projection light projected by the projection unit and the captured image luminance that is the luminance of the projection light in the image obtained by imaging the projection light by the imaging unit. The first captured image is derived on the basis of the derived reference projection light brightness, the projection light brightness at which the captured image brightness starts to change linearly with respect to the change of the projection light brightness. And a determination unit that determines a luminance output of the projection unit that projects the second captured image;
Comprising
Control system. The control unit
The image projected by the projection unit is a captured image captured by the imaging unit, and a change in luminance or color is repeated periodically, and a plurality of waves having different phases indicating the cycle of the change in luminance or color are mutually different Based on the first captured image and the second captured image in which the reference object projected by the projection unit is captured by the imaging unit, the reference object in the second captured image for each of the first captured images. The phase of each pixel is calculated based on the position on the captured image plane where is captured,
Based on the phase of each pixel and the calculated, and the position of the captured image plane of the imaging unit, have rows correspondence between positions on the projection image plane of the projection portion,
The projection light luminance based on the projection light luminance that is the luminance of the projection light projected by the projection unit and the captured image luminance that is the luminance of the projection light in the image obtained by imaging the projection light by the imaging unit. The first captured image is derived on the basis of the derived reference projection light brightness, the projection light brightness at which the captured image brightness starts to change linearly with respect to the change of the projection light brightness. Determining a luminance output of the projection unit that projects the second captured image;
Control method. To the control computer,
The image projected by the projection unit is a captured image captured by the imaging unit, and a change in luminance or color is repeated periodically, and a plurality of waves having different phases indicating the cycle of the change in luminance or color are mutually different Based on the first captured image and the second captured image in which the reference object projected by the projection unit is captured by the imaging unit, the reference object in the second captured image for each of the first captured images. The phase of each pixel is calculated with the position on the captured image plane taken as the base point,
Based on the calculated phase of each pixel, the position on the captured image plane of the imaging unit and the position on the projection image plane of the projection unit are associated ,
The projection light luminance based on the projection light luminance that is the luminance of the projection light projected by the projection unit and the captured image luminance that is the luminance of the projection light in the image obtained by imaging the projection light by the imaging unit. The first image pickup is performed based on the derived reference projection light luminance, wherein the projection light luminance starts to change linearly with respect to the change of the projection light luminance. Determining the luminance output of the projection unit that projects the image and the second captured image;
Control program.