JPWO2020165976A1 - Simulation equipment and simulation method - Google Patents

Abstract

The simulation device (10) measures a three-dimensional measuring device including a projection device that projects light onto a measurement object and an image pickup device that captures an imaging space including the measurement object irradiated with the projected light from the projection device. A measurement condition acquisition unit (101) that acquires measurement condition information indicating conditions, and a virtual image generation unit (102) that generates a virtual image that reproduces the image output by the imaging device based on the measurement condition information. , Executes a three-dimensional measurement process that measures the three-dimensional position of the surface of the object to be measured using a virtual captured image, and outputs a three-dimensional measurement calculation unit (103) that obtains the measured value and a simulation result that includes the measured value. It is characterized by including an output unit (104).

Description

The present invention relates to a simulation device, a simulation method, and a simulation program capable of simulating the measurement results of a three-dimensional measuring device.

Conventionally, a device that performs three-dimensional measurement by using a projection device that projects light onto a measurement object and an imaging device that captures the measurement object is known. For example, Patent Document 1 discloses a three-dimensional measuring device that performs three-dimensional measurement by an active stereo method. The three-dimensional measuring device disclosed in Patent Document 1 projects a geometric pattern from a projection device onto a measurement object, sets a plurality of detection points on the geometric pattern in a captured image output by the imaging device, and sets the detection points as detection points. The three-dimensional position of the surface of the corresponding measurement object is calculated based on the triangular survey method.

Japanese Unexamined Patent Publication No. 2015-203652

However, in the technique described in Patent Document 1, in order to obtain the measurement result, it is necessary to configure the actual environment and prepare a sample of the measurement object. Therefore, there is a problem that the measurement result in the actual environment cannot be evaluated when there is no actual environment or when the number of samples of the measurement object is small.

The present invention has been made in view of the above, and an object of the present invention is to obtain a simulation apparatus capable of evaluating measurement results even when there is no actual environment or when the number of samples of measurement objects is small. And.

In order to solve the above-mentioned problems and achieve the object, the simulation apparatus according to the present invention includes a projection device that projects light onto a measurement object and a photographing object that is irradiated with the projection light from the projection device. A virtual captured image that reproduces the captured image output by the imaging device based on the measurement condition acquisition unit that acquires the measurement condition information indicating the measurement conditions of the three-dimensional measuring device equipped with the imaging device that captures the space. A 3D measurement calculation unit that obtains the measured value by executing a 3D measurement process that measures the 3D position of the surface of the object to be measured using the virtual captured image It is characterized by including an output unit for outputting a simulation result including the simulation result.

According to the present invention, there is an effect that the measurement result can be evaluated even when there is no actual environment or the number of samples of the measurement object is small.

The figure which shows the functional structure of the simulation apparatus which concerns on Embodiment 1 of this invention. A flowchart showing the operation of the simulation apparatus shown in FIG. The figure which shows the detailed functional structure of the measurement condition acquisition part shown in FIG. The figure which shows an example of the display screen output by the simulation apparatus shown in FIG. The figure which shows the functional structure of the virtual photograph image generation part which concerns on Embodiment 2 of this invention. A flowchart showing the operation of the simulation device having the virtual captured image generation unit shown in FIG. The figure which shows the functional structure of the simulation apparatus which concerns on Embodiment 3 of this invention. The figure which shows the functional structure of the virtual photograph image generation part shown in FIG. A flowchart showing the operation of the simulation apparatus shown in FIG. A flowchart showing the details of step S304 shown in FIG. A flowchart showing the details of step S305 shown in FIG. The figure which shows an example of the display screen output by the simulation apparatus shown in FIG. The figure which shows the functional structure of the optical reproduction image generation part which concerns on Embodiment 4 of this invention. The figure which shows the detailed functional structure of the sensor viewpoint data generation part shown in FIG. The figure which shows the detailed functional structure of the map generation part shown in FIG. A flowchart showing the operation of the simulation apparatus according to the fourth embodiment of the present invention. The figure which shows an example of the display screen output by the simulation apparatus which concerns on Embodiment 5 of this invention. The figure which shows the functional structure of the simulation apparatus which concerns on Embodiment 6 of this invention. A flowchart showing the operation of the simulation apparatus shown in FIG. The figure which shows an example of the display screen of the simulation apparatus shown in FIG. The figure which shows the functional structure of the simulation apparatus which concerns on Embodiment 7 of this invention. A flowchart showing the operation of the simulation apparatus shown in FIG. The figure which shows the functional structure of the simulation apparatus which concerns on Embodiment 8 of this invention. A flowchart showing the operation of the simulation apparatus shown in FIG. 23. The figure which shows an example of the display screen output by the simulation apparatus shown in FIG. The figure which shows the dedicated hardware for realizing the function of the simulation apparatus which concerns on Embodiment 1-8 of this invention. The figure which shows the structure of the control circuit for realizing the function of the simulation apparatus which concerns on Embodiment 1-8 of this invention. The figure which shows an example of the hardware configuration for realizing the function of the simulation apparatus which concerns on Embodiment 1-8 of this invention.

Hereinafter, the simulation apparatus, simulation method, and simulation program according to the embodiment of the present invention will be described in detail with reference to the drawings. The present invention is not limited to this embodiment.

Embodiment 1.
FIG. 1 is a diagram showing a functional configuration of the simulation device 10 according to the first embodiment of the present invention. The simulation device 10 includes a measurement condition acquisition unit 101, a virtual captured image generation unit 102, a three-dimensional measurement calculation unit 103, and an output unit 104.

The simulation device 10 simulates the measurement result of the three-dimensional measurement device based on the measurement condition information indicating the measurement conditions of the three-dimensional measurement device that measures the three-dimensional position of the surface of the object to be measured by using the active stereo method. Has a function. The three-dimensional measuring device assumed here includes a projection device that projects light onto a measurement object and an imaging device that captures a photographing space including the measurement object irradiated with the projected light from the projection device. The projected light from the projection device indicates the projection pattern used for the three-dimensional measurement. Hereinafter, the term "projected light" refers to light indicating a projection pattern. The three-dimensional measuring device can perform a three-dimensional measurement process for measuring the three-dimensional position of the surface of the object to be measured based on the projection pattern included in the captured image output by the imaging device. Further, the simulation device 10 may simulate the output result of the system using the three-dimensional measuring device.

The measurement condition acquisition unit 101 acquires measurement condition information indicating the measurement conditions of the three-dimensional measuring device. Details of the measurement condition information will be described later. The measurement condition acquisition unit 101 inputs the acquired measurement condition information to the virtual captured image generation unit 102.

The virtual captured image generation unit 102 is a virtual CG (Computer Graphics) image that reproduces the captured image output by the imaging device included in the three-dimensional measuring device based on the measurement condition information input from the measurement condition acquisition unit 101. Generate a captured image. The virtual captured image generation unit 102 inputs the generated virtual captured image to the three-dimensional measurement calculation unit 103.

The three-dimensional measurement calculation unit 103 executes the three-dimensional measurement process executed by the three-dimensional measurement device using the virtual photographed image input from the virtual photographed image generation unit 102, and acquires the measured value. The three-dimensional measurement calculation unit 103 inputs the acquired measured value to the output unit 104.

The output unit 104 outputs a simulation result including the measurement value input from the three-dimensional measurement calculation unit 103.

FIG. 2 is a flowchart showing the operation of the simulation device 10 shown in FIG. The measurement condition acquisition unit 101 of the simulation device 10 acquires the measurement condition information (step S101). The measurement condition acquisition unit 101 inputs the acquired measurement condition information to the virtual captured image generation unit 102.

The virtual captured image generation unit 102 generates a virtual captured image based on the measurement condition information (step S102). The virtual captured image generation unit 102 inputs the generated virtual captured image to the three-dimensional measurement calculation unit 103.

The three-dimensional measurement calculation unit 103 executes a three-dimensional measurement calculation using the virtual captured image and acquires the measured value (step S103). The three-dimensional measurement calculation unit 103 inputs the acquired measured value to the output unit 104. The output unit 104 outputs a simulation result including the measured value (step S104).

The simulation device 10 can generate a virtual captured image that reproduces the captured image output by the imaging device based on the measurement condition information of the three-dimensional measuring device, and execute the three-dimensional measurement process using the virtual captured image. can. With such a configuration, it is possible to verify the three-dimensional measurement on the simulation without actually installing the projection device and the imaging device and acquiring the actual data. Therefore, it is not necessary to install the 3D measuring device at the site, adjust the hardware and software of the 3D measuring device, and collect the actual data of the measurement object, and the time required for the work is shortened. It is possible to reduce human and physical costs. Therefore, it is possible to specify an appropriate measurement condition by performing a simulation while setting various measurement conditions of the three-dimensional measuring device. Therefore, it is possible to shorten the design period of the three-dimensional measuring device and the verification period of trial and error performed before the three-dimensional measuring device is introduced to the site and put into operation.

FIG. 3 is a diagram showing a detailed functional configuration of the measurement condition acquisition unit 101 shown in FIG. The measurement condition acquisition unit 101 acquires measurement condition information indicating the measurement conditions of the three-dimensional measuring device. The measurement condition acquisition unit 101 includes a projection condition information acquisition unit 201, a shooting condition information acquisition unit 202, a measurement object information acquisition unit 203, and a non-measurement object information acquisition unit 204.

The projection condition information acquisition unit 201 acquires projection condition information indicating the projection condition of the projection device. The projection condition information can include information that can identify at least one of the performance and usage conditions of the projection device. The performance of the projection device is, for example, the resolution and the angle of view, and the state of the projection device is, for example, the position and orientation of the projection device and the state of focus. Further, the projection condition information can include information indicating a projection pattern. The information indicating the projection pattern includes information indicating the pattern of the projection pattern. When there are a plurality of projection patterns, the information indicating the projection patterns may further include information indicating the number of projection patterns. The pattern of the projection pattern is a stripe pattern in which stripes of a predetermined thickness are arranged according to a certain rule for each projection pattern, a dot-like pattern arranged in an irregular arrangement, and the intensity of the projected light in the projection pattern. It may be a smoothly changed gradation pattern or a combination thereof. The above pattern is an example, and a projection pattern of any pattern can be used. The number of projection patterns is any one or more. Further, the information indicating the pattern of the projection pattern may be, for example, information indicating the type of the pattern, or information indicating the intensity distribution of light on a predetermined projection surface.

The shooting condition information acquisition unit 202 acquires shooting condition information indicating the shooting conditions of the imaging device. The shooting condition information can include information that can identify at least one of the performance and the usage state of the imaging device. The performance of the image pickup device is, for example, the resolution and the angle of view, and the state of the image pickup device is, for example, the position and orientation of the image pickup device and the focus state.

The measurement object information acquisition unit 203 acquires information on the measurement object, for example, measurement object information which is information indicating a state and characteristics such as a shape, a position and a posture of the measurement object. The characteristic of the measurement object is, for example, the reflection characteristic of the measurement object. Information indicating the reflection characteristics of the object to be measured includes, for example, at least one of the color, diffuse reflectance, and mirror reflectance of the object to be measured.

The non-measurement object information acquisition unit 204 acquires the non-measurement object information which is the information about the non-measurement object which is the object other than the measurement object or the ambient light other than the projected light existing in the photographing space. The non-measurement object is, for example, a container, a pedestal, or a jig for holding the measurement object. The non-measurement object is not limited to the above example, and an object other than the measurement object that can be reflected in the photographed image taken by the actual photographing device, for example, a wall, a window, another landscape, and the above object. It also includes light such as illumination light that illuminates. The non-measurement object information can include at least one of the position, shape and characteristics of the non-measurement object and information indicating the state of ambient light.

The information indicating the shape of the object included in the measurement object information and the non-measurement object information is a combination of meshes such as 3D-CAD (3Dimensional-Computer-Assisted Drawing), a primitive such as a sphere or a rectangular parallelepiped, or an aggregate thereof. Can be expressed as. The reflection characteristics of an object are used to reproduce the appearance of the object when exposed to light. The information indicating the state of ambient light is used to obtain a shadow effect similar to that of an image actually taken by an imaging device. Further, the object may be in a floating state in the air, or the floor surface and the box for storing the object may be set so that the object exists inside the box. It is preferable that the reflection characteristics of the floor surface and the box can be set in the same manner as the object.

As described above, since the projection condition information and the shooting condition information including the performance such as the resolution and the angle of view of the projection device and the image pickup device can be set individually, the measurement error that may occur in the actual three-dimensional measurement device can be set more accurately. It is possible to reproduce. For example, when the resolution of the image pickup device is lower than the resolution of the projection device, even if the projection pattern of the projection device is made finer, the pattern of the projection pattern cannot be finely discriminated from the captured image of the projection pattern. It causes errors and defects in three-dimensional measurement. Since the virtual captured image generation unit 102 generates a virtual captured image based on the resolution of the imaging device, the fineness of the identifiable projection pattern of the imaging device can be found by analyzing the virtual captured image. Thereby, the limit value of the fineness of the projection pattern that can be discriminated by the imaging device can be estimated as the upper limit value of the resolution of the projection device. Alternatively, the performance of the imaging device corresponding to the resolution of the projection device can be examined. That is, since it is necessary to use the projection device and the imaging device of the three-dimensional measuring device having the performances commensurate with each other, it is preferable to examine the performance of the projection device and the imaging device using the simulation results.

The virtual captured image generation unit 102 generates a virtual captured image that reproduces the captured image output by the imaging device based on the measurement condition information. The virtual shooting image generation unit 102 reproduces the arrangement of the objects existing in the shooting space in the virtual shooting space based on the measurement target information and the non-measurement target information included in the measurement condition information, and the shooting condition information. Based on, the part in the shooting space indicated by the virtual shot image can be specified. Further, the virtual captured image generation unit 102 can reproduce the shadow generated by the projected light in the imaging space by using the reflection model based on the projection condition information, the measurement object information, and the non-measurement object information. The virtual captured image generation unit 102 can reproduce the position indicating the boundary of the shadow generated by the projected light in the captured image output by the imaging device with the pixel value. Here, the error included in the position indicating the boundary of the shadow reduces the visibility of the light in the captured image. The decrease in light visibility means that the three-dimensional position where the projected light is irradiated in the shooting space and the detection result of the projected light irradiation position on the shot image do not match, or the projection device is two-dimensional. When projecting a projection pattern, it means that the shadow boundary of light, for example, the pattern boundary which is the boundary between the area where the projected light is irradiated and the area where the projected light is not irradiated cannot be detected correctly. Therefore, when the visibility of light is lowered, an error occurs in the measurement result of the three-dimensional measurement or the measurement becomes impossible, and the quality of the three-dimensional measurement is deteriorated.

Factors that cause an error in the position of the shadow boundary of light in the captured image may include those caused by the projection device and those caused by the imaging device. The error factors caused by the projection device include mutual reflection in which the projected light from the projection device is reflected by the first surface and then incident on the second surface to illuminate the second surface, and the projection device is out of focus. It is possible that the light is out of focus. The second surface may be the surface of an object different from the first surface, or may be a different part of the same object as the first surface.

Possible error factors caused by the image pickup device include image distortion due to distortion of the lens of the image pickup device, random noise appearing irregularly in the image, and image blur due to the image pickup device being out of focus. The phenomenon treated as an error factor is not limited to the above, and may include any phenomenon that causes a change in the projection state of light or a change in the captured image.

As an example of the above-mentioned situation in which mutual reflection is likely to occur, there is a case where two or more objects are arranged close to each other. The light beam emitted from the projection device may be reflected on the surface of the first object in the photographing space, and the reflected light may illuminate the second object different from the first object. It is also conceivable that the light reflected by the part X of the first object existing in the photographing space illuminates the part Y different from the part X of the same first object.

Mutual reflection can occur not only in the projected light from the projection device, but also in ambient light such as indoor lighting and outdoor sunlight. In addition, the reflection of light rays is not limited to one time, and multiple reflections may occur. However, since light absorbs energy each time it is reflected, if the number of reflections is repeated, the sensitivity of the light falls below the observation sensitivity of the image pickup apparatus and often does not affect the captured image. Therefore, mutual reflection due to reflected light more than a predetermined number of times may be ignored.

The intensity of light due to mutual reflection can be determined based on the intensity of light before reflection, the reflection characteristics of the reflection point, and the traveling path of light, in addition to the same reflection model as when generating a virtual captured image.

When mutual reflection occurs, a shadow boundary of light may be observed in addition to the pattern boundary that should be observed on the image. In the part where the boundary other than the pattern boundary is observed, the light irradiation direction cannot be calculated correctly, which causes a measurement error in the three-dimensional measurement.

Since the virtual photographed image generation unit 102 generates a virtual photographed image in consideration of the influence of the mutual reflection of light, the measurement value output by the three-dimensional measurement calculation unit 103 is a measurement target in which mutual reflection may actually occur. The measurement error that can occur when three-dimensional measurement is performed on an object is reproduced. Therefore, it is possible to grasp the magnitude and appearance tendency of the measurement error before actually configuring the three-dimensional measuring device. If the appearance tendency of measurement error can be grasped in advance, a method of reducing the influence of mutual reflection will be examined by changing the measurement conditions such as the positional relationship between the projection device and the imaging device or the arrangement method of the measurement object. You can also do it. Therefore, it is possible to improve the accuracy of three-dimensional measurement.

When the projection device is out of focus, the shadow boundary of the light is blurred and the pattern boundary is blurred. Therefore, for the purpose of specifying the light irradiation direction, the analysis accuracy of the projection pattern on the captured image is lowered, and the light irradiation is performed. The direction may be calculated incorrectly. When the shooting space has a certain depth or more, it is possible that the projection device cannot be focused on the entire shooting space. In such a case, it is necessary to study a calculation method that can accurately analyze the projection pattern even when the projection device is out of focus. However, even if you try to verify the analysis method of the projection pattern using the actual data, which is the data obtained by actually operating the projection device and the imaging device and observing the object to be measured, it is necessary to know the true position of the pattern boundary in the actual data. It is difficult. Therefore, it is not possible to strictly judge whether the verification result is correct. On the other hand, the simulation device 10 can acquire the true position of the pattern boundary based on the measurement condition information, and generates a virtual captured image that reproduces the blurring of the pattern boundary due to the out-of-focus of the projection device. .. Therefore, as a result of improving the projection pattern analysis method, it becomes possible to easily determine whether or not the detection result of the pattern boundary is improved. Therefore, it becomes possible to streamline the development of the projection pattern analysis algorithm required for three-dimensional measurement.

In addition, since the relationship between the degree of focus shift of the projection device and the measurement result of the three-dimensional measurement can be simulated, it is necessary to verify how much the focus shift is within the permissible range. It is possible to estimate the required performance such as the range of depth of field where the projection device is in focus. Therefore, the simulation device 10 has an effect that the design of the three-dimensional measuring device can be facilitated.

Further, the virtual captured image generation unit 102 can calculate the image distortion as an effect of the distortion aberration caused by the lens of the image pickup apparatus. As an example of the image distortion model, the correspondence between the pixels before and after the distortion occurs in the image can be obtained by using the following mathematical formula (1).

_{Here, (x u, y} u) is the image coordinates of the undistorted _{image, (x d, y} d) is the image coordinates of the image with a distortion, K is a coefficient indicating the degree of strain And r is the distance from the center of the image to the pixel of interest. The pixel of interest is a pixel at arbitrary coordinates (x _d , y _d ) of the distorted image. The virtual captured image generation unit 102 may sequentially select all the pixels of the distorted image as the pixels of interest. Many image distortions have been proposed, from simplified models to detailed models. The simulation device 10 can use any model including the mathematical formula (1) as the calculation formula of the image distortion.

The virtual captured image generation unit 102 can reproduce random noise by setting the appearance probability and intensity of noise for each pixel or for each fixed region on the image. The virtual photographed image generation unit 102 determines whether or not to add noise to the pixel or area using the appearance probability, and if it is determined to add noise, changes the color of the pixel or area based on the set intensity. .. As a method of specifying the intensity, the rate of change with respect to the color of the original pixel or region may be used, or an integer value may be used. The intensity may be indicated using a fixed rate of change or an integer value, or may have a range. Further, the intensity can take both a positive value meaning an increase in the brightness of the pixel and a negative value meaning a decrease in the brightness of the pixel.

The virtual photographed image generation unit 102 can reproduce the image blur caused by the out-of-focus of the image pickup apparatus by using the color information of the pixels around the pixel of interest. Any pixel on the image may be selected as the pixel of interest. As a method of calculating the color after the image blurring occurs, there are a method of using the average value of the colors in the pixel of interest and the pixels around it, and Gaussian smoothing in which pixels closer to the pixel of interest are synthesized at a higher ratio. Using Gaussian smoothing has the advantage that image blur can be reproduced more accurately than the method of taking an average value. Further, the method of taking the average value has an advantage that the processing time is faster than the method using Gaussian smoothing. The image blur also changes depending on the distance from the image pickup device to the object. Objects that are in focus have less blur, and objects that are out of focus are imaged as a larger and blurry image. By further using information such as the change in the degree of blurring due to the distance and the in-focus distance, the virtual photographed image generation unit 102 can reproduce the blurring closer to the actual photographed image. The information may be acquired as measurement condition information, or may be obtained by calculation from other information included in the measurement condition information. A specific method for the virtual captured image generation unit 102 to reproduce the error factor will be described in Examples 2 and 3, but the reproduction method is not limited thereto.

The three-dimensional measurement calculation unit 103 executes the three-dimensional measurement process by inputting the virtual captured image. The three-dimensional measurement process executed by the three-dimensional measurement calculation unit 103 may be the same as the three-dimensional measurement process performed by the three-dimensional measurement device using the captured image taken in the actual environment. The three-dimensional measurement process includes a process of identifying the projection pattern irradiated on the object to be measured and a process of measuring the distance. The projection pattern identification process acquires luminance information of pixels at the same position from a plurality of captured images, determines whether or not each pixel is illuminated by a projection device, and acquires a combination thereof. In addition to the above, there is a method of analyzing the pattern of the local projection pattern around the pixel of interest as the pattern identification process. If the local projection pattern is designed to be unique in the entire pattern, it is possible to determine where the portion of the entire projection pattern projected from the projection device that is illuminated around the pixel of interest is located. Can be identified. The purpose of each pattern recognition method is to uniquely obtain the vector from the projection device to the pixel of interest.

When the vector from the projection device to the pixel of interest is obtained by the projection pattern identification process, the three-dimensional measurement calculation unit 103 contains sensor information indicating the arrangement conditions such as the positions and orientations of the projection device and the image pickup device, which are included in the measurement condition information. Based on the above, the distance measurement process is performed based on the principle of triangulation. The details of the distance measurement process depend on the projection pattern used. Since the projection pattern used by the simulation device 10 may be any projection pattern, the distance measurement process is selected according to the projection pattern to be used.

FIG. 4 is a diagram showing an example of a display screen 20 output by the simulation device 10 shown in FIG. The display screen 20 includes a processing result display area 21, a measurement condition display area 22, a measurement condition list display area 23, a display content selection area 24, an execute button 25, a save button 26, and an end button 27. include. The simulation result is displayed in the processing result display area 21. In the measurement condition display area 22, individual measurement conditions are displayed, and the displayed measurement conditions can be changed and input. In the measurement condition list display area 23, the saved measurement conditions are displayed in a list. When one is selected from the list of measurement conditions displayed in the list display area 23, the selected measurement conditions are displayed in the measurement condition display area 22. In the display content selection area 24, an operation unit for selecting the content to be displayed in the processing result display area 21 is displayed. Here, the captured image reproduced by the virtual captured image generation unit 102 and the measurement data are options. It is displayed.

The execution button 25 is an operation unit for executing the simulation process using the measurement conditions displayed in the measurement condition display area 22. The save button 26 is an operation unit for saving the measurement conditions displayed in the measurement condition display area 22. The end button 27 is an operation unit for ending the simulation process of the three-dimensional measurement.

When the execution button 25 is operated, the measurement condition acquisition unit 101 acquires the measurement conditions displayed in the measurement condition display area 22 and inputs them to the virtual captured image generation unit 102. When the virtual captured image generation unit 102 generates a virtual captured image and inputs it to the three-dimensional measurement calculation unit 103, and the three-dimensional measurement calculation unit 103 executes the three-dimensional measurement process and inputs the measurement result to the output unit 104, it is output. The unit 104 outputs the processing results obtained in the process of the simulation processing such as the virtual captured image and the measured value, the input data to the simulation processing such as the measurement condition information, and the like to the processing result display area 21.

The output unit 104 may perform processing such as emphasizing the point of interest, adjusting the contrast, and removing noise so that the output is easy for the user to see. The display screen 20 shown in FIG. 4 is an example, and is not limited to the example of FIG. By using the display screen 20 shown in FIG. 4, the user can confirm the repeated simulation result while sequentially adjusting the measurement conditions.

As described above, according to the first embodiment of the present invention, a virtual captured image that reproduces the captured image output by the actual imaging device is generated, and three-dimensional measurement processing is performed based on the virtual captured image. .. By adopting such a configuration, it is possible to obtain the measurement result of the three-dimensional measurement without actually configuring the three-dimensional measuring device. Further, the simulation device 10 can reproduce the error of the position of the light included in the captured image acquired by the actual three-dimensional measuring device. The error is caused by, for example, image distortion due to distortion, random noise, image blur, mutual reflection, and the like. When actually configuring a 3D measurement device, it is difficult to add these errors to the actual captured image later, but by reproducing these errors on the virtual captured image, the simulation result of 3D measurement The accuracy of Further, if the projection pattern and the error can be reproduced with good reproducibility in the virtual photographed image, it is possible to use a normal process for the three-dimensional measurement process.

Embodiment 2.
FIG. 5 is a diagram showing a functional configuration of the virtual captured image generation unit 102 according to the second embodiment of the present invention. The virtual captured image generation unit 102 includes an optical reproduction image generation unit 301 and an image quality deterioration processing unit 302. Although not shown, an apparatus including the virtual captured image generation unit 102 shown in FIG. 5 and showing the same configuration as the simulation apparatus 10 shown in FIG. 1 is referred to as a simulation apparatus 12 according to the second embodiment. Since the simulation device 12 has the same configuration as that of the first embodiment shown in FIG. 1 except for the virtual captured image generation unit 102, detailed description thereof will be omitted here. Hereinafter, the same components as those in the first embodiment will be described with reference to the reference numerals shown in FIG. 1, and the parts different from the first embodiment will be mainly described.

The optical reproduction image generation unit 301 performs an optical simulation based on the measurement condition information and generates an optical reproduction image that reproduces the captured image. The image quality deterioration processing unit 302 performs image quality deterioration processing on the optically reproduced image according to an error factor. The virtual captured image generation unit 102 sets the image after the deterioration processing as a virtual captured image.

FIG. 6 is a flowchart showing the operation of the simulation device 12 having the virtual captured image generation unit 102 shown in FIG. The measurement condition acquisition unit 101 acquires the measurement condition information (step S201). The measurement condition acquisition unit 101 inputs the acquired measurement condition information to the virtual captured image generation unit 102.

The optical reproduction image generation unit 301 of the virtual captured image generation unit 102 generates an optical reproduction image based on the measurement condition information (step S202). The optical reproduction image generation unit 301 inputs the generated optical reproduction image to the image quality deterioration processing unit 302.

The image quality deterioration processing unit 302 executes image quality deterioration processing on the optically reproduced image (step S203). The image quality deterioration processing unit 302 inputs the image after the image quality deterioration processing to the three-dimensional measurement calculation unit 103 as a virtual captured image.

The three-dimensional measurement calculation unit 103 performs a three-dimensional measurement process using the virtual captured image and obtains a measurement result (step S204). The three-dimensional measurement calculation unit 103 inputs the measurement result to the output unit 104. The output unit 104 outputs a simulation result including the measurement result (step S205).

The optical reproduction image generation unit 301 calculates the imaging position in the photographing space corresponding to each pixel in the image, and whether or not the calculated imaging position is irradiated with the projected light, that is, the projected light of the projection device is the imaging position. Is determined. First, the optical reproduction image generation unit 301 calculates a _{vector V cam} passing through each pixel from _{the optical center O cam} of the imaging device based on the sensor information and the information indicating the arrangement and characteristics of the object existing in the photographing space. do. The optical reproduction image generation unit 301 detects a _{point P obj} on the surface of the object that first intersects the _{vector V cam.} By such a calculation, it is possible to grasp the object imaged in each pixel.

Subsequently, the optical reproduction image generation unit 301 _{determines whether or not the point P obj} on the surface of the object is illuminated by the projection device. The optical reproduction image generation unit 301 first calculates a vector V _{proj from the optical center O proj} of the projection device toward the point P _obj. The optical reproduction image generation unit 301 determines whether or not the _{vector V proj} is included in the range in which the projection pattern is irradiated from the projection device by using the sensor information and the pattern information indicating the projection pattern of the projection device. .. When included, the point P _obj on the surface of the object can be determined to be illuminated by the projection device.

In addition to the above calculation result, the optical reproduction image generation unit 301 can determine the color of each pixel by using the information indicating the reflection characteristic and the state of ambient light included in the measurement condition information. Typical reflection models used to determine color include Lambertian reflection for diffuse reflection and Phong reflection model for specular reflection. However, the reflection model used by the optical reproduction image generation unit 301 is not limited to these, and any reflection model can be used.

In the generation of the optically reproduced image, since the image is an aggregate of discrete pixels, the boundary of the object and the boundary of the projection pattern of the projection device may be included in the range imaged by one pixel. .. In that case, it is more natural to use a pixel color as a color in which the colors of two or more objects constituting the boundary are mixed. However, the above-mentioned vector V _cam can detect only one intersection in the photographing space. Therefore, among the objects constituting the boundary, only the color of the object that intersects the _{vector V cam is determined as the pixel color.} As a result, an image may be generated in which the boundaries of objects and projection patterns look unnatural. This is a kind of phenomenon caused by so-called quantization error, and is often a problem in the field of image processing.

In order to solve the problem of the phenomenon caused by the quantization error, an optically reproduced image is created at a resolution higher than the resolution of the virtual photographed image, and the image is reduced to the image size in the image quality deterioration process. By adding, there is a method of generating a virtual photographed image. For example, the optical reproduction image generation unit 301 generates an optical reproduction image at a resolution four times the resolution of the virtual captured image to be finally output. In this case, the color of each pixel of the virtual captured image can be determined by using the color information of the four pixels closest to the pixel in the optically reproduced image. As a result, it is possible to generate a virtual captured image in which the representation near the boundary between the object and the projection pattern is natural.

The optical reproduction image generation unit 301 generates an optical reproduction image by using, for example, information on mutual reflection and focus shift of the projection device, which are error factors caused by the projection device, among the error factors. Since these affect the projection state of the projection pattern, for example, when the focus shift occurs, the optical reproduction image generation unit 301 blurs the shadow boundary of light including the pattern boundary in the optical reproduction image. The optical reproduction image generation unit 301 makes, for example, the brightness of the pixel including the incident point where the reflected light reflected by the first surface is incident on the second surface higher than the brightness when the influence of mutual reflection is not taken into consideration. Therefore, mutual reflection can be reproduced. Further, the optical reproduction image generation unit 301 can reproduce the blurring of the light shadow boundary by adjusting the brightness of the pixel corresponding to the light shadow boundary. As a specific method for reproducing the mutual reflection and a specific method for reproducing the blurring of the shadow boundary of light due to the out-of-focus of the projection device, for example, the method described in the third embodiment may be used. can.

The image quality deterioration processing unit 302 deteriorates the image quality of the optically reproduced image by using, for example, information on image distortion, random noise, and out-of-focus of the image pickup device, which are error factors caused by the image pickup device. For example, the image quality deterioration processing unit 302 performs a filtering process for flattening a change in brightness in an image to blur the outline of an object, a shadow boundary of light, or the like, or change the brightness of pixels at randomly selected positions. By doing so, the image quality of the optically reproduced image can be deteriorated.

According to the simulation apparatus 12 according to the second embodiment of the present invention, it is possible to obtain an optically reproduced image which is an image reproducing an optical effect including a shadow due to light being shielded by an object by an optical simulation. can. Further, by performing the deterioration processing of the image quality on the optically reproduced image, it becomes possible to accurately reproduce the actual photographed image.

Further, by explicitly separating the process of reproducing the optical phenomenon and the process of reproducing the deterioration of the image quality, it becomes easy to analyze the error factors of the three-dimensional measurement independently. When verifying the measurement error of 3D measurement using an actual 3D measurement device, it is difficult to determine the ratio of each error factor affecting it, but by using the simulation device 12, it is possible to use the simulation device 12. It is possible to separate and verify the measurement error caused by the projection device and the measurement error caused by the imaging device. Therefore, it is possible to accurately design a three-dimensional measuring device and accurately examine the performance of devices such as a projection device and an imaging device. Further, in order to verify using an actual three-dimensional measuring device in an actual environment, it is necessary to prepare equipment and measurement objects corresponding to various measurement conditions, which requires a great deal of cost and time. According to the simulation device 12, since the examination can be carried out by simulation, the three-dimensional measuring device can be evaluated under more measurement conditions, and there is an effect that the product guarantee becomes easy.

Embodiment 3.
FIG. 7 is a diagram showing a functional configuration of the simulation device 13 according to the third embodiment of the present invention. The simulation device 13 includes a measurement condition acquisition unit 101, a virtual captured image generation unit 102, a three-dimensional measurement calculation unit 103, an output unit 104, and an error information acquisition unit 105. Hereinafter, the parts different from the simulation device 12 will be mainly described.

The simulation device 13 has an error information acquisition unit 105 in addition to the configuration of the simulation device 12. The error information acquisition unit 105 acquires error information indicating an error factor represented by a virtual captured image from the outside. The error information includes at least one of the strength of the error and the order of addition for each type of error factor. The error information acquisition unit 105 inputs the acquired error information to the virtual captured image generation unit 102.

FIG. 8 is a diagram showing a functional configuration of the virtual captured image generation unit 102 shown in FIG. 7. In the simulation device 13, the virtual photographed image generation unit 102 includes an optical reproduction image generation unit 301, an image quality deterioration processing unit 302, and an error factor determination unit 303.

Based on the error information, the error factor determination unit 303 determines the processing conditions of the error factor addition processing including at least one of the strength and the addition order of the error factor addition processing that generates the optically reproduced image and performs the deterioration processing of the image quality. do. The error factor determination unit 303 inputs the determined processing conditions to the optical reproduction image generation unit 301.

The optical reproduction image generation unit 301 generates an optical reproduction image based on the measurement condition information, the error information, and the processing conditions input from the error factor determination unit 303. The image quality deterioration processing unit 302 performs image quality deterioration processing on the optically reproduced image based on the measurement condition information and the error information.

FIG. 9 is a flowchart showing the operation of the simulation device 13 shown in FIG. The measurement condition acquisition unit 101 acquires the measurement condition information (step S301). The measurement condition acquisition unit 101 inputs the acquired measurement condition information to the virtual captured image generation unit 102.

The error information acquisition unit 105 acquires error information (step S302). The error information acquisition unit 105 inputs the acquired error information to the virtual captured image generation unit 102. The process of step S301 and the process of step S302 may be performed in parallel.

The error factor determination unit 303 of the virtual captured image generation unit 102 determines the processing conditions of the error factor addition processing including at least one of the strength of the error factor and the addition order (step S303). The error factor determination unit 303 inputs the determined processing conditions to the optical reproduction image generation unit 301.

The optical reproduction image generation unit 301 generates an optical reproduction image by an optical simulation based on the processing conditions and the measurement condition information (step S304). The optical reproduction image generation unit 301 inputs the generated optical reproduction image to the image quality deterioration processing unit 302.

The image quality deterioration processing unit 302 executes the image quality deterioration processing of the optically reproduced image based on the measurement condition information and the error information (step S305). The image quality deterioration processing unit 302 inputs the image after the image quality deterioration processing to the three-dimensional measurement calculation unit 103 as a virtual captured image.

The three-dimensional measurement calculation unit 103 executes the three-dimensional measurement process using the virtual captured image and obtains the measurement result (step S306). The three-dimensional measurement calculation unit 103 inputs the measurement result to the output unit 104. The output unit 104 outputs a simulation result including the measurement result (step S307).

FIG. 10 is a flowchart showing the details of step S304 shown in FIG. The optical reproduction image generation unit 301 first generates an optical reproduction image that does not include an error factor (step S401). As described above, the optical reproduction image generation unit 301 discriminates between the projected area, which is a region directly illuminated by the projected light, and the non-projected region, which is an unilluminated region, and draws the projected region in a brighter color than the non-projected region. do.

Subsequently, the optical reproduction image generation unit 301 acquires the order of adding error factors from the error information (step S402). Further, the optical reproduction image generation unit 301 acquires the type of error factor and the intensity for each type from the error information (step S403). Hereinafter, the optical reproduction image generation unit 301 determines whether or not to add an error to the optical reproduction image according to the addition order acquired in step S402. The optical reproduction image generation unit 301 first determines whether or not to add mutual reflection to the optical reproduction image based on the error information (step S404).

When adding mutual reflection (step S404: Yes), the optical reproduction image generation unit 301 adds mutual reflection to the optical reproduction image (step S405). Specifically, when mutual reflection occurs, the reflected light reflected by the first surface of the object is incident on the second surface. At this time, the optical reproduction image generation unit 301 sets the brightness of the pixel including the incident point where the reflected light is incident on the second surface as a pixel whose brightness is increased by mutual reflection. The optical reproduction image generation unit 301 increases the brightness based on the position and orientation of the object including the first surface and the object including the second surface, and the surface characteristics of the first surface and the second surface. decide. The brightness of each pixel of the optically reproduced image is the second brightness obtained by adding the amount of increase in brightness due to mutual reflection to the first brightness of the pixel when the influence of mutual reflection is not taken into consideration.

In the process of step S405, the optical reproduction image generation unit 301 does not necessarily have to calculate the brightness increase due to mutual reflection in the reflection of all light. For example, for a surface with low mirror reflectance of light, the amount of increase in brightness may be smaller than the resolution of the image, so for a surface with mirror reflectance below the threshold, there is no increase in brightness due to mutual reflection. Can be regarded as.

When the optical reproduction image generation unit 301 finishes the process of adding the mutual reflection in step S405, it determines whether or not the error adding process is finished (step S406). When the error addition processing is completed (step S406: Yes), the optical reproduction image generation unit 301 ends the error addition processing and inputs the optical reproduction image to the image quality deterioration processing unit 302. If the error addition process is not completed (step S406: No), the optical reproduction image generation unit 301 returns to the process of step S403.

When no mutual reflection is added (step S404: No), the optical reproduction image generation unit 301 determines whether or not to add the optical blur of the projection device (step S407).

When adding the optical blur of the projection device (step S407: Yes), the optical reproduction image generation unit 301 adds the optical blur of the projection device to the optical reproduction image (step S408). The blurring of the light shadow boundary can be expressed by the brightness of the pixels. Specifically, the optical reproduction image generation unit 301 is the same as the first area, which is the area where light is projected when it is assumed that there is no focus shift in the optical reproduction image, based on the measurement condition information, and the projection device. It is determined from the second region, which is a region where light is not projected when it is assumed that there is no out-of-focus. The optical reproduction image generation unit 301 identifies pixels within a predetermined distance from the boundary between the first region and the second region based on the discrimination result. The maximum brightness of these pixels is the brightness of the region where light is projected, and the minimum brightness is the brightness of the region where light is not projected. The brightness of a pixel may be determined near the boundary based on the distance between the pixel of interest and the boundary. The brightness of the pixels may be determined so that the closer the pixel of interest is to the first region, the closer to the maximum value of brightness, and the closer the pixel of interest is to the second region, the closer to the minimum value of brightness. By changing the brightness based on the shortest distance between the pixel of interest and the first region in this way, it is possible to reproduce a state in which the projection device is out of focus.

When the optical reproduction image generation unit 301 finishes the process of step S408, the optical reproduction image generation unit 301 proceeds to step S406. When the optical blur of the projection device is not added (step S407: No), the optical reproduction image generation unit 301 determines whether or not to add the ambient light to the optical reproduction image (step S409).

When the ambient light is added (step S409: Yes), the optical reproduction image generation unit 301 adds the ambient light to the optical reproduction image (step S410). Ambient light is represented by at least one of the brightness and color of the pixels. When the optical reproduction image generation unit 301 finishes the process of step S410, the optical reproduction image generation unit 301 proceeds to step S406. When no ambient light is added (step S409: No), the optical reproduction image generation unit 301 ends the error addition process.

FIG. 11 is a flowchart showing the details of step S305 shown in FIG. The image quality deterioration processing unit 302 first acquires the order of adding error factors based on the error information (step S501). The image quality deterioration processing unit 302 further acquires the type of error factor and the intensity of the error based on the error information (step S502). Hereinafter, the image quality deterioration processing unit 302 determines whether or not to add an error according to the addition order acquired in step S501.

First, the image quality deterioration processing unit 302 determines whether or not to add image distortion (step S503). When it is determined that the image distortion is to be added (step S503: Yes), the image quality deterioration processing unit 302 adds the image distortion to the optically reproduced image (step S504). After adding the image distortion, the image quality deterioration processing unit 302 determines whether or not to end the image quality deterioration processing (step S505). When it is determined that the image quality deterioration processing is finished (step S505: Yes), the image quality deterioration processing unit 302 ends the processing. When it is determined that the image quality deterioration processing is not completed (step S505: No), the image quality deterioration processing unit 302 returns to the processing in step S502.

When it is determined that the image distortion is not added (step S503: No), the image quality deterioration processing unit 302 subsequently determines whether or not to add random noise (step S506). When it is determined that random noise is to be added (step S506: Yes), the image quality deterioration processing unit 302 adds random noise to the optically reproduced image (step S507). When the process of step S507 is completed, the image quality deterioration processing unit 302 proceeds to the process of step S505.

When it is determined that the random noise is not added (step S506: No), the image quality deterioration processing unit 302 determines whether or not to add the image blur (step S508). When it is determined that the image blur is not added (step S508: No), the image quality deterioration processing unit 302 ends the process. When it is determined that the image blur is to be added (step S508: Yes), the image quality deterioration processing unit 302 adds the image blur to the optically reproduced image (step S509). When the process of step S509 is completed, the image quality deterioration processing unit 302 proceeds to the process of step S505.

The image quality deterioration processing unit 302 may add only one type of error factor to the optical reproduction image, or may add the same error factor to the optical reproduction image a plurality of times. For example, when the addition of random noise and the addition of image blur are alternately executed a plurality of times, it is possible to obtain an effect in which color unevenness occurs in the appearance of the target object. In this way, various image quality effects can be reproduced by combining the order of addition of error factors and the intensity.

FIG. 12 is a diagram showing an example of the display screen 30 output by the simulation device 13 shown in FIG. 7. The display screen 30 includes a processing result display area 21, a measurement condition display area 22, a measurement condition list display area 23, a display content selection area 24, an execute button 25, a save button 26, and an end button 27. It has an error factor setting area 28.

The display screen 30 has an error factor setting area 28 in addition to the components of the display screen 20 described in the first embodiment. Hereinafter, the same parts as the display screen 20 will be omitted, and the parts different from the display screen 20 will be mainly described.

The error factor setting area 28 is an area for setting the strength of the error and the order of adding the error for each type of error factor. The user can input and set the strength of the error and the order of addition in the error factor setting area 28. The error information acquisition unit 105 can acquire the error information displayed in the error factor setting area 28 when the execution button 25 is operated.

As described above, the simulation device 13 according to the third embodiment of the present invention can execute the simulation reflecting the error information set by the user by the error information acquisition unit 105. With such a configuration, the user can evaluate the three-dimensional measurement performed under the desired operating conditions. Further, since the type of error factor, the order of addition and the intensity of adding the error to the optically reproduced image can be selected, there is an effect that the variation of the reproducible image quality can be increased. Since it is possible to perform tests by randomly changing the operating conditions and usage environment, it is possible to perform tests with a combination of conditions that the designer could not have expected in advance, and the test contents will be comprehensive. As a result, the effect of increasing the reliability of the three-dimensional measuring device can be expected.

Embodiment 4.
FIG. 13 is a diagram showing a functional configuration of the optical reproduction image generation unit 301 according to the fourth embodiment of the present invention. The optical reproduction image generation unit 301 includes a sensor viewpoint data generation unit 401, a map generation unit 402, and an image composition unit 403.

A device (not shown) including the optical reproduction image generation unit 301 shown in FIG. 13 and having the same configuration as the simulation device 13 shown in FIG. 7 is referred to as a simulation device 14 according to the fourth embodiment. The simulation device 14 has the same configuration as the simulation device 13 shown in FIG. 7, and the configuration of the optical reproduction image generation unit 301 of the virtual photographed image generation unit 102 is different from that of the simulation device 13. Hereinafter, the description of the components similar to those of the simulation device 13 will be omitted, and the parts different from the simulation device 13 will be mainly described.

The sensor viewpoint data generation unit 401 generates sensor viewpoint data including a first image showing the photographing space seen from the imaging device and distance data from each of the imaging device and the projection device to an object in the imaging space. The sensor viewpoint data generation unit 401 inputs the generated sensor viewpoint data to the map generation unit 402.

FIG. 14 is a diagram showing a detailed functional configuration of the sensor viewpoint data generation unit 401 shown in FIG. The first image generated by the sensor viewpoint data generation unit 401 includes a bright image in which the entire shooting space is represented by the brightness when the projected light is irradiated, and a dark image in which the shooting space is not irradiated with the projected light. including. The sensor viewpoint data generation unit 401 includes a bright image generation unit 501 that generates a bright image, a dark image generation unit 502 that generates a dark image, and a distance data generation unit 503 that generates distance data.

The sensor viewpoint data generation unit 401 generates a first image including a bright image generated by the bright image generation unit 501, a dark image generated by the dark image generation unit 502, and distance data generated by the distance data generation unit 503. Output the sensor viewpoint data including.

Returning to the description of FIG. The sensor viewpoint data output by the sensor viewpoint data generation unit 401 is input to the map generation unit 402. Based on the measurement condition information, the map generation unit 402 generates an irradiation map which is a map showing different numerical values of the first region where the light from the projection device is illuminated and the second region where the light from the projection device is not illuminated. do. For example, in the irradiation map, the first region can be represented by "1" and the second region can be represented by "0". The map generation unit 402 inputs the generated irradiation map to the image composition unit 403.

FIG. 15 is a diagram showing a detailed functional configuration of the map generation unit 402 shown in FIG. The map generation unit 402 includes an irradiation area calculation unit 601, an irradiation map generation unit 602, an optical blur reproduction unit 603, a reflection area calculation unit 604, and a reflection map generation unit 605.

The irradiation area calculation unit 601 calculates an area where the light from the projection device can be illuminated based on the measurement condition information. The irradiation area calculation unit 601 inputs information indicating the calculated area to the irradiation map generation unit 602.

The irradiation map generation unit 602 uses the information from the irradiation area calculation unit 601 and the measurement condition information to obtain a first region in which the light from the projection device is illuminated and a second region in which the light from the projection device is not illuminated. Generate an irradiation map, which is a map indicated by different numerical values. The irradiation map generation unit 602 inputs the generated irradiation map to the optical blur reproduction unit 603.

The reflection area calculation unit 604 uses the measurement condition information and the sensor viewpoint data to reflect the light emitted from the projection device once on an object in the photographing space, and the intensity and direction of the light rays are changed. By associating the three-dimensional position of the point illuminated by with the coordinates on the first image, the reflection region, which is the region illuminated by the reflected light rays in the first image, is calculated. The reflection area calculation unit 604 inputs the calculated information indicating the reflection area to the reflection map generation unit 605. The reflection map generation unit 605 uses the information indicating the reflection area and the measurement condition information to generate a reflection map, which is a map in which the projection device indicates the reflected area and the area not illuminated by the reflected light with different numerical values for each projection pattern. Generate. For example, in the reflection map, the reflection area can be represented by "1" and the area not illuminated by the reflected light can be represented by "0". The reflection map generation unit 605 inputs the generated reflection map to the light blur reproduction unit 603.

The light blur reproduction unit 603 reproduces the effect of light blur on the irradiation map. Specifically, the light blur reproduction unit 603 adjusts the value of the irradiation map from 0 to 1 according to the degree of blur of the shadow boundary of light. The light blur reproduction unit 603 reproduces the effect of light blur on the reflection map as well as the irradiation map. Specifically, the light blur reproduction unit 603 adjusts the value of the irradiation map from 0 to 1 according to the degree of blur of the shadow boundary of light. The optical blur reproduction unit 603 outputs an irradiation map and a reflection map after adjusting the values.

The image synthesizing unit 403 generates an optically reproduced image by synthesizing a bright image and a dark image included in the first image based on the information of the irradiation map and the reflection map. Specifically, the image synthesizing unit 403 sets the brightness of each pixel of the optically reproduced image to be the brightness obtained from the pixels at the same position in the bright image or the dark image based on the value of the irradiation map, so that the bright image And the dark image are combined. Further, when the values of the irradiation map and the reflection map are adjusted by the light blur reproduction unit 603, the image synthesizing unit 403 can weight the brightness of each pixel based on the adjusted values.

The number of virtual captured images is equal to the number of projection patterns. Therefore, as the number of projection patterns increases, there is a problem that the processing time for creating a virtual captured image becomes long. Therefore, it is effective to implement a process that is common when creating a virtual captured image by irradiating different projection patterns so that it only needs to be performed once. Examples of common processing include the creation of bright and dark images of the imaging space viewed from the viewpoint of the imaging device, the calculation of distance data from the imaging device and the projection device to an object existing in the imaging space, and the like.

The calculation of the area illuminated by the projected light from the projection device differs depending on the projection pattern. However, since the calculation of the region where the projected light is shielded and shaded by the shadow of the object is constant regardless of the projection pattern, it can be extracted as a common process.

When the projection device is out of focus, the pattern projected on the scene is out of focus, so that the pattern recognition accuracy may decrease, and as a result, the accuracy of the three-dimensional measurement may decrease. The blurring of the pattern light does not occur when the image pickup device takes a picture, but occurs when the pattern light is projected from the projection device. Therefore, the phenomenon cannot be reproduced by the method of adding blur to the virtual photographed image.

Therefore, an irradiation map showing an area illuminated by the projected light in the virtual photographed image is created. The irradiation map can be defined as a two-dimensional array having the same number of elements as the virtual captured image, and each element has a real number value from 0 to 1. The area illuminated by the projected light is expressed as 1, the area not reached by the projected light is expressed as 0, and the area illuminated with a lower intensity than usual due to the influence of the blur of the projected light is expressed as a value larger than 0 and smaller than 1. can. This can be expressed by a mathematical formula as shown in the following mathematical formula (2).

Here, I _{i, j} is the brightness in the image coordinates (i, j) of the virtual captured image, P _{i, j} is the value in the coordinates (i, j) of the irradiation map, and B _{i, j} is. , The brightness at the image coordinates (i, j) of the bright image, and S _{i, j} is the brightness at the image coordinates (i, j) of the dark image.

When creating an irradiation map, it is first assumed that the projection device is in focus, that is, there is no blurring of the projected light, and the area illuminated by the projection pattern is calculated. At this point, the value of each element of the irradiation map is 0 or 1. By applying a smoothing filter such as Gaussian smoothing to this irradiation map, the blur of the projected light is reproduced. By applying Gaussian smoothing, the change in the value of the irradiation map near the boundary between the area illuminated by the pattern light and the area not illuminated by the pattern light becomes smooth. By smoothing the change in the value of the boundary portion, the effect of blurring the boundary of the projected light can be obtained, and the blur of the projected light can be reproduced.

FIG. 16 is a flowchart showing the operation of the simulation device 14 according to the fourth embodiment of the present invention. The sensor viewpoint data generation unit 401 generates a bright image and a dark image in the bright image generation unit 501 and the dark image generation unit 502 (step S601). The sensor viewpoint data generation unit 401 generates distance data for each viewpoint in the distance data generation unit 503 in parallel with step S601 (step S602). The sensor viewpoint data generation unit 401 inputs the sensor viewpoint data including the generated bright image, dark image, and distance data to the map generation unit 402.

The map generation unit 402 calculates the irradiation area in the irradiation area calculation unit 601 (step S603). Subsequently, the map generation unit 402 generates an irradiation map in the irradiation map generation unit 602 (step S604).

The map generation unit 402 acquires the order in which the error factors are added (step S605). Subsequently, the map generation unit 402 acquires the type and intensity of the error factor (step S606). The map generation unit 402 determines whether or not to add mutual reflection (step S607). When it is determined that mutual reflection is to be added (step S607: Yes), the map generation unit 402 causes the reflection area calculation unit 604 to calculate the reflection area (step S608) and causes the reflection map generation unit 605 to generate a reflection map (step S607: Yes). S609).

When it is determined that the mutual reflection is not added (step S607: No), the processes of step S608 and step S609 are omitted. Subsequently, the map generation unit 402 determines whether or not to add optical blur (step S610). When it is determined that the light blur is to be added (step S610: Yes), the light blur reproduction unit 603 of the map generation unit 402 reproduces the light blur on the irradiation map (step S611).

When it is determined that the optical blur is not added (step S610: No), the process of step S611 is omitted. Subsequently, the map generation unit 402 determines whether or not to add ambient light (step S612). When it is determined that the ambient light is to be added (step S612: Yes), the irradiation map generation unit 602 of the map generation unit 402 adds the ambient light to the irradiation map (step S613). When it is determined that the ambient light is not added (step S612: No), the process of step S613 is omitted.

The map generation unit 402 determines whether or not to end the error addition process (step S614). When it is determined that the error addition process is not completed (step S614: No), the process returns to step S606. When it is determined that the error addition processing is completed (step S614: Yes), the image synthesizing unit 403 executes the image synthesizing process (step S615).

As described above, according to the simulation apparatus 14 according to the fourth embodiment of the present invention, a plurality of first images, irradiation maps, and reflection maps, which are images that individually reproduce optical phenomena, are created. It is possible to simplify the process of generating an optically reproduced image and the data structure when the projection pattern of is used. In addition, by expressing the blurring effect of the shadow boundary of light caused by the out-of-focus using the irradiation map and expressing the effect of mutual reflection using the reflection map, the out-of-focus and mutual of the projection device can be expressed by simple arithmetic processing. It has the effect of being able to reproduce captured images in which complex optical phenomena such as reflection are combined.

When an actual three-dimensional measuring device is configured and evaluated, only the result in a state where all the optical phenomena are mixed can be obtained. On the other hand, according to the simulation device 14, each of a plurality of types of optical phenomena can be reproduced individually. Therefore, it is possible to know in more detail which factor caused the measurement error caused by the three-dimensional measurement. Such a configuration has the effect of facilitating the design and performance evaluation of the three-dimensional measuring device.

Embodiment 5.
The simulation device 15 (not shown) according to the fifth embodiment has the same configuration as the simulation device 13 shown in FIG. 7. The output display screen of the simulation device 15 is different from that of the simulation device 13. Hereinafter, the same parts as the simulation device 13 will be omitted, and the parts different from the simulation device 13 will be mainly described.

FIG. 17 is a diagram showing an example of a display screen 40 output by the simulation device 15 according to the fifth embodiment of the present invention. The simulation device 15 can output a simulation result including at least a part of the measurement condition information and at least one of the virtual captured images.

The display screen 40 can include an adjustment item display area 41, a save button 42, an end button 43, and a measurement result display area 44. The adjustment item display area 41 is an area in which the setting values of the items to be adjusted are displayed in a list. The measurement result display area 44 is an area for displaying the measured values when the items of the measurement conditions to be adjusted are set to the respective set values displayed in the list. The save button 42 is an operation unit for performing an operation for saving the measurement result, and the end button 43 is an operation unit for performing an operation for ending the process.

On the display screen 40, the set value of the item of the measurement condition to be adjusted, the measured value which is the processing result, the virtual photographed image obtained in the processing process as the intermediate result, the irradiation map, and the like are displayed side by side. Further, the difference result based on the processing result at any of the set values may be displayed together, or the portion where the difference in the processing result is large may be highlighted.

As described above, according to the simulation device 15 according to the fifth embodiment of the present invention, the user can output the measurement condition information, the virtual photographed image, and the like together with the measured value, so that the user can perform a three-dimensional operation for each measurement condition. You can check the process of the original measurement result. At this time, by displaying the processing results and the intermediate results corresponding to a plurality of measurement conditions side by side, it is possible to compare and examine the set values of the measurement conditions. Therefore, there is an effect that the arrangement and performance of the projection device and the image pickup device can be easily examined.

Embodiment 6.
FIG. 18 is a diagram showing a functional configuration of the simulation device 16 according to the sixth embodiment of the present invention. The simulation device 16 includes a measurement condition acquisition unit 101, a virtual photographed image generation unit 102, a three-dimensional measurement calculation unit 103, an output unit 104, an error information acquisition unit 105, an evaluation reference data generation unit 106, and measurement evaluation. It has a part 107.

The simulation device 16 has an evaluation reference data generation unit 106 and a measurement evaluation unit 107 in addition to the configuration of the simulation device 13. Hereinafter, detailed description of the configuration similar to that of the simulation device 13 will be omitted, and parts different from the simulation device 13 will be mainly described.

The evaluation standard data generation unit 106 uses the measurement condition information to generate evaluation standard data that serves as an evaluation standard for the simulation result. The evaluation standard data generation unit 106 inputs the generated evaluation standard data to the measurement evaluation unit 107. The measurement evaluation unit 107 evaluates the simulation result using the evaluation reference data and acquires the simulation evaluation. The output unit 104 outputs the simulation evaluation result in addition to the simulation result.

FIG. 19 is a flowchart showing the operation of the simulation device 16 shown in FIG. Since the operation shown in FIG. 19 is the same as that in FIG. 9 from step S301 to step S307, detailed description thereof will be omitted. In the present embodiment, after the three-dimensional measurement process is completed, the evaluation reference data generation unit 106 generates the evaluation reference data (step S701). Subsequently, the measurement evaluation unit 107 performs a measurement evaluation process of comparing the measurement data obtained by the simulation with the evaluation reference data and calculating a quantitative evaluation value (step S702). The output unit 104 outputs the simulation evaluation (step S703).

The evaluation reference data includes the distance data from the imaging device included in the sensor viewpoint data, the processing result of three-dimensional measurement when the virtual captured image before adding the error factor is input, and the processing such as the irradiation map. Data obtained in the process and actual measurement data obtained by actually measuring the object to be measured can be considered.

Theoretically, as the simulation evaluation, it is conceivable to use the ratio of the missing area where the measurement data cannot be obtained to the measurable area which is the area where the measurement data can be obtained, the measurement error when compared with the evaluation reference data, and the like. .. The measurable region can be, for example, a region in which no loss has occurred in the measurement data obtained by inputting the virtual captured image before adding the error factor. In addition, statistics such as mean value, variance, standard deviation, and maximum value obtained by comparing the evaluation reference data and the simulation result may be used as the evaluation index.

When the actual measurement data is used as the evaluation reference data, it is also useful to provide feedback so that the simulation evaluation is improved, that is, the difference between the actual measurement data and the simulation result is small.

As a feedback method, for example, it is conceivable to generate a plurality of different simulation results and acquire a setting value having the best simulation evaluation. In order to obtain a plurality of simulation results, for example, there is a method of changing the value of the surface characteristic of the object to be measured under the measurement conditions or changing the intensity of the error factor.

Parameters such as the surface characteristics of the object to be measured and the strength of error factors when the simulation evaluation is the best can be defined as the optimum simulation settings. By providing the user with this optimum simulation setting, a more accurate verification experiment can be performed.

FIG. 20 is a diagram showing an example of the display screen 50 of the simulation device 16 shown in FIG. The display screen 50 includes an evaluation reference data display area 51 in addition to the components of the display screen 40.

On the display screen 50, the evaluation reference data is displayed along with a plurality of set values of the adjustment target items of the measurement conditions. In addition, an intermediate result obtained from a simulation of a three-dimensional measurement process using a reference value may also be displayed. Further, the display screen 50 can further display the simulation evaluation.

As described above, in the simulation device 16 according to the sixth embodiment of the present invention, evaluation reference data can be obtained in addition to the simulation results. The evaluation reference data is data that serves as a reference when evaluating the simulation result, and is actual measurement data or the like. The simulation device 16 can facilitate the examination of the simulation result by displaying the measured value included in the simulation result and the evaluation reference data side by side.

Further, the simulation device 16 can obtain a simulation evaluation in which the simulation result is evaluated using the evaluation reference data. With such a configuration, measurement errors and defects can be quantitatively grasped. Therefore, it becomes easy to examine the performance of the imaging device and the projection device, and to determine the suitability of the three-dimensional measurement for each measurement object. Further, when the performance evaluation method is defined by the standard, it is possible to easily determine the suitability by obtaining the simulation evaluation using the performance evaluation method.

Embodiment 7.
FIG. 21 is a diagram showing a functional configuration of the simulation device 17 according to the seventh embodiment of the present invention. The simulation device 17 has an object recognition processing unit 108 and a recognition evaluation unit 109 in addition to the configuration of the simulation device 16 according to the sixth embodiment. Hereinafter, the parts different from the simulation device 16 will be mainly described.

The object recognition processing unit 108 inputs the simulation result output by the three-dimensional measurement calculation unit 103 and the measurement condition information output by the measurement condition acquisition unit 101, and inputs the position and orientation of the object existing in the photographing space and the object. The recognition result including at least one of the gripping position which is the gripping position is acquired. The object recognition processing unit 108 inputs the recognition result to the recognition evaluation unit 109.

The recognition evaluation unit 109 evaluates the recognition result based on the measurement condition information. Specifically, the recognition evaluation unit 109 acquires the recognition evaluation result including at least one of the estimation accuracy of the position and orientation included in the recognition result and the estimation accuracy of the gripping position. The recognition evaluation unit 109 inputs the recognition result of the object and the recognition evaluation result to the output unit 104. The output unit 104 outputs the recognition result and the recognition evaluation result in addition to the simulation result and the simulation evaluation.

FIG. 22 is a flowchart showing the operation of the simulation device 17 shown in FIG. Since the operations of steps S301 to S307 and steps S701 to S703 are the same as those in FIG. 19, description thereof will be omitted here. Hereinafter, the parts different from those in FIG. 19 will be mainly described.

When the three-dimensional measurement process shown in step S306 is completed, the object recognition processing unit 108 executes the object recognition process and acquires the recognition result (step S801). When the recognition result is obtained, the recognition evaluation unit 109 executes a recognition evaluation process for evaluating the recognition result (step S802). When the recognition evaluation result is obtained, the output unit 104 outputs the recognition evaluation result (step S803). Here, in step S803, the recognition result may be output in addition to the recognition evaluation result.

One of the purposes of performing three-dimensional measurement of an object is to recognize the position and orientation of the object to be measured. For example, an application program that causes a robot to grasp an object is applicable. When the position and orientation of the object to be gripped are not known in advance, it is necessary to sense the object on the spot and recognize the position and orientation of the object or the position where the robot can grip.

Therefore, in the seventh embodiment of the present invention, the object recognition processing unit 108 that performs the object recognition processing by inputting the simulation result of the three-dimensional measurement is provided. Any algorithm may be used as the algorithm for object recognition in the object recognition processing unit 108. The object recognition algorithm may input a three-dimensional point group that treats the result of the three-dimensional measurement as a set of points in the three-dimensional space, or may input a depth image expressing the three-dimensional measurement result as a two-dimensional image.

Furthermore, using the result of object recognition, the recognition result such as the estimation accuracy of the position and orientation of the recognized object and the estimation accuracy of the gripping position is also evaluated. When evaluating an algorithm that recognizes the position and orientation of an object, the problem in many cases is that the true value of the position and orientation of the object is unknown. Since the true value is not known, it is difficult to quantitatively judge whether the object is good or bad even if the recognition result of the position and orientation of the object is output. However, since the position and orientation of the object are known in the simulation, it is possible to quantitatively evaluate the recognition result.

As described above, according to the simulation device 17 according to the seventh embodiment of the present invention, it is possible to verify the object recognition using the simulation results. With such a configuration, it is possible to evaluate the performance of object recognition without actually configuring a three-dimensional measuring device. Since the position and orientation of the object to be recognized are known in the simulation space, the object recognition result can be compared with the true value, which has the effect of facilitating the quantitative evaluation of the object recognition performance.

In a system that grips an object recognized by a robot, it is very important to quantitatively evaluate whether or not the accuracy of object recognition is sufficient. This is because the success of gripping by the robot is greatly affected by the accuracy of object recognition. However, for example, assuming that a plurality of objects are gripped from a bulky state in which they are stacked irregularly, it is difficult to know the true value of the position and orientation of the bulky objects. Therefore, even if the test is performed using an actual machine including a three-dimensional measuring device and a robot, the exact accuracy of object recognition cannot be quantitatively grasped. According to the present invention, there is an advantage that the true value can be easily obtained by simulation and the accuracy of object recognition can be quantitatively evaluated.

Embodiment 8.
FIG. 23 is a diagram showing a functional configuration of the simulation device 18 according to the eighth embodiment of the present invention. The simulation device 18 has an object grip evaluation unit 110 in addition to the configuration of the simulation device 17 according to the seventh embodiment.

Hereinafter, the description of the same configuration as that of the simulation device 17 will be omitted, and the parts different from the simulation device 17 will be mainly described.

The object grasping evaluation unit 110 is based on the measurement condition information output by the measurement condition acquisition unit 101, the simulation result output by the three-dimensional measurement calculation unit 103, and the recognition result output by the object recognition processing unit 108. Acquire the gripping evaluation result of the object for which the gripping success probability is evaluated. The object grip evaluation unit 110 inputs the grip evaluation result to the output unit 104. The output unit 104 also outputs the grip evaluation result.

FIG. 24 is a flowchart showing the operation of the simulation device 18 shown in FIG. 23. Since the operations of steps S301 to S307, steps S701 to S703, and steps S801 to S803 are the same as those in FIG. 22, the description thereof will be omitted here. Hereinafter, the parts different from those in FIG. 22 will be mainly described.

When the object recognition result is obtained, the object grip evaluation unit 110 executes the object grip evaluation (step S901). The object grip evaluation unit 110 inputs the grip evaluation result to the output unit 104. The output unit 104 outputs the grip evaluation result (step S902). The operation of step S901 can be executed in parallel with the recognition evaluation process.

By using the object recognition result, it is possible to obtain information on the position where the robot hand grips the object. By using the information, it is possible to simulate the position of contact with the object and the magnitude and direction of the force generated in the robot hand or the object when the robot hand is moved to the gripping position and the gripping operation is executed.

Knowing the magnitude and direction of the force generated by the robot hand and the object makes it possible to estimate the success of the grip. For example, consider the case where the robot hand is a parallel hand having two claws. Gripping fails when the gripped object comes off from the gap between the claws of the robot hand. A situation corresponding to this case is considered to be a case where the force applied to the gripping object in the direction perpendicular to the closing direction of the parallel hand is larger than the frictional force between the robot hand and the gripping object. Therefore, it is possible to determine whether or not the gripping is successful if there is information such as the friction coefficient of the surface of the robot hand and the gripping object, the weight of the gripping object, and the gripping force of the robot hand.

However, as a method for evaluating the success probability of gripping, not only a method of estimating the force generated between the robot hand and the gripping object but also an arbitrary evaluation method may be used. In the eighth embodiment of the present invention, the object grasping evaluation unit 110 having such an object grasping simulation function by a robot is provided. An example of the output screen according to the eighth embodiment is shown in FIG. FIG. 25 is a diagram showing an example of a display screen 60 output by the simulation device 18 shown in FIG. 23. The display screen 60 has a recognition and grip evaluation result display area 61 in addition to the display contents of the display screen 30. As a method of outputting the recognition result and the grip evaluation result, different symbols may be output depending on whether the recognition or grip is successful or unsuccessful, or the cause of the failure may be output at the time of failure. Some quantitative value used for evaluation may be displayed.

As described above, according to the simulation device 18 according to the eighth embodiment of the present invention, since the gripping success rate can be evaluated on the simulation, there is an effect that the preliminary verification before constructing the robot system becomes easy. ..

In order to build a system that can perform a series of operations from object recognition to gripping using an actual robot, it takes a long time to install a 3D measuring device, adjust the object recognition software, and adjust the robot operation. It costs money. Furthermore, a test in which an object is repeatedly gripped in real space also requires a long time and trial and error. According to the present invention, since the gripping test can be performed by simulation, the test can be performed in a shorter period of time than before.

In addition, when a gripping test is performed using an actual robot system and an attempt is made to analyze the cause of gripping failure, gripping success or gripping is performed in a state where all failure factors are mixed, from errors in 3D measurement to robot motion adjustment errors. Only the result of failure will be obtained. Therefore, it is difficult to analyze where to modify to improve the success rate of gripping. In the present invention, since the cause of failure can be isolated and verified by simulation, the cause can be investigated and improved in a short period of time.

Subsequently, the hardware configurations of the simulation devices 10, 12, 13, 14, 15, 16, 17, and 18 according to the first to eighth embodiments of the present invention will be described. Measurement condition acquisition unit 101, virtual captured image generation unit 102, three-dimensional measurement calculation unit 103, output unit 104, error information acquisition unit 105, evaluation reference data generation unit 106, measurement evaluation unit 107, object recognition processing unit 108, recognition evaluation The unit 109 and the object grip evaluation unit 110 are realized by a processing circuit. These processing circuits may be realized by dedicated hardware, or may be control circuits using a CPU (Central Processing Unit).

When the above processing circuits are realized by dedicated hardware, these are realized by the processing circuit 90 shown in FIG. FIG. 26 is a diagram showing dedicated hardware for realizing the functions of the simulation devices 10, 12, 13, 14, 15, 16, 17, and 18 according to the first to eighth embodiments of the present invention. The processing circuit 90 is a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination thereof.

When the above processing circuit is realized by a control circuit using a CPU, this control circuit is, for example, a control circuit 91 having the configuration shown in FIG. 27. FIG. 27 is a diagram showing a configuration of a control circuit 91 for realizing the functions of the simulation devices 10, 12, 13, 14, 15, 16, 17, and 18 according to the first to eighth embodiments of the present invention. As shown in FIG. 27, the control circuit 91 includes a processor 92 and a memory 93. The processor 92 is a CPU, and is also called a central processing unit, a processing unit, an arithmetic unit, a microprocessor, a microcomputer, a DSP (Digital Signal Processor), or the like. The memory 93 is, for example, a non-volatile or volatile semiconductor memory such as a RAM (Random Access Memory), a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable ROM), or an EEPROM (registered trademark) (Electrically EPROM). Magnetic discs, flexible discs, optical discs, compact discs, mini discs, DVDs (Digital Versatile Disk), etc.

When the above processing circuit is realized by the control circuit 91, it is realized by the processor 92 reading and executing a program stored in the memory 93 corresponding to the processing of each component. The memory 93 is also used as a temporary memory in each process executed by the processor 92.

Further, the functions of the simulation devices 10, 12, 13, 14, 15, 16, 17, and 18 according to the first to eighth embodiments of the present invention can also be realized by using the hardware having the configuration shown in FIG. 28. .. FIG. 28 is a diagram showing an example of a hardware configuration for realizing the functions of the simulation devices 10, 12, 13, 14, 15, 16, 17, and 18 according to the first to eighth embodiments of the present invention. The functions of the simulation devices 10, 12, 13, 14, 15, 16, 17, and 18 according to the first to eighth embodiments of the present invention use the input device 94 and the output device 95 in addition to the processor 92 and the memory 93. Can be realized. The input device 94 is an input interface that accepts input operations from users such as a keyboard, a mouse, and a touch sensor. The output device 95 is, for example, a display device, and can display an output screen to the user. When a touch panel is used, the display device of the touch panel is the output device 95, and the touch sensor superimposed on the display device is the input device 94.

For example, the functions of the measurement condition acquisition unit 101 and the output unit 104 of the simulation devices 10, 12, 13, 14, 15, 16, 17, and 18 may be realized only by the processor 92, or the processor 92 and the input device 94. Alternatively, it may be realized by the output device 95 or an interface with them.

The configuration shown in the above-described embodiment shows an example of the content of the present invention, can be combined with another known technique, and is one of the configurations without departing from the gist of the present invention. It is also possible to omit or change the part.

For example, the functions of the simulation devices 10, 12, 13, 14, 15, 16, 17, and 18 according to the first to eighth embodiments of the present invention may be realized on one hardware or a plurality of functions. It may be distributed in hardware.

Further, the display screens 20, 30, 40, 50, and 60 shown above are examples, and various changes can be made.

10, 12, 13, 14, 15, 16, 17, 18 Simulation device, 20, 30, 40, 50, 60 Display screen, 21 Processing result display area, 22 Measurement condition display area, 23 List display area, 24 Display contents Selection area, 25 execute button, 26, 42 save button, 27, 43 end button, 28 error factor setting area, 41 adjustment item display area, 44 measurement result display area, 51 evaluation standard data display area, 61 recognition and grasp evaluation result Display area, 90 processing circuit, 91 control circuit, 92 processor, 93 memory, 101 measurement condition acquisition unit, 102 virtual shooting image generation unit, 103 three-dimensional measurement calculation unit, 104 output unit, 105 error information acquisition unit, 106 evaluation criteria Data generation unit, 107 measurement evaluation unit, 108 object recognition processing unit, 109 recognition evaluation unit, 110 object grip evaluation unit, 201 projection condition information acquisition unit, 202 shooting condition information acquisition unit, 203 measurement object information acquisition unit, 204 non Measurement target information acquisition unit, 301 optical reproduction image generation unit, 302 image quality deterioration processing unit, 303 error factor determination unit, 401 sensor viewpoint data generation unit, 402 map generation unit, 403 image composition unit, 501 bright image generation unit, 502 Dark image generation unit, 503 distance data generation unit, 601 irradiation area calculation unit, 602 irradiation map generation unit, 603 light blur reproduction unit, 604 reflection area calculation unit, 605 reflection map generation unit.

In order to solve the above-mentioned problems and achieve the object, the simulation apparatus according to the present invention includes a projection device that projects light onto a measurement object and a photographing object that is irradiated with the projection light from the projection device. A virtual captured image that reproduces the captured image output by the imaging device based on the measurement condition acquisition unit that acquires the measurement condition information indicating the measurement conditions of the three-dimensional measuring device equipped with the imaging device that captures the space. A 3D measurement calculation unit that obtains the measured value by executing a 3D measurement process that measures the 3D position of the surface of the object to be measured using the virtual captured image It is characterized by including an output unit for outputting a simulation result including the simulation result. Based on the measurement condition information, the virtual captured image generation unit generates a virtual captured image in which the error included in the position indicating the boundary of the shadow generated by the projected light in the captured image is reproduced by the pixel value.

Claims (29)

Measurement conditions indicating the measurement conditions of a three-dimensional measuring device including a projection device that projects light onto a measurement object and an imaging device that captures an imaging space including the measurement object irradiated with the projected light from the projection device. Measurement condition acquisition unit to acquire information and
A virtual captured image generation unit that generates a virtual captured image that reproduces the captured image output by the imaging device based on the measurement condition information.
A three-dimensional measurement calculation unit that executes a three-dimensional measurement process that measures the three-dimensional position of the surface of the measurement object using the virtual captured image and obtains the measured value.
An output unit that outputs simulation results including the measured values, and
A simulation device characterized by being equipped with. The measurement condition information includes information indicating the projection conditions of the projection device, information indicating the imaging conditions of the imaging device, information indicating the shape of the measurement object, and information indicating the state of the measurement object. The simulation apparatus according to claim 1, further comprising at least one of information indicating the characteristics of the measurement object. The information indicating the projection condition is information that can specify the position / orientation, resolution, and focus of the projection device.
The information indicating the shooting conditions is information that can specify the position / orientation, resolution, and focus of the imaging device.
The information indicating the state of the measurement object includes at least one of the position and orientation of the measurement object.
The simulation apparatus according to claim 2, wherein the information indicating the characteristics of the measurement object includes at least one of the color, the diffuse reflectance, and the mirror surface reflectance of the measurement object. Any one of claims 1 to 3, wherein the measurement condition information further includes information indicating at least one of a position, a shape, and a characteristic of an object other than the measurement object constituting the photographing space. The simulation apparatus described in. The simulation apparatus according to any one of claims 1 to 4, wherein the measurement condition information further includes information indicating a state of ambient light. Based on the measurement condition information, the virtual captured image generation unit generates the virtual captured image in which the error included in the position indicating the boundary of the shadow generated by the projected light in the captured image is reproduced by the pixel value. The simulation apparatus according to any one of claims 1 to 5, which is characterized. The virtual captured image generation unit is caused by mutual reflection illuminated by the light reflected on the surface of the object, blurring of the projected light due to the out-of-focus of the projection device, and curvature aberration of the lens of the imaging device. A claim characterized by generating the virtual captured image that reproduces the error caused by at least one of the image distortion, the random noise, and the image blur caused by the image pickup device being out of focus. Item 6. The simulation apparatus according to Item 6. The virtual captured image generation unit
An optical reproduction image generation unit that generates an optical reproduction image that reproduces the photographed image by an optical simulation based on the measurement condition information.
The optical reproduction image includes an image quality deterioration processing unit that performs image quality deterioration processing based on the error.
The simulation apparatus according to claim 6 or 7, wherein the image after the deterioration processing of the image quality is used as the virtual captured image. The virtual captured image generation unit
An error information acquisition unit that acquires error information indicating the cause of the error represented by the virtual captured image,
With more
The optical reproduction image generation unit generates the optical reproduction image based on the measurement condition information and the error information, and generates the optical reproduction image.
The simulation apparatus according to claim 8, wherein the image quality deterioration processing unit performs image quality deterioration processing on the optically reproduced image based on the measurement condition information and the error information. The error information includes the strength of the error for each type of the cause of the error.
9. The virtual photographed image generation unit is characterized in that, based on the error information, the intensity of the deterioration processing applied to the optically reproduced image is determined for each type of the cause of the error. The simulation device described. The error information includes the order of addition for each type of the cause of the error.
9. 10. The simulation apparatus according to 10. The virtual captured image generation unit calculates the traveling path of the light based on the measurement condition information, and when the projected light is reflected on the first surface and the reflected light is incident on the second surface, the first The invention according to any one of claims 1 to 11, wherein the brightness of the pixel including the incident point of the reflected light on the second surface is increased as compared with the first brightness that does not consider the influence of the reflected light. Simulation equipment. The virtual captured image generation unit uses the amount of increase in the brightness of the pixel to increase the brightness as the surface characteristics of the first surface, the surface characteristics of the second surface, and the second surface of the first surface. The simulation apparatus according to claim 12, wherein the simulation apparatus is determined based on an attitude toward a surface. Based on the measurement condition information, the virtual captured image generation unit calculates a first region which is a region where the projected light is irradiated and a second region which is a region where the projected light is not irradiated. Claims 1 to 13 are characterized in that the brightness of a pixel within a predetermined distance from the boundary between the first region and the second region is determined based on the distance between the pixel and the boundary. The simulation apparatus according to any one of the above items. In the virtual captured image generation unit, the brightness of the pixel within a predetermined distance from the boundary is equal to or less than the maximum value which is the brightness when the projected light is irradiated, and the projected light is not irradiated. The simulation apparatus according to claim 14, wherein the brightness is at least a minimum value. The virtual captured image generation unit brings the brightness of the pixel closer to the maximum value as the pixel is closer to the first region, and reduces the brightness of the pixel to the minimum value as the pixel is closer to the second region. The simulation apparatus according to claim 15, characterized in that they are brought closer to each other. The virtual captured image generation unit
Using the measurement condition information, sensor viewpoint data including a first image showing the imaging space seen from the imaging device and distance data from each of the imaging device and the projection device to an object in the imaging space is obtained. The sensor viewpoint data generator to be generated and
Based on the measurement condition information, a map generation unit that generates an irradiation map that is a map showing a region irradiated with the projected light and a region not irradiated with the projected light with different numerical values in the first image.
Have,
The simulation apparatus according to any one of claims 1 to 16, wherein the virtual captured image is generated based on the irradiation map and the first image. The first image includes a bright image in which the entire shooting space is represented by the brightness when the projected light is irradiated, and a dark image in which the shooting space is not irradiated with the projected light.
The 17th claim, wherein the virtual photographed image generation unit generates the virtual photographed image by synthesizing the bright image and the dark image for each pixel based on the value of the irradiation map. Simulation equipment. The map generator
Using the measurement condition information and the sensor viewpoint data, the three-dimensional position of the point irradiated with the projected light is associated with the coordinates on the first image, and the projected light is generated in the first image. An irradiation area calculation unit that calculates an irradiation area, which is an irradiation area,
An irradiation map generation unit that generates the irradiation map for each projection pattern indicated by the projected light using the irradiation region and the measurement condition information.
18. The simulation apparatus according to claim 18. The map generator
It further has an optical blur reproduction unit that reproduces the blur of light by adjusting the value of the irradiation map according to the degree of blur of the shadow boundary of the light.
Based on the adjusted value of the irradiation map, the brightness of the bright image and the brightness of the dark image are weighted, and the weighted brightness is summed to determine the brightness of each pixel of the virtual captured image. The simulation apparatus according to claim 19. The map generator
Using the measurement condition information and the sensor viewpoint data, a point at which a light beam emitted from the projection device is reflected by an object in the photographing space and is irradiated with a reflected light ray, which is a light ray whose intensity and direction are changed. By associating the three-dimensional position of the above with the coordinates on the first image, a reflection area calculation unit that calculates a reflection area that is a region to which the reflected light rays are irradiated in the first image, and a reflection area calculation unit.
A reflection map that uses the reflection region and the measurement condition information to generate a reflection map, which is a map showing the reflection region and the region not irradiated with the reflected light with different numerical values for each projection pattern indicated by the projected light. Generator and
Have,
The simulation apparatus according to any one of claims 18 to 20, wherein the virtual captured image is generated by using the reflection map, the irradiation map, and the first image. The map generator adjusts the value of the reflection map according to the intensity of the reflected light and the degree of blurring of the shadow boundary of the projected light, and based on the adjusted values of the reflection map and the irradiation map. The simulation apparatus according to claim 21, wherein each of the brightness of the bright image and the brightness of the dark image is weighted, and the weighted brightness is summed to determine the brightness of each pixel of the virtual captured image. The simulation apparatus according to any one of claims 1 to 22, wherein the output unit outputs the simulation result including at least a part of the measurement condition information and at least one of the virtual captured images. .. The measurement condition acquisition unit acquires a plurality of the measurement condition information and obtains the plurality of measurement condition information.
The claim is characterized in that the output unit outputs a list of simulation results in which the information of one or more items designated as adjustment target items among the plurality of measurement condition information and the measured values are associated with each other. The simulation apparatus according to any one of 1 to 23. An evaluation standard data generation unit that generates evaluation standard data, which is an evaluation standard for the simulation results, using the measurement condition information.
A measurement evaluation unit that evaluates the simulation results using the simulation results and the evaluation reference data,
With more
The simulation apparatus according to any one of claims 1 to 24, wherein the output unit outputs an evaluation result in addition to the simulation result. Based on the simulation result, an object recognition processing unit that acquires at least one of the position and orientation of the object in the photographing space and the gripping position that is the position where the object can be gripped as a recognition result.
A recognition evaluation unit that acquires a recognition evaluation result including at least one of the estimation accuracy of the position and the posture and the estimation accuracy of the gripping position by using the recognition result and the measurement condition information.
With more
The simulation apparatus according to any one of claims 1 to 25, wherein the output unit further outputs the recognition evaluation result. An object grasping evaluation unit that evaluates a successful gripping probability of an object in the photographing space by using the measurement condition information and the recognition result.
26. The simulation apparatus according to claim 26. Simulates the three-dimensional measurement processing of a three-dimensional measuring device including a projection device that projects light onto a measurement object and an imaging device that captures an imaging space including the measurement object irradiated with the projected light from the projection device. Simulation device
A step of acquiring measurement condition information indicating the measurement conditions of the three-dimensional measuring device, and
Based on the measurement condition information, a step of generating a virtual captured image that reproduces the captured image output by the imaging device, and
A step of executing a three-dimensional measurement process for measuring the three-dimensional position of the surface of the measurement object using the virtual photographed image and obtaining a measured value, and a step of obtaining the measured value.
The step of outputting the simulation result including the measured value and
A simulation method characterized by including. Measurement conditions indicating the measurement conditions of a three-dimensional measuring device including a projection device that projects light onto a measurement object and an imaging device that captures an imaging space including the measurement object irradiated with the projected light from the projection device. Steps to get information and
Based on the measurement condition information, a step of generating a virtual captured image that reproduces the captured image output by the imaging device, and
A step of executing a three-dimensional measurement process for measuring the three-dimensional position of the surface of the measurement object using the virtual photographed image and obtaining a measured value, and a step of obtaining the measured value.
The step of outputting the simulation result including the measured value and
A simulation program characterized by having a computer execute.

Images