DE112019006855T5 - SIMULATION DEVICE AND SIMULATION METHOD - Google Patents

Abstract

A simulation device (10) includes: a measurement condition obtaining unit (101) that receives measurement condition information indicating a measurement condition of a spatial measurement device that includes a projecting device for projecting light onto a measurement object to be measured and an imaging device for taking an image of an imaging space, which includes the measurement object which is irradiated with projected light from the projecting device; a virtual captured image generation unit that generates a virtual captured image that reproduces a captured image output by the imaging device based on the measurement condition information; a spatial measurement arithmetic unit that performs a spatial measurement process of determining a spatial position on a surface of the measurement object by using the virtual captured image to obtain a measurement value; and an output unit (104) that outputs a simulation result containing the measurement value.

Description

area

The present invention relates to a simulation device, a simulation method and a simulation program which are capable of simulating a measurement result of a device for spatial measurements.

General state of the art

Devices that take spatial measurements using a projecting device that projects light onto a measurement object to be measured and an imaging device that records an image of the measurement object are previously known. Thus, Patent Document 1 discloses a spatial measurement apparatus that makes spatial measurements using an active stereo method. The spatial measurement apparatus disclosed in Patent Document 1 projects a geometric pattern onto a measurement object using a projecting device, sets a plurality of detection points in the geometric pattern on a captured image output by an imaging device, and calculates spatial positions on a surface of the measurement object which correspond to the detection points based on the triangulation method.

List of citations

Patent literature

Patent Document 1: Japanese Patent Application Laid-Open No. 2015-203652

Brief description

Technical task

In the case of the technique described in Patent Document 1, however, it is necessary to provide a real environment and prepare samples of a measurement object in order to obtain measurement results. Accordingly, there has been a problem that measurement results in a real environment cannot be evaluated in a case where there is no real environment or in a case where the number of samples of a measurement object is small.

The present invention has been made in view of the above, and an underlying object is to provide a simulation apparatus capable of evaluating measurement results even in a case where there is no real environment or in a case where the number the samples of a measurement object is small, is able.

Solution of the task

In order to achieve the above objects and to achieve the object, a simulation device according to the present invention comprises: a measurement condition obtaining unit for obtaining measurement condition information indicating a measurement condition of a spatial measurement device that includes a projecting device for projecting light onto a measurement object to be measured and an imaging device Device for picking up an image of an imaging space including the measurement object, which is irradiated with projected light from the projecting device; a virtual captured image generation unit for generating a virtual captured image that reproduces a captured image output by the imaging device based on the measurement condition information; a spatial measurement arithmetic unit for performing a spatial measurement process of determining a spatial position on a surface of the measurement object by using the virtual captured image to obtain a measurement value; and an output unit for outputting a simulation result including the measured value.

Advantageous Effects of the Invention

According to the present invention, there is produced an advantageous effect that evaluation of measurement results is enabled even in a case where there is no real environment or in a case where the number of samples of a measurement object is small .

Figure list

• 1 Fig. 13 is a diagram showing a functional configuration of a simulation device according to a first embodiment of the present invention.
• 2 FIG. 13 is a flow chart illustrating the operation of the FIG 1 illustrated simulation device represents.
• 3 FIG. 13 is a diagram showing a detailed functional configuration of a measurement condition obtaining unit shown in FIG 1 is shown.
• 4th Fig. 13 is a diagram showing an example of a display screen represented by the in 1 simulation device shown is output.
• 5 Fig. 13 is a diagram showing a functional configuration of a virtual captured image generation unit according to a second embodiment of the present invention.
• 6th FIG. 13 is a flowchart showing the operation of a simulation apparatus employing the FIG 5 has shown generation unit for virtual recorded images.
• 7th Fig. 13 is a diagram showing a functional configuration of a simulation device according to a third embodiment of the present invention.
• 8th represents a functional configuration of a generation unit for virtual recording images, which are shown in 7th is shown.
• 9 FIG. 13 is a flow chart illustrating the operation of the FIG 7th illustrated simulation device represents.
• 10 Fig. 3 is a flow chart showing details of the in 9 illustrated step S304.
• 11 Fig. 3 is a flow chart showing details of the in 9 illustrated step S305.
• 12th Fig. 13 is a diagram showing an example of a display screen represented by the in 7th simulation device shown is output.
• 13th Fig. 13 is a diagram showing a functional configuration of an optically reproduced image generation unit according to a fourth embodiment of the present invention.
• 14th FIG. 13 is a diagram showing a detailed functional layout of a sensor viewpoint data generation unit shown in FIG 13th is shown.
• 15th Fig. 13 is a diagram showing a detailed functional configuration of a card generation unit shown in 13th is shown.
• 16 Fig. 13 is a flowchart showing the operation of a simulation device according to a fourth embodiment of the present invention.
• 17th Fig. 13 is a diagram showing an example of a display screen output by a simulation device according to a fifth embodiment of the present invention.
• 18th Fig. 13 is a diagram showing a functional configuration of a simulation device according to a sixth embodiment of the present invention.
• 19th FIG. 13 is a flow chart illustrating the operation of the FIG 18th illustrated simulation device represents.
• 20th FIG. 13 is a diagram showing an example of a display screen on the simulation apparatus shown in FIG 18th is shown.
• 21 Fig. 13 is a diagram showing a functional configuration of a simulation device according to a seventh embodiment of the present invention.
• 22nd FIG. 13 is a flow chart illustrating the operation of the FIG 21 illustrated simulation device represents.
• 23 Fig. 13 is a diagram showing a functional configuration of a simulation device according to an eighth embodiment of the present invention.
• 24 FIG. 13 is a flow chart illustrating the operation of the FIG 23 illustrated simulation device represents.
• 25th Fig. 13 is a diagram showing an example of a display screen represented by the in 23 simulation device shown is output.
• 26th Fig. 13 is a diagram showing dedicated hardware for implementing the functions of the simulation devices according to the first to eighth embodiments of the present invention.
• 27 Fig. 13 is a diagram showing a configuration of a control circuit for implementing the functions of the simulation devices according to the first to eighth embodiments of the present invention.
• 28 Fig. 13 is a diagram showing an example of a hardware configuration for implementing the functions of the simulation devices according to the first to eighth embodiments of the present invention.

Description of embodiments

A simulation apparatus, a simulation method, and a simulation program according to certain embodiments of the present invention will be described in detail below with reference to the drawings. It should be noted that the present invention is not limited to the embodiments.

First embodiment

1 Fig. 16 is a diagram showing a functional configuration of a simulation device 10 according to a first embodiment of the present invention. The simulation device 10 includes a measurement condition maintenance unit 101 , a generation unit for virtual recording images 102 , a computing unit for spatial measurements 103 and an output unit 104 .

The simulation device 10 has functions of simulating a measurement result of a spatial measurement device that measures spatial positions on a surface of a measurement object to be measured, which is done by using the active stereo method based on measurement condition information indicating measurement conditions of the spatial measurement device. The device for spatial measurements assumed here includes a projecting device that projects light onto the measurement object, and an imaging device that records images of an imaging space that contains the measurement object, which is irradiated with projected light from the projecting device. The projected light from the projecting device has a projection pattern that is used for spatial measurement. In the following, “projected light” refers to light that has the projection pattern. The spatial measurement device can perform a process of spatial measurement of spatial positions on a surface of the measurement object on the basis of a projection pattern included in a captured image output by the imaging device. The simulation device 10 can also simulate an output result of a system using the apparatus for spatial measurements.

The measurement condition maintenance unit 101 receives the measurement condition information indicating the measurement conditions of the spatial measurement device. Details of the measurement condition information are described below. The measurement condition maintenance unit 101 inputs the measurement condition information obtained into the generation unit for virtual recording images 102 a.

The generation unit for virtual recording images 102 generates a captured virtual image that is a computer graphic (CG) image that reproduces a captured image output by the imaging device included in the spatial measurement apparatus based on the information obtained from the measurement condition obtaining unit 101 input measurement condition information takes place. The generation unit for virtual recording images 102 sends the generated virtual recording image to the computing unit for spatial measurements 103 a.

The computing unit for spatial measurements 103 adopts the spatial measurement process performed by the spatial measurement apparatus by using that of the virtual captured image generation unit 102 entered virtual recording image in order to obtain a measured value. The computing unit for spatial measurements 103 sends the received measured value to the output unit 104 a.

The output unit 104 outputs a simulation result that corresponds to the calculation unit for spatial measurements 103 the entered measured value.

2 FIG. 13 is a flow chart illustrating the operation of the FIG 1 illustrated simulation device 10 represents. The measurement condition maintenance unit 101 the simulation device 10 obtains the measurement condition information (step S101). The measurement condition maintenance unit 101 inputs the measurement condition information obtained into the generation unit for virtual recording images 102 a.

The generation unit for virtual recording images 102 generates a captured virtual image based on the measurement condition information (step S102). The generation unit for virtual recording images 102 sends the generated virtual recording image to the computing unit for spatial measurements 103 a.

The computing unit for spatial measurements 103 performs spatial measurement calculation by using the captured virtual image to obtain a measurement value (step S103). The computing unit for spatial measurements 103 sends the received measured value to the output unit 104 a. The output unit 104 outputs a simulation result including the measurement value (step S104).

The simulation device 10 can perform the spatial measurement process by generating a captured virtual image that reproduces a captured image output by the imaging device based on the measurement condition information of the spatial measurement device and by using the captured virtual image. Such a configuration enables the spatial measurement to be checked by simulation without providing a projecting device and an imaging device and obtaining real data. This eliminates the need of a workflow of installing the spatial measurement device on site, a workflow of controlling hardware and software of the spatial measurement device, and a workflow of acquiring real data of the measurement object, which shorten the time and labor costs required for workflows as well Can reduce material costs. Thus, the simulation with differently set measurement conditions of the device for spatial measurements enables Simulation thus suitable measurement conditions to be determined. This can thus reduce the time required to set up the spatial measurement device and the time required for trial and error verification before installing and operating the spatial measurement device in the field.

3 Fig. 13 is a diagram showing a detailed functional configuration of the measurement condition obtaining unit 101 represents that in 1 is shown. The measurement condition maintenance unit 101 receives the measurement condition information indicating the measurement conditions of the spatial measurement device. The measurement condition maintenance unit 101 includes a projection condition information obtaining unit 201 , a mapping condition information obtaining unit 202 , a measurement object information acquisition unit 203 and a non-measurement object information acquisition unit 204 .

The projection condition information obtaining unit 201 obtains projection condition information indicating projection conditions of the projecting device. The projection condition information may include information capable of specifying at least one of the performance and the state of use of the projecting device. Examples of the performance of the projecting device include resolution and angle of view, and examples of the state of the projecting device include a position and posture as well as the focusing state of the projecting device. The projection condition information can also include information indicating the projection pattern. The information specifying the projection pattern includes information specifying the design or construction of the projection pattern. It should be noted that, in a case where there are a plurality of projection patterns, the information indicating the projection pattern may further include information indicating the number of the projection patterns. The design of the projection pattern may be a stripe pattern in which stripes having a thickness predetermined for each projection pattern are arranged according to a certain rule, a dot pattern in which dots are arranged irregularly, a gradual pattern which is a projection pattern, which continuously changes the strength of the projected light, or a combination of these. The above-described configurations are examples, and a projection pattern of any configuration can be used. The number of projection patterns is any number greater than or equal to one. In addition, the information specifying the design of the projection pattern can be information specifying the type of design or information specifying the intensity distribution of the light on a predetermined projection plane, for example.

The imaging condition information obtaining unit 202 obtains imaging condition information indicating imaging conditions of the imaging device. The imaging condition information may include information capable of specifying at least one of the performance and the state of use of the imaging device. Examples of the performance of the imaging device include resolution and angle of view, and examples of the condition of the imaging device include the position and posture as well as the focus condition of the imaging device.

The measurement object information preservation unit 203 receives information about the measurement object, for example measurement object information, which is information that specifies the shape, the state such as the position and the posture as well as characteristics of the measurement object, as an example. The characteristics of the measurement object are, for example, the reflection characteristics of the measurement object. The information indicating the reflection characteristics of the measurement object includes at least one of the color, diffuse reflection, and specular reflection of the measurement object, for example.

The non-measurement object information obtaining unit 204 receives non-measurement object information, which is information relating to a non-measurement object, that is to say an object other than the measurement object or ambient light that is not the projected light present in the imaging space. The non-measurement object is, for example, a container, a holder or a frame with which the measurement object is held. It should be noted that the non-measurement object is not limited to the aforementioned examples and also includes an object other than the measurement object that can be captured in an image captured by an actual image capturing device, such as a wall, a window, other surrounding objects, as well Light such as illuminating light with which the aforementioned object is illuminated. The non-measurement object information may include at least one of the position, posture, and characteristics of the non-measurement object as well as information indicating the state of the ambient light.

It should be noted that the information indicating the shape of an object, which belongs to the measurement object information and the non-measurement object information, is based on a combination of grids such as, for example, a three-dimensional computer-aided drawing (3D CAD) from Primitives like a sphere or a right-angled parallelepiped or a set of them can be expressed. The reflective characteristics of an object are used to reproduce what the object looks like when illuminated with light. The information indicating the state of the ambient light is used to obtain such a shadow effect as if an image had actually been taken by the imaging device. In addition, an object may be in the air, or a floor surface and a box for holding an object may be placed and the object may be inside the box. It is preferable that the reflection characteristics of the floor surface and the box can also be adjusted in a manner similar to the object.

Since the projection condition information and the imaging condition information, which include the performance such as the resolution and the angle of view of the projecting device and the imaging device, can be set individually as described above, measurement errors that may occur in an actual spatial measurement device can be avoided can be reproduced with higher accuracy. For example, in a case where the resolution of the imaging device is lower than that of the projecting device, the configuration of the projection pattern cannot be precisely determined from the captured image of the projection pattern even if a finer projection pattern of the projecting device is used, which is Causes errors and defects in spatial measurements. As the generation unit for virtual recording images 102 If a virtual captured image is generated on the basis of the resolution of the imaging device, the fineness of the projection pattern, which can be determined by the imaging device, can be obtained by analyzing the virtual captured image. Thus, the limit of the fineness of the projection pattern, which can be determined with the aid of the imaging device, can be estimated as the upper limit of the resolution of the projecting device. Alternatively, the performance of the imaging device, which corresponds to the resolution of the projecting device, can be considered. Accordingly, since the projecting device and the imaging device included in the spatial measurement device must have corresponding performances, it is preferable to consider the performances of the projecting device and the imaging device by using the simulation results.

The generation unit for virtual recording images 102 generates a virtual captured image that reproduces a captured image output by the imaging device based on the measurement condition information. The generation unit for virtual recording images 102 can, based on the measurement object information and the non-measurement object information included in the measurement condition information, reproduce in a virtual imaging space the arrangement of objects existing in an imaging space and a part in the imaging space shown in the captured virtual image on the basis of the imaging condition information. The generation unit for virtual recording images 102 can further reproduce a shadow cast by projected light in the imaging space by using a reflection model based on the projection condition information, the measurement object information, and the non-measurement object information. The generation unit for virtual recording images 102 can reproduce, with pixel values, the position corresponding to the boundary of the shadow cast by the projected light in a captured image output by the imaging device. It should be noted that an error contained in the position corresponding to the boundary of the shadow lowers the visibility of the light in the captured image. Lower visibility of the light means that a spatial position in the imaging space that is irradiated with the projected light and a detection result of an irradiation position of the projected light in the recorded image do not correspond to one another, or that, in a case in which the projecting device projects a two-dimensional projection pattern, a A boundary of a shadow of light, such as a pattern boundary which is a boundary between a region irradiated with the projected light and a region not irradiated with the projected light, cannot be correctly detected. As a result, when the visibility of the light decreases, an error is caused in the measurement result of the spatial measurement, or the measurement becomes impossible, which lowers the quality of the spatial measurement.

Factors for errors in the position of a boundary of a shadow of light in a captured image may include those caused by the projecting device and those caused by the imaging device. Factors for errors caused by the projecting device may be interreflection of projected light from the projecting device that is reflected by a first surface and then hits a second surface to illuminate the second surface to reduce light blurring, caused by the projecting device being out of focus, or the like. The second surface can be a surface of an object other than the object with the first surface, or it can be part of the same object with the first surface.

Factors for errors caused by the imaging device can include image distortion due to distortion of a lens of the imaging device or the like, random noise that appears irregularly in the image, image blurring that occurs because the imaging device is not in focus is located, or the like count. It should be noted that the phenomena regarded as error factors are not limited to the above and may include any phenomenon that causes a change in the light projection condition or a change in a captured image.

An example of a situation where the interreflection is likely to occur is a case where two or more objects are close to each other. A light beam emitted from the projecting device can be reflected from a surface of a first object in an imaging space, and the reflected light can illuminate a second object different from the first object. Alternatively, light that is reflected from a part X of a first object which is present in an imaging space can illuminate a part Y which differs from the part X of one and the same first object.

In addition to interreflection of projected light from the projecting device, interreflection of ambient light such as room lighting and outside sunlight may occur in a manner similar to that of the projected light. In addition, the number of times a light beam is reflected is not limited to one, and a light beam can be reflected multiple times. However, since light energy is absorbed every time light is reflected, the more the reflection frequency increases, the light energy is likely to become less than the perceptual sensitivity of the imaging device, with the result that the light does not affect a captured image. As a result, the interreflection of light reflected a predetermined number of times or more can be disregarded.

The light intensity resulting from the interreflection can be determined based on the light intensity before the reflection, the reflection characteristics at a reflection point and a propagation path of the light, in addition to a reflection model similar to that used for generating a virtual captured image.

When interreflection occurs, a shadow boundary of light can be observed in addition to a pattern boundary to be observed on an image. At a position where a boundary other than a pattern boundary is observed, the direction of light irradiation cannot be correctly calculated, thereby causing a measurement error in spatial measurement.

Because the generation unit for virtual recording images 102 If a virtual recording image is generated in consideration of the influence of the interreflection of light, a measurement error that can occur in a spatial measurement of a measurement object that can actually cause interreflection is generated in one of the arithmetic unit for spatial measurements 103 output measured value is reproduced. For this reason, the extent and tendency of the occurrence of a measurement error can be obtained before the spatial measurement apparatus is actually provided. If the tendency for a measurement error to occur can be obtained in advance, a method of reducing the influence of interreflection by changing measurement conditions such as relative positions of the projecting device and the imaging device or an arrangement method of a measurement object can be considered. The accuracy of a spatial measurement can thus be improved.

When the projecting device is defocused, a shadow boundary of light is blurred and a pattern boundary becomes indistinct. Therefore, the accuracy of analyzing a projection pattern on a captured image for the purpose of determining the direction of light irradiation may be lowered and the direction of light irradiation may be incorrectly calculated. In a case where an imaging space has a depth of a predetermined value or greater, the projecting apparatus may not be able to focus on the entire imaging space. In such a case, it is necessary to provide a calculation method capable of analyzing a projection pattern even when the projecting device is out of focus. Even if an attempt is made to verify a method of analyzing a projection pattern by using real data, which is data obtained by observing a measurement object by actually operating the projecting device and the imaging device, it is difficult to find a real position a model boundary in the real data. Therefore, it cannot be determined exactly whether or not a verification result is correct. In contrast, the simulation device 10 obtain an actual position of a pattern boundary based on the measurement condition information and generate a virtual captured image that reproduces a blurring of the pattern boundary caused by defocusing of the projecting device. Thus, as a result of an improvement in the method of analyzing a projection pattern, it can be easily determined whether a result of detecting a pattern boundary is improved or not. Algorithms for analyzing a projection pattern, which are necessary for spatial measurements, can therefore be developed more efficiently.

Furthermore, the relationship between the degree of defocusing of the projecting device and the measurement result of a spatial measurement can be simulated, which enables verification of which degree of defocusing gives errors in a measurement result within an allowable range, and an estimation of a required performance such as a range the depth at which the projecting device is in focus. The simulation device 10 consequently produces an advantageous effect in that it is able to facilitate the design of the spatial measurement device.

In addition, the generation unit for virtual recorded images 102 calculate the image distortion as the effect of the distortion of a lens of the imaging device. As an example of a model of image distortion, the relationship between pixels before and after occurrence of distortion on an image can be obtained by using the following formula (1).
[Formula 1] $$\begin{array}{l}{\text{x}}_{\text{u}}={\text{x}}_{\text{d}}+\left({\text{x}}_{\text{d}}-{\text{x}}_{\text{c}}\right)\left({\text{K}}_1{\text{<figure-callout id="2" label="r" filenames="DE112019006855T5\_0005.png,DE112019006855T5\_0013.png" state="{{state}}">r</figure-callout>}}^2+{\text{K}}_2{\text{r}}^{\mathrm{4th}}+\cdots \right)\\ {\text{y}}_{\text{u}}={\text{y}}_{\text{d}}+\left({\text{y}}_{\text{d}}-{\text{y}}_{\text{c}}\right)\left({\text{K}}_1{\text{<figure-callout id="2" label="r" filenames="DE112019006855T5\_0005.png,DE112019006855T5\_0013.png" state="{{state}}">r</figure-callout>}}^2+{\text{K}}_2{\text{r}}^{\mathrm{4th}}+\cdots \right)\end{array}$$

In the formula, (x _u , y _u ) stand for image coordinates in an image without distortion, (x _d , y _d ) for image coordinates in an image with distortion, K for a coefficient indicating a degree of distortion, and r for the distance from the center of the image to a target pixel. The target pixel is a pixel at given coordinates (x _d , y _d ) in an image with distortion. The generation unit for virtual recording images 102 can sequentially select each pixel in the distorted image as the target pixel. A large number of image distortions have been proposed, ranging from simplified models to detailed models. For the simulation device 10 Any model including the formula (1) as a formula for calculating the image distortion can be used.

The generation unit for virtual recording images 102 can reproduce random noise by determining the probability of occurrence and the intensity of the noise at each pixel or in any given region in an image. The generation unit for virtual recording images 102 determines whether to apply noise to a pixel or region by using the occurrence probability, and when it determines that noise is to be applied, it changes the color of the pixel or region based on the specified intensity. As a method of specifying the intensity, a rate of change with respect to the color of the original pixel or region or an integer value can be used. The intensity can be expressed in terms of a fixed rate of change or a fixed integer value, or it can have a specific range of values. Furthermore, the intensity can assume both a positive value, which indicates an increase in the luminance of a pixel, and a negative value, which indicates a decrease in the luminance.

With the help of color information on pixels in the vicinity of a target pixel, the generation unit for virtual recorded images 102 reproduce image blurring caused by defocusing of the imaging device. Any pixel in an image can be selected as a target pixel. Examples of a method of calculating color after image blurring has occurred include a method of using an average color value of a target pixel and surrounding pixels, Gaussian blurring, which is to generate a composite value where a pixel that is closer to a target pixel , has a higher proportion, and the like. In order to enable image blur to be reproduced more correctly, the use of Gaussian blurring is more advantageous than the method of using an average value. In contrast, the method of using an average value is advantageous in that the processing time is shorter than that of the Gaussian blurring method. Furthermore, an image blurring also changes depending on the distance from the imaging device to an object. A captured image of an object that is in focus is less blurred, and a captured image of an object that is not in focus is significantly more blurred. The additional use of information such as a change in the degree of blurring, which is dependent on the distance and the focus distance, offsets the generation unit for virtual recorded images 102 able to reproduce a blur similar to that in an actual captured image. The information may be obtained as the measurement condition information, or it may be obtained by calculation based on other information included in the measurement condition information. It should be noted that examples of the specific method for reproducing error factors by the virtual captured image generation unit 102 Although the second and third embodiments are described, the reproducing method is not limited thereto.

The computing unit for spatial measurements 103 performs the spatial measurement process in response to an input of a virtual captured image. The one by the computing unit for spatial measurements 103 The spatial measurement process performed may be the same as the spatial measurement process performed by the spatial measurement apparatus by using an image captured in a real environment. The process of spatial measurement includes a process of identifying a projection pattern that is emitted onto a measurement object and a distance measurement process. The process of identifying a projection pattern includes obtaining luminance-related information about pixels in the same position from a plurality of captured images, determining whether or not each of the pixels is illuminated by the projecting device, and obtaining combinations thereof. In addition to the foregoing, the process of identifying a projection pattern may alternatively be a method of analyzing the layout of a local projection pattern surrounding a target pixel. If the local projection pattern is designed to be unique throughout the pattern, the position of the part emitted to the vicinity of the target pixel in the entire projection pattern projected by the projecting device can be identified. Each of the methods for pattern identification aims to obtain a vector once from the projecting device to a target pixel.

When the vector is obtained from the projecting device to a target pixel through the process of identifying a projection pattern, the spatial measurement arithmetic unit takes 103 the distance measuring process according to the triangulation principle on the basis of sensor information indicating arrangement conditions such as the positions and postures of the projecting device and the imaging device contained in the measuring condition information. The details of the distance measurement process differ depending on the projection pattern. Since that through the simulation device 10 The projection pattern used can be any projection pattern, the distance measuring process is selected depending on the projection pattern used.

4th is a diagram showing an example of a display screen 20th represented by the in 1 illustrated simulation device 10 is issued. The display screen 20th has a processing result display region 21 , a measurement condition display region 22nd , a measurement condition list display region 23 , a selection region for displayed content 24 , a run button 25th , a save button 26th and a stop button 27 on. A simulation result is displayed in the processing result display region 21 displayed. Individual measurement conditions are displayed in the measurement condition display region 22nd is displayed, and a change in the displayed measurement conditions can be entered therein. A list of stored measurement conditions is displayed in the measurement condition list display region 23 displayed. When a measurement condition is selected from the measurement condition list displayed in the list display region 23 is displayed, the selected measurement condition is displayed in the measurement condition display region 22nd displayed. An operation section for selecting contents displayed in the processing result display region 21 should be displayed is in the selection region for displayed content 24 displayed in the one by the generation unit for virtual captured images 102 reproduced recorded image and measurement data are displayed as options herein.

The run button 25th is an operation section for performing a simulation process by using that in the measurement condition display region 22nd displayed measurement condition. The save button 26th is an operation section for storing the information in the measurement condition display region 22nd displayed measurement condition. The exit button 27 is an operation section for completing the process of simulating spatial measurements.

When the run button 25th is operated, the measurement condition maintenance unit receives 101 those in the measurement condition display region 22nd displayed measurement condition and enters the received measurement condition into the generation unit for virtual recording images 102 a. When the generation unit for virtual recording images 102 a virtual recording image is generated and the generated virtual recording image is transferred to the computing unit for spatial measurements 103 and the computing unit for spatial measurements 103 carries out the process of spatial measurement and transfers the measurement result to the output unit 104 gives the output unit 104 a captured virtual image, a processing result such as a measurement value obtained in the course of a simulation process, input data such as the measurement condition information for the simulation process, and the like to the processing result display region 21 a.

In order to make the output easily visible to the user, the output unit 104 Perform processes such as highlighting a target point, adjusting contrast, and removing noise. The in 4th display screen shown 20th it is an example and the display screen is not based on the example of 4th limited. Using the in 4th displayed display screen 20th enables the user to check the simulation result while the measurement conditions are successively set.

As described above, according to the first embodiment of the present invention, a captured virtual image reproducing a captured image output by the actual imaging device is generated, and the spatial measurement process is performed based on the captured virtual image. This configuration makes it possible to obtain a measurement result of spatial measurement without actually providing the spatial measurement device. Furthermore, the simulation device 10 able to reproduce errors in the position of light contained in a captured image obtained by the actual spatial measurement device. The errors are caused by image distortion due to distortion, random noise, image blurring, interreflection and the like. While the errors are difficult to add to an image after the image is actually captured in a case where the spatial measurement apparatus is actually provided, reproducing the errors in a virtual captured image can improve the accuracy of the simulation result of the spatial measurement . In addition, if the projection pattern and defects can be reproduced in a captured virtual image with high reproducibility, a normal process can be used for the process of spatial measurement.

Second embodiment

5 is a diagram showing a functional configuration of a generation unit for virtual captured images 102 according to a second embodiment of the present invention. The generation unit for virtual recording images 102 contains a generation unit for optically reproduced images 301 and an image quality deterioration processing unit 302 . Although not shown, a device which incorporates the in 5 Generation unit shown for virtual recording images 102 and has a configuration similar to that of the in 1 illustrated simulation device 10 resembles, as a simulation device 12th referred to according to the second embodiment. As the simulation device 12th has a configuration other than the generation unit for virtual captured images 102 that of the in 1 is similar to the first embodiment illustrated, a detailed description thereof is not repeated here. In the following, components similar to those in the first embodiment will be described using the method shown in FIG 1 and mainly differences from the first embodiment are described.

The generation unit for optically reproduced images 301 performs an optical simulation based on the measurement condition information and generates an optically reproduced image that reproduces a captured image. The image quality deterioration processing unit 302 performs an image quality deterioration process depending on error factors on the optically reproduced image. The generation unit for virtual recording images 102 provides the image subjected to the impairment process as a virtual recorded image.

6th Fig. 13 is a flow chart showing the operation of the simulation device 12th represents the in 5 Generation unit shown for virtual recording images 102 contains. The measurement condition maintenance unit 101 obtains measurement condition information (step S201). The measurement condition maintenance unit 101 inputs the measurement condition information obtained into the generation unit for virtual recording images 102 a.

The generation unit for optically reproduced images 301 the generation unit for virtual recording images 102 generates an optically reproduced image based on the measurement condition information (step S202). The generation unit for optically reproduced images 301 inputs the generated optically reproduced image into the image quality deterioration processing unit 302 a.

The image quality deterioration processing unit 302 performs the image quality deterioration process on the optically reproduced image (step S203). The image quality deterioration processing unit 302 sends an image subjected to the image quality impairment process as a virtual recording image to the computing unit for spatial measurements 103 a.

The computing unit for spatial measurements 103 performs the spatial measurement process by using the captured virtual image to obtain a measurement result (step S204). The computing unit for spatial measurements 103 sends the measurement result to the output unit 104 a. The output unit 104 outputs a simulation result including the measurement result (step S205).

The generation unit for optically reproduced images 301 calculates an imaging position in an imaging space that corresponds to each pixel in an image, and determines whether the calculated imaging position is illuminated with projected light becomes or not, that is, whether or not projected light from the projecting device reaches the imaging position. The generation unit for optically reproduced images 301 first computes a vector V _cam passing through each pixel from the optical center O _{cam of} the imaging device based on sensor information and information indicating the arrangement and characteristics of objects present in the imaging space. The generation unit for optically reproduced images 301 detects a point P _obj on a surface of an object that first intersects the vector Vcam. This calculation makes it possible to obtain an object that is imaged at each pixel.

The generation unit then determines for optically reproduced images 301 whether or not the point P _obj on the surface of the object is irradiated by the projecting device. The generation unit for optically reproduced images 301 first computes a vector V _{proj which is guided} from the optical center O _{proj of} the projecting device to the point P _obj . The generation unit for optically reproduced images 301 determines whether the vector V _{proj is included} in an area irradiated with a projection pattern by the projecting device using the sensor information and pattern information indicative of the projection pattern of the projecting device. When the vector V _{proj is included} in the area, it can be determined that the point P _obj on the surface of the object is irradiated by the projecting device.

In addition to the above calculation result, the optically reproduced image generation unit can 301 determine the color of each pixel using the reflection characteristics and the information indicating the state of the ambient light contained in the measurement condition information. Typical reflection models used to determine the color are Lambertian reflection in the case of diffuse reflection and the Phong reflection model for directional reflection. That of the generation unit for optically reproduced images 301 However, the reflection model used is not limited to these models, and any reflection model can be used.

It should be noted that, in generating an optically reproduced image, since an image is a series of individual pixels, a boundary of an object or a boundary of a projection pattern of the projecting device may be included in an area imaged on a pixel. In this case, a color from a mixture of the colors of two or more objects forming the border would be more natural than the color of the pixel. With the above-mentioned vector Vcam, however, only one interface can be recorded in the imaging space. For this reason, only the color of the object intersecting with the vector V _cam among the objects forming the boundary is determined as the color of the pixel. As a result, an image in which boundaries of objects and a projection pattern appear unnatural may be generated. This is a kind of phenomenon caused by so-called quantization errors, a problem that is common in the field of image processing.

In order to solve the problem of phenomena caused by quantization errors, there is a method of generating a virtual captured image by generating an optically reproduced image with a resolution higher than that of the virtual captured image and applying a process of reducing the image to a Image size using the image quality degradation process. For example, the generation unit generates for optically reproduced images 301 an optically reproduced image with a resolution that corresponds to four times the resolution of a virtual recorded image that is ultimately to be output. In this case, the color of each pixel in the virtual captured image can be determined from color information for four pixels that are closest to the pixel in the optically reproduced image. As a result, a virtual recording image with a natural appearance in the area of the boundaries of objects and a projection pattern can be generated.

The generation unit for optically reproduced images 301 generates an optically reproduced image based on information about interreflection and defocusing of the projecting device, which are error factors caused by the projecting device, as an example of error factors. Since these affect the projected state of a projection pattern when defocus occurs, for example, the generation unit effects optically reproduced images 301 that shadow boundaries of light, including pattern boundaries in the optically reproduced image, become blurred. The generation unit for optically reproduced images 301 can reproduce interreflection by causing the luminance of a pixel having a point of incidence where light reflected from a first surface is incident on a second surface to be higher than the luminance in the case where the influence of interreflection is not taken into account will. Furthermore, the generation unit for optically reproduced images 301 reproduce a blurring of a shadow boundary of the light by adjusting the luminance of a pixel corresponding to the shadow boundary of the light. For a specific method of reproducing interreflection and a specific method for reproducing blurring of a shadow boundary of light caused by defocusing the projecting device, for example, methods described in the third embodiment can be used.

The image quality deterioration processing unit 302 affects the image quality of an optically reproduced image based on information on image distortion, random noise and defocusing of the imaging device, which are error factors caused by the imaging device, for example. For example, the image quality deterioration processing unit may 302 affect the image quality of an optically reproduced image by performing a filtering process by smoothing changes in luminance in the image to cause the outlines of an object, the shadow boundaries of light and the like to be blurred, or by increasing the luminance of pixels randomly selected positions changes.

With the simulation device 12th According to the second embodiment of the present invention, an optically reproduced image, which is an image in which optical effects including shadows reproduced by objects blocking light, can be obtained through optical simulation. In addition, the image quality deterioration process is performed on the optically reproduced image, which enables an actually captured image to be reproduced with high accuracy.

Furthermore, the process of reproducing optical phenomena and the process of reproducing an impairment of the image quality are explicitly separated from one another, which facilitates the analysis of respective error factors in spatial measurements independently of one another. In a case where an actual spatial measurement apparatus is used to verify the measurement errors of spatial measurements, it is difficult to distinguish the proportions of the influence of the respective error factors. With the use of the simulation device 12th However, it is possible to differentiate between measurement errors that are caused by the projecting device and measurement errors that are caused by the imaging device in order to verify the measurement errors. Thus, the configuration of the spatial measurement device and the performances of devices such as the projecting device and imaging device can be properly viewed. In addition, in order to perform verification using an actual spatial measurement device in a real environment, devices and measurement objects associated with various measurement conditions must be provided, which requires a lot of cost and time. With the simulation device 12th For example, the consideration can be carried out by simulation, which brings about advantageous effects, which are that an evaluation of a device for spatial measurements with more measurement conditions is enabled and product assurance is promoted.

Third embodiment.

7th Fig. 16 is a diagram showing a functional configuration of a simulation device 13th according to a third embodiment of the present invention. The simulation device 13th includes the measurement condition maintenance unit 101 , the generation unit for virtual recording images 102 , the computing unit for spatial measurements 103 , the output unit 104 and a failure information obtaining unit 105 . The following are mainly differences from the simulation device 12th described.

The simulation device 13th includes the failure information preservation unit 105 in addition to the design of the simulation device 12th . The error information preservation unit 105 receives, from the outside, error information indicating error factors which are manifested in a virtual recorded image. The error information includes at least one of the intensity and adding order of errors, which depend on the kinds of the error factors. The error information preservation unit 105 passes the received error information into the generation unit for virtual recorded images 102 a.

8th represents a functional configuration of the generation unit for virtual recording images 102 represent that in 7th is shown. In the simulation device 13th contains the generation unit for virtual recording images 102 the generation unit for optically reproduced images 301 and the image quality deterioration processing unit 302 and an error factor determination unit 303 .

The error factor determination unit 303 determines a processing condition of an error factor adding process including at least one of the intensity and the addition order in the error factor adding process based on the error information. The error factor adding process includes generating an optically reproduced image and performing the image quality deterioration process on the optically reproduced image. The error factor determination unit 303 gives the certain processing condition to the generation unit for optically reproduced images 301 a.

The generation unit for optically reproduced images 301 generates an optically reproduced image based on the measurement condition information, the error information, and the processing condition obtained from the error factor determination unit 303 was entered. The image quality deterioration processing unit 302 performs the image quality deterioration process on the optically reproduced image based on the measurement condition information and the error information.

9 FIG. 13 is a flow chart illustrating the operation of the FIG 7th illustrated simulation device 13th represents. The measurement condition maintenance unit 101 obtains the measurement condition information (step S301). The measurement condition maintenance unit 101 inputs the measurement condition information obtained into the generation unit for virtual recording images 102 a.

The error information preservation unit 105 obtains the error information (step S302). The error information preservation unit 105 passes the received error information into the generation unit for virtual recorded images 102 a. It should be noted that the process in step S301 and the process in step S302 can be performed in parallel with each other.

The error factor determination unit 303 the generation unit for virtual recording images 102 determines the processing condition of the error factor adding process including at least one of the intensity and adding order of error factors (step S303). The error factor determination unit 303 gives the certain processing condition to the generation unit for optically reproduced images 301 a.

The generation unit for optically reproduced images 301 generates an optically reproduced image through optical simulation based on the processing condition and the measurement condition information (step S304). The generation unit for optically reproduced images 301 inputs the generated optically reproduced image into the image quality deterioration processing unit 302 a.

The image quality deterioration processing unit 302 performs the image quality deterioration process on the optically reproduced image based on the measurement condition information and the error information (step S305). The image quality deterioration processing unit 302 sends an image subjected to the image quality impairment process as a virtual recording image to the computing unit for spatial measurements 103 a.

The computing unit for spatial measurements 103 performs the spatial measurement process by using the captured virtual image to obtain a measurement result (step S306). The computing unit for spatial measurements 103 sends the measurement result to the output unit 104 a. The output unit 104 outputs a simulation result including the measurement result (step S307).

10 Fig. 3 is a flow chart showing details of the in 9 illustrated step S304. The generation unit for optically reproduced images 301 first generates an optically reproduced image that does not contain any error factors (step S401). As described above, the generation unit distinguishes between optically reproduced images 301 between projected regions, which are regions that are directly illuminated with projected light, and unprojected regions, i.e. regions that are not illuminated with projected light, and provides the projected regions in colors that are brighter than the colors of the unprojected regions are.

Subsequently, the generation unit receives for optically reproduced images 301 the adding order of error factors from the error information (step S402). The generation unit for optically reproduced images 301 obtains the kinds of the error factors and the intensities depending on the kinds from the error information (step S403). The generation unit then determines for optically reproduced images 301 whether or not to add an error to the optically reproduced image according to the adding order obtained in step S402. The generation unit for optically reproduced images 301 first determines whether or not to add interreflection to the optically reproduced image based on the error information (step S404).

When interreflection is to be added (step S404: Yes), the optically reproduced image generation unit adds 301 the interreflection is added to the optically reproduced image (step S405). More specifically, when interreflection occurs, light reflected from a first surface of an object is incident on a second surface. In this case, the generation unit sets for optically reproduced images 301 the luminance of a pixel which has the point of impact at which the reflected light strikes the second surface as the luminance increased by the interreflection. The generation unit for optically reproduced images 301 determines the amount of increase in luminance based on positions and postures of the object having the first face and the object having the second face and the surface characteristics of the first face and the second face. The luminance each Pixels in the optically reproduced image become a second luminance obtained by adding the amount of luminance increase due to interreflection to a first luminance of the pixel at which the effect of interreflection is not considered.

It should be noted that the generation unit for optically reproduced images 301 in the process of step S405, the luminance increase due to interreflection does not necessarily have to be calculated for each light reflection. For example, since the amount of luminance increase may be less than the resolution of the image on a surface with a low specular light reflection, it can be assumed that the luminance on a surface with a specular reflection at or below a threshold value is not increased by interreflection.

Upon completion of the process of adding interreflection in step S405, the optically reproduced image generation unit determines 301 whether or not the error adding process is completed (step S406). When the error adding process is completed (step S406: Yes), the optically reproduced image generation unit closes 301 executes the error adding process and outputs the optically reproduced image to the image quality deterioration processing unit 302 a. If the error adding process is not completed (step S406: No), the optically reproduced image generation unit returns 301 back to the process in step S403.

If no interreflection is to be added (step S404: No), the optically reproduced image generation unit determines 301 whether or not to add a light blur to the projecting device (step S407).

When a light blur from the projecting device is to be added (step S407: Yes), the optically reproduced image generation unit adds 301 light blur of the projecting device is added to the optically reproduced image (step S408). A blurring at a shadow boundary of the light can be expressed by the luminance of pixels. Specifically, the generation unit distinguishes between optically reproduced images 301 between a first region, which is a region in which light is projected in the optically reproduced image when it is assumed that there is no defocus, and a second region, which is a region in which light is not projected, when it is assumed that there is also no defocusing of the projecting device, which is done on the basis of the measurement condition information. The generation unit for optically reproduced images 301 identifies pixels within a predetermined distance from the boundary between the first region and the second region based on the result of the discrimination. The highest luminance among these pixels is the luminance in the region where light is projected, and the lowest luminance is the luminance in the region where light is not projected. The luminance of pixels in the vicinity of a boundary can be determined based on the distance between a target pixel and the boundary. The luminance of pixels can be determined closer to the maximum luminance because a target pixel is in closer proximity to the first region and can be determined closer to the minimum luminance because a target pixel is in closer proximity to the second region. As a result of thus changing the luminance based on the shortest distance between a target pixel and the first region, the defocused state of the projecting device can be reproduced.

Upon completion of the process in step S408, the optically reproduced image generation unit goes 301 to step S406. When there is no light blur to be added to the projecting device (step S407: No), the optically reproduced image generation unit determines 301 whether or not to add ambient light to the optically reproduced image (step S409).

When ambient light is to be added (step S409: Yes), the optically reproduced image generation unit adds 301 ambient light is added to the optically reproduced image (step S410). The ambient light is expressed by at least one of luminance and color of pixels. Upon completion of the process in step S410, the optically reproduced image generation unit goes 301 to step S406. If there is no ambient light to be added (step S409: No), the optically reproduced image generation unit closes 301 the bug adding process.

11 Fig. 3 is a flow chart showing details of the in 9 illustrated step S305. The image quality deterioration processing unit 302 first obtains the adding order of error factors based on the error information (step S501). The image quality deterioration processing unit 302 further obtains the types of error factors and the intensities of errors based on the error information (step S502). The image quality deterioration processing unit then determines 302 whether or not to add an error according to the adding order obtained in step S501.

The image quality deterioration processing unit 302 first determines whether a Image distortion is to be added or not (step S503). When it is determined that image distortion is to be added (step S503: Yes), the image quality deterioration processing unit adds 302 image distortion is added to the optically reproduced image (step S504). After adding the image distortion, the image quality deterioration processing unit determines 302 whether or not to complete the image quality deterioration process (step S505). When it is determined that the image quality deterioration process is to be completed (step S505: Yes), the image quality deterioration processing unit closes 302 the process. When it is determined that the image quality deterioration process is not to be completed (step S505: No), the image quality deterioration processing unit returns 302 back to the process in step S502.

If it is determined that the image distortion is not to be added (step S503: No), the image quality deterioration processing unit determines 302 then whether or not to add random noise (step S506). When it is determined that random noise is to be added (step S506: Yes), the image quality deterioration processing unit adds 302 adds random noise to the optically reproduced image (step S507). Upon completion of the process in step S507, the image quality deterioration processing unit goes 302 to the process in step S505.

When it is determined that no random noise is to be added (step S506: No), the image quality deterioration processing unit determines 302 whether or not to add image blur (step S508). When it is determined that no image blur is to be added (step S508: No), the image quality deterioration processing unit closes 302 the process. If it is determined that image blur is to be added (step S508: Yes), the image quality deterioration processing unit adds 302 add image blur to the optically reproduced image (step S509). Upon completion of the process in step S509, the image quality deterioration processing unit goes 302 to the process in step S505.

It should be noted that the image quality deterioration processing unit 302 can add only one kind of error factor to the optically reproduced image, or can add the same error factor to the optically reproduced image several times. For example, if the addition of random noise and the addition of image blur are alternately made several times, an effect such as color irregularity can be produced in the appearance of a corresponding object. As described above, various picture quality effects can be reproduced by combinations of the order of addition and intensities of error factors.

12th is a diagram showing an example of a display screen 30th represented by the in 7th illustrated simulation device 13th is issued. The display screen 30th assigns the processing result display region 21 , the measurement condition display region 22nd , the measurement condition list display region 23 , the selection region for selected content 24 who have favourited the Run button 25th who have favourited Save button 26th who have favourited the quit button 27 and an error factor setting region 28 on.

The display screen 30th indicates the error factor setting region 28 in addition to the components of the display screen described in the first embodiment 20th on. The following is the description of parts similar to those on the display screen 20th are similar, not repeated; It mainly describes differences from the display screen 20th .

The error factor setting region 28 is a region for setting intensities of defects and the adding order of defects according to the kinds of the defect factors. A user can see the intensities and the adding order of errors in the error factor setting region 28 input. When the run button 25th is operated, the error information obtaining unit 105 the error information obtained in the error factor setting region 28 are displayed.

As described above, it is in the simulation device 13th according to the third embodiment of the present invention with the failure information obtaining unit 105 possible to run a simulation that reflects the error information specified by the user. This design enables the user to evaluate spatial measurements made under desired operating conditions. Furthermore, since the kinds of error factors, the addition order in which errors are added to an optically reproduced image, and the intensities of the errors can be selected, an advantageous effect of increasing variations in reproducible image quality is brought about. Since tests with randomly changed operating conditions and usage environments can also be carried out, tests with combinations of conditions that were not expected in advance by the designer can also be carried out, which makes the tests comprehensive and therefore leads to an effect of increasing the Reliability of the device for spatial measurements can be expected.

Fourth embodiment. 13th is a diagram showing a functional configuration of a generation unit for optically reproduced images 301 according to a fourth embodiment of the present invention. The generation unit for optically reproduced images 301 contains a generation unit for sensor position data 401 , a map generation unit 402 and an image combining unit 403 .

It should be noted that a device, which is not shown and which is shown in FIG 13th Generation unit shown for optically reproduced images 301 and has a configuration similar to that of the in 7th illustrated simulation device 13th resembles, as a simulation device 14th is designated according to the fourth embodiment. The simulation device 14th has an interpretation that is similar to that of in 7th illustrated simulation device 13th is similar, but with respect to the configuration of the generation unit for optically reproduced images 301 the generation unit for virtual recording images 102 from the simulation device 13th differs. The following is the description of components similar to those of the simulation device 13th are similar, not repeated; The main differences to the simulation device are described 13th .

The generation unit for sensor position data 401 generates sensor point of view data including a first image of an imaging space from the point of view of the imaging device and distance data from each of the imaging device and the projecting device to objects in the imaging space. The generation unit for sensor position data 401 sends the generated sensor position data to the map generation unit 402 a.

14th is a schematic showing a detailed functional configuration of the in 13th shown generation unit for sensor position data 401 represents. The first by the generation unit for sensor position data 401 The generated image includes a light image in which the entire image space is displayed with such a brightness as in the case that the image space is irradiated with projected light, and a dark image in which the image space is not irradiated with projected light. The generation unit for sensor position data 401 contains a generation unit for bright images 501 that generates a bright image and a generation unit for dark images 502 , which generates a dark image, and a generation unit for distance data 503 that generates distance data.

The generation unit for sensor position data 401 outputs sensor standpoint data including a first image included by the bright image generation unit 501 bright image generated and one generated by the generation unit for dark images 502 generated dark image, and by the generation unit for distance data 503 include generated distance data.

The description resumes 13th Relation. The output of the sensor standpoint data by the sensor standpoint data generation unit 401 becomes in the card generation unit 402 entered. The card generation unit 402 generates an irradiation map that is a map showing a first region that is irradiated with light from the projecting device and a second region that is not irradiated with light from the projecting device, based on various numerical values from each other based on the measurement condition information. For example, the first region can be represented by “1” and the second region by “0” on the exposure map. The card generation unit 402 puts the generated exposure map into the image combination unit 403 a.

15th is a schematic showing a detailed functional configuration of the in 13th map generation unit shown 402 represents. The card generation unit 402 includes an exposure region calculation unit 601 , an exposure map generation unit 602 , a light blur reproduction unit 603 , a reflection region calculation unit 604 and a reflection map generation unit 605 .

The exposure region calculation unit 601 calculates a region that can be irradiated with light from the projecting device based on the measurement condition information. The exposure region calculation unit 601 inputs information indicating the calculated region into the exposure map generation unit 602 a.

The exposure map generation unit 602 generates an exposure map that is a map showing a first region that is illuminated with light from the projecting device and a second region that is not illuminated with light from the projecting device by using different ones Numerical values by using the information from the exposure region calculating unit 601 and the measurement condition information. The exposure map generation unit 602 sends the generated exposure map to the light blur reproduction unit 603 a.

The reflection region calculation unit 604 provides a link between coordinates on the first image and the spatial position of a point irradiated with a reflected light beam, which is light emitted by the projecting device, which is reflected once by an object in the imaging space and whose intensity and direction are thus changed is done by using the measurement condition information and the sensor point of view data to calculate a reflection region that is a region irradiated with the reflected light beam in the first image. The reflection region calculation unit 604 inputs information indicating the calculated reflection region into the reflection map generation unit 605 a. The reflection map generation unit 605 generates a reflection map which is a map showing a reflection region and a region which is not irradiated with reflected light by using different numerical values from each other for each projection pattern of the projecting device based on the information which the reflection region and the Specify measurement condition information. For example, on the reflection map, the reflection region can be represented by “1” and the region that is not illuminated with reflected light by “0”. The reflection map generation unit 605 puts the generated reflection map into the light blur reproduction unit 603 a.

The light blur reproduction unit 603 reproduces an effect of blurring light on an exposure map. Specifically, the light blur reproduction unit represents 603 a value of the exposure map to a real number from 0 to 1 depending on the degree of uncertainty of the shadow boundaries of the light. The light blur reproduction unit 603 also reproduces an effect of blurring light on a reflection map in a manner similar to the exposure map. Specifically, the light blur reproduction unit represents 603 a value of the exposure map to a real number from 0 to 1 depending on the degree of uncertainty of the shadow boundaries of the light. The light blur reproduction unit 603 outputs the exposure map and the reflection map subjected to the setting of the values.

The image combining unit 403 combines the light image and the dark image contained in the first image based on the information on the exposure map and the reflection map to generate an optically reproduced image. Concretely, the image combining unit combines 403 the light image and the dark image by adjusting the luminance of each pixel in the optically reproduced image to the luminance obtained from the pixel at the same position in the light image or the dark image based on the values in the exposure map. Furthermore, the image combination unit 403 in a case where the values in the exposure map and the reflection map are determined by the light blur reproduction unit 603 adjusted, weight the luminance of each pixel based on the adjusted values.

The number of virtual recorded images corresponds to the number of projection patterns. Thus, there is a problem that as the number of projection patterns increases, the processing time for generating virtual captured images becomes longer. An implementation is thus effective in which processes which are used for illuminating with different projection patterns in the generation of virtual recorded images only have to be carried out once. Examples of the processes in use include generating a light image and a dark image of an imaging space from the point of view of the imaging device, and calculating distance data from the imaging device and the projecting device to objects present in the imaging space.

The calculation results of a region irradiated with projected light from the projecting device differ in different projection patterns. The calculation of regions that are located behind objects and are therefore in the shade because the projected light is blocked, on the other hand, is constant regardless of the projection patterns and can thus be extracted as a common process.

In a case where the projecting device is defocused, a pattern projected onto a scene is fuzzy, which can lower the accuracy of pattern recognition, which can lower the accuracy of spatial measurements. The blurring of the pattern light does not occur during imaging by the imaging device, but does occur during the pattern projection by the projecting device. As a result, the method of adding a blur to a captured virtual image cannot reproduce the phenomenon.

For this reason, an irradiation map is generated which indicates a region irradiated with projected light in a virtual recorded image. The exposure map can be defined as a two-dimensional arrangement that has the same number of elements as the virtual recorded image, each element having a value of a real number from 0 to 1. A region irradiated with projected light can be expressed by 1, a region to which the projected light does not reach can be expressed by 0, and a region in which blurring of projected light occurs and is thereby irradiated with an intensity which lower than normal intensity can be expressed by a value greater than 0 and less than 1. This can be expressed as in the following formula (2).
[Formula 2] $${\text{I.}}_{\text{i},\text{j}}={\text{P.}}_{\text{i},\text{j}}{\text{B.}}_{\text{i},\text{j}}+\left(1-{\text{P.}}_{\text{i},\text{j}}\right){\text{S.}}_{\text{i},\text{j}}$$

In the formula I _{i, j stands} for the luminance at the image coordinates (i, j) in the virtual recorded image, P _{i, j stands} for the value at the coordinates (i, j) in the exposure map, B _{i, j stands} for is the luminance at the image coordinates (i, j) in the light image and S _{i, j stands} for the luminance at the image coordinates (i, j) in the dark image.

To generate the irradiation map, a state is first assumed in which the projecting device is focused, that is, a state in which the projected light is not blurred, and a region irradiated with a projection pattern is calculated. At this point, the value of each element on the exposure map is 0 or 1. A smoothing filter such as Gaussian blur is applied to the exposure map to reproduce the blurring of projected light. As a result of the application of Gaussian blurring, changes in the values in the irradiation map in the area of the boundary between a region which is irradiated with pattern light and a region which is not irradiated with pattern light are smoothed. The smoothing of the changes in the values at the boundary brings about an effect that the boundary of the projected light is blurred, and therefore the blurring of the projected light can be reproduced.

16 Fig. 13 is a flow chart showing the operation of the simulation device 14th according to the fourth embodiment of the present invention. The generation unit for sensor position data 401 generates a light image and a dark image using the light image generation unit 501 and the generation unit for dark images 502 (Sensor position data S601). The generation unit for sensor position data 401 generates distance data at each station using the distance data generation unit 503 in parallel with step S601 (step S602). The generation unit for sensor position data 401 inputs the sensor position data including the generated light image, dark image and distance data into the map generation unit 402 a.

The card generation unit 402 calculates an irradiation region using the irradiation region calculating unit 601 (Step S603). The map generation unit then generates 402 an exposure map using the exposure map generation unit 602 (Step S604).

The card generation unit 402 obtains the adding order of error factors (step S605). The card generation unit then receives 402 the types and the intensity of the error factors (step S606). The card generation unit 402 determines whether or not to add interreflection (step S607). If interreflection is to be added (step S607: Yes), the map generation unit causes 402 the reflection region calculation unit 604 to calculate a reflection region (step S608), and it causes the reflection map generation unit 605 to generate a reflection map (step S609).

If it is determined that no interreflection is to be added (step S607: No), the processes in steps S608 and S609 are omitted. The card generation unit then determines 402 whether or not to add light blur (step S610). If it determines that light blur is to be added (step S610: Yes), the light blur reproduction unit reproduces 603 the card generation unit 402 a light blur on the exposure map (step S611).

If it is determined that there is no light blur to be added (step S610: No), the process in step S611 is omitted. The card generation unit then determines 402 whether or not to add ambient light (step S612). If it is determined that ambient light is to be added (step S612: Yes), the exposure map generation unit adds 602 the card generation unit 402 Add ambient light on the exposure map (step S613). When it is determined that no ambient light is to be added (step S612: No), the process in step S613 is omitted.

The card generation unit 402 determines whether or not to complete the error adding process (step S614). When it is determined that the error adding process is not to be completed (step S614: No), the process returns to step S606. When it is determined that the error adding process is to be completed (step S614: Yes), the image combining unit takes 403 perform an image combining process (step S615).

As described above, the simulation device 14th According to the fourth embodiment of the present invention, generates a first image that is an image in which optical phenomena are individually reproduced, an exposure map and a reflection map, which simplify the processes for generating an optically reproduced image and the data structure using a plurality of projection patterns can. Of Further, the effect of blurring of shadow boundaries of light caused by defocusing is expressed by using an illumination map, and the effect of interreflection is expressed by using a reflection map, which brings about an advantageous effect that it enables reproduction of a captured image , in which complicated optical phenomena such as defocusing of the projecting device and interreflection are combined by simple calculation processes.

In a case where an actual spatial measurement apparatus is provided for evaluation, results can only be obtained in a state where all of the optical phenomena are mixed. In contrast, according to the simulation device 14th each of a variety of kinds of optical phenomena can be individually reproduced. In this way, factors that cause measurement errors in spatial measurements can be determined in more detail. This configuration brings about an advantageous effect that the design and performance evaluation of a spatial measurement apparatus is facilitated.

Fifth embodiment.

A simulation device 15th , which is not shown, according to a fifth embodiment has a configuration similar to that of the FIGS 7th illustrated simulation device 13th is similar. The simulation device 15th differs from the simulation device in a display screen that is output 13th . The following is the description of parts similar to those of the simulation device 13th are similar, not repeated; The main differences to the simulation device are described 13th .

17th is a diagram showing an example of a display screen 40 represents that by the simulation device 15th according to the fifth embodiment of the present invention. The simulation device 15th can output a simulation result that further includes at least one of at least part of the measurement condition information and a virtual captured image.

The display screen 40 can be a customization aspect display region 41 , a save button 42 , a quit button 43 and a measurement result display region 44 include. At the adjustment aspect display region 41 it is a region that displays a list of fixed values for an aspect to be adjusted. The measurement result display region 44 is a region in which measured values are displayed, each associated with one of the specified values displayed in the list, in relation to which the aspect of the measurement conditions to be adjusted is specified. The save button 42 is an operation section for performing an operation of storing a measurement result, and the exit button 43 is an operation section for performing an operation of completing a process.

On the display screen 40 For example, the set values of an aspect of the measurement conditions to be adjusted, measurement values that are processing results, and virtual captured images, exposure maps and the like obtained as intermediate results during the process are displayed side by side. Furthermore, results relating to differences from a processing result in each of the set values can also be displayed, and processing results having large differences can be highlighted.

As described above, the simulation device 15th According to the fifth embodiment of the present invention, the measurement condition information, the captured virtual image and the like are also output in addition to the measurement value, which enables the user to check the processes by which a spatial measurement result is obtained under each measurement condition. In this case, the processing results and the intermediate results associated with a plurality of measurement conditions are displayed side by side, which enables comparison between specified values of measurement conditions for assessment. Thus, an advantageous effect is brought about which is to facilitate the assessment of the arrangement and the performances of the projecting device and the imaging device.

Sixth embodiment.

18th Fig. 16 is a diagram showing a functional configuration of a simulation device 16 according to a sixth embodiment of the present invention. The simulation device 16 includes the measurement condition maintenance unit 101 , the generation unit for virtual recording images 102 , the computing unit for spatial measurements 103 , the output unit 104 , the failure information preservation unit 105 , an evaluation reference data generation unit 106 and a measurement evaluation unit 107 .

The simulation device 16 contains the evaluation reference data generation unit 106 and the measurement evaluation unit 107 in addition to the configuration of the simulation device 13th . The following is the detailed description of the configuration that is the same as that of the simulation device 13th is similar, not repeated; The main differences to the simulation device are described 13th .

The evaluation reference data generation unit 106 generates evaluation reference data, which is a reference for evaluating a simulation result, by using the measurement condition information. The evaluation reference data generation unit 106 inputs the generated evaluation reference data to the measurement evaluation unit 107 a. The measurement evaluation unit 107 evaluates a simulation result based on the evaluation reference data and receives a simulation evaluation. The output unit 104 also outputs a simulation evaluation result in addition to a simulation result.

19th FIG. 13 is a flow chart illustrating the operation of the FIG 18th illustrated simulation device 16 represents. The in 19th In the operation shown, steps S301 to S307 are similar to those in FIG 9 hence their detailed explanation is not repeated. In the present embodiment, the evaluation reference data generation unit generates 106 after completing the spatial measurement process, the evaluation reference data (step S701). The measurement evaluation unit then takes 107 a measurement evaluation process of comparing the measurement data obtained by simulation with the evaluation reference data to calculate a quantitative evaluation value (step S702). The output unit 104 outputs a simulation evaluation (step S703).

The evaluation reference data may be: the distance data from the imaging device included in the sensor position data, a processing result of spatial measurement when a captured virtual image is input before error factors are added, data such as an exposure map obtained during the process, or actual measurement data obtained by actually measuring a measurement object.

The simulation evaluation can theoretically be a ratio of a faulty region in which no measurement data can be obtained to a measurable region, i.e. a region in which measurement data can be obtained, to measurement errors that are obtained when compared with the evaluation reference data , or something like that. For example, the measurable region may be a region in which there is no defect in measurement data obtained when a virtual captured image is input before error factors are added. Furthermore, statistical data such as an average value, a variance, a standard deviation and a maximum value obtained by comparison between the evaluation reference data and a simulation result can be used as evaluation indexes.

In a case where actual measurement data are used as evaluation reference data, it is also useful to provide feedback in order to provide a better simulation evaluation, i.e. to reduce the difference between the actual measurement data and the simulation result.

One method for feedback may be to generate a variety of different simulation results and obtain a fixed value with which the best simulation score is obtained, for example. In order to obtain a variety of simulation results, there are a method of changing values of the surface characteristics of a measurement object under a measurement condition and a method of changing the intensities of error factors, for example.

Parameters such as the surface characteristics of a measurement object and the intensities of error factors when the best simulation evaluation is obtained can be defined as the optimal simulation setting. The optimal simulation setting is provided to the user, which enables more correct verification tests to be performed.

20th is a diagram showing an example of a display screen 50 the in 18th illustrated simulation device 16 represents. The display screen 50 includes an evaluation reference data display region 51 in addition to the components of the display screen 40 .

On the display screen 50 the evaluation reference data is displayed side by side with a plurality of specified values of an aspect of measurement conditions to be adjusted. Furthermore, intermediate results obtained by simulating the process of spatial measurement can also be displayed using a reference value. The display screen 50 can also display the simulation rating.

As described above, the simulation device 16 According to the sixth embodiment of the present invention, the evaluation reference data can be obtained in addition to the simulation result. The evaluation reference data are used as a reference for evaluating a simulation result, and they are actual measurement data or the like. The simulation device 16 can the Facilitate consideration of simulation results by displaying the measured value contained in the simulation result and the evaluation reference data side by side.

Furthermore, the simulation device 16 able to obtain a simulation evaluation that evaluates a simulation result by using the evaluation reference data. This configuration enables measurement errors and defects to be determined quantitatively. This facilitates operations such as examining the performances of an image capturing device and the projecting device and determining whether or not a spatial measurement is appropriate for each measurement object. Further, in a case where a performance evaluation method is defined by a standard, the simulation evaluation is obtained by using the performance evaluation method that facilitates the determination of whether or not spatial measurement is appropriate.

Seventh embodiment.

21 Fig. 16 is a diagram showing a functional configuration of a simulation device 17th according to a seventh embodiment of the present invention. The simulation device 17th includes an object recognition processing unit 108 and a recognition evaluation unit 109 in addition to the design of the simulation device 16 according to the sixth embodiment. The following are mainly differences from the simulation device 16 described.

If one by the computing unit for spatial measurements 103 output simulation result and that by the measurement condition maintenance unit 101 outputted measurement condition information is inputted to the object recognition processing unit 108 a recognition result containing at least one of positions and postures of objects present in an imaging space and a gripping position which is a position at which an object can be gripped. The object recognition processing unit 108 inputs the recognition result into the recognition evaluation unit 109 a.

The recognition evaluation unit 109 evaluates the recognition result based on the measurement condition information. Specifically, the recognition evaluation unit receives 109 a recognition evaluation result including at least one of the accuracy of the estimation of positions and postures included in the recognition result and the accuracy of the estimation of the grasping position. The recognition evaluation unit 109 outputs the object recognition result and the recognition evaluation result to the output unit 104 a. The output unit 104 outputs the recognition result and the recognition evaluation result in addition to the simulation result and the simulation evaluation.

22nd FIG. 13 is a flow chart illustrating the operation of the FIG 21 illustrated simulation device 17th represents. Since the operations in steps S301 to S307 and steps S701 to S703 are similar to those in 19th are similar, their explanation is not repeated here. The following are mainly differences too 19th described.

After completing the spatial measurement process illustrated in step S306, the object recognition processing unit takes 108 perform an object recognition process to obtain a recognition result (step S801). Upon receiving a recognition result, the recognition evaluation unit takes 109 a recognition evaluation process for evaluating the recognition result (step S802). Upon receipt of a recognition evaluation result, the output unit gives 104 the recognition evaluation result (step S803). It should be noted that, in step S803, the recognition result can also be output in addition to the recognition evaluation result.

One purpose of taking a spatial measurement on an object is, among other things. in the detection of the position and posture of a measuring object. For example, an application program can be used with which a robot is made to grasp an object. In a case where the position and posture of an object to be grasped are not known beforehand, it is necessary to detect the object on the spot to determine the position and posture of the object or the position at which the robot moves the object can grasp, recognize.

In the seventh embodiment of the present invention, for this reason, is the object recognition processing unit 108 that performs the object recognition process when a simulation result of spatial measurement is input. An algorithm used for object recognition by the object recognition processing unit 108 any algorithm can be used. A spatial point group in which the result of a spatial measurement is used as a set of points in a three-dimensional space can be input into the object recognition algorithm, or a depth image which expresses the result of a spatial measurement in a two-dimensional image can be input into the object recognition algorithm will.

In addition, the result of an object detection is used to determine the To evaluate recognition results such as, for example, the accuracy of the estimation of the position and posture of the recognized object and the accuracy of the estimation of the gripping position. A problem when evaluating an algorithm for recognizing the position and posture of an object is in many cases that real values relating to the position and posture of the object are not known. Because the real values are not known, it is difficult to quantitatively determine whether or not an outputted recognition result of the position and posture of an object is correct. In the simulation, however, the position and posture of an object are known, which enables a quantitative evaluation of a recognition result.

As described above, the simulation device enables 17th According to the seventh embodiment of the present invention, the verification of the object recognition by using a simulation result. This design enables a performance evaluation of the object recognition without actually providing a device for spatial measurements. Since the position and posture of an object to be recognized are known in a simulation space, an object recognition result can be compared with real values, which brings about an effect of facilitating the quantitative evaluation of the object recognition performance.

For a system in which a robot recognizes and grasps an object, it is very important to quantitatively evaluate whether the accuracy of object recognition is sufficient or not. This is because the accuracy of the object detection has a significant impact on whether or not the robot succeeds in grasping it. For example, in a case where an object is gripped in the heaped state in which a plurality of objects are irregularly piled up, it is difficult to know real values of positions and postures of the objects loosely in a heap. Therefore, even when tests are carried out by using actual devices including the spatial measurement device and the robot, the exact accuracy of object recognition cannot be obtained quantitatively. The present invention is advantageous in easily obtaining real values through simulation and enabling quantitative evaluation of the accuracy of object recognition.

Eighth embodiment.

23 Fig. 16 is a diagram showing a functional configuration of a simulation device 18th according to an eighth embodiment of the present invention. The simulation device 18th includes an object grasping evaluation unit 110 in addition to the configuration of the simulation device 17th according to the seventh embodiment.

The following is the description of the configuration that is the same as that of the simulation device 17th is similar, not repeated; The main differences to the simulation device are described 17th .

The object gripping assessment unit 110 obtains a gripping judgment result of an object of which the success probability of gripping based on the measurement condition information obtained by the measurement condition obtaining unit 101 are output, the simulation result that is generated by the computing unit for spatial measurements 103 is output, and the recognition result obtained by the object recognition processing unit 108 is output, is evaluated. The object gripping assessment unit 110 outputs the seizure evaluation result to the output unit 104 a. The output unit 104 also outputs the capture assessment result.

24 FIG. 13 is a flow chart illustrating the operation of the FIG 23 illustrated simulation device 18th represents. Since the operations in steps S301 to S307, steps S701 to S703, and steps S801 to S803 are similar to those in 22nd are similar, their detailed explanation is not repeated here. The following are mainly differences too 22nd described.

Upon receiving the object recognition result, the object grasping evaluation unit takes 110 an object grasping evaluation (step S901). The object gripping assessment unit 110 outputs the seizure evaluation result to the output unit 104 a. The output unit 104 outputs the seizure judgment result (step S902). The operation of step S901 can be performed in parallel with the recognition evaluation process.

By using the object recognition result, information on the position at which a robot hand grasps an object can be obtained. The information can be used to simulate the position at which the robot hand comes into contact with the object and the amounts and directions of forces acting on the robot hand or the object when the robot hand is moved to the gripping position and the gripping operation is performed .

If the amounts and directions of the forces acting on the robot hand and the object are known, it is possible to estimate whether that is Seizing will succeed or not. For example, assume a case where the robot hand is a parallel hand with two grippers. A case where a gripped object falls through the gap between the grippers of the robot hand corresponds to a failure to grasp. A situation corresponding to this case may be that a force acting on the gripped object in a direction perpendicular to the closing direction of the parallel hand is greater than a frictional force between the robot hand and the gripped object. Therefore, information such as coefficients of friction of the surfaces of the robot hand and the gripped object, the weight of the gripped object, and the gripping force of the robot hand enable a determination of whether or not the gripping will succeed.

The method for evaluating the probability of success of the gripping is not limited to the method for estimating the forces that act between the robot hand and the gripped object; instead, any evaluation method can be used. In the eighth embodiment of the present invention, the object grasping evaluation unit is 110 which has such a function of simulating grasping of an object by a robot. An example of an output screen in the eighth embodiment is shown in FIG 25th shown. 25th is a diagram showing an example of a display screen 60 represented by the in 23 illustrated simulation device 18th is issued. The display screen 60 includes a recognition and seizure evaluation result display region 61 in addition to displaying the display screen 30th . The method of outputting the recognition result and the seizure judgment result may be outputting the success or failure of the recognition or grasping using various symbols in the case of success and the failure, outputting a cause of failure in the case of failure, or displaying consist of a quantitative value used for valuation.

As described above, the simulation device is 18th according to the eighth embodiment of the present invention, is able to evaluate a success rate of grasping by simulation, which brings about an advantageous effect that is to facilitate pre-verification before a robot system is constructed.

To construct a system that enables a variety of workflows from detecting to grasping an object by using an actual robot, it takes a significant amount of time and money to install a spatial measurement device, adjust object recognition software, adjust Work processes of the robot and the like are required. In addition, a long time and trial and error are also required for tests in which grasping an object is repeated in a real space. According to the present invention, gripping tests can be performed by simulation, which enables tests to be performed in a shorter period of time than those of the prior art.

In a case where an actual robot system is used to conduct seizure tests and an analysis of the cause of seizure failures is attempted, results of the success of grasping or failure of grasping can only be obtained in a state in which all the failure factors caused by errors ranging from spatial measurements to errors in setting up the robot's workflows. As a result, it is difficult to analyze what corrections should be made to improve the success rate of the grasping. Within the scope of the present invention, the failure factors can be distinguished from one another and verified by simulation, which enables causes and improvements to be determined in a short period of time.

The following is a hardware configuration of the simulation devices 10 , 12th , 13th , 14th , 15th , 16 , 17th and 18th according to the first to eighth embodiments of the present invention. The measurement condition maintenance unit 101 , the generation unit for virtual recording images 102 , the computing unit for spatial measurements 103 , the output unit 104 , the failure information preservation unit 105 , the evaluation reference data generation unit 106 , the measurement evaluation unit 107 , the object recognition processing unit 108 , the recognition evaluation unit 109 and the object grasping evaluation unit 110 are implemented by a processing circuit. The processing circuit can be implemented by dedicated hardware or a control circuit using a central processing unit (CPU).

In a case where the processing circuit is implemented by dedicated hardware, the components are implemented by the in 26th processing circuit shown 90 implemented. 26th is a scheme showing dedicated hardware for implementing the functions of the simulation devices 10 , 12th , 13th , 14th , 15th , 16 , 17th and 18th according to the first to eighth embodiments of the present invention. The processing circuit 90 is a single circuit, a compound circuit, a programmed processor, a parallel programmed processor, an application-specific integrated circuit (ASIC), a field-programmable gate arrangement (FPGA) or a combination thereof.

In a case where the processing circuit is implemented by a control circuit using a CPU, the control circuit is a control circuit 91 with an in 27 configuration shown, as an example. 27 is a diagram showing a configuration of the control circuit 91 for implementing the functions of the simulation devices 10 , 12th , 13th , 14th , 15th , 16 , 17th and 18th according to the first to eighth embodiments of the present invention. As in 27 shown includes the control circuit 91 a processor 92 and a memory 93 . The processor 92 is a CPU and is also referred to as a central processing device, processing device, computing device, microprocessor, microcomputer, digital signal processor (DSP) or the like. The memory 93 is a non-volatile or volatile semiconductor memory such as a random access memory (RAM), a read-only memory (ROM), a flash memory, an erasable programmable ROM (EPROM) or an electrically erasable programmable ROM (EEPROM; Registered Trademark), a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, a digital versatile disk (DVD) or the like, for example.

In a case where the processing circuit is controlled by the control circuit 91 is implemented, the processing circuitry is implemented by the processor 92 implemented, which reads and executes programs that the processes of the respective components that are in the memory 93 are stored. Furthermore, the memory 93 also used as temporary storage in processes run by the processor 92 be performed.

In addition, the functions of the simulation devices 10 , 12th , 13th , 14th , 15th , 16 , 17th and 18th according to the first to eighth embodiments of the present invention also by hardware with an in 28 can be implemented. 28 Fig. 13 is a diagram showing an example of a hardware configuration for implementing the functions of the simulation devices 10 , 12th , 13th , 14th , 15th , 16 , 17th and 18th according to the first to eighth embodiments of the present invention. The functions of the simulation devices 10 , 12th , 13th , 14th , 15th , 16 , 17th and 18th According to the first to eighth embodiments of the present invention, by using an input device 94 and an output device 95 in addition to the processor 92 and the memory 93 implemented. The input device 94 is an input interface such as a keyboard, mouse, or touch sensor that receives operations input by a user. The output device 95 is a display device, for example, that can display screens to the user. In a case where a touch panel is used, a display device of the touch panel corresponds to the output device 95, and a touch sensor overlaid on the display device corresponds to the input device 94.

For example, the functions of the measurement condition maintenance unit 101 and the output unit 104 of the simulation devices 10 , 12th , 13th , 14th , 15th , 16 , 17th and 18th through the processor 92 or they can be implemented by the processor 92 and the input device 94 or the output device 95 or interfaces thereof are implemented.

The constructions set forth in the above embodiments are examples of the present invention and can be combined with other known techniques or partially omitted or modified without departing from the scope of the present invention.

For example, the functions of the simulation devices 10 , 12th , 13th , 14th , 15th , 16 , 17th and 18th according to the first to eighth embodiments of the present invention can be implemented by one piece of hardware, or they can be processed in a distributed manner by a plurality of pieces of hardware.

In addition, they are the display screens described above 20th , 30th , 40 , 50 and 60 about examples that can be modified in different ways.

List of reference symbols

10, 12, 13, 14, 15, 16, 17, 18

Simulation device;

20, 30, 40, 50, 60

Display screen;

21

Processing result display region;

22nd

Measurement condition display region;

23

List display region;

24

Selection region for selected content;

25th

Run button;

26, 42

Save button;

27, 43

Exit button;

28

Error factor setting region;

41

Adjustment aspect display region;

44

Processing circuit; Measurement result display region;

51

Evaluation reference data display region;

61

Detection and seizure evaluation result display region;

90 91

Control circuit;

92

Processor;

93

Storage;

101

Measurement condition maintenance unit;

102

Generation unit for virtual recorded images;

103

Computing unit for spatial measurements;

104

Output unit;

105

Failure information preservation unit;

106

Evaluation reference data generation unit;

107

Measurement evaluation unit;

108

Object recognition processing unit;

109

Recognition evaluation unit;

110

Object recognition processing unit;

201

Projection condition information obtaining unit;

202

Mapping condition information obtaining unit;

203

Measurement object information preservation unit;

204

Non-measurement object information obtaining unit;

301

Generation unit for optically reproduced images;

302

Image quality deterioration processing unit;

303

Error factor determination unit;

401

Generation unit for sensor position data;

402

Card generation unit;

403

Image combining unit;

501

Generation unit for bright images;

502

Generation unit for dark images;

503

Distance data generation unit;

601

Exposure region calculation unit;

602

Exposure map generation unit;

603

Light blur reproduction unit;

604

Reflection region calculation unit;

605

Reflection map generation unit.

QUOTES INCLUDED IN THE DESCRIPTION

This list of the documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent literature cited

• JP 2015203652 [0003]

Claims (29)

Simulation device comprising: a measurement condition obtaining unit for obtaining measurement condition information indicating a measurement condition of a spatial measurement device that includes a projecting device for projecting light onto a measurement object and an imaging device for capturing an image of an imaging space containing the measurement object, which is projected with light from the projecting device is irradiated includes; a virtual captured image generation unit for generating a virtual captured image that reproduces a captured image output by the imaging device based on the measurement condition information; a spatial measurement arithmetic unit for performing a spatial measurement process of determining a spatial position on a surface of the measurement object by using the virtual captured image to obtain a measurement value; and an output unit for outputting a simulation result that contains the measured value. Simulation device according to Claim 1 wherein the measurement condition information includes at least one of information indicating a projection condition of the projecting device, information indicating an imaging condition of the imaging device, information indicating a shape of the measuring object, information indicating a state of the measuring object, and information indicating the characteristics of the measurement object are included. Simulation device according to Claim 2 wherein the information indicating the projection condition is information that enables a position and a posture, a resolution and a focus of the projecting device to be determined, the information indicating the imaging condition is information that enables that a position and a posture, a resolution and a focus of the imaging device are determined, the information indicating the state of the measurement object, at least one of a position and a posture of the measurement object, and the information indicating the characteristics of the measurement object , contain at least one of the color, diffuse reflection and specular reflection of the target. Simulation device according to one of the Claims 1 until 3 wherein the measurement condition information further includes information indicating at least one of a position, a shape, and characteristics of an object other than the measurement object included in the imaging space. Simulation device according to one of the Claims 1 until 4th wherein the measurement condition information further includes information indicating a state of the ambient light. Simulation device according to one of the Claims 1 until 5 wherein the captured virtual image generation unit generates the captured virtual image with a pixel value reproducing an error contained at a position corresponding to a boundary of a shadow cast by the projected light in the captured image based on the measurement condition information he follows. Simulation device according to Claim 6 wherein the generation unit for virtual recorded images generates the virtual recorded image, the error being reproduced which is caused by at least one occurrence factor of interreflection caused by exposure to light reflected from a surface of an object, blurring of the projected light caused by defocusing of the projecting Device, image distortion caused by distortion of an object of the imaging device, random noise, and image blurring caused by defocusing of the imaging device. Simulation device according to Claim 6 or 7th wherein the captured virtual image generation unit includes: an optically reproduced image generation unit for generating an optically reproduced image, the captured image being reproduced by optical simulation based on the measurement condition information; and an image quality deterioration processing unit for performing an image quality deterioration process on the optically reproduced image based on the defect, and an image that has been subjected to the image quality deterioration process is used as the captured virtual image. Simulation device according to Claim 8 wherein the captured virtual image generation unit further includes: an error information obtaining unit for obtaining error information indicating an occurrence factor of the error to be expressed in the captured virtual image, the optically reproduced image generation unit based on the optically reproduced image of the measurement condition information and the error information, and the image quality deterioration processing unit performs an image quality deterioration process on the optically reproduced image based on the measurement condition information and the error information. Simulation device according to Claim 9 wherein the defect information includes intensities of defects depending on types of occurrence factors of defects, and the virtual captured image generation unit determines an intensity of the deterioration process to be performed on the optically reproduced image depending on types of occurrence factors of defects based on the defect information. Simulation device according to Claim 9 or 10 wherein the defect information includes an addition order of defects according to kinds of occurrence factors of the defects, and the virtual captured image generation unit determines an order of the deterioration process to be performed on the optically reproduced image according to kinds of occurrence factors of the defects based on the defect information . Simulation device according to one of the Claims 1 until 11 wherein the virtual captured image generation unit calculates a propagation path of the light based on the measurement condition information, and when the projected light is reflected from a first surface and reflected light is incident on a second surface, a luminance of a pixel that is an incident point of the reflected light on the second surface, increased such that it is higher than a first luminance in which the influence of the reflected light is not taken into account. Simulation device according to Claim 12 wherein the captured virtual image generation unit determines an amount of increase in luminance of the pixel based on surface characteristics of the first surface, surface characteristics of the second surface, and a position of the first surface with respect to the second surface. Simulation device according to one of the Claims 1 until 13th wherein the virtual captured image generation unit has a first region that is a region that is irradiated with the projected light and a second region that is a region that is not irradiated with the projected light on the basis of the measurement condition information, and determines a luminance of a pixel within a predetermined distance from a boundary between the first region and the second region based on a distance between the pixel and the boundary. Simulation device according to Claim 14 , wherein the generation unit for virtual recorded images sets the luminance of the pixel within the predetermined distance from the boundary such that it is less than or equal to a maximum value that corresponds to a luminance with illumination with the projected light, and greater than or equal to a is the minimum value which corresponds to a luminance without irradiation with the projected light. Simulation device according to Claim 15 wherein the captured virtual image generation unit sets the luminance of the pixel closer to the maximum value when the pixel is in closer proximity to the first region, and sets the luminance of the pixel closer to the minimum value when the pixel is in closer proximity to the second region is located. Simulation device according to one of the Claims 1 until 16 wherein the virtual captured image generation unit includes: a sensor standpoint data generation unit for generating sensor standpoint data including a first image of the imaging space from the viewpoint of the imaging device and distance data from each of the imaging device and the projecting device to an object in the imaging space Use of the measurement condition information; and a map generation unit for generating an exposure map which is a map indicating, with different numerical values, a region that is irradiated with the projected light and a region that is not irradiated with the projected light in the first image , based on the measurement condition information, and the virtual captured image generation unit generates the virtual captured image based on the exposure map and the first image. Simulation device according to Claim 17 , wherein the first image includes a bright image in which the entire imaging space is displayed with a brightness of the irradiation with the projected light, and a dark image in which the imaging space is not irradiated with the projected light, and the generation unit for virtual Pickup Images the virtual pickup image by combining the light image and the dark image in units of Pixels generated based on the values on the exposure map. Simulation device according to Claim 18 wherein the map generation unit includes: an irradiation region calculating unit for calculating an irradiation region, which is a region irradiated with the projected light, in the first image by associating a spatial position of a point irradiated with the projected light with coordinates in the first image by using the measurement condition information and the sensor point of view data; and an exposure map generation unit for generating the exposure map for each projection pattern of the projected light by using the exposure region and the measurement condition information. Simulation device according to Claim 19 wherein the map generation unit further includes: a light blur reproduction unit for reproducing blurring of the light by adjusting the values on the exposure map depending on a degree of blurring of a shadow boundary of the light, and the map generating unit each of the luminances of the light image and the dark image based on the adjusted Values are weighted on the exposure map and the weighted luminances are summed in order to determine a luminance of each pixel in the virtual recorded image. Simulation device according to one of the Claims 18 until 20th wherein the map generation unit includes: a reflection region calculation unit for calculating a reflection region, which is a region irradiated with a reflected light beam, in the first image by associating a spatial position of a point irradiated with the reflected light beam with coordinates on the first image Using the measurement condition information and the sensor standpoint data, wherein the reflected light beam is a light beam emitted from the projecting device, reflected from an object in the imaging space, and the intensity and direction of which are changed; and a reflection map generation unit for generating a reflection map, which is a map indicating, with various numerical values, the reflection region and a region which is not irradiated with the reflected light beam, for each projection pattern of the projected light by using the reflection region and the measurement condition information , and the captured virtual image is generated by using the reflection map, the exposure map, and the first image. Simulation device according to Claim 21 wherein the map generation unit adjusts values in the reflection map depending on an intensity of the reflected light beam and a degree of blurring of a shadow boundary of the projected light, weighting each of the luminances of the light image and the dark image based on the adjusted values on the reflection map and the exposure map, and sums the weighted luminances to determine a luminance of each pixel in the virtual captured image. Simulation device according to one of the Claims 1 until 22nd , wherein the output unit outputs the simulation result, which furthermore contains at least one of at least part of the measurement condition information and the virtual recorded image. Simulation device according to one of the Claims 1 until 23 wherein the measurement condition obtaining unit receives a plurality of pieces of the measurement condition information, and the output unit outputs a list of simulation results each relating information on one or more aspects specified as aspects to be adjusted by the pieces of the measurement condition information with the measurement value. Simulation device according to one of the Claims 1 until 24 , further comprising: an evaluation reference data generation unit for generating evaluation reference data, which is a reference for evaluating the simulation result, by using the measurement condition information; and a measurement evaluation unit for evaluating the simulation result by using the simulation result and the evaluation reference data, the output unit outputting an evaluation result in addition to the simulation result. Simulation device according to one of the Claims 1 until 25th , further comprising: an object recognition processing unit for obtaining, as a recognition result, at least one of a position and a posture of an object in the imaging space and a gripping position which is a position at which the object is able to be gripped , based on the simulation result; and a recognition evaluation unit for obtaining a recognition evaluation result including at least one of an accuracy of an estimation of the position and the posture and an accuracy of an estimation of the gripping position Using the recognition result and the measurement condition information, wherein the output unit continues to output the recognition evaluation result. Simulation device according to Claim 26 , further comprising: an object gripping evaluation unit for evaluating the probability of succeeding in grasping an object in the imaging space by using the measurement condition information and the recognition result. Simulation method that is executed by a simulation device that simulates a process of spatial measurement of a device for spatial measurements that includes a projecting device for projecting light onto a measurement object to be measured and an imaging device for capturing an image of an imaging space containing the measurement object, which is irradiated with projected light from the projecting device, the simulation method comprising: a step of obtaining measurement condition information indicating a measurement condition of the spatial measurement apparatus; a step of generating a virtual captured image, reproducing a captured image output by the imaging device based on the measurement condition information; a step of performing a spatial measurement process of determining a spatial position on a surface of the measurement object by using the virtual captured image to obtain a measurement value; and a step of outputting a simulation result that contains the measured value. Simulation program that causes a computer to do the following: a step of obtaining measurement condition information indicating a measurement condition of a device for spatial measurements, which a projecting device for projecting light onto a measurement object to be measured and an imaging device for taking an image of an imaging space containing the measurement object, which with projected light from the projecting device is irradiated includes; a step of generating a virtual captured image, reproducing a captured image output by the imaging device based on the measurement condition information; a step of performing a spatial measurement process of determining a spatial position on a surface of the measurement object by using the virtual captured image to obtain a measurement value; and a step of outputting a simulation result that contains the measured value.

Images