JP2021128041A - Information processing apparatus, information processing method, and program - Google Patents

Abstract

To accurately calculate the normal line of an object even if a surface of the object is a rough surface.SOLUTION: An information processing apparatus of the present invention has: acquisition means that acquires the reflection characteristics of a target object based on a picked-up image of the target object; first calculation means that calculates a first normal line direction of an attention position in the target object based on the acquired reflection characteristics, a first direction in which incident light is incident on the target object, and a second direction of the peak of light that is the incident light incident from the first direction and reflected on the target object; and determination means that determines a condition for calculating a second normal line direction in the attention position based on the calculated first normal line direction.SELECTED DRAWING: Figure 2

Description

The present invention relates to a technique for calculating a normal on the surface of an object.

The light radiated to the object causes diffuse reflection that is diffusely reflected inside the object from the irradiation point and then is emitted again from the irradiation point, and surface reflection that is reflected on the surface of the object. There is a method of obtaining the normal direction of the object surface by using this surface reflection. When the surface of the target object is a smooth surface, the light incident on the surface of the target object is reflected in a specific direction, and the angle formed by the normal direction and the incident direction of the target object is defined as the incident angle. Assuming that the angle formed by the linear direction and the reflection direction is the reflection angle, the incident angle and the reflection angle match. Therefore, by measuring the incident direction and the reflection direction, the normal direction can be estimated from the direction of the bisectors in those directions (for example, Patent Document 1).

Japanese Unexamined Patent Publication No. 2018-136693

However, when the surface of the target object is rough and rough, the light incident on the surface of the target object is reflected with a spread in a certain direction, but the angle of incidence formed by the normal direction and the incident direction of the target object and the method of the target object The reflection angles formed by the linear direction and the peak direction of the reflected light do not exactly match. Therefore, the accuracy of the normal direction calculated on the assumption that the incident angle and the reflection angle match may be low.

The present invention has been made in view of the above problems, and an object of the present invention is to calculate the normal of an object with high accuracy even if the surface of the object is a rough surface.

The information processing apparatus of the present invention includes an acquisition means for acquiring the reflection characteristics of the target object based on an image of the target object, the acquired reflection characteristics, and a first method in which incident light is incident on the target object. A first calculation means for calculating the first normal direction of the position of interest in the target object based on the direction and the second direction of the peak of the reflected light in the target object with respect to the incident light incident from the first direction. It is characterized by having a determination means for determining a condition for calculating the second normal direction at the attention position based on the calculated first normal direction.

According to the present invention, even if the surface of the object is rough, the normal of the object can be calculated with high accuracy.

It is a schematic diagram for demonstrating the surface reflection in a rough surface. It is a schematic diagram for demonstrating the outline of the calculation method of a normal direction. It is a figure which shows the appearance of the normal calculation system. It is a block diagram which shows the hardware configuration of an information processing apparatus. It is a block diagram which shows the functional structure of an information processing apparatus. It is a flowchart which shows the process of 1st Embodiment. It is a flowchart which shows the process of acquiring an image. It is a flowchart which shows the process of imaging a target object and a white plate. It is a block diagram which shows the functional structure of the normal calculation part. It is a flowchart which shows the process of calculating the normal information. It is a flowchart which shows the process of calculating the variable angle reflection characteristic information. It is a flowchart which shows the process of calculating the 1st normal information. It is a flowchart which shows the process of calculating the 2nd normal information. It is a block diagram which shows the functional structure of an image acquisition control part and a normal calculation part. It is a flowchart which shows the process of 2nd Embodiment. It is a flowchart which shows the process of calculating the 1st normal information. It is a flowchart which shows the process of calculating a lighting position. It is a flowchart which shows the process of calculating the 2nd normal information.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. It should be noted that the following embodiments do not limit the present invention, and not all combinations of features described in the present embodiment are essential for the means for solving the present invention. The same configuration will be described with the same reference numerals.

[First Embodiment]
FIG. 1A is a schematic diagram for explaining surface reflection when the surface of the target object is a rough surface. 101 is the target object, 102 is the rough surface, and 103 is the general normal direction of the rough surface to be measured. Here, the global normal direction of the rough surface is defined as the direction perpendicular to the plane when the rough surface is approximated by the plane. 104 is the incident direction of the light incident on the rough surface 102, and 105 is the specular reflection direction. Here, the specular reflection direction 105 is a direction in which the angle formed by the incident direction 104 and the global normal direction 103 is equal to the angle formed by the global normal direction 103 and the specular reflection direction 105. The normal direction 103, the incident direction 104, and the specular reflection direction 105 are on the same plane. Further, 106 is the reflected light having a spread, and 107 is the direction of the peak of the reflected light.

FIG. 1B is a schematic diagram for explaining the distribution of a plurality of minute surfaces constituting the rough surface 102 in the normal direction. Reference numeral 108 denotes an axis indicating the magnitude of the inclination of the normal of the minute surface. The origin is the global normal direction of the rough surface. Further, 109 is an axis indicating the frequency. It is assumed that the distribution of the plurality of minute surfaces constituting the rough surface 102 in the normal direction is symmetrical with respect to the origin.

FIG. 1C is a schematic diagram for explaining the incident angle dependence of the reflectance of surface reflection on a minute surface. It is assumed that the surface of the minute surface constituting the rough surface is a smooth surface. Reference numeral 111 denotes an axis indicating the magnitude of the incident angle formed by the normal direction and the incident direction of the minute surface. Reference numeral 112 denotes an axis indicating the reflectance. The reflectance of surface reflection on a smooth surface follows the so-called Fresnel equation, but as shown in FIG. 1 (c), it is a downwardly convex monotonic increasing function and has an extremum at an incident angle of 0 °. Further, paying attention to the reflectance in the vicinity of the angle theta ₁ of 113, theta is larger reflectance theta ₁ is greater than angle than ₁ less than the angle reflectance. Therefore, since the reflectance increases as the inclination of the minute surface increases, the incident angle and the reflection angle do not match on the rough surface. Focusing on the reflectance in the vicinity of the angle θ ₂ of 114, it is said that the reflectance of the angle larger than _{θ 2 is} larger than the reflectance of the angle smaller than _{θ 2} as in the vicinity _{of the angle θ 1 of 113.} The features are the same, but the difference is small. That is, if the angle formed by the incident direction 104 and the global normal direction 103 is small, the angle formed by the specular reflection direction 105 and the peak direction 107 of the reflected light becomes small. However, since it is not possible to know in advance the global normal direction 103 of the rough surface to be measured, the incident direction 104 in which the angle formed by the incident direction 104 and the global normal direction 103 becomes small is also obtained in advance. I can't know.

2 (a) and 2 (b) are schematic views for explaining the outline of the calculation method of the normal direction in the present embodiment. First, the rough surface is irradiated with light from an arbitrary direction 201, and the intensity distribution of the reflected light 202 having a spread is measured. The direction 203 of the peak of the reflected light is calculated from the measured intensity distribution of the reflected light 202. Next, the direction 204 of the bisector is calculated from the arbitrary direction 201 and the direction 203 of the peak of the reflected light. The direction 204 of the bisector is a temporary normal direction 204', which is the result of the first calculation of the normal direction. Subsequently, as shown in FIG. 2 (b), the rough surface is irradiated with light from the direction 205 opposite to the temporary normal direction 204', the intensity distribution of the reflected light 206 having a spread is measured, and the measured reflected light is measured. The direction 207 of the peak of the reflected light is calculated from the intensity distribution of 206. Next, the direction 208 of the bisector is calculated from the opposite direction 205 and the peak direction 207. The direction 208 of the bisector is the calculation result of the second normal direction. The direction 208 of the bisector is closer to the true normal direction (global normal direction 103) than the tentative normal direction 204'. In this way, the conditions for calculating the second normal direction are determined based on the calculation of the first normal direction. The condition here is the direction in which the rough surface is irradiated with light when the second normal direction is calculated. Next, by applying the determined condition and calculating the normal direction for the second time, the normal direction of the target object can be calculated with higher accuracy.

Next, a normal calculation system for realizing the above-mentioned normal direction calculation method will be described.
FIG. 3 is a diagram showing the appearance of the normal calculation system of the first embodiment. FIG. 3A is a front view of the normal calculation system as viewed from the front, and FIG. 3B is a top view of the normal calculation system as viewed from the top. Reference numeral 301 denotes a target object to be measured. The target object 301 will be described as a substantially flat body having surface irregularities of several mm such as a printed matter, cloth, or a painted surface. Reference numeral 302 denotes an object fixing jig for fixing the target object 301. Reference numeral 303 denotes an imaging device such as a DSLR (Digital Single Lens Reflex camra) that images the target object 301. As the image pickup apparatus 303, for example, a DSLR having a CMOS type area sensor of 8688 × 5792 pixels can be used. In the present embodiment, the image pickup apparatus 303 has a photoelectric conversion characteristic of obtaining a signal value linear with respect to the brightness on the target object 301. It is assumed that the image data captured and stored by the image pickup apparatus 303 has the luminance information of one gray channel in each pixel, and the channel is quantized by 16 bits.

Further, 304 is a stage for changing the position of the image pickup apparatus 303. In the present embodiment, in order to perform imaging on the target object 301 from various directions, the position of the imaging device 303 is changed to perform imaging, but the method of changing the position of the imaging device 303 is in this case. Not limited. For example, a plurality of image pickup devices may be fixed on a hemispherical fixing jig, and the position of the image pickup device may be changed by switching the image pickup device to be used. Further, the fixing jig for fixing the stage 304 and the imaging device does not have to be hemispherical as long as the target object 301 can be imaged from various directions, and may be, for example, a two-dimensional grid shape. The hemispherical shape can make the distance from the image pickup apparatus 303 to the target object 301 substantially constant. On the other hand, in the two-dimensional grid pattern, the distance from the image pickup apparatus 303 to the target object 301 cannot be made constant, so that additional processing such as resolution conversion processing is required.

Further, the 305 is a lighting device composed of a light source such as a plurality of LEDs (Light Emitting Diodes) for irradiating the target object 301. In the lighting device 305, each light source is arranged on the optical fixing jig 306. In the present embodiment, in order to irradiate the target object 301 with light from various directions, the lighting device 305 composed of a plurality of light sources is used, but a stage in which the position of the light sources can be changed may be used. .. Further, the optical fixing jig 306 is not hemispherical, and may be, for example, a two-dimensional lattice as long as the target object 301 can be irradiated with light from various directions. Further, the 307 is an information processing device that controls the image pickup device 303, the stage 304, and the lighting device 305 to perform a series of processes for acquiring an image of the target object 301.

FIG. 4 is a block diagram showing a hardware configuration of the information processing device 307. The information processing device 307 includes a CPU 401, a ROM 402, and a RAM 403. The information processing device 307 includes a VC (video card) 404, a general-purpose I / F (interface) 405, a SATA (Serial ATA) I / F 406, and a NIC (network interface card) 407. The CPU 401 uses the RAM 403 as a work memory to execute an OS (operating system) and various programs stored in a ROM 402, an HDD (hard disk drive) 412, and the like. Further, the CPU 401 controls each configuration via the system bus 408. The process according to the flowchart described later is executed by the CPU 401 after the program code stored in the ROM 402, the HDD 412, or the like is expanded into the RAM 403. A display 414 is connected to the VC 404. An input device 410 such as a mouse or keyboard, an image pickup device 303, a stage 304, and a lighting device 305 are connected to the general-purpose I / F 405 via a serial bus 409. A general-purpose drive 413 that reads and writes HDD 412 and various recording media is connected to SATAI / F406 via the serial bus 411. The NIC 407 inputs and outputs information to and from an external device. The CPU 401 uses various recording media mounted on the HDD 412 or the general-purpose drive 413 as a storage location for various data. The CPU 401 displays a GUI (graphical user interface) provided by the program on the display 414, and receives an input such as a user instruction received via the input device 410.

FIG. 5 is a block diagram showing a functional configuration of the information processing device 307. The information processing device 307 includes an image pickup driver 501, a lighting driver 502, a stage driver 503, an input driver 504, a display driver 505, an image acquisition control unit 506, a normal calculation unit 507, and a UI management unit 508. The image pickup driver 501 controls the image pickup device 303 based on an instruction from the image acquisition control unit 506. The lighting driver 502 controls the lighting device 305 based on an instruction from the image acquisition control unit 506. Specifically, the lighting driver 502 individually turns on and off a plurality of light sources of the lighting device 305. The stage driver 503 controls the stage 304 based on an instruction from the image acquisition control unit 506. Specifically, the stage driver 503 starts and ends the movement of the stage 304. The input driver 504 controls the input device 410 based on the instruction from the UI management unit 508. The display driver 505 controls the display 414 based on the instruction from the UI management unit 508. The image acquisition control unit 506 controls the image pickup driver 501, the illumination driver 502, and the stage driver 503 to perform a series of processes for acquiring the captured image. The normal calculation unit 507 calculates the normal information of the target object from the image acquired by the image acquisition control unit 506. The normal information calculated in this embodiment is the normal direction for each position of the target object. The UI management unit 508 manages the user interface function. Specifically, the UI management unit 508 manages the information input by the user to the input device 410 and displays the measurement result on the display 414.

FIG. 6 is a flowchart showing an example of processing by the information processing apparatus 307.
In S601, the UI management unit 508 receives and sets condition settings such as imaging conditions of the imaging device 303, stage moving conditions of the stage 304, and lighting conditions of the lighting device 305 from the user. The imaging conditions set here are the aperture value, shutter speed, ISO sensitivity, and the like. Further, the stage movement conditions to be set are a plurality of positions for imaging on the stage 304, a movement order, and the like. The lighting conditions to be set are the position of the light source to be lit, the lighting order, and the like.

In S602, the image acquisition control unit 506 changes the position of the image pickup device 303 by moving the stage 304, and acquires an image by performing irradiation by the illumination device 305 on the target object and imaging by the image pickup device 303. Perform a series of processes. The specific processing of S602 will be described later.
In S603, the normal calculation unit 507 calculates the normal information of the target object from the image acquired in S602. The specific processing of S603 will be described later.
In S604, the UI management unit 508 displays the normal information of the target object calculated in S603 on the display 414, and ends the process. In S604, the UI management unit 508 may directly store the entire image of the target object in a recording medium such as HDD 412 without displaying it on the display 414.

FIG. 7 is a flowchart showing a process of acquiring an image in S602 of FIG. In the present embodiment, in order to correct the difference in illuminance that differs depending on the location of the target object, imaging for calibration is performed separately from the imaging of the target object.
In S701, the image acquisition control unit 506 acquires an image of the target object.
In S702, the image acquisition control unit 506 acquires an image captured for calibration. Since the brightness of the light source changes with time but does not depend on the order, it is preferable that the time difference between S701 and S702 is small, and the order of S701 and S702 may be reversed. Further, S702 may be performed before and after S701.

FIG. 8A is a flowchart showing a specific process in S701 of FIG. 7. The user fixes the target object 301 on the object fixing jig 302 in advance.
In S801, the image acquisition control unit 506 repeatedly performs the processes of S802 to S806 at each imaging position on the stage 304.
In S802, the image acquisition control unit 506 moves the image pickup apparatus 303 to a position designated based on the stage movement condition set in S601 via the stage 304.
In S803, the image acquisition control unit 506 repeats the processes of S804 to S806 for each position of the plurality of light sources of the lighting device 305.
In S804, the image acquisition control unit 506 lights the light source specified in the lighting device 305 based on the lighting conditions set in S601.
In S805, the image acquisition control unit 506 performs image pickup by the image pickup apparatus 303 based on the image pickup conditions set in S601.
In S806, the image acquisition control unit 506 turns off the light source specified in the lighting device 305 based on the lighting conditions set in S601.

Next, the image acquisition control unit 506 returns to S803, changes the light source position, and performs the processes of S804 to S806. When the processing of S804 to S806 is completed at all the light source positions, the image acquisition control unit 506 returns to S801 and changes the position of the image pickup apparatus 303 to perform the processing of S803 to S806. The image acquisition control unit 506 ends the processing of S701 when the processing of S803 to S806 is completed at all the imaging positions. As described above, in the present embodiment, the imaging position of the imaging device 303 is changed and the light source position of the lighting device 305 is changed to perform imaging. Since the change of the imaging position of the imaging device 303 is mechanical and the change of the light source position of the lighting device 305 is non-mechanical, the loop of changing the light source position of the lighting device 305 and the loop of changing the imaging position of the imaging device 303 are used. The imaging time can be shortened by placing it inside.

FIG. 8B is a flowchart showing a specific process in S702 of FIG. 7. In addition, while FIG. 8 (a) images the target object 301, in FIG. 8 (b), a white plate is imaged in order to correct the difference in illuminance that differs depending on the location of the target object. Therefore, the user changes the object object 301 fixed on the object fixing jig 302 to a white plate and fixes it. The white plate preferably has a uniform color, a small mirror surface, and a high diffusivity. The processing of the image acquisition control unit 506 in the flowchart of FIG. 8B is the same as that of the flowchart of FIG. 8A, and the description thereof will be omitted.

FIG. 9 is a block diagram showing a detailed functional configuration of the normal calculation unit 507 that executes the process of calculating the normal information in S603 of FIG. The normal calculation unit 507 includes a variable angle reflection characteristic calculation unit 901, a first normal calculation unit 902, and a second normal calculation unit 903.

FIG. 10 is a flowchart showing a process of calculating the normal information in S603 of FIG.
In S1001, the variable angle reflection characteristic calculation unit 901 obtains the variable angle reflection characteristic information of the target object from the images of the target object and the white plate captured by changing the imaging position of the image pickup device 303 and the light source position of the illumination device 305. calculate. The specific processing of S1001 will be described later.
In S1002, the first normal calculation unit 902 calculates the first normal information by analyzing the variable angle reflection characteristic information calculated by the variable angle reflection characteristic calculation unit 901. The specific processing of S1002 will be described later.
In S1003, the second normal calculation unit 903 analyzes the variable angle reflection characteristic information calculated by the variable angle reflection characteristic calculation unit 901 based on the first normal information calculated by the first normal calculation unit 902. The second normal information is calculated by doing so. Here, the first normal information is the calculation result of the first normal information described above, and the second normal information is the calculation result of the second normal information described above.

FIG. 11 is a flowchart showing a process of calculating the variable angle reflection characteristic information in S1001 of FIG.
In S1101 and S1102, the variable angle reflection characteristic calculation unit 901 repeats the processes of S1103 to S1106 for each image in which the imaging position of the imaging device 303 is the same and the light source position of the lighting device 305 is the same.
In S1103, the variable angle reflection characteristic calculation unit 901 reads one image of the target object captured in S701.
In S1104, the variable angle reflection characteristic calculation unit 901 reads one image of the white plate imaged in S702.
In S1105, the variable angle reflection characteristic calculation unit 901 calculates the reflectance information R (i, j) by the following (Equation 1).

Here, the image of the target object is Obj (i, j), and the image of the white plate is Ref (i, j). (I, j) represents the pixel position. By calculating using the image of the white plate in this way, the difference in illuminance that differs depending on the location of the target object is corrected. Using the image Obj_black (i, j), which is an image of the target object without turning on all the light sources of the lighting device 305, and the image Ref_black (i, j), which is an image of the white plate, according to the following (Equation 2). The reflectance information R'(i, j) may be calculated.

By doing so, the noise of the fixed pattern can be reduced, so that the normal direction can be calculated with higher accuracy.

In S1106, the variable angle reflection characteristic calculation unit 901 calculates the direction of the lit light source of the lighting device 305 and the direction of the imaging device 303 for each position of the target object. Specifically, the variable angle reflection characteristic calculation unit 901 includes the coordinates in the three-dimensional space of the image pickup device 303 and the coordinates in the three-dimensional space of the lit light source of the illumination device 305, which are stored in advance in a recording medium such as HDD412. , Read the coordinates of each position of the target object in the three-dimensional space. Next, the variable angle reflection characteristic calculation unit 901 sets the incident angle (for each position of the target object) from the coordinates of the lit light source of the lighting device 305 in the three-dimensional space and the coordinates of each position of the target object in the three-dimensional space. θi, φi) are calculated. Further, the variable angle reflection characteristic calculation unit 901 calculates the reflection angle (θj, φj) from the coordinates of the image pickup apparatus 303 in the three-dimensional space and the coordinates of each position of the target object in the three-dimensional space.

Next, the variable angle reflection characteristic calculation unit 901 returns to S1102, changes the light source position, and performs the processes of S1103 to S1106. When the processing of S1103 to S1106 is completed at all the light source positions, the variable angle reflection characteristic calculation unit 901 returns to S1101 and changes the position of the image pickup apparatus 303 to perform the processing of S1102 to S1106. The variable angle reflection characteristic calculation unit 901 performs the processing of S1102 to S1106 at all the positions of the image pickup apparatus 303.
In S1107, the variable angle reflection characteristic calculation unit 901 aggregates the reflectance information calculated in S1105 and the geometric condition information calculated in S1106, and outputs the variable angle reflection characteristic information. The variable angle reflection characteristic information (reflection characteristic) is the reflectance information at the incident angle (θi, φi) and the reflection angle (θj, φj) for each position of the target object. The variable angle reflection characteristic calculation unit 901 stores the output variable angle reflection characteristic information in a recording medium such as HDD 412.

In this embodiment, the diffuse reflection component and the surface reflection component are used without being separated, but the diffuse reflection component and the surface reflection component can be separated and the variable angle reflection characteristic of the surface reflection component can be used. good. As a processing method for separating the diffuse reflection component and the surface reflection component, there are known methods such as a method using a polarizing filter and a method using a reflection model. By utilizing the variable angle reflection characteristic of the surface reflection component, it is possible to calculate the normal direction with higher accuracy in the case of an object object in which the diffuse reflection component differs depending on the angle.

FIG. 12 is a flowchart showing a process of calculating the first normal information in S1002 of FIG.
In S1201, the first normal calculation unit 902 reads the variable angle reflection characteristic information calculated in S1001 from a recording medium such as HDD 412. In this embodiment, the case where the variable angle reflection characteristic information calculated in S1001 is used will be described, but the case is not limited to this case. The first normal calculation unit 902 may read the variable angle reflection characteristic information acquired by another acquisition device.

In S1202, the first normal calculation unit 902 sets the position (attention position) of the target object and repeats the processes of S1203 to S1205 for each position.
In S1203, the first normal calculation unit 902 sets the direction (first direction) of the incident light. This direction (first direction) corresponds to the arbitrary direction 201 of FIG. 2A described above. The direction of the incident light to be set is stored in advance in a recording medium such as HDD 412.

In S1204, the first normal calculation unit 902 reads the reflectance information for each direction of the reflected light in the direction of the incident light set in S1203 at the position (attention position) of the target object set in S1202 in S1201. However, it is extracted from the variable angle reflection characteristic information. The first normal calculation unit 902 calculates the direction of the peak of the reflected light (second direction) from the reflectance information for each direction of the extracted reflected light. When extracting the reflectance information for each direction of the reflected light, interpolation processing such as spline interpolation may be used. By using the interpolation process, it is possible to improve the calculation accuracy in the normal direction even if the variable angle reflection characteristic information is sparse. Further, when calculating the direction of the peak of the reflected light from the reflectance information for each direction of the extracted reflected light, the direction in which the maximum value of the reflectance information is calculated is calculated as the direction of the peak of the reflected light. Not limited to the case, for example, an interpolation process such as spline interpolation may be used. Further, the direction of the peak of the reflected light may be calculated by fitting a monomodal function to the reflectance information for each direction of the extracted reflected light by using an optimization method such as the least squares method. .. By using the optimization method, it is possible to improve the calculation accuracy in the normal direction even if the variable angle reflection characteristic information is sparse.

In S1205, the first normal calculation unit 902 calculates the direction of the bisector from the direction of the incident light set in S1203 and the direction of the peak of the reflected light calculated in S1204. When calculating the direction of the bisector, the first normal calculation unit 902 normalizes so that the norm of the calculation direction is 1.
Next, the first normal calculation unit 902 returns to S1202, changes the position of the target object, and performs the processes of S1203 to S1205. The first normal calculation unit 902 performs the processes of S1203 to S1205 at all the positions of the target object.
In S1206, the first normal calculation unit 902 outputs the direction of the bisection line calculated in S1205 for each position of the target object as the first normal direction. The first normal calculation unit 902 stores the first normal information, which is a collection of information in the first normal direction for each position of the target object, in a recording medium such as HDD 412.

FIG. 13 is a flowchart showing a process of calculating the second normal information in S1003 of FIG.
In S1301, the second normal calculation unit 903 reads the variable angle reflection characteristic information calculated in S1001 from a recording medium such as HDD 412. This process is the same as S1201.

In S1302, the second normal calculation unit 903 repeats the processes of S1303 to S1305 for each position (attention position) of the target object.
In S1303, the second normal calculation unit 903 calculates the second normal direction for each position of the target object based on the first normal direction calculated by the first normal calculation unit 902. To determine. Specifically, the second normal calculation unit 903 reads the first normal information calculated in S1003 from a recording medium such as HDD 412, and the direction opposite to the first normal direction is the direction of the incident light (third). Direction) and set the direction of the incident light. This direction (third direction) corresponds to the opposite direction 205 of FIG. 2B described above.

In S1304, the second normal calculation unit 903 reads the reflectance information for each direction of the reflected light in the direction of the incident light set in S1303 at the position (attention position) of the target object set in S1302 in S1301. However, it is extracted from the variable angle reflection characteristic information. The second normal calculation unit 903 calculates the direction of the peak of the reflected light (fourth direction) from the reflectance information for each direction of the extracted reflected light. As in S1204, a process for improving the calculation accuracy in the normal direction may be used.
In S1305, the second normal calculation unit 903 calculates the direction of the bisector from the direction of the incident light set in S1303 and the direction of the peak of the reflected light calculated in S1304. When calculating the direction of the bisector, the second normal calculation unit 903 normalizes so that the norm of the calculation direction is 1.
Next, the second normal calculation unit 903 returns to S1302, changes the position of the target object, and performs the processes of S1303 to S1305. The second normal calculation unit 903 performs the processes of S1303 to S1305 at all positions of the target object.
In S1306, the second normal calculation unit 903 outputs the direction of the bisection line calculated in S1305 for each position of the target object as the second normal direction. The second normal calculation unit 903 stores the second normal information, which is the aggregate of the information in the second normal direction for each position of the target object, in a recording medium such as HDD 412.

As described above, according to the present embodiment, the second normal calculation unit 903 has the second normal line at the position of interest of the target object based on the first normal direction calculated by the first normal calculation unit 902. Determine the conditions for calculating the direction. Therefore, the second normal direction calculated by the second normal calculation unit 903 can be made closer to the true normal direction at the position of interest of the target object, and the surface of the target object is high even if it is a rough surface. The normal direction can be calculated accurately.
Further, the second normal calculation unit 903 sets the target object based on the direction of the incident light based on the conditions determined based on the first normal direction and the direction of the peak of the reflected light in the target object. Calculate the second normal direction of the position of interest. Therefore, by repeating the process of calculating the normal direction, the normal direction calculated by the second normal calculation unit 903 can be brought closer to the true normal direction at the position of interest of the target object, and the target can be obtained. Even if the surface of the object is rough, the normal direction can be calculated with high accuracy.

In the present embodiment, the case where the calculated second normal direction is set to the normal direction at the position of interest of the target object has been described, but the present invention is not limited to this case, and the third normal direction or a plurality of normal directions is described. The normal direction of the second time may be the normal direction at the position of interest of the target object.
For example, the second normal calculation unit 903 calculates the third normal direction by performing the same processing as in S1301 to S1306 using the second normal direction for each position of the target object calculated in S1305. However, the third normal direction may be the normal direction at the position of interest of the target object. That is, in S1303, the second normal calculation unit 903 determines the direction opposite to the previously calculated second normal direction as the direction of the incident light (fifth direction), and sets the direction of the incident light. In S1304, the second normal calculation unit 903 calculates the direction (sixth direction) of the peak of the reflected light in the set direction of the incident light. In S1305, the second normal calculation unit 903 calculates the direction of the bisector from the direction of the incident light and the direction of the peak of the reflected light. In S1306, the second normal calculation unit 903 outputs the direction of the bisection line calculated for each position of the target object as the third normal direction. The second normal calculation unit 903 similarly repeats S1301 to S1306 to obtain the (N + 1) normal direction using the Nth normal direction for each position of the target object calculated in S1305. It may be calculated and the normal (N + 1) normal direction may be set as the normal direction at the position of interest of the target object. By repeating the process of calculating the normal direction three times or more in this way, the normal direction of the target object can be calculated with higher accuracy.

[Second Embodiment]
In the present embodiment, a normal calculation system capable of calculating the normal direction of the target object with higher accuracy by determining the second measurement condition based on the first measurement will be described. Since the configuration of the normal calculation system and the functional configuration of the information processing device in the second embodiment are the same as those in the first embodiment, the description thereof will be omitted, and the points different from those in the first embodiment will be mainly described.

FIG. 14 is a block diagram showing a detailed functional configuration of the image acquisition control unit 506 and the normal calculation unit 507 according to the second embodiment. The image acquisition control unit 506 includes a first condition setting unit 1401, a first image acquisition unit 1402, a second condition setting unit 1403, and a second image acquisition unit 1404. Further, the normal calculation unit 507 includes a first normal angle reflection characteristic calculation unit 1405, a first normal calculation unit 1406, a lighting position calculation unit 1407, a second variable angle reflection characteristic calculation unit 1408, and a second method. It has a line calculation unit 1409.

FIG. 15 is a flowchart showing an example of processing by the information processing apparatus 307.
In S1501, the first condition setting unit 1401 receives condition settings such as an imaging condition of the imaging device 303, a stage moving condition of the stage 304, and a lighting condition of the lighting device 305 from the user via the UI management unit 508, and the first condition setting unit 1401 receives the condition setting. Set as the condition of 1. In the first embodiment, the position of the image pickup device 303 and the position of each light source of the lighting device 305 are changed to take an image, but in the present embodiment, the lighting device 305 is turned on in order to calculate the first normal direction. The position of the light source to be used is set to one, and the position of the image pickup device 303 is changed to perform imaging. Therefore, the imaging conditions and the stage movement conditions set here are the same as those in the first embodiment, but the lighting conditions are set for one light source to be turned on. The position of the light source to be lit is arbitrary.

In S1502, the first image acquisition unit 1402 acquires the first image based on the first condition set in S1501. Specifically, the first image acquisition unit 1402 changes the position of the image pickup device 303 via the stage 304, and performs irradiation by the illumination device 305 on the target object 301 and imaging by the image pickup device 303. A series of processes for acquiring the image of 1 is executed.
In S1503, the first variable angle reflection characteristic calculation unit 1405 calculates the first variable angle reflection characteristic information based on the first image by the same processing as the flowchart of FIG. 11 of the first embodiment.
In S1504, the first normal calculation unit 1406 calculates the first normal information based on the first variable angle reflection characteristic information calculated in S1503. The specific processing of S1504 will be described later.

In S1505, the lighting position calculation unit 1407 calculates the lighting position based on the first normal information. The specific processing of S1505 will be described later.
In S1506, the second condition setting unit 1403 receives the imaging condition of the imaging device 303 and the stage moving condition of the stage 304 from the user via the UI management unit 508, and together with the lighting condition of the lighting device 305 calculated in S1505 and the like. , Set as the second condition.
In S1507, the second image acquisition unit 1404 acquires the second image based on the second condition set in S1506. Specifically, the second image acquisition unit 1404 changes the position of the image pickup device 303 via the stage 304, and performs irradiation by the illumination device 305 on the target object 301 and imaging by the image pickup device 303. A series of processes for acquiring the image of 2 is executed.

In S1508, the second variable angle reflection characteristic calculation unit 1408 calculates the second variable angle reflection characteristic information by the same processing as in S1503 based on the second image.
In S1509, the second normal calculation unit 1409 calculates the second normal information based on the second variable angle reflection characteristic information calculated in S1508. The specific processing of S1509 will be described later.
In S1510, the UI management unit 508 displays the second normal information calculated in S1509 as the normal information of the target object on the display 414, and ends the process. In S1510, the UI management unit 508 may directly store the entire image of the target object in a recording medium such as HDD 412 without displaying it on the display 414.

FIG. 16 is a flowchart showing a process of calculating the first normal information in S1504 of FIG. In this embodiment, one light source that lights up as described above is set.
In S1601, the first normal calculation unit 1406 reads the first variable angle reflection characteristic information calculated in S1503.
In S1602, the first normal calculation unit 1406 sets the position (attention position) of the target object and repeats the processes of S1603 and S1604 for each position.
In S1603, the first normal calculation unit 1406 reads the reflectance information for each direction of the reflected light at the position (attention position) of the target object set in S1602 from the first variable angle reflection characteristic information read in S1601. Extract. The first normal calculation unit 1406 calculates the direction of the peak of the reflected light (second direction) from the reflectance information for each direction of the extracted reflected light.

In S1604, the first normal calculation unit 1406 calculates the direction of the bisector from the direction of the incident light and the direction of the peak of the reflected light calculated in S1603. When calculating the direction of the bisector, the first normal calculation unit 1406 normalizes so that the norm of the calculation direction is 1.
Next, the first normal calculation unit 1406 returns to S1602, changes the position of the target object, and performs the processes of S1603 and S1604. The first normal calculation unit 1406 performs the processes of S1603 and S1604 at all positions of the target object.
In S1605, the first normal calculation unit 1406 outputs the direction of the bisection line calculated in S1604 for each position of the target object as the first normal direction. The first normal calculation unit 1406 stores the first normal information, which is a collection of information in the first normal direction for each position of the target object, in a recording medium such as HDD 412.

FIG. 17 is a flowchart showing a process of calculating the lighting position in S1505 of FIG.
In S1701, the lighting position calculation unit 1407 reads the first normal information stored in S1605.
In S1702, the lighting position calculation unit 1407 performs the processing of S1703 for each position of the target object.
In S1703, the lighting position calculation unit 1407 calculates the coordinates of the intersection where the first normal direction and the hemisphere of the optical fixing jig 306 intersect.
Next, the lighting position calculation unit 1407 returns to S1702, changes the position of the target object, and performs the process of S1703. The lighting position calculation unit 1407 performs S1703 at all positions of the target object.
In S1704, the lighting position calculation unit 1407 calculates the average intersection coordinates by calculating the average value of the calculated intersection coordinates.
In S1705, the lighting position calculation unit 1407 determines the conditions for calculating the second normal direction based on the first normal direction calculated by the first normal calculation unit 1406. Specifically, the lighting position calculation unit 1407 identifies the light source of the lighting device 305 closest to the average intersection coordinates, determines the light source position to be lit, and outputs the light source position information to the second condition setting unit 1403. .. The direction (third direction) of the incident light incident from this light source position corresponds to the opposite direction 205 of FIG. 2B described above.

FIG. 18 is a flowchart showing a process of calculating the second normal information in S1509 of FIG.
In S1801, the second normal calculation unit 1409 reads the second variable angle reflection characteristic information calculated in S1508.
In S1802, the second normal calculation unit 1409 repeats the processes of S1803 and S1804 for each position (position of interest) of the target object.
In S1803, the second normal calculation unit 1409 extracts the reflectance information for each direction of the reflected light at the position (attention) of the target object set in S1702 from the second variable angle reflection characteristic information read in S1801. do. The second normal calculation unit 1409 calculates the direction of the peak of the reflected light (fourth direction) from the reflectance information for each direction of the extracted reflected light.
In S1804, the second normal calculation unit 1409 calculates the direction of the bisector from the direction of the incident light and the direction of the peak of the reflected light calculated in S1803. When calculating the direction of the bisector, the second normal calculation unit 1409 standardizes so that the norm of the calculation direction is 1.
Next, the second normal calculation unit 1409 returns to S1802, changes the position of the target object, and performs the processes of S1803 and S1804. The second normal calculation unit 1409 performs the processes of S1803 and S1804 at all the positions of the target object.
In S1805, the second normal calculation unit 1409 outputs the direction of the bisection line calculated in S1804 for each position of the target object as the second normal direction. The second normal calculation unit 1409 stores the second normal information, which is the aggregate of the information in the second normal direction for each position of the target object, in a recording medium such as HDD 412.

As described above, according to the present embodiment, the lighting position calculation unit 1407 calculates the second normal direction at the position of interest of the target object based on the first normal direction calculated by the first normal calculation unit 1406. Determine the conditions for doing so. Therefore, the second normal direction calculated by the second normal calculation unit 903 can be made closer to the true normal direction at the position of interest of the target object, and the surface of the target object is high even if it is a rough surface. The normal direction can be calculated accurately.
Further, since the second image acquisition unit 1404 acquires the second image based on the conditions determined based on the first normal direction, the number of images to be acquired can be reduced and the imaging speed can be increased. Can be done.

In the present embodiment, the case where the position of the light source to be lit by the lighting device 305 is fixed, the position of the image pickup device 303 is changed, and the normal direction is calculated using the captured image has been described. Not limited. On the contrary, the normal direction may be calculated using the image obtained by fixing the position of the image pickup device 303 and changing the position of the light source to be lit by the lighting device 305. That is, in S1502 and S1507, the position of the image pickup apparatus 303 may be fixed, and the position of the light source to be lit by the illumination apparatus 305 may be changed to perform imaging. By doing so, the mechanical movement by the image pickup apparatus 303 can be reduced, so that the image pickup can be speeded up.

[Other Embodiments]
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or recording medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

Of the above-mentioned processing units, the normal calculation unit 507 and the like may be processed by using a machine-learned trained model instead. In that case, for example, a plurality of combinations of input data and output data to the processing unit are prepared as training data, knowledge is acquired from them by machine learning, and output data for the input data is obtained based on the acquired knowledge. Generates a trained model that outputs as a result. The trained model can be constructed by, for example, a neural network model. Then, the trained model performs the processing of the processing unit by operating in collaboration with the CPU, GPU, or the like as a program for performing the same processing as the processing unit. The trained model may be updated after a certain process if necessary.

Although the present invention has been described above with various embodiments, the present invention is not limited to these embodiments, and changes can be made within the scope of the present invention, and the above-described embodiments can be combined in a timely manner. You may.

307: Information processing device 507: Normal calculation unit 901: Variable angle reflection characteristic calculation unit 902: First normal calculation unit 903: Second normal calculation unit 1405: First variable angle reflection characteristic calculation unit 1406: 1st normal calculation unit 1407: Lighting position calculation unit 1408: 2nd variable angle reflection characteristic calculation unit 1409: 2nd normal calculation unit

Claims (10)

An acquisition means for acquiring the reflection characteristics of the target object based on an image obtained by capturing the target object, and
Based on the acquired reflection characteristics, the first direction in which the incident light is incident on the target object, and the second direction of the peak of the reflected light in the target object with respect to the incident light incident from the first direction. A first calculation means for calculating the first normal direction of the position of interest in the target object, and
A determination means for determining a condition for calculating the second normal direction at the attention position based on the calculated first normal direction, and a determination means.
An information processing device characterized by having. The object is based on the acquired reflection characteristics, a third direction based on the determined conditions, and a fourth direction of a peak of reflected light in the target object with respect to incident light incident from the third direction. The information processing apparatus according to claim 1, further comprising a second calculation means for calculating the second normal direction of the position of interest in the object. The second calculation means is
The information processing apparatus according to claim 2, wherein the direction opposite to the first normal direction is the third direction. The second calculation means is
The fourth direction is calculated based on the acquired reflection characteristics and the third direction.
The information processing apparatus according to claim 2 or 3, wherein the direction of the bisection line between the third direction and the fourth direction is calculated as the second normal direction of the position of interest in the target object. The second calculation means is
The information processing apparatus according to any one of claims 2 to 4, wherein the second normal direction is output as the normal direction of the position of interest in the target object. The determination means is
Based on the Nth normal direction calculated by the iterative processing by the second calculation means, the conditions for calculating the (N + 1) normal direction at the attention position are determined.
The second calculation means is
The information processing apparatus according to any one of claims 2 to 4, wherein the first (N + 1) normal direction is output as the normal direction of the position of interest in the target object. The first calculation means is
The second direction is calculated based on the acquired reflection characteristics and the first direction.
The invention according to any one of claims 1 to 6, wherein the direction of the bisection line between the first direction and the second direction is calculated as the first normal direction of the position of interest in the target object. Information processing equipment. The acquisition means
The information processing apparatus according to any one of claims 1 to 7, further comprising a processing means for separating a diffuse reflection component and a surface reflection component. The acquisition step of acquiring the reflection characteristics of the target object based on the image obtained by capturing the target object, and
Based on the acquired reflection characteristics, the first direction in which the incident light is incident on the target object, and the second direction of the peak of the reflected light in the target object with respect to the incident light incident from the first direction. The first calculation step of calculating the first normal direction of the position of interest in the target object, and
A determination step for determining a condition for calculating the second normal direction at the attention position based on the calculated first normal direction, and a determination step.
An information processing method characterized by having. A program for causing a computer to function as each means of the information processing apparatus according to any one of claims 1 to 8.

Images