JP2021081241A - Image processing device, image processing method, and program - Google Patents

Abstract

To provide an image processing device and an image processing method which suppress deterioration in accuracy of acquiring a bidirectional reflectance property.SOLUTION: An image processing device for calculating a bidirectional reflectance property of an object includes: acquisition means for acquiring a taken image of the object, which contains a plurality of channels; and calculation means for calculating a bidirectional reflectance property corresponding to a target channel among the plurality of channels on the basis of a bidirectional reflectance property corresponding to a channel different from the target channel.SELECTED DRAWING: Figure 5

Description

The present invention relates to an image processing technique for obtaining reflection characteristics of an object.

In the field of computer graphics (CG) that expresses the material of an object and the texture of paint, the bidirectional reflectance distribution function BRDF that expresses the eccentric reflection characteristic that changes depending on the illumination direction and observation direction, or the two-dimensional eccentric reflection characteristic The measurement data of SVBRDF representing the above is used. BRDF is an abbreviation for Bidirectional Reflectance Distribution Function. SVBRDF is an abbreviation for Spatially Variing Bidirectional Reflectance Distribution. Patent Document 1 discloses a measuring device that acquires measurement data indicating the reflected light intensity of an object under measurement conditions set based on either or both of a light projection angle and a light reception angle.

JP-A-2017-78628

When measuring the variable angle reflection characteristic using a camera, the measuring device disclosed in Patent Document 1 determines the quality of measurement conditions based on the SN ratio of the captured image and determines the number of measurements. However, since the measurement range of the camera is not taken into consideration, it may not be possible to suppress a decrease in measurement accuracy due to saturation of pixel values.

The present invention has been made in view of the above problems, and an object of the present invention is to provide a process for suppressing a decrease in acquisition accuracy of variable angle reflection characteristics.

The image processing device according to the present invention is an image processing device that calculates the variable angle reflection characteristic of an object, and includes an acquisition means for acquiring an captured image of the object including a plurality of channels, and the plurality of channels. Among them, it is characterized by having a calculation means for calculating the eccentric reflection characteristic corresponding to the target channel based on the eccentric reflection characteristic corresponding to a channel different from the target channel.

According to the present invention, it is possible to suppress a decrease in the acquisition accuracy of the variable angle reflection characteristic.

It is a figure which shows an example of the hardware configuration of an image processing apparatus. It is a figure which shows an example of the functional structure of the image processing apparatus of 1st Embodiment. It is a flowchart which shows an example of the processing of the image processing apparatus of 1st Embodiment. It is a figure which shows an example of the structure of the acquisition device. It is a figure which shows an example of the measurement data of each channel. It is a figure for demonstrating the variable angle reflection characteristic estimated in 1st Embodiment. It is a figure which shows an example of the functional structure of the image processing apparatus of 2nd Embodiment. It is a flowchart which shows an example of the processing of the image processing apparatus of 2nd Embodiment. It is a figure for demonstrating various measurement data in 2nd Embodiment.

A mode for carrying out the present invention will be described with reference to the drawings. However, the components described in this embodiment are merely examples, and are not intended to limit the scope of the present invention to them.

[First Embodiment]
<Hardware configuration of image processing device>
FIG. 1 is a diagram showing an example of the hardware configuration of the image processing device 1 according to the present embodiment. The image processing device 1 is composed of, for example, a computer 11. The CPU 101 uses the RAM 103 as a work memory to execute an OS (operating system) and various programs stored in a ROM 102, an HDD (hard disk drive) 17, and the like. Further, the CPU 101 controls each configuration via the system bus 107. The process according to the flowchart described later is executed by the CPU 101 after the program code stored in the ROM 102 or the HDD 17 is expanded into the RAM 103. The general-purpose I / F (interface) 104 is a serial bus interface such as USB, and an input device 13 such as a mouse or a keyboard is connected via the serial bus 12. The SATA (Serial ATA) I / F 105 is a serial bus interface, and a general-purpose drive 18 for reading and writing HDD 17 and various recording media is connected via the serial bus 16. The CPU 101 uses various recording media mounted on the HDD 17 or the general-purpose drive 18 as a storage location for various data. The VC (video card) 106 is a video interface to which the display 15 is connected. The CPU 101 displays a UI (user interface) provided by the program on the display 15 and receives an input such as a user instruction received via the input device 13.

<Functional configuration of image processing device>
FIG. 2 is a diagram showing an example of the functional configuration of the image processing device 1. The CPU 101 functions as the functional configuration shown in FIG. 2 by reading and executing a program stored in the ROM 102 or the HDD 17 using the RAM 103 as a work memory. It should be noted that it is not necessary that all of the processes shown below are executed by the CPU 101, and even if the image processing device 1 is configured so that a part or all of the processes are performed by one or a plurality of processing circuits other than the CPU 101. Good.

The image processing device 1 stores captured image acquisition unit 201, imaging condition acquisition unit 202, image determination unit 203, first variable angle reflection characteristic calculation unit 204, second variable angle reflection characteristic calculation unit 205, and data storage. It has a part 206.
The captured image acquisition unit 201 acquires a plurality of captured images (captured image group). The captured image group is an RGB color image obtained by capturing an object (hereinafter referred to as a sample) to be measured under different imaging conditions. The imaging condition acquisition unit 202 acquires the imaging conditions corresponding to each captured image. The image determination unit 203 refers to the captured image group, calculates the reliability for each RGB channel, and determines the quality of each RGB channel based on the reliability.

The first variable angle reflection characteristic calculation unit 204 performs estimation processing of the variable angle reflection characteristic based on the pixel value of each image in the channel that is judged to be good by the image determination unit 203 and the imaging condition. The second variable angle reflection characteristic calculation unit 205 includes pixel values of each image in the channel rejected by the image determination unit 203, imaging conditions, and other channels obtained by the first variable angle reflection characteristic calculation unit 204. Based on the variable angle reflection characteristic of, the angled reflection characteristic is estimated.

The data storage unit 206 is composed of various recording media mounted on the HDD 17 and the general-purpose drive 18. The data storage unit 206 stores the captured image or information such as the variable angle reflection characteristic calculated by the first variable angle reflection characteristic calculation unit 204 and the second variable angle reflection characteristic calculation unit 205. Detailed processing in each part will be described later.

<Operation of image processing device>
FIG. 3 is a flowchart showing an example of processing of the image processing device 1 in the present embodiment. Hereinafter, the processing of the image processing apparatus 1 will be described in detail with reference to FIG. The process shown in the flowchart is executed by the CPU 101 after the program code stored in the ROM 102 is expanded in the RAM 103. Further, the process shown in the flowchart starts when the user operates the input device 13 to input an instruction and the CPU 101 receives the input instruction.

In S301, the captured image acquisition unit 201 acquires a plurality of captured images (captured image group) from the data storage unit 206. It is assumed that the captured image group is recorded in advance in the data storage unit 206. A method of acquiring a group of captured images recorded in the data storage unit 206 will be described with reference to FIG.

FIG. 4 is a diagram schematically showing the configuration of the acquisition device 4 for acquiring the captured image group and the imaging conditions. The acquisition device 4 has a lighting device 410. The lighting device 410 can arrange a plurality of light sources 412 on the control board 411 and individually control the lighting of each light source 412. The illuminating device 410 illuminates the sample 420 installed on the sample table 430. The acquisition device 4 has an image pickup device 440. The image pickup apparatus 440 uses, for example, a 3-band RGB camera having a 1000 × 1000 pixel CCD image sensor. The acquisition device 4 turns on one of the light sources 412 mounted on the lighting device 410, images the sample surface irradiated by the turned on light source 412 with the image pickup device 440, and then turns off the light source 412. The light source 412 to be lit is changed and the execution is performed sequentially. As a result, the image pickup apparatus 440 records the reflected light on the sample surface when light is incident from different directions at the zenith angle 450 as an image pickup image. Each recorded image is an RGB color image in TIF format, and an RGB value (0-65535) normalized to 16 bits is stored in each pixel. It is assumed that the pixel value of the captured image has a linearity with the radiance value of the imaged object. It is also possible to acquire the two-dimensional distribution of the variable angle reflection characteristic by calculating the variable angle reflection characteristic described later for each pixel position of the captured image group, but in the present embodiment, in order to simplify the explanation, it is possible to obtain the two-dimensional distribution in advance. Only one pixel corresponding to a predetermined predetermined pixel position will be described as a processing target. In general, the incident illuminance on the sample surface differs depending on the distance between the light source and the subject or the orientation characteristic of the light source. Therefore, it is desirable to correct the pixel value of each image by the following equation using the captured image obtained by imaging the complete diffuse reflector under the same conditions as the imaging condition of the sample surface.

Here, S _i (x, y, b) is the pixel value of the channel b in the _{captured image S i of the sample.} Similarly, R _i (x, y, b) is the pixel value in the captured image R _i of perfectly diffuse reflector. Further, I _i (x, y, b) is a pixel value of _{the image I i corrected by (Equation 1).} Further, c is a fixed value, and i is an identification number unique to the light source that is lit at the time of imaging. As described above, the coordinates (x, y) indicating the pixel position are fixed values that are predetermined. It is assumed that the image pickup apparatus 440 has a telecentric optical system having a telecentric lens, and the light receiving direction is uniform in each pixel. Further, the distance from the light source to the sample surface is sufficiently large, and the incident direction can be regarded as uniform at each coordinate of the sample surface corresponding to each pixel in the imaging region.

In S302, the imaging condition acquisition unit 202 acquires the imaging conditions of each captured image acquired in S301 from the data storage unit 206. The imaging conditions are the zenith angles of the direction of the incident light incident on the sample surface (hereinafter, the incident direction) and the direction in which the reflected light reflected by the sample surface is received by the imaging device (hereinafter, the light receiving direction). θ and each azimuth angle φ. Specifically, the imaging conditions are a zenith angle θ _{i in} the incident direction, an azimuth φ _{i in} the incident direction, a zenith angle θ _{o in} the light receiving direction, and an azimuth φ _o in the light receiving direction. Note that FIG. 4 shows the zenith angle θ _{i in the incident direction, the zenith angle θ o in} the light receiving direction, and the normal line L of the sample surface of the sample 420. Imaging conditions are stored in an image S _i acquired in S301, the four corresponding relationship between the angle are described LUT (Look Up Table) format. In the present embodiment, as described with reference to FIG. 4, only _{the zenith angle θ i in the incident direction differs between the captured images among the acquired imaging conditions.}

Above, the processing of S301～S302, zenith angle theta _i of the incident direction, the azimuth angle phi _i of the incident direction, zenith angle theta _o of the light receiving direction, the imaging conditions of the azimuth angle phi _o of the light receiving direction, the coordinates of the sample surface A pixel value based on the reflection characteristic of (x, y) can be obtained for each channel b. Hereinafter, the pixel value with the channel b and the zenith angle θ _i in the incident direction as variables is referred to as S (b, θ _i ). In this embodiment, the fixed values (x, y), the azimuth φ _{i in} the incident direction, the zenith angle θ _{o in the} light receiving direction, and the azimuth φ _o are omitted. Further, the pixel value S (b, θ _i ) is referred to as measurement data.

FIG. 5 is a diagram showing an example of measurement data. Here, the horizontal axis of the graph is the zenith angle θ _i in the incident direction, and the vertical axis is the pixel value S (b, θ _i ). The plots of the graphs of FIGS. 5A, 5B, and 5C correspond to the measurement data of R channel, G channel, and B channel, respectively. In the processes of S303 to S305 described later, the measurement data is referred to, and the parameters of the model function that approximates the measurement data of each channel are derived. Hereinafter, the variable angle reflection characteristic shall _{refer to the equation f (b, θ i} ) which approximates the measurement data. Further, in the graph shown in FIG. 5, the equation f (b, θ _i ) is represented by a broken line.

In S303, the image determination unit 203 determines the quality of each RGB channel. The image determination unit 203 calculates the reliability of the measurement data for each channel, and the channel having the measurement data with the highest reliability is regarded as good, and the other channels are rejected. Here, the reliability T (b) of the channel b is derived by the following equation.

Here, k is determined by the following equation.

According to (Equation 2) and (Equation 3), the measurement data of the channel having a small number of saturated pixel values and a large number of pixel values having a large value has higher reliability. On the other hand, the measurement data of a channel having many saturated pixel values and many small pixel values has lower reliability. It is highly possible that a channel that is judged to be good has a large amount of data in which information is less likely to be lost due to saturation and the influence of noise is relatively small. For example, when the measurement data shown in FIGS. 5A to 5C are obtained for each of the RGB channels, the measurement data of the R channel shown in FIG. 5A is determined to be good. On the other hand, the measurement data of the G channel shown in FIG. 5 (b) and the measurement data of the B channel shown in FIG. 5 (c) are determined to be negative.

The determination method for each channel is not limited to the above method, and any one or more channels may be used for good determination. For example, the image determination unit 203 may count the number of non-saturated pixel values for each channel to obtain the reliability. Further, the image determination unit 203 may make a pass / fail determination by comparing a predetermined threshold value with the reliability without performing comparison between channels.

In S304, the first variable angle reflection characteristic calculation unit 204 estimates the variable angle reflection characteristic of the channel D1 which is judged to be good. The estimated variable angle reflection characteristic is defined by the following equation.

(Equation 4) is a normal distribution function of _{the variance σ D1} and the mean m _{D1 with a D1} as the weighting coefficient. These three parameters are optimized by using a known gradient descent method so that the cumulative square error between each pixel value of the channel D1 judged to be good and the estimated value by (Equation 4) is minimized. Derived by. The cumulative squared error, which is the objective function, does not include the error between the saturated pixel value and the estimated value. The variable angle reflection characteristic derived by this process is shown by the broken line 510 in FIG. 5 (a). In addition to the normal distribution, a known model formula such as a Cook-Torrance model may be used as the model function. Hereinafter, the above-mentioned variable angle reflection characteristic derived by referring only to the measurement data of the channel to be estimated is referred to as a first variable angle reflection characteristic.

In S305, the second variable angle reflection characteristic calculation unit 205 estimates the variable angle reflection characteristic of the channel D2 that has been determined to be negative. The estimated variable angle reflection characteristic is defined by the following equation.

(Equation 5) is a normal distribution function similar to S304, but uses _{the value m D1 derived in S304 for the average, which is one of the parameters.} Therefore, the second variable angle reflection characteristic calculation unit 205 derives only the two parameters of _{the weighting coefficient a D2} and the variance σ _{D2 by the optimization process.} In the derivation, as in S304, an optimization process using a known gradient descent method is performed so that the cumulative square error between each pixel value of the channel D2 determined to be rejected and the estimated value by the model formula is minimized. Derived by The cumulative squared error, which is the objective function, does not include the error between the saturated pixel value and the estimated value. Hereinafter, the above-mentioned variable angle reflection characteristic derived by referring to the measurement data of the channel to be estimated and the parameters obtained by referring to the measurement data of the other channel will be referred to as a second variable angle reflection characteristic.

As described above, according to the processes of S304 to S305, by reducing the number of parameters to be derived, it is possible to suppress a decrease in estimation accuracy even when saturated pixel values are included. The estimation accuracy will be described with reference to FIG.
FIG. 6 is a diagram showing an example of measurement data. In the graph shown in FIG. 6, the horizontal axis is the zenith angle θ _i in the incident direction and the vertical axis is the pixel value S (b, θ _i ) as in FIG. Further, the plot in the graph shown in FIG. 6 (a) is the measurement data of the channel judged to be good, and the plot in the graph shown in FIG. 6 (b) has many saturated pixel values, and the channel judged to be negative. It is the measurement data of. Here, in order to facilitate the qualitative grasp of the estimation accuracy, the broken line in each graph will be described as the true value of the variable angle reflection characteristic of the sample. In the measurement data illustrated in FIG. 6A, as shown in the figure, there is an error from the true value depending on the camera noise and the model accuracy. High robustness can be obtained against errors. As a result, even in the processing of S304 using only the measurement data of the target channel, the first variable angle reflection characteristic close to the true variable angle reflection characteristic can be estimated as shown by the solid line in the figure. .. When the information near the peak is lost due to the saturation of the pixel value as in the measurement data illustrated in FIG. 6B, the robustness against the error is reduced in the processing of S304 using only the measurement data of the target channel. To do. As a result, the estimation accuracy of the first variable angle reflection characteristic is lowered, and as shown by the solid line in FIG. 6B, the first variable angle reflection characteristic having a large error from the true value is estimated. On the other hand, in the processing of S305, the average value corresponding to the peak direction obtained by referring to the measurement data of the most reliable channel is used as a constraint condition for parameter derivation. In general, the peak direction of the variable angle reflection characteristic largely depends on the normal vector of the sample surface, so that the peak directions can be regarded as the same even when the channels are different. Therefore, in the processing of S305 with reference to the highly reliable measurement data of another channel, the second variable angle reflection characteristic closer to the true value can be estimated as shown by the solid line in FIG. 6 (c). As described above, the image processing apparatus 1 refers to the more reliable estimation result of the other channel when estimating the variable angle reflection characteristic corresponding to each RGB channel. Therefore, the number of parameters to be estimated can be reduced, and a decrease in estimation accuracy can be suppressed.

As described above, the image processing apparatus 1 calculates the angled reflection characteristic of the target channel based on the angled reflection characteristic of the other channel among the plurality of channels. Therefore, even when the reliability of the measurement data of the target channel is low, it is possible to suppress a decrease in the accuracy of the calculated variable angle reflection characteristic.
Further, the image processing device 1 calculates the first variable angle reflection characteristic based on the pixel value of the first channel of the captured image, and sets the pixel value of the second channel of the captured image and the first variable angle reflection characteristic. Based on this, the second variable angle reflection characteristic is calculated. Therefore, even when the reliability of the measurement data of the second channel is low, it is possible to suppress a decrease in the accuracy of the second variable angle reflection characteristic.

[Second Embodiment]
In the first embodiment, when estimating the eccentric reflection characteristic of any of the RGB channels at the target pixel position (hereinafter, coordinates), the eccentric reflection characteristic of the other channel on the same coordinate was referred to. In the present embodiment, when estimating the angled reflection characteristic of any of the RGB channels of the target coordinates, the angled reflection characteristic is estimated using the angled reflection characteristic on the other coordinates. As a result, it is possible to suppress a decrease in estimation accuracy even when the angled reflection characteristics differ greatly between channels having the same coordinates.

<Functional configuration of image processing device>
FIG. 7 is a diagram showing an example of the functional configuration of the image processing device 2.
The image processing device 2 stores captured image acquisition unit 701, imaging condition acquisition unit 702, image determination unit 703, first variable angle reflection characteristic calculation unit 704, second variable angle reflection characteristic calculation unit 705, and data storage. It has a part 706 and a part 706. The captured image acquisition unit 701 and the imaging condition acquisition unit 702 have the same functions as the captured image acquisition unit 201 and the imaging condition acquisition unit 202 in the first embodiment, and measure each RGB channel in each pixel in the captured image. Get the data. The image determination unit 703 has the same function as the image determination unit 203 in the first embodiment, and determines the quality of each RGB channel in each pixel. The first variable angle reflection characteristic calculation unit 704 has the same function as the first variable angle reflection characteristic calculation unit 204 in the first embodiment, and the first variable angle reflection of the channel that is judged to be good by the image determination unit 703. The characteristics are calculated for each pixel.

The second variable angle reflection characteristic calculation unit 705 includes a similar pixel search unit 7051, a data synthesis unit 7052, and a parameter derivation unit 7053, and calculates the variable angle reflection characteristic of the channel rejected by the image determination unit 203. The similar pixel search unit 7051 is based on the first variable angle reflection characteristic calculated by the first variable angle reflection characteristic calculation unit 704, and is a pixel (similar pixel) having other coordinates similar to the first variable angle reflection characteristic. Search for a pair. The data synthesis unit 7052 synthesizes the measurement data of the channel rejected by the image determination unit 703 between the similar pixels. The parameter derivation unit 7053 refers to the measurement data synthesized by the data synthesis unit 7052, and derives the variable angle reflection characteristic of the channel rejected by the image determination unit 703. The data storage unit 706 has the same function as the data storage unit 206 in the first embodiment, and holds various information.

<Operation of image processing device>
FIG. 8A is a flowchart showing an example of processing of the image processing device 2 in the present embodiment.
In S801, the captured image acquisition unit 701 acquires the captured image from the data storage unit 706 as in the first embodiment. The captured image is an image captured by the acquisition device 4, and the processing acquires the same number (1000 × 1000) of measurement data of each RGB channel as the number of camera pixels corresponding to the image pickup device 440.
In S802, the imaging condition acquisition unit 702 acquires the imaging condition from the data storage unit 706 as in the first embodiment.

In S803, the image determination unit 703 determines the quality of each RGB channel for each pixel, as in the first embodiment.
In S804, the first variable angle reflection characteristic calculation unit 704 calculates the variable angle reflection characteristic by referring to the measurement data of the channel that is judged to be good in each pixel, as in the first embodiment.
In S805, the second variable angle reflection characteristic calculation unit 705 calculates the variable angle reflection characteristic of the channel rejected in each pixel. The process will be described in detail below.

<Processing of the second variable angle reflection characteristic calculation unit>
FIG. 8B is a flowchart for explaining the processing of the second variable angle reflection characteristic calculation unit 705. It is assumed that the series of processes of S811 to S813, which will be described later, is repeated for each pixel.
In S811, the similar pixel search unit 7051 refers to the first variable angle reflection characteristic, and selects a pixel having a similar processing pixel and the first variable angle reflection characteristic. Specifically, the similarity shown in the following equation is calculated for each pixel that is a candidate for similar pixels (hereinafter, candidate pixel), and the candidate pixel having the maximum similarity is used as a similar pixel. The candidate pixel refers to a pixel on a coordinate different from that of the target pixel, and a channel in which the target pixel is judged to be good and a pixel in which a good judgment of the same channel is obtained.

_{Here, a t, m t, σ} t is a parameter indicating a first variable angle reflection characteristics of the processed pixel, weighting coefficients for each normal distribution function, average, corresponds to the dispersion. Similarly, a _s , m _s , and σ _s are parameters indicating the first variable angle reflection characteristic of the candidate pixel. Further, k ₁ , k ₂ , and k ₃ are predetermined weighting coefficients. The method for selecting similar pixels does not need to be limited to the above method as long as it is a method for selecting pixels having similar variable angle reflection characteristics. For example, the pixel having the smallest square error of only one of the parameters defining the first variable angle reflection characteristic may be selected, or the measurement data referred to when estimating the first variable angle reflection characteristic may be directly used. Pixels may be selected by comparison.

In S812, the data synthesizing unit 7052 integrates the measurement data of the target pixel and similar pixels in the same channel in the rejected channel. Here, various measurement data will be described with reference to FIG. An example will be described in which the channel obtained by determining good and the first variable angle reflection characteristic is calculated is defined as the R channel, and the data of the G channel determined to be negative is integrated. Further, the target pixel is referred to as p1 and the similar pixel is referred to as p2.

Graphs 901 to 905 shown in FIG. 9 are graphs in which the horizontal axis is the zenith angle in the incident direction and the vertical axis is the pixel value, and various measurement data are plotted. The graph 901 shows the measurement data of the R channel and the first variable angle reflection characteristic which are judged to be good in the target pixel p1. The graph 902 shows the measurement data of the R channel and the first variable angle reflection characteristic which are judged to be good in the similar pixel p2. Graph 903 represents the measurement data of the G channel that was rejected in the target pixel p1. Graph 904 represents the measurement data of the G channel that was rejected in the similar pixel p2. Here, even if the coordinates are different, if the variable angle reflection characteristics are similar in a specific channel as shown in Graphs 901 and 902, the sample has the same texture between p1 and p2. Therefore, it is highly possible that the other channels have the same variable angle reflection characteristics as shown by the broken lines in the graphs 903 and 904. Therefore, the data synthesis unit 7052 treats each measurement data as data showing the same variable angle reflection characteristic, and generates integrated measurement data as shown in Graph 905.

In S813, the parameter derivation unit 7053 estimates the variable angle reflection characteristic of the rejected channel. The estimation process is equivalent to the process by the first variable angle reflection characteristic calculation unit 704 in S804, but the measurement data to be referred to is the measurement data generated by the data synthesis unit 7052 in S812. As a result, the number of useful data increases, and even when information is lost due to saturation of pixel values or the like, it is highly possible that robustness against noise can be maintained and a decrease in estimation accuracy can be suppressed.

As described above, the image processing apparatus 2 in the present embodiment makes an estimation by referring to the variable angle reflection characteristics of other channels and other coordinates when estimating the variable angle reflection characteristics corresponding to each RGB channel. Expand the number of required data. As a result, the robustness against noise can be improved, and the deterioration of the accuracy of the variable angle reflection characteristic can be suppressed.

[Other Embodiments]
In the above embodiment, three channels of RGB have been described assuming a general color camera, but the processing of the above embodiment is applied to at least two or more channels having different sensitivity characteristics depending on the wavelength. Is possible. In this case, the first variable angle reflection characteristic calculation unit derives the first variable angle reflection characteristic by referring only to the measurement data in the channel having the higher reliability among at least two channels. On the other hand, the second variable angle reflection characteristic calculation unit derives the second variable angle reflection characteristic by referring to the measurement data in the channel having the lower reliability among at least two channels and the first variable angle reflection characteristic. To do.

In the above embodiment, the case of selecting which of the variable angle reflection characteristic calculation units to estimate the variable angle reflection characteristic according to only the measurement data for each channel has been described, but the user instruction obtained via the input device 13 has been described. It may be the case of selecting based on. For example, information indicating the reliability of each pixel may be displayed to the user via the display 15, and the user who refers to the information may individually instruct which variable angle reflection characteristic calculation unit is applied.

In the above embodiment, the pixel value group (measurement data) in which the incident direction is a variable is acquired by using a plurality of captured images, but the measurement data and the acquisition method of the measurement data are not limited to this case. .. For example, when the sample surface has uniform variable angle reflection characteristics, it is possible to acquire a pixel value group in which the light receiving direction is a variable as measurement data by using one captured image. Specifically, the acquisition device 4 provided with the image pickup device 440 having a non-telecentric optical system is used to image the sample surface. As a result, even with one captured image, it is possible to obtain pixel values corresponding to different light receiving directions in each pixel or at least two different pixels.

In the above embodiment, the case where the first variable angle reflection characteristic is always used when estimating the second variable angle reflection characteristic has been described, but whether or not to use the first variable angle reflection characteristic is adaptively determined. You may. For example, when the reliability calculated by the image determination unit is less than a predetermined threshold value, the same estimation method as the first variable angle reflection characteristic may be applied without using the measurement data of similar pixels.

In the second embodiment, the case where the first variable angle reflection characteristic of the similar pixel is always used when estimating the second variable angle reflection characteristic has been described, but the first variable angle reflection characteristic of the similar pixel is used. You may adaptively decide whether or not to do so. For example, if the similarity calculated when selecting similar pixels is equal to or greater than a predetermined threshold value, even if the same estimation method as the first variable angle reflection characteristic is applied without using the measurement data of similar pixels. Good.

Of the above-mentioned processing units, the image determination units 203, 703, the first variable angle reflection characteristic calculation units 204, 704, the second variable angle reflection characteristic calculation units 205, 705, and the like are machine learning instead. The trained model that has been trained may be used instead. In that case, for example, a plurality of combinations of input data and output data to the processing unit are prepared as training data, knowledge is acquired from them by machine learning, and output data for the input data is obtained based on the acquired knowledge. Generates a trained model that outputs as a result. The trained model can be constructed by, for example, a neural network model. Then, the trained model performs the processing of the processing unit by operating in collaboration with the CPU, GPU, or the like as a program for performing the same processing as the processing unit. The trained model may be updated after a certain process if necessary.

The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

Although the present invention has been described above with various embodiments, the present invention is not limited to these embodiments, and changes can be made within the scope of the present invention, and the above-described embodiments can be combined in a timely manner. You may.

201: Captured image acquisition unit 204: First variable angle reflection characteristic calculation unit 205: Second variable angle reflection characteristic calculation unit

Claims (17)

An image processing device that calculates the variable angle reflection characteristics of an object.
An acquisition means for acquiring a captured image of the object including a plurality of channels, and
An image processing apparatus comprising: a calculation means for calculating a variable angle reflection characteristic corresponding to a target channel among the plurality of channels based on a variable angle reflection characteristic corresponding to a channel different from the target channel. .. The calculation means is
A first calculation means for calculating the first variable angle reflection characteristic based on the pixel value of the first channel of the captured image, and
The first aspect of the present invention is characterized in that the second calculation means for calculating the second variable angle reflection characteristic based on the pixel value of the second channel of the captured image and the first variable angle reflection characteristic is provided. The image processing apparatus described. The first calculation means is
The first variable angle reflection characteristic is calculated by using the channel having the smaller saturated pixel value among at least two channels as the first channel.
The second calculation means is
The image processing apparatus according to claim 2, wherein the second channel is used as the second channel, whichever of at least two channels has a large number of saturated pixel values, to calculate the second variable angle reflection characteristic. The first calculation means is
The first variable angle reflection characteristic is calculated by using the channel having the smaller saturated pixel value and the larger pixel value among at least two channels as the first channel.
The second calculation means is
2. The second aspect of the present invention is characterized in that the second variable angle reflection characteristic is calculated by using the channel having a large number of saturated pixel values and a large number of pixel values having a small value among at least two channels as the second channel. Or the image processing apparatus according to 3. The first calculation means is
The first variable angle reflection characteristic is calculated by using the channel having the smallest saturated pixel value among the plurality of channels as the first channel.
The second calculation means is
Any one of claims 2 to 4, wherein the second channel is a channel different from the channel having the smallest saturated pixel value among the plurality of channels, and the second variable angle reflection characteristic is calculated. The image processing apparatus according to. It also has a determination means for determining the reliability for each channel.
The first calculation means is
The first variable angle reflection characteristic is calculated by using the channel having the higher reliability among at least two channels as the first channel.
The second calculation means is
The image processing according to any one of claims 2 to 5, wherein the channel having the lower reliability among at least two channels is used as the second channel to calculate the second variable angle reflection characteristic. apparatus. The acquisition means
Acquire at least two captured images with different imaging conditions,
The calculation means is
The image processing apparatus according to any one of claims 1 to 6, wherein the variable angle reflection characteristic is calculated based on at least two captured images. Further, it has a condition acquisition means for acquiring imaging conditions for each of the two captured images.
The calculation means is
The image processing apparatus according to claim 7, wherein the variable angle reflection characteristic is calculated based on different imaging conditions. The acquisition means
At least one captured image with different imaging conditions for each of the two pixels is acquired, and
The calculation means is
The image processing apparatus according to any one of claims 1 to 8, wherein the variable angle reflection characteristic is calculated based on the two pixels. It further has a condition acquisition means for acquiring the imaging conditions for each of the two pixels.
The calculation means is
The image processing apparatus according to claim 9, wherein the variable angle reflection characteristic is calculated based on different imaging conditions. The imaging conditions are
The image processing apparatus according to any one of claims 7 to 10, further comprising a zenith angle of incident light incident on the object or a zenith angle of reflected light reflected by the object. The imaging conditions are
The image processing apparatus according to any one of claims 7 to 11, further comprising an azimuth angle of incident light incident on the object or an azimuth angle of reflected light reflected by the object. The calculation means is
The image processing apparatus according to any one of claims 1 to 12, wherein the variable angle reflection characteristic is calculated by a normal distribution function. Further having a search means for searching pixel positions having similar variable angle reflection characteristics,
The calculation means is
The image processing apparatus according to any one of claims 1 to 13, wherein the variable angle reflection characteristic is calculated based on the pixel values at similar pixel positions. The search means
The image processing apparatus according to claim 14, wherein a pixel position having the smallest square error of a parameter defining a normal distribution function is selected. It is an image processing method that calculates the variable angle reflection characteristics of an object.
An acquisition step of acquiring a captured image of the object including a plurality of channels, and
An image processing method characterized by having a calculation step of calculating a variable angle reflection characteristic corresponding to a target channel among the plurality of channels based on a variable angle reflection characteristic corresponding to a channel different from the target channel. .. A program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 15.

Images