JP2017215247A - Image processor, image processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To accurately acquire color characteristics of an object.SOLUTION: Disclosed is an image processor for generating a color image representing color distribution of an object, the image processor including: acquisition means that acquires a first reflection image obtained by picking up first reflectance, which represents diffuse reflectance and surface reflectance of the object and a second reflection image obtained by picking up a second reflectance which represents the diffuse reflectance; and generation means that generates the color image based on the first reflection image and the second reflection image acquired by the acquisition means.SELECTED DRAWING: Figure 5

Description

  The present invention relates to a technique for acquiring color characteristics of an object.

  In recent years, there has been a demand for a technique for faithfully acquiring the color characteristics of an object for the purpose of reproducing and preserving artwork such as paintings. As a method of acquiring color characteristics, a method of acquiring light reflected from an object with an imaging device or the like is common. The light reflected from the object can be classified into surface reflected light reflected on the surface of the object and diffuse reflected light that comes out of the surface of the object again after the light that has entered the object is scattered and absorbed. The surface reflected light is a component that reflects in the regular reflection direction with respect to the incident light among all the components of the reflected light, and is recognized as a reflection of light (gloss) and is difficult to recognize as a color. Therefore, as a method for acquiring only diffuse reflection light, Patent Document 1 discloses that a polarizer is installed for each of the light source and the imaging device, and the polarizer of the light source and the polarizer of the imaging device are in an orthogonal Nicols state. A method for obtaining only diffuse reflected light by setting is described.

Japanese Patent Application Laid-Open No. 2014-120791

  In the case of an object having fine irregularities on the surface, the surface reflected light reflected on the object surface is reflected in various directions by the fine irregularities, so even in a direction other than the regular reflection direction, Diffuse reflected light and surface reflected light are included. Such light is scattered light in all directions and is smaller than the amount of light reflected in the regular reflection direction, so it cannot be felt as gloss, but it may be so large that it cannot be ignored with respect to diffusely reflected light. In particular, there are art objects such as oil paintings that have different uneven patterns in one image, and when the reflection characteristic obtained by removing the surface reflection light as in Patent Document 1 is acquired as the color characteristic, the object is obtained. In some cases, colors could not be acquired correctly. Therefore, an object of the present invention is to provide an image processing apparatus that acquires the color characteristics of an object with high accuracy based on information on each position of the object.

  The present invention is an image processing apparatus that generates a color image representing a color distribution of an object, and is obtained by photographing first reflected light representing diffuse reflected light and surface reflected light of the object. Acquisition means for acquiring a first reflection image and a second reflection image obtained by photographing the second reflection light representing the diffuse reflection light; a first reflection image acquired by the acquisition means; An image processing apparatus comprising a generating unit configured to generate the color image based on the two reflected images.

  According to the present invention, it is possible to acquire the color characteristics of an object with high accuracy.

1 is a block diagram illustrating a schematic configuration of an image processing apparatus according to a first embodiment. 1 is a block diagram illustrating a functional configuration of an image processing apparatus according to a first embodiment. 10 is a flowchart of processing executed by the image processing apparatus according to the first embodiment. The figure which shows the optical system in Example 1. Flowchart of color image generation processing in Embodiment 1 The figure which shows the relationship between the surface unevenness | corrugation in the target object handled in Example 1, and reflection characteristics FIG. 6 is a diagram for explaining divided shooting in the second embodiment. Flowchart of processing executed by image processing apparatus according to embodiment 2 Flowchart of color image generation processing in Embodiment 2 FIG. 9 is a block diagram illustrating a functional configuration of an image processing apparatus according to a third embodiment. Flowchart of processing executed by image processing apparatus according to embodiment 3 Flowchart of color image generation processing in Embodiment 3 Flowchart of color image generation processing in Embodiment 4 Block diagram showing a functional configuration of an image processing apparatus in Embodiment 5 Flowchart of processing executed by image processing apparatus in embodiment 5. User interface in embodiment 5 Material map in Example 5 Flowchart of color image generation processing in embodiment 5

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the following embodiments do not limit the present invention, and all combinations of features described below are not necessarily essential as means for solving the present invention. In addition, the same code | symbol is provided about the same component.

[Example 1]
<About schematic configuration of image processing apparatus>
FIG. 1 is a block diagram illustrating a schematic configuration of an image processing apparatus 1 according to the present embodiment. The image processing apparatus 1 includes an input unit 101, a display unit 102, a data storage unit 103, a CPU 104, a ROM 105, a RAM 106, and a communication unit 107. The input unit 101 is a device for inputting data that can accept an instruction from a user, and includes a pointing device such as a mouse and a keyboard. The display unit 102 is a device for displaying a GUI or the like and presenting information to the user, and is, for example, a CRT or a liquid crystal display. The data storage unit 103 is a device for storing data such as image data and programs, and normally a hard disk is used. The CPU 104 is a device for comprehensively controlling all elements constituting the image processing apparatus 1 including the above-described elements, and is involved in all processing of each element. The ROM 105 and RAM 106 provide the CPU 104 with programs, data, work areas, and the like necessary for the processing. When a control program necessary for processing is stored in the data storage unit 103 or the ROM 105, it is read by the RAM 106 and executed by the CPU 104. The communication unit 107 is an interface for performing communication between devices. For example, a known communication method device such as Ethernet, USB, IEEE, Bluetooth, or the like is used. When the image processing apparatus 1 receives a program via the communication unit 107, the program is recorded in the data storage unit 103 and then read and executed in the RAM 106, or directly read from the communication unit 107 into the RAM 106 and executed. The Although the image processing apparatus 1 has various elements other than the above-described ones, the description thereof is omitted because it is not the main point of the present invention.

<Functional configuration of image processing apparatus>
FIG. 2 is a block diagram illustrating a functional configuration of the image processing apparatus 1 according to the present embodiment. The image processing apparatus 1 includes a diffusion / surface reflection image acquisition unit 201, a diffuse reflection image acquisition unit 202, an image holding unit 203, a color image generation unit 204, and an image output unit 205. The diffusion / surface reflection image acquisition unit 201 acquires a diffusion / surface reflection image from the image holding unit 203. A diffusion / surface reflection image is an image obtained by capturing light including diffuse reflection light and surface reflection light of an object by photographing the object. The diffuse reflection image acquisition unit 202 acquires a diffuse reflection image from the image holding unit 203. A diffuse reflection image is an image obtained by acquiring diffuse reflection light of an object by photographing the object. The diffuse / surface reflection image and the diffuse reflection image are acquired using an optical system (see FIG. 4) described later, and are held by the image holding unit 203. Note that when holding the diffusion and surface reflection image and a diffuse reflection image by the image holding unit 203, for either you obtain these images in advance how to later. The color image generation unit 204 distributes the color of the target object based on the diffusion / surface reflection image acquired by the diffusion / surface reflection image acquisition unit 201 and the diffuse reflection image acquired by the diffusion reflection image acquisition unit 202. A color image representing is generated. Although the appearance of the object differs depending on the positional relationship between the observer and the object, the color of the object represented by the color information included in the color image in this specification refers to the object in front. It is a color when observed from (0 degree direction). The image output unit 205 outputs an image based on the color image generated by the color image generation unit 204.

<About processing executed by image processing apparatus>
Hereinafter, processing executed by the image processing apparatus 1 in the present embodiment will be described. However, in this embodiment, since the image held in the image holding unit 203 is an image acquired in advance using a predetermined optical system, this optical system will be described before explaining the processing of the image processing apparatus 1. This will be described with reference to FIG.

  As shown in FIG. 4, an object 401 that is a measurement (colorimetry) target is fixed on a sample stage 402. The light source 403 projects light onto the object 401. As the light source 403, for example, a single-plate monochrome DLP projector using an LED light source with 640 × 480 pixels can be used, but the invention is not limited to this. Any object can be used as the light source 403 as long as the object 401 can be irradiated with light. The imaging devices 404 and 405 are devices including a DSLR (Digital Single Lens Reflex camera) or the like for photographing the object 401. As the imaging devices 404 and 405, for example, a combination of a DSLR having a CMOS area sensor with 8688 × 5792 pixels and a macro lens with a focal length of 100 mm can be used, but the present invention is not limited to this. A tilt lens or the like may be combined so that the surface of the object 401 is in focus. In this embodiment, the imaging devices 404 and 405 have photoelectric conversion characteristics that obtain a linear signal value with respect to the luminance on the object 401. In addition, it is assumed that the image data acquired by the imaging devices 404 and 405 has RGB 3 channel color information in each pixel, and each channel is quantized with 8 bits. The imaging device 404 and the imaging device 405 may be installed at arbitrary positions as long as they are different positions, but as shown in FIG. 4, they should be installed at positions that are opposite to each other with the light source 403 as a reference. Is desirable.

  The light source 403 is provided with a polarizer 406, the imaging device 404 is provided with a polarizer 407 and a polarizer 408, and the imaging device 405 is provided with a polarizer 409 and a polarizer 410. Here, the direction in which the linearly polarized light is transmitted through the polarizer 407 is parallel to the direction in which the linearly polarized light is transmitted through the polarizer 406 installed in the light source 403, and the states of the polarizer 406 and the polarizer 407 at this time are parallel. Nicole state. Further, the direction in which the linearly polarized light is transmitted through the polarizer 408 is perpendicular to the direction through which the linearly polarized light is transmitted through the polarizer 406, and the state of the polarizer 406 and the polarizer 408 at this time is an orthogonal Nicols state. Similar to the imaging device 404, the imaging device 405 is provided with a polarizer 406 and a parallel Nicol polarizer 409, and a polarizer 406 and a crossed Nicol polarizer 410. In the optical system illustrated in FIG. 4, it is possible to switch the polarizer to be used for each of the polarizer 407 and the polarizer 408, and the polarizer 409 and the polarizer 410. That is, the diffusion / surface reflection image of the object 401 is acquired by using the polarizers 407 and 409 in a parallel Nicol state with respect to the polarizer 406 of the light source when photographing with the imaging devices 404 and 405. In addition, a diffuse reflection image of the object 401 is obtained by cutting the surface reflected light using the polarizers 408 and 410 in a crossed Nicols state with respect to the polarizer 406 of the light source at the time of photographing with the imaging devices 404 and 405. . Hereinafter, for convenience of explanation, the imaging device 404 is also referred to as a left camera and the imaging device 405 is also referred to as a right camera. A diffusion / surface reflection image and a diffuse reflection image acquired by photographing with the left camera 404 and the right camera 405 are held in the image holding unit 203 in advance.

Hereinafter, processing executed by the image processing apparatus 1 will be described with reference to FIG. First, in step S <b> 301, the diffusion / surface reflection image acquisition unit 201 holds an image I _{p, L} captured using the polarizer 407 in a parallel Nicol state and the left camera 404 with respect to the polarizer 406 of the light source 403. Obtained from the unit 203. The image acquired in this step is an image obtained by the left camera 404 acquiring the reflected light including the diffuse reflected light and the surface reflected light of the object 401. In step S <b> 302, the diffusion / surface reflection image acquisition unit 201 captures the images I _{p, R} captured using the polarizer 409 and the right camera 405 in a parallel Nicol state with respect to the polarizer 406 of the light source 403. Get from. In step S < _b > 303, the diffuse reflection image acquisition unit 202 acquires _, from the image holding unit 203, images I _{c, L} captured using the polarizer 408 in the direct Nicole state and the left camera 404 with respect to the polarizer 406 of the light source 403. To do. The image acquired in this step is an image obtained by the left camera 404 acquiring reflected light that does not substantially include surface reflected light and is dominated by diffuse reflected light. In step S304, the color image generation unit 204 converts the image I _{p, L} acquired in step S301, the image I _{p, R} acquired in step S302, and the image I _{c, L} acquired in step S303. Based on this, a color image I of the object 401 is generated. Details of step S304 will be described later. In step S305, the image output unit 205 displays an image based on the color image generated in step S304 on the display unit 102. In the following, for convenience of explanation, an image obtained by photographing in a state where the polarizer of the light source and the polarizer of the imaging device are in the parallel Nicols state is also referred to as a parallel image, and the polarizer of the light source and the polarizer of the imaging device are in the orthogonal Nicols state An image obtained by photographing in is also called an orthogonal image.

<Details of Step S304>
Hereinafter, the color image generation processing in step S304 will be described with reference to FIG. In step S501, the color image generation unit 204 sets a variable (x, y) representing the coordinates of the image to an initial value (0, 0). In step S502, the color image generation unit 204 determines the standard deviation σ between the pixel value at the coordinates (x, y) of the parallel image I _{p, L} and the pixel value at the coordinates (x, y) of the parallel image I _{p, R. p} (x, y) is calculated by the following equation. In the following, an image I _{p, L} of the coordinates (x, y) the pixel value at I _p, L (x, y), the image I _p, the pixel value at the coordinates (x, y) of the _R I _{p, R} (x, y) etc. When the parallel images taken by the left and right cameras are different, the change in luminance is the largest among hue, luminance, and saturation. Therefore, in this embodiment, the expressions (1) and (2) are used based on the G channel pixel values G _{p, L} (x, y) and G _{p, R} (x, y) containing a large amount of luminance information. The standard deviation σ _p (x, y) is calculated. Note that the standard deviation calculation method is not limited to that using the G channel. For example, the RGB value may be converted into the CIELab value by a known method, and the standard deviation may be calculated using the L * value.

In step S503, the color image generation unit 204 determines the magnitude relationship between σ _p (x, y) calculated in step S502 and a preset threshold Th1 using Expression (3). Here, the determination process in step S503 will be described with reference to FIG. FIG. 6 is a diagram showing the relationship between surface irregularities and reflection characteristics in an object, and FIG. 6A shows the reflection characteristics when light is reflected by a substantially smooth surface 601, and FIG. Indicates the reflection characteristics when light is reflected by the surface 604 having fine irregularities. In FIG. 6A, a thick solid line 602 indicates diffuse reflection light of the object, and a broken line 603 indicates surface reflection light of the object. As shown in FIG. 6A, when the surface of the object is substantially smooth, the light irradiated to the object is reflected around the direction of the reflection angle equal to the incident angle. Therefore, as indicated by a cross in the figure, the amount of reflected light (reflection intensity) acquired at two different observation angles varies greatly depending on the observation angle. In such a case, the component of the surface reflected light is unevenly distributed around the direction of the reflection angle substantially the same as the incident angle, and generally the surface reflected light is not included in the direction (0 degree direction) in which the object is observed. Therefore, only the diffuse reflected light 602 is acquired in the 0 degree direction. On the other hand, in FIG. 6B, a thick solid line 605 indicates diffuse reflection light of the object, and a broken line 606 indicates surface reflection light of the object. As shown in FIG. 6B, when the object has fine irregularities on the surface, the light irradiated on the object is scattered in various directions on the surface. Therefore, as indicated by a cross in the figure, the reflection intensity acquired at two different observation angles does not change greatly even when the observation angle changes. In such a case, since the component of the surface reflected light is also present in the 0 degree direction, the reflected light including the diffuse reflected light 605 and the surface reflected light 606 is acquired in the 0 degree direction. In this embodiment, when the standard deviation σ _p (x, y) of the parallel images acquired by the left camera and the right camera installed at different positions (observation angles) satisfies Expression (3), FIG. It judges that it is in the state of and advances to Step S504. On the other hand, when the expression (3) is not satisfied, it is determined that the state shown in FIG.

σ _p (x, y)> Th1 Expression (3)

In step S <b> 504, the color image generation unit 204 uses the polarizer 408 and the left camera 404 as the pixel value at the coordinates (x, y) of the color image I as shown in Expression (4). The pixel value at the coordinates (x, y) of _{c and L} is substituted.

I (x, y) = I _{c, L} (x, y) (4)

In step S505, the color image generation unit 204 uses the polarizer 407 and the left camera 404 as the pixel value at the coordinates (x, y) of the color image I, as shown in Expression (5). The pixel value at the coordinates (x, y) of _{p and L} is substituted.

I (x, y) = I _{p, L} (x, y) (5)

  In step S506, the color image generation unit 204 determines whether all pixels have been processed. If the determination result in step S506 is true, the series of processes ends. If the determination result is false, the process proceeds to step S507. In step S507, the color image generation unit 204 updates the value of the variable (x, y) indicating the coordinates of the target pixel to be processed, and the process returns to step S502.

The color image generated by the above processing is an image in which the pixels of the orthogonal images I _{c, L and} the pixels of the parallel images I _{p, L} are selectively arranged. By generating a color image in this way, it becomes possible to grasp the color when observing the object with high accuracy.

  In this embodiment, when generating a color image, a standard deviation is calculated in order to acquire information on changes in reflection intensity over a plurality of observation angles, and a threshold is determined using the calculated standard deviation. Yes. However, the parameter used for this determination is not limited to the standard deviation, and may be a variance of pixel values or a difference between pixel values, for example.

  In the present embodiment, in order to acquire information regarding changes in reflection intensity over a plurality of observation angles, an object is photographed by two imaging devices (that is, a left camera and a right camera) installed at different positions. . However, if it is possible to acquire information on the change in reflection intensity over a plurality of observation angles, an optical system in which two or more imaging devices are installed at different positions may be used. May be taken.

  In this embodiment, in the color image generation process, the color image is generated based on the pixel value of the image acquired by the left camera, but the color image may be generated based on the pixel value of the image acquired by the right camera. .

  In the present embodiment, an image acquired by the imaging device photographing light that has passed through a polarizer in a parallel Nicol state with respect to the polarizer of the light source is used as the diffusion / surface reflection image. However, the diffuse / surface reflection image is not limited to this, and any image acquired by photographing light including diffuse reflection light and surface reflection light may be used. For example, an image acquired with a configuration in which a polarizer is not installed in the imaging apparatus may be used as the diffusion / surface reflection image.

[Example 2]
In the first embodiment, in order to acquire information related to changes in reflection intensity over a plurality of observation angles, images acquired by shooting a target with a plurality of imaging devices installed at different positions are used. However, an image acquired by one imaging device may be used as long as information on changes in reflection intensity over a plurality of observation angles can be acquired. In this embodiment, the object is divided and photographed by one imaging device so that the photographing areas overlap. In the following, for convenience of explanation, it is assumed that divided shooting is performed using the left camera 404 of the optical system shown in FIG. In the present embodiment, the description common to the first embodiment is simplified or omitted.

  FIG. 7 is a diagram for explaining divided shooting in the present embodiment. In FIG. 7, a solid line 701 indicates an object, and broken lines 702, 703, 704, and 705 indicate one-shot shooting areas at the time of divided shooting. In this embodiment, as shown in FIG. 7, the object 701 is divided and photographed so that one-shot photographing areas (the photographing area 702 and the photographing area 703) partially overlap. And the information regarding the change of the reflection intensity over a plurality of observation angles is acquired in the overlapping area where the imaging areas partially overlap.

<About processing executed by image processing apparatus>
Hereinafter, processing executed by the image processing apparatus 1 according to the present embodiment will be described with reference to FIG. In step S <b> 801, the diffusion / surface reflection image acquisition unit 201 captures the diffusion / surface reflection image group for the entire imaging region, which is captured using the polarizer 407 in the parallel Nicol state and the left camera 404 with respect to the polarizer 406 of the light source 403. (Parallel image group) is acquired. In this embodiment, the object is photographed by dividing it into four (see FIG. 7), and four images are acquired in this step. In step S <b> 802, the diffuse reflection image acquisition unit 202 acquires a diffuse reflection image group for the entire imaging region, which is captured using the polarizer 408 in the crossed Nicols state and the left camera 404 with respect to the polarizer 406 of the light source 403. In step S803, the color image generation unit 204 generates a color image I based on the image group acquired in steps S801 and S802. Details of step S803 will be described later. In step S804, the image output unit 205 outputs an image based on the color image generated in step S803.

<Details of Processing in Step S803>
Hereinafter, the color image generation processing in step S803 will be described with reference to FIG. In step S901, the color image generation unit 204 sets variables (x, y) representing the position on the surface of the target object to initial values. In this embodiment, it is assumed that the image group acquired in step S801 and the image group acquired in step S802 have been aligned in advance. When the alignment is not performed, the corresponding point is searched for in the image of each imaging region, and the variable (x, y) is set to the initial value. For example, a known block matching method can be used as a corresponding point search method. Since the corresponding point search method is not the main point of the present invention, a description thereof will be omitted.

In step S902, the color image generation unit 204 extracts the G channel pixel value at the coordinates (x, y) from the diffuse / surface reflection image group for the entire imaging region acquired in step S801, and uses the equations (6) and (6). The standard deviation σ (x, y) is calculated from (7). In these equations, N represents the number of divisions of the region (4 in this embodiment), and G _{p, k} (x, y) represents the coordinates of the diffusion / surface reflection image (parallel image) for the kth region ( It represents the pixel value of the G channel at x, y).

  In step S903, the color image generation unit 204 determines whether or not Expression (8) is satisfied. In Expression (8), Th2 is a preset threshold value. If the determination result in step S903 is true, it is determined that the change in the reflection intensity over a plurality of observation angles is large (see FIG. 6A), and the process proceeds to step S904. On the other hand, when the determination result of step S903 is false, it is determined that the change in the reflection intensity over a plurality of observation angles is small (see FIG. 6B), and the process proceeds to step S905.

σ (x, y)> Th2 (8)

  In step S904, the color image generation unit 204 sets the G channel value closest to the average value of the G channels in the orthogonal image group acquired in step S802 to the pixel value at the coordinates (x, y) of the color image I. The pixel value at (x, y) of the image having is substituted.

  In step S905, the color image generation unit 204 sets the G channel value closest to the average value of the G channels in the parallel image group acquired in step S801 to the pixel value at the coordinates (x, y) of the color image I. The pixel value at (x, y) of the image having is substituted.

  Steps S906 and S907 are the same as steps S506 and S507 in the first embodiment.

  According to the present embodiment, color information for observing an object can be acquired with high accuracy as in the first embodiment, based on an image group acquired by one imaging apparatus.

  In this embodiment, as shown in FIG. 7, the object is imaged by dividing it into four parts. However, the number of divisions is not limited to four and may be any number as long as the imaging regions partially overlap. .

  In this embodiment, divided shooting is performed using the left camera 404, but divided shooting may be performed using the right camera 405.

[Example 3]
In Example 1 and Example 2, in order to acquire information regarding the change in reflection intensity over a plurality of observation angles, photographing is performed using an imaging device in which a polarizer in a parallel Nicol state is installed with respect to a polarizer of a light source. The diffuse / surface reflection image (parallel image) is used. In the present embodiment, the unevenness information of the object is used in order to acquire information regarding the change in reflection intensity over a plurality of observation angles. In the present embodiment, the description common to the above embodiment is simplified or omitted.

  FIG. 10 is a block diagram illustrating a functional configuration of the image processing apparatus 1 according to the third embodiment. The image processing apparatus 1 includes a diffuse / surface reflection image acquisition unit 1001, a diffuse reflection image acquisition unit 1002, an image holding unit 1003, an uneven image acquisition unit 1004, a color image generation unit 1005, and an image output unit 1006. Have. The diffuse / surface reflection image acquisition unit 1001, the diffuse reflection image acquisition unit 1002, and the image holding unit 1003 are the same as the diffusion / surface reflection image acquisition unit 201, the diffuse reflection image acquisition unit 202, and the image holding unit 203 of the first embodiment. It is. The concavo-convex image acquisition unit 1004 acquires a concavo-convex image representing the concavo-convex information for each position of the object from the image holding unit 1003. In addition, the uneven | corrugated image in a present Example is a bitmap image of 8-bit gray scale in which each pixel has the height with respect to a reference plane as a pixel value. The color image generation unit 1005 has a diffusion / surface reflection image acquired by the diffusion / surface reflection image acquisition unit 1001, a diffuse reflection image acquired by the diffusion reflection image acquisition unit 1002, and unevenness acquired by the unevenness image acquisition unit 1004. A color image is generated based on the image. The image output unit 1006 is the same as the image output unit 205 of the first embodiment.

<About processing executed by image processing apparatus>
Hereinafter, processing executed by the image processing apparatus 1 in the present embodiment will be described with reference to FIG.

  In step S <b> 1101, the diffusion / surface reflection image acquisition unit 1001 captures a diffusion / surface reflection image (parallel image) Ip photographed using an imaging device in which a polarizer in a parallel Nicol state is installed with respect to the polarizer of the light source. Obtained from the holding unit 1003. In step S <b> 1102, the diffuse reflection image acquisition unit 1102 receives from the image holding unit 1003 a diffuse reflection image (orthogonal image) Ic photographed using an imaging device in which a polarizer in a crossed Nicol state is installed with respect to the polarizer of the light source. get. In step S <b> 1103, the uneven image acquisition unit 1004 acquires the uneven image H of the object from the image holding unit 1003. In step S1104, the color image generation unit 1005 generates a color image based on the parallel image Ip acquired in step S1101, the orthogonal image Ic acquired in step S1102, and the uneven image H acquired in step S1103. Details of step S1104 will be described later. In step S1105, the image output unit 1006 outputs an image based on the color image generated in step S1104.

<Details of Processing in Step S1104>
Hereinafter, the color image generation processing in step S1104 will be described with reference to the flowchart of FIG.

In step S1201, the color image generation unit 1005 sets a variable (x, y) representing the coordinates of the image to an initial value (0, 0). In step S1202, the color image generation unit 1005 refers to the range of a × b as a neighboring pixel of the coordinates (x, y) of the uneven image H acquired in step S1103, and the standard deviation σ _h (x, y) is calculated by Equation (9) and Equation (10).

  In step S1203, the color image generation unit 1005 determines whether or not Expression (11) is satisfied. In Formula (11), Th3 is a preset threshold value. If the determination result in step S1203 is true, the surface of the object is determined to be smooth (see FIG. 6A), and the process proceeds to step S1204. On the other hand, when the determination result of step S1203 is false, it is determined that the object has fine unevenness on the surface (see FIG. 6B), and the process proceeds to step S1205.

σ _h (x, y) <Th3 Expression (11)

  In step S1204, the color image generation unit 1005 substitutes the pixel value at the coordinates (x, y) of the orthogonal image Ic for the pixel value at the coordinates (x, y) of the color image I as shown in Expression (12). To do.

I (x, y) = Ic (x, y) (12)

  In step S1205, the color image generation unit 1005 substitutes the pixel value at the coordinates (x, y) of the parallel image Ip for the pixel value at the coordinates (x, y) of the color image I as shown in Expression (13). To do.

I (x, y) = Ip (x, y) (13)

  Steps S1206 and S1207 are the same as steps S506 and S507 in the first embodiment.

  According to the present embodiment, by generating a color image based on the standard deviation of the concavo-convex image, it becomes possible to acquire the color information of the object with high accuracy as in the first embodiment.

[Example 4]
In the first embodiment, the color image is generated by selectively arranging the pixels of the diffuse / surface reflection image I _{p, L and} the pixels of the diffuse reflection image I _{c, L} according to the coordinates. In the present embodiment, the diffusion / surface reflection image I _{p, L} , the composite pixel of the diffusion / surface reflection image I _{p, L} and the diffuse reflection image I _{c, L} , and the diffuse reflection image I _{c , L} pixels are selectively arranged according to the coordinates to generate a color image. In the present embodiment, the contents different from the first embodiment will be described in detail, and the description common to the first embodiment will be simplified or omitted.

<Details of Step S304>
The processing executed by the image processing apparatus 1 in the present embodiment is the same as that in the first embodiment (see FIG. 3). Therefore, hereinafter, the color image generation processing in step S304 in the present embodiment will be described with reference to FIG.

  Steps S1301 and S1302 are the same as steps S501 and S502 of the first embodiment. In step S1303, the color image generation unit 204 determines whether or not Expression (14) is satisfied. In formula (14), Th4 is a preset threshold value. If the determination result of step S1303 is true, it is determined that the change in reflection intensity over a plurality of observation angles is large, and the process proceeds to step S1305. On the other hand, if the determination result of step S1303 is false, the process proceeds to step S1304.

σ _p (x, y)> Th4 Expression (14)

  In step S1304, the color image generation unit 204 determines whether or not Expression (15) is satisfied. In Formula (15), Th5 is a preset threshold value. If the determination result of step S1304 is true, it is determined that the change in reflection intensity over a plurality of observation angles is moderate, and the process proceeds to step S1306. On the other hand, if the determination result of step S1304 is false, it is determined that the change in reflection intensity over a plurality of observation angles is small, and the process proceeds to step S1307.

σ _p (x, y)> Th5 Expression (15)

In step S1306, the color image generation unit 204 obtains a pixel value at the coordinates (x, y) of the color image I using Expression (16). As shown in Expression (16), as the pixel at the coordinates (x, y) of the color image I, the pixel at the coordinates (x, y) of the surface / diffuse reflection image I _{p, L} and the diffuse reflection image I _{c, L} A pixel obtained by synthesizing the pixel at the coordinates (x, y) is used.

I (x, y) = Th4 / (Th4-Th5) × I _{p, L} (x, y) + Th5 / (Th4-Th5) × I _{c, L} (x, y) (16)

  Step S1305, step S1307, step S1308, and step S1309 are the same as step S504, step S505, step S506, and step S507 of the first embodiment, respectively.

  According to the present embodiment, the color information of the object can be acquired with higher accuracy than in the first embodiment.

  In this embodiment, a case where information relating to changes in reflection intensity over a plurality of observation angles is acquired using images acquired by shooting an object with a plurality of imaging devices installed at different positions (example). 1) is described. However, the present embodiment is also applicable to a case where an object is divided and photographed with a single imaging device (Example 2) or an uneven image of the object is used (Example 3) so that the imaging regions overlap. Applicable.

[Example 5]
In the above-described embodiment, the standard deviation of the object is derived, and the color image is generated by estimating the change in the reflection intensity over a plurality of observation angles using the derived standard deviation. In this embodiment, a color image is generated by estimating a change in reflection intensity over a plurality of observation angles based on information specifying the type of material of the object. For example, when the material of the object is metal or the like, it can be estimated that the change in reflection intensity over a plurality of observation angles is large because the surface has high smoothness (see FIG. 6A). On the other hand, when the object is an oil painting or the like, the surface has fine irregularities, and it can be estimated that the change in the reflection intensity over a plurality of observation angles is small (see FIG. 6B). In the present embodiment, the description common to the above embodiment is simplified or omitted.

  FIG. 14 is a block diagram illustrating a functional configuration of the image processing apparatus 1 according to the present embodiment. The image processing apparatus 1 includes a diffuse / surface reflection image acquisition unit 1401, a diffuse reflection image acquisition unit 1402, an image holding unit 1403, a material information acquisition unit 1404, a color image generation unit 1405, and an image output unit 1406. Have. The diffuse / surface reflection image acquisition unit 1401, the diffuse reflection image acquisition unit 1402, and the image holding unit 1403 are the same as the diffusion / surface reflection image acquisition unit 201, the diffuse reflection image acquisition unit 202, and the image holding unit 203 of the first embodiment. It is. The material information acquisition unit 1404 acquires information (material information) regarding the types of materials constituting the object. The material information in the present embodiment is bitmap format data composed of pixel values that specify the material, and is input by the user via the user interface displayed on the display unit 102. In the present embodiment, the material information is also referred to as a material map. The color image generation unit 1405 is acquired by the diffusion / surface reflection image acquired by the diffusion / surface reflection image acquisition unit 1401, the diffuse reflection image acquired by the diffuse reflection image acquisition unit 1402, and the material information acquisition unit 1404. A color image is generated based on the material information. An image output unit 1406 is the same as the image output unit 205 of the first embodiment.

<About processing executed by image processing apparatus>
Hereinafter, processing executed by the image processing apparatus 1 in the present embodiment will be described with reference to FIG. Steps S1501 and S1502 are the same as steps S1101 and S1102 in the third embodiment.

  In step S1503, the material information acquisition unit 1404 acquires material information (material map) of the object. FIG. 16 shows an example of a user interface used by a user to input a material map. As illustrated, the user interface includes an image display unit 1601, a material designation unit 1603, and an OK button 1605. An image 1602 of the object is displayed on the image display unit 1601. In this embodiment, the diffusion / surface reflection image (parallel image) acquired in step S1501 is displayed. The user designates each pixel of the diffuse / surface reflection image 1602 with the pointer 1604 and selects one of the radio buttons displayed on the material designation unit 1603 to designate the material in each pixel of the target object. Note that a material may be specified for each block of a predetermined size including a plurality of pixels instead of each pixel. In the present embodiment, the material designating unit 1603 includes a radio button for designating a material having a large change in reflection intensity over a plurality of angles, a radio button for designating a material having a small change, And a radio button for designating that the change is moderate. However, the number of radio buttons is not limited to three and may be two or four or more. When the user performs the above operation on all pixels and presses an OK button 1605, the input of the material map is completed. Through the above processing, a material map (denoted as M) in which the types of materials constituting the object are described for each pixel is obtained. FIG. 17 shows an example of the material map M. As shown in the figure, the material map is bitmap format data composed of pixel values for each pixel that designates the material. In this embodiment, the value corresponding to high gloss is set to 0, the value corresponding to low gloss is set to 1, and the value corresponding to gloss is set to 2, but the correspondence between the type of material and the possible pixel values is as follows. If it is clear, the pixel value to be used is not limited to these, and an arbitrary value may be used.

  In step S1504, the color image generation unit 1405, the diffuse / surface reflection image (parallel image) acquired in step S1501, the diffuse reflection image (orthogonal image) acquired in step S1502, and the material information acquired in step S1503. Based on the above, a color image is generated. Details of step S1504 will be described later.

  Step S1505 is the same as step S1105 of the third embodiment.

<Details of Step S1504>
Hereinafter, the color image generation processing in step S1504 will be described with reference to FIG.

  Step S1801 is the same as step S501 in the first embodiment.

  In step S1802, the color image generation unit 1405 acquires the value M (x, y) at the coordinates (x, y) of the material map M acquired in step S1503.

  In step S1803, the color image generation unit 1405 determines the value M (x, y) acquired in step S1802. As a result of the determination in step S1803, if M (x, y) = 0, the process proceeds to step S1804. If M (x, y) = 1, the process proceeds to step S1805. If M (x, y) = 2, The process proceeds to step S1806.

  Step S1804, step S1805, and step S1806 are the same as step S1305, step S1307, and step S1306 of the fourth embodiment. Steps S1807 and S1808 are the same as steps S506 and S507 in the first embodiment.

  According to the present embodiment, it is possible to obtain the color information of the object with high accuracy without deriving the standard deviation.

  In this embodiment, the pixel value of the color image is combined with the pixel value of the diffuse / surface reflection image, the pixel value of the diffuse reflection image, and the pixel of the diffusion / surface reflection image and the pixel of the diffuse reflection image. A case where any one of the pixel values is substituted for each pixel (Example 4) is described. However, this embodiment can also be applied to the first to third embodiments.

[Other Examples]
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

DESCRIPTION OF SYMBOLS 1 ... Image processing apparatus 204 ... Color image generation part

Claims (11)

An image processing device that generates a color image representing a color distribution of an object,
A first reflection image obtained by photographing the first reflected light representing the diffuse reflected light and the surface reflected light of the object and a second reflected light obtained by photographing the second reflected light representing the diffuse reflected light. Acquisition means for acquiring two reflection images;
An image processing apparatus comprising: a generation unit configured to generate the color image based on the first reflection image and the second reflection image acquired by the acquisition unit.   The generation unit substitutes, for each pixel, either the pixel value of the first reflection image or the pixel value of the second reflection image as the pixel value of the color image. The image processing apparatus according to claim 1, wherein:   The generation means includes a pixel value of the first reflection image, a pixel value of the second reflection image, a pixel of the first reflection image, and a second reflection image as the pixel value of the color image. The image processing apparatus according to claim 1, wherein the color image is generated by substituting one of pixel values of a pixel obtained by combining the first pixel and the second pixel for each pixel.   And a determination unit that determines which pixel value is to be substituted as a pixel value of the color image based on a plurality of first reflection images acquired by photographing the object from a plurality of different positions. The image processing apparatus according to claim 2, wherein the image processing apparatus is an image processing apparatus.   The image processing apparatus further includes a determination unit that determines which pixel value is substituted as a pixel value of the color image based on a plurality of first reflection images acquired by dividing and photographing the object from the same position. The image processing apparatus according to claim 2 or 3.   The determination unit calculates a standard deviation of each pixel from the plurality of first reflection images, and substitutes any pixel value based on a determination result of a magnitude relationship between the calculated standard deviation and a predetermined threshold value. The image processing apparatus according to claim 4, wherein it is determined whether or not to perform.   The image processing apparatus according to claim 2, further comprising a determination unit that determines which pixel value is substituted as a pixel value of the color image based on the uneven image of the object.   The determination unit calculates a standard deviation of each pixel from the uneven image, and determines which pixel value is substituted based on a determination result of a magnitude relationship between the calculated standard deviation and a predetermined threshold value. The image processing apparatus according to claim 7.   4. The apparatus according to claim 2, further comprising a determination unit that determines which pixel value is substituted as a pixel value of the color image based on material information indicating a material in each pixel of the object. 5. Image processing apparatus. An image processing method for generating a color image representing a color distribution of an object,
A first reflection image obtained by photographing the first reflected light representing the diffuse reflected light and the surface reflected light of the object and a second reflected light obtained by photographing the second reflected light representing the diffuse reflected light. An acquisition step of acquiring two reflection images;
An image processing apparatus comprising: a generation step of generating the color image based on the first reflection image and the second reflection image acquired in the acquisition step.   The program for making a computer perform the method of Claim 10.

Images