JP2021077237A - Information processing apparatus, display unit, information processing method, and program - Google Patents

Abstract

To easily perform color matching of an image for the appearance of a subject.SOLUTION: An information processing apparatus 300 controls display of information including a first image for reference and a second image to be corrected. The information processing apparatus 300 has a reflected light ratio derivation unit 408 and a display control unit 410. The reflected light ratio derivation unit 408 derives the ratio between a diffused reflection component and a mirror reflection component included in pixel values of the first image, based on the intensity of reflected light for every angle in the subject, shape information on the subject, and a geometric condition corresponding to the first image in picking up an image of the subject. The display control unit 410 displays, on display means, the first image, the second image, and the derived ratio altogether.SELECTED DRAWING: Figure 4

Description

The present invention relates to a technique for editing texture information of an image.

In recent years, with the improvement of printing and computer graphics (CG) technology, it is becoming possible to express the texture of an object in an image. On the other hand, there is known a technique of acquiring information on the appearance texture of a subject by measuring the subject while changing the lighting and the direction of observation in various ways and approximating the measured value with a reflection model. By using the acquired texture information for material settings in CG software, the texture of the subject virtually placed in the three-dimensional space by rendering is reproduced under lighting that does not actually exist, but it is an approximate value. It is a reproduction using, and the color of the appearance is different from the real thing. In reproducing the texture of a subject using an approximate value, it is necessary for the user to edit the texture in order to bring the rendering result closer to the actual appearance. In texture editing, the reflected light is generally separated into a specular reflected light having an intensity distribution in the normal reflection direction and a diffuse reflected light whose intensity does not change significantly in any reflection direction. Patent Document 1 separates the pixel values of the pixels constituting the image into a mirror-reflected light component due to mirror reflection and a diffuse-reflected light component due to diffuse reflection, and at least one of the mirror-reflected light component and the diffuse-reflected light component is used. A technique for changing the color of a three-dimensional object in an image by changing the color is described.

Japanese Unexamined Patent Publication No. 2001-52144

When each reflection component is controlled independently as in Patent Document 1, diffuse reflection light and specular reflection light are not associated with each other with respect to the color in the image, and the color matching of the image is performed with respect to the appearance of the subject. There is a problem that it is difficult.

The present invention has been made in view of the above problems, and is to provide a technique capable of easily color-matching an image with respect to the appearance of a subject.

The information processing device according to one aspect of the present invention is an information processing device that controls the display of information including the first image for reference and the second image to be corrected, and the reflected light intensity for each angle in the subject. , The ratio of the diffuse reflection component and the mirror reflection component included in the pixel value of the first image based on the shape information of the subject and the geometric conditions corresponding to the first image when the subject is imaged. It is characterized by having a derivation means for deriving the above, and a display control means for displaying the first image, the second image, and the derived ratio together on the display means.

According to the present invention, it is possible to easily perform color matching of an image with respect to the appearance of a subject.

The figure which shows the example of the color matching under the geometric condition which the diffuse reflection light contributes. The figure which shows the example of the color matching in the geometrical condition where diffuse reflection light and specular reflection light are mixed. Block diagram showing a hardware configuration example of an information processing device Block diagram showing an example of logical configuration of an information processing device Diagram showing an example of how to express texture information Flowchart showing the flow of processing executed by the information processing device Diagram showing GUI example The figure which shows the angle difference between a half vector and a normal under the imaging geometric condition. Diagram showing the frequency distribution of the angle difference between the half vector and the normal in the reference image The figure which shows the variation intensity with respect to the angle difference between a half vector and a normal Block diagram showing an example of logical configuration of an information processing device Flowchart showing the flow of processing executed by the information processing device Diagram showing GUI example The figure which shows the example of a specular reflection distribution image Block diagram showing an example of logical configuration of an information processing device Flowchart showing the flow of processing executed by the information processing device Diagram showing GUI example Block diagram showing an example of logical configuration of an information processing device Flowchart showing the flow of processing executed by the information processing device Diagram showing GUI example Block diagram showing an example of logical configuration of an information processing device Flowchart showing the flow of processing executed by the information processing device Diagram showing GUI example

Hereinafter, embodiments of the present invention will be described with reference to the drawings. It should be noted that the following embodiments do not limit the present invention, and not all combinations of features described in the present embodiment are essential for the means for solving the present invention. The same configuration will be described with the same reference numerals.

<< Embodiment 1 >>
First, editing of the texture information of the subject in the image will be described. When color matching of a rendered image is performed while referring to a target image (hereinafter referred to as a reference image) representing the appearance of a real object, the user independently controls the specular reflected light component and the diffuse reflected light component of the rendered image. .. At this time, if the specular reflected light component and the diffuse reflected light component are not associated with each other with respect to the color of the image, it is difficult for the user to perform color matching of the rendered image.

FIG. 1 is a diagram showing an example of color matching when a reference image is used in which mirror-reflected light is not included and only diffuse-reflected light is included. FIG. 1A shows the geometric conditions when the reference image is captured. FIG. 1 (b) shows the flow of color matching under the geometric conditions shown in FIG. 1 (a). Since the reference image does not have a gloss change according to the shape of the subject, the user recognizes that the color of the image before color correction being observed is due to the diffuse reflection light. Then, the user can bring the color of the rendered image, which is the image after the color correction, closer to the reference image by changing the color of the recognized diffuse reflected light in the image before the color correction. On the other hand, FIG. 2 is a diagram showing an example of color matching when a reference image including diffuse reflected light and specular reflected light is used. FIG. 2A shows the geometric conditions when the reference image is captured. FIG. 2B shows the flow of color matching under the geometric conditions shown in FIG. 2A. Since the reference image has a gloss change according to the shape of the subject, the user recognizes that the color of the image before color correction, which is being observed, comes from the specularly reflected light. Then, the user changes the color of the recognized specularly reflected light in the image before the color correction. However, the color observed in the reference image is a combination of the specular reflected light and the diffuse reflected light, and the color cannot be brought close to the reference image only by editing the specular reflected light. This happens because the user cannot recognize that the diffuse and specular light are associated with each other for the colors in the reference image.

Therefore, in the present embodiment, the ratio of the diffuse reflection light and the specular reflection light in the reference image is based on the threshold value set for the deviation from the normal reflection condition obtained from the geometric conditions of imaging and illumination and the shape information of the subject. Derived and the ratio of each derived reflected light is displayed as a numerical value on the GUI. As a result, the user can easily edit the texture while confirming the influence of each reflected light on the apparent color by referring to the ratio of each reflected light displayed on the GUI.

(Device configuration of information processing device)
The hardware configuration of the information processing apparatus according to this embodiment will be described with reference to FIG. FIG. 3 is a diagram showing a hardware configuration example of the information processing device 300 according to the present embodiment.

The information processing device 300 includes a CPU 301, a RAM 302, a ROM 303, a secondary storage device 304, an input interface (hereinafter referred to as an input IF) 305, and an output interface (hereinafter referred to as an output IF) 306. The components of the information processing apparatus 300 are connected to each other by the system bus 307. The information processing device 300 is connected to the input device 308 via the input IF 305. The information processing device 300 is connected to the external storage device 309 via the input IF 305 and the output IF 306. The information processing device 300 is connected to the display device 310 via the input IF 305 and the output IF 306.

The CPU (Central Processing Unit) 301 executes a program stored in the ROM 303, the secondary storage device 304, or the like using the RAM 302 as a work memory, and comprehensively controls each component of the information processing device via the system bus 307. .. As a result, various processes described later are executed. The secondary storage device 304 is a storage device that stores various data handled by the information processing device 300, and a hard disk drive (HDD) is used in the present embodiment. The CPU 301 writes data to the secondary storage device 304 and reads data stored in the secondary storage device 304 via the system bus 307. In addition to the HDD, various storage devices such as an optical disk drive and a flash memory can be used for the secondary storage device 304.

The RAM (Random Access Memory) 302 includes a work area used by the CPU 301 to execute various processes. The ROM (Read Only Memory) 303 stores programs, data, and the like for causing the CPU 301 to execute various processes described later.

The input IF305 is, for example, a serial bus interface such as USB or IEEE1394. The information processing device 300 receives data, instructions, and the like from an external device via the input IF 305. In the present embodiment, the information processing device 300 acquires data from an external storage device 309 (for example, a storage medium such as a hard disk, a memory card, a CF card, an SD card, or a USB memory) via an input IF 305. Further, in the present embodiment, the information processing device 300 acquires the user's instruction input to the input device 308 (for example, a mouse, keyboard, tablet) or the display device 310 via the input IF 305. The display device 310 is a display device such as a liquid crystal display, and displays data or the like output from the information processing device 300 via the output IF 306 by the CPU 301. As described above, when a tablet or smartphone is used as the information processing device 300, the input device 308 and the display device 310 are laminated to form a touch panel.

The output IF 306 is a serial bus interface such as USB or IEEE 1394. The output IF 306 may be, for example, a video output terminal such as DVI or HDMI (registered trademark). The information processing device 300 outputs data or the like to an external device via the output IF 306. In the present embodiment, the information processing device 300 stores data in the external storage device 309 via the output IF 306. Further, in the present embodiment, the information processing device 300 displays data processed by the CPU 301 on the display device 310 via the output IF 306 (for example, image data, a graphical user interface (GUI), processing progress and processing of image processing). Results etc.) are output.

Although there are components of the information processing apparatus other than the above, the description thereof will be omitted because they are not the main objects of the present invention.

(Logical configuration of information processing device)
Next, the processing performed by the information processing apparatus 300 will be described with reference to FIG. FIG. 4 is a block diagram showing an example of a logical configuration of the information processing apparatus 300. The information processing device 300 functions as the logical configuration shown in FIG. 4 by executing the program stored in the ROM 303 by the CPU 301 using the RAM 302 as the work memory. It should be noted that it is not necessary that all of the processes shown below are executed by the CPU 301, and even if the information processing apparatus 300 is configured so that a part or all of the processes are performed by one or a plurality of processing circuits other than the CPU 301. Good.

The information processing device 300 includes a texture color information acquisition unit 401, a texture variation angle information acquisition unit 402, a shape information acquisition unit 403, an imaging / lighting condition acquisition unit 404, a reference image acquisition unit 405, and a color correction information acquisition unit 406. The information processing device 300 further includes an initial rendered image generation unit 407, a reflected light ratio derivation unit 408, a rendered image generation unit 409, and a display control unit 410.

The texture color information acquisition unit 401 acquires the diffuse reflection color C _d (x, y, z) and the specular reflection color C _s (x, y, z) of the subject for which texture editing is performed. The diffuse reflection color C _d (x, y, z) and the specular reflection color C _s (x, y, z) of the subject are texture color information at a three-dimensional position (x, y, z) on the subject. , Information acquired by a known method and stored in advance in a secondary storage device 304 or the like.

Textured bending information acquisition unit 402, the diffuse reflection bending strength of the object to perform texture edit _{A d (x, y, z} , l n, v n), specular bending strength _{A s (x, y, z} , l _n , v _n ) is acquired. The diffusion reflection bending strength of the subject _{A d (x, y, z} , l n, v n) and specular bending strength _{A s (x, y, z} , l n, v n) , the angle of the object It is texture variation information representing each reflected light intensity, and is information acquired by a known method and stored in advance in a secondary storage device 304 or the like. This data will be described with reference to FIG. FIG. 5 is a diagram for explaining a method of expressing texture information. As shown in FIG. 5, this data is obtained as follows from the captured image obtained by imaging the subject while changing the position of the illumination (light source) and the observation direction (position and orientation of the camera). That is, the above data shows the reflection characteristics at the position (x, y, z) of the subject as a specular reflection component indicating the spread of the reflected light in the normal reflection direction and a diffuse reflection component indicating the reflected light diffused in all directions. It is obtained by approximating it as the sum of. Note that specular reflection _{component, C s (x, y,} z) · A s (x, y, z, l n, v n) represented by. The diffuse reflection component is represented by C _d (x, y, z) and _Ad (x, y, z, l _n , v _n ). The diffuse reflection color C _d (x, y, z) holds the RGB value of the diffuse reflection light at the position (x, y, z). The specular reflection color C _s (x, y, z) holds the RGB value of the specular reflection light at the position (x, y, z). The diffuse reflection variable angle intensity _Ad (x, y, z, l _n , v _{n ) is such that the illumination is incident light vector l n} = (l _x , l _y , l _z ) with respect to the position (x, y, z). ), And the diffuse reflection intensity when the emitted light vector v _n = (v _x , v _y , v _{z) is reflected.} Specular bending strength _{A s (x, y, z} , l n, v n) , the illumination is incident at an incident light vector l _n, holds a specular reflection intensity when reflected by the exit light vector v _n.

The shape information acquisition unit 403 acquires a normal line n (x, y, z) representing the surface shape of the subject. The normal line n (x, y, z) is data holding the normal line n (x, y, z) = (n _x , n _y , n _{z) of the subject at the position (x, y, z).} Is. The normal n (x, y, z) is associated with the diffuse reflection color C _d (x, y, z) and the specular reflection color C _s (x, y, z) acquired by the texture color information acquisition unit 401. Has been done. The normals n (x, y, z) are the diffuse reflection variation intensity _Ad (x, y, z, l _n , v _n ) and the specular reflection variation intensity Ad (x, y, z, l n, v n) acquired by the texture variation information acquisition unit 402. a _s are associated (x, y, z, l n, v n) and. The normal line n (x, y, z) is the shape information of the subject, which is the information acquired by a known method and stored in advance in the secondary storage device 304 or the like.

The imaging / illumination condition acquisition unit 404 separately acquires the imaging condition V and the illumination condition L of the reference image acquired by imaging the subject with the camera. The imaging condition V is a geometric condition for receiving the light reflected by the subject irradiated with the light, and holds _{the internal parameter v in} and the external parameter v _{ex of the camera at the time of imaging. The illumination condition L holds an illumination vector l n} indicating the direction of illumination at the time of imaging and an illumination intensity l _a indicating the intensity of illumination, which are geometric conditions for irradiating light. The imaging condition V and the illumination condition L of the reference image are information stored in advance in the secondary storage device 304 or the like. The imaging condition V and the illumination condition L of the reference image are also referred to as geometric conditions of the captured image.

The reference image acquisition unit 405 acquires a reference image I representing a first image for reference, which is obtained by separately capturing a subject with a camera for comparison with a rendered image at the time of texture editing. The reference image I is selected from the captured images used for estimating the texture information. The reference image I is information stored in advance in the secondary storage device 304 or the like. Hereinafter, the pixel value at the two-dimensional position (u, v) of the reference image is defined as I (u, v).

Color correction information acquisition unit 406 performs diffuse reflection color _{C d (x, y, z} ) and diffuse reflection correcting value p _d for correcting the specular reflection color _{C s (x, y, z} ) to correct the It acquires specular correction value p _s for. The diffuse reflection correction value p _d and the specular reflection correction value p _s are color correction information. Each correction value is a coefficient for performing color editing by multiplying RGB of each reflected color, and holds a coefficient for each of RGB. The color correction information is acquired by the slider on the slider bar of the area U705 adjusted by the user operation on the GUI screen 700, which will be described in detail later.

Initial rendered image generation unit 407, diffuse reflection color, specular color, diffuse reflection bending strength, specular bending strength, the normal imaging conditions, to produce an initial rendered image R _o from lighting conditions. The diffuse reflection color and the specular reflection color are the diffuse reflection color C _d (x, y, z) and the specular reflection color C _s (x, y, z) acquired by the texture color information acquisition unit 401. The diffuse reflection eccentricity and the specular reflection eccentricity are the diffuse reflection eccentricity _Ad (x, y, z, l _n , v _n ) and the specular reflection eccentricity acquired by the texture eccentricity information acquisition unit 402. _{a s (x, y, z} , l n, v n) is. The normal is the normal n (x, y, z) acquired by the shape information acquisition unit 403. The imaging condition and the illumination condition are the imaging condition V and the illumination condition L acquired by the imaging / illumination condition acquisition unit 404. That is, the initial rendering image generation unit 407, acquired by the acquiring unit 401 to 404, the texture color information of an object, texture deformation information, shape information, and generates an initial rendered image R _o from the geometric conditions of the reference image. Initial rendered image generation unit 407 initial rendered image R _o generated in the display control module 410 and displayed as a rendered image prior to texture editing GUI screen when performing color editing.

Reflected light ratio deriving unit 408, normal, diffuse reflection bending strength, deriving specular bending strength, imaging conditions, the ratio r _d and the ratio r _s of the specular reflection light of the diffuse reflected light in the reference image from the illumination conditions .. The normal is the normal n (x, y, z) acquired by the shape information acquisition unit 403. The diffuse reflection eccentricity and the specular reflection eccentricity are the diffuse reflection eccentricity _Ad (x, y, z, l _n , v _n ) and the specular reflection eccentricity acquired by the texture eccentricity information acquisition unit 402. _{a s (x, y, z} , l n, v n) is. The imaging condition and the illumination condition are the imaging condition V and the illumination condition L acquired by the imaging / illumination condition acquisition unit 404. The reference image I is a reference image I acquired by the reference image acquisition unit 405. That is, the reflected light ratio deriving unit 408, acquired by the acquiring unit 402 to 404, texture deformation information of the object, the shape information, the ratio r _d of the diffuse reflected light in the reference image from the geometric conditions of the reference image and the specular reflection light The ratio r _{s of} is derived.

The rendered image generation unit 409 obtains the following information from the diffuse reflection color, the specular reflection color, the diffuse reflection eccentricity, the mirror reflection eccentricity, the normal, the imaging condition, the illumination condition, the diffuse reflection correction value, and the specular reflection correction value. Derived. In other words, the rendered image generation unit 409, the corrected diffuse reflection color _{C 'd (x, y,} z), corrected specular color _{C' s (x, y,} z) to derive. Rendered image generation unit 409, derived correction after diffuse reflection color _{C 'd (x, y,} z) and the corrected specular color _{C' s (x, y,} z) to derive the rendered image R from. The diffuse reflection color and the specular reflection color are the diffuse reflection color C _d (x, y, z) and the specular reflection color C _s (x, y, z) acquired by the texture color information acquisition unit 401. The diffuse reflection eccentricity and the specular reflection eccentricity are the diffuse reflection eccentricity _Ad (x, y, z, l _n , v _n ) and the specular reflection eccentricity acquired by the texture eccentricity information acquisition unit 402. _{a s (x, y, z} , l n, v n) is. The normal is the normal n (x, y, z) acquired by the shape information acquisition unit 403. The imaging condition and the illumination condition are the imaging condition V and the illumination condition L acquired by the imaging / illumination condition acquisition unit 404. The diffuse reflection correction value and the specular reflection correction value are the diffuse reflection correction value _pd and the specular reflection correction value p _s acquired by the color correction information acquisition unit 406. That is, the rendering image generation unit 409 derives the corrected diffuse reflection color and the corrected specular reflection color from the texture color information, texture variation angle information, shape information, geometric conditions of the reference image, and color correction information of the subject, and then derives the corrected diffuse reflection color and the corrected specular reflection color. Generate a rendered image R.

The display control unit 410 displays the initial rendered image, the ratio of diffuse reflected light, the ratio of specular reflected light, the corrected diffuse reflected color, the corrected mirror reflected color, the reference image, and the rendered image on the GUI screen when color editing is performed. indicate. The initial rendered image is generated by the initial rendering image generation unit 407 initially rendered image R _o. Ratio and the ratio of the specular reflection light of the diffuse reflection light is the ratio r _d and the ratio r _d of the diffuse reflection light of the diffuse reflection light derived by reflected light ratio deriving unit 408. The reference image is the reference image I acquired by the reference image acquisition unit 405. Corrected diffuse reflection color and the corrected specular color is corrected derived by the rendering image generation unit 409 diffuse reflection color _{C 'd (x, y,} z) and the corrected specular color C' _s (x, y , Z). The rendered image is a rendered image R generated by the rendered image generation unit 409.

(Operation in information processing device)
FIG. 6 is a flowchart showing a flow of processing executed by the information processing apparatus 300. FIG. 7 is a diagram showing an example of GUI used in this embodiment. Hereinafter, the details of the operation in the information processing apparatus 300 will be described with reference to the flowchart of FIG. The following operation is started when a user operation is performed on the input device 308 and a predetermined instruction by the user operation is input via the input IF 305. Hereinafter, each step (step) is represented by adding S before the reference numeral.

In S401, the information processing device 300 displays the GUI screen 700 shown in FIG. 7 on the display device 310. On the GUI screen 700, the reference image is displayed, the information necessary for editing the texture is acquired according to the input by the user operation, the rendered image generated based on the acquired information is displayed, and the result of the texture editing is saved. Can be done. Hereinafter, the display control for the display device 310 is performed by the display control unit 410.

In S402, the texture color information acquisition unit 401, the texture variation angle information acquisition unit 402, the shape information acquisition unit 403, the imaging / lighting condition acquisition unit 404, and the reference image acquisition unit 405 each information from the data storage destination set by the user. To get. That is, the texture color information acquisition unit 401 acquires the diffuse reflection color C _d (x, y, z) and the specular reflection color C _s (x, y, z), which are the texture color information of the subject. Textured bending information acquisition unit 402 is a texture deformation color information of the object, the diffuse reflection bending strength _{A d (x, y, z} , l n, v n) and specular bending strength A _s (x, y , Z, l _n , v _n ). The shape information acquisition unit 403 acquires the normal n (x, y, z) which is the shape information of the subject. The imaging / illumination condition acquisition unit 404 acquires the imaging condition V and the illumination condition L of the reference image of the subject. The reference image acquisition unit 405 acquires the reference image I of the subject. The area U701 on the GUI screen 700 is an area for setting a data reading source, and a user sets a data storage destination for reference images, imaging / lighting conditions, shapes, and color correction pre-texture information. When the data read button 711 is pressed by the user operation, the read process of reading the data from the storage destination is started. The color correction pre-texture information includes diffuse reflection color C _d (x, y, z), specular reflection color C _s (x, y, z), diffuse reflection variable angle intensity _Ad (x, y, z, l _n ,). v _n), shows specular bending strength _{a s (x, y, z} , l n, the data and its storage location to hold the storage destination of v _n) in the list. _{The diffuse reflection color C d} (x, y, z) and the specular reflection color C _s (x, y, z) acquired by the texture color information acquisition unit 401 are the initial rendering image generation unit 407 and the rendering image generation unit 409. Is output to. Obtained by texture deformation information acquisition unit 402, the diffuse reflection bending strength _{A d (x, y, z} , l n, v n), specular bending strength _{A s (x, y, z} , l n, v _n ) is output to the initial rendered image generation unit 407, the reflected light ratio derivation unit 408, and the rendered image generation unit 409. The normal n (x, y, z) acquired by the shape information acquisition unit 403 is output to the initial rendering image generation unit 407, the reflected light ratio derivation unit 408, and the rendering image generation unit 409. The imaging condition V and the illumination condition L of the reference image acquired by the imaging / illumination condition acquisition unit 404 are output to the initial rendered image generation unit 407, the reflected light ratio deriving unit 408, and the rendered image generation unit 409. The reference image I acquired by the reference image acquisition unit 405 is output to the display control unit 410 and displayed in the area U702 of the GUI screen 700 in S405 described later.

In S403, the initial rendering image generation unit 407, diffuse reflection color C _d, specular color C _s, diffuse reflection bending strength A _d, specular bending strength A _s, the normal n (x, y, z) , _{The initial rendered image Ro} is generated based on the imaging condition V and the illumination condition L. Diffuse reflection color C _d, specular color C _s, diffuse reflection bending strength A _d, specular bending strength A _s, the normal n (x, y, z) , the imaging condition V, the illumination condition L is the S402 This is the information obtained. Initial rendered image generation unit initial rendered image R _o produced by 407 is output to the display control unit 410, is displayed in the area U703 of the GUI screen 700 in S405 will be described later. The pixel value _Ro (u, v) of the initial rendered image _Ro is derived based on the following equations (1) and (2) by the rendering equation for the reflection characteristic.

Here, the position (x, y, z) on the subject corresponding to the position (u, v) on the image is derived from the internal parameter and the external parameter of the camera. C (x, y, z, l _n , v _n ) represents the reflection characteristic at the position (x, y, z). C (x, y, z, l n, v n) , the diffuse reflection bending strength _{A d (x, y, z} , l n, v n), specular bending strength A _s (x, y, z , L _n , v _n ), multiplied by the diffuse reflection color C _d (x, y, z) and the specular reflection color C _s (x, y, z). Dot represents the inner product operation between vectors, and when the inner product becomes negative, it is replaced with 0. The observation vector v _n (x, y, z) at the position (x, y, z) is the camera position (x _c , y _c , z _c ) and the subject position (x, y) of the external parameter of the imaging condition V. , Z), it is derived from the following equations (3) and (4).

In S404, the reflected light ratio deriving unit 408, the diffuse reflection bending strength A _d, specular bending strength A _s, the normal n (x, y, z) , the imaging condition V, in the reference image I from illumination condition L deriving a ratio r _d and the ratio r _s of the specular reflection light of the diffuse reflected light. Diffuse reflection bending strength A _d, specular bending strength A _s, the normal n (x, y, z) , the imaging condition V, the illumination condition L is data acquired in S402. Ratio r _d and the ratio r _s of the specular reflection light of the diffuse reflected light in the reference image I derived by the reflected light ratio deriving unit 408 is output to the display control unit 410, the area of the GUI screen 700 in S405 described later U704 Is displayed in. Since the ratio of the reflected light is the light in which the specular reflected light is distributed in the regular reflection direction, pay attention to how much the geometric condition at the time of imaging in each pixel of the reference image I deviates from the geometric condition of the regular reflection. Derived by. The deviation of the geometric condition from specular reflection at the time of imaging is the _{half vector h (representing the angle between the incident light vector l n and the emitted light vector v n} (x, y, z) at the position (x, y, z). It is obtained from the angle difference θ between x, y, z) and the normal n (x, y, z). The half vector h (x, y, z) is derived from the following equation (5). Further, the angle difference θ between the half vector h (x, y, z) and the normal n (x, y, z) is derived from the following equation (6).

Here, with respect to the relationship of the angle difference θ between the half vector h (x, y, z) and the normal n (x, y, z) with respect to the imaging geometric condition, FIGS. 8 (a) and 8 (b) are used. Will be explained. Half vector when FIG. 8 (a), the position _{(x 1, y 1, z} 1) output light vector v _n and the incident light vector l _n in the _{(x 1, y 1, z} 1) is in the specular reflection condition It is a figure which shows the relationship between h (x ₁ , y ₁ , z ₁ ) and the normal n (x ₁ , y ₁ , z _1). When the incident light vector l _n and the emitted light vector v _n (x ₁ , y ₁ , z ₁ ) are specular reflection conditions, the half vector h (x ₁ , y ₁ , z ₁ ) and the normal n (x ₁ , z 1) y ₁ , z ₁ ) match. Therefore, the angle difference θ between the half vector h (x ₁ , y ₁ , z ₁ ) and the normal n (x ₁ , y ₁ , z _{1) is 0.} On the other hand, in FIG. 8 (b), the incident light vector l _n and the emitted light vector v _n (x ₂ , y ₂ , z _{2 ) at the positions (x 2} , y ₂ , z ₂ ) deviate from the specular reflection condition. It is a figure which shows the relationship between the half vector h (x ₂ , y ₂ , z ₂ ) and the normal n (x ₂ , y ₂ , z _{2) at the time.} When the incident light vector l _n and the emitted light vector v _n (x ₂ , y ₂ , z ₂ ) deviate from the specular reflection condition, the half vector h (x ₂ , y ₂ , z ₂ ) and the normal n (x) ₂ , y ₂ , z ₂ ) have an angle difference θ. From this, it is possible to determine whether or not the imaging condition is specular reflection from the magnitude of the value of the angle difference θ. Derivation of ratio r _d and the ratio r _s of the specular reflection light of the diffuse reflected light in the reference image I sets a threshold value T to separate the specular reflection condition corresponding to the specular reflection light, the geometric condition corresponding to the diffuse reflected light Do it by. The threshold T, specular bending strength A _s from the intensity distribution with respect to the specular reflection direction of the (x, y, z, l n, v n), the intensity with respect to the maximum intensity around the specular reflection is 10% or less Set the angle difference θ. FIG. 9A is a diagram showing a reference image. FIG. 9B is a diagram showing a frequency distribution of the angle difference between the half vector and the normal in the reference image shown in FIG. 9A. For the reference image shown in FIG. 9 (a), when considering the frequency distribution f (theta) of the angle difference theta in the reference image I as shown in FIG. 9 (b), the diffuse reflection ratio r _d and specular light The ratio r _s is derived from the following equations (7) and (8).

Here, t is the interval width of the angle difference θ in the frequency distribution. T is the threshold value described above.

In S405, the information processing apparatus 300 includes a reference image I acquired in S402, and the initial rendered image R _o generated in S403, the ratio r _d of the diffuse reflection light derived by S404, specular reflection light The ratio r _s of is displayed on the GUI screen 700. The reference image I is displayed in the area U702 of the GUI screen 700. Initial rendered image R _o is displayed in an area U703 of the GUI screen 700. Ratio r _d and the ratio r _s of the specular reflection light of the diffuse reflection light is displayed in the area U704 of the GUI screen 700.

In S406, the color correction information acquisition unit 406 acquires the diffuse reflection correction value p _d and the specular reflection correction value p _s , which are color correction information, from the GUI screen 700. The user gives a coefficient for correcting RGB of each reflected color by operating the slider on the slider bar displayed in the area U705.

In S407, the rendering image generation unit 409 uses the texture color information, texture variation angle information, shape information, geometric conditions, and color correction information to obtain the corrected diffuse reflection color _C'd (x, y, z) and the corrected specular reflection. color _{C 's (x, y,} z) to derive, generate a rendered image R. The rendered image R generated by the rendered image generation unit 409 is output to the display control unit 410, and is displayed in the area U703 of the GUI screen 700 in S408 described later. The texture color information is information including the diffuse reflection color C _d (x, y, z) and the specular reflection color C _s (x, y, z), and is the information acquired in S402. Information texture bending information, including the diffuse reflection bending strength _{A d (x, y, z} , l n, v n) and specular bending strength _{A s (x, y, z} , l n, v n) a This is the information acquired in S402. The shape information is information including the normal n, and is the information acquired in S402. The geometric condition is information including the imaging condition V and the illumination condition L, and is the information acquired in S402. The color correction information is information including the diffuse reflection correction value p _d and the specular reflection correction value p _s , and is the information acquired in S406. The corrected diffuse reflection color _C'd (x, y, z) is derived by the following equation (9). Corrected specular color _{C 's (, x, y} z) is derived by the following equation (10).

The diffuse reflection correction value _pd is a coefficient for correcting RGB of the diffuse reflection color C _d (x, y, z), and the correction is performed by multiplying each of RGB by a value. The specular reflection correction value p _s is a coefficient for correcting RGB of the specular reflection color C _s (x, y, z), and the correction is performed by multiplying each of RGB by a value. Rendered image generation unit 409, the information obtained and the derived corrected diffuse reflection color _{C 'd (x, y,} z) and the corrected specular color _{C' s (x, y,} z) from a rendering image The pixel value R (u, v) of R is derived based on the following equation (11).

Among the above symbols, "C'(x, y, z, l _n , v _n )" is the same as the derivation formula in the initial rendered image generation in S403. C'(x, y, z, l _n , v _n ) represents the reflection characteristic at the position (x, y, z).

In S408, the information processing apparatus 300 displays the rendered image R generated in S407 on the GUI screen 700. The image displayed in the area U703 on the GUI screen 700 is replaced with the rendered image R so that the user can confirm the color-corrected image.

In S409, the information processing apparatus 300 determines whether or not to end the editing work. The editing work includes color editing, which is the editing of the texture information of the subject in the image. When the information processing apparatus 300 obtains a determination result that the editing work is completed (YES in S409), the information processing apparatus 300 shifts the processing to S410. When the information processing apparatus 300 obtains a determination result that the editing work is not completed (NO in S409), the processing shifts to S406, and the processing after S406 is performed. Whether or not to end the editing work is determined by whether or not the editing end button 712 in the area U706 of the GUI screen 700 is pressed by a user operation. When correction is performed by a user operation other than pressing the edit end button 712 (NO in S409), such as color correction being performed by the slider on the slider bar in the area U705 of the GUI screen 700, the process is shifted to S406. Color correction processing is performed after S406.

In S410, the information processing device 300 stores the texture information after color correction. The data storage destination is specified by the user in the area U706 of the GUI screen 700. Here, texture data after the color correction is corrected diffuse reflection color _{C 'd (x, y,} z), corrected specular color _{C' s (x, y,} z), diffuse reflection bending strength A _d ( _{x, y, z, l n} , v n), holds specular bending strength _{a s (x, y, z} , l n, the storage destination of v _n) in the list.

With the above configuration, the ratio of the diffuse reflection light to the specular reflection light in the reference image is derived based on the deviation of the imaging geometric condition of the reference image subject from the normal reflection condition, and is presented to the user through the GUI screen 700. As a result, the user recognizes the ratio of the reflected light that contributes to the color of the reference image, and since the specular reflection light component and the diffuse reflection light component are associated with each other with respect to the color of the image, the diffuse reflection in the texture editing Both light and specular light can be edited appropriately.

In the present embodiment, the ratio of the diffuse reflected light and the specular reflected light in the reference image is derived based on the threshold value set for the angle difference between the half vector and the normal. The ratio of the diffuse reflection light to the specular reflection light in the reference image may be derived based on the diffuse reflection eccentricity and the specular eccentricity at each angle difference. FIG. 10 is a diagram showing the variation intensity with respect to the angle difference between the half vector and the normal line. As shown in FIG. 10, it can be confirmed that the ratio of the diffuse reflected light and the specular reflected light changes for each angle difference. Therefore, it is possible to derive the ratio of each reflected light in consideration of the variation intensity for each angle difference. Ratio r _d of the diffuse reflection light at this time is derived by the following equation (13). The ratio r _s of the specular reflection light at this time is derived by the following equation (14).

Here, θ (x, y, z) is the angle difference θ between the half vector h (x, y, z) at the position (x, y, z) and the normal n (x, y, z). .. f (θ (x, y, z)) is the frequency of the angle difference θ between the half vector h (x, y, z) and the normal n (x, y, z) in the reference image I (u, v). is there. The position (x, y, z) on the subject corresponding to the position (u, v) on the image is obtained from the internal and external parameters of the camera. By these calculation formulas, it is possible to derive a more strict ratio of diffuse reflected light and specular reflected light by considering the variation intensity of diffuse reflected light and specular reflected light for each angle difference.

<< Embodiment 2 >>
In the first embodiment, an example of supporting texture editing by displaying the ratio of diffuse reflected light and specular reflected light in the reference image as a numerical value on the GUI has been described. In the present embodiment, an example in which texture editing is supported by visualizing and displaying the ratio of diffuse reflected light and specular reflected light to each position of the image as an image on the GUI will be described. This makes it possible to confirm the ratio of the reflected light at the position that the user wants to pay attention to.

(Device configuration of information processing device)
Since the device configuration of the information processing device of the present embodiment is the same as the device configuration of the information processing device 300, the description thereof will be omitted.

(Logical configuration of information processing device)
Next, the processing performed by the information processing apparatus of the present embodiment will be described with reference to FIG. FIG. 11 is a block diagram showing a logical configuration example of the information processing apparatus 1100 of the present embodiment. The information processing apparatus 1100 functions as the logical configuration shown in FIG. 11 by executing the program stored in the ROM 303 by the CPU 301 using the RAM 302 as the work memory. It should be noted that it is not necessary that all of the processes shown below are executed by the CPU 301, and even if the information processing apparatus 1100 is configured so that a part or all of the processes are performed by one or a plurality of processing circuits other than the CPU 301. Good.

The information processing device 1100 includes a texture color information acquisition unit 401, a texture variation angle information acquisition unit 402, a shape information acquisition unit 403, an imaging / lighting condition acquisition unit 404, a reference image acquisition unit 405, and a color correction information acquisition unit 406. Further, the information processing device 1100 has an initial rendered image generation unit 407 and a rendered image generation unit 409. The information processing device 1100 further includes a reflected light distribution derivation unit 1101 and a display control unit 1102. The texture color information acquisition unit 401, the texture variation angle information acquisition unit 402, the shape information acquisition unit 403, the imaging / lighting condition acquisition unit 404, the reference image acquisition unit 405, and the color correction information acquisition unit 406 are logical configurations in the information processing device 300. The same applies to the above, and the description thereof will be omitted. Further, the initial rendered image generation unit 407 and the rendered image generation unit 409 are also the same as the logical configuration in the information processing apparatus 300, and the description thereof will be omitted.

The reflected light distribution derivation unit 1101 determines the ratio of the diffuse reflected light and the mirror reflected light at the position (u, v) of the reference image I from the normal line, the diffuse reflection variable angle intensity, the mirror reflection variable angle intensity, the imaging condition, and the illumination condition. It is derived and the reflected light distribution image r is generated. The normal is the normal n (x, y, z) acquired by the shape information acquisition unit 403. The diffuse reflection eccentricity and the specular reflection eccentricity are the diffuse reflection eccentricity _Ad (x, y, z, l _n , v _n ) and the specular reflection eccentricity acquired by the texture eccentricity information acquisition unit 402. _{a s (x, y, z} , l n, v n) is. The imaging condition and the illumination condition are the imaging condition V and the illumination condition L acquired by the imaging / illumination condition acquisition unit 404. That is, the reflected light distribution derivation unit 1101 derives the ratio of the diffuse reflected light and the mirror surface reflected light at the position (u, v) of the reference image I from the texture variation information, the shape information, and the geometric conditions of the reference image of the subject. The reflected light distribution image r is generated.

The display control unit 1102 displays the initial rendered image, the reflected light distribution image, the corrected diffuse reflection color, the corrected mirror reflection color, the reference image, and the rendered image on the GUI screen when color editing is performed. The initial rendered image are generated in the initial rendering image generation unit 407 initially rendered image R _o. The reflected light distribution image is a reflection distribution image r derived by the reflected light distribution derivation unit 1101. The reference image is the reference image I acquired by the reference image acquisition unit 405. Corrected diffuse reflection color and the corrected specular color is corrected derived by the rendering image generation unit 409 diffuse reflection color _{C 'd (x, y,} z) and the corrected specular color C' _s (x, y , Z). The rendered image is a rendered image R generated by the rendered image generation unit 409.

(Operation in information processing device)
FIG. 12 is a flowchart showing a flow of processing executed by the information processing apparatus 1100. FIG. 13 is a diagram showing an example of GUI used in this embodiment. Hereinafter, the details of the operation in the information processing apparatus 1100 will be described with reference to the flowchart of FIG. The following operation is started when a user operation is performed on the input device 308 and a predetermined instruction by the user operation is input via the input IF 305. In the present embodiment, the display control for the display device 310 is performed by the display control unit 1102.

Since S401 to S403 and S406 to S410 have the same processing contents as the steps having the same reference numerals in the first embodiment, the description thereof will be omitted.

In S1101, the reflected light distribution deriving section 1101, the diffuse reflection bending strength A _d, in specular bending strength A _s, the normal n, the imaging condition V, the position of the reference image I from illumination conditions L (u, v) The ratio of the diffuse reflected light and the mirror surface reflected light is derived, and the reflected light distribution image r is generated. The reflected light distribution image r generated by the reflected light distribution derivation unit 1101 is output to the display control unit 1102 and displayed in the area U1301 of the GUI screen 700 in S1102 described later. Diffuse reflection bending strength A _d, specular bending strength A _s, the normal n, the imaging condition V, the illumination condition L, the reference image I is data acquired in S402. First, the position of the reference image I (u, v) the ratio of the specular reflection light at r _s (u, v) is derived by (15) below.

Here, the position (x, y, z) on the subject corresponding to the position (u, v) on the image is associated with the internal parameter and the external parameter of the camera. Next, the ratio r _s (u, v) of the specularly reflected light is converted into the pixel value r (u, v) of the reflected light distribution image r for display on the GUI. In the reflected light distribution image r, the position (u, v) where the ratio of the specular reflected light exceeds 80% is displayed in white, and the position (u, v) where the ratio of the specular reflected light is less than 80% is displayed as black. The color of the image is converted as follows.

In S1102, the information processing apparatus 1100 displays the reference image I acquired in S402, the initial rendered image R ₀ generated in S403, and the reflected light distribution image r generated in S1101 on the GUI screen 1300. To do. The reference image I is displayed in the area U702 of the GUI screen 1300. The initial rendered image R ₀ is displayed in the area U703 of the GUI screen 1300. The reflected light distribution image r is displayed in the area U1301 of the GUI screen 1300.

With the above configuration, the ratio of the reflected light at each position of the reference image is presented to the user as a reflected light distribution image through the GUI screen 1300. As a result, the user recognizes the ratio of the reflected light that contributes to the color at each position of the reference image, and the mirror surface reflected light component and the diffuse reflected light component are correlated with each other with respect to the color of the image. Both diffuse and mirror-reflected light can be edited appropriately in.

FIG. 14 is a diagram showing an example of a specular reflection distribution image displayed in the area U1301 of the GUI screen 1300. FIG. 14A shows the threshold image 1401 displayed in the area U1301. FIG. 14B shows the contour image 1402 displayed in the region U1301. FIG. 14 (c) shows the intensity distribution image 1403 displayed in the region U1301. In the present embodiment, the threshold image 1401 as shown in FIG. 14A is generated as the reflected light distribution image based on whether the ratio of the specularly reflected light exceeds 80%, but the threshold value is changed to another value. You may. Further, in order to present the ratio of specular reflection in more detail, a contour image 1402 in which the threshold value changes depending on the region as shown in FIG. 14 (b) and an intensity distribution image 1403 in which the threshold value changes depending on the position as shown in FIG. 14 (c). May be generated.

<< Embodiment 3 >>
In the first embodiment and the second embodiment, an example in which the texture is edited using only one input reference image has been described. In the present embodiment, an example will be described in which texture editing can be performed by color matching of a plurality of input reference images under geometric conditions by using a reference image group including a plurality of reference images. Further, in the present embodiment, it is possible to efficiently edit the texture by ranking the input images so that the images having geometric conditions suitable for color matching between the reference image and the rendered image can be referred to. Become.

(Device configuration of information processing device)
Since the device configuration of the information processing device of the present embodiment is the same as the device configuration of the information processing device 300, the description thereof will be omitted.

(Logical configuration of information processing device)
Next, the processing performed by the information processing apparatus of the present embodiment will be described with reference to FIG. FIG. 15 is a block diagram showing a logical configuration example of the information processing apparatus 1500 of the present embodiment. The information processing device 1500 functions as the logical configuration shown in FIG. 15 by executing the program stored in the ROM 303 by the CPU 301 using the RAM 302 as the work memory. It should be noted that it is not necessary that all of the processes shown below are executed by the CPU 301, and even if the information processing apparatus 1500 is configured so that a part or all of the processes are performed by one or a plurality of processing circuits other than the CPU 301. Good.

The information processing device 1500 includes a texture color information acquisition unit 401, a texture variation angle information acquisition unit 402, a shape information acquisition unit 403, and a color correction information acquisition unit 406. Further, the information processing device 1500 has an initial rendered image generation unit 407 and a rendered image generation unit 409. The information processing device 1500 further includes an imaging / lighting condition group acquisition unit 1501, a reference image group acquisition unit 1502, a reference image number acquisition unit 1503, a reflected light ratio group / reference image table creation unit 1504, and a reference image / geometric condition selection unit. It has 1505 and a display control unit 1506. The texture color information acquisition unit 401, the texture variation angle information acquisition unit 402, the shape information acquisition unit 403, and the color correction information acquisition unit 406 are the same as the logical configuration in the information processing device 300, and the description thereof will be omitted. Further, the initial rendered image generation unit 407 and the rendered image generation unit 409 are also the same as the logical configuration in the information processing apparatus 300, and the description thereof will be omitted.

The imaging / lighting condition group acquisition unit 1501 acquires the imaging condition group V _n and the illumination condition group L _{n corresponding to the separately acquired reference image group In n} , which will be described later. Here, n is an index for the reference image, and numbers 1 ... q are assigned to q images. _{The image pickup condition group V n} and the illumination condition group L _n acquired by the image pickup / illumination condition group acquisition unit 1501 are output to the reflected light ratio group / reference image table creation unit 1504 and the reference image / geometric condition selection unit 1505. The imaging condition group V _n and the illumination condition group L _n of the reference image group are information stored in advance in the secondary storage device 304 or the like. The imaging condition group V _n and the illumination condition group L _n of the reference image group are also referred to as a geometric condition group of the reference image group.

Reference image group acquiring unit 1502 acquires a reference image group I _n for comparison with rendered images during texture editing. Reference image group I _n acquired by the reference image group acquiring unit 1502 is output to the reference image-geometric-condition selection unit 1505. The reference image group I _n is a pre-stored information such as secondary storage devices 304.

The reference image number acquisition unit 1503 acquires the reference image number m used for selecting the reference image. The reference image number m acquired by the reference image number acquisition unit 1503 is output to the reference image / geometric condition selection unit 1505. The reference image number m is obtained from the reference number of the area U1702 selected by the user operation on the GUI screen 1700, which will be described in detail later.

Reflected light ratio group and the reference image table creation unit 1504, normal, diffuse reflection bending strength, specular bending strength, the imaging condition group, the ratio group r _{d_n} of diffuse reflected light in the reference image I _n the illumination condition group _{The ratio group r s_n} of the mirror-reflected light is derived. The normal is the normal n (x, y, z) acquired by the shape information acquisition unit 403. The diffuse reflection eccentricity and the specular reflection eccentricity are the diffuse reflection eccentricity _Ad (x, y, z, l _n , v _n ) and the specular reflection eccentricity acquired by the texture eccentricity information acquisition unit 402. _{a s (x, y, z} , l n, v n) is. The imaging condition group and the illumination condition group are the imaging condition group V _n and the illumination condition group L _n acquired by the imaging / illumination condition group acquisition unit 1501. That is, the reflected light fraction group and the reference image table creation unit 1504, shape information, to derive the texture deformation information, the rate set of the diffuse reflection light in the reference image I _n from the geometric condition group and percentage groups of the specular reflection light. Then, the reflected light fraction group and the reference image table creation unit 1504 selects the rate group ratio group and specular reflection light of the diffuse reflected light in the derived reference image I _n, the geometric conditions suitable for color matching between the rendered image A reference image table T is created for this purpose. Created by the reflected light fraction group and the reference image table creation unit 1504, the ratio group r _{d_n} of diffuse reflected light, the ratio group r _{s_n} specular light, the reference image table T is outputted to the reference image-geometric-condition selection unit 1505 Will be done.

The reference image / geometric condition selection unit 1505 is used to refer to the reference image and its geometric condition and diffusion from the ratio group of diffuse reflected light and mirror surface reflected light, the reference image table, the reference image group, the geometric condition group, and the reference image number when editing the texture. Select and acquire the ratio of reflected light and the ratio of mirror-reflected light. The diffused reflected light ratio group, the mirror surface reflected light ratio group, and the reference image table are the reflected light ratio group / reference image table creation unit 1504, and the diffused reflected light ratio group _{rd_n} and the mirror surface reflected light ratio group. r _{s_n} , reference image table T. Reference image group is a reference image group I _n acquired by the reference image group acquiring unit 1502. Geometric condition group is a geometric condition group acquired by the imaging-illumination condition group acquiring unit 1501 includes an imaging condition group V _n and the illumination condition group L _n corresponding to the reference image group I _n. The reference image number is the reference image number m acquired by the reference image number acquisition unit 1503. The imaging condition V and the illumination condition L, which are the geometric conditions of the reference image selected by the reference image / geometric condition selection unit 1505, are output to the rendering image generation unit 409.

The display control unit 1506 displays the initial rendered image, the reference image, the ratio of diffuse reflected light, the ratio of specular reflected light, the corrected diffuse reflected color, the corrected mirror reflected color, and the rendered image on the GUI screen when color editing is performed. indicate. The initial rendered image is generated by the initial rendering image generation unit 407 initially rendered image R _o. The reference image is the reference image I selected by the reference image / geometric condition selection unit 1505. Ratio and the ratio of the specular reflection light of the diffuse reflection light is the ratio r _d and the ratio r _d of the diffuse reflection light is selected in the reference image-geometric-condition selection unit 1505 diffuse reflected light. Corrected diffuse reflection color and the corrected specular color is corrected derived by the rendering image generation unit 409 diffuse reflection color _{C 'd (x, y,} z) and the corrected specular color C' _s (x, y , Z). The rendered image is a rendered image R generated by the rendered image generation unit 409.

(Operation in information processing device)
FIG. 16 is a flowchart showing a flow of processing executed by the information processing apparatus 1500. FIG. 17 is a diagram showing an example of GUI used in this embodiment. Hereinafter, the details of the operation in the information processing apparatus 1500 will be described with reference to the flowchart of FIG. The following operation is started when a user operation is performed on the input device 308 and a predetermined instruction by the user operation is input via the input IF 305. In the present embodiment, the display control for the display device 310 is performed by the display control unit 1506.

Since S401, S403, and S406 to S410 have the same processing contents as the steps having the same reference numerals in the first embodiment, the description thereof will be omitted.

In S1501, the texture color information acquisition unit 401, the texture variation angle information acquisition unit 402, the shape information acquisition unit 403, the imaging / lighting condition group acquisition unit 1501, and the reference image group acquisition unit 1502 are stored from the data storage destination set by the user. Get each information. That is, the texture color information acquisition unit 401 acquires the diffuse reflection color C _d (x, y, z) and the specular reflection color C _s (x, y, z), which are the texture color information of the subject. Textured bending information acquisition unit 402, a texture deformation color information of the object, the diffuse reflection bending strength _{A d (x, y, z} , l n, v n) and specular bending strength A _s (x, y , Z, l _n , v _n ). The shape information acquisition unit 403 acquires the normal n (x, y, z) which is the shape information of the subject. The imaging / illumination condition group acquisition unit 1501 acquires the imaging condition group V _n and the illumination condition group L _n , which are geometric condition groups of the reference image group of the subject. Reference image group acquiring unit 1502 acquires a reference image group I _n of the subject. The area U1701 on the GUI screen 1700 is an area for setting a data read source, and a user sets a data storage destination for a reference image group, an imaging / lighting condition group, a shape, and color correction pre-texture information. When the data read button 711 is pressed by the user operation, the read process of reading the data from the storage destination is started. _{The diffuse reflection color C d} (x, y, z) and the specular reflection color C _s (x, y, z) acquired by the texture color information acquisition unit 401 are the initial rendering image generation unit 407 and the rendering image generation unit 409. Is output to. Obtained in texture bending information acquisition unit 402, the diffuse reflection bending strength A _d, specular bending strength A _s is the initial rendering image generator 407, the reflected light fraction group and the reference image table creation unit 1504, the rendering image It is output to the generation unit 409. The normal n (x, y, z) acquired by the shape information acquisition unit 403 is output to the initial rendering image generation unit 407, the reflected light ratio group / reference image table creation unit 1504, and the rendering image generation unit 409. _{The image pickup condition group V n} and the illumination condition group L _n of the reference image group acquired by the image pickup / illumination condition group acquisition unit 1501 are sent to the reflected light ratio group / reference image table creation unit 1504 and the reference image / geometric condition selection unit 1505. It is output. Reference image group I _n of the object obtained by the reference image group acquiring unit 1502 is output to the reference image-geometric-condition selection unit 1505. _{Of the reference image group In n} of the subject acquired by the reference image group acquisition unit 1502, the reference image I corresponding to the reference image number m acquired by the reference image number acquisition unit 1503 described later is GUI in S1504 described later. It is displayed in the area U1702 of the screen 1700.

In S1502, the reflected light ratio group / reference image table creation unit 1504 has the ratio of diffuse reflected light corresponding to the reference image group from the normal line, the diffuse reflection eccentricity intensity, the mirror reflection eccentricity intensity, the imaging condition group, and the illumination condition group. Create the group r _{d_n} and the ratio group r _{s_n of the mirror-reflected light.} The normal line, diffuse reflection eccentricity intensity, and specular reflection eccentricity intensity are the normal line n (x, y, z) and the diffuse reflection eccentricity intensity _Ad (x, y, z, l _n ,) acquired in S1501. v _n), a specular bending strength _{a s (x, y, z} , l n, v n). The imaging condition group and the illumination condition group are also the imaging condition group V _n and the illumination condition group L _n acquired in S1501. Then, the reflected light fraction group and the reference image table creation unit 1504, the reference image table from ranking of the reference image group based on each rate group at the rate group r _{s_n} ratio group r _{d_n} and specular reflected light created diffuse reflected light Create T. The creation of the reference image table T will be described in detail. In texture editing, the user edits the color of diffusely reflected light and the color of specularly reflected light, respectively. Therefore, it is desirable to select an image containing a large amount of color information of each reflected light as the reference image used for color matching. Therefore, by ranking the reference image group in descending order of the ratio of diffuse reflection and the ratio of specular reflection, the reference image suitable for color matching can be preferentially referred to. First, S404 and similar, based on the imaging condition group V _n of the reference image group I _n the illumination condition group L _n-, the proportion group ratio group r _{d_n} the specular reflection light of the diffuse reflection light for each reference image in the first embodiment Derivation of r _{s_n.} Here, n corresponds to the index of the reference image group. _{Next, a diffuse reflection reference image table T d in} which the indexes of the reference image groups are arranged in descending order of diffuse reflection _{and a specular reflection reference image table T s in} which the indexes of the reference image groups are arranged in descending order of specular reflection are created. Finally, using the diffuse reflection reference image table T _d and the specular reflection reference image table T _s , the reference image table T is created as shown in the following equation.

In S1503, the reference image number acquisition unit 1503 acquires the reference image number m selected by the user operation in the area U1702 of the GUI screen 1700. FIG. 17 shows an example in which the selected 1 is acquired as the reference image number m by pressing the left arrow button or the right arrow button by the user operation with the reference number. The acquisition of the reference image number is not limited to this, and a number directly input by the user operation may be acquired as the reference image number. Thereafter, reference image-geometric-condition selection unit 1505, the reference image group I _n, the imaging condition group V _n, the lighting condition group L _n, the reference image table T, the ratio group r _{d_n} of diffuse reflected light, the proportion group specular light Select the following information from r _{s_n and the reference image number m.} That is, reference image-geometric-condition selection unit 1505, first GUI screen 1700 reference image I to be displayed on the proportion r _d of the diffuse reflected light, the proportion of specularly reflected light r _s, imaging conditions V, and illumination conditions L select. Reference image group I _n, the imaging condition group V _n, the lighting condition group L _n is the information acquired in S1501. The reference image table T, the ratio group r _{d_n} of the diffuse reflected light, and the ratio group r _{s_n} of the specular reflected light are the information created in S1502. The reference image number m is information acquired by the reference image acquisition unit 1503. The reference image number m is designated from the reference number in the area U1702 of the GUI screen 1700, and is usually set to an initial value of 1. The initial value is not limited to this and may be other than 1. The reference image number m acquired by the reference image acquisition unit 1503 is output to the reference image / geometric condition selection unit 1505. Here, on the basis of the reference image number m obtained, a reference image-geometric-condition selection unit 1505 obtains the index of the m-th reference image group stored in the reference image table T, GUI from the reference image group I _n Select the reference image I to be displayed on the screen. Reference image-geometric-condition selection unit 1505, further, the ratio r _d of the diffuse reflected light from the ratio group r _{d_n} of diffuse reflected light, the proportion of specularly reflected light from the ratio group r _{s_n} specular light r _s, the imaging condition group V _The imaging condition V _{is selected from n,} and the illumination condition L is selected from the illumination condition group L n. Reference image-geometric-condition reference image I which is selected by the selection section 1505, the ratio r _d of the diffuse reflected light, the ratio r _s of the specular reflection light is output to the display control unit 1506, the GUI screen 1700 at S1504 will be described later It is displayed in the areas U1702 and U704.

In S1504, information processing apparatus 1500, the reference image I selected in S1503, the ratio r _d of the diffuse reflected light, the initial rendered image R _o the GUI screen generated by the proportion of specularly reflected light r _s, S403 1700 Display on. The reference image I is displayed in the area U1702 of the GUI screen 1700. The initial rendered image R ₀ is displayed in the area U703 of the GUI screen 1700. Ratio r _d and the ratio r _s of the specular reflection light of the diffuse reflected light is respectively displayed in the region U704.

In S1505, the information processing apparatus 1500 determines whether or not the reference image number has been changed by the user operation. The reference image number is changed by editing the numerical value of the reference number in the area U1702 of the GUI screen 1700. When the information processing apparatus 1500 obtains a determination result that the reference image number has not been changed by the user operation (NO in S1505), the information processing apparatus 1500 shifts the process to S409. When the information processing apparatus 1500 obtains the determination result that the reference image number has been changed by the user operation (YES in S1505), the information processing apparatus 1500 shifts the process to S1506.

In S1506, the reference image / geometric condition selection unit 1505 refers to the index of the reference image table T corresponding to the changed reference image number m. This reference, reference image-geometric-condition selection unit 1505, the reference image group I _n, the ratio group r _{d_n} of diffuse reflected light, the ratio group r _{s_n} specular light, the imaging condition group V _n, the lighting condition group L _n , Select the following information. That is, reference image-geometric-condition selection unit 1505, a reference image I, the ratio r _d diffuse specular light, the ratio r _s of the specular reflection light, the imaging condition V, selects the illumination condition L. The information selected by the reference image / geometric condition selection unit 1505 is output to the rendering image generation unit 409. After these data are selected, the rendered image generation unit 409 generates a rendered image R corresponding to the reference image I selected by the reference image / geometric condition selection unit 1505. The information processing apparatus 1500 updates the images to be displayed on U1702 and U703 of the GUI screen 1700 to the selected reference image I and the rendered image R, respectively. The information processing device 1500 updates the image displayed on the U1702 of the GUI screen 1700 with the reference image I selected by the reference image / geometric condition selection unit 1505. Further, the image displayed on the U703 of the GUI screen 1700 is updated to the rendered image R generated by the rendered image generation unit 409 based on the information selected by the reference image / geometric condition selection unit 1505.

With the above configuration, a reference image group for referencing a plurality of geometric conditions in texture editing is input, and texture editing under a plurality of geometric conditions is executed. Editing can be made more efficient by determining the priority of the reference image in editing based on the ratio of diffuse reflected light and specularly reflected light.

In the present embodiment has been numeric display used in Embodiment 1 as a display on the GUI screen ratio r _d and the ratio r _s of the specular reflection light of the diffuse reflected light, the threshold image shown in the second embodiment, the contour line An image, an intensity distribution image, or the like may be used as a ratio display of reflected light.

<< Embodiment 4 >>
In the first to third embodiments, an example in which the entire subject is uniformly edited when the texture is edited has been described. In the present embodiment, an example in which a region for texture editing is specified by the user and diffuse reflection and specular reflection in that region are edited will be described.

(Device configuration of information processing device)
Since the device configuration of the information processing device of the present embodiment is the same as the device configuration of the information processing device 300, the description thereof will be omitted.

(Logical configuration of information processing device)
Next, the processing performed by the information processing apparatus of the present embodiment will be described with reference to FIG. FIG. 18 is a block diagram showing a logical configuration example of the information processing device 1800 of the present embodiment. The information processing device 1800 functions as the logical configuration shown in FIG. 18 by executing the program stored in the ROM 303 by the CPU 301 using the RAM 302 as the work memory. It should be noted that it is not necessary that all of the processes shown below are executed by the CPU 301, and even if the information processing device 1800 is configured so that a part or all of the processes are performed by one or a plurality of processing circuits other than the CPU 301. Good.

The information processing device 1800 includes a texture color information acquisition unit 401, a texture variation angle information acquisition unit 402, a shape information acquisition unit 403, an imaging / lighting condition acquisition unit 404, a reference image acquisition unit 405, a color correction information acquisition unit 406, and an initial rendering. It has an image generation unit 407. The information processing device 1800 further includes a color correction region acquisition unit 1801, a reflected light ratio derivation unit 1802, a rendered image generation unit 1803, and a display control unit 1804. The texture color information acquisition unit 401, the texture variation angle information acquisition unit 402, the shape information acquisition unit 403, the imaging / lighting condition acquisition unit 404, the reference image acquisition unit 405, and the color correction information acquisition unit 406 are logical configurations in the information processing device 300. Since it is the same as, the description thereof will be omitted. Further, since the initial rendered image generation unit 407 has the same logical configuration as that of the information processing apparatus 300, the description thereof will be omitted.

The color correction area acquisition unit 1801 acquires the color correction area w, which is an area for performing color correction on the image. The color correction area w acquired by the color correction area acquisition unit 1801 is output to the reflected light ratio derivation unit 1802 and the rendered image generation unit 1803.

Reflected light ratio deriving unit 1802, normal, diffuse reflection bending strength, specular bending strength, imaging conditions, lighting conditions, specular and percentage r _d of the diffuse reflection light in the color correction region of the reference image I from the color correction region The ratio r _{s of the} reflected light is derived. The normal is the normal n (x, y, z) acquired by the shape information acquisition unit 403. The diffuse reflection eccentricity and the specular reflection eccentricity are the diffuse reflection eccentricity _Ad (x, y, z, l _n , v _n ) and the specular reflection eccentricity acquired by the texture eccentricity information acquisition unit 402. _{a s (x, y, z} , l n, v n) is. The imaging condition and the illumination condition are the imaging condition V and the illumination condition L acquired by the imaging / illumination condition acquisition unit 404. The color correction area w is a color correction area w acquired by the color correction area acquisition unit 1801. That is, the reflected light ratio deriving unit 1802 is the ratio of the diffuse reflected light in the color correction region of the reference image I from the shape information, the texture variation information, the geometric condition, and the color correction region acquired by the acquisition units 402 to 404, 1801. to derive the ratio r _s of r _d and specular reflection light.

The rendered image generation unit 1803 is based on the diffuse reflection color, the specular reflection color, the diffuse reflection eccentricity, the mirror reflection eccentricity, the normal, the imaging condition, the illumination condition, the color correction area, the diffuse reflection correction value, and the specular reflection correction value. , Derives the following information. In other words, the rendered image generation unit 1803, the corrected diffuse reflection color _{C 'd (x, y,} z), corrected specular color _{C' s (x, y,} z) to derive. The rendered image generation unit 1803, since the obtained information, derived correction after diffuse reflection color _{C 'd (x, y,} z) and the corrected specular color _{C' s (x, y,} z) and rendered The image R is derived. The diffuse reflection color and the specular reflection color are the diffuse reflection color C _d (x, y, z) and the specular reflection color C _s (x, y, z) acquired by the texture color information acquisition unit 401. The diffuse reflection eccentricity and the specular reflection eccentricity are the diffuse reflection eccentricity _Ad (x, y, z, l _n , v _n ) and the specular reflection eccentricity acquired by the texture eccentricity information acquisition unit 402. _{a s (x, y, z} , l n, v n) is. The normal is the normal n (x, y, z) acquired by the shape information acquisition unit 403. The imaging condition and the illumination condition are the imaging condition V and the illumination condition L acquired by the imaging / illumination condition acquisition unit 404. The color correction area is the color correction area w acquired by the color correction area acquisition unit 1801. The diffuse reflection correction value and the specular reflection correction value are the diffuse reflection correction value _pd and the specular reflection correction value p _s acquired by the color correction information acquisition unit 406. That is, the rendering image generation unit 1803 derives the corrected diffuse reflection color and the corrected mirror reflection color from the texture color information, texture variation angle information, shape information, geometric condition of the reference image, color correction area, and color correction information of the subject. Then, the rendered image R is generated.

The display control unit 1804 colors the initial rendered image, the ratio of the diffuse reflected light and the ratio of the specular reflected light in the color correction region of the reference image I, the corrected diffuse reflected color, the corrected mirror surface reflected color, the reference image, and the rendered image. Displayed on the GUI screen when editing. The initial rendered image is generated by the initial rendering image generation unit 407 initially rendered image R _o. Ratio of proportions and specular reflection light of the diffuse reflection light is the ratio r _s ratio r _d and specular reflection light of the diffuse reflection light in the color correction region of the reference image I derived by the reflected light ratio deriving unit 1802. The reference image is the reference image I acquired by the reference image acquisition unit 405. Corrected diffuse reflection color and corrected specular color is corrected derived by rendering image generating unit 1803 diffuse reflection color _{C 'd (x, y,} z) and the corrected specular color C' _s (x, y , Z). The rendered image is a rendered image R generated by the rendered image generation unit 1803.

(Operation in information processing device)
FIG. 19 is a flowchart showing a flow of processing executed by the information processing apparatus 1800. FIG. 20 is a diagram showing an example of GUI used in this embodiment. Hereinafter, the details of the operation in the information processing apparatus 1800 will be described with reference to the flowchart of FIG. The following operation is started when a user operation is performed on the input device 308 and a predetermined instruction by the user operation is input via the input IF 305. In the present embodiment, the display control for the display device 310 is performed by the display control unit 1804.

Since S401 to S403, S405, S406, and S408 to S410 have the same processing contents as the steps having the same reference numerals in the first embodiment, the description thereof will be omitted.

In S1801, the reflected light ratio deriving unit 1802 diffuses from the diffuse reflection eccentricity, the mirror reflection eccentricity, the normal line, the imaging condition, the illumination condition, and the color correction area w to the area set in the color correction area w. to derive the ratio r _d and the ratio r _s of the specular reflection light of the reflected light. Diffuse reflection eccentricity, specular reflection eccentricity, normal, imaging conditions, and illumination conditions are the diffuse reflection eccentricity _Ad (x, y, z, l _n , v _n ) acquired in S402, mirror surface. reflection bending strength _{a s (x, y, z} , l n, v n), the normal n (x, y, z) , the imaging condition V, a lighting condition L. The color correction area w is information acquired by the color correction area acquisition unit 1801. The color correction area w is data in which 1 is filled in the position (u, v) on the image of the subject to be corrected and 0 is filled in the position (u, v) not to be corrected, and the initial value. 1 is set for all areas. Therefore, at S1801, the reflected light fraction deriving section 1802 derives a ratio r _d and the ratio r _s of the specular reflection light of the diffuse reflection light in the entire reference image I. A method of deriving these proportions r _d, r _s is the same as S404 of the first embodiment, description thereof is omitted. The proportion of the diffuse reflection light derived by reflected light ratio deriving unit 1802 r _d, an initial rendered image R ₀ generated by the reference image I, S403 obtained in the ratio r _s, S402 of the specular reflected light, the processing of S1801 It is displayed on the GUI screen 2000 in S405 following. Ratio r _d and the ratio r _s of the specular reflection light of the diffuse reflection light is displayed in the area U704 of the GUI screen 2000. The reference image I is displayed in the area U702 of the GUI screen 2000. The initial rendered image R ₀ is displayed in the area U703 of the GUI screen 2000.

In S1802, the color correction area acquisition unit 1801 acquires the color correction area w indicating the area to be color-corrected. The acquisition of the color correction area in S1802 is executed by directly designating the area for color correction by the user operation by the user via the GUI screen 2000. The area U2002 on the GUI screen 2000 is an example thereof, and is specified by enclosing the area on an image to be color-corrected by a user operation. 1 is set in the color correction area w in the position (u, v) corresponding to the area U2002 surrounded by the thick line, and the color correction area (u, v) in the position (u, v) corresponding to the outside of the area U2002 surrounded by the thick line is set. 0 is set in w.

In S1803, the reflected light ratio deriving unit 1802 sets the value w (u, v) of the color correction region w from the color correction region, the diffuse reflection eccentricity intensity, the mirror reflection eccentricity intensity, the normal line, the imaging condition, and the illumination condition. deriving a ratio r _d and the ratio r _s of the specular reflection light of the diffuse reflected light in 1 become area. The color correction area is the color correction area w acquired by the color correction area acquisition unit 1801 in S1802. The diffuse reflection eccentricity and the specular reflection eccentricity are the diffuse reflection eccentricity _Ad (x, y, z, l _n , v _n ) acquired by the texture variation information acquisition unit 402 in S402, and the mirror surface. reflection bending strength _{a s (x, y, z} , l n, v n) is. The normal is the normal n (x, y, z) acquired by the shape information acquisition unit 403 in S402. The imaging condition and the illumination condition are the imaging condition V and the illumination condition L acquired by the imaging / illumination condition acquisition unit 404 in S402. The value w (u, v) color correction area w = 1 and becomes the ratio r _s of the proportion of diffuse reflected light r _d and specular reflected light region, the value of the color correction region w the derivation process in S404 of Embodiment 1 It is derived by executing for the area where w (u, v) = 1.

In S1803, the rendering image generation unit 1803 derives the corrected diffuse reflection color and the corrected specular reflection color from the texture color information, texture variation angle information, shape information, geometric conditions, color correction area, and color correction information. The texture color information is information including the diffuse reflection color C _d (x, y, z) and the specular reflection color C _s (x, y, z), and is the information acquired in S402. Information texture bending information, including the diffuse reflection bending strength _{A d (x, y, z} , l n, v n) and specular bending strength _{A s (x, y, z} , l n, v n) a This is the information acquired in S402. The shape information is information including the normal n, and is the information acquired in S402. The geometric condition is information including the imaging condition V and the illumination condition L, and is the information acquired in S402. The color correction area is information including the color correction area w, and is the information selected in S1802. The color correction information is information including the diffuse reflection correction value p _d and the specular reflection correction value p _s , and is the information acquired in S406. The rendered image generation unit 1803, acquired by information derived the corrected diffuse reflection color _{C 'd (x, y,} z), corrected specular color _{C' s (x, y,} z) from the rendering Generate image R. First, the corrected diffuse reflection color _C'd (x, y) obtained _{by applying the diffuse reflection correction value pd} and the specular reflection correction value p _s to the region where the value w (u, v) = 1 of the color correction region w. , z) and the corrected specular color _{C 's (x, y,} z) to derive.

Here, the position (x, y, z) on the subject corresponding to the position (u, v) on the image is obtained from the internal parameter and the external parameter of the camera. Derived correction after diffuse reflection color _{C 'd (x, y,} z) and the corrected specular color _{C' s (x, y,} z) from the acquired information, the pixel value R (u rendered image R, v) is derived based on the equations (11) and (12) in S407 of the first embodiment. The rendered image R generated by the rendered image generation unit 1803 is output to the display control unit 1506, and is displayed in the area U703 of the GUI screen 2000 in S408 following the processing of S1803.

With the above configuration, the texture editing for each area is executed by designating the area for texture editing from the GUI screen 2000 by the user operation. As a result, different texture editing can be applied to each material and color correction corresponding to the material can be performed. Therefore, even if the subject is composed of a plurality of materials having different textures, the texture editing is performed for each area according to the material. be able to.

In the present embodiment, the area for color editing is specified, but the color of the material to be selected, the feature amount of the texture distribution, etc. are specified, and an area similar to the specified information is selected inside the information processing apparatus. Then, you may execute texture editing. Although performed numerical display used in Embodiment 1 as a display on the GUI screen ratio r _d and the ratio r _s of the specular reflection light of the diffuse reflected light, the threshold image shown in the second embodiment, the contour line image, the intensity A distribution image or the like may be used as a ratio display of reflected light. Further, color correction may be performed using reference images under a plurality of geometric conditions as in the third embodiment.

<< Embodiment 5 >>
In the first to fourth embodiments, the diffuse reflection color and the specular reflection color are changed independently in the texture editing. In the present embodiment, the color correction value is not given to each of the diffuse reflection color and the mirror surface reflection color, but the color correction value is given based on the color in which the reflected lights are added, and the color is diffused inside the information processing apparatus. An example of applying color correction to each reflected color based on the ratio of the reflected light and the mirror surface reflected light will be described. As a result, the user can perform color correction without worrying about the ratio of reflected light.

(Device configuration of information processing device)
Since the device configuration of the information processing device of the present embodiment is the same as the device configuration of the information processing device 300, the description thereof will be omitted.

(Logical configuration of information processing device)
Next, the processing performed by the information processing apparatus of the present embodiment will be described with reference to FIG. FIG. 21 is a block diagram showing a functional configuration example of the information processing apparatus 2100 of the present embodiment. The information processing apparatus 2100 functions as the logical configuration shown in FIG. 21 by executing the program stored in the ROM 303 by the CPU 301 using the RAM 302 as the work memory. It should be noted that it is not necessary that all of the processes shown below are executed by the CPU 301, and even if the information processing apparatus 2100 is configured so that a part or all of the processes are performed by one or a plurality of processing circuits other than the CPU 301. Good.

The information processing device 2100 includes a texture color information acquisition unit 401, a texture variation angle information acquisition unit 402, a shape information acquisition unit 403, an imaging / lighting condition acquisition unit 404, a reference image acquisition unit 405, an initial rendering image generation unit 407, and reflected light. It has a ratio derivation unit 408. The information processing device 2100 further includes a color correction information acquisition unit 2101, a diffuse reflection color / specular reflection color correction unit 2102, a rendering image generation unit 2103, and a display control unit 2104. The texture color information acquisition unit 401, the texture variation angle information acquisition unit 402, the shape information acquisition unit 403, the imaging / lighting condition acquisition unit 404, and the reference image acquisition unit 405 have the same logical configuration as that of the information processing device 300. The explanation is omitted. Further, since the initial rendered image generation unit 407 and the reflected light ratio derivation unit 408 are the same as the logical configuration in the information processing apparatus 300, the description thereof will be omitted.

The color correction information acquisition unit 2101 acquires a color correction value p which is color correction information for correcting _{the diffuse reflection color C d} (x, y, z) and the specular reflection color C _{s (x, y, z).} To do. The color correction value p is a coefficient for adjusting the color of the rendered image R, and holds a coefficient for each of RGB. The color correction value p acquired by the color correction information acquisition unit 2101 is output to the diffuse reflection color / specular reflection color correction unit 2102.

The diffuse reflection color / specular reflection color correction unit 2102 corrects from the ratio of diffuse reflection light, the ratio of specular reflection light, diffuse reflection color, specular reflection color, diffuse reflection variation intensity, mirror reflection variation intensity, and color correction value. rear diffuse reflection color _{C 'd (x, y,} z), corrected specular color _{C' s (x, y,} z) to derive. Diffusion rate and the ratio of the specular reflection light of the reflected light is the ratio r _d and the ratio r _s of the specular reflection light of the diffuse reflection light derived by reflected light ratio deriving unit 408. The diffuse reflection color and the specular reflection color are the diffuse reflection color C _d (x, y, z) and the specular reflection color C _s (x, y, z) acquired by the texture color information acquisition unit 401. The diffuse reflection eccentricity and the specular reflection eccentricity are the diffuse reflection eccentricity _Ad (x, y, z, l _n , v _n ) and the specular reflection eccentricity acquired by the texture eccentricity information acquisition unit 402. _{a s (x, y, z} , l n, v n) is. _{The diffuse reflection color C d} (x, y, z) and the specular reflection color C _s (x, y, z) acquired by the texture color information acquisition unit 401. The color correction value is the color correction value p acquired by the color correction information acquisition unit 2101. That is, the diffuse reflection color / mirror surface reflection color correction unit 2102 is based on the ratio of the diffuse reflection light, the ratio of the mirror surface reflection light, the texture color information, the texture variation angle information, and the color correction information, and the corrected diffuse reflection color _C'd (x). , y, z) and the corrected specular color C _'s (x, y, derives the z). Then, after correction derived by the diffuse reflection color, specular color correction unit 2102 diffuse reflection color _{C 'd (x, y,} z) and the corrected specular color _{C' s (x, y,} z) is rendered It is output to the image generation unit 2103 and the display control unit 2104.

The rendered image generation unit 2103 derives the rendered image R from the corrected diffuse reflection color, the corrected mirror surface reflection color, the diffuse reflection variation intensity, the mirror reflection variation intensity, the normal, the imaging condition, and the illumination condition. The corrected diffuse reflection color and the corrected specular reflection color are the corrected diffuse reflection color _C'd (x, y, z) and the corrected specular reflection color C'd (x, y, z) derived by the diffuse reflection color / specular reflection color correction unit 2102. _s (x, y, z). The diffuse reflection eccentricity and the specular reflection eccentricity are the diffuse reflection eccentricity _Ad (x, y, z, l _n , v _n ) and the specular reflection eccentricity acquired by the texture eccentricity information acquisition unit 402. _{a s (x, y, z} , l n, v n) is. The normal is the normal n (x, y, z) acquired by the shape information acquisition unit 403. The imaging condition and the illumination condition are the imaging condition V and the illumination condition L acquired by the imaging / illumination condition acquisition unit 404. That is, the rendered image generation unit 2103 derives the rendered image R from the corrected diffuse reflection color, the corrected specular reflection color, the texture variation angle information, the shape information, and the geometric conditions of the reference image.

The display control unit 2104 displays the initial rendered image, the reference image, the corrected diffuse reflection color, the corrected specular reflection color, and the rendered image on the GUI screen when color editing is performed. The initial rendered image is generated by the initial rendering image generation unit 407 initially rendered image R _o. The reference image is the reference image I acquired by the reference image acquisition unit 405. The corrected diffuse reflection color and the corrected specular reflection color are the corrected diffuse reflection color _C'd (x, y, z) and the corrected specular reflection color C'd (x, y, z) derived by the diffuse reflection color / specular reflection color correction unit 2102. _s (x, y, z). The rendered image is a rendered image R generated by the rendered image generation unit 2103.

(Operation in information processing device)
FIG. 22 is a flowchart showing a flow of processing executed by the information processing apparatus 2100. FIG. 23 is a diagram showing an example of GUI used in this embodiment. Hereinafter, the details of the operation in the information processing apparatus 2100 will be described with reference to the flowchart of FIG. The following operation is started when a user operation is performed on the input device 308 and a predetermined instruction by the user operation is input via the input IF 305. In the present embodiment, the display control unit 2104 controls the display on the display device 310.

Since S401 to S404 and S409 to S410 have the same processing contents as the steps having the same symbols in the first embodiment, the description thereof will be omitted.

In S2101, the information processing apparatus 2100 displays the reference image I acquired in S402 and the initial rendered image R ₀ generated in S403 on the GUI screen 2300. The reference image I is displayed in the area U702 of the GUI screen 2300. The initial rendered image R ₀ is displayed in the area U703 of the GUI screen 2300.

In S2102, the color correction information acquisition unit 2101 acquires the color correction value p which is the color correction information. The color correction value is a coefficient for correcting RGB of the rendered image R, and is acquired by a user operation by positioning the slider on the slider bar of the area U2301 of the GUI screen 2300.

In S2103, the diffuse reflection color / specular color correction unit 2102 uses the diffuse reflection light ratio, the specular reflection light ratio, the diffuse reflection color, the specular reflection color, the diffuse reflection eccentricity, the specular reflection eccentricity, and the color correction value. after correction from the p diffusion reflection color C _'d, corrected specular color C' to derive the _s. Proportion of diffuse reflected light, the ratio of the specular reflection light is the ratio r _d, the ratio of the specular reflection light r _s of the diffuse reflection light derived in S404. The diffuse reflection color and the specular reflection color are the diffuse reflection color C _d (x, y, z) and the specular reflection color C _s (x, y, z) acquired in S402. Diffuse reflection bending strength, specular bending strength was obtained in S402, the diffuse reflection bending strength _{A d (x, y, z} , l n, v n), specular bending strength A _s (x, y, z, l _n , v _n ). The color correction value is the color correction value p acquired in S2102. First, the diffuse reflection _{correction value p d} and the specular reflection correction value p _{s for the diffuse reflection color C d} (x, y, z) and the specular reflection color C _s (x, y, z) are derived. The image in which the color correction value p is applied to the pixel value R (u, v) of the rendered image R has the diffuse reflection color C _d (x, y, z) and the specular reflection color C _s (x, y, z), respectively. The rendered image is obtained by applying the diffuse reflection correction value p _d and the specular reflection correction value p _s. This relationship is expressed by the following equations (19), (20), and (21).

Further, based on the ratio r _s ratio r _d and specular reflected light diffuse reflected light, to determine the percentage of each reflected light to render the image, the diffuse reflection correcting value p _d and specular reflection correcting value p _s, The constraint shown by the following equation (22) is given.

Based on the above equations, the diffuse reflection correction value p _d and the specular reflection correction value p _s are derived from the following equations (23) and (24).

By multiplying this diffuse reflection correction value p- _{d by the diffuse reflection color C d} (x, y, z) as in S407 of the first embodiment _{, the corrected diffuse reflection color C'd} (x, y, z) ) Is derived. Moreover, the specular reflection correction value p _s, similarly to S407 of Embodiment 1, By multiplying the specular color _{C s (x, y, z} ), corrected specular color _{C 's (x, y,} z ) Is derived.

In S2104, rendering image generation unit 2103, the corrected diffuse reflection color C _'d, corrected specular color C' _s, diffuse reflection bending strength A _d, specular bending strength A _s, the normal n, the imaging condition The rendered image R is derived from V and the lighting condition L. The rendered image R is derived based on the equation (3) of S404 of the first embodiment. Corrected diffuse reflection color _{C 'd (x, y,} z), corrected specular color _{C' s (x, y,} z) is the information derived by the S2103. Diffuse reflection bending strength _{A d (x, y, z} , l n, v n), specular bending strength _{A s (x, y, z} , l n, v n), the normal n (x, y, z), the imaging condition V, and the illumination condition L are the information acquired in S402.

With the above configuration, in texture editing, color correction of each reflected light color is executed based on the ratio of diffuse reflected light and specular reflected light from the color correction value for the image. As a result, the user can perform color correction without worrying about the ratio of reflected light.

In the present embodiment, the color correction is uniformly applied to all the regions, but as in the fourth embodiment, the region to be color-corrected may be specified. Further, color correction may be performed using reference images under a plurality of geometric conditions as in the third embodiment.

[Other Embodiments]
Of the above-mentioned processing units, the initial rendered image generation unit 407, the reflected light ratio derivation unit 408, 1802, the rendered image generation unit 409, 2103, and the like are replaced with machine-learned trained models. It may be used for processing. In that case, for example, a plurality of combinations of input data and output data to the processing unit are prepared as training data, knowledge is acquired from them by machine learning, and output data for the input data is obtained based on the acquired knowledge. Generates a trained model that outputs as a result. The trained model can be constructed by, for example, a neural network model. Then, the trained model performs the processing of the processing unit by operating in collaboration with the CPU, GPU, or the like as a program for performing the same processing as the processing unit. The trained model may be updated after a certain process if necessary.

In the above-described embodiment, the diffuse reflection color and the specular reflection color are expressed as RGB, but the present invention is not limited to this, and CIE L * a * b * or an XYZ color system may be used.

Further, in the above-described embodiment, the information on the reflection characteristics is treated as the diffuse reflection eccentricity intensity and the specular reflection eccentricity intensity having the intensities corresponding to the illumination direction and the observation direction, but the present invention is not limited thereto. You may input the parameters approximated by the existing Torrance-Sparrow model, Phong model, etc., and internally convert them into the intensities for each angle before use.

The GUI shown in the above-described embodiment is an example for realizing the configuration, and any form may be used as long as it is a mechanism capable of input / output similar to the above-described embodiment.

The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

300 Information processing device 408 Reflected light ratio derivation unit 410 Display control unit

Claims (13)

An information processing device that controls the display of information including a first image for reference and a second image to be corrected.
Diffuse included in the pixel value of the first image based on the reflected light intensity for each angle in the subject, the shape information of the subject, and the geometric conditions corresponding to the first image when the subject is imaged. Derivation means for deriving the ratio of the reflection component and the specular reflection component,
A display control means for displaying the first image, the second image, and the derived ratio together on the display means.
An information processing device characterized by having. An acquisition means for acquiring color correction information used for color correction of the second image, and
It further has a generation means for generating a third image by correcting the color of the second image based on the color correction information.
The information processing device according to claim 1, wherein the display control means updates the second image displayed on the display means with the third image. The information processing apparatus according to claim 2, wherein the generation means generates the third image in which the diffuse reflection component and the specular reflection component are associated with each other. Further having a designating means for designating an area for color correction in the second image,
The information processing apparatus according to claim 2 or 3, wherein the generation means corrects the color of the designated region in the second image. The information processing apparatus according to any one of claims 2 to 4, wherein the acquisition means acquires the color correction information based on an instruction from a user via a user interface. The information processing device according to claim 5, wherein the display control means displays a slider for causing the user to adjust the color correction information on the display means. The information according to any one of claims 1 to 6, wherein the display control means displays the derived ratio as a numerical value, or displays the distribution of the derived ratio as an image. Processing equipment. The derivation means
The ratio is derived by setting a threshold value for the deviation from the specular reflection condition obtained from the geometric condition and the shape information.
Or
The information according to any one of claims 1 to 7, wherein the ratio is derived based on the intensity of the diffuse reflection component and the intensity of the specular reflection component with respect to the deviation from the normal reflection condition. Processing equipment. An image group acquisition means for acquiring a plurality of the first images having different geometric conditions at the time of imaging, and
A condition acquisition means for acquiring geometric conditions corresponding to each of the plurality of first images, and
A means for creating a table for ranking images to be referred to when correcting the color of the second image based on the ratio derived from the acquired plurality of geometric conditions and the shape information. The information processing apparatus according to any one of claims 1 to 8, further comprising. The information according to any one of claims 1 to 9, wherein the geometric condition is a geometric condition of irradiation of light and reception of light reflected by the subject irradiated with the light. Processing equipment. A display means for displaying the first image for reference, the second image to be corrected, and the ratio of the diffuse reflection component and the specular reflection component included in the pixel value of the first image together.
An update means for updating the second image displayed on the display means to the second image corrected according to a user's instruction, and
A display device characterized by having. A program for causing a computer to function as each means of the information processing apparatus according to any one of claims 1 to 10. An information processing method that controls the display of information including a first image for reference and a second image to be corrected.
Diffuse included in the pixel value of the first image based on the reflected light intensity for each angle in the subject, the shape information of the subject, and the geometric conditions corresponding to the first image when the subject is imaged. Derivation step to derive the ratio of the reflection component and the specular reflection component,
A display control step for displaying the first image, the second image, and the derived ratio together on the display means.
An information processing method characterized by having.

Images