JP7463697B2 - Gloss acquisition state calculation device, gloss acquisition state calculation method, gloss acquisition state calculation program, terminal, and gloss acquisition state display program - Google Patents

Description

This embodiment relates to a gloss acquisition state calculation device, a gloss acquisition state calculation method, a gloss acquisition state calculation program, a terminal, and a gloss acquisition state display program.

Traditionally, in the production of 3DCG (three-dimensional computer graphics), CG designers create realistic CG images by setting textures for three-dimensional shapes formed by modeling real-world objects.

A known technique for reproducing a real object as a three-dimensional image in CG is a technique using texture mapping, as proposed in Patent Document 1. In a technique using texture mapping, multiple images captured from multiple directions are prepared in advance. When reproducing the three-dimensional image, an appropriate image is selected for the viewpoint, and the selected image is pasted onto the three-dimensional shape model. However, the lighting conditions of the three-dimensional image reproduced as CG and the real object at the time of capture often differ, and as a result, the three-dimensional image and the real object often look different.

The method of Non-Patent Document 1 is used as a method for making the appearance of a three-dimensional image closer to that of a real object. In the method of Non-Patent Document 1, the change in reflectance due to the lighting direction or line of sight direction is represented by a bi-directional reflectance distribution function (BRDF). In the method of Non-Patent Document 1, the BRDF is modeled and represented by a reflection constant. Many measurement methods have been proposed for estimating this reflection constant.

As a measurement method for estimating the reflection constant, a method is often used in which a large number of images taken with different light source directions or viewing directions are used to estimate the reflection constant for each pixel. However, since the BRDF represents a hemispherical distribution of reflected light for the object surface, it is necessary to take detailed images from many angles in order to estimate the reflection constant. In Patent Document 2, a large number of images are obtained by taking images while changing the illumination direction of the light source, and the reflection constant is estimated for each pixel. In the method of Patent Document 2, lighting with a two-dimensional spread is used to capture the sharp luster of real objects such as metals, which makes it possible to reduce the number of images taken.

JP 2000-348213 A Patent No. 3962588

SIGGRAPH Computer Graphics Proceedings, Annual Conference Series, 1997, p379-387”Object Shape and Reflectance Modeling from Observation”

To estimate the reflection constant, it is necessary to obtain information on the gloss transition, which is a characteristic of the reflected light of an object when a light source moves across the object's surface area. In many estimation methods, the reflection constant is estimated by optimizing the constants of a reflection model for the gloss transition. However, for a three-dimensional object with a complex shape, it is difficult to determine in advance in what direction the camera and light source should be positioned to obtain the gloss transition. In this case, it becomes necessary to perform multiple shots with different relative positions between the camera and light source. However, in these multiple shots, many images are taken that do not contain gloss information and are not necessary for estimation.

This embodiment has been made in consideration of such circumstances, and provides a gloss acquisition state calculation device, a gloss acquisition state calculation method, a gloss acquisition state calculation program, a terminal, and a gloss acquisition state display program that can acquire the gloss transition of a real object with a small number of shots, regardless of the reflection characteristics, three-dimensional shape, and fine surface shape of the real object.

A gloss acquisition state calculation device according to a first aspect includes an image acquisition means for acquiring a group of images obtained by photographing an object with a camera in association with movement of at least one of the object, a light source illuminating the object, and a camera photographing the object, a shape acquisition means for acquiring three-dimensional shape information of the object, a radiance transition estimation means for estimating radiance transition information, which is information on the transition of radiance of specular reflected light across the group of images, for each surface region of the object represented by the three-dimensional shape information, and a calculation means for calculating acquisition state information indicating an acquisition state of the radiance transition information for each surface region. The calculation means uses a normal to the surface region calculated from the three-dimensional shape information, a position of the camera represented by camera position information, which is information on the position of the camera across the group of images, and a position of the light source represented by light source position information, which is information on the position of the light source across the group of images, to determine for each surface region whether or not there is a position of the light source in a specular reflection direction with the surface region as a reflecting surface relative to the position of the camera, and calculates the acquisition state information based on a result of the determination of whether or not at least the position of the light source is present.

A gloss acquisition state calculation method according to a second aspect includes acquiring a group of images obtained by photographing an object with a camera in association with movement of at least one of the object, a light source illuminating the object, and a camera photographing the object, acquiring three-dimensional shape information of the object, estimating radiance transition information, which is information on the transition of radiance of specular reflected light across the group of images, for each surface region of the object indicated by the three-dimensional shape information, and calculating acquisition state information indicating an acquisition state of the radiance transition information for each surface region. The calculation of the acquisition state information includes determining, for each surface region, whether or not there is a position of the light source that is in a specular reflection direction with the surface region as a reflecting surface with respect to the position of the camera, using a normal of the surface region calculated from the three-dimensional shape information, a position of the camera indicated by camera position information, which is information on the position of the camera across the group of images, and a position of the light source indicated by light source position information, which is information on the position of the light source across the group of images, and calculating the acquisition state information based on a result of the determination of whether or not there is at least a position of the light source.

A gloss acquisition state calculation program according to a third aspect causes a computer to execute the following: acquiring a group of images obtained by photographing an object with a camera in association with the movement of at least one of the object, a light source illuminating the object, and a camera photographing the object; acquiring three-dimensional shape information of the object; estimating radiance transition information, which is information on the transition of radiance of specular reflected light across the group of images, for each surface region of the object indicated by the three-dimensional shape information; and calculating acquisition state information indicating an acquisition state of the radiance transition information for each surface region. The calculation of the acquisition state information includes: determining, for each surface region, whether or not there is a position of the light source that is in a specular reflection direction with the surface region as a reflecting surface with respect to the position of the camera, using a normal of the surface region calculated from the three-dimensional shape information, a position of the camera indicated by camera position information, which is information on the position of the camera across the group of images, and a position of the light source indicated by light source position information, which is information on the position of the light source across the group of images, and calculating the acquisition state information based on a result of the determination of whether or not there is at least a position of the light source.

According to this embodiment, it is possible to provide a gloss acquisition state calculation device, a gloss acquisition state calculation method, a gloss acquisition state calculation program, a terminal, and a gloss acquisition state display program that can acquire the gloss transition of a real object with a small number of shots, regardless of the reflection characteristics, three-dimensional shape, and fine surface shape of the real object.

FIG. 1 is a diagram illustrating an example of the configuration of a gloss acquisition state calculation device according to an embodiment. FIG. 2 is a diagram showing an example of a format of polygon data as an example of shape information. FIG. 3 is a diagram showing an example of the format of the vertex data. FIG. 4 is a block diagram showing a configuration of an example of an image analyzing apparatus. FIG. 5 is a functional block diagram of the processor. FIG. 6 is a diagram showing an example of the configuration of a captured image information table indicating the correspondence between camera information and light source information. FIG. 7 is a schematic diagram of a commonly used object reflection model. FIG. 8 is a diagram showing radiance transition information. FIG. 9A is a diagram showing a first setting example of the threshold range. FIG. 9B is a diagram showing a second setting example of the threshold range. FIG. 9C is a diagram showing a third example of setting the threshold range. FIG. 9D is a diagram showing a fourth example of setting the threshold range. FIG. 10 is a display example of an instruction image. FIG. 11 is a flowchart showing an example of the operation of the gloss acquisition state calculation method. FIG. 12A is a diagram showing a display example of an instruction image. FIG. 12B is a diagram showing a display example of an instruction image. FIG. 12C is a diagram showing a display example of an instruction image. FIG. 12D is a diagram showing a display example of an instruction image.

Embodiments of the present invention will now be described with reference to the drawings. FIG. 1 is a diagram showing an example of the configuration of a gloss acquisition state calculation device according to one embodiment. In FIG. 1, the gloss acquisition state calculation device 1 includes an image analysis device 2, a shape measurement device 3, a camera 4, a light source 5, and an image display device 6.

The image analysis device 2 estimates the shape and texture of the object 7 based on information about the three-dimensional shape of the object 7 acquired from the shape measurement device 3, the image of the object 7 acquired from the camera 4, position information of the camera 4, and position information of the light source 5. The shape of the object 7 includes the overall shape and the fine surface shape. The fine surface shape is the shape of each of the fine surfaces in the surface region set on the object 7. The fine surface shape includes information on normals that represent the corresponding faces. Furthermore, the texture of the object includes information on the reflection characteristics of each fine surface shape. Details of the image analysis device 2 will be described later.

The shape measurement device 3 measures information about the three-dimensional shape of the object 7 and sends the shape information obtained as a result of the measurement to the image analysis device 2. The shape measurement device 3 is configured to measure information about the three-dimensional shape of the object 7, for example, by an active measurement method. The active measurement method is a method in which measurement light, such as a laser or pattern light, is actively irradiated onto the surface of the object 7 and the reflected light from the object 7 is received to perform measurement. The shape measurement device 3 is not particularly limited as long as it can measure information about the three-dimensional shape of the object 7. For example, the shape measurement device 3 may be configured to measure information about the three-dimensional shape of the object 7 by a passive measurement method that uses images obtained by photography, such as a volume intersection method and a stereo method. The images used in the passive measurement method may be images taken by the shape measurement device 3 or images taken by the camera 4.

Whether the measurement method is the active one or the passive one described above, much of the shape information obtained as a result of the measurement is composed of point cloud data. The image analysis device 2 may convert the point cloud data into polygon data and store this polygon data as shape information in storage, for example, in a format as shown in FIG. 2. Most polygon data is composed of triangular polygons. For this reason, the polygon data is expressed as triangular vertex data as shown in FIG. 2. The image analysis device 2 may also store data in the format shown in FIG. 3 as each vertex data in storage. The vertex data shown in FIG. 3 is data that has information on the coordinates and normals of each vertex. The normals of the faces that make up the polygon can be calculated from the coordinates and normals of the vertices stored as shown in FIG. 3.

Camera 4 is configured to capture an image of an object and generate an image of the object. Camera 4 includes, for example, an optical system and an image sensor. The optical system is an optical system configured to form an image of the object on the image sensor. The image sensor is an image sensor such as a CCD image sensor or a CMOS image sensor, and converts light incident from the object into an electrical signal to generate an image of the object. Here, in the embodiment, camera 4 is assumed to be fixed at a predetermined position.

The light source 5 emits illumination light for illuminating the object. The light source 5 is, for example, an LED light source, but is not limited thereto. In addition, in order to facilitate measurement of reflected light, which will be described later, the light source 5 is preferably a line light source or a surface light source. A line light source is a light source configured to emit diffuse illumination light from a line-shaped light-emitting portion. A surface light source is a light source configured to emit diffuse illumination light from a surface-shaped light-emitting portion. However, the light source 5 may be a point light source. Here, the light source 5 is configured so that the relative position between the camera 4 and the object 7 can be changed. For example, the light source 5 is attached to a robot arm and configured so that the position can be changed three-dimensionally. The specific configuration of the light source 5 is not particularly limited as long as the relative position between the camera 4 and the object 7 can be changed. In addition, the light source 5 does not necessarily need to be configured so that it can be moved mechanically. For example, the light source 5 may be configured so that the user can hold it in his or her hand and move it.

The image display device 6 is a display device such as a liquid crystal display or an organic EL display. The image display device 6 is connected to, for example, the image analysis device 2 so as to be able to communicate with it, and displays various images sent from the image analysis device 2. These images include, for example, three-dimensional images of the object.

Figure 4 is a block diagram showing an example of the configuration of the image analysis device 2. The image analysis device 2 includes a processor 21, a ROM 22, a RAM 23, a storage 24, an input interface 25, an output interface 26, and a communication interface 27. Each of these is connected to each other via a bus 28. The image analysis device 2 may be an electronic device such as a personal computer (PC), a tablet terminal, or a smartphone.

The processor 21 is a processor that controls various operations of the image analysis device 2. The processor 21 is, for example, a CPU, but is not limited to this. The processor 21 may be an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), a GPU (Graphic Processing unit), etc. Furthermore, the processor 21 does not necessarily have to be composed of one CPU, etc., but may be composed of multiple CPUs, etc.

The ROM 22 stores the startup program of the image analysis device 2, etc. The RAM 23 is a main storage device for the processor 21, etc.

The storage 24 is a storage such as a hard disk or a solid state drive. The storage 24 stores various programs, parameters, etc., which are used by the processor 21 and are necessary for controlling the image analysis device 2. These various programs include a gloss acquisition state calculation program. The storage 24 also stores various data such as shape information of the object as the measurement result of the shape measurement device 3, the image generated by the camera 4, the position of the camera 4, and the position of the light source 5.

The input interface 25 includes a keyboard, a mouse, a touch panel, etc. In response to a user's operation via the input interface 25, a signal indicating the content of the user's operation is input to the processor 21 via the bus 28. The processor 21 executes various processes in response to the signal indicating the content of the operation.

The output interface 26 includes, for example, an interface for communication with the image display device 6. The output interface 26 sends image data to the image display device 6 at an appropriate timing when an image is displayed on the image display device 6. The output interface 26 may also include an interface for communication with a printer.

The communication interface 27 includes interfaces for communication between the image analysis device 2 and the shape measurement device 3, the camera 4, and the light source 5. The communication interface 27 may include an interface for communication via wired communication, or an interface for communication via wireless communication.

Figure 5 is a functional block diagram of the processor 21. The processor 21 includes an image acquisition unit 211, a shape acquisition unit 212, a camera information acquisition unit 213, a light source information acquisition unit 214, a radiance transition estimation unit 215, a calculation unit 216, a pointer image generation unit 217, a light source control unit 218, and a texture information estimation unit 219. The image acquisition unit 211, the shape acquisition unit 212, the camera information acquisition unit 213, the light source information acquisition unit 214, the radiance transition estimation unit 215, the calculation unit 216, the pointer image generation unit 217, the light source control unit 218, and the texture information estimation unit 219 are realized by the processor 21 executing a gloss acquisition state calculation program stored in the storage 24, for example. The image acquisition unit 211, shape acquisition unit 212, camera information acquisition unit 213, light source information acquisition unit 214, radiance transition estimation unit 215, calculation unit 216, instruction image generation unit 217, light source control unit 218, and texture information estimation unit 219 may be realized by hardware that performs similar processing to each other.

The image acquisition unit 211 acquires images from the camera 4. These images are an image group including multiple images generated by photographing the object 7 as the light source 5 moves. The image group may be generated by capturing multiple still images with the camera 4, or may be generated by capturing a video once with the camera 4.

The shape acquisition unit 212 acquires shape information of the object 7 from the shape measurement device 3. As described above, the shape information of the object 7 may be polygon data.

The camera information acquisition unit 213 acquires camera information of the camera 4 and stores the acquired camera information in the storage 24. The camera information includes the position of the camera 4. The camera information may also include the shooting conditions of the camera 4, such as the focal length.

The position of the camera 4 can be derived in advance by photographing a checkerboard or the like with the camera 4. The position of the camera 4 may also be derived by extracting feature points from an image of the object 7 acquired by the image acquisition unit 211 and performing camera calibration based on the feature points.

The light source information acquisition unit 214 acquires light source information of the light source 5 and stores the acquired light source information in the storage 24. The light source information includes the position of the light source 5, the radiance of the illumination light, the color temperature, etc.

When the light source 5 is configured to be mechanically movable by a robot arm or the like, the position of the light source 5 can be accurately obtained from sensor information of a sensor attached to the robot arm or the like. On the other hand, when the light source 5 is configured to be moved, for example, by a user's hand, if the light source 5 is provided with a position sensor and a multi-axis acceleration sensor, the light source position can be estimated from sensor information of the position sensor, acceleration sensor, etc.

Furthermore, the light source position may be estimated from an image obtained by the camera 4. For example, when a light source is captured in the image, the light source information acquisition unit 214 projects the image captured by the camera 4 onto the polygon data based on the camera information. Then, the light source information acquisition unit 214 regards the position of a projection area having a luminance value of, for example, a predetermined threshold or more among the projection areas as the position of the light source. In this method, the light source information acquisition unit 214 may also estimate the shape of the light source from the shape of the projection area. Also, when a light source is captured in the image, the light source information acquisition unit 214 may estimate the position and shape of the light source by an explicit designation of the light source area by the user. Also, if a reference object having a known shape and reflection characteristics is captured in the image together with the target object 7, even if the light source is not captured in the image, the light source information acquisition unit 214 may detect reflected light from this reference object from the image and estimate the position of the light source based on the detected reflected light. At this time, if a marker is attached to the reference object, the position of the reference object can be easily estimated from the image by detecting the marker. Also, when the light source is fixed, the position and shape of the light source estimated from a specific image captured so that the light source is visible may always be used.

In addition to the position of the light source, the radiance may also be obtained from the image. If a reference object is placed together with the target object 7, the light distribution of the light source can be calculated based on the light reflected from the reference object.

The camera information and light source information are stored in storage 24 in association with each other using the shooting time as an index. Figure 6 is a diagram showing an example of the configuration of a captured image information table showing the correspondence between camera information and light source information. As shown in Figure 6, a column for each camera information and light source information is provided for each record, corresponding to the column for the captured image number. When the object 7 is captured as a video, the camera information and light source information are stored for each frame of the video.

Returning now to the explanation of FIG. 5, the radiance transition estimation unit 215 estimates radiance transition information of the object 7 from the radiance of the incident light of the light source 5. The radiance transition information is information that represents the transition of radiance across a group of images. In other words, the radiance transition information represents the gloss transition of the object 7. The reason for estimating the radiance transition information is that, in order to estimate the reflection characteristics of the object 7, it is necessary to measure how the radiance changes as the light source 5 moves, rather than just measuring the radiance at the peak. Here, the surface region is a region that is set on the object 7. The surface region may be, for example, a colored portion corresponding to one or more polygons. The surface region may also be, for example, a portion corresponding to one or more pixels.

The calculation unit 216 calculates acquisition status information indicating the acquisition status of the radiance transition information for each surface region of the object 7 based on the radiance of the object 7, the camera information, the light source information, and the shape information of the object 7. The acquisition status information is information indicating whether or not the radiance transition information has been acquired.

The acquisition state information will be further described below. Fig. 7 is a schematic diagram of a commonly used reflection model of an object. In the reflection model of Fig. 7, incident light from a light source 71 is irradiated onto a microscopic surface 70 of an object, and reflected light from the microscopic surface 70 is observed at a viewpoint 72. In Fig. 7, N is a normal vector of the microscopic surface 70. L is a vector representing the direction of the light source 71 based on the reflection position. V is a vector representing the direction of the viewpoint 72 based on the reflection position. In addition, _θr is the angle between vector N and vector V, and _θL is the angle of incidence and the angle of reflection.

It is generally known in the field of CG that if there is no light transmission through the microsurface 70, the reflected light from the microsurface 70 is composed of the sum of diffuse reflected light 73 and specular reflected light 74. This type of reflection model is called a dichromatic reflection model. The dichromatic reflection model is expressed by a model equation that represents the respective characteristics of the diffuse reflected light 73 and specular reflected light 74.

When the radiance of the incident light from the light source 71 is IL, the radiance Id of the diffusely reflected light 73 is expressed by the following formula (1). Here, kd in formula (1) is the reflectance of the diffusely reflected component. Also, f _id is a predetermined function with kd, N, and L as parameters. In formula (1), the radiance Id does not change with the vector V. In other words, formula (1) shows that the diffusely reflected light 73 does not change depending on the position of the viewpoint 72, and is radiated with the same luminance in all directions.
Id = IL × kd × |L||N|cosθ _L = IL × f _id (kd, N, L) (1)

In contrast, specular reflected light 74 generally emits strong light in the regular reflection direction relative to the direction L of the light source 71. This causes the observer to perceive a highlight on the surface of the object. The brightness Is of specular reflected light according to the Cook-Torrance reflection model that is often used in CG is expressed by the following formula (2).
Is = IL×F/π×DG／(N・L)(N・V) = IL×f _is (ks, σ, N, L, V) (2)
In formula (2), F is a Fresnel term that indicates the degree of specular reflection on the microscopic surface. G is a geometric attenuation term that indicates the self-shadowing caused by the microscopic surface of the object. Furthermore, D is a Beckmann distribution term that indicates the variation of the microscopic surface. This D term indicates the spread of specularly reflected light and differs depending on the roughness of the object's surface.
The right side of equation (2) is a simplified version of the left side of equation (2). In the right side of equation (2), radiance Is is expressed using a predetermined function f _is with parameters ks, σ, N, L, and V. Here, ks is the specular reflectance, and σ is the surface roughness. In order to estimate the surface roughness σ, it is necessary to obtain not only the radiance for a light source at a specific position, but also the transition of the radiance associated with the movement of the light source. The calculation unit 216 determines whether or not radiance transition information indicating such a transition of the radiance has been obtained.

The calculation unit 216 calculates the normal for each surface region of the object 7 based on the shape information of the object 7 acquired by the shape measurement device 3. The calculation unit 216 then determines whether the light source 5 is passing through a position in the specular reflection direction of the camera 4, which is calculated from the normal for each surface region, and generates first acquisition status information based on this determination result. The first acquisition status information may be, for example, a numerical value indicating whether the light source 5 is passing through a position in the specular reflection direction of the camera 4. In this case, for example, the first acquisition status information may be generated to indicate 1 when the light source 5 is passing through, and 0 when the light source is not passing through.

7, the specular reflection direction vector L' with respect to the line of sight direction vector V of the camera 4 is derived by the following formula (3). When the direction vector L of the light source 5 passes through the specular reflection direction vector L', the calculation unit 216 determines that the light source 5 has passed a position that is in the specular reflection direction of the camera 4. When the light source 5 is located at a position that is in the specular reflection direction of the camera 4, the gloss of the object 7 reaches its peak.
L' = 2 × N × (V × N) - V (3)
Here, when the light source 5 is a light source having a size such as a line light source or a surface light source, the radiance transition estimation unit 215 determines that the light source 5 has passed a position where a part of the light source 5 is in the specular reflection direction of the camera 4 when the light source 5 passes along the specular reflection direction vector L'.

The calculation unit 216 also determines whether a transition in radiance sufficient to estimate the surface roughness σ has been acquired, and generates second acquisition status information based on this determination result. The second acquisition status information may be, for example, a numerical value indicating whether a transition in radiance sufficient to estimate the surface roughness σ has been acquired. In this case, for example, the second acquisition status information may be generated to indicate 1 when acquisition has been achieved, and 0 when acquisition has not been achieved.

Figure 8 is a diagram showing radiance transition information. The horizontal axis of Figure 8 is the change in the image, that is, the change in the position of the light source. The vertical axis of Figure 8 is the value of radiance normalized by the value of radiance corresponding to the gloss peak. When the light source 5 moves continuously, the radiance also transitions continuously as shown in the graph of Figure 8. Here, the markers in the graph indicate measurement points that are points where the radiance value can actually be measured. When a specified number or more of measurement points are obtained within a threshold range indicated by a ratio to the radiance value of the gloss peak, the calculation unit 216 determines that radiance transition information sufficient for estimating the surface roughness σ has been obtained. The radiance of the gloss peak does not have to be the radiance of the gloss peak strictly. For example, the maximum radiance among the measurement points may be used as the radiance of the gloss peak.

Here, the threshold range may be specified by the user, but may also be set in advance. An example of a setting method for setting the threshold range in advance is shown below. The threshold range may also be set based on a different concept from the example setting method below.

Figure 9A is a diagram showing a first setting example of the threshold range. The first setting example is based on the idea that it is sufficient to acquire the minimum amount of radiance transition information sufficient to estimate the surface roughness σ. In the first example, the threshold range is set to a range of 20-100% of the radiance of the gloss peak. Then, when three or more measurement points have been acquired, including the measurement point corresponding to the gloss peak and two or more measurement points included in the threshold range, the calculation unit 216 determines that sufficient radiance transition information has been acquired to estimate the surface roughness σ.

Figure 9B is a diagram showing a second setting example of the threshold range. The second setting example is an example of setting the threshold range for an object having a highly glossy surface such as a metal surface or a plastic surface. In the case of a highly glossy object, as shown in Figure 9B, the radiance shows a steep change at the gloss peak. Therefore, in the case of a highly glossy object, there is a possibility that a sufficient number of measurement points cannot be obtained unless the threshold range is widened. For this reason, in the second example, the threshold range is set to a range of 10-100% of the radiance of the gloss peak. Then, when three or more measurement points, including the measurement point corresponding to the gloss peak and two or more measurement points included in the threshold range, have been obtained, the calculation unit 216 determines that radiance transition information sufficient for estimating the surface roughness σ has been obtained.

Figure 9C is a diagram showing a third setting example of the threshold range. The third setting example is an example of setting the threshold range for an object having a surface with low gloss, such as a cloth surface, a textured surface, or a leather surface. For an object with low gloss, as shown in Figure 9C, the distribution is wide with respect to the change in the position of the light source. Therefore, for an object with low gloss, if the threshold range is too wide, there is a possibility that more measurement points than necessary will be acquired. For this reason, in the third example, the threshold range is set to a range of 50-100% of the radiance of the gloss peak. Then, when three or more measurement points, including the measurement point corresponding to the gloss peak and two or more measurement points included in the threshold range, have been acquired, the calculation unit 216 determines that radiance transition information sufficient to estimate the surface roughness σ has been acquired.

Figure 9D is a diagram showing a fourth setting example of the threshold range. The fourth setting example is a setting example when a highly accurate estimation of gloss is required. When a highly accurate estimation of gloss is required, for example, two threshold ranges are set as shown in Figure 9D. Threshold range 1 is set to a range of 70-100% of the radiance of the gloss peak. Threshold range 2 is set to a range of 40-70% of the radiance of the gloss peak. Then, when five or more measurement points have been acquired, including the measurement point corresponding to the gloss peak, two or more measurement points included in threshold range 1, and two or more measurement points included in threshold range 2, the calculation unit 216 determines that radiance transition information sufficient to estimate the surface roughness σ has been acquired. Note that in the fourth setting example, when a more accurate estimation of gloss is required, three or more threshold ranges may be set.

Now, let us return to the explanation of FIG. 5. The indication image generating unit 217 generates an indication image based on the acquisition state information calculated by the calculation unit 216. The indication image is an image that distinguishes between surface areas where radiance transition information, i.e., gloss transition, has been acquired and surface areas where it has not been acquired. The indication image is displayed, for example, on the image display device 6. FIG. 10 is an example of an indication image. The indication image of FIG. 10 is generated by making the pixel values of the surface area where radiance transition information has been acquired in the texture model generated by pasting a predetermined texture on the polygon model of the object 7 different from the pixel values of the other surface areas. In such an indication image, the display area 61 where radiance transition information has been acquired is displayed in a color different from the other surface areas. The user can distinguish between the part where radiance transition information has been acquired and the part where it has not been acquired by looking at the texture model displayed on the image display device 6. This allows the user to move the light source 5 so that radiance transition information can be acquired for the surface area where radiance transition information has not been acquired, and to perform re-imaging with the camera 4.

Returning now to the explanation of FIG. 5, the light source control unit 218 moves the light source 5 based on the acquisition state information calculated by the calculation unit 216. Specifically, the light source control unit 218 moves the light source 5 to a position where radiance transition information can be acquired.

The texture information estimation unit 219 uses the camera information, light source information, and shape information of the object 7 stored in the storage 24 to estimate the reflection constant and normal for each surface area of the object 7.

The reflection constant and normal are derived by estimating the optimal solution of the parameters that minimize the error between the radiance Io of the surface area and the calculated radiance Io'. For example, when equations (1) and (2) are used as the reflection model of the object, the radiance Io' is calculated by the following equation (4).
Io' = Id + Is = IL × {f _id (kd, N, L) + f _is (ks, σ, N, L, V)} (4)
In formula (4), vectors L and V are known because they are calculated from the camera information and light source information stored in the storage 24. The normal vector N of the surface region is expressed by adding the normal vector Np of the polygon in the surface region and the normal vector Nm of the micro surface in the surface region. As described above, the normal vector Np is known because it is calculated from the shape information acquired by the shape measurement device 3. Therefore, by solving the problem of minimizing the square error J shown in the following formula (5), the reflection constant (kd, ks, σ) and the micro normal Nm for each surface region can be estimated. This estimation is performed for all surface regions of the object 7, and the overall shape and reflection characteristics of the object 7 can be estimated.
J = 1 / 2 Σ _n (Io _n -Io _n ') ² (5)

Figure 11 is a flowchart showing an example of the operation of the gloss acquisition state calculation device according to this embodiment. Here, the target object 7 in the example of Figure 11 is an object having a surface that does not transmit light.

In step S1, the processor 21 acquires shape information representing the macroscopic three-dimensional shape of the object 7 from the shape measurement device 3. As mentioned above, the measurement method used by the shape measurement device 3 is not limited to a specific method.

In step S2, the processor 21 converts the point cloud data as shape information into polygon data that is easy to handle in CG. The processor 21 also calculates the normals of each polygon. The processor 21 then stores the polygon data of each polygon in the storage 24. As described above, the polygon data includes the coordinates of the vertices of the polygon and information on the normals.

In step S3, the processor 21 moves the light source 5 so that the transition of the radiance of the object 7 can be obtained. In the first time, the processor 21 may move the light source 5 to a predetermined position. In the second time and thereafter, the processor 21 sequentially changes the position of the light source 5 so that the transition of the radiance of the object 7 can be obtained.

In step S4, the processor 21 executes image capture by the camera 4. The processor 21 then stores in the storage 24 the image generated by the camera 4 in association with camera information such as the position of the camera 4 at the time of image capture and the image capture conditions, and light source information such as the position of the light source 5 at the time of image capture and the radiance of the object 7.

In step S5, the processor 21 determines whether the light source 5 has passed through a position that corresponds to the specular reflection direction of the camera 4, based on the normal vector N of the polygon corresponding to the surface region of the object 7, the line-of-sight vector V of the camera 4, and the direction vector L of the light source 5. If it is determined in step S5 that the light source 5 has not passed through a position that corresponds to the specular reflection direction of the camera 4, the process returns to step S3. In this case, the processor 21 moves the light source 5 to another position. If it is determined in step S5 that the light source 5 has passed through a position that corresponds to the specular reflection direction of the camera 4, the process proceeds to step S6.

In step S6, the processor 21 calculates the radiance of the specularly reflected light at each light source position from the light source information stored in the storage 24. The processor 21 generates radiance transition information by normalizing the radiance at each light source position with the radiance value corresponding to the gloss peak. The processor 21 then sets a threshold range of radiance indicated as a ratio to the radiance value for the gloss peak.

In step S7, the processor 21 determines whether or not a specified number of measurement points or more have been obtained within the threshold range. If it is not determined in step S7 that the specified number of measurement points or more have not been obtained, the process returns to step S3. In this case, the processor 21 moves the light source 5 to another position. If it is determined in step S7 that the specified number of measurement points or more have been obtained, the process proceeds to step S8.

In step S8, the processor 21 causes the image display device 6 to display a texture model of the object 7 based on the polygon data acquired in step S2. Furthermore, the processor 21 colors the surface region for which it is determined that sufficient radiance transition information has been acquired. While the three-dimensional image is being displayed, the processor 21 may accept an operation by the user to rotate, enlarge or reduce the texture model of the object 7. When such an operation is performed, the processor 21 causes the image display device 6 to re-display the three-dimensional image based on the rotated, enlarged or reduced polygon data.

In step S9, the processor 21 determines whether or not the transition in radiance has been obtained for all surface regions. If it is determined in step S9 that the transition in radiance has not been obtained for all surface regions, the process proceeds to step S10. If it is determined in step S9 that the transition in radiance has been obtained for all surface regions, the process proceeds to step S11. The process may also be configured to proceed to step S11 if the user instructs the completion of measurement in step S9.

In step S10, the processor 21 changes the surface area to be processed to a surface area for which the radiance transition has not yet been obtained. The process then returns to step S3. In this case, the processor 21 moves the light source 5 to another position so that the radiance transition for another surface area can be obtained.

In step S11, the processor 21 estimates the fine surface shape and reflection constant for each surface region by using the radiance transition information, the camera information, the light source information, and the shape information of the object 7. Then, the processor 21 ends the processing of FIG. 11.

As described above, according to this embodiment, it is determined for each surface region whether radiance transition information sufficient for estimating the reflectance characteristics to reproduce the texture of the object 7, i.e., gloss transition, has been acquired. By performing imaging based on the result of this determination, it is possible to acquire just the right amount of gloss transition sufficient for estimating the reflectance characteristics. Therefore, radiance information unnecessary for estimating the reflectance characteristics is not used in the estimation calculation of the reflectance characteristics, and efficient estimation of the reflectance characteristics can be achieved.

In addition, since it is only necessary to capture images of surface areas where a gloss transition sufficient for estimating reflectance characteristics has not been obtained, the number of images taken can be reduced.In addition, since it is only necessary to capture images of surface areas where a gloss transition sufficient for estimating reflectance characteristics has not been obtained, the arrangement of the light source and camera does not need to be determined in advance based on the shape and reflectance characteristics of the object.In addition, since the light source is configured to be movable in three dimensions, there is no problem in which the camera is unable to capture gloss due to the positional relationship between the surface area and the light source.

[Modification 1]
Modifications of the embodiment will be described below. In the embodiment, the camera 4 and the object 7 are fixed, and the object 7 is photographed while the light source 5 is moved. However, it is not necessary to photograph the object 7 while the light source 5 is moved. For example, the light source 5 and the object 7 may be fixed, and the object 7 may be photographed while the camera 4 is moved. Also, the camera 4 and the light source 5 may be fixed, and the object 7 may be photographed while the object 7 is moved. Furthermore, the object 7 may be photographed while two or more of the camera 4, the light source 5, and the object 7 are moved.

[Modification 2]
In the embodiment, the Cook-Torrance model is exemplified as a reflection model used to estimate the reflection characteristics. Any model may be used to estimate the reflection characteristics, not limited to the Cook-Torrance model. For example, an anisotropic reflection model such as the Ward model may be used for an object that emits a reflection intensity that differs depending on the camera direction or the light source direction, such as an object having a hairline-finished metal surface.

When generating an image that reproduces the texture using CG, the acquired shape information, reflection characteristics, and normal information are used to reproduce the appearance and texture of the object under any lighting environment. However, the reflection model used during reproduction must be the same as the reflection model used when estimating the reflection characteristics.

[Modification 3]
In the embodiment described above, it is determined that radiance transition information sufficient for estimating the reflectance characteristics has been acquired when the light source 5 passes through a position in the specular reflection direction of the camera 4 and a predetermined number or more of measurement points within the threshold range have been acquired. It may also be determined that radiance transition information sufficient for estimating the reflectance characteristics has been acquired when only one of these conditions is satisfied.

[Modification 4]
In the above-described embodiment, the instruction image is generated by making the pixel values of the surface region for which radiance transition information in the texture model of the object can be obtained different from the pixel values of the other surface regions. However, the instruction image may be generated by other methods.

Figure 12A is a diagram showing another example of the display of an indication image. The indication image in Figure 12A is generated by making the pixel values of a portion 62 corresponding to a surface region of a polygon model of an object to which no texture has been applied, for which radiance transition information has been obtained, different from the pixel values of other portions.

Figure 12B is a diagram showing yet another display example of an indication image. The indication image in Figure 12B is generated by making the pixel values of a portion 63 of a texture map (developed view) 63 of an object identified by shape information, which corresponds to a surface region for which radiance transition information was obtained, different from the pixel values of other portions.

Figure 12C is a diagram showing yet another display example of an instruction image. The instruction image in Figure 12C is generated by making the pixel values of surface region 64b, from which radiance transition information has been acquired, of texture model 64a generated by pasting an image acquired by camera 4 as a texture onto a polygon model of object 7 different from the pixel values of other surface regions.

Figure 12D is a diagram showing yet another display example of an instruction image. The example of Figure 12D is a suitable display example when a user photographs an object 7 while holding a terminal 8 such as a smartphone equipped with a camera 4 and an image display device 6 in his/her hand. A gloss acquisition state display program is installed in the terminal 8 so that the instruction image can be displayed. Furthermore, the terminal 8 has a communication interface configured to send the image obtained by the camera 4 to the image analysis device 2 and to receive the instruction image generated by the image analysis device 2.

In the example of FIG. 12D, the image is generated by making the pixel values of the portion corresponding to the surface region for which radiance transition information has been acquired different from the pixel values of the other portions of the images sequentially obtained when the object 7 is photographed by the camera 4 of the terminal 8 from different directions, such as direction 8a and direction 8b.

Here, the terminal 8 is equipped with at least a camera 4 and an image display device 6. The light source 5 may or may not be equipped in the terminal 8. Furthermore, if the camera 4 of the terminal 8 is a twin-lens camera, the shape information of the object 7 can be acquired by the stereo method, and therefore the shape measurement device 3 is not necessary. Of course, the shape of the object 7 may be measured by a shape measurement device 3 separate from the terminal 8.

Furthermore, the terminal 8 may be configured to be capable of performing processing equivalent to that of the image analysis device 2. For example, the terminal 8 may have a gloss acquisition state calculation program installed so that it can perform processing equivalent to that of the image analysis device 2.

The present invention is not limited to the above-described embodiments, and can be modified in various ways in the implementation stage without departing from the gist of the invention. The embodiments may also be implemented in appropriate combination, in which case the combined effects can be obtained. Furthermore, the above-described embodiments include various inventions, and various inventions can be extracted by combinations selected from the multiple constituent elements disclosed. For example, if the problem can be solved and an effect can be obtained even if some constituent elements are deleted from all the constituent elements shown in the embodiments, the configuration from which these constituent elements are deleted can be extracted as an invention.

The following will addendum to inventions that can be extracted from the embodiments of the present invention.
[1] An image acquisition means for acquiring a group of images obtained by photographing an object with a camera in accordance with movement of at least one of an object, a light source that illuminates the object, and a camera that photographs the object;
A shape acquisition means for acquiring three-dimensional shape information of the object;
a radiance transition estimation means for estimating, for each surface region of the object represented by the three-dimensional shape information, radiance transition information which is information on a transition of the radiance of specular reflected light across the group of images;
A calculation means for calculating, for each of the surface regions, acquisition state information indicating an acquisition state of the radiance transition information;
A gloss acquisition state calculation device comprising:
[2] A camera information acquisition means for acquiring camera information of the camera when the image group is captured;
a light source information acquisition means for acquiring light source information of the light source when the image group is captured;
Further equipped with
the light source information includes light source position information that is information on the position of the light source across at least the group of images;
The camera information includes at least camera position information that is information on the position of the camera across the group of images.
The gloss acquisition state calculation device according to [1].
[3] further comprising a texture information estimating unit configured to estimate a reflection constant and a normal of a predetermined reflection model for each of the surface regions using the radiance transition information, the three-dimensional shape information, the camera position information, and the light source position information.
The gloss acquisition state calculation device according to [2].
[4] The calculation means:
using a normal to the surface area calculated from the three-dimensional shape information, a position of the camera indicated by the camera position information, and a position of the light source indicated by the light source position information, determining for each surface area whether or not there is a position of the light source that is in a regular reflection direction with the surface area as a reflecting surface with respect to the position of the camera;
calculating the acquisition state information based on a result of determining whether or not the position of the light source is present;
The gloss acquisition state calculation device according to [2] or [3].
[5] The calculation means:
calculating a peak of a plurality of radiances indicated by the radiance transition information for each of the surface regions, and determining whether or not a predetermined number of radiances are included in a threshold range calculated based on the peak;
calculating the acquisition state information based on a result of determining whether or not at least a predetermined number of the radiances are present;
The gloss acquisition state calculation device according to any one of [1] to [4].
[6] The threshold range is specified by a user.
The gloss acquisition state calculation device according to [5].
[7] An indication image generating means for generating an indication image having a plurality of pixels each corresponding to any one of the surface regions, each pixel having a pixel value based on the acquisition state information of the corresponding surface region.
The gloss acquisition state calculation device according to any one of [1] to [6].
[8] The image group is a video obtained by continuously photographing the object in accordance with continuous movement of any one of the light source, the object, or the camera.
The gloss acquisition state calculation device according to any one of [1] to [7].
[9] The light source is a line light source or a surface light source.
The gloss acquisition state calculation device according to any one of [1] to [8].
[10] Acquiring a group of images obtained by photographing the object with a camera in accordance with movement of at least one of the object, a light source that illuminates the object, and a camera that photographs the object;
acquiring three-dimensional shape information of the object;
estimating radiance transition information, which is information on transitions of radiance of specular reflected light across the set of images, for each surface region of the object represented by the three-dimensional shape information;
calculating acquisition state information indicating an acquisition state of the radiance transition information for each of the surface regions;
A gloss acquisition state calculation method comprising:
[11] Acquiring a group of images obtained by photographing the object with a camera in accordance with movement of at least one of the object, a light source that illuminates the object, and a camera that photographs the object;
acquiring three-dimensional shape information of the object;
estimating radiance transition information, which is information on transitions of radiance of specular reflected light across the set of images, for each surface region of the object represented by the three-dimensional shape information;
calculating acquisition state information indicating an acquisition state of the radiance transition information for each of the surface regions;
A gloss acquisition state calculation program for causing a computer to execute the above.
[12] A terminal connected to be able to communicate with the gloss acquisition state calculation device according to [7],
A transmitting means for transmitting the image group to the gloss acquisition state calculation device;
A receiving means for receiving the instruction image from the gloss acquisition state calculation device;
A display means for displaying the instruction image;
A terminal comprising:
[13] A gloss acquisition state display program for a terminal connected to be able to communicate with the gloss acquisition state calculation device according to [7],
transmitting the set of images to the gloss acquisition state calculation device;
receiving the instruction image from the gloss acquisition state calculation device;
displaying the instruction image on a display means;
A gloss acquisition status display program for causing the terminal computer to execute the above.
[14] The shape acquisition means acquires the three-dimensional shape information by estimation from the image group.
The gloss acquisition state calculation device according to [1].
[15] The light source information acquisition means acquires the light source information by estimation from the image group.
The gloss acquisition state calculation device according to [2].
[16] The camera information acquisition means acquires the camera information by estimation from the image group.
The gloss acquisition state calculation device according to [2].
[17] The instruction image is a texture image corresponding to the three-dimensional shape information.
The gloss acquisition state calculation device according to [7].
[18] A part or all of the pixels of the instruction image correspond one-to-one to each of the surface regions.
The gloss acquisition state calculation device according to [7].
[19] The correspondence between the pixels of the instruction image and the surface region is based on the set of images captured by the camera.
The gloss acquisition state calculation device according to [7].
[20] The method further includes a control unit that controls a position of any one of the object, the light source, and the camera based on the acquisition state information.
The gloss acquisition state calculation device according to [1].

1...3D image processing system, 2...image analysis device, 3...shape measurement device, 4...camera, 5...light source, 6...image display device, 7...object, 8...terminal, 21...processor, 22...ROM, 23...RAM, 24...storage, 25...input interface, 26...output interface, 27...communication interface, 28...bus, 211...image acquisition unit, 212...shape acquisition unit, 213...camera information acquisition unit, 214...light source information acquisition unit, 215...radiance transition, 216...calculation unit, 217...pointed image generation unit, 218...light source control unit, 219...texture information estimation unit.

Claims (11)

an image acquisition means for acquiring a group of images obtained by photographing the object with a camera in accordance with movement of at least one of the object, a light source that illuminates the object, and a camera that photographs the object;
A shape acquisition means for acquiring three-dimensional shape information of the object;
a radiance transition estimation means for estimating, for each surface region of the object represented by the three-dimensional shape information, radiance transition information which is information on a transition of the radiance of specular reflected light across the group of images;
A calculation means for calculating, for each of the surface regions, acquisition state information indicating an acquisition state of the radiance transition information;
Equipped with
The calculation means is
determining, for each surface region, whether or not there is a position of the light source that is in a specular reflection direction with the surface region as a reflecting surface, with respect to the camera position, using a normal of the surface region calculated from the three-dimensional shape information, a position of the camera indicated by camera position information that is information on the position of the camera across the image group, and a position of the light source indicated by light source position information that is information on the position of the light source across the image group;
calculating the acquisition state information based on a result of determining whether or not the position of the light source is present;
Gloss acquisition state calculation device. a texture information estimating unit configured to estimate a reflection constant and a normal of a predetermined reflection model for each of the surface regions using the radiance transition information, the three-dimensional shape information, the camera position information, and the light source position information,
The gloss acquisition state calculation device according to claim 1 . The calculation means is
calculating a peak of a plurality of radiances indicated by the radiance transition information for each of the surface regions, and determining whether or not a predetermined number of radiances are included in a threshold range calculated based on the peak;
calculating the acquisition state information based on a result of determining whether or not at least a predetermined number of the radiances are present;
The gloss acquisition state calculation device according to claim 1 or 2 . The threshold range is specified by a user.
The gloss acquisition state calculation device according to claim 3 . an indication image generating means for generating an indication image having a plurality of pixels each corresponding to any one of the surface regions, each pixel having a pixel value based on the acquisition state information of the corresponding surface region;
The gloss acquisition state calculation device according to any one of claims 1 to 4 . The image group is a video obtained by continuously photographing the object in accordance with continuous movement of any one of the light source, the object, or the camera.
The gloss acquisition state calculation device according to any one of claims 1 to 5 . The light source is a line light source or a surface light source.
The gloss acquisition state calculation device according to any one of claims 1 to 6 . acquiring a group of images obtained by photographing the object with a camera in accordance with movement of at least one of the object, a light source that illuminates the object, and a camera that photographs the object;
acquiring three-dimensional shape information of the object;
estimating radiance transition information, which is information on transitions of radiance of specular reflected light across the set of images, for each surface region of the object represented by the three-dimensional shape information;
calculating acquisition state information indicating an acquisition state of the radiance transition information for each of the surface regions;
Equipped with
Calculating the acquisition state information
determining, for each surface region, whether or not there is a position of the light source that is in a specular reflection direction with the surface region as a reflecting surface, with respect to the camera position, using a normal of the surface region calculated from the three-dimensional shape information, a position of the camera indicated by camera position information that is information on the position of the camera across the image group, and a position of the light source indicated by light source position information that is information on the position of the light source across the image group;
calculating the acquisition state information based on a result of determining whether or not the position of the light source is present;
Gloss acquisition state calculation method. acquiring a group of images obtained by photographing the object with a camera in accordance with movement of at least one of the object, a light source that illuminates the object, and a camera that photographs the object;
acquiring three-dimensional shape information of the object;
estimating radiance transition information, which is information on transitions of radiance of specular reflected light across the set of images, for each surface region of the object represented by the three-dimensional shape information;
calculating acquisition state information indicating an acquisition state of the radiance transition information for each of the surface regions;
A gloss acquisition state calculation program for causing a computer to execute the above .
Calculating the acquisition state information
determining, for each surface region, whether or not there is a position of the light source that is in a specular reflection direction with the surface region as a reflecting surface, with respect to the camera position, using a normal of the surface region calculated from the three-dimensional shape information, a position of the camera indicated by camera position information that is information on the position of the camera across the image group, and a position of the light source indicated by light source position information that is information on the position of the light source across the image group;
calculating the acquisition state information based on a result of determining whether or not the position of the light source is present;
Gloss acquisition state calculation program . A terminal connected to be able to communicate with the gloss acquisition state calculation device according to claim 5 ,
A transmitting means for transmitting the image group to the gloss acquisition state calculation device;
A receiving means for receiving the instruction image from the gloss acquisition state calculation device;
A display means for displaying the instruction image;
A terminal comprising: A gloss acquisition state display program for a terminal communicably connected to the gloss acquisition state calculation device according to claim 5 ,
transmitting the set of images to the gloss acquisition state calculation device;
receiving the instruction image from the gloss acquisition state calculation device;
displaying the instruction image on a display means;
A gloss acquisition status display program for causing the terminal computer to execute the above.