JP2020119333A - Image processing method, image processing device, image processing system, imaging device, program, and storage medium - Google Patents

Abstract

To provide an image processing method capable of estimating normal information with high accuracy in a general object having no limitation on a reflection characteristic and a shape.SOLUTION: An image processing method includes a first step for acquiring a first normal map of an object space and reference data as input data, a second step for acquiring learning information, and a third step for estimating a second normal map on the basis of the input data and the learning information. The first normal map is either a normal map acquired by using a photometric stereo method or a shape from shading method or a normal map acquired by using a distance map. The reference data includes a photographic image obtained by photographing the object space. The third step estimates the second normal map by executing a step for executing the n-th linear transformation and the n-th non-linear transformation based on the learning information of the input data until an integer n becomes N(≥2) from 1 to generate intermediate data, and a step for executing the N+1-th linear transformation based on the learning information of the intermediate data.SELECTED DRAWING: Figure 3

Description

The present invention relates to an image processing method, an image processing device, an image processing system, an imaging device, a program, and a storage medium for acquiring normal line information of a subject.

A method is known in which surface normal information (hereinafter referred to as normal information) is acquired as shape information of a subject from a captured image obtained by capturing an image of the subject with an imaging device such as a digital camera. As a method of acquiring the normal line information, there are a shape from shading method and a photometric stereo method. In the shape-from-shading method, normal line information can be estimated from a single captured image, but it is necessary to assume that the reflectance of the target object is uniform and that the shape of the subject changes smoothly. The photometric stereo method is a method of assuming reflection characteristics based on a surface normal of a subject and a light source direction, and estimating normal line information from luminance information of the subject at a plurality of light source positions and the assumed reflection characteristics. The normal information can be estimated under a smaller number of assumptions than the shape-from-shading method by using captured images captured at a plurality of light source positions. A Lambert reflection model that follows Lambert's cosine law is often used as the assumed reflection characteristic of the subject.

Generally, reflection on an object includes specular reflection and diffuse reflection. The specular reflection is regular reflection on the object surface and Fresnel reflection according to the Fresnel equation on the object surface (interface). Diffuse reflection is reflection in which light returns after being transmitted through the surface of an object and then scattered inside the object. The specularly reflected light cannot be expressed by the Lambert's cosine law described above.If specular reflected light is included in the reflected light from the subject observed by the image pickup device, the normal information can be obtained by the shape from shading method or photometric stereo method. Cannot be accurately estimated. Even in the shaded area where the light from the light source does not illuminate, deviation from the assumed reflection model occurs, and normal information cannot be accurately estimated. In addition, the diffuse reflection component also deviates from Lambert's cosine law in a subject with a rough surface or a semitransparent body. Furthermore, normal information cannot be accurately estimated even when mutual reflection occurs or even in a metal body or a transparent body where a diffuse reflection component is not observed.

Patent Document 1 discloses a method of highly accurately obtaining normal line information from a plurality of normal line candidates obtained by using four or more light sources by removing the influence of specular reflection components. In addition, Non-Patent Document 1 discloses a method of estimating normal line information from one captured image by applying a convolutional neural network.

JP, 2010-122158, A

D. Eigen, et al. "Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture", 1.4ar34(141).

However, the method disclosed in Patent Document 1 cannot estimate normal line information when there are a plurality of captured images affected by the specular reflection component. In addition, normal information cannot be estimated even when a shaded portion is generated, a subject having a reflection characteristic deviated from Lambert's cosine law, mutual reflection occurs, or a subject in which a diffuse reflection component is not observed. .. The method disclosed in Non-Patent Document 1 is less likely to cause a failure due to the inability to estimate normal information, but has a low estimation accuracy with respect to the normal map acquired with high accuracy by the photometric stereo method or the like.

The present invention relates to an image processing method, an image processing device, an image processing system, an imaging device, an image processing program, and a storage device, which are capable of estimating normal line information with high accuracy in a general subject having no reflection characteristic or shape constraint. The purpose is to provide a medium.

An image processing method according to one aspect of the present invention includes a first step of acquiring a first normal map of a subject space and reference data as input data, and a second step of acquiring learning information learned in advance. And a third step of estimating a second normal map based on the input data and the learning information, wherein the first normal map is obtained by using a photometric stereo method or a shape from shading method. The normal map obtained by using the normal map and the normal map obtained by using the distance map, the reference data includes a captured image of the subject space, and in the third step, N is an integer of 2 or more, n Is an integer from 1 to N, the n-th linear conversion based on the learning information and the n-th nonlinear conversion are sequentially executed on the input data until n becomes 1 to N, and the intermediate data Is generated, and a step of executing the (N+1)th linear transformation on the intermediate data based on the learning information is performed, whereby the second normal map is estimated.

An image processing method according to another aspect of the present invention includes a first step of acquiring a first normal line map of a subject space and reference data as input data, and a second step of acquiring learning information learned in advance. And a third step of estimating a second normal map based on the input data and the learning information, and the reference data is a region of the captured image obtained by capturing the subject space based on the characteristics of the subject. At least one of a labeled label map, a reliability map indicating the reliability of the normal information included in the first normal map, and a distance map is included. In the third step, N is an integer of 2 or more, When n is an integer from 1 to N, the n-th linear conversion based on the learning information and the n-th nonlinear conversion are sequentially executed on the input data until n becomes 1 to N The second normal map is estimated by executing the step of generating data and the step of executing the (N+1)th linear transformation based on the learning information with respect to the intermediate data.

According to the present invention, an image processing method, an image processing apparatus, an image processing system, an image pickup apparatus, an image processing program, and an image processing method capable of estimating normal line information with high accuracy in a general subject having no reflection characteristic or shape constraint. A storage medium can be provided.

3 is an external view of the image pickup apparatus of Embodiment 1. FIG. 3 is a block diagram of the image pickup apparatus of Embodiment 1. FIG. 6 is a flowchart showing a second normal map estimation process of the first embodiment. It is a figure which shows the network structure of CNN(Convolutional Neural Network) which is one of the deep learning (Examples 1 and 2). 5 is a flowchart showing learning of learning information according to the first embodiment. 6 is an external view of an image processing system of Example 2. FIG. 5 is a block diagram of an image processing system of Example 2. FIG. 9 is a flowchart illustrating a second normal map estimation process according to the second embodiment.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In each of the drawings, the same members are designated by the same reference numerals, and duplicated description will be omitted.

In the present invention, the second normal line obtained by improving the first normal line map by using deep learning (also referred to as deep learning) based on the first normal line map having a broken portion or an area with insufficient accuracy is used. Estimate the map. The normal line map is a map in which normal line information of the subject space is arranged as an image, and the normal line information indicates a normal direction vector and each degree of freedom representing the normal line. In addition to the first normal map, at least one of a captured image, a label map, a confidence map, or a distance map is input as input for deep learning, and the second normal can be obtained by referring to these data. Estimate the map. By using the input data consisting of the first normal map and at least one of the photographed image, the label map, the reliability map, and the distance map, the correspondence relationship is learned by deep learning, so that the normal line can be accurately obtained. Information can be estimated.

In this embodiment, a case where the image processing method of the present invention is applied to the image pickup apparatus 100 will be described. FIG. 1 is an external view of the image pickup apparatus 100. FIG. 2 is a block diagram of the image pickup apparatus 100.

The imaging device 100 includes an imaging unit 101 that acquires an image of the subject space as a captured image. The image pickup unit 101 includes an image forming optical system 101a that collects light incident from a subject space, and an image pickup element 101b having a plurality of pixels. The image sensor 101b is, for example, a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal-Oxide Semiconductor) sensor. The image processing unit 102 acquires a partial area of the first normal map of the subject space, and estimates normal information (second normal map) for the input data combined with the corresponding area of the captured image. I do. The image processing unit 102 has a learning unit 102a, an estimation unit 102b, and an acquisition unit 102c. When estimating the second normal map, the learning information stored in the storage unit 103 is called and used. The estimated second normal map is displayed on the display unit 104 such as a liquid crystal display or saved in the recording medium 105. Instead of the second normal map, an image (for example, a rendering image) generated based on the second normal map may be displayed on the display unit 104 or stored in the recording medium 105. However, the first normal map, the captured image, or the input data may be stored in the recording medium 105, and the second normal map may be estimated at an arbitrary timing. The light source 110 is selectively turned on at the time of image capturing, and enables the image capturing unit 101 to perform image capturing under a plurality of different light source environments. In the present embodiment, the light source 110 is configured integrally with the imaging device 100, but may be an external device different from the imaging device 100. The system controller 106 performs the series of controls described above.

Hereinafter, the estimation of the second normal map executed by the image processing unit 102 will be described with reference to FIG. FIG. 3 is a flowchart showing the estimation process of the second normal map.

In step S101, the acquisition unit 102c acquires a plurality of photographed images of a subject under different light source environments and learning information. The learning information is information learned in advance for connecting the estimated second normal map to the input data described later.

In step S102, the acquisition unit 102c acquires, as the input data, the partial area of the first normal map and the partial areas of the plurality of captured images of the subject captured under different light sources. In the present embodiment, as the first normal map, one obtained by the photometric stereo method from a plurality of captured images obtained by capturing the subject under different light source environments by the acquisition unit 102c in advance is used. Further, the areas acquired from each of the plurality of captured images are selected so as to be at the same position on the images. Note that position shift processing such as electronic camera shake correction processing may be performed on each of the plurality of captured images. Further, it is not necessary to use all the captured images as the input data, and one captured image may be used, or a captured image different from the captured image used for the photometric stereo method may be used. The estimation of the second normal map is performed for each partial area with the input data as a unit. The partial area may be a partial area of each captured image, and may be the entire captured image. Further, the method of acquiring the first normal map is not limited to the photometric stereo method, but may be acquired based on the shape-from-shading method or the distance map of the subject space acquired from the parallax image. In this case, in step S101, a captured image required for the first normal map acquisition method and input data may be acquired, and it may be captured using a predetermined light source or only ambient light. Only one captured image or parallax image may be acquired. The first normal map may be stored in the recording medium 105 in advance. When the first normal map is acquired from the recording medium 105, only the captured image necessary for the input data needs to be acquired in step S101.

In step S103, the estimation unit 102b estimates the partial normal information from the input data using the learning information. In step S103, the input data is processed using a multilayer neural network. Hereinafter, the details of the estimation performed in step S103 will be described with reference to FIG.

FIG. 4 shows the network structure of CNN (Convolutional Neural Network), which is one of the deep learning methods. However, as long as it is a multilayer neural network, a method other than CNN (for example, DBN (Deep Belief Network)) may be used.

The CNN has a plurality of layered structures, and linear conversion and non-linear conversion based on learning information are executed in each layer. The nonlinear conversion may or may not be based on the learning information. Here, when n is an integer from 1 to N, the nth layer is the nth layer, and the linear conversion and the nonlinear conversion in the nth layer are the nth linear conversion and the nth nonlinear conversion, respectively. Call it. However, N is an integer of 2 or more. The input data 201 is subjected to convolution (first linear conversion by a plurality of linear functions) with each of the plurality of filters 202 in the first layer. After that, conversion (first nonlinear conversion) is performed by a nonlinear function called an activation function (AF in FIG. 4). Further, the plurality of input data 201 are drawn because they have a plurality of channels. In this embodiment, in addition to the number of channels of the first normal map, the input data has three channels of RGB (Red, Green, Blue) for each photographed image used for the input data. The number of channels in the first normal map differs depending on the method of expressing the normal information. If the three-dimensional component of the normal vector is assigned to each channel, there are three channels, and if the direction of the normal vector is expressed by two angles, there are two channels. Even if the input data has RGB channels, each color may be individually input to the CNN.

There are a plurality of filters 202, and the convolution of each of them and the input data 201 is individually calculated. The coefficient of the filter 202 is determined from the learning information. The learning information may be the filter coefficient itself or a coefficient when the filter is fitted with a predetermined function. The number of channels of each filter 202 matches that of the input data 201 and becomes a three-dimensional filter (the third dimension represents the number of channels). Further, a constant (which can be negative) determined from the learning information may be added to the result of convolution.

Examples of the activation function f(x) include the following equations (1) to (3).

The formula (1) is called a sigmoid function, the formula (2) is called a hyperbolic tangent function, and the formula (3) is called ReLU (Rectified Linear Unit). Max in Expression (3) represents a MAX function that outputs the maximum value among the arguments. Equations (1) to (3) are all monotonically increasing functions. Alternatively, Maxout may be used as the activation function. Maxout is a MAX function that outputs the signal value that is the maximum value in each pixel of the plurality of images that are the output of the n-th linear conversion.

The input data that has been subjected to the first linear conversion and the first nonlinear conversion is referred to as first conversion input data 203. Each channel component of the first converted input data 203 is generated from the convolution of the input data 201 and the filter 202. Therefore, the number of channels of the first conversion input data 203 is the same as the number of filters 202.

In the second layer, with respect to the first transformed input data 203, convolution (second linear transformation) with a plurality of filters 204 determined from the learning information as in the first layer, and non-linearity by the activation function The conversion (second nonlinear conversion) is performed. The filter 204 used in the second layer is typically not the same as the filter 202 used in the first layer. The size and number of filters do not have to match. However, the number of channels of the filter 204 matches the number of channels of the first conversion input data 203. By repeating the same calculation up to the Nth layer, the intermediate data 210 is obtained. Finally, the partial normal information 212 obtained by estimating the second normal map for the input data 201 is obtained from the convolution of the intermediate data 210 and the filter 211 and the addition of the constant (N+1th linear conversion) in the (N+1)th layer. .. The filter 211 and the constant are also determined from the learning information. The number of channels of the partial normal information 212 depends on the method of expressing the normal information and is equal to the number of channels of the first normal map. Therefore, the number of filters 211 is the same as the number of channels of the partial normal information 212. The component of each channel of the partial normal information 212 is obtained from the calculation including the convolution of the intermediate data 210 and the filter 211. Note that the sizes of the input data 201 and the partial normal information 212 do not have to match. Since data does not exist outside the input data 201 during convolution, the size of the convolution result becomes small if the calculation is performed only in the area where the data exists. However, the size can be maintained by providing periodic boundary conditions. In addition, in the present embodiment, the coefficients of the respective filters relating to the m-th linear transformation (m=1 to N+1) are all different.

The reason why deep learning can exhibit high performance is that high nonlinearity can be obtained by performing nonlinear conversion many times with a multilayer structure. If there is no activation function responsible for non-linear conversion and the network is configured only with linear conversion, there is a single-layer linear conversion equivalent to that, no matter how many layers are used. There is no. It is said that deep learning is more likely to produce higher performance because a stronger multilayer can be obtained by using more layers. Generally, the case of having at least three layers or more is called deep learning. By using the deep learning configured as described above, it is possible to estimate the normal vector information (second normal vector map) in which the breakdown portion is reduced.

In step S104, the estimation unit 102b determines whether or not the estimation of the second normal map has been completed for a predetermined area in the captured image. If the partial normal information is generated for all the predetermined areas, the process proceeds to step S105. If not, the process returns to step S102, and the input data for which the normal information is not estimated is acquired.

In step S105, the estimation unit 102b outputs the second normal line map. The second normal map is generated by combining the generated partial normal information. When the input data is the entire image area, the partial normal information may be directly used as the second normal map.

With the above processing, the second normal map can be estimated by referring to the captured image from the first normal map.

Even if the parallax image is not used when acquiring the first normal map, a plurality of captured images with different viewpoints may be input to obtain the second normal map. Since the reflection angle of light changes depending on the viewpoint, the influence of the reflection characteristics of the subject also changes. The estimation accuracy can be improved by inputting a plurality of captured images that have received different light reflections for the same subject.

Hereinafter, learning of learning information will be described with reference to FIG. FIG. 5 is a flowchart showing learning of learning information. The learning may be performed by the learning unit 102a or may be performed by an arithmetic device other than the imaging device 100 before the estimation of the second normal map. In this embodiment, a case where learning is performed by the learning unit 102a will be described.

In step S201, the learning unit 102a acquires a pair of learning data. The pair of learning data is the learning input data including the first normal map and the captured image, and the learning normal map with higher accuracy than the first normal map of the same subject. Here, although the first normal map differs depending on the acquisition method, there is a broken portion (low accuracy or a portion that cannot be acquired) caused by a low resolution or a condition that each acquisition method is not good at. In the present embodiment, an estimation process for correcting the first normal map to the more accurate second normal map is performed. Therefore, in the learning data, it is necessary to align the conditions under which the first normal map fails and the effects of the failure with the time of estimation. Therefore, the first normal map used at the time of learning is the data acquired in the same manner as the first normal map used at the time of estimation, and the photographed image used at the time of learning should correspond to the photographed image used at the time of estimation, the number of images, and photographing conditions. Is desirable. As the highly accurate normal information (learning normal map) used as learning data, it is necessary to use normal information that is more accurate than the first normal map and has no broken portions. The accuracy of the estimated second normal map changes depending on the accuracy of this normal information.

It is desirable that the high-accuracy normal line information used as the learning data includes subjects having various shapes and normal vectors, and subjects having various reflection characteristics. For example, when there is no glossy subject in the learning data, the learning data using the first normal map, the captured image, the reference data described later, and the second normal map for the subject having the same reflection characteristic is obtained. not exist. Therefore, there is a possibility that the effect of estimating the normal information for the same subject cannot be sufficiently obtained.

As a method for preparing the learning data, a simulation may be used or actually measured information may be used. When performing a simulation, an image corresponding to a captured image is generated by performing CG rendering on a 3D model with reflection characteristics, and based on an acquisition method equivalent to that of the acquisition unit 102c from the generated image, It is sufficient to acquire the normal map of No. 1. Since the 3D model is known, the normal map for learning is known.

When the measured information is used, a subject whose shape is known (a subject whose normal line information is known) is photographed, and normal line information corresponding to the first normal line map is obtained based on an acquisition method equivalent to that of the acquisition unit 102c. Just get it. As a result, accurate normal line information can be obtained. Also in this case, the learning normal map is already known.

Further, it is also possible to train a subject whose shape is unknown by using normal line information (second normal line map) acquired by using a different method. As an example, the shape can be acquired by laser ranging, the shape can be acquired from the image illuminated by structured illumination, the shape can be acquired from the reflection direction of the specular reflection and the incident direction of the light source, or with a contact-type measuring device. There is a method of acquiring the shape. By selecting an appropriate acquisition method according to conditions such as the size, shape, and reflection characteristics of the subject, normal line information can be obtained for a general subject with higher accuracy than the first normal line map. ..

Here, the conditions under which the first normal map easily breaks will be described. First, the photometric stereo method will be described. When the normal line is obtained by the photometric stereo method, the normal line direction is obtained from the change in brightness when the images are taken under different light source environments. Some photometric stereo methods correspond to general reflection characteristics, but in many cases diffuse reflection according to the Lambert model is assumed because the number of images to be photographed and the calculation load increase. Therefore, a false normal is obtained when specular reflection is observed. Further, the normal line easily breaks down due to a change in the brightness due to a shadow portion where the incident light is shielded, a shade portion where the incident light does not hit, or mutual reflection where reflected light from another object enters. For a subject with a rough surface or a semi-transparent body with strong internal scattering, the reflection characteristics deviate from the Lambert model, and for a transparent body such as a glossy metal body or glass, diffuse reflection is not observed. The line is easy to fail. Further, when the ambient light is strong, the contribution of the incident light from the light source is relatively small, and the acquisition accuracy is reduced. A Lambert model is often assumed also in the shape-from-shading method, and a similar subject is likely to fail. In addition, when assuming a continuous change in the normal line, fine irregularities may not be reflected.

When the normal information is acquired from the distance map, incorrect normal information is estimated when the distance discontinuously changes at the boundary with respect to the distant object. Since the normal information corresponds to the differentiation of the distance information, it is also strongly affected by the noise in the distance map. Further, there are cases where the distance map itself has a failure depending on the method of acquiring the distance map, and in this case, the normal line information also fails. For example, when acquiring a distance from a multi-viewpoint image based on the correlation between viewpoints, a specular reflector or a transparent body that has a different appearance for each viewpoint, occlusion, an object with a small amount of texture with a small change in brightness at each position, or a periodic object. It is easy for the distance to collapse due to subjects with different structures. When the distance is acquired by the Time of flight method, the distance cannot be acquired by a specular reflector or a transparent body that does not return light, or the accuracy of acquisition may be reduced by noise in a low reflection object or in an environment with strong external light. ..

In step S202, the learning unit 102a acquires a plurality of learning pairs from the learning data. The learning pair consists of learning input data and learning partial normal information. The input data for learning is acquired from the first normal map and the captured image. The size of the learning input data is the same as the size of the input data acquired in step S102. The learning partial normal information is acquired from the learning normal map such that the center of the area is at the same subject position as the learning input data. The size of the learning partial normal information is the same as the size of the partial normal information generated in step S103.

In step S203, the learning unit 102a acquires learning information from a plurality of learning pairs by learning. Learning uses the same network structure as the estimation of the second normal map. In this embodiment, learning input data is input to the network structure of FIG. 4, and an error between the output result and the learning partial normal information is calculated. The error of the normal information may be the difference between the components, or may be a value obtained by subtracting 1 from the inner product of the normal vector of the learning partial normal information and the normal vector of the output result. In order to minimize the error, the coefficient of each filter used in the first layer to the N+1th layer and the constant to be added (learning information) are updated and optimized by using, for example, the back propagation method (Backpropagation). .. The initial value of each filter and constant may be determined from random numbers, for example. Further, pre-training such as Auto Encoder for pre-learning the initial value may be performed for each layer.

The method of inputting all the learning pairs into the network structure and updating the learning information by using all the information is called batch learning. In batch learning, the computational load increases as the number of learning pairs increases. A learning method in which only one learning pair is used to update learning information and a different learning pair is used for each update is called online learning. In online learning, the amount of calculation does not increase even if the number of learning pairs increases, but it is greatly affected by noise existing in one learning pair. Therefore, it is desirable to use the mini-batch method, which is located between these two methods, for learning. In the mini-batch method, some pairs are extracted from all learning pairs and the learning information is updated using them. In the next update, we will extract and use different learning pairs. By repeating this, the defects of batch learning and online learning can be reduced, and a high estimation effect can be easily obtained.

In step S204, the learning unit 102a outputs the learned learning information. In this embodiment, the output learning information is stored in the storage unit 103.

By the above processing, it is possible to learn the learning information capable of estimating the normal line information with high accuracy in a general subject having no restriction on the reflection characteristic or the shape.

Further, in addition to the above processing, a device for improving the performance of CNN may be used together. For example, in order to improve robustness, pooling, which is dropout or downsampling, may be performed in each layer of the network.

Further, in the present embodiment, the input data is acquired from the partial area of the captured image, but it may be acquired from the image after performing arbitrary image processing on the captured image. For example, a diffuse reflection image that is not affected by specular reflection may be generated from the captured image, and the input data may be acquired from the diffuse reflection image. As a result, even for subjects having different reflection characteristics, learning and estimation can be performed by focusing on the components excluding specular reflection, and the estimation accuracy can be improved.

With the configuration described above, it is possible to estimate the normal line information with high accuracy in a general subject having no restriction on the reflection characteristic or shape.

In this embodiment, a case where the image processing method of the present invention is applied to the image processing system 500 will be described. FIG. 6 is an external view of the image processing system 500. FIG. 7A is a block diagram of the image processing system 500.

The image processing system 500 includes an imaging device 300 that obtains a captured image, an image processing device 301 that estimates a second normal map, and a server 305 that performs learning. In the present embodiment, at least one of a label map, a reliability map, or a distance map is used as reference data when estimating the second normal map, instead of the captured image used in the first embodiment. .. This enables highly accurate estimation of normal line information.

The basic configuration of the image capturing apparatus 300 is the same as that of the image capturing apparatus 100 according to the first embodiment except for the image processing unit that estimates the second normal map and learns learning information. The captured image captured by the image capturing apparatus 300 is stored in the storage unit 302 in the image processing apparatus 301. The image processing apparatus 301 is connected to the network 304 by wire or wirelessly, and similarly accesses the server 305 connected to the network 304. The server 305 includes a learning unit 307 that learns learning information for estimating the second normal map from the first normal map and reference data acquired by the acquisition unit 311, and a storage unit that stores the learning information. 306 and. The estimation unit 303 acquires the learning information from the storage unit 306 and estimates the second normal line map. The second normal map is output to at least one of the display device 308, the recording medium 309, and the output device 310. An image (for example, a rendering image) generated based on the second normal map may be output instead of the second normal map.

The display device 308 is, for example, a liquid crystal display or a projector. The user can work while confirming the normal line information during the processing via the display device 308. The recording medium 309 is, for example, a semiconductor memory, a hard disk, a server on the network, or the like. The output device 310 is, for example, a printer or the like. The image processing apparatus 301 may have a function of performing development processing and other image processing as needed.

Hereinafter, the estimation of the second normal map executed by the image processing apparatus 301 will be described with reference to FIG. FIG. 8 is a flowchart showing the estimation process of the second normal line map.

In step S301, the estimation unit 303 acquires a captured image from the storage unit 302.

In step S302, the estimation unit 303 acquires the first normal line map and the reference data. The first normal map is acquired in advance by the acquisition unit 311. In this embodiment, the reference data is at least one of a label map, a reliability map, and a distance map. In addition, you may use a picked-up image as reference data like Example 1. The first normal map and the distance map may be acquired as described in the first embodiment. The label map is a distribution in which each area of the captured image is labeled based on the characteristics of the subject. The characteristics of the subject refer to, for example, transmission/reflection characteristics. By labeling each material of the subject, it is possible to know what transmission/reflection characteristics the subject has. Further, when acquiring the first normal map such as a specular reflection area, a mutual reflection area, or a shadow area, an area that causes a failure may be acquired and labeled. The reliability map is a distribution indicating how reliable the normal information is at each position of the first normal map. The reliability map is, for example, when the first normal map is acquired by the photometric stereo method, the difference amount between the brightness value estimated from the first normal map and the Lambert model and the actual brightness value. Can be generated from.

In step S303, the estimation unit 303 determines a network structure to be used and learning information, a size of input data, and an area to be estimated based on the reference data. In this embodiment, the estimation unit 303 estimates the normal line information using the CNN described with reference to FIG. Further, in this embodiment, since the learning information individually learned according to the reference data is used, the network structure and the size of the input data are the same as those used at the time of learning. By using the learning information according to the acquired reference data, the second normal map can be estimated with higher accuracy.

When a label map is used as the reference data, areas that are likely to fail in the first normal map are labeled according to their types, and areas that are less likely to fail are also labeled. For example, in the case where a subject such as a metal body or a transparent body having different reflection characteristics is labeled, learning information learned for each type of subject is used. This enables more accurate estimation. The network structure to be used, learning information, and the size of input data are determined by the learning conditions. Further, the region where the estimation is performed may be limited to the region labeled as the region where the failure is likely to occur, whereby the estimation can be performed at high speed.

When the reliability map is used as the reference data, the degree of possibility that a failure has occurred in the first normal map is represented by the reliability. Therefore, for example, the smaller the reliability, the larger the size of the input data. Also, the size of the input data may be determined so as to include a region with high reliability. This enables more accurate estimation. Further, the region where the estimation is performed may be limited to the region where the reliability is lower than a predetermined threshold value, whereby the estimation can be performed at high speed.

Next, when a distance map is used as the reference data, the acquisition accuracy of the first normal map may change depending on the distance. For example, when the light sources of the image pickup apparatus 300 are sequentially turned on and the normal line is obtained by the photometric stereo method, the light source is fixed to the image pickup apparatus 300. The difference in the direction of the light source between the captured images becomes small, and it becomes difficult for the brightness to change. The accuracy of normal acquisition decreases as the change in brightness is less likely to occur, so the distance map corresponds to the reliability of the normal. Therefore, the size of the input data may be changed according to the distance map. Further, at the point where the distance map changes discontinuously, it is the boundary area of the distant object and it is easy to calculate an incorrect normal line when acquiring the normal map from the distance map. Therefore, in the boundary area, learning information learned using the data of the boundary area may be used, or the estimation may be executed only in the boundary area.

The network structure includes not only the size of filters used in each layer, but also the number of filters used in one layer, the number of layers, and the like.

The learning information is learned according to the reference data, and the corresponding learning information is used. This enables more accurate estimation.

In step S304, the estimation unit 303 acquires input data from the first normal map and reference data.

In step S305, the estimation unit 303 generates partial normal information from the learning information.

In step S306, the estimation unit 303 determines whether or not the estimation of the default area has been completed. If the estimation of the predetermined area is completed, the process proceeds to step S307, and if not, the process returns to step S304.

In step S307, the estimation unit 303 outputs the second normal line map.

Note that step 304 may be executed before steps S302 and S303. In that case, in steps S302 and S303, the first normal map and reference data are acquired for the partial area acquired in step 304, and corresponding learning information and the like are acquired.

Hereinafter, learning of learning information executed by the learning unit 307 will be described. In this embodiment, different learning information is learned for each reference data. The learning method follows the flowchart of FIG.

First, a case where learning data is generated by simulation (CG rendering) will be described. For example, a pair of learning data corresponding to the reflection characteristic is obtained by setting a predetermined reflection characteristic, generating a rendering image from the normal information, and acquiring a first normal map from the rendering image. When associating a plurality of similar reflection characteristics as one label, it is desirable to obtain learning data for each of the similar reflection characteristics. Steps S201 to S204 are executed on the learning data, and then the same procedure is repeated for the subject having different reflection characteristics. Even when the reliability map or the distance map is used as the reference data, the same procedure may be learned for each characteristic of the subject region.

A case will be described in which learning data is generated using a real subject whose shape is already known. In this case, learning data is obtained by acquiring a first normal map and reference data used for estimation for a subject having a known shape. When the learning information is changed according to the distance to the subject, the captured image, the first normal map, and the reference data may be acquired while changing the distance to the subject. When the reflection characteristics of the subject are known, learning data may be acquired while changing to a subject having a different reflection characteristic. Even if the reflection characteristics are not exactly known, learning data may be acquired while changing the material. In this case, since it is difficult to acquire the reliability map in advance, it is necessary to acquire the reliability map at the same time as the first normal map and acquire the learning data for the subject spaces of various reliability. Then, the learning information is generated by executing steps S201 to S204 for each learning data that falls into the same classification.

In the present embodiment, the image processing system 500 shown in FIG. 7A has been described as an example of the image processing system to which the present invention can be applied, but the present invention is not limited to this. For example, as shown in FIG. 7B, the server 401 may have the functions of the image processing device 301 and the server 305 of FIG. 7A. In this case, the server 401 acquires captured images output from an image output device such as an imaging device, a smartphone, a tablet terminal, and a personal computer (PC) via the network 402, and the present invention is applied to the acquired captured images. The image processing method of is executed.

With the configuration described above, it is possible to estimate the normal line information with high accuracy in a general subject having no restriction on the reflection characteristic or shape.
(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. It can also be realized by the processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

According to each embodiment, an image processing device, an imaging device, an image processing method, an image processing program, and a storage medium capable of accurately estimating normal line information in a general subject having no reflection characteristic or shape constraint are provided. Can be provided.

Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes can be made within the scope of the gist thereof.

201 Input data 210 Intermediate data

Claims (19)

A first step of acquiring a first normal map of the subject space and reference data as input data;
A second step of acquiring learning information learned in advance;
A third step of estimating a second normal map based on the input data and the learning information,
The first normal map is either a normal map acquired using a photometric stereo method or a shape from shading method, and a normal map acquired using a distance map,
The reference data includes a captured image of the subject space,
In the third step, when N is an integer greater than or equal to 2 and n is an integer from 1 to N, the nth linear conversion based on the learning information and the nth nonlinear conversion are performed on the input data. Is executed in order from n to 1 to N to generate intermediate data, and a step of performing the (N+1)th linear conversion on the intermediate data based on the learning information is executed. The image processing method, wherein the second normal map is estimated by The image processing method according to claim 1, wherein the captured images are a plurality of images obtained by capturing the subject space under a plurality of different light source environments. In the first step, a partial area of the first normal map and a partial area of the reference data are acquired as the input data,
The image processing method according to claim 2, wherein the input data includes partial areas at the same position extracted from each of the plurality of images. The reference data includes a label map that labels each area of the captured image based on the characteristics of the subject, a reliability map that represents the reliability of normal line information included in the first normal line map, and a distance map. The image processing method according to claim 1, wherein at least one of them is included. A first step of acquiring a first normal map of the object space and reference data as input data;
A second step of acquiring learning information learned in advance,
A third step of estimating a second normal map based on the input data and the learning information,
The reference data is a label map that labels each area of a captured image in which the subject space is captured based on the characteristics of the subject, and a reliability map that indicates the reliability of normal line information included in the first normal line map. , And at least one of the distance maps,
In the third step, when N is an integer of 2 or more and n is an integer from 1 to N, the nth linear conversion and the nth nonlinear conversion based on the learning information are performed on the input data. Is executed in order from n to 1 to N to generate intermediate data, and a step of performing (N+1)th linear conversion on the intermediate data based on the learning information is executed. The image processing method, wherein the second normal map is estimated by In the first step, a partial area of the first normal map and a partial area of the reference data are acquired as input data,
The image processing method according to claim 4, wherein the partial region is extracted based on at least one of the label map, the reliability map, and the distance map. 7. The partial area is an area of a label representing an object whose accuracy is low in the first normal map of the label map or an area of low reliability of the reliability map. The image processing method described in. The learning information includes learning input data including a first normal map and reference data regarding a subject space different from the subject space, and a learning method different from the first normal map corresponding to the learning input data. Information learned based on the line map,
The image processing method according to claim 1, wherein the learning input data includes a subject whose accuracy is low in the first normal map. 9. The image processing method according to claim 8, wherein the learning input data includes at least one of a subject that is specularly reflected, a subject that has a rough surface, a metal body, a transparent body, and a translucent body. The image processing method according to claim 1, wherein the learning information corresponding to the linear conversion is determined based on the reference data. The image processing method according to claim 10, wherein the size of the filter used for the linear conversion is determined based on the reference data. The image processing method according to claim 1, wherein the first normal map is a normal map acquired based on a photometric stereo method or a shape-from-shading method. .. An acquisition unit that acquires the first normal map of the object space and the reference data as input data, and acquires learning information learned in advance;
An estimation unit that estimates a second normal map based on the input data and the learning information,
The first normal map is either a normal map acquired using a photometric stereo method or a shape from shading method, and a normal map acquired using a distance map,
The reference data includes a captured image of the subject space,
When N is an integer greater than or equal to 2 and n is an integer from 1 to N, the estimation unit performs the nth linear conversion and the nth nonlinear conversion on the input data based on the learning information. , N in sequence from 1 to N to generate intermediate data, and performing the N+1th linear transformation based on the learning information on the intermediate data. An image processing apparatus characterized by estimating a second normal map. An acquisition unit that acquires the first normal map of the object space and the reference data as input data, and acquires learning information learned in advance;
An estimation unit that estimates a second normal map based on the input data and the learning information,
The reference data is a label map that labels each area of a captured image in which the subject space is captured based on the characteristics of the subject, and a reliability map that indicates the reliability of normal line information included in the first normal line map. , And at least one of the distance maps,
When N is an integer greater than or equal to 2 and n is an integer from 1 to N, the estimation unit performs the nth linear conversion and the nth nonlinear conversion on the input data based on the learning information. , N in sequence from 1 to N to generate intermediate data, and performing the N+1th linear transformation based on the learning information on the intermediate data. An image processing apparatus characterized by estimating a second normal map. The image processing apparatus according to claim 13, further comprising a storage unit that stores the learning information. The image processing apparatus according to claim 13,
An image processing apparatus, comprising: an image output device that outputs a captured image of a subject space to the image processing device. An imaging unit that acquires an image of the subject space as a captured image,
An image pickup apparatus comprising: an image processing unit that executes the image processing method according to claim 1 on the captured image. A program for causing a computer to execute the image processing method according to any one of claims 1 to 12. A computer-readable storage medium storing the program according to claim 18.