JP6910763B2 - Processing equipment, processing systems, imaging equipment, processing methods, programs, and recording media - Google Patents

Description

The present invention relates to a processing device, a processing system, an imaging device, a processing method, a program, and a recording medium.

By acquiring more physical information about the subject, it is possible to generate an image based on the physical model in the image processing after imaging. For example, it is possible to generate an image in which the appearance of the subject is changed. The appearance of the subject is determined by information such as the shape information of the subject, the reflectance information of the subject, or the light source information. Since the physical behavior of the reflected light emitted from the light source and reflected by the subject depends on the local surface normal, it is particularly effective to use the surface normal information instead of the three-dimensional shape as the shape information.

Conventionally, an illuminance difference stereo method is known in which the reflection characteristics based on the surface normal of the subject and the direction of the light source are assumed, and the surface normal is determined from the reflection characteristics assumed to be the brightness information of the subject at a plurality of light source positions. (See, for example, Non-Patent Document 1). As the reflection characteristic of the subject, a Lambertian reflection model that follows Lambert's cosine law is often used.

Generally, the reflected light of an object has each component of specular reflected light and diffuse reflected light. Specular reflection light is regular reflection on the surface of an object, and refers to Fresnel reflection according to Fresnel's equation on the surface (interface) of an object. Diffuse reflected light refers to light that is scattered inside an object and returned after passing through the surface of the subject. Since the specular reflection component cannot be expressed by Lambert's cosine law, if the specular reflection component is included in the reflected light from the subject observed by the imaging device, the surface normal is calculated accurately using the illuminance difference stereo method. Can't. Even in the shaded area where the light from the light source does not hit, there is a deviation from the assumed reflection model, and it is not possible to accurately acquire the surface normal information of the subject. For example, Patent Document 1 discloses a method of obtaining a true surface normal from a plurality of surface normal candidates obtained by using four or more light sources.

Further, an imaging device including a flash device as a light source is known in order to secure the amount of light when photographing a low-illuminance subject. For example, Patent Document 2 discloses an imaging device that controls the amount of light emitted by a flash device based on a photometric value of the amount of reflected light and distance information to a subject.

JP-A-2010-122158 Japanese Patent No. 3880148

Yasuyuki Matsushita, "Illuminance Difference Stereo", Information Processing Society of Japan Research Report, Vol. 2011-CVIM-177, No. 29, pp. 1-12, 2011

In order to acquire the surface normal of a subject by using the illuminance difference stereo method in an imaging device such as a digital camera, a plurality of images having different luminance information for each irradiation light source position are required. In the illuminance difference stereo method, the surface normal is calculated based on the difference in brightness between a plurality of images. If the amount of light emitted by the light source is not appropriate, phenomena such as overexposure and underexposure may occur in the subject portion of the surface normal calculation target included in the image, and the difference in brightness between a plurality of images cannot be calculated accurately. In the part where the difference in brightness between a plurality of images cannot be calculated accurately, the calculated surface normal deviates greatly from the true surface normal. Non-Patent Document 1, Patent Document 1, and Patent Document 2 do not disclose the control of the amount of light emitted from each light source at the time of image acquisition used in the illuminance difference stereo method. Further, although a plurality of light sources are individually made to emit light, it is complicated to calculate the appropriate amount of light emitted from each of the plurality of light sources.

In view of these problems, the present invention has a processing device, a processing system, and an imaging device capable of easily controlling the amount of light emitted from a light source so as to be suitable for the illuminance difference stereo method and calculating surface normals with high accuracy. , Processing methods, programs, and recording media.

The processing device as one aspect of the present invention is a processing device that sequentially irradiates a subject with light from three or more light sources having different positions to acquire three or more images, and the three or more light sources. The first photometric value of the reflected light from the subject, the information on the amount of light emitted by the light source in the preliminary light emission, and the information on the amount of light emitted by the light source in the preliminary light emission, and the three or more light sources are acquired without making the preliminary light emission. a second photometric value of the reflected light from the subject, based on imaging conditions, have a control unit for controlling the light emission amount of each light source in acquiring the three or more images, and the position of each light source The irradiation angle is fixed, and the image is acquired at a subject distance equal to or greater than a predetermined subject distance .
The processing device as another aspect of the present invention is a processing device that sequentially irradiates a subject with light from three or more light sources having different positions to acquire three or more images, and is a processing device for acquiring three or more images. A distance calculation unit that calculates the distance, a first light measurement value of the reflected light from the subject acquired by simultaneously pre-emission of the three or more light sources, information on the amount of light emitted by the light source in the pre-emission, and information on the amount of light emitted by the light source in the preliminary light emission. When acquiring the three or more images based on the second photometric value of the reflected light from the subject, the imaging conditions, and the subject distance, which are acquired without preliminarily emitting the three or more light sources. It has a control unit that controls the amount of light emitted from each light source, and is characterized in that the position and irradiation angle of each light source are fixed.
The processing device as another aspect of the present invention is a processing device that sequentially irradiates a subject with light from three or more light sources having different positions to acquire three or more images, and is a processing device that obtains three or more images. An acquisition unit that acquires light source information including at least one of the irradiation angles, a first photometric value of the reflected light from the subject acquired by simultaneously pre-emitting the three or more light sources, and the light source in the pre-emission. Based on the information on the amount of light emitted by the above, the second metering value of the reflected light from the subject acquired without pre-illuminating the three or more light sources, the imaging conditions, and the light source information, the three or more. It has a control unit that controls the amount of light emitted from each light source when acquiring the image of the above, and is characterized in that the acquisition of the image is performed at a subject distance equal to or greater than a predetermined subject distance.

Further, the processing system as another aspect of the present invention is a processing system that sequentially irradiates a subject with light from three or more light sources having different positions to acquire three or more images. A photometric unit that acquires the first photometric value of the reflected light from the subject when the above light sources are simultaneously pre-emitted , the first photometric value, information on the amount of light emitted by the light source in the pre-emission, and the above 3. Control the amount of light emitted by each light source when capturing three or more images based on the second photometric value of the reflected light from the subject acquired without pre-emission of two or more light sources and the imaging conditions. a control unit which possess, located an irradiation angle of each light source is fixed, the acquisition of the image is characterized to be performed by the object distance more than a predetermined subject distance.

Further, the image pickup device as another aspect of the present invention includes an image pickup unit that sequentially irradiates a subject with light from three or more light sources having different positions to acquire three or more images, and the three or more image pickup devices. A metering unit that acquires the first metering value of the reflected light from the subject when the light sources are simultaneously pre-emitted , the first metering value, information on the amount of light emitted by the light source in the pre-emission, and the above three. Based on the second photometric value of the reflected light from the subject acquired without preliminarily emitting the above light sources and the imaging conditions , the amount of light emitted by each light source when capturing the three or more images is controlled. a control unit, based on the luminance information of the three or more images, possess a normal calculation unit for calculating a surface normal information, the position and the irradiation angle of each light source is fixed, the image Is obtained at a subject distance greater than or equal to a predetermined subject distance .

Further, the processing method as another aspect of the present invention is to sequentially irradiate a subject with light from three or more light sources having different positions and fixed positions and irradiation angles to obtain a predetermined subject distance or more. This is a processing method for acquiring three or more images taken at the subject distance of , the first photometric value of the reflected light from the subject when the three or more light sources are preliminarily emitted at the same time, and the above 3. A step of acquiring a second photometric value of the reflected light from the subject and an imaging condition acquired without making one or more light sources preliminarily emit light , the first photometric value, the second photometric value, and the imaging. and conditions, based on the amount of light emission of the information by the light source in the preliminary light emission, and having the steps of: controlling the light emission amount of each light source in acquiring the three or more images.

According to the present invention, a processing device, a processing system, an imaging device, a processing method, a program, which can easily control the amount of light emitted from a light source so as to be suitable for the illuminance difference stereo method and can calculate a surface normal with high accuracy. And a recording medium can be provided.

It is an external view of the image pickup apparatus which concerns on embodiment of this invention (Examples 1 and 2). It is a block diagram of the image pickup apparatus of Example 1. FIG. It is a figure which shows the processing system (Examples 1 and 2). It is a flowchart which shows the calculation process of the surface normal information of Example 1. FIG. It is a relationship diagram of the light receiving part of an image sensor and the pupil of an image pickup optical system. It is a schematic diagram of an imaging unit. It is a figure which shows the imaging state. It is explanatory drawing of the appropriate light emission amount calculation. It is a relationship diagram of a surface normal and a reflected luminance value. It is a relationship diagram of a surface normal and a reflected luminance value. It is a block diagram of the image pickup apparatus of Example 2. It is a flowchart which shows the calculation process of the surface normal information of Example 2. It is explanatory drawing of the Torrance-Sparrow model.

Hereinafter, examples of the present invention will be described in detail with reference to the drawings. In each figure, the same member is given the same reference number, and duplicate description is omitted.

The illuminance difference stereo method assumes the reflection characteristics of the subject based on the surface normal of the subject and the direction from the subject to the light source (light source direction), and the surface method is based on the reflection characteristics assuming the brightness information of the subject at a plurality of light source positions. This is a method for calculating line information. If the reflectance is not uniquely determined given a given surface normal and the position of the light source, the reflectance can be approximated by a Lambertian reflection model that follows Lambert's cosine law. As shown in FIG. 12, the specular reflection component depends on the bisector of the light source vector s and the line-of-sight direction vector v and the angle α formed by the surface normal line n. Therefore, the reflection characteristic may be a characteristic based on the line-of-sight direction. Further, the luminance information may be obtained by photographing each subject when the light source is on and when the light source is off and taking the difference between them to remove the influence of a light source other than the light source such as ambient light.

Hereinafter, the case where the reflection characteristics are assumed in the Lambertian reflection model will be described. The brightness value of the reflected light is i, the Lambert diffusion reflectance of the object is ρd, the intensity of the incident light is E, the unit vector indicating the direction from the object to the light source (light source direction vector) is s, and the unit surface normal vector of the object. When n is, the luminance i is expressed by the following equation (1) from Lambert's cosine law.

Let each component of different M (M ≧ 3) light source vectors be s ₁ , s ₂ , ..., s _M , and the brightness values for each component of the light source vector be i ₁ , i ₂ , ... i _M. , Equation (1) is represented by the following equation (2).

The left side of equation (2) is the luminance vector of M rows and 1 column, the right side [s ₁ ^T , ... s _M ^T ] is the incident light matrix S indicating the light source direction of M rows and 3 columns, and n is 3 rows and 1 column. Unit plane normal vector of. When M = 3, Eρ _d n is represented by the following equation (3) ^{using the inverse matrix S -1} of the incident light matrix S.

The norm of the vector on the left side of the equation (3) _{is the product of the intensity E of the incident light and the Lambert diffuse reflectance ρ d} , and the normalized vector is calculated as the surface normal vector of the object. That is, since the intensity E of the incident light and the Lambert diffuse reflectance ρ _d appear in the conditional equation only in the form of a product, if Eρ _d is regarded as one variable, the equation (3) is 2 of the unit surface normal vector n. It can be regarded as a simultaneous equation that determines three unknown variables together with the degree of freedom. Therefore, each variable can be determined by acquiring the luminance information using at least three light sources. If the incident light matrix S is not an invertible matrix, there is no inverse matrix. Therefore, it is necessary to select _{each component s 1 to s 3 of the incident light matrix S so that the incident light matrix S becomes an invertible matrix.} That is, it is desirable to select the component s3 linearly independently of the components s1 and s2.

Further, when M> 3, more conditional expressions can be obtained than the unknown variables to be obtained. Therefore, the unit surface normal vector n may be calculated from three arbitrarily selected conditional expressions by the same method as in the case of M = 3. .. When four or more conditional equations are used, the incident light matrix S is no longer an invertible matrix. Therefore, for example, a Moore-Penrose pseudo-inverse matrix may be used to calculate an approximate solution. Further, the unit surface normal vector n may be calculated by a fitting method or an optimization method.

If the reflection characteristics of the subject are assumed by a model different from the Lambertian reflection model, the conditional equation may differ from the linear equation for each component of the unit surface normal vector n. In that case, if a conditional expression equal to or greater than the unknown variable is obtained, a fitting method or an optimization method can be used.

Further, when M> 3, a plurality of conditional expressions of 3 or more and M-1 or less can be obtained, so that a plurality of solution candidates of the unit surface normal vector n can be obtained. In this case, a solution may be selected from a plurality of solution candidates using yet another condition. For example, the continuity of the unit surface normal vector n can be used as a condition. When calculating the unit normal line n for each pixel of the imaging device, if the surface normal line at the pixel (x, y) is n (x, y) and n (x-1, y) is known. The solution that minimizes the evaluation function represented by the following equation (4) may be selected.

Further, if n (x + 1, y) and n (x, y ± 1) are also known, the solution in which the following equation (5) is minimized may be selected.

If there is no known surface normal and there is indefiniteness of the surface normal at all pixel positions, the solution is such that the sum of all pixels of equation (5) represented by the following equation (6) is minimized. May be selected.

It should be noted that the surface normals of pixels other than the nearest neighbors may be used, or an evaluation function weighted according to the distance from the pixel position of interest may be used.

Further, as another condition, the luminance information at an arbitrary light source position may be used. In the diffuse reflection model represented by the Lambertian reflection model, the closer the unit surface normal vector and the light source direction vector are, the greater the brightness of the reflected light. Therefore, the unit surface normal vector can be determined by selecting a solution close to the light source direction vector having the largest brightness value among the brightness values in a plurality of light source directions.

Further, in the specular reflection model, if the light source vector is s and the unit vector in the direction from the object to the camera (camera line-of-sight vector) is v, the following equation (7) holds.

As shown in the equation (7), if the light source direction vector s and the camera line-of-sight vector v are known, the unit surface normal vector n can be calculated. When the surface is rough, the specular reflection also has a widening of the emission angle, but it spreads near the solution obtained as a smooth surface, so it is sufficient to select the candidate closest to the solution for the smooth surface from among the multiple solution candidates. .. Moreover, the true solution may be determined by averaging a plurality of solution candidates.

In the illuminance difference stereo method, it is assumed that the intensity E of the incident light is constant under the conditions of each light source direction, and the brightness value of the reflected light from the subject is accurately detected. Further, in an imaging device provided with a flash device, if the amount of light emitted is too large, the amount of exposure of the subject portion becomes too large. Therefore, a phenomenon called overexposure, which is called overexposure, occurs in the subject portion, and an accurate luminance value cannot be obtained. On the contrary, when the amount of light emitted is too small, the amount of exposure of the subject portion becomes too small. Therefore, a phenomenon called black crushing in which the gradation property is lost occurs in the subject portion, and an accurate luminance value cannot be obtained. That is, if the amount of light emitted from a plurality of light sources having different positions (intensity of incident light in each light source direction) is not appropriate, overexposure or underexposure occurs in the subject portion, and an accurate luminance value cannot be obtained. Therefore, the calculated surface normal deviates significantly from the true surface normal. Therefore, it is necessary to appropriately control the amount of light emitted from each light source, but since a plurality of light sources are used, the calculation of the appropriate amount of light emitted becomes complicated.

FIG. 1 is an external view of the image pickup apparatus 1000A of this embodiment, and FIG. 2 is a block diagram of the image pickup apparatus 1000A of this embodiment. The image pickup apparatus 1000A includes an image pickup unit 100 for photographing a subject and a light source unit 200. The image pickup unit 100 includes an image pickup optical system 101 and an image pickup element 102. In this embodiment, the light source unit 200 is composed of eight light sources arranged concentrically at equal intervals about the optical axis of the imaging optical system 101. Since at least three light sources are required when performing the illuminance difference stereo method, the light source unit 200 may include three or more light sources. Further, in the present embodiment, the light source unit 200 arranges a plurality of light sources concentrically at equal intervals about the optical axis of the imaging optical system 101, but the present invention is not limited to this. Further, in this embodiment, an LED (Light Emitting Diode) is used as each light source of the light source unit 200, but another light source such as a xenon lamp may be used. Further, in this embodiment, the light source unit 200 is built in the image pickup apparatus 1000A, but may be detachably attached. The release button 300 is a button for activating shooting and autofocus.

The image pickup optical system 101 includes a diaphragm 101a, and forms an image of light emitted from the subject on the image pickup element 102. Further, the imaging optical system 101 may be a variable magnification optical system in which the imaging magnification can be changed by moving each lens group. In this embodiment, the imaging optical system 101 is built in the imaging device 1000A, but may be detachably attached to the imaging device 1000A like a single-lens reflex camera. The image sensor 102 is composed of a photoelectric conversion element such as a CCD sensor or a CMOS sensor, and images a subject. The analog electric signal generated by the photoelectric conversion of the image pickup device 102 is converted into a digital signal by the A / D converter 103 and input to the image processing unit 104.

The image processing unit 104 acquires the surface normal information of the subject in addition to the image processing generally performed on the digital signal. The surface normal information is information for determining at least one candidate for one degree of freedom of the surface normal, information for selecting a true solution from a plurality of solution candidates for the surface normal, and the obtained surface normal. Information about the validity of. The image processing unit 104 includes a photometric unit 104a that measures the reflected light from the subject, and a light emission amount control unit 104b that controls an appropriate light emission amount of each light source based on the photometric value of the photometric unit 104a. Further, the image processing unit 104 includes a normal calculation unit 104c for calculating surface normal information and a subject distance calculation unit 104d for calculating distance information to the subject. The output image processed by the image processing unit 104 is stored in an image recording unit 109 such as a semiconductor memory or an optical disk. Further, the output image may be displayed on the display unit 105. In this embodiment, the light measuring unit 104a, the light emitting amount control unit 104b, the normal calculation unit 104c, and the subject distance calculation unit 104d are built in the image pickup device 1000A, but are configured separately from the image pickup device 1000A as described later. May be done.

The information input unit 108 supplies the image pickup conditions (aperture value, exposure time, ISO sensitivity, number of shots, etc.) selected by the user to the system controller 110. The irradiation light source control unit 106 controls the light emitting state of the light source unit 200 in response to an instruction output from the system controller 110. The image pickup control unit 107 acquires an image under desired shooting conditions selected by the user based on the information output from the system controller 110. The ROM 111 stores various programs executed by the system controller 110 and data required for the various programs. The light source information acquisition unit 112 acquires light source information such as the position of each light source of the light source unit 200 and the irradiation angle when the light source emits light. In this embodiment, each light source position and irradiation angle of the light source unit 200 are fixed, but the light source unit 200 may be configured so that each light source position and irradiation angle can be arbitrarily set. In this case, the light source information acquisition unit 112 may acquire light source information such as each light source position and irradiation angle set at the time of shooting. When each light source position and irradiation angle are fixed as in this embodiment, the light source information may be stored in the ROM 111. In this case, since the ROM 111 functions as the light source information acquisition unit, it is not necessary to include the light source information acquisition unit 112.

The calculation process of the surface normal information of this embodiment will be described with reference to the flowchart of FIG. FIG. 3 is a flowchart showing a calculation process of the surface normal information of this embodiment. The calculation process of the surface normal information of this embodiment is executed by the system controller 110 and the image processing unit 104 according to a processing program for making the computer function as a processing device. The processing program may be recorded on a computer-readable recording medium, for example.

In step S101, the system controller 110 sets the first imaging condition (aperture value, exposure time, ISO sensitivity, number of shots, etc.) set by the user from the information input unit 108 as the imaging condition.

In step S102, the system controller 110 is interlocked with the half-pressing operation of the release button 300, and the image sensor 102 receives the reflected light from the subject in a state where the light source is not pre-flashed (pre-flashed). The A / D converter 103 converts the analog electric signal generated by the photoelectric conversion of the image sensor 102 into a digital signal and outputs it to the image processing unit 104. The metering unit 104a acquires the second metering value based on the digital signal input to the image processing unit 104. The second photometric value is a photometric value in a state where the light source does not emit preliminary light, that is, a photometric value of the reflected light of the subject irradiated with only ambient light. Further, in this embodiment, the photometric unit 104a acquires the photometric value based on the image signal obtained by the image sensor 102, but a sensor for photometry provided separately may acquire the photometric value.

In step S103, the system controller 110 causes the subject distance calculation unit 104d to calculate the subject distance. Here, a method of calculating the subject distance will be described. The subject distance calculation unit 104d may estimate the subject distance from the position of the focus lens when autofocusing or manually focusing by the user in step S101. Further, the subject distance calculation unit 104d may acquire a plurality of parallax images taken from different viewpoints and estimate the subject distance by the stereo method. In the stereo method, the depth is estimated by triangulation from the parallax amount of the corresponding points of the subject in the acquired plurality of parallax images, the position information of each viewpoint, and the focal length of the optical system. The subject distance may be an average value of depths calculated at corresponding points in the subject, or may be a depth at a specific point of the subject. When estimating the subject distance from the parallax image, as shown in FIG. 4, the image pickup units of the plurality of parallax images make a plurality of light beams that have passed through different regions of the pupil of the image pickup optical system different from each other. It has an image pickup system that guides a light receiving unit (pixel) to perform photoelectric conversion.

FIG. 4 is a diagram showing the relationship between the light receiving portion of the image sensor and the pupil of the image pickup optical system. A plurality of pairs (pixel pairs) of G1 pixels and G2 pixels, which are light receiving units, are arranged in the image sensor. A plurality of G1 pixels are collectively referred to as a G1 pixel group, and a plurality of G2 pixels are collectively referred to as a G2 pixel group. The pair of G1 pixels and the G2 pixel have a conjugate relationship with the exit pupil EXP of the imaging optical system via a common (that is, one for each pixel pair) microlens ML. Further, a color filter CF is provided between the microlens ML and the light receiving portion.

FIG. 5 is a schematic view of an imaging system assuming that there is a thin-walled lens at the position of the exit pupil EXP instead of the microlens ML of FIG. The G1 pixel receives the luminous flux that has passed through the P1 region of the exit pupil EXP, and the G2 pixel receives the luminous flux that has passed through the P2 region of the exit pupil EXP. An object does not necessarily have to be present in the object point OSP being imaged, and the luminous flux passing through the object point OSP is incident on the G1 pixel or the G2 pixel depending on the region (position) in the pupil through which the object is being imaged. The passage of the luminous flux through different regions in the pupil corresponds to the separation of the incident light from the object point OSP by the angle (parallax). That is, among the G1 pixels and G2 pixels provided for each microlens ML, the image generated by using the output signal from the G1 pixel and the image generated by using the output signal from the G2 pixel are separated from each other. It becomes a plurality of (here, a pair) parallax images having. In the following description, receiving light flux passing through different regions in the pupil by different light receiving portions (pixels) is referred to as pupil division.

In FIGS. 4 and 5, even if the above-mentioned conjugate relationship is not perfect or the P1 region and the P2 region partially overlap due to the position of the exit pupil EXP being displaced, a plurality of obtained cases are obtained. The image can be treated as a parallax image.

As yet another example, a parallax image may be acquired by providing a plurality of imaging optical systems in one imaging device. Further, a parallax image may be acquired by capturing the same subject using a plurality of cameras.

In step S104, the system controller 110 causes the light source information acquisition unit 112 to acquire the light source information. The light source information of this embodiment is at least one information of each light source position of the light source unit 200 and the irradiation angle at the time of light emission of the light source.

It should be noted that step S103 and step S104 may be executed between steps S102 and S106.

In step S105, the system controller 110 causes the irradiation light source control unit 106 to simultaneously make a preliminary emission (pre-emission) of the light source used for the main shooting with a preset amount of light. The amount of pre-emission is preferably set lower than that at the time of the main imaging in order to reduce the deterioration of the detection accuracy of the photometric value due to the occurrence of overexposure. In the illuminance difference stereo method, a plurality of light sources having different positions are individually emitted to acquire a plurality of images, but in this embodiment, preliminary emission for detecting the photometric value is performed by simultaneously emitting the light sources. By executing the preliminary light emission only once, the control load can be reduced as compared with the case where the preliminary light emission is executed for each light source.

In step S106, the system controller 110 receives the reflected light from the subject by the image sensor 102 in synchronization with the preliminary light emission in step S105. The A / D converter 103 converts the analog electric signal generated by the photoelectric conversion of the image sensor 102 into a digital signal and outputs it to the image processing unit 104. The photometric unit 104a acquires the first photometric value based on the digital signal input to the image processing unit 104. The first photometric value is a photometric value in a state where the light source emits preliminary light, that is, a photometric value of the reflected light of the subject irradiated with the ambient light and the preliminary light emitted from the light source. Further, the light emission amount control unit 104b calculates an appropriate light emission amount for each light source based on the first imaging condition, the first and second photometric values set in step S101, the light source information, and the subject distance.

Here, with reference to FIG. 6, the operation at the time of calculating the appropriate light emission amount of this embodiment will be described. FIG. 6 is a diagram showing an imaging state when the imaging device 1000A acquires surface normal information of the subject 2000. First, the photometric unit 104a measures the reflected light of the subject irradiated with only ambient light (sunlight) without causing the light source to preliminarily emit light, and acquires a second photometric value. Next, the image pickup apparatus 1000A preliminarily emits all the light sources used in the main shooting. For the sake of simplicity, only the preliminary emission light (light A and light B) of some light sources is shown in the figure, but in reality, other light sources are also preliminary emission. The preliminary emission light from each light source applied to the subject 2000 is reflected by the subject 2000 and then incident on the image pickup unit 100. The photometric unit 104a acquires the first photometric value based on the luminance value information of the image signal obtained by capturing the subject 2000. The light emission amount control unit 104b calculates an appropriate light emission amount of all the light sources used in the main shooting based on the first imaging condition, the first and second photometric values, the light source information, and the subject distance.

In the illuminance difference stereo method, the same subject is irradiated with light from different light sources, and the surface normal information is calculated based on the change in brightness. Therefore, the brightness information based only on the light emitted from the light source is important, and the environment. Luminance information based on light (sunlight) is not required when calculating the surface normal. Therefore, when the intensity of the ambient light is strong, it is important to adjust the amount of light emission so as to maintain the gradation of the luminance information based only on the light emitted from the light source. As described above, the first photometric value is the photometric value of the reflected light of the subject irradiated with the ambient light and the preliminary emission light of the light source, and the second photometric value is the photometric value of the reflected light of the subject irradiated with only the ambient light. The value. In this embodiment, the light emission amount control unit 104b calculates the minimum light emission amount based on the difference between the first and second photometric values so as to maintain the gradation of the luminance information only by the light emitted from the light source. In this way, by calculating the minimum emission amount based on the difference between the first and second photometric values, the gradation of the low-luminance region of the luminance information based only on the emission light from the light source is lowered (black crushing). ) Can be suppressed. Further, when the image signal based on the reflected light of the subject due to the ambient light and the light emitted from the light source is overexposed, the gradation of the high-luminance region of the luminance information based only on the emitted light from the light source is deteriorated. In this embodiment, the gradation of the luminance information based only on the emitted light of the light source is maintained based on the first photometric value which is the photometric value of the reflected light of the subject irradiated with the ambient light and the preliminary emission light of the light source. Calculate the maximum amount of light emitted. The light emission amount control unit 104b sets the light emission amount that can secure the gradation of the luminance information most between the minimum light emission amount and the maximum light emission amount as the appropriate light emission amount. When the maximum light emission amount is larger than the minimum light emission amount, the light emission amount control unit 104b preferably sets the maximum light emission amount as an appropriate light emission amount. By setting the maximum light emission amount as an appropriate light emission amount, it is possible to maintain high gradation of luminance information while avoiding overexposure. Further, in this embodiment, the case where the intensity of the ambient light is strong has been described, but when the intensity of the ambient light is weak such as in a dark room photography, step S102 may be omitted.

With reference to FIG. 7, the concept of calculating the appropriate amount of light emission when all the light sources are preliminarily emitted at the same time will be described. FIG. 7 is an explanatory diagram for calculating the appropriate light emission amount. The two light sources 200A and 200B are arranged so that the center is located at a distance h from the center of the imaging unit 100, respectively. Further, the ball subject 2001 is arranged at a position separated from the imaging unit 100 by a distance D. For the sake of simplicity, only two light sources 200A and 200B are shown in the figure.

FIG. 8 is a diagram showing the relationship between the surface normal θ of the spherical subject 2001 and the standardized reflection luminance value i when the light source is made to emit light in the configuration shown in FIG. Graph 11 shows the reflected luminance value when the light source 200A is made to emit light with the light emitting amount E, and graph 12 shows the reflected luminance value when the light source 200B is made to emit light with the light emitting amount E. Here, the spherical subject 2001 is sufficiently smaller than the image pickup apparatus 1000A, and the reflection characteristic follows the Lambertian reflection model. Further, it is assumed that each light source is a point light source that radiates with uniform intensity in all directions. At this time, the light source direction vector for each surface of the sphere subject 2001 is calculated by the distance h and the distance D, and the graphs 11 and 12 have the characteristics represented by the equation (1). Graph 11 is a cosine function starting from -10 degrees, which has the maximum reflected luminance value with respect to a surface normal of -10 degrees. Further, the graph 12 is a cosine function starting from 10 degrees, which has the maximum reflected luminance value with respect to a surface normal of 10 degrees. That is, the light sources 200A and 200B are arranged at positions where the angles with respect to the sphere subject 2001 are -10 degrees and 10 degrees, respectively, with respect to the line connecting the sphere subject 2001 and the imaging unit 100.

Graph 14 shows the combined reflection luminance value when the light sources 200A and 200B are simultaneously emitted with the light emission amount E, that is, the combined light emission amount 2E. The reflected luminance value of the sphere subject 2001 with respect to the surface normal is the sum of the graphs 11 and 12. Here, consider the case of photographing another subject having only a predetermined surface normal. For example, the combined reflection luminance value when the light sources 200A and 200B are simultaneously emitted with a light emission amount E on a subject having a surface normal of 30 degrees is set as a limit value at which overexposure does not occur. At this time, since the combined light emission amount of the two light sources is 2E, if the light emission amount of each light source to be individually emitted during the main shooting is calculated as 2E, overexposure occurs when the light source 200B is emitted. Resulting in.

Graph 15 shows the maximum value of the reflected luminance value when each light source is individually emitted with a light emission amount of 2E. In the graph 15, the reflected luminance values for all the surface normals are higher than those in the graph 14 in which the two light sources are simultaneously emitted with the combined emission amount 2E. That is, in order to calculate the maximum amount of light emission that avoids overexposure from the first photometric value when all the light sources are preliminarily emitted at the same time, it is necessary to consider the reflection characteristics of the subject.

Graph 13 shows the maximum value of the reflected luminance value when the maximum emission amount of each light source is uniformly set to 1.5E by considering the Lambertian reflection model of the subject. In the graph 13, the reflected luminance value is lower in the entire range of the surface normal of -50 degrees to 50 degrees as compared with the graph 14 in which the two light sources are simultaneously emitted with the combined light emission amount of 2E. Therefore, it is possible to avoid the occurrence of overexposure in a wide surface normal range.

FIG. 9 is a diagram showing the relationship between the surface normal θ and the normalized reflection luminance i when the distance from the imaging unit 100 to the spherical subject 2001 is half (D / 2) of FIG. By halving the distance from the image pickup unit 100 to the ball subject 2001, the light sources 200A and 200B have angles of -20 degrees and 20 degrees with respect to the ball subject 2001 with respect to the line connecting the ball subject 2001 and the image pickup unit 100, respectively. It is placed in position. Graph 21 shows the reflected luminance value when the light source 200A is made to emit light with the light emitting amount E, and graph 22 shows the reflected luminance value when the light source 200B is made to emit light with the light emitting amount E. Graph 24 shows the combined reflection luminance value when the light sources 200A and 200B are simultaneously emitted with the light emission amount E, that is, the combined light emission amount 2E. Graph 25 shows the maximum value of the reflected luminance value when each light source is individually emitted with a light emission amount of 2E. Graph 23 shows the maximum value of the reflected luminance value when the maximum emission amount of each light source is uniformly set to 1.3E by considering the Lambertian reflection model of the subject.

The change in the set value of the maximum light emission amount is due to the fact that the light source direction vector with respect to the subject changes depending on the distance to the subject even when the light source position and the irradiation angle are fixed. That is, assuming that the reflection characteristic of the subject is a Lambertian reflection model, the light emission amount control unit 104b can calculate a uniform maximum light emission amount for each light source based on the light source information, the subject distance, and the first photometric value. It becomes. By setting all the light sources to the same light emission amount, it is possible to omit the correction processing of the luminance information due to the difference in the light emission amount, and it is possible to reduce the processing load at the time of calculating the surface normal. Further, the setting table of the maximum light emission amount based on the light source information and the subject distance may be saved in the ROM 111. In this case, the light emission amount control unit 104b can calculate the appropriate light emission amount of the light source by referring to the maximum light emission amount setting table stored in the ROM 111.

As described above, when all the light sources are preliminarily emitted at the same time in order to reduce the control load of the appropriate emission amount, it is possible to avoid overexposure for a wide range of surface normal subjects.

In step S107, the system controller 110 determines whether or not the light emission amount control unit 104b can control all the light sources used for the illumination difference stereo type imaging via the irradiation light source control unit 106 with the appropriate light source amount calculated in step S106. judge. If there is an uncontrollable light source, the process proceeds to step S108, and if it does not exist, the process proceeds to step S109.

In step S108, the system controller 110 causes the image pickup control unit 107 to change the image pickup condition of the image pickup unit 100 from the first image pickup condition to the second image pickup condition. Specifically, when the appropriate light emission amount calculated in step S106 is smaller than the controllable light emission amount, for example, the aperture value of the aperture 101a is set large (dark), the exposure time is set short, or ISO. The sensitivity may be set low. The imaging conditions may be changed by combining the above settings. When the appropriate light emission amount calculated in step S106 is larger than the controllable light emission amount, for example, the aperture value is set small (bright), the exposure time is set long, or the ISO sensitivity is set high. do it. The imaging conditions may be changed by combining the above settings. Further, if the minimum light emission amount calculated in step S106 is larger than the maximum light emission amount, the dynamic range of the image sensor 102 is insufficient. Therefore, HDR (high dynamic range) shooting is set at the time of main shooting. .. As HDR photography, a generally known method of acquiring and synthesizing a plurality of images having different exposure conditions may be used. In HDR shooting in the illuminance difference stereo system, it is sufficient to acquire a plurality of images having different exposure conditions for one light source emission. That is, the number of images to be captured is set, such as how many images are acquired for one light source. Further, the light emission amount control unit 104b may correct the light emission amount based on the changed imaging conditions. Further, the imaging conditions may be changed for each imaging corresponding to one light source emission, or may be the same conditions for all imaging.

In step S109, the system controller 110 causes the light emission amount control unit 104b to set the light emission amount of each light source via the irradiation light source control unit 106. In this embodiment, the light emission amount control unit 104b is set so that the light emission amount of each light source is the same.

In step S110, the system controller 110 is linked to the full pressing operation of the release button 300 to take an image of the subject at a plurality of light source positions. Specifically, the system controller 110 sequentially irradiates the subject with light from at least three or more light sources having different positions from each other in the light source unit 200 via the irradiation light source control unit 106, and images the subject through the image pickup control unit 107. The unit 100 is made to image the subject. The A / D converter 103 forms a captured image (luminance information) by A / D converting the analog signal output from the image sensor 102, and outputs the captured image (luminance information) to the image processing unit 104. The image processing unit 104 may perform normal development processing or various image correction processing for image generation.

In step S111, the system controller 110 causes the normal calculation unit 104c to calculate the surface normal information based on the luminance information corrected in step S110. The normal calculation unit 104c calculates the surface normal information using the above-mentioned illuminance difference stereo method. The image recording unit 109 saves the surface normal information and the image information, and the flow is completed. The image recording unit 109 may store information obtained by adding information on the amount of light emitted and information on imaging conditions to the image information. By adding the light emission amount information and the imaging condition information to the image information and saving the information, it is possible to execute the luminance information correction process and the surface normal calculation process later.

As described above, in the present embodiment, the luminance information can be acquired with high accuracy by preliminarily emitting light from each light source at the same time and setting the light emission amount of each light source to an appropriate light emission amount based on the photometric value. It is possible to maintain the calculation accuracy of the surface normal.

In this embodiment, the surface normal information of the subject is calculated in the imaging device 1000A, but as shown in FIG. 2B, the surface normal information of the subject is calculated by using a processing system 500 different from that of the imaging device 1000A. May be calculated. The processing system 500 shown in FIG. 2B includes a processing device 501, a normal calculation unit 502, an imaging unit 503, a light source unit 504, a photometric unit 505, and a subject distance calculation unit 506. The processing device 501 includes a light emitting amount control unit 501a. When calculating the surface normal information using the processing system 500, first, the photometric unit 505 measures the reflected light from the subject in a state where the light source does not pre-flash according to the instruction of the processing device 501. Next, the light source unit 504 preliminarily emits at least three or more light sources according to the instruction of the processing device 501, and the photometric unit 505 measures the reflected light from the subject. Next, the subject distance calculation unit 506 calculates the subject distance, and the light emission amount control unit 501a calculates the appropriate light emission amount for each light source based on the imaging conditions, the photometric value, the light source information, and the subject distance. Subsequently, the light emission amount control unit 501a controls the light source unit 504 so as to irradiate the subject with light of an appropriate light source amount from at least three or more light source positions, and the image pickup unit 503 acquires an image of each light source position. It is sufficient that the light emitting amount control unit 501a can control the light emitting amount of each light source, and other processing such as lighting of the light source unit 200 may be executed by another part of the processing device 501. Finally, the normal calculation unit 502 calculates the surface normal information using the image captured by the image pickup unit 503. The processing system 500 may include at least a processing device 501. Further, the processing device 501 may include a normal calculation unit 502. Further, the image pickup unit 503 and the light source unit 504 may be individual devices, or the light source unit 504 may be built in the image pickup unit 503.

FIG. 10 is a block diagram of the image pickup apparatus 1000B of this embodiment. Since the appearance configuration of the image pickup apparatus 1000B is the same as the appearance configuration of the image pickup apparatus 1000A of the first embodiment, detailed description thereof will be omitted. The internal configuration of the image pickup apparatus 1000B is the same as the internal configuration of the image pickup apparatus 1000A, except that the subject distance calculation unit 104d and the light source information acquisition unit 112 are not provided. Therefore, a detailed description of the internal configuration will be omitted, and only a part different from the image pickup apparatus 1000A will be described.

The surface normal information calculation process of this embodiment will be described with reference to the flowchart of FIG. FIG. 11 is a flowchart showing a calculation process of the surface normal information of this embodiment. The calculation process of the surface normal information of this embodiment is executed by the system controller 110 and the image processing unit 104 according to a processing program for making the computer function as a processing device. The processing program may be recorded on a computer-readable recording medium, for example.

Since the steps up to step S201 and step S202 are the same as the steps up to step S101 and step S102 of the first embodiment, the description thereof will be omitted.

In step S203, the system controller 110 causes the irradiation light source control unit 106 to simultaneously make a preliminary emission (pre-emission) of the light source used for the main shooting with a preset amount of light. The amount of pre-emission is preferably set lower than that at the time of the main imaging in order to reduce the deterioration of the detection accuracy of the photometric value due to the occurrence of overexposure. In the illuminance difference stereo method, a plurality of light sources having different positions are individually emitted to acquire a plurality of images, but in this embodiment, preliminary emission for detecting the photometric value is performed by simultaneously emitting the light sources. By executing the preliminary light emission at one time, the control load can be reduced as compared with the case where the preliminary light emission is executed for each light source.

In step S204, the system controller 110 receives the reflected light from the subject by the image sensor 102 in synchronization with the preliminary light emission in step S203. The A / D converter 103 converts the analog electric signal generated by the photoelectric conversion of the image sensor 102 into a digital signal and outputs it to the image processing unit 104. The photometric unit 104a acquires the first photometric value based on the digital signal input to the image processing unit 104. The first photometric value is a photometric value in a state where the light source emits preliminary light, that is, a photometric value of the reflected light of the subject irradiated with the ambient light and the preliminary light emitted from the light source. Further, the light emission amount control unit 104b calculates an appropriate light emission amount for each light source based on the first imaging condition set in step S101 and the first and second photometric values. In this embodiment, unlike the first embodiment, each light source of the light source unit 200 is arranged at a fixed position and the irradiation angle is also fixed, so that the light source information and the subject distance are not acquired. At this time, the closer the subject distance is, the larger the light source direction vector becomes, and it is necessary to set the maximum amount of light emission lower. For example, as described with reference to FIGS. 8 and 9, when the subject distance is D, the maximum light emission amount is 1.5E, and when the subject distance is D / 2, the maximum light emission amount is 1.3E. In other words, by setting the minimum subject distance condition at the time of shooting, it is possible to calculate the maximum amount of light emission that avoids overexposure within the set distance condition range without using the subject distance. In this embodiment, by setting the minimum subject distance condition at the time of shooting, the appropriate light emission amount of each light source is reduced by using the correction coefficient at the time of calculating the maximum light emission amount. With the above configuration, it is possible to calculate the appropriate amount of light emission by a method in which all light sources are preliminarily emitted at the same time without providing a subject distance calculation unit and a light source information acquisition unit, which simplifies the device and reduces the processing load. be able to.

Since the steps from step S205 to step S209 are the same as the steps from step S107 to step S111 in the first embodiment, the description thereof will be omitted.

As described above, in the present embodiment, the luminance information can be acquired with high accuracy even when the subject distance calculation unit and the light source information acquisition unit are omitted, so that the surface normal calculation accuracy can be maintained.

Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and modifications can be made within the scope of the gist thereof.

104 Image processing unit (processing device)
104b Appropriate light emission amount calculation unit (control unit)

Claims (16)

It is a processing device that sequentially irradiates a subject with light from three or more light sources having different positions to acquire three or more images.
The first photometric value of the reflected light from the subject obtained by simultaneously pre-emitting the three or more light sources, the information on the amount of light emitted by the light source in the pre-emission, and the pre-emission of the three or more light sources. a second photometric value of the reflected light from the object to be acquired without causing, based on the imaging conditions, have a control unit for controlling the light emission amount of each light source in acquiring the three or more images,
A processing device characterized in that the position and irradiation angle of each of the light sources are fixed, and the acquisition of the image is performed at a subject distance equal to or greater than a predetermined subject distance. The processing apparatus according to claim 1, further comprising a normal calculation unit that calculates surface normal information based on the luminance information of the three or more images. It is a processing device that sequentially irradiates a subject with light from three or more light sources having different positions to acquire three or more images.
A distance calculating unit for calculating the object distance to the subject,
The first photometric value of the reflected light from the subject obtained by simultaneously pre-emitting the three or more light sources, the information on the amount of light emitted by the light source in the pre-emission, and the pre-emission of the three or more light sources. Control to control the amount of light emitted from each light source when acquiring the three or more images based on the second photometric value of the reflected light from the subject, the imaging condition, and the subject distance. Has a part
The processing device location and irradiation angle of each light source shall be the feature that it is fixed. The processing device according to claim 3 , wherein the control unit controls the light emission amount of each light source so that the light emission amount of each light source becomes smaller as the subject distance is shorter. It is a processing device that sequentially irradiates a subject with light from three or more light sources having different positions to acquire three or more images.
An acquisition unit which acquires a light source information including at least one of the position and the irradiation angle of the light source,
The first photometric value of the reflected light from the subject obtained by simultaneously pre-emitting the three or more light sources, the information on the amount of light emitted by the light source in the pre-emission, and the pre-emission of the three or more light sources. Control to control the amount of light emitted from each light source when acquiring the three or more images based on the second photometric value of the reflected light from the subject, the imaging condition, and the light source information. Has a part
Processing apparatus you characterized in that acquisition of the images is performed by the object distance more than a predetermined subject distance. The processing apparatus according to claim 5 , wherein the light source information is a position and an irradiation angle of each light source. Based on the light emission amount of each light source, wherein the control unit is set, the processing apparatus according to any one of claims 1 to 6, further comprising an imaging control unit for changing the imaging condition. The processing apparatus according to any one of claims 1 to 7, wherein the imaging condition is at least one of the exposure time of the imaging unit, the ISO sensitivity, the aperture value, and the number of shots. The processing apparatus according to any one of claims 1 to 8 , wherein the control unit controls the amount of light emitted from each of the three or more light sources so that the amount of light emitted from the three or more light sources becomes the same. It is a processing system that sequentially irradiates a subject with light from three or more light sources having different positions to acquire three or more images.
A metering unit that acquires the first photometric value of the reflected light from the subject when the three or more light sources are preliminarily emitted at the same time.
The first photometric value, information on the amount of light emitted by the light source in the preliminary light emission, the second photometric value of the reflected light from the subject acquired without causing the three or more light sources to perform preliminary light emission, and imaging conditions. based on, have a control unit for controlling the light emission amount of each light source at the time of imaging the three or more images,
A processing system characterized in that the position and irradiation angle of each light source are fixed, and the image is acquired at a subject distance equal to or greater than a predetermined subject distance. The processing system according to claim 10 , further comprising a normal calculation unit that calculates surface normal information based on the luminance information of the three or more images. The processing system according to claim 10 or 11 , further comprising a light source unit including three or more light sources having different positions from each other. An imaging unit that sequentially irradiates a subject with light from three or more light sources having different positions to acquire three or more images.
A photometric unit that acquires the first photometric value of the reflected light from the subject when the three or more light sources are preliminarily emitted at the same time.
The first photometric value, information on the amount of light emitted by the light source in the preliminary light emission, the second photometric value of the reflected light from the subject acquired without causing the three or more light sources to perform preliminary light emission, and imaging conditions. Based on, a control unit that controls the amount of light emitted from each light source when capturing the three or more images, and
Based on the luminance information of the three or more images, it possesses a normal calculation unit for calculating a surface normal information, and
An imaging device characterized in that the position and irradiation angle of each light source are fixed, and the image is acquired at a subject distance equal to or greater than a predetermined subject distance. Acquires three or more images taken at a subject distance equal to or greater than a predetermined subject distance by sequentially irradiating the subject with light from three or more light sources having different positions and fixed positions and irradiation angles. It is a processing method to make
The first photometric value of the reflected light from the subject when the three or more light sources are preliminarily emitted at the same time, and the first photometric value of the reflected light from the subject acquired without pre-embossing the three or more light sources. 2 Photometric values, steps to acquire imaging conditions, and
Based on the information of the first photometric value, the second photometric value, the imaging condition, and the light emission amount of the light source in the preliminary light emission, the light emission amount of each light source when acquiring the three or more images. A processing method comprising: a step of controlling. A program for operating a computer as the processing device according to any one of claims 1 to 9. A computer-readable recording medium on which the program according to claim 15 is recorded.

Images