JP2017103578A - Control device, imaging device, imaging system, normal information acquisition system, image processing device, control method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve antiglare effect to a person and to maintain surface normal accuracy to objects other than the person.SOLUTION: A system controller 111 for sequentially irradiating light to a subject from a plurality of three or more mutually different positions to image the subject at each irradiation determines whether the face of the subject is detected before irradiating the light from the plurality of positions, selects invisible light without selecting visible light as light to be irradiated when the face of the subject is detected, and selects the visible light as the light to be irradiated when the face of the subject is not detected.SELECTED DRAWING: Figure 3

Description

  The present invention relates to a control device, an imaging device, an imaging system, a normal vector information acquisition system, an image processing device, a control method, and a program.

  Non-Patent Document 1 provides a method of selecting a light source in a wavelength range in which a light source in a plurality of wavelength ranges is provided and a reflection characteristic is closest to a Lambertian reflection model in an illuminance difference stereo method for acquiring normal information of a local surface of a subject Has proposed. In Patent Document 1, 3 or more light sources used for the illuminance difference stereo method have different wavelengths to obtain surface normals of the head and neck surface of the human body and reduce glare of the subject. A method using infrared light for the above light source has been proposed.

JP 2013-31650 A

Takashi Takashi, “Improving the accuracy of photometric stereo using multiwavelength images”, Information Processing Society of Japan, Vol. 2012-CVIM-183, no. 12, pp. 1-8, 2012

  In the illuminance difference stereo method, when a subject is a person, the person feels dazzling when using visible light (has discomfort), but Non-Patent Document 1 does not consider such a problem. Thus, it is conceivable to apply the infrared light source to all subjects by applying Patent Document 1, but there is a risk that the accuracy of normal information acquired for subjects other than a person may be reduced.

  The present invention provides a control device, an imaging device, an imaging system, a normal information acquisition system, an image processing device, and a control method capable of improving the anti-glare effect on a person and maintaining the surface normal accuracy with respect to a subject other than the person. And to provide a program.

  The control device according to the present invention is a control device that sequentially irradiates a subject with light from a plurality of different positions of three or more and images the subject at each irradiation, before the light is irradiated from the plurality of positions. In addition, it is determined whether or not the face of the subject has been detected. If the face is detected, invisible light is selected as the irradiation light without selecting visible light, and the face is not detected. In this case, the visible light is selected as the light to be irradiated.

  According to the present invention, a control device, an imaging device, an imaging system, and a normal information acquisition system capable of improving the antiglare effect on a person and maintaining the accuracy of normal information acquired for a subject other than the person. An image processing apparatus, a control method, and a program can be provided.

1 is an external perspective view of an imaging apparatus according to the present invention. (Example 1) It is a block diagram of the imaging device shown in FIG. (Example 1) It is a flowchart which shows the image processing method (normal line estimation method) which the system controller shown in FIG. 2 performs. It is a figure for demonstrating the face detection of S103 shown in FIG. 1 is an external perspective view of an imaging apparatus according to the present invention. (Example 2) It is the schematic which shows the structure of the imaging part shown in FIG. (Example 2) It is the schematic which shows the structure of the light source means shown in FIG. (Example 2) It is a flowchart which shows the image processing method (normal line estimation method) which the system controller shown in FIG. 2 performs. (Example 2) It is a figure which shows the simplified Torrance-Sparrow model.

  When capturing an image of a subject (still image shooting) with an imaging device, the shape information of the subject, in particular, surface normal information, is acquired along with luminance information, so that, for example, the appearance of the subject is changed in image processing after imaging. It is possible to generate an image. The appearance of the subject is determined by factors such as subject shape information, subject reflectance information, and light source information. It is particularly effective to use surface normal information instead of a three-dimensional shape as shape information. This is because the physical behavior of the reflected light when the light emitted from the light source is reflected by the subject depends on the local surface normal.

  The illuminance-difference stereo method, which is a method for acquiring surface normal information, acquires distance information by a method such as triangulation using a laser beam or binocular stereo, and obtains a three-dimensional shape obtained from the distance information as a surface normal. Compared to the method of converting to information, the apparatus becomes smaller and the accuracy of the normal information to be acquired becomes higher. The illuminance difference stereo method assumes reflection characteristics based on the surface normal of the subject and the light source direction, and determines the surface normal from the luminance information of the subject at a plurality of light source positions and the assumed reflection characteristics. A Lambertian reflection model that follows Lambert's cosine law is often used as the reflection characteristic of the subject. Assuming that the reflectance is uniquely determined when a certain surface normal and light source position are given, if the subject's reflectance is unknown, it can be approximated by a Lambert reflection model that follows Lambert's cosine law. Good.

  FIG. 9 is a diagram illustrating a simplified Torrance-Sparrow model. Since the reflectance of the specular reflection component depends on the angle α formed by the bisector of the light source vector s and the line-of-sight vector v and the surface normal n, the reflection characteristic is based on the line-of-sight direction. In the illuminance difference stereo method, images taken when a known light source is turned on and off are taken, and the difference between these images is taken to eliminate the influence of light sources other than known light sources such as ambient light. May be. Hereinafter, a case where a Lambertian reflection model is assumed will be described.

  The luminance of the reflected light is i, the Lambertian diffuse reflectance of the object is ρd, the intensity of the incident light is E, the unit vector (light source direction vector) indicating the direction from the object to the light source is s, and the unit surface normal vector of the object is Let n. At this time, the following equation is established from Lambert's cosine law.

  Here, if the luminance values obtained in different M (M is 3 or more (M ≧ 3)) light source directions s1, s2,..., SM are i1, i2,. From this, the following equation is established.

  Here, the left side is a luminance vector of M rows and 1 column, the right side [s1T,... SMT] is an incident light matrix S indicating the light source direction of M rows and 3 columns, and n is a unit surface normal vector of 3 rows and 1 column. It is. In the case of M = 3, Eρdn is obtained as follows by multiplying the inverse matrix of the incident light matrix S from the left.

  The norm of the vector on the left side is the product of E and ρd, and the normalized vector is obtained as the surface normal vector of the object. Since E and ρd appear in the conditional expression only in the form of a product, when Eρd is viewed as one variable, it can be regarded as a simultaneous equation that determines three unknown variables together with the two degrees of freedom of the unit surface normal vector. Therefore, three equations are obtained by acquiring luminance information under three light source conditions, and a solution can be determined.

  When the incident light matrix S is not regular, there is no inverse matrix, so it is necessary to select the light source directions s1 to s3 so that S is regular. That is, it is desirable to select s3 linearly independent of s1 and s2. On the other hand, when M> 3, more conditional expressions are obtained than the unknown variable to be obtained. In this case, the surface normal vector can be obtained from three arbitrarily selected conditional expressions by the same method as described above. When four or more conditional expressions are used, the incident light matrix S is not a square matrix. For example, an approximate solution may be obtained using a Moore-Penrose pseudo inverse matrix. Further, the solution may be obtained by a known fitting method or optimization method without using matrix calculation.

  When the reflection characteristic of the object is assumed by a model other than the Lambertian reflection model, the conditional expression may not be a linear equation for each component of the unit surface normal vector n. Even in such a case, if an equation equal to or greater than the unknown variable is obtained for the unknown variable, a known fitting method or optimization method can be used. Further, when M> 3, a plurality of combinations of conditional expressions of 3 or more and M−1 or less are obtained, so that a plurality of surface normal candidates can be obtained. In this case, a solution may be selected by a method described later.

  When a plurality of surface normal vector solution candidates are obtained, a solution may be selected from yet another condition. For example, continuity of surface normal vectors can be used as a condition. When calculating the surface normal for each pixel of the imaging device, if n (x−1, y) is known as the surface normal n (x, y) at the pixel (x, y), A solution that minimizes the evaluation function represented by can be selected.

  If n (x + 1, y) and n (x, y ± 1) are also known, a solution that minimizes the evaluation function expressed by the following equation may be selected.

  If there is no known surface normal and the surface normal is indefinite at all pixel positions, the solution may be selected so that the sum of all the pixels expressed by the following equation is minimized.

  The above is an example, and surface normal information at pixels other than the nearest neighbor may be used, or an evaluation function weighted according to the distance from the pixel position of interest may be used.

  In addition, luminance information at an arbitrary light source position may be used. In the diffuse reflection model typified by the Lambert reflection model, the brightness of the reflected light increases as the surface normal vector and the light source direction are closer. Therefore, it is possible to select a surface normal vector that is closer to the direction in which the luminance is larger from the luminance values in a plurality of light source directions. In the specular reflection model, the following equation holds for a smooth surface where the unit vector (camera line-of-sight vector) in the direction from the object to the camera is v.

  Therefore, if the light source direction vector s and the camera line-of-sight vector v are known, the surface normal n can be calculated. In general, when the surface is rough, the specular reflection also has an emission angle spread, but it spreads in the vicinity of the solution obtained as a smooth surface, so select the candidate closest to the solution for the smooth surface from multiple solution candidates Can do.

  Further, the true solution may be determined by the average of a plurality of solution candidates.

  FIG. 1 is an external perspective view of an imaging apparatus 1 according to Embodiment 1 of the present invention. The imaging device 1 includes a digital camera, a television camera, and the like. The imaging device 1 may be a lens-integrated type, or may be a lens interchangeable type such as a single-lens reflex camera or a non-reflex camera (mirrorless camera).

  The imaging device 1 includes an imaging unit 100, a light source unit 200, and a release button 300.

  The imaging unit 100 images a subject (not shown).

  The light source means 200 irradiates each of visible light and invisible light from a plurality of three or more different positions. The light source means 200 is provided with a visible light source means 210 that emits visible light that can be seen by human eyes, and near-infrared light that is invisible light that cannot be seen by human eyes. And an invisible light source means 220 for irradiating. The visible light source means 210 includes eight visible light sources 212 that each emit visible light. The invisible light source means 220 includes eight invisible light sources 222 that each emit invisible light. The visible light source 212 and the invisible light source 222 have the same size, and are provided concentrically around the imaging unit 100.

  The invisible light source unit 220 is provided outside the visible light source unit 210, and the corresponding visible light source 212 and invisible light source 222 are located on a line extending in the radial direction from the imaging optical axis in a plane perpendicular to the imaging optical axis. ing. The further away the light source means 200 is from the imaging optical axis, the higher the surface normal acquisition accuracy (estimation accuracy) of the surface of the subject. Further, the higher the light amount of the light source means 200 or the higher the sensitivity of the image sensor, the higher the accuracy of obtaining normal line information on the surface of the subject. Since the sensitivity of the image sensor to invisible light is lower than that to visible light, the invisible light source means 220 is arranged outside the visible light source means 210 as shown in FIG.

  In the present embodiment and the embodiments described later, the wavelength of visible light will be described as 400 nm to 700 nm. The invisible light source includes near infrared light (wavelength 700 nm to 2500 nm), infrared light (wavelength 2500 nm to 1 mm), and ultraviolet light (wavelength 250 nm to 400 nm). In this embodiment and the embodiments described later, invisible light will be described as an example of near infrared light.

  Further, the number of light sources is not limited to eight, and it is sufficient if light sources are provided at a plurality of three or more different positions necessary for the illuminance difference stereo method. When the imaging device is an interchangeable lens type, a lens device that forms an optical image of a subject with the imaging device body and can be attached to the imaging device body constitutes an imaging system (camera system).

  In this case, the light source means 200 that irradiates each of visible light and invisible light from a plurality of three or more different positions may be provided in the imaging apparatus main body or may be provided in the lens apparatus. Further, the light source means 200 may be provided as a light source device that is separate from the imaging device. In this case, the light source device is communicably connected to the imaging device via a wired or wireless connection, and is controlled by the imaging device.

  The release button 300 transmits a SW1 signal to the system controller 111 described later in response to the half press by the user, and transmits a SW2 signal to the system controller 111 in response to the full press by the user. The release button 300 functions as an instruction means for instructing shooting preparation and shooting. In this embodiment, half-pressing the release button 300 corresponds to a shooting preparation instruction, and full-pressing the release button 300 corresponds to a shooting instruction.

  FIG. 2 is a block diagram of the imaging apparatus 1. The imaging apparatus 1 includes an imaging unit 100, an A / D converter 103, an image processing unit 104, a display unit 105, a light source control unit 106, an imaging control unit 107, an information input unit 108, a reflection characteristic model recording unit 109, and an image recording unit 110. The system controller 111 is included. Further, as described above, the imaging apparatus 1 includes the light source means 200.

  In the present embodiment, the imaging unit 100 includes an imaging optical system 101, an imaging element 102, and various driving means (or moving means).

  The imaging optical system 101 forms an optical image (subject image) of a subject on the imaging surface of the image sensor 102, and includes a zoom lens (magnification lens) 101a, a diaphragm 101b, a focus lens 101c, and a wavelength selective imaging filter FL101d. Note that the ellipse in FIG. 2 schematically shows one of these lenses, and this is the same in the examples described later.

  The zoom lens (magnification lens) 101a is moved in the optical axis direction by a driving unit (not shown) to change the focal length. The diaphragm 101b is disposed at the exit pupil position of the imaging optical system 101, is driven by a driving unit (not shown), and adjusts the amount of light incident on the imaging element 102. The focus lens 101c is moved in the optical axis direction by a driving unit (not shown) to perform focus adjustment. The wavelength selective imaging filter FL101d functions as means for changing (decreasing) the transmittance of invisible light, and is moved by the moving means 101e so as to be inserted into and removed from the optical path of the imaging optical system 101. When irradiating visible light, the system controller 111 inserts the wavelength-selective imaging filter FL101d into the optical path, and when irradiating invisible light, the system controller 111 moves the moving means 101e via the imaging control means 107. To control. That is, in the case of irradiating invisible light, the transmittance of invisible light is increased more than in the case of irradiating visible light.

  The image pickup element 102 is configured by a photoelectric conversion element such as a CCD sensor or a CMOS sensor, and picks up an object by photoelectrically converting an optical image formed by the image pickup optical system 101. The image sensor 102 includes, for example, a color filter having a Bayer array on the photoelectric conversion element group, but the color filter array is not limited to this. In the Bayer array, 1 pixel of R (red), 2 pixels of G (green), and 1 pixel of B (blue) are regularly arranged in 4 pixels of 2 × 2. The two G pixels are arranged in a diagonal direction, and the R pixel and the B pixel are provided in a diagonal direction orthogonal thereto.

  In the normal line information acquisition method of the present invention, the subject is imaged by sequentially irradiating the subject with light from a plurality of light sources having three or more different positions, and luminance information is acquired. The subject is picked up by the image sensor 102 to become a picked-up image, and the luminance information is acquired from the picked-up image of the subject by the image processing means 104. Here, “luminance information” is a pixel value (luminance value) of each pixel of RGB.

  In the present invention, visible light source means and invisible light source means are used as light sources, but each pixel of RGB has sensitivity to invisible light (for example, near infrared light). In the illuminance difference stereo method, changes in luminance information (luminance differences) are required at a plurality of positions instead of the luminance information itself. Therefore, even if the image sensor 102 does not include a dedicated pixel sensitive only to invisible light, the illuminance difference stereo. Luminance information required by the method can be acquired. Needless to say, the image sensor 102 may include pixels that are not sensitive to visible light but sensitive to invisible light.

  The present invention makes it possible to generate an image in which the appearance of a subject is changed in image processing after shooting. For this purpose, it is necessary to acquire not only the normal information of the surface of the subject but also the captured image of the subject with respect to visible light. When the image sensor 102 has a normal Bayer color filter and illuminates the subject using only near infrared light, each pixel of RGB has sensitivity to near infrared light, so the photographed image is reddish. It becomes an image. When the imaging element 102 includes a dedicated pixel that has sensitivity only to near-infrared light, the subject image can be corrected by subtracting the pixel value of the dedicated pixel from each pixel value of the RGB pixel. Of course, as will be described later, this method need not be used.

  The A / D converter 103 converts the analog electrical signal generated by the image sensor 102 into a digital signal and outputs the digital signal to the image processing unit 104.

  The image processing unit 104 performs color conversion, white balance, gamma correction and other image processing, resolution conversion processing, image compression processing, and the like on the digital signal. Further, the image processing unit 104 estimates (acquires) normal line information of the local surface of the subject. The image processing unit 104 obtains normal information on the surface of the subject based on the face detection unit 104a that detects a face in the captured image of the subject obtained by the imaging element 102 and the luminance information obtained from the captured image of the subject. Normal information acquisition means 104b.

  The normal line information acquisition unit 104b may be configured as an independent dedicated image processing apparatus (normal line information acquisition apparatus) or a personal computer (PC) without being provided in the image processing unit 104. In this case, software for executing the normal information acquisition method and the image processing method for changing the appearance of the image of the present invention is installed in the PC. This software can be embodied as a program for causing a computer to execute these methods, and such a program can be stored in a storage means (for example, a non-transitory computer-readable storage medium).

  The imaging device or the imaging system described above and the normal information acquisition device constitute a normal information acquisition system.

  The display unit 105 displays the output image processed by the image processing unit 104, various control information, normal information, and images before and after the appearance is changed, and includes a liquid crystal display, various lamps, and the like. The light source control means 106 performs light emission control (light emission on / off and light emission amount) of the light source means 200 based on information from the system controller 111.

  The imaging control unit 107 controls the operations of the imaging optical system 101 and the imaging element 102 based on information from the system controller 111. More specifically, the imaging control unit 107 moves the focus lens and the zoom lens, the aperture value by the aperture 101b, the exposure time, the insertion / removal of the wavelength selective imaging filter FL101d by the moving unit 101e, and the energization to the imaging element 102. Control etc. When the imaging optical system 101 is provided in the lens apparatus, the lens control unit of the lens apparatus has a partial function of the imaging control unit 107, and the lens control unit is a camera control unit of the imaging apparatus body. Controlled by.

  The information input means 108 outputs data representing information input by the user selecting desired photographing conditions (aperture value, exposure time, focal length, etc.) to the system controller 111, such as an output switch, dial, button, lever, touch screen, etc. It is the input means containing.

  In the present embodiment, the system controller 111 automatically switches the light emission of the light source means 200, but the present invention also allows the user to switch manually. In this case, an imaging apparatus that sequentially irradiates light from a plurality of different positions at three or more and images the subject at the time of each irradiation, each of visible light and invisible light is converted into a plurality of three or more different light. A light source means 200 for irradiating from a position and an information input means 108 are provided. The information input unit 108 functions as an operation unit that allows a user to manually switch between irradiation of visible light and invisible light.

  The reflection characteristic model recording means 109 is a storage means for storing the reflection characteristic model data of the subject by invisible light and the Lambertian reflection model described above, and is constituted by a semiconductor memory, an optical disk, or the like.

  The image recording unit 110 is a storage unit that records various types of information such as an output image processed by the image processing unit 104, surface normal information, and luminance information, and includes a semiconductor memory, an optical disk, and the like.

  The system controller 111 is a control unit that controls the operation of each unit of the imaging apparatus 1 and includes a microcomputer. In particular, the system controller 111 functions as a control device that sequentially irradiates the subject with light from a plurality of different positions of three or more and images the subject at each irradiation. The system controller 111 starts shooting preparation operations such as AWB (auto white balance) processing, EF (flash pre-flash) processing, face detection, etc., in accordance with the SW1 signal generated by the release button 300. Further, the system controller 111 drives a shutter drive circuit (not shown) in accordance with the SW2 signal generated by the release button 300, guides the photographing light flux to the image sensor 102, and performs exposure (photographing). Further, the system controller 111 controls a series of photographing processing operations from reading a signal from the image sensor 102 to writing image data in the image recording unit 110 according to the SW2 signal. Further, the system controller 111 executes the control method shown in FIG.

  FIG. 3 is a flowchart showing an image processing method (normal line estimation method) executed by the system controller 111, and “S” represents a step (process). The flowchart shown in FIG. 3 can be embodied as a program for causing a computer to execute the processing of each step. Such a program can be stored in the image recording unit 110 or other storage unit (for example, a non-transitory computer). A readable storage medium). Y represents “Yes”, and N represents “No”.

  In S101, the system controller 111 sets desired shooting conditions (aperture value, exposure time, focal length, etc.) input from the information input unit 108 by the user.

  In S102, the system controller 111 determines whether or not the release button 300 is half-pressed based on whether or not the SW1 signal is received. When half-pressed, the imaging apparatus enters a shooting preparation state, performs pre-shooting, and stores the pre-shot image in the image recording unit 110 or a storage unit (not shown). In pre-photographing, a captured image of a subject is acquired by capturing the subject in a state where neither the visible light source unit 210 nor the invisible light source unit 220 emits light. When the image sensor 102 does not have a pixel that is sensitive only to invisible light (that is, a pixel that is not sensitive to visible light and sensitive to invisible light), the image processing unit 104 performs each of the RGB pixels at this time. The pixel value is recorded in the image recording unit 110. This information can be used to correct an image when the subject includes a face.

  In S103, the system controller 111 determines whether the face of the subject is detected in the pre-captured image obtained in S102 based on the detection result of the face detection unit 104a. The face detection unit 104a performs horizontal bandpass filter processing on the pre-captured image data, and performs vertical bandpass filter processing on the image data that has been subjected to the horizontal bandpass filter processing. Thereby, an edge component is detected from the image data.

  Next, the face detection unit 104a performs pattern matching on the detected edge component to extract a candidate group of eyes, nose, mouth, and ears. Next, the face detection unit 104a determines a candidate satisfying a preset condition (for example, conditions such as the distance and inclination of two eyes) from the extracted eye candidate group as an eye pair. Narrow down the candidate group of eyes by excluding the candidate group that does not make a pair.

  Next, the face detection unit 104a associates the narrowed-down eye candidate group with other parts (nose, mouth, ears) that form a face corresponding to the eye, and determines a face area candidate. A preset non-face condition filtering process is performed on the candidate. The face detection unit 104a excludes face area candidates determined not to satisfy the face condition, detects the remaining face area candidates as face areas, and outputs face information indicating the detection result of the face area to the system controller 111.

  FIG. 4 is a diagram illustrating an example of a detected human face area 1001. The system controller 111 stores the face information in the image recording unit 110 or a storage unit (not shown). If no face is detected by the face detection unit 104a, the flow proceeds to S104, and if a face is detected, the flow proceeds to S105.

  In S104, the system controller 111 selects all light sources (a total of 16 visible light source means 210 and invisible light source means 220) as light to be emitted to the light source control means 106 (as light sources for performing the illuminance difference stereo method). Order to do. In S104, only the visible light source unit 210 need not be selected and the invisible light source unit 220 need not be selected. However, the invisible light source unit 220 is not selected without selecting the visible light source unit 210. This is because in the latter case, there is a risk that the normal information acquisition accuracy may be reduced. The reason for selecting both is that the normal information acquisition accuracy improves as the number of selected light sources increases.

  In S105, the system controller 111 instructs the light source control means 106 to select the invisible light source means 220 without selecting the visible light source means 210 as the light to be irradiated. This is because when a face is detected, the subject includes a person, and when the visible light source means 210 is turned on, the person feels dazzling (has discomfort). Thereby, it is possible to reduce discomfort due to glare when estimating the normal information of the subject person.

  In S106, the system controller 111 determines whether the release button 300 is fully pressed based on whether the SW2 signal is received. When fully pressed, the imaging apparatus 1 enters a shooting state and starts actual shooting.

  In step S107, the system controller 111 controls the imaging control unit 107 and the light source control unit 106 to sequentially illuminate light sources at three or more different positions, thereby imaging a subject and generating a plurality of captured images. get.

  When the flow from S104 to S107 is performed, 16 light sources sequentially emit light, and when the flow from S105 to S107 is performed, 8 light sources sequentially emit light. In synchronization with each light emission, the image sensor 102 images a subject to generate a plurality of captured images, and the image processing unit 104 acquires luminance information for each captured image. The subject is imaged at a plurality of light source positions by causing the image sensor 102 to image in accordance with the timing.

  When the invisible light source unit 220 is turned on, the imaging control unit 107 transmits the invisible light to the imaging element 102 by retracting the wavelength selective imaging filter FL101d. That is, the system controller 111 moves the moving unit via the imaging control unit 107 so as to insert the filter into the optical path when irradiating visible light, and retract the filter from the optical path when irradiating invisible light. 101e is controlled.

  In this embodiment, the wavelength-selective imaging filter FL101d is moved, but the transmittance and reflectance are reduced by changing the filter composition by electrical control to adjust the transmission of invisible light to the imaging element 102. Also good. In addition, when the image sensor 102 includes pixels that are sensitive only to near-infrared light, the resolution is reduced, but wavelength separation is performed without using a wavelength selective filter (redness due to near-infrared light is removed by correction). It is also possible to obtain an appropriate subject image.

  In S <b> 108, the system controller 111 uses the digital signal output from the image sensor 102 and A / D converted by the A / D converter 103 to the normal information acquisition unit 104 b of the image processing unit 104 to use the normal information. Is estimated (acquired). At this time, the image processing means 104 may perform normal development processing and various image correction processing for image generation. The normal line information acquisition unit 104b estimates a normal line based on a change in luminance information depending on a light source position using an illuminance difference stereo method.

  When the flow from S104 to S107 is performed, the reflection characteristic of the general subject is assumed to be a Lambertian reflection model recorded in the reflection characteristic model recording unit 109, and the illuminance difference stereo method is used based on the change in the luminance information in each wavelength region. The surface normal is calculated individually. As shown in Non-Patent Document 1, since the reflection characteristic varies depending on the wavelength of irradiation depending on the material and color of the subject, the surface normal calculated in each wavelength range is different including an error, but the average value thereof is different. By taking the error can be reduced. Further, by using the method shown in Non-Patent Document 1, it is possible to improve the surface normal acquisition accuracy by determining and selecting a light source in a wavelength region where the reflection characteristic is closest to the Lambertian reflection model. In this way, even when a general subject (subject other than a person) is targeted, it is possible to suppress a decrease in surface normal acquisition accuracy based on changes in luminance information in a plurality of wavelength ranges. It becomes.

  When the flow from S105 to S107 is performed, the reflection characteristics of the human face and skin in the invisible light region are different from those of the Lambert reflection model. An estimation error can occur. Therefore, in this embodiment, human reflection characteristic data for invisible light is acquired in advance, a human reflection characteristic model is created based on the reflection characteristic data, and stored in the reflection characteristic model recording means 109. . By estimating the normal information of the person based on the reflection characteristic model for a person and the illuminance difference stereo method, the surface normal of the person can be accurately calculated.

  According to the present embodiment, when a face is detected, the dullness in estimating the normal information of the person is reduced by selecting the invisible light source means 220 without selecting the visible light source means 210. can do. Furthermore, by providing a human reflection characteristic model for invisible light used in the illuminance difference stereo method, it is possible to accurately estimate the human surface normal. In addition, by using a light source in a plurality of wavelength regions for a general subject, it is possible to estimate normal information that is robust to a general wider range of subjects.

  FIG. 5 is an external perspective view of the imaging apparatus 2 according to the second embodiment of the present invention. The imaging device 2 includes a release button 300, two imaging units 400 and 500 that image a subject, and eight light source means 600. The light source means 600 is a broadband light source capable of emitting visible light and invisible light from each of a plurality of positions.

  FIG. 6A is a schematic diagram illustrating the configuration of the imaging unit 400. The imaging unit 400 is an imaging unit for photographing a visible light region, and photoelectrically converts a first imaging optical system 401 that forms an optical image of the subject and an optical image of the subject formed by the first imaging optical system 401. A first imaging element 402. The first imaging optical system 401 transmits light having a wavelength in the visible light region and forms an image on the first imaging element 402, but does not transmit invisible light. The first imaging optical system 401 includes a zoom lens (magnification lens) 401a, a diaphragm 401b, a focus lens 401c, and a wavelength selective imaging filter FL401d, and these functions are the same as those in the first embodiment.

  FIG. 6B is a schematic diagram illustrating the configuration of the imaging unit 500. The imaging unit 500 includes a second imaging optical system 501 that forms an optical image of the subject, and a second imaging element 502 that photoelectrically converts the optical image of the subject formed by the second imaging optical system 402. The second imaging optical system 501 includes a zoom lens (magnification lens) 501a, a diaphragm 501b, and a focus lens 501c. These functions are the same as those in the first embodiment, but do not include a wavelength-selective imaging filter. . That is, the second imaging optical system 501 transmits light having a wavelength in the visible light region and invisible light, and forms an image on the second imaging element 502. That is, the second imaging optical system 501 has a higher transmittance for invisible light than the first imaging optical system 401. For this reason, the imaging unit 500 functions as an imaging unit for invisible light region imaging.

  FIG. 7 is a schematic diagram showing the configuration of the light source means 600. The light source means 600 includes a wavelength-selective light source filter FL620 that is disposed on the image sensor side of the light emitting unit 610 and reduces transmission of invisible light, and a moving means 630 that removably moves the filter on the optical axis. ing. The driving by the moving unit 630 is controlled by the light source control unit 106. The system controller 111 moves through the light source control means 106 so that the wavelength selective light source filter FL620 is inserted into the optical path when irradiating visible light, and the filter is retracted from the optical path when irradiating invisible light. Control means 630.

  In addition, the number of the light emission parts of each light source means is not limited to one. R (wavelength 620 nm to 750 nm, for example, 700 nm), G (wavelength 495 nm to 570 nm, for example, 546.1 nm) B (wavelength 430 nm to 495 nm, for example, 435.8 nm), near infrared light emitting part It may be provided with a plurality. In that case, since the irradiation wavelength can be selected by selecting the light emitting unit 610, the irradiation wavelength can be controlled without using the wavelength selective light source filter FL620. Except for the configuration of the imaging units 400 and 500 and the light source unit 600, the configuration is the same as the block diagram shown in FIG.

  FIG. 8 is a flowchart showing an image processing method (normal line estimation method) executed by the system controller 111, and “S” represents a step (process). The flowchart shown in FIG. 3 can be embodied as a program for causing a computer to execute the processing of each step. Such a program can be stored in the image recording unit 110 or other storage unit (for example, a non-transitory computer). A readable storage medium). Y represents “Yes”, and N represents “No”. Only steps different from those in FIG. 3 will be described below.

  S201 to S203 are the same as S101 to S103. In S204, the system controller 111 causes the light source control unit 106 to select light in the entire wavelength region as light to be irradiated (as a light source for performing the illuminance difference stereo method) when a face is not detected in S103. “Full wavelength region” means the full wavelength region that the light source means 600 can irradiate. In response to this, the light source control means 106 controls a driving means (not shown) to retract the wavelength selective light source filter FL620 from the optical path.

  In S205, when a face is detected in S103, the system controller 111 causes the light source control unit 106 to select invisible light (near infrared light) as light to be irradiated, and does not select visible light. In response to this, the light source control means 106 controls a driving means (not shown) to insert the wavelength selective light source filter FL620 into the optical path. This is because when a face is detected, a person is included in the subject and the person feels dazzling when irradiated with visible light. Thereby, it is possible to reduce discomfort due to glare when estimating the normal information of the subject person.

  Instead of inserting / removing the wavelength selective light source filter FL620, the composition of the filter may be changed by electrical control, and the wavelength to be irradiated may be controlled by changing the transmittance or reflectance.

  S206 is the same as S106. In S207, the system controller 111 controls the imaging control unit 107 and the light source control unit 106 to image a subject at a plurality of light source positions, and causes the image processing unit 104 to acquire luminance information. In particular, when invisible light is irradiated, the imaging control unit 107 controls the imaging unit 500 for invisible light imaging to acquire luminance information. In this embodiment, the imaging unit is configured with two optical systems for visible light and invisible light, thereby simultaneously obtaining luminance information in both wavelength regions without retracting the wavelength selective imaging filter. Therefore, simplification of control can be realized. S208 is the same as S108.

  According to the present embodiment, when a face is detected, glare when estimating normal information of a person can be reduced by selecting invisible light without selecting visible light. Further, by providing a human reflection characteristic model for invisible light used in the illuminance difference stereo method, it is possible to accurately estimate the surface normal of the person. In addition, by using a light source in a plurality of wavelength regions for a general subject, it is possible to estimate normal information that is robust to a general wider range of subjects.

  Although the embodiments of the present invention have been described above, the present invention can be variously modified and changed within the scope of the gist thereof. The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  The present invention can be applied to a method, an apparatus, a system, and the like that acquire normal information.

111 ... System controller (control means, control device), 210 ... Visible light source means, 220 ... Invisible light source means

Claims (27)

A control device that sequentially irradiates a subject with light from a plurality of three or more different positions and images the subject at the time of each irradiation,
Before irradiating the light from the plurality of positions, determine whether the face of the subject is detected,
If the face is detected, select invisible light as the light to irradiate without selecting visible light,
When the face is not detected, the visible light is selected as the irradiation light.   The control device according to claim 1, wherein when the face of the subject is not detected, the control device further selects the invisible light as the light to be irradiated. An imaging device that sequentially irradiates a subject with light from three or more different positions and images the subject at each irradiation,
Light source means for irradiating each of visible light and invisible light from a plurality of three or more different positions;
An operation means for enabling manual switching between irradiation of the visible light and the invisible light;
An imaging device comprising: An image sensor for imaging a subject;
Face detection means for detecting a face in the captured image of the subject obtained by the image sensor;
Control means for sequentially irradiating the subject with light from a plurality of three or more different positions, and causing the imaging device to image the subject at each irradiation;
Have
The control means includes
Before irradiating the light from the plurality of positions, determine whether the face detection means has detected the face of the subject,
When the face detection means detects the face, select invisible light without selecting visible light as light to be irradiated,
When the face detection unit does not detect the face, the visible light is selected as the light to be irradiated.   The imaging apparatus according to claim 4, further comprising a light source unit that irradiates each of the visible light and the invisible light from a plurality of three or more different positions.   The light source means comprises: a visible light source means for irradiating the visible light; and an invisible light source means for irradiating the invisible light provided at a position different from the visible light source means. Item 6. The imaging device according to Item 5.   The imaging apparatus according to claim 6, wherein the invisible light source unit is provided outside the visible light source unit. An optical system for forming an optical image of the subject, the optical system comprising means for changing the transmittance of the invisible light;
The said control means makes the transmittance | permeability of the said invisible light in the said optical system higher when irradiating the said invisible light than when irradiating the said visible light. The imaging apparatus of Claim 1. The optical system is
A filter that reduces the transmittance of the invisible light;
Moving means for moving the filter so that it can be inserted into and removed from the optical path of the optical system;
Have
The control means controls the moving means to insert the filter into the optical path when irradiating the visible light, and to retract the filter from the optical path when irradiating the invisible light. The imaging device according to claim 8. The optical system includes a filter that changes the transmittance of the invisible light,
The said control means changes the transmittance | permeability or reflectance of the said invisible light by the said filter by electrical control between the case where the said visible light is irradiated, and the case where the said invisible light is irradiated. Item 9. The imaging device according to Item 8.   The image pickup device according to claim 4, wherein the image pickup device includes a pixel having no sensitivity to the visible light and having a sensitivity to the invisible light.   The imaging apparatus according to claim 5, wherein the light source unit includes a broadband light source that irradiates the visible light and the invisible light from each of the plurality of positions. The light source means includes
A filter that reduces the transmittance of the invisible light;
Moving means for moving the filter so that it can be inserted into and removed from the optical path of the light source means;
Have
The control means controls the moving means to insert the filter into the optical path when irradiating the visible light, and to retract the filter from the optical path when irradiating the invisible light. The imaging apparatus according to claim 12.   13. The light source unit according to claim 12, wherein the light source means includes light emitting units for light having a wavelength of 620 nm to 750 nm, light having a wavelength of 495 nm to 570 nm, light having a wavelength of 430 nm to 495 nm, and the invisible light. Imaging device. A first optical system for forming an optical image of the subject;
A second optical system having a higher transmittance of the invisible light than the first optical system and forming an optical image of the subject;
Further comprising
The image sensor is
A first image sensor that photoelectrically converts the optical image formed by the first optical system;
A second image sensor that photoelectrically converts the optical image formed by the second optical system;
The imaging apparatus according to any one of claims 4 to 7, and 12 to 14.   16. The normal line information acquisition unit for acquiring normal line information on the surface of the subject based on luminance information obtained from a captured image of the subject. Imaging device. Storage means for storing a reflection characteristic model of the subject by the invisible light;
The normal line information acquisition unit acquires the normal line information based on the reflection characteristic model stored in the storage unit when the face detection unit detects the face. 16. The imaging device according to 16. An imaging device that sequentially irradiates a subject with invisible light from three or more different positions, and images the subject at each irradiation,
Storage means for storing a reflection characteristic model of the subject by the invisible light;
Normal information acquisition means for acquiring normal information of the surface of the subject based on luminance information obtained from the captured image of the subject and the reflection characteristic model stored in the storage means;
An imaging device comprising: An imaging device according to claim 4,
A light source device that irradiates each of the visible light and the invisible light from a plurality of three or more different positions;
An imaging system comprising:   The imaging system according to claim 19, further comprising a lens device that forms an optical image of the subject and is attachable to the imaging device. The lens device includes means for changing the transmittance of the invisible light,
The said control means raises the transmittance | permeability of the said invisible light in the optical system which forms the optical image of the said subject, when irradiating the said invisible light rather than the case where the said visible light is irradiated. Item 21. The imaging system according to Item 20. The imaging device according to any one of claims 4 to 14,
A normal information acquisition device for acquiring normal information of the surface of the subject based on luminance information obtained from a captured image of the subject;
A normal information acquisition system comprising: An imaging system according to claim 19 to 21,
A normal information acquisition device for acquiring normal information of the surface of the subject based on luminance information obtained from a captured image of the subject;
A normal information acquisition system comprising: A lens device that forms an optical image of a subject and is attachable to the imaging device according to claim 4,
The lens apparatus has light source means for irradiating each of the visible light and the invisible light from a plurality of three or more different positions. An image processing apparatus for obtaining normal information of a surface of a subject from a captured image of the subject,
Storage means for storing a reflection characteristic model of the subject by invisible light;
Normal information acquisition means for acquiring normal information of the surface of the subject based on luminance information obtained from the captured image and the reflection characteristic model stored in the storage means;
An image processing apparatus comprising: A control method for sequentially irradiating a subject with light from a plurality of different positions from three or more, and imaging the subject at each irradiation,
Determining whether a face of the subject is detected before irradiating the light from the plurality of positions;
If the face of the subject is detected, the step of selecting invisible light without selecting visible light as the irradiation light;
If the face of the subject is not detected, selecting the visible light as the light to irradiate;
A control method characterized by comprising:   A program for causing a computer to execute the control method according to claim 26.