JP2008016918A - Image processor, image processing system, and image processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a technology of realizing an image with high resolution by imaging a light source image resulting from imaging a light source and estimating light source information without the need for mount of an additional imaging apparatus. <P>SOLUTION: A wide angle image imaging section 101 uses an imaging apparatus 1001 capable of imaging a wide angle image and automatically and consecutively images an object. An imaging apparatus information acquisition section 102 acquires imaging apparatus information representing a state of the imaging apparatus 1001 in its imaging state. A light source information estimate section 103 uses an imaged wide angle image and the acquired imaging apparatus information to estimate the light source information including the position, the color and the illuminance of the light source, and an image high resolution section 104 uses the estimated light source information to bring the imaged wide angle image to have a high resolution. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a technique for estimating light source information such as the position, direction, brightness, spectrum, and color of a light source at the time of image capturing, and a technique for increasing the resolution of an image using the estimated light source information.

  With the spread of camera-equipped mobile phones and digital cameras, the importance of image processing has increased. As such image processing, there are various types such as high resolution of an image known as digital zoom and augmented reality in which an image created by Computer-Graphics is superimposed and displayed.

  These image processes are performed based on the “appearance” of the subject recorded on the image sensor by image capturing. The appearance of the object is obtained when light from the light source is reflected on the surface of the subject and the reflected light is received by the image sensor.

  Therefore, light source information is very important in image processing. That is, it is very effective to acquire light source information and use it for image capturing and image processing. For example, in Patent Document 1, when a virtual object is superimposed and displayed on the real world, a reflected light from irradiation light reflected on the virtual object or a shadow generated by the virtual object is added by measuring a light source environment in the real world. To do.

The light source information is useful not only for image processing, but also when an imager who is a subject performing an imaging action acquires an image with an imaging apparatus. For example, in Patent Document 2, the light source position is detected, and the information is used to display the front light state or the back light state to the photographer. It is possible to easily capture portrait photographs of backlight (half backlight) using shadows and shadows.
JP-A-11-175762 JP-A-8-160507

In order to realize such image processing, an image obtained by imaging a light source, that is, a light source image is necessary. The light source image
(1) Capture a different direction from the subject, and
(2) It is desirable to have a wide range of images close to the whole sky. This is due to the following reason.

  (1) A general subject, for example, a person, is often illuminated from above by street lamps or indoor lighting. Further, when the light source is in the same direction as the subject, the subject may be backlit and the subject may become dark. Therefore, the photographer wants to avoid that the subject and the light source enter the same image.

  (2) How many light sources are unknown to the photographer.

  For this reason, in the above-mentioned patent document 1 and patent document 2, in order to acquire light source information, the optical axis is turned to the zenith direction, and the imaging device equipped with the fisheye lens is used.

  However, in order to image the fisheye lens in a direction different from the subject as described above, a second imaging device different from the imaging device that images the subject is required. This is a great burden in terms of cost. In particular, wearable cameras and camera-equipped mobile phones that are worn on the body are highly demanded for miniaturization, so increasing the number of imaging devices is a big problem in terms of size.

  Furthermore, since the imaging device that images the subject is different from the imaging device that images the light source, alignment (calibration) between the two imaging devices is required. This is because the positional relationship between the subject and the light source is particularly important in estimating the light source information. That is, when the subject and the light source are imaged by separate imaging devices, it is necessary to associate them with each other, but this operation is very complicated.

  In view of the above problems, the present invention captures a light source image in which a light source is captured without mounting an additional imaging device, for example, in an apparatus such as a wearable camera, estimates light source information, and increases the height of the image. An object is to realize resolution.

  As an image processing apparatus and method, an imaging apparatus that automatically and continuously captures a subject using an imaging apparatus capable of capturing a wide-angle image and represents the state of the imaging apparatus when the imaging is performed Information is obtained, light source information including the position, color and illuminance of the light source is estimated using the captured wide-angle image and the acquired imaging device information, and the wide-angle image is estimated using the estimated light source information. The resolution is increased.

  According to the present invention, an object is automatically and continuously imaged using an imaging device capable of imaging a wide-angle image, so that the possibility that a light source around the object is reflected in the captured wide-angle image increases. That is, a light source image can be obtained together with the subject image. Then, light source information including the position, color, and illuminance of the light source is estimated using the captured wide-angle image and the imaging device information when the imaging is performed, and the wide-angle is calculated using the estimated light source information. The image is increased in resolution. That is, it is possible to acquire a light source image and realize high resolution of the image without mounting an additional imaging device other than for subject imaging.

  According to the present invention, it is possible to capture a light source image around a subject without mounting an additional imaging device other than the subject imaging that is the original purpose, and estimate light source information from the light source image thus captured. can do. Thereby, various image processing such as high-power digital zoom processing can be realized.

  In the first aspect of the present invention, using an imaging device capable of capturing a wide-angle image, the wide-angle image imaging unit that automatically and continuously images the subject, and the imaging when the wide-angle image imaging unit captures the image Using an imaging device information acquisition unit that acquires imaging device information representing the status of the imaging device, a wide-angle image captured by the wide-angle image imaging unit, and imaging device information acquired by the imaging device information acquisition unit, A light source information estimation unit that estimates light source information including a position, color, and illuminance of a light source, and an image resolution enhancement unit that increases the resolution of the wide-angle image using the light source information estimated by the light source information estimation unit. Provided is an image processing apparatus.

  In the second aspect of the present invention, the imaging device information acquisition unit uses at least one of an output of an angle sensor mounted on the imaging device and an image captured by the wide-angle image imaging unit, The image processing device according to the first aspect is provided for acquiring the imaging device information.

  According to a third aspect of the present invention, there is provided the image processing device according to the first aspect, wherein the wide-angle image capturing unit sets an exposure time of a pixel of the image capturing device to a different value for each region.

  In the fourth aspect of the present invention, the wide-angle image capturing unit sets a pixel exposure time shorter in a region corresponding to a peripheral portion of the lens of the imaging device than in a region corresponding to the central portion of the lens. An image processing apparatus according to the third aspect is provided.

  According to a fifth aspect of the present invention, the image processing device includes a shape information acquisition unit that acquires surface normal information or three-dimensional position information of the subject as shape information, and the image high-resolution unit includes the shape information acquisition unit. The image processing apparatus according to the first aspect is provided for increasing the resolution of the wide-angle image using the acquired shape information.

  In the sixth aspect of the present invention, the image resolution increasing unit separates the wide-angle image into a diffuse reflection component and a specular reflection component, and individually increases the resolution of the separated diffuse reflection component and specular reflection component. An image processing apparatus according to the fifth aspect is provided.

  According to a seventh aspect of the present invention, there is provided the image processing device according to the fifth aspect, wherein the image resolution increasing unit decomposes the wide-angle image into parameters and individually increases the resolution of the decomposed parameters. .

  According to an eighth aspect of the present invention, there is provided the image processing device according to the first aspect, wherein the image resolution increasing unit sets an enlargement magnification for increasing the resolution to a different value for each region of the image.

  In the ninth aspect of the present invention, the image resolution increasing unit sets the magnification in the region corresponding to the peripheral portion of the lens of the imaging device higher than the region corresponding to the central portion of the lens. An image processing apparatus according to an aspect is provided.

  In a tenth aspect of the present invention, the light source environment determination unit that determines whether or not the light source environment for the imaging device has changed is provided, and the light source information estimation unit determines that the light source environment has changed by the light source environment determination unit. When this is done, the image processing apparatus of the first aspect for estimating light source information is provided.

  In an eleventh aspect of the present invention, the light source environment determining unit determines whether or not the imaging device has moved from the output of an angle sensor mounted on the imaging device, and the light source environment has changed when the imaging device has moved. The image processing apparatus according to the tenth aspect is determined.

  In a twelfth aspect of the present invention, the imaging environment determination unit determines whether or not the imaging device has moved using an image captured by the wide-angle image imaging unit, and the light source environment has changed when the imaging device has moved. The image processing apparatus according to the tenth aspect is determined.

  In a thirteenth aspect of the present invention, the light source information estimation unit includes an estimated light source information determination unit that determines whether or not the light source information is accurately estimated, and the light source information estimation unit accurately estimates the light source information. If it is determined by the estimated light source information determination unit that it is not, the image processing apparatus according to the first aspect that performs light source information estimation again is provided.

  In a fourteenth aspect of the present invention, the estimated light source information determination unit estimates the direction of sunlight or the position of the sun using the output of a GPS (Global Positioning System) sensor and time information, and the estimated direction or position Is provided with the light source information estimated by the light source information estimation unit, thereby providing the image processing apparatus according to the thirteenth aspect for performing the determination.

  In the fifteenth aspect of the present invention, the estimated light source information determination unit estimates the image of the subject using the shape information of the subject and the light source information estimated by the light source information estimation unit, The image processing apparatus according to the thirteenth aspect that performs determination by comparing with an image captured by the wide-angle image capturing unit.

  In the sixteenth aspect of the present invention, the estimated light source information determining unit determines the color information of the image captured by the wide-angle image capturing unit by comparing the color information of the light source estimated by the light source information estimating unit. An image processing apparatus according to the thirteenth aspect is provided.

  In a seventeenth aspect of the present invention, as an image processing system for increasing the resolution of an image, the wide-angle image captured by the wide-angle image capturing unit, including the wide-angle image capturing unit and the imaging device information acquiring unit of the first aspect, And the communication terminal that transmits the imaging device information acquired by the imaging device information acquisition unit, the light source information estimation unit and the image resolution enhancement unit of the first aspect, and the wide-angle image transmitted from the communication terminal And the server that receives the imaging device information, provides the wide-angle image to the light source information estimation unit and the image resolution enhancement unit, and provides the imaging device information to the light source information estimation unit, The server is provided for instructing the contents of light source information estimation and image resolution enhancement.

  In the eighteenth aspect of the present invention, a first step of automatically and continuously imaging a subject using an imaging device capable of capturing a wide-angle image, and when the imaging is performed in the first step Using the second step of acquiring imaging device information representing the status of the imaging device, the wide-angle image captured in the first step, and the imaging device information acquired in the second step, a light source A third step of estimating light source information including the position, color, and illuminance, and a fourth step of increasing the resolution of the wide-angle image using the light source information estimated in the third step. An image processing method is provided.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(First embodiment)
FIG. 1 is a block diagram showing the configuration of the image processing apparatus according to the first embodiment of the present invention. In FIG. 1, reference numeral 101 denotes a wide-angle image capturing unit that automatically and continuously captures an object using an imaging device 1001 that can capture a wide-angle image. An imaging device information acquisition unit 103 that acquires imaging device information representing the status of the device 1001, 103 uses the wide-angle image captured by the wide-angle image imaging unit 101 and the imaging device information acquired by the imaging device information acquisition unit 102. A light source information estimation unit 104 that estimates light source information including the position, color, and illuminance of the light source, and 104 uses a light source information estimated by the light source information estimation unit 103 to generate a high-angle image captured by the wide-angle image imaging unit 101. It is an image high-resolution part for increasing the resolution.

  Here, the wide-angle image imaging unit 101, the imaging device information acquisition unit 102, the light source information estimation unit 103, and the image resolution enhancement unit 104 are realized, for example, by executing a program by the CPU 1021. However, all or part of these functions may be realized by hardware. The memory 1022 stores the captured image captured by the wide-angle image capturing unit 101 and the image capturing apparatus information acquired by the image capturing apparatus information acquiring unit 102.

  FIG. 2 is a schematic diagram illustrating a usage form of a wearable camera as a device on which the image processing apparatus according to the present embodiment is mounted. In FIG. 2, a photographer 1003 who is a subject performing an imaging action has a wearable camera 1000 installed on his / her shoulder. Reference numeral 1017 denotes the optical axis direction of the imaging device of the wearable camera 1000. As shown in FIG. 2, it is effective that the image processing apparatus according to the present embodiment is used in a wearable camera or the like.

  The imaging apparatus 1001 is configured by a CMOS, a CCD sensor, or the like, and is mounted with, for example, a fisheye lens, an all-around camera, or a lens with a short focal length so that a wide-angle image can be captured. In addition, a triaxial angle sensor 1020 is mounted on the imaging apparatus 1001. The wide-angle image capturing unit 101 uses the image capturing apparatus 1000 to automatically and continuously capture a subject. That is, in the present embodiment, imaging is continuously performed without an aggressive operation such as a photographer pressing an imaging switch. The wide-angle image captured at this time is highly likely to include a light source. The captured wide-angle image is stored in the memory 1022.

  FIG. 3 is a schematic diagram showing a captured wide-angle image. In FIG. 3, the visual field range 1004 of the imaging apparatus 1001 includes a subject 1018 and a light source 1005 that irradiates the subject 1018. That is, the light source pixel 1007 and the subject image 1019 are reflected in the wide-angle image 1006 captured by the imaging device 1001. In this manner, the light source 1005 and the subject 1018 can be captured simultaneously by capturing an image at a wide angle. In addition, when using a lens with a short focal distance, it is desirable to capture all images that the photographer is interested in. For this reason, it is desirable to use a lens having a viewing angle of 110 degrees or more so that a guided field of view, which is a region of the field of view that is not clearly visible but is worried about the presence or movement of an object, can be captured.

  The imaging device information acquisition unit 102 acquires imaging device information representing the state of the imaging device 1001 when imaging is performed by the wide-angle image imaging unit 101. Specifically, for example, the output of the angle sensor 1020 and the focal length information of the imaging device 1001 are acquired as imaging device information. The acquired imaging device information is stored in the memory 1022. FIG. 4 is a schematic diagram showing a part of information held in the memory 1022. For a certain captured image, the angle sensor output and the focal length are stored as imaging device information.

The attitude information of the imaging apparatus 1001 is expressed by the following 3 × 3 matrix Rlight using the output of the angle sensor 1020.

  This 3 × 3 matrix Rlight representing the posture information of the imaging apparatus 1001 is referred to as a camera posture matrix. Here, (α, β, γ) is a value in the roll, pitch, and yaw angle representation of the sensor output attached to the camera, and is represented by the amount of movement from a certain reference point. The roll, pitch, and yaw angle representation, as shown in FIG. 5, refers to any rotation by rotating the rotation around the z axis, then the pitch around the new y axis, and finally the new x This is represented by a three-stage rotation of yaw, which is rotation around an axis.

Rx (α), Ry (β), and Rz (γ) are matrices for converting the roll, pitch, and yaw angles into x-axis rotation, y-axis rotation, and z-axis rotation, and are expressed by the following equations.

  Such a method for acquiring camera posture information from an angle sensor or an angular velocity sensor attached to the camera may use an existing method (for example, “Takayuki Okaya,“ 3 by fusion of a mechanical sensor and an image sensor). Dimensional shape restoration ", Information Processing Society of Japan, 2005-CVIM-147, pp. 123-130, 2005").

  When the imaging apparatus 1001 can zoom, the zoom information is also acquired as focal length information. Further, when the imaging apparatus 1001 has a fixed focus, the focal length information is also acquired. The focal length information can be acquired by performing camera calibration widely used in the field of image processing.

  The imaging device information acquisition unit 102 may obtain the optical axis direction, position information, and the like of the imaging device 1001 as imaging device information using an image captured by the wide-angle image imaging unit 101. For example, using the images taken at times t1 and t2, the relative positional relationship P1 and the camera posture matrix Rlight between times t1 and t2 are obtained using an optical flow.

  An optical flow is a vector connecting two temporally continuous points on a subject corresponding to a certain point on the subject, that is, a vector connecting the corresponding points, and there is a geometric constraint between the corresponding points and the camera movement. The formula holds. For this reason, when the corresponding point satisfies a certain condition, the motion of the camera can be calculated.

  For example, an 8-point method is known as a method for obtaining the relative positional relationship of imaging devices at different times from an optical flow (HC Longuet-Higgins, “A computer algorithm for reconstructing a scene from two projections”). , Nature, vol.293, pp.133-135, 1981). This method calculates camera motion from a set of eight or more corresponding points that are stationary between two images. In addition, since such a method for obtaining corresponding points between two images is a generally well-known method, detailed description thereof is omitted (for example, Carlo Tomasi and Takeo Kanade, “Detection and Tracking of Point Features”, Carnegie Mellon University Technical Report, CMU-CS-91-132, April 1991).

  The light source information estimation unit 103 uses the wide-angle image captured by the wide-angle image imaging unit 101 and the imaging device information acquired by the imaging device information acquisition unit 102 stored in the memory 1022, and the light source information for the current imaging device 1001. To get. This process will be described.

  Here, it is assumed that the direction of the light source is estimated. First, in a wide-angle image, a pixel having a sufficiently high luminance value is extracted as a pixel capturing a light source, that is, a light source pixel. In FIG. 3, in the captured wide-angle image 1006, the luminance value of the region 1007 where the light source 1005 is captured is very high. Thus, threshold processing is used to extract pixels having a luminance value higher than a predetermined threshold as light source pixels.

The light source direction is estimated from the light source pixels thus obtained. This process requires a relational expression between the pixel position (u, v) of the image pickup apparatus and the actual position (xf, yf) on the image pickup element called an image coordinate system. Considering the influence of lens distortion and the like, the relationship between the pixel position (u, v) and the actual position (xf, yf) can be obtained by the following equation.

  Where (Cx, Cy) is the pixel center position, s is the scale factor, (dx, dy) is the size [mm] of one pixel of the image sensor, Ncx is the number of image sensors in the x direction, and Nfx is the number of effective pixels in the x direction. , Κ1, and κ2 are distortion parameters indicating lens distortion.

Further, the relationship between the camera coordinate system (x, y, z) and the image coordinate system (xf, yf) shown in FIG. It is calculated by the following formula.
Here, f represents the focal length of the imaging device. That is, if the camera parameters (Cx, Cy), s, (dx, dy), Ncx, Nfx, f, κ1, and κ2 are known, the pixel position (u, v) is determined by (Expression 1) and (Expression 2). And the camera coordinate system (x, y, z) can be converted.

  Normally, Ncx and Nfx are known if an image sensor can be specified, and (Cx, Cy), s, (dx, dy), κ1, κ2, and f are known by performing so-called camera calibration (for example, Roger Y. Tsai, “An Efficient and Accurate Camera Calibration Technique for 3D Machine Vision”, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Miami Beach, FL, 1986, pp. 364-374, 1986). These parameters do not change even if the position or orientation of the imaging device changes. Such parameters are called camera internal parameters.

  Therefore, camera calibration is performed before imaging, and camera internal parameters (Cx, Cy), s, (dx, dy), Ncx, Nfx, f, κ1, and κ2 are specified. As these values, those attached at the time of purchase of the imaging apparatus may be used. In addition, when the camera is not fixed focus but can be zoomed, the focal length f at each zoom may be obtained individually so that it can be selected when necessary. Then, the focal length f may be held together with the captured image.

Using the above information, the light source direction is estimated from the light source pixels. When the pixel position of the light source pixel is (ulight, vlight), the light source direction Llight can be expressed by the following equation.
However,
By the way, since Llight is expressed in the camera coordinate system when the light source pixel is imaged, it is expressed again in the current camera coordinate system Lnow. This can be expressed by the following equation.
Here, Rlight represents the camera posture matrix when the light source pixel is photographed, and Rnow represents the current camera posture matrix.

  By performing the above processing, the light source direction vector Lnow is estimated. In this way, the direction of the light source is estimated.

  Further, by using the movement of the imaging apparatus 1001, not only the direction of the light source but also a three-dimensional position may be obtained.

  FIG. 7 is a schematic diagram for explaining this process. In FIG. 7, 1001A and 1002A indicate light source direction vectors estimated as the imaging device at time t = t1, and 1001B and 1002B indicate light source direction vectors estimated as the imaging device at time t = t2. Here, if the relative positional relationship and orientation of the imaging device at times t1 and t2 are known, the light source should exist at the intersection where the light source vectors 1002A and 1002B are extended. That is, the three-dimensional position of the light source is obtained as follows.

The camera posture matrix of the imaging device, the three-dimensional position of the imaging device, and the estimated light source direction vector at time t1 are R1, P1, and L1, respectively, and the light source estimated as the camera posture matrix of the imaging device at time t2. The direction vectors are R2 and L2, respectively. However, it is assumed that the position of the imaging device is the origin O (0, 0, 0) at time t2. At this time, the light source position Plight satisfies the following equation.
However, s and m are arbitrary constants. If all the estimated values are correct and no noise exists, the light source position Plight can be obtained by solving simultaneous equations for s and m in (Equation 3) and (Equation 4). However, since there is usually an influence of noise, the light source position is obtained using the least square method.

First, consider the following function f (m, s).
Here, m and s satisfy the following relational expression.
That means
Therefore, the light source position Plight is obtained by solving (Equation 5) and (Equation 6) as simultaneous equations relating to m and s and substituting the obtained m and s into (Equation 3) or (Equation 4). In this way, the position of the light source is estimated.

  Note that, as described above, the relative three-dimensional position P1 of the imaging device at time t1 (relative positional relationship between the imaging devices at time t1 and t2) is calculated from the image captured by the wide-angle image capturing unit 101, as described above. It is determined by using a flow.

  Further, the luminance and color of the light source can be obtained by obtaining the luminance value and RGB value of the light source pixel. Moreover, you may make it detect the spectrum of a light source by acquiring an image with a multispectral camera. By acquiring the spectrum of the light source in this way, it is known that an image with high color reproducibility can be synthesized in higher resolution and augmented reality described later (for example, “Toshiro Uchiyama, Masaru Tsuchida, Masahiro Yamaguchi , Hideaki Haneishi, Nagaaki Oyama, “Light Source Environment Measurement and Spectrum-Based Image Generation Using Multispectral Imaging”, IEICE Technical Report PRMU2005-138, pp.7-12, 2006 ”).

  The light source information acquisition unit 103 may acquire illuminance information of the light source as the light source information. This may be done by using an illuminometer that matches the image pickup apparatus 1001 in the optical axis direction. As the illuminance meter, a photocell illuminance meter or the like that reads a photocurrent generated by incident light by connecting a microampere meter to the photovoltaic cell may be used.

  The image enhancement unit 104 increases the resolution of the image captured by the wide-angle image capturing unit 101 using the light source information estimated by the light source information estimation unit 103.

  When a wide-angle image is captured by an element such as a CCD or CMOS, an image with a lower resolution is obtained than when captured by a narrow-angle lens. Therefore, in the image capturing apparatus according to the present embodiment, the light source information estimated by the light source information estimation unit 103 is used to improve the resolution of the subject.

  The image resolution increasing unit 104 will be described in detail. Higher resolution of images is known as digital zoom. A method that uses interpolation, a method that uses multiple images to estimate a camera degradation model (for example, “Masayuki Tanaka, Masatoshi Okutomi,“ Reconstruction Super "Resolution processing acceleration algorithm and its accuracy evaluation", IEICE Transactions D-II, vol.J88-D-II, No.11, pp.2200-2209, 2005, etc.), using learning A method or the like may be used. Here, the resolution enhancement processing of an image using learning will be described.

  First, the concept of image resolution enhancement processing using learning will be described. The image resolution increasing unit 104 uses the following four pieces of input information.

○ Diffuse reflection image of the subject ○ Specular reflection image of the subject ○ Three-dimensional shape information of the subject ○ Light source position / color / illuminance

  Here, the diffuse reflection image is an image obtained by imaging only the diffuse reflection component that is a matte reflection component in the input image. Similarly, the specular reflection image is a specular surface that is a metric in the input image. Only the reflection component is imaged. Here, the diffuse reflection component is a reflection that occurs on the surface of a non-glossy object and is a component that is uniformly scattered in all directions. On the other hand, the specular reflection component is a component that strongly reflects in the direction opposite to the incident light with respect to the normal, like reflection on a mirror surface. Assuming a dichroic reflection model, the brightness of an object is expressed as the sum of a diffuse reflection component and a specular reflection component. As will be described later, the specular reflection image and the diffuse reflection image can be acquired by, for example, imaging the subject while rotating the polarization filter.

  FIG. 8A shows an image obtained by imaging an object (tumbler) irradiated with a light source with an imaging device. It can be seen that a specular reflection appears at the top of the figure. On the other hand, FIGS. 8B and 8C show the result of separating the image of FIG. 8A into a diffuse reflection image and a specular reflection image by a method described later. The diffuse reflection image is removed from the light and the surface texture information is clear, but the stereoscopic effect is lost. On the other hand, in the specular reflection image, fine shape information appears clearly, but on the contrary, texture information is lost. In other words, the input image is a superposition of two images including completely different information. By separating the image into a diffuse reflection image and a specular reflection image and processing them separately, a finer and higher resolution processing can be performed.

  A learning-based method is used for the high resolution processing. The learning base is to prepare low-resolution and high-resolution image sets in advance and learn the correspondence. At this time, in order to make the high resolution processing work even in an image other than the prepared image, the feature amount extracted from the image is learned instead of learning the image itself.

  FIG. 9 is a block diagram illustrating a configuration example of the image resolution increasing unit 104. 9 increases the resolution of the wide-angle image captured by the wide-angle image capturing unit 101 and stored in the memory 1022 using the light source information estimated by the light source information estimation unit 103. In addition, the shape information acquisition unit 204 acquires normal information or three-dimensional position information on the surface of the subject as shape information. The shape information acquired by the shape information acquisition unit 204 is used for the image high-resolution processing in the image high-resolution unit 104.

  The image resolution increasing unit 104 separates the given wide-angle image into a diffuse reflection component and a specular reflection component, and individually increases the resolution of the separated diffuse reflection component and specular reflection component. Hereinafter, each processing will be described.

  The diffuse reflection / specular reflection separation unit 202 separates the given wide-angle image into a diffuse reflection component and a specular reflection component.

First, the reflection characteristics of an object will be described. Assuming a dichroic reflection model, the brightness of an object is expressed by the following equation as the sum of a diffuse reflection component and a specular reflection component.
Here, I is the luminance value of the subject imaged by the imaging apparatus, Ia is the ambient light component, Id is the diffuse reflection component, and Is is the specular reflection component. Here, the ambient light component is indirect light in which light from a light source is scattered by an object or the like. It is scattered throughout the space and gives a slight brightness to shadows where direct light does not reach. For this reason, it is usually handled as noise.

  Considering that the ambient light component is sufficiently small and can be ignored as noise, the image can be separated into a diffuse reflection component and a specular reflection component. As described above, the diffuse reflection component depends on the texture information, whereas the specular reflection image depends on fine shape information, and these components exhibit very different characteristics. Therefore, in order to increase the resolution of the image, the input image is separated into a diffuse reflection image and a specular reflection image, and the resolution of each image is increased by different methods, so that a very high resolution high resolution image is obtained. Can be obtained. For this reason, it is necessary to first separate the diffuse reflection image and the specular reflection image.

Various separation methods have been conventionally proposed. For example,
A method using a polarizing filter utilizing the difference in the degree of polarization between specular reflection and diffuse reflection (for example, Japanese Patent No. 3459981).
A method of separating a specular reflection region by rotating an object and using a multispectral camera (for example, Japanese Patent Application Laid-Open No. 2003-85531).
・ Uses images of objects illuminated from various directions, synthesizes a linearized image that is an image in an ideal state where no specular reflection occurs, and uses the linearized image to create specular reflection and shadow areas. Separation methods (for example, "Ishii Ikunori, Fukui Kotaro, Mukakawa Yasuhiro, Takenaga Takeshi," Linearization of images based on optical phenomenon classification ", IPSJ Journal, vol.44, no. SIG5 (CVIM6 ), Pp.11-21, 2003 ").
and so on.

  Here, a method using a polarizing filter is used. FIG. 10 shows a wearable camera 1000 equipped with the image processing apparatus according to this embodiment. As shown in FIG. 10, the imaging apparatus 1001 is provided with a linear polarization filter 1009A having a rotation mechanism (not shown). In addition, an illumination device 1005 to which the linear polarization filter 1009B is attached is provided.

Here, regarding the subject illuminated by the illumination device 1005 to which the linear polarization filter 1009B is attached, the imaging device 1001 captures a plurality of images while rotating the linear polarization filter 1009A by a rotation mechanism. Here, focusing on the fact that the illumination is linearly polarized, the reflected light intensity changes as shown in FIG. 11 with respect to the rotation angle ψ of the polarizing filter 1009A. Here, assuming that the diffuse reflection component of the reflected light is Id and the specular reflection component is Is, the maximum value Imax and the minimum value Imin of the reflected light luminance are expressed by the following equations.
That is, the diffuse reflection component Id and the specular reflection component Is of the reflected light can be obtained from the following equations.

  FIG. 12 shows the flow of this processing. First, the polarizing filter 1009A is rotated by the rotation mechanism (step S301), an image is taken and held in the memory (step S302). Next, it is confirmed whether or not a predetermined number of images held in the memory have been captured (step S303). At this time, when a sufficient number of images for detecting the minimum value and the maximum value of the reflected light luminance has not been captured yet (No in step S303), the polarizing filter is rotated again (step S301), and the imaging is repeated. On the other hand, when a sufficient number of images have been captured (Yes in step S303), the minimum and maximum values of reflected light luminance are detected using the captured image data (step S304), (Equation 8) And (Equation 9) are used to separate the diffuse reflection component and the specular reflection component (step S305). In this process, a minimum value and a maximum value may be obtained for each pixel from a plurality of images, but here, fitting of a sin function is used. This process will be described.

The reflected light luminance I with respect to the polarization filter angle ψ shown in FIG. 11 can be approximated by a sin function as follows.
Here, A, B, and C are constants. From (Expression 8) and (Expression 9)
That is, the diffuse reflection component and the specular reflection component can be separated by obtaining A, B, and C of (Expression 10) from the captured image.

By the way, (Equation 10) can be developed as follows.
However,
That is, the diffuse reflection component and the specular reflection component can be separated by obtaining A, B, and C that minimize the following evaluation formula.
Here, Ii indicates the reflected light intensity at the polarizing filter angle ψi. Here, when the least square method is used, each parameter is estimated as follows.
As described above, the diffuse reflection component and the specular reflection component are separated by using (Expression 11) to (Expression 19). In this case, since there are three unknown parameters, it is sufficient to capture at least three images in which the rotation angle of the polarization filter is changed.

  For this reason, instead of providing the rotation mechanism of the linear polarization filter 1009A, imaging devices having different polarization directions for each pixel may be used. FIG. 13 schematically shows a pixel of such an imaging apparatus. Here, 1010 indicates each pixel, and a straight line in each pixel indicates the polarization direction. That is, this imaging apparatus has pixels having four types of polarization directions of 0 °, 45 °, 90 °, and 135 °. Then, by treating four types of pixels as one pixel like a Bayer array as indicated by a thick line 1010 in FIG. 13, images with four different polarization directions can be simultaneously captured. Such an imaging apparatus may use, for example, a photonic crystal device.

  Further, as the illumination device 1005, polarized illumination such as a liquid crystal display may be used. For example, a liquid crystal display mounted on the wearable camera as an interface may be used.

  Of course, instead of rotating the polarizing filter 1009A of the imaging device 1001, the polarizing filter 1009B of the illumination device 1005 may be rotated. Further, instead of installing a polarizing filter on both the imaging device 1001 and the illumination device 1005, it may be installed only on one side such as the imaging device side, and the diffuse reflection component and the specular reflection component may be separated using independent component analysis. It does not matter (see, for example, Japanese Patent No. 3459981).

  The shape information acquisition unit 204 acquires surface normal information that is shape information of the subject or three-dimensional position information of the subject. As means for acquiring the shape information of the subject, for example, an existing method such as a slit light projection method, a pattern light projection method, or a laser radar method may be used.

  Of course, acquisition of shape information is not limited to these methods. For example, stereo vision using multiple cameras, motion stereo method using camera movement, illuminance difference stereo method using images taken while changing the position of the light source, and using subjects such as millimeter waves and ultrasound A method for measuring the distance of a transparent object, and a method using the polarization characteristics of reflected light (for example, US Pat. No. 5,028,138 and “Daisuke Miyazaki, Katsushi Ikeuchi,” Surface shape of transparent object by polarization ray tracing method) The estimation method of "The Institute of Electronics, Information and Communication Engineers Journal D-II, vol. J88-D-II, No. 8, pp. 1432-1439, 2005") may be used. Here, the illuminance difference stereo method will be described.

The illuminance difference stereo method is a method for estimating the normal direction and reflectance of a subject using three or more images having different light source directions. For example, “H. Hayakawa,“ Photometric Stereo under a light source with arbitrary motion ”, Journal of the Optical Society of America A, vol.11, pp.3079-89, 1994” reflects more than 6 reflections on the image. By acquiring points with the same rate as known information and using them as constraint conditions, the following parameters are estimated while the position information of the light source is unknown.
-Subject information: Normal direction and reflectance of each point on the image-Light source information: Light source direction and illuminance at the observation point of the subject

  Here, the illuminance difference stereo method using only the diffuse reflection image separated by the diffuse reflection / specular reflection separation method described above is performed. Originally, since this method assumes that the subject is completely diffusely reflected, a large error occurs in a subject where specular reflection exists. However, the estimation error due to the presence of specular reflection can be eliminated by using only the separated diffuse reflection image. Of course, as will be described later, the process may be performed with a diffuse reflection image from which the shadow area has been removed by the shadow removal unit 205.

Diffuse reflection images with different light source directions are represented by a luminance matrix Id as follows.
Here, iff (p) indicates the luminance value at the pixel p of the diffuse reflection image in the light source direction f. Further, the number of pixels of the image is P pixels, and the number of images taken in different light source directions is F. By the way, from the Lambertian model, the luminance value of the diffuse reflection image can be expressed as follows.
Where ρp is the reflectance (albedo) of the pixel p, np is the normal vector of the pixel p, tf is the incident illuminance of the light source f, and Lf is the direction vector of the light source f.
The following expressions are derived from (Expression 20) and (Expression 21).
However,
Here, R is a surface reflection matrix, N is a surface normal matrix, L is a light source direction matrix, T is a light source intensity matrix, S is a surface matrix, and M is a light source matrix.

Here, using singular value decomposition, (Equation 22) can be expanded as follows.
However,
And E represents a unit matrix. U ′ is a P × 3 matrix, U ″ is a P × (F-3) matrix, Σ ′ is a 3 × 3 matrix, Σ ″ is an (F-3) × (F-3) matrix, and V ′ is 3. The × F matrix, V ″ is an (F-3) × F matrix. Here, U ″, V ″ are considered to be orthogonal bases of signal components U ′ and V ′, that is, noise components. Here, using singular value decomposition, (Equation 24) can be modified as follows.

That is, by solving (Equation 25), shape information and light source information can be obtained simultaneously, but the following 3 × 3 matrix A indefiniteness remains.
Here, A is an arbitrary 3 × 3 matrix. In order to acquire shape information and light source information, it is necessary to obtain this matrix A. For example, it is sufficient that the reflectance is known to be equal at six or more points on the screen. For example, if the reflectance of any 6 points k1 to k6 is equal,
From (Expression 23), (Expression 26) and (Expression 28),
further,
Then, (Equation 29) is as follows.
Here, since the matrix B is a symmetric matrix according to (Equation 30), the unknown number of the matrix B is 6. That is, if it is known that the reflectance is equal at six or more points on the screen, (Equation 31) can be solved.

  If the matrix B is known, the matrix A can be solved by using singular value decomposition in (Equation 30).

  Furthermore, shape information and light source information are acquired from (Expression 26) and (Expression 27).

As described above, the following information can be obtained by capturing three or more images while changing the light source direction in a subject in which six or more pixels having the same reflectance are known.
Subject information: Normal direction vector and reflectance of each point on the imageLight source information: Light source direction vector and radiance at the observation point of the subject

  However, the reflectance of the subject obtained by the above processing and the radiance of the light source are relative, and in order to obtain the absolute value, the reflectance is known at six or more points on the screen. Different known information is required.

  If the positional relationship between the light source and the imaging device is known, the distance between the imaging device and the subject or the three-dimensional position may be obtained. This will be described with reference to the drawings.

  FIG. 14 is a schematic diagram for explaining this process. In FIG. 14, 1001 is an imaging device, 1005A and 1005B are light sources, 1012 is a subject observation point O, 1002A and 1002B are light source directions of each light source at the subject observation point O, and 1013 is an imaging measure at the subject observation point O. The line-of-sight direction is shown.

  First, since the positional relationship between the light source and the imaging device is known, the three-dimensional positional relationships La and Lb between the imaging device 1001 and the light sources 1005A and 1005B are known. If the imaging apparatus 1001 is calibrated, the line-of-sight direction 1013 of the imaging apparatus 1001 is also known. Therefore, the observation point O1012 of the subject exists on the line-of-sight direction 1013. Further, the light source directions 1002A and 1002B of the respective light sources at the observation point O of the subject are known by the above-described illuminance difference stereo method. If the distance Lv between the imaging device 1001 and the observation point O1012 is positive (Lv> 0), there is only one observation point O that satisfies such a positional relationship. Therefore, the position of the observation point O1012 is known, and the distance Lv between the imaging device 1001 and the observation point O1012 is obtained.

  For example, when a light source is installed in the imaging device like a flash of a digital camera, the positional relationship between the light source and the imaging device can be obtained from design information.

  In addition, the shape information acquisition unit 204 may acquire the surface normal direction of the subject using the polarization characteristics of the reflected light. This process will be described with reference to FIG.

  In FIG. 15, reference numeral 1001 denotes an imaging device, 1005 denotes a light source, 1012 denotes an observation point O, 1009 denotes a linear polarization filter having a rotation mechanism (not shown), and 1014 denotes a normal direction. When imaging is performed while rotating the polarizing filter 1009 by a rotating mechanism in a state where natural light is irradiated as a light source, the reflected light intensity is a sin function with a period π as shown in FIG.

  Here, the angles ψmax and ψmin of the polarizing filter for measuring the maximum value Imax and the minimum value Imin of the reflected light intensity are considered. When a plane including the imaging device 1001, the light source 1005, and the observation point O1012 is an incident surface, ψmax is a direction in which the polarization direction of the polarization filter 1009 is perpendicular to the incidence surface, and ψmin is a direction in which the polarization direction of the polarization filter 1009 is incident. It is known that the direction is parallel to the direction.

As described above, the reflected light component having the polarization characteristic is the specular reflection component reflected from the surface of the observation point O, and the non-polarized component is the diffuse reflection component. From this, the observation point O where the intensity difference between the maximum value Imax and the minimum value Imin of the reflected light intensity occurs is an observation point having a strong specular reflection component, that is, light is specularly reflected (normal direction 1014 of the observation point O). Are the bisector directions of the light source direction from the observation point O and the imaging device direction from the observation point O). Therefore, the normal direction 1014 also exists in the incident surface. Therefore, by estimating ψmax or ψmin, it can be estimated that the normal direction 1014 exists in the following plane.
A plane that passes through the imaging device 1001 and includes the polarization direction ψmin (or the vertical direction of ψmax) of the polarizing filter 1009

  Here, ψmax or ψmin is estimated by performing the above-described sin function fitting process.

  Further, two different planes including the normal direction 1014 can be estimated by changing the position of the imaging device 1001 and performing similar processing. The normal direction 1014 is estimated by obtaining the intersection line between the two estimated planes. At this time, it is necessary to estimate the movement amount of the imaging apparatus 1001. This may be performed by using the above-described 8-point method or the like.

  Of course, similarly to the diffuse reflection / specular reflection separation unit 202, an imaging device having a different polarization direction for each pixel may be used.

  Of course, instead of changing the position of the imaging device 1001, a plurality of imaging devices may be installed to obtain the normal direction 1014.

  As described above, the normal information of the surface is acquired by the illuminance difference stereo method and the method using the polarization characteristic. On the other hand, in a method such as slit light projection or stereo vision, three-dimensional position information of the subject is acquired. The normal information on the surface of the subject is tilt information in a minute space of the three-dimensional position information of the subject, and both are shape information of the subject.

  Through the above processing, the shape information acquisition unit 204 acquires surface normal information or object three-dimensional position information, which is object shape information.

The following information is acquired by the above processing.
○ Diffuse reflection image of subject ○ Specular reflection image of subject ○ Three-dimensional shape information of subject ○ Light source position / illuminance

  The shadow removal unit 205 estimates a shadow area in the image and performs a shadow removal process. Various methods have been proposed for such shadow removal and shadow area estimation processing. For example, if a shadow area has a low luminance value, and a pixel whose luminance value is equal to or less than a threshold value is estimated as a shadow area, Good.

  Further, when the 3D shape information is acquired by the shape information acquisition unit 204, ray tracing, which is a rendering method widely used in the field of Computer-Graphics, may be used. Rendering is performed by calculating environment coordinate data such as object coordinate data, light source, and viewpoint position, while ray tracing is performed by tracing the rays that reach the viewpoint in reverse. Therefore, it is possible to calculate how much shadow is generated at which place by using ray tracing.

  Next, the diffuse reflection image and the specular reflection image separated by the diffuse reflection / specular reflection separation unit 202 are each increased in resolution by different methods. First, processing of the diffuse reflection image will be described.

  The albedo estimation unit 206 estimates the albedo of the subject using the diffuse reflection image separated by the diffuse reflection / specular reflection separation unit 202. Since albedo is not affected by light source information, processing that is robust to light source fluctuations can be realized by performing processing using an albedo image.

This process will be described. From (Equation 21), the following relationship holds for the diffuse reflection component.
Here, θi represents an angle formed by the normal direction vector of the subject and the light source direction vector. Here, the angle θi is known by the light source information acquisition unit 103 and the shape information acquisition unit 204. As will be described later, since the incident illuminance tf of the light source can also be estimated, the albedo rp of the subject is obtained from (Equation 32).

  At this time, when cos θi has a value of 0 or less, that is, when it is an attached shadow, (Equation 32) has no meaning because the albedo becomes negative or division by 0. However, since such a pixel is removed by the shadow removal unit 205 described above, no problem occurs.

Of course, instead of obtaining the albedo of the subject, a pseudo albedo obtained by multiplying the albedo by the radiance of the light source may be obtained and used according to the following equation.

  Next, a description will be given of increasing the resolution of the albedo image thus obtained.

  The albedo resolution increasing unit 207 increases the resolution of the albedo image estimated by the albedo estimation unit 206. This process will be described in detail.

  As described above, an albedo image is an image that expresses a reflectance characteristic unique to a subject that does not depend on an optical phenomenon such as specular reflection or shading of light. The high resolution in the present embodiment is based on subject prior learning because subject information is essential. Here, high resolution based on texton (texture feature amount of an image) is used.

  FIG. 17 is a diagram showing the concept of high resolution based on textons. The low-resolution image LR (number of pixels N × N) input at the time of execution is interpolated and enlarged to M × M times so that the number of pixels matches the target number of pixels. This pixel number MN × MN image is referred to as an exLR image. In the exLR image, the high frequency component of the image is lost and the image becomes blurred. Sharpening this blurred image is nothing but high resolution.

  Next, the luminance value of the exLR image is converted for each pixel into a T-dimensional texton based on the multi-resolution by multi-resolution conversion WT. Processing such as wavelet transformation or pyramid structure decomposition is used for this transformation. As a result, a total of MN × MN T-dimensional texton vectors are generated for each pixel of the exLR image. Next, in order to improve versatility, clustering is performed on the texton vectors, and L input representative texton vectors are selected and generated. These L texton vectors are converted from previously learned database information to generate T-dimensional high-resolution texton vectors. For this conversion, table reading and linear and nonlinear conversion in a T-dimensional multidimensional feature vector space are used. The high-resolution texton vector is returned to the image luminance value by inverse transform IWT such as inverse wavelet transform or pyramid structure reconstruction, and the high-resolution image HR is completed.

  In this process, a great deal of time is required for the search and the table read process in the clustering process of MN × MN T-dimensional texton vectors, and it is difficult to cope with high speed such as moving images. Therefore, 1) clustering processing is performed on the LR image. 2) Change table reading to linear matrix conversion. Added the improvement. In this processing, the fact that one pixel of the LR image corresponds to a cell of M × M pixels of the HR image is used to perform a linear matrix conversion from T dimension to T dimension for each cell, and the space inside the cell Continuity can be maintained. The linear matrix to be used is optimally selected from the result of clustering. When cell boundary discontinuity becomes a problem, processing such as partially overlapping blocks of matrix processing units may be added.

  FIG. 18 is a diagram schematically showing the improved points. The LR image is subjected to WT transformation to obtain L (here, L = 3) representative feature vectors in the T-dimensional feature amount space. Each feature vector is accompanied by a different linear matrix. What saved this state is nothing but a high resolution database.

  The details of the image processing method will be described below using an example in which N = 32, M = 4, that is, a resolution of 4 × 4 times is applied to a 32 × 32 pixel low resolution image. The albedo image is an (RGB) color image, but the color image is converted from (RGB) to luminance color difference (YCrCB) and handled as an independent color component image. Normally, at about 2x2 magnification, there is no sense of incongruity even if the color component is added as it is with a low-resolution color component with a luminance Y component only, but at 4x4 or higher, it is essential to increase the resolution of the color signal. Therefore, each component is handled in the same manner. Hereinafter, processing of only one component image of a color image will be described.

(During learning)
FIG. 19 is a PAD diagram illustrating the flow of the learning process, and FIG. 20 is a diagram illustrating the relationship between the processing target pixel and the processing target cell of the image processed in the learning process. Hereinafter, description will be made using FIGS. 19 and 20 alternately.

  First, in S311 to S313, a low resolution image LR image, a high resolution image HR image, and an enlarged image exLR image of a low resolution image are input. These images are all generated from HR, and are set to have no pixel shift in imaging. Bicubic interpolation is used to generate an exLR image from an LR image. In FIG. 20, a high-resolution image HR (number of pixels 128 × 128), a low-resolution LR image (number of pixels 32 × 32), and an exLR image (number of pixels 128 × 128) in which LR is matched with HR by the number of pixels. Three types of images are prepared.

  In S314, the LR image is converted into text. Specifically, a two-dimensional discrete stationary wavelet transform (SWT transformation) using a Haar basis is performed. Assuming that the SWT conversion has two layers (2-step), a six-dimensional LRW image (number of pixels 32 × 32 = 1024) is generated. Originally, a two-dimensional stepwise two-dimensional discrete stationary wavelet transform results in a seven-dimensional feature vector, but the LL component image of the lowest frequency is close to the average luminance information of the image, and only the remaining six components are used to store this. To do.

  In S315, a total of 1024 6-dimensional vectors of the textonized LRW images are clustered up to Cmax. Here, for example, clustering is performed to Cmax = 512 using the K-means method. A set of 512 resulting texton vectors is referred to as cluster C. There is no problem if all 1024 textons are used without clustering.

  In S316, the LR pixel determined as the same cluster of the cluster C is determined. Specifically, the pixel value of the LR image is replaced with each texton number of cluster C.

  In step S317, while repeating the processing for all the textons in cluster C, the pixel cells of exLR and the pixel cells of the HR image corresponding to the texton are searched, and the corresponding cell numbers are stored. Since this search only needs to be performed for the number of pixels of the LR image, the search time is greatly reduced when the magnification is high.

  Here, correspondence between the pixels of the LR image and the pixel cells of the exLR image and the HR image will be described with reference to FIG. In FIG. 20, it is assumed that two pixels 2001 and 2002 are determined to be the same cluster of C (cluster number: Ci = 0) on the LR image. Then, it is considered that pixel cells such as 2003 and 2004 on the exLR image enlarged while maintaining the positional relationship as it is, and 2005 and 2006 on the HR image correspond to each other. Stored as having the corresponding texton. The number of pixels contained in the pixel cell is equal to the enlargement ratio 4 × 4 = 16.

  Next, in step S318, the pixel cell group is converted into a texton with a pair of exLR image and HR image. Specifically, a two-dimensional discrete stationary wavelet transform is performed to generate an exLRW image and an HRW image.

  In S319 and S320, texton pairs obtained from the HRW image and the exLRW image are accumulated in the form of a matrix. Each format is a 6 × Data_num matrix. Here, Data_num is (number of pixels of one cell) × (number of searched cells), and in the above example of Ci = 0, two cells are searched and 16 × 2 = 32.

In S321, a 6 × 6 matrix M is calculated by the least square method from a total of 2 × 4 × 4 = 128 feature vectors belonging to both of them, and in S322, it is combined with the cluster number K = 0 and the database CMat. Store and store in (K). In the least square method in S322, first, the exLR and HR texton matrices accumulated in S319 and S320 are respectively Lf and Hf (size: 6 × Data_num), and the matrix to be obtained is M (6 × 6) as follows. Can be executed.

  Next, the same processing is repeated for the cluster number K = 1 until K = 511. That is, CMat is a group of 6 × 6 transformation matrices defined for each cluster number.

  Finally, in S323 and S324, the used cluster C and the learned transformation matrix CMat are output. The cluster C thus obtained and the learned transformation matrix CMat are accumulated in the albedo DB 208.

  FIG. 21 is a diagram showing processing of a two-dimensional discrete stationary wavelet transform. In the normal wavelet transform, the image is reduced each time the decomposition hierarchy proceeds with the same filter bank configuration, but in the two-dimensional discrete steady wavelet transform, the transformed image size does not change even if the decomposition hierarchy proceeds. The multi-resolution analysis is performed by upsampling (↑) the two types of filters of the scaling function F and the wavelet function G and increasing them by a power of 2. In the Haar basis, specific numerical values of F and G and the state of upsampling are as shown in Table 1.

  When the cA image, which is an LL component, is advanced one layer and subjected to wavelet decomposition, four types of images are generated as shown in FIG. 21 by alternately convolving the F and G filters one-dimensionally. 1) F in the row direction and F: cA image in the column direction (LL component) 2) F in the row direction and G: cDh image in the column direction (LH component) 3) F: cDv image in the G direction in the row direction and the column direction (HL component) 4) G: cDd image (HH component) in the G direction in the row direction and in the column direction.

FIG. 22 is an example of an image result when the two-dimensional discrete stationary wavelet transform is performed on the test image. A texton vector is a series of values corresponding to each pixel of 1-STEP and 2-STEP converted images of these wavelets.
This is a 7-dimensional vector. However, except for cA2, which is a 2-step LL component, high-resolution conversion is performed using only the 6-dimensional vector portion, and the cA2 component is stored.

  Here, the number of steps of the wavelet transform is 2-STEP in both S314 and S318. As the number of steps increases, the rough features of the image can be expressed in texton. In the present invention, the number of steps is variable, but in the case of S314 for clustering LR images, 1-STEP may not be sufficient information as the surrounding pixel status, and is therefore 2-STEP. On the other hand, in the case of S318 for generating a texton for increasing the resolution of an exLR image, it is experimentally possible that an image of 3-STEP is better than 2-STEP at a magnification of 8 × 8, for example. Has been confirmed. For this reason, it is desirable to determine the number of steps in relation to the magnification.

(When executing high-resolution processing)
FIG. 23 is a PAD diagram showing a flow of processing at the time of execution, and FIG. 24 is a diagram showing a relationship with pixel cells of processing at the time of execution.

  First, in S331 and S332, an LR image and an exLR image obtained by enlarging the LR image are input. As in the learning, the number of pixels of the LR image = 32 × 32 and the number of pixels of the exLR image = 128 × 128. Here, the exLR image generation method is based on the bicubic method, similar to the method of generating the exLR image of the learning image in S313 of FIG.

  Next, in S333 and S334, the cluster C obtained at the time of learning, the transformation matrix CMat, and the albedo DB 208 are read and input.

  In S335, the LR image is converted into text. Specifically, as shown in FIG. 24, a two-dimensional discrete stationary wavelet transform (SWT transformation) using a Haar basis is performed. Assuming that the SWT conversion has two layers (2-step), a six-dimensional LRW image (number of pixels 32 × 32 = 1024) is generated. Originally, a two-dimensional stepwise two-dimensional discrete stationary wavelet transform results in a seven-dimensional feature vector, but the lowest frequency LL component image is close to the average luminance information of the image, and only the remaining six components are used to store this. To do.

  Next, in S336, for each texton, the texton vector of the shortest distance in the cluster C (Cmax textons) is searched to obtain the texton number (Ci). This corresponds to assigning texton numbers C0, C1,..., Cn to the pixels 2011, 2012,..., 2013 on one line of the LR image in FIG.

  Next, the process proceeds to S337, but after this, it becomes a repetitive process of processing each cell of the HR image in the scanning line order. Specifically, in FIG. 24, when the exLR image cells 2014, 2015,..., 2016 are processed, the corresponding HR image cells 2023, 2024,.

  In S337, the corresponding cell region of the exLR image is converted into text. Specifically, a two-dimensional discrete stationary wavelet transform is performed to generate an exLRW image. Cells 2017, 2018,..., 2019 are generated.

  In S338, the conversion matrix CMat in the corresponding cell is determined by subtracting the conversion matrix CMat from the texton number. This process is performed as shown in FIG. The LRW image has already been assigned texton numbers such as pixel 2011 = C0, pixel 2012 = C1,..., Pixel 2013 = Cn. This is applied to cells 2017, 2018,..., 2019 of the exLRW image in which the positional relationship is stored, and in each cell, a separate 6 × 6 transformation matrix M is obtained from Mat with C0, C1,. You can choose.

In S339, the transformation matrix M is applied to each cell. This is for all textons LTi (i = 1-16) in the cell,
Should be implemented. By repeating these steps, cells 2020, 2021,..., 2022 of HRW images are generated from cells 2017, 2018,.

  Next, the 2-step LL component of the exLRW image is added to the 6-dimensional textons in the high-resolution cells to generate 7-dimensional textons.

  In S340, the 7-dimensional texton in each cell is converted into an image by inverse SWT conversion. The above is repeated for all cells of the exLR image.

  Inverse SWT (ISWT) conversion can be realized by the signal flow shown in FIG. It is almost the same expression as FIG. In normal wavelet inverse transformation, the filter bank configuration remains the same, and the image is enlarged each time the decomposition hierarchy advances. In contrast, in this inverse transform, the size of the transformed image remains unchanged even when the decomposition hierarchy progresses, and the two types of filters, scaling function F and wavelet function G1, are downsampled (↓) and shortened to a power of 2. As a result, multi-resolution analysis is performed. On the Haar basis, specific values of F and G1 and the state of downsampling are as shown in Table 2.

  As described above, the resolution of one component of the albedo image is increased. By performing this process on all the albedo images, a high-resolution albedo image is synthesized.

  At this time, normalization of the image may be performed so that processing can be performed even if the size, posture, orientation, etc. of the subject included in the albedo image changes. It is conceivable that the resolution enhancement processing using texton does not sufficiently exhibit the accuracy of resolution enhancement when the size and orientation of the albedo image are different from the learning data. Therefore, a plurality of sets of albedo images are prepared to solve this problem. In other words, an image obtained by rotating the albedo image by 30 degrees is synthesized, and the resolution is increased for all the images, so that the posture and orientation are changed. In this case, when searching for the shortest distance texton in step S336 of FIG. 23, which is a PAD diagram of the above-mentioned “when executing high resolution processing”, a plurality of LR images obtained from the rotated images are used. Each texton is searched for the shortest distance texton, and the nearest texton is searched to obtain the texton number (Ci).

  Further, in order to cope with the change in size, an albedo image in which the size of the image is changed may be synthesized.

  Also, with reference to the actual size, for example, an enlargement / reduction process may be performed so that an image of 5 cm × 5 cm always has 8 × 8 pixels, and a texton may be generated for the image. Since the size of the subject is already known by the shape information acquisition unit 204, the variation in size can be achieved by creating a texton with an image of the same size at both “learning” and “when executing high resolution processing”. It does not matter even if it corresponds to.

  Also, instead of rotating the albedo image at the time of executing the high resolution processing, the albedo image at the time of learning is rotated to create a plurality of sets of textons, and the cluster C and the learned transformation matrix CMat May be stored in the albedo DB 208.

  Further, it may be possible to estimate what the input subject is and to estimate the orientation of how the estimated subject is rotating. For such processing, a widely used image recognition technique may be used. This is because, for example, a tag such as RFID is set on the subject, the tag information is recognized to recognize what the subject is, and the shape information of the subject is estimated from the tag information, and an image or subject The posture may be estimated from the shape information (see, for example, JP-A-2005-346348).

  The diffusion image high resolution unit 209 synthesizes a high resolution diffusion image from the high resolution albedo image synthesized by the albedo high resolution unit 207. This process will be described.

  As described above, the albedo image is obtained by dividing the diffusion component image by the inner product of the light source vector and the normal direction vector of the subject. Therefore, by multiplying the albedo image by the inner product of the light source direction vector estimated by the light source information estimation unit 103 and the high-density normal direction vector of the subject obtained by the parameter resolution increasing unit 213 described later, a high resolution Composite diffusion images of When a plurality of light sources are estimated by the light source information estimation unit 103, a high-resolution diffusion image is synthesized for each light source, and the images are added together to synthesize a single high-resolution diffusion image. .

  Through the above processing, it is possible to synthesize a diffusion image with high resolution. Here, the high resolution processing is performed using the albedo image. However, instead of the albedo image, the diffusion image may be directly increased in resolution. In this case, the learning process may be performed using a diffusion image.

  Next, the process for increasing the resolution of the specular reflection image will be described. Here, the image is decomposed into parameters, and the density is increased for each parameter. This process will be described in order.

  The parameter estimation unit 210 represents the subject using the normal information of the subject acquired by the shape information acquisition unit 204, the diffuse reflection image and the specular reflection image separated by the diffuse reflection / specular reflection separation unit 202. Estimate the parameters. Here, a method using a Cook-Torrance model widely used in the field of Computer-Graphics will be described.

In the Cook-Torrance model, a specular reflection image is modeled as follows.

  Here, Ei is the incident illuminance, ρs, λ is the bidirectional reflectance of the specular reflection component at the wavelength λ, n is the normal direction vector of the subject, V is the line-of-sight vector, L is the light source direction vector, H is the line-of-sight vector and illumination. An intermediate vector of direction vectors, β represents an angle between the intermediate vector H and the normal direction vector n. Further, Fλ is a Fresnel coefficient which is a ratio of reflected light from the dielectric surface obtained from the Fresnel equation, D is a microfacet distribution function, and G is a geometric attenuation factor representing the influence of shading due to unevenness of the object surface. Further, nλ is the refractive index of the subject, m is a coefficient indicating the roughness of the subject surface, and Ij is the radiance of the incident light. Ks is a coefficient of the specular reflection component.

Furthermore, when the Lambertian model of (Expression 21) is used, (Expression 7) is developed as follows.
However,
Here, ρd is the reflectance (albedo) of the diffuse reflection component, dpx and dpy are the lengths of one pixel in the x direction and y direction, and r is the distance from the observation point O of the imaging device. Kd is a coefficient satisfying the following relational expression.
Sr is a constant for expressing the difference in luminance value between the diffuse reflection component and the specular reflection component, and indicates that the diffuse reflection component reflects energy from the subject in all directions. FIG. 26 is a schematic diagram for explaining the constant Sr. In FIG. 26, the diffuse reflection component energy reflected at the observation point O spreads in a hemispherical shape. Here, since the imaging apparatus 1001 is separated from the observation point O by r, the ratio Sr between the energy reaching one imaging element of the imaging apparatus and the total energy reflected at the observation point O is expressed by (Equation 44). .

  From the above, the parameter estimation unit 210 estimates parameters from (Expression 33) to (Expression 44).

  To summarize the above relational expressions, the known parameters for parameter estimation and the parameters to be estimated are as follows.

(Known parameters)
○ Ambient light component Ia
○ Diffuse reflection component Id
○ Specular reflection component Is
○ Normal direction vector n of the subject
○ Light source direction vector L
○ Gaze vector V
○ Intermediate vector H
The angle β between the intermediate vector H and the normal direction vector n.
○ Lengths dpx and dpy in the x direction and y direction of one pixel of the imaging device.
○ Distance r between imaging device and observation point O

(Parameters to be estimated)
○ Incident illuminance Ei
○ Specular reflection coefficient ks
○ Subject surface roughness m
○ Subject refractive index ηλ
Here, the coefficient kd of the diffuse reflection component and the reflectance (albedo) ρd of the diffuse reflection component are also unknown parameters. However, since only the parameter of the specular reflection component is estimated, no estimation process is performed here.

  FIG. 27 is a diagram showing a flow of processing of the parameter estimation unit 210. The process consists of the following two stages.

First, the incident illuminance Ei is obtained using the light source information (step S351). This uses the position information of the light source acquired by the light source information estimation unit 103, the distance information between the imaging device and the subject obtained by the shape information acquisition unit 204, and the light source illuminance obtained by the light source information estimation unit 103. This is obtained from the following equation.
Here, Ii is the incident illuminance of the light source 1005 measured by the illuminance meter 1015 installed in the imaging device 1001, R1 is the distance between the imaging device 1001 and the light source 1005, R2 is the distance between the light source 1005 and the observation point O, and θ1. Represents the angle formed between the normal 1014 at the observation point O and the light source direction 1002C, and θ2 represents the angle formed between the optical axis direction 1017 and the light source direction 1002A in the imaging apparatus 1001 (see FIG. 28). Here, when the size of the subject is considered to be sufficiently larger than the distance R2 between the light source 1005 and the observation point O, the distance R2 is equal at all the observation points O on the subject. Therefore, in (Equation 46), (R1 / R2) is a constant and there is no need to actually measure.

Next, unknown parameters m, ηλ, and ks are estimated using the simplex method (step S352). The simplex method is a method of optimizing functions by assigning variables to the vertices of figures called simplex and changing the size and shape of the simplex (Noboru Ohta, “Basics of Color Reproduction Optics”, pp.90-92, Corona). A simplex is a set of (n + 1) points in an n-dimensional space. Here, n is the number of unknowns to be estimated, and is “3” here. Therefore, the simplex is a tetrahedron. The vertex of the simplex is represented by a vector xi, and a new vector is defined as follows.
However,
Indicates xi that maximizes and minimizes the function f (xi), respectively.

Further, the three types of operations used in this method are defined as follows.
1. Mirror image:

2. Expansion:

3. Shrinkage:
Here, α (> 0), β (> 1), and γ (1>γ> 0) are coefficients.

The simplex method is based on the expectation that the function value in the mirror image becomes small by selecting the largest function value among the vertices of the simplex. If this expectation is correct, the minimum value of the function is obtained by repeating the same process. That is, while updating the parameter given as the initial value by three types of operations, the parameter update is repeated until the error from the target indicated by the evaluation function becomes less than the threshold. Here, m, ηλ, ks as parameters, and a specular reflection component image calculated from (Expression 33) represented by (Expression 52) as an evaluation function and the specular reflection obtained by the diffuse reflection / specular reflection separation unit 202. The difference ΔIs from the component image was used.
However, is (i, j) ′ and is (i, j) are the calculated specular image estimated value Is ′ and the specular reflection component image Is obtained by the diffuse reflection / specular reflection separation unit 202, respectively. The luminance value of (i, j), Ms (i, j), is a function that takes 1 when the pixel (i, j) has a specular reflection component and 0 otherwise.

  This process will be described in detail. FIG. 29 is a flowchart for explaining the flow of this process.

  First, 0 is substituted into counters n and k for storing the number of repetitive calculation updates, and initialization is performed (step S361). Here, the counter n is a counter that stores how many times the initial value is changed, and k is a counter that stores how many times the candidate parameter is updated by the simplex with respect to a certain initial value.

Next, random values are used to determine initial values of estimation parameter candidate parameters m ′, ηλ ′, and ks ′ (step S362). At this time, from the physical constraint conditions of each parameter, the generation range of the initial value was determined as follows.

  Next, the candidate parameter obtained in this way is substituted into (Expression 33) to obtain an estimated value Is ′ of the specular reflection image (step S363). Further, a difference ΔIs between the calculated estimated value Is ′ of the specular reflection image and the specular reflection component image obtained by the diffuse reflection / specular reflection separation unit 202 is obtained from (Equation 52), and this is calculated as an evaluation function of the simplex method. (Step S364). If ΔIs obtained in this way is sufficiently small (Yes in step S365), the parameter estimation is successful, and candidate parameters m ′, ηλ ′, and ks ′ are selected as the estimation parameters m, ηλ, and ks, and the process ends. On the other hand, if ΔIs is large (No in step S365), the candidate parameters are updated by the simplex method.

  Before updating candidate parameters, evaluate the number of updates. First, 1 is added to the counter k storing the number of updates (step S366), and the size of the counter k is determined (step S367). If the counter k is sufficiently large (No in step S367), the repetitive calculation is sufficiently performed, but since it has fallen to the local minimum, it is determined that the optimum value is not reached even if the update is repeated as it is. Change the value to get out of the local minimum. Therefore, 1 is added to the counter n and 0 is added to the counter k (step S371). Here, it is determined whether or not the value of the counter n is higher than the threshold value, and it is determined whether to continue the processing as it is or to terminate the processing because it cannot be processed (step S372). Here, when n is larger than the threshold value (No in step S372), it is determined that this image cannot be estimated, and the process ends. On the other hand, if n is smaller than the threshold value (Yes in step S372), the initial value is again selected from random numbers within the range of (Equation 53) (step S362), and the process is repeated. For example, 100 may be selected as the threshold for such k.

  On the other hand, when the counter k is equal to or smaller than the threshold value in step S367 (Yes in step S367), the candidate parameter is changed using (Equation 49) to (Equation 51) (step S368). This process will be described later.

Next, it is determined whether the candidate parameter thus transformed is meaningful as a solution (step S369). That is, by repeating the simplex method, there is a possibility that the deformed parameter has a physically meaningless value (for example, the roughness parameter m is a negative value, etc.), which is removed. For example, the following conditions may be given, and if this condition is satisfied, it may be determined as a meaningful parameter, and if not satisfied, it may be determined as a meaningless parameter.
These values can be obtained from the subject. For example, the refractive index ηλ is a value determined by the material of the subject. For example, it is known that the plastic is 1.5 to 1.7, and the glass is 1.5 to 1.9. Therefore, these values may be used. That is, when the subject is plastic, the refractive index ηλ may be 1.5 to 1.7.

  If the deformed parameter satisfies (Equation 54) (Yes in step S369), the candidate parameter is considered to be a meaningful value, so it is set as a new candidate parameter (step S370), and the updating process is repeated (step S370). S363). On the other hand, if the deformed parameter does not satisfy (Equation 54) (No in step S369), the updating process for the initial value is terminated, and the update is performed with the new initial value (step S371).

Here, the deformation process in step S368 will be described in detail. FIG. 30 is a flowchart showing the flow of this process. Here, the candidate parameters m ′, ηλ ′, and ks ′ are expressed as vectors, which are set as parameters x. That is,

  First, using (Equation 47) to (Equation 49), the parameter xr subjected to the mirror image operation is calculated, and the difference ΔIs (xr) from the specular reflection component image at xr is calculated according to (Equation 52) ( Step S381). Next, ΔIs (xr) obtained in this way is compared with ΔIs (xs) having the second worst evaluation function (step S382). If ΔIs (xr) is smaller than ΔIs (xs) (Yes in step S382), the evaluation value ΔIs (xr) subjected to the mirror image operation is compared with ΔIs (xl) having the best evaluation value at present (step). S383). If ΔIs (xr) is larger (No in step S383), xh having the worst evaluation value is changed to xr (step S384), and the process ends.

  On the other hand, when ΔIs (xr) is smaller than ΔIs (xl) (Yes in step S383), an expansion process is performed using (Equation 46), and the difference ΔIs between the parameter xe and the specular reflection component image at xe is performed. (xe) is calculated (step S385). Next, ΔIs (xe) thus obtained is compared with ΔIs (xr) obtained by mirror image operation (step S386). If ΔIs (xe) is smaller than ΔIs (xr) (Yes in step S386), xh having the worst evaluation value is changed to xe (step S387), and the process is terminated.

  On the other hand, if ΔIs (xe) is larger than ΔIs (xr) (No in step S386), xh having the worst evaluation value is changed to xr (step S387), and the process ends.

  If ΔIs (xr) is larger than ΔIs (xs) in step S382 (No in step S382), the evaluation value ΔIs (xr) subjected to the mirror image operation and ΔIs (xh) having the worst evaluation value are obtained. Compare (step S388). If ΔIs (xr) is smaller than ΔIs (xh) (Yes in step S388), xh having the worst evaluation value is changed to xr (step S389), and the contraction operation is performed using (Equation 50). The difference ΔIs (xc) between the parameter xc subjected to the above and the specular reflection component image at xc is calculated (step S390). On the other hand, if ΔIs (xr) is larger than ΔIs (xh) (No in step S388), the difference ΔIs () between the parameter xc subjected to the contraction operation without changing xh and the specular reflection component image at xc. xc) is calculated (step S390).

  Next, ΔIs (xc) thus obtained is compared with ΔIs (xh) having the worst evaluation value (step S391). Here, if ΔIs (xc) is smaller than ΔIs (xh) (Yes in step S391), xh having the worst evaluation value is changed to xc (step S392), and the process is terminated.

On the other hand, if ΔIs (xc) is larger than ΔIs (xh) (No in step S391), all candidate parameters xi (i = 1, 2, 3, 4) are changed according to the following formula, and the process is terminated. .

  By repeating the above process, the unknown parameters m, ηλ, and ks in the specular reflection image are estimated.

  Through the above processing, all unknown parameters can be estimated.

  The model used for parameter estimation does not need to be a Cook-Torrance model. For example, a Torrance-Sparrow model, a Phong model, a simple Torrance-Sparrow model (for example, “K. Ikeuchi and K. Sato,“ Determining reflectance properties of an object using range and brightness images ”, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.13, no.11, pp.1139-1153, 1991”).

  Further, the parameter estimation method does not need to be a simplex method, and for example, a general parameter estimation method such as a gradient method or a least square method may be used.

  Further, the above processing may be performed for each pixel, or region division may be performed and an equal parameter set may be estimated for each region. When processing is performed for each pixel, a sample in which a known parameter such as a normal direction vector n, a light source direction vector L, or a line-of-sight vector V of the subject is changed is acquired by moving the light source, the imaging device, or the subject. It is desirable. Further, when the processing is performed for each region, it is desirable to perform optimal parameter estimation by changing the region division so that the variation in parameters obtained for each region is reduced.

  The normal information densification unit 211 densifies the surface normal information acquired by the shape information acquisition unit 204. This is realized as follows.

  First, the normal information of the surface acquired by the shape information acquisition unit 204 is projected onto the image acquired by the image capturing unit 201, and the normal direction corresponding to each pixel in the image is obtained. Such processing is based on conventional camera calibration processing (for example, “Hiroki Unten, Katsushi Ikeuchi,“ Texturing method of 3D geometric model for virtualizing real objects ”, CVIM-149-34, pp.301 -316, 2005 ”).

  At this time, the normal vector is expressed in polar coordinates, and the values are θi and φi (see FIG. 31). Through the above processing, images of θ and φ that are normal components are produced. High-resolution normal information is estimated by increasing the resolution of the θ image and the φ image thus obtained by the same method as the albedo resolution increasing unit 207 described above. At this time, a learning process is performed before the densification process, and the cluster C and the learned transformation matrix CMat for the θ and φ components of the normal are accumulated in the normal DB 212.

  Further, it is desirable that the above processing is performed only on the area that is not removed as a shadow by the shadow removal unit 205. This is to prevent an error from occurring in the parameter estimation process due to the presence of a shadow.

  In the parameter estimation unit 210, a controllable light source installed in the vicinity of the imaging device may be used. The light source may be a digital camera flash. In this case, a flash image captured by irradiating the flash and a non-flash image captured without irradiating the flash may be captured continuously in time, and parameter estimation may be performed using the difference image. The positional relationship between the imaging device and the flash that is the light source is known, and the three-dimensional position, color, and intensity, which are the light source information of the flash, can be measured in advance. Further, since the imaging device and the flash are installed in a very close place, an image with little shadow can be taken. Therefore, the parameter can be estimated for most pixels in the image.

  Further, the parameter densification unit 213 densifies the parameter obtained by the parameter estimation unit 210. Here, simple linear interpolation is performed to increase the density of all parameters. Of course, a densification method using learning such as the albedo resolution enhancement unit 207 described above may be used.

  Further, the densification method may be switched for each parameter. For example, it is considered that the refractive index ηλ of the subject, which is an estimation parameter, does not change even when the density is increased. Therefore, the refractive index ηλ of the subject is increased in density by simple interpolation, and the diffuse reflection component coefficient kd, the specular reflection component coefficient ks, and the diffuse reflection component reflectivity (albedo) ρd are high using learning. The densification process may be performed.

  The specular reflection image high-resolution unit 214 uses the high-density normal information estimated by the normal-line information densification unit 211 and the parameters densified by the parameter high-resolution unit 214 to obtain a high-resolution specular surface. Composite reflection images. A high-resolution specular reflection image is synthesized by substituting high-density parameters into (Expression 33) to (Expression 41).

  Here, for example, a value estimated only for the incident illuminance Ei may be multiplied by a coefficient l (for example, l = 2) so that the luminance value becomes higher than the actual specular reflection image. This is to increase the texture of the subject by increasing the luminance value of the specular reflection image. Similarly, a specular reflection image that is more intense than actual may be synthesized by setting the roughness m of the subject surface to a value larger than the estimated value.

  The shadow generation unit 215 synthesizes a shadow image to be superimposed on the high-resolution diffuse reflection image and the high-resolution specular reflection image generated by the diffuse reflection image high-resolution unit 209 and the specular reflection image high-resolution unit 214. This may be done by using ray tracing used in the shadow removal unit 205 described above.

  Here, it is assumed that the image resolution increasing unit 104 has knowledge about the three-dimensional shape of the subject to be imaged. The shadow generation unit 215 acquires the three-dimensional shape data of the subject, and estimates the three-dimensional posture and the three-dimensional position of the subject from the appearance of the subject in the captured image. An example of estimating the 3D position and 3D posture from the appearance when the subject is the cornea of the human eye is "K.Nishino and SKNayar," The World in an Eye ", in Proc. Of Computer Vision and Pattern Recognition CVPR '04, vol.I, pp444-451, Jul., 2004. " Although there are limited subjects that can estimate the three-dimensional position and the three-dimensional posture from the appearance, the technique of the above document can be applied to the present invention as long as such a subject.

  If the three-dimensional posture and the three-dimensional position of the subject are estimated, normal information of the subject surface can be calculated at an arbitrary position on the subject. The above process is repeated in the captured image to calculate normal information on the subject surface. Furthermore, it is possible to increase the density of the three-dimensional shape of the subject by increasing the density of the normal information of the subject using the high density normal information estimated by the normal information density increasing unit 211. . A high-resolution shadow image is estimated by performing ray tracing using the high-density three-dimensional shape obtained in this way and the parameters densified by the parameter-enhancing unit 213.

  The rendering unit 216 includes a high-resolution diffuse reflection image combined by the diffuse reflection image high-resolution unit 209, a high-resolution specular reflection image combined by the specular reflection image high-resolution unit 214, and a shadow combined by the shadow generation unit 215. Synthesize images and synthesize high-resolution output images.

  As described above, the image processing apparatus according to the present embodiment can perform image resolution enhancement processing.

  In this example, only the specular reflection image is increased in resolution by using parameter estimation. However, it is also possible to perform parameter estimation in a diffuse reflection image to increase the resolution.

This process will be described. As described above, the unknown parameters of the diffuse reflection image are the following two parameters.
○ Coefficient kd of diffuse reflection component
○ Reflectance of diffuse reflection component (Albedo) ρd
Therefore, these parameters are estimated. FIG. 32 is a diagram showing a flow of parameter estimation processing of the diffuse reflection image. After the process of the specular reflection image parameter estimation unit 210 shown in FIG. 27, the following two steps are further performed.

First, kd is estimated from the following equation using (Equation 45) and ks obtained by parameter estimation of the specular reflection image (step S353).

Further, using (Equation 43), the reflectance (albedo) ρd of the diffuse reflection image is estimated from the following equation (step S354).

  Through the above processing, all unknown parameters can be estimated. It is only necessary to increase the resolution of the diffuse reflection image by densifying the parameters obtained in this way by the same method as the parameter densification unit 213.

  In addition, when the wide-angle image capturing unit 101 captures an image, it is desirable to capture the image without irradiating a flash. This is because when a subject that causes specular reflection such as a mirror is present in the field of view of the imaging apparatus 1001, the flash is reflected and is erroneously estimated as a light source pixel. For this reason, it is desirable to use an imaging device capable of imaging a wide dynamic range such as a cooled CCD camera or multiple exposure imaging.

  The wide-angle image capturing unit 101 may set the exposure time of the pixels of the image capturing apparatus 1001 to a different value for each region. Different exposure times can be set by changing the charge accumulation time when the imaging device 1001 is a CCD, for example.

  In FIG. 2, assuming that the optical axis direction 1017 of the wearable camera 1000 is the same direction as the front direction of the photographer 1003, the subject facing the photographer is likely to be imaged at the center of the lens. The subject facing the photographer 1003 is considered to be the subject that the photographer 1003 is interested in. For this reason, in the area corresponding to the center of the lens in which the subject is shown, it is effective to increase the exposure time of the pixels and take an image with high sensitivity. On the other hand, the light source may be sunlight or street lamps outdoors, or illumination laid on the ceiling indoors. For this reason, in the region corresponding to the lens peripheral portion, it is possible to reduce the possibility that the light source pixel having a high luminance value is saturated by making the exposure time of the pixel shorter than that of the lens central portion.

  Of course, when the optical axis direction 1017 of the wearable camera 1000 is different from the front direction of the photographer 1003, the exposure time of the pixels corresponding to the front direction of the photographer 1003 may be relatively increased. In order to detect the front direction of the imager 1003, for example, when the imager 1003 wears the wearable camera 1000, the imager 1003 may instruct the wearable camera 1000 about the front direction.

  Further, the image resolution increasing unit 104 may set the enlargement magnification for resolution enhancement to a different value for each area of the image.

  In order to capture a wide-angle image, it is necessary to use a wide-angle lens or a fish-eye lens with a short focal length. However, since these lenses are greatly affected by lens distortion, the resolution is not uniform between the central portion and the peripheral portion of the lens. Usually, the resolution at the center of the lens is the highest, and the resolution becomes lower toward the periphery. For this reason, you may make it set the magnification with respect to the area | region corresponding to a lens peripheral part high with respect to the area | region corresponding to a lens center part. Thereby, an enlarged image with uniform resolution can be generated.

  As described above, there is a high possibility that the subject facing the photographer is captured at the center of the lens. The subject facing the imager is considered to be the subject that the imager is interested in. On the other hand, the light source may be sunlight or street lamps outdoors, or illumination laid on the ceiling indoors. Usually, when the resolution enhancement processing of an image of 2 × 2 or more is performed, there is a problem that the higher the enlargement magnification is, the more the image becomes blurred and the image quality deteriorates. However, by reducing the magnification at the center of the lens compared to the periphery, it is possible to increase the resolution of a wide-angle image while suppressing deterioration in the image quality of the subject that the photographer is interested in.

  In addition, an enlarged image with uniform resolution may be generated by increasing the magnification of the region with low resolution compared to the region with high resolution.

  In addition, a method of using a reflection mirror to capture a wide-angle image is known (see, for example, Japanese Patent No. 2939087). Even when such an imaging apparatus is used, the resolution is not uniform in the image. Therefore, the resolution in the captured image is detected, and based on the result, an enlarged image with a uniform resolution is generated by setting the enlargement magnification of the low resolution area higher than that of the high resolution area. It doesn't matter.

  In order to detect the resolution in the captured image, internal parameters of the camera may be acquired by performing widely known camera calibration. This is because an imaging range for each pixel of the imaging apparatus can be estimated by using information such as focal length, focal position, lens distortion, and image sensor size, which are internal parameters. .

  As described above, according to the present embodiment, a subject is automatically and continuously imaged using an imaging device capable of imaging a wide-angle image, so that a light source image can be obtained together with the subject image. That is, it is possible to accurately acquire the light source image at that time when the subject is imaged. Then, light source information including the position, color, and illuminance of the light source is estimated using the captured wide-angle image and the imaging device information when the image is captured, and the wide-angle image is used using the estimated light source information. The resolution can be increased. That is, it is possible to acquire a light source image and realize high resolution of the image without mounting an additional imaging device other than for subject imaging. Furthermore, since the subject and the light source can be acquired as one image, there is no need to bother the alignment of the light source position and the subject.

(Second Embodiment)
FIG. 33 is a block diagram showing a configuration of an image processing apparatus according to the second embodiment of the present invention. 33, the same reference numerals are given to the same components as those of the image processing apparatus according to the first embodiment of the present invention shown in FIG. 1, and the detailed description thereof is omitted here.

  The configuration of FIG. 33 includes a light source environment determination unit 105 in addition to the configuration of FIG. The light source environment determination unit 105 determines whether or not the light source environment for the imaging apparatus 1001 has changed. The light source information estimation unit 103 determines that the light source environment has changed when the light source environment determination unit 105 determines that the light source environment has changed. Estimate.

  Specifically, for example, the light source environment determination unit 105 determines whether or not the imaging device 1001 has moved, and determines that the light source environment for the imaging device 1001 has changed when the imaging device 1001 has moved. As described above, when the position of the light source is estimated as the light source information, two images in which the position and orientation of the imaging device 1001 are changed are necessary. Therefore, efficient light source estimation processing can be performed by performing light source estimation processing by the light source information estimation unit 103 only when the imaging apparatus 1001 moves.

  For example, the light source environment determination unit 105 determines whether the imaging device 1001 has moved from the output of the angle sensor 1020 mounted on the imaging device 1001. That is, when the output of the angle sensor 1020 changes, it may be determined that the imaging device 1001 has moved. In the case where an angular velocity sensor or an acceleration sensor is mounted on the imaging apparatus 1001, it may be determined that the imaging apparatus 1001 has moved when a change in the output is detected.

  Further, instead of using the output of the angle sensor 1020, it may be determined whether the imaging apparatus 1001 has moved using an image captured by the wide-angle image capturing unit 101. For example, a difference between wide-angle images captured in time series is obtained, and the difference images are combined. If the amount of change is large, it may be determined that the imaging device 1001 has moved. Alternatively, the movement amount or the posture change amount of the imaging apparatus 1001 may be obtained from the wide-angle image using the above-described 8-point method or the like.

  Further, the light source environment determination unit 105 may determine whether or not the light source environment for the imaging device 1001 has changed using the information on the light source pixels described above. That is, when the luminance value or position of the light source pixel changes, it can be determined that the light source environment is likely to be changed. Therefore, only the light source pixel is extracted from the wide-angle image, and when the position or the luminance value is changed, it is determined that the light source environment has changed and the light source information estimation process is performed.

  As described above, according to the present embodiment, it is determined whether or not the light source environment for the imaging apparatus has changed, and when it is determined that the light source environment has changed, the light source information is estimated. And the resolution of the captured image can be increased.

(Third embodiment)
FIG. 34 is a block diagram showing a configuration of an image processing apparatus according to the third embodiment of the present invention. 34, the same reference numerals are given to the same components as those of the image processing apparatus according to the first embodiment shown in FIG. 1, and detailed description thereof is omitted here.

  The configuration in FIG. 34 includes an estimated light source information determination unit 106 in addition to the configuration in FIG. The estimated light source information determination unit 106 determines whether the light source information estimation unit 103 accurately estimates the light source information. When the estimated light source information determination unit 106 determines that the light source information is not accurately estimated, the light source information estimation unit 103 performs the light source information estimation process again.

  The process of the estimated light source information determination unit 106 will be described. Here, a method using a GPS (Global Positioning System) sensor and time information will be described. It is considered that sunlight is present as a light source when imaging is performed outdoors in the daytime on a fine day. Therefore, the direction of sunlight or the position of the sun is estimated, and the estimated direction or position is compared with the light source information estimated by the light source information estimation unit 103. Thereby, it is determined whether or not the light source information is accurately estimated. The estimation of sunlight may be performed from the output of the GPS sensor 1024 installed in the imaging apparatus 1001 and time information acquired using, for example, a radio clock 1025.

  Moreover, you may make it utilize the condition of the building around a photographer from map information using GPS information. It is possible to determine whether or not sunlight is blocked from the state of the building.

  Further, the estimated light source information determination unit 106 may use the shape information of the subject. This is because the image of the subject is estimated from the shape information of the subject and the light source information estimated by the light source information estimation unit 103, and the image actually captured by the wide-angle image capturing unit 101 is compared with the estimated image. It is determined whether or not the light source information is accurately estimated.

  This process will be described in detail. When the above-described shape information acquisition unit 204 is installed in the imaging apparatus 1001, normal information on the subject surface, which is the shape information of the subject, distance information from the imaging apparatus 1001, information on the three-dimensional position of the subject, and the like can be acquired. Therefore, using the normal information on the surface of the subject or the three-dimensional position information of the subject, the light source information, and further the camera parameter information, as shown in FIG. get.

FIG. 35 is a schematic diagram showing the positional relationship between the observation point O, the imaging device 1001 and the light source 1005. Here, an angle formed by the line-of-sight direction V and the normal direction n is θV, and an angle formed by the light source direction L and the normal direction n is θL. Here, it is known that a pixel satisfying the following relationship is a pixel in which light is regularly reflected, and its luminance value is very high.
○ θL = θV
○ Light source direction L, normal direction n, and line-of-sight direction V exist on the same plane

  Therefore, when a pixel satisfying the above condition is detected from an image captured by the wide-angle image capturing unit 101 and the luminance value thereof is sufficiently large or sufficiently larger than the luminance value of a neighboring pixel, the light source information estimation unit 103 Determines that the light source information is accurately estimated. On the other hand, when the luminance value of the detected pixel is not sufficiently large or smaller than the luminance value of the neighboring pixel, it is determined that the light source information estimated by the light source information estimation unit 103 is incorrect.

  Further, the estimated light source information determination unit 106 accurately estimates the light source information by comparing the color information of the image captured by the wide-angle image capturing unit 101 with the color of the light source estimated by the light source information estimation unit 103. You may make it determine whether it is.

  For example, the estimated color of the light source is compared with the color information of the specular reflection component. First, it is assumed that the subject is a dielectric. At this time, the image captured by the wide-angle image capturing unit 101 is separated into a specular reflection component and a diffuse reflection component using the method described in the explanation of the diffuse reflection / specular reflection separation unit 202. Here, it is known that the color information of the separated specular reflection component is equal to the color information of the light source. Therefore, the color information of the separated specular reflection component is compared with the color of the light source estimated by the light source information estimation unit 103. If the color information is sufficiently close, it is determined that the light source information estimated by the light source information estimation unit 103 is correct. On the other hand, when the color information is not close enough, it is determined that the light source information estimated by the light source information estimation unit 103 is incorrect. Or you may make it compare the color information of the separated specular reflection component, and the color information of a light source pixel.

  Of course, the color information of the light source may be compared with the color information of the diffuse reflection component. The color information of the diffuse reflection component is determined by the color information specific to the subject and the color information of the light source. Therefore, the color information of the light source and the color information of diffuse reflection cannot have a complementary color relationship. Therefore, for example, when the light source color information is blue but the diffuse reflection color is yellow, the light source information estimation unit 103 estimates the color information of the light source and the diffuse reflection color information in a complementary color relationship. It is determined that the light source information is incorrect. Of course, the color information of the light source and the color information of the image captured by the wide-angle image capturing unit 101 may be used.

  If the estimated light source information determination unit 106 determines that the estimated light source information is incorrect, the wide-angle image at that time is marked, and the wide-angle image is not used for light source estimation in the subsequent processing. It doesn't matter. This is effective when the light source information estimation unit 103 performs estimation later, not when the wide-angle image imaging unit 101 performs imaging, as in the editing process.

  As described above, according to the present embodiment, it is determined whether or not the light source information is accurately estimated, and when it is determined that the light source information is not accurately estimated, the light source information is estimated again. Accurate light source information is reliably estimated, and the captured image can be increased in resolution with high accuracy.

(Fourth embodiment)
FIG. 36 is a block diagram showing a configuration of an image processing system according to the fourth embodiment of the present invention. 36, the same reference numerals are given to the same components as those of the image processing apparatus according to the first embodiment shown in FIG. 1, and the detailed description thereof is omitted here.

  In FIG. 36, a wide-angle image imaging unit 101 and an imaging device information acquisition unit 102 are provided in a communication terminal 1100 having a communication function. The light source information estimation unit 103 and the image resolution enhancement unit 104 are provided in a server 1101 that is an external device connected to the communication terminal 1100 via a network. That is, in this embodiment, the communication terminal 1000 does not perform all processing, but only acquires a wide-angle image and imaging device information, and the light source information estimation and image resolution enhancement are executed on the server 1101 side. To do.

  The server 1101 and the communication terminal 1100 are located away from each other, and the server 1101 communicates with the communication terminal 1100 via the network. The communication terminal 1000 includes an information transmission unit 107, and the information transmission unit 107 transmits the wide-angle image captured by the wide-angle image imaging unit 101 and the imaging device information acquired by the imaging device information acquisition unit 102 to the server 1101. Further, the communication terminal 1100 instructs the server 1101 to estimate the light source information and increase the image resolution.

  The server 1101 has an information receiving unit 108, and the information receiving unit 108 receives a wide-angle image and imaging device information transmitted by the information transmitting unit 107 of the communication terminal 1100. The light source information acquisition unit 103 estimates light source information using the received wide-angle image and imaging device information. Further, the image resolution increasing unit 104 increases the resolution of the received wide-angle image using the light source information estimated by the light source information estimation unit 103. The estimation of the light source information and the resolution enhancement of the image are performed according to the contents instructed from the communication terminal 1100.

  Thus, by providing the light source information estimation unit 103 and the image resolution enhancement unit 104 in the server 1101, the calculation load on the communication terminal 1100 can be reduced.

  The image processing method of the present invention is also effective for image processing other than increasing the resolution of an image. Here, an augmented reality display that superimposes and displays an image (CG image) created by Computer-Graphics will be described.

  When superimposing and displaying an image created by Computer-Graphics, it is important to superimpose a CG image on an actual image with brightness, hue, and shadow without a sense of incongruity (see, for example, Patent Document 1). Since the image processing method of the present invention can accurately acquire the position and direction of the light source, luminance, color, and spectrum information, such processing is possible. In this case, means for superimposing and displaying the CG image on the actual image may be provided instead of the image resolution increasing unit 104. As such a superimposed display method, for example, a known method such as Patent Document 1 may be used.

  As described above, image processing such as high-resolution digital zoom processing can be performed by using the image processing apparatus of the present invention.

  According to the present invention, it is possible to acquire a light source image and estimate light source information without mounting an additional imaging device. For this reason, for example, in a digital camera or digital video camera attached to a moving body such as a person or a car, it is useful for performing image processing such as high resolution of an image.

1 is a block diagram illustrating a configuration of an image processing apparatus according to a first embodiment of the present invention. It is a schematic diagram which shows the utilization form of the wearable camera as an apparatus by which the image processing apparatus which concerns on this invention is mounted. It is a schematic diagram which shows the imaged wide angle image. It is a schematic diagram which shows a part of information hold | maintained at memory. It is a schematic diagram for demonstrating roll * pitch * yaw angle expression. It is a schematic diagram for demonstrating the relationship between a camera coordinate system and an image coordinate system. It is a schematic diagram for demonstrating the process which estimates the three-dimensional position of a light source using that an imaging device moves. It is a figure which shows the example which isolate | separated the image into the diffuse reflection image and the specular reflection image. It is a block diagram which shows the structural example of the image high resolution part. It is a figure which shows the wearable camera by which the image processing apparatus which concerns on embodiment of this invention is mounted. It is a graph which shows the change of reflected light intensity when rotating a polarizing filter when irradiated with linearly polarized light. It is a flowchart which shows the flow of a separation process of a specular reflection image and a diffuse reflection image using a polarizing filter. It is a schematic diagram for demonstrating the imaging device from which a polarization direction differs for every pixel. It is a schematic diagram for demonstrating the process which calculates | requires the distance of a to-be-photographed object and a three-dimensional position using an illuminance difference stereo method. It is a schematic diagram for demonstrating the acquisition process of the shape information using the polarization characteristic of reflected light. It is a graph which shows the change of reflected light intensity when rotating a polarizing filter, when natural light is irradiated. It is a schematic diagram which shows the concept of the high resolution processing based on texton. It is a conceptual diagram for demonstrating the high resolution process based on the texton using linear matrix conversion. It is a PAD figure which shows the flow of the learning process in the high resolution processing based on texton. It is a schematic diagram for demonstrating the learning process in the high resolution processing based on texton. It is a figure which shows the process of a two-dimensional discrete stationary wavelet transform. It is an example of an image result at the time of implementing a two-dimensional discrete stationary wavelet transform to a test image. It is a PAD figure which shows the flow of the process at the time of execution in the high resolution processing based on texton. It is a schematic diagram for demonstrating the process at the time of the high resolution process based on texton. It is a figure which shows the process of a two-dimensional discrete stationary inverse wavelet transform. It is a schematic diagram for demonstrating the constant Sr for expressing the difference of the luminance value of a diffuse reflection component and a specular reflection component. It is a figure which shows the flow of the parameter estimation process of the specular reflection image in an image high resolution process. It is a conceptual diagram for demonstrating each parameter of the type | formula showing incident illumination intensity. It is a flowchart which shows the flow of the parameter estimation process by a simplex method. It is the flowchart which showed the flow of the parameter update process in a simplex method. It is a schematic diagram for demonstrating polar coordinate expression. It is a figure which shows the flow of the parameter estimation process of the diffuse reflection image in an image high resolution process. It is a block diagram which shows the structure of the image processing apparatus which concerns on the 2nd Embodiment of this invention. It is a block diagram which shows the structure of the image processing apparatus which concerns on the 3rd Embodiment of this invention. It is a schematic diagram which shows the positional relationship of the observation point of a to-be-photographed object, an imaging device, and a light source. It is a block diagram which shows the structure of the image processing system which concerns on the 4th Embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 Wide-angle image imaging part 102 Imaging device information acquisition part 103 Light source information estimation part 104 Image high resolution part 105 Light source environment determination part 106 Estimated light source information determination part 204 Shape information acquisition part 1001 Imaging apparatus 1020 Angle sensor 1024 GPS sensor 1100 Communication terminal 1101 server

Claims (18)

A wide-angle image capturing unit that automatically and continuously images a subject using an imaging device capable of capturing a wide-angle image;
An imaging device information acquisition unit that acquires imaging device information representing a situation of the imaging device when imaging is performed by the wide-angle image imaging unit;
A light source information estimation unit that estimates light source information including a position, a color, and illuminance of a light source using the wide angle image captured by the wide angle image capturing unit and the imaging device information acquired by the imaging device information acquisition unit; ,
An image processing apparatus comprising: an image high-resolution unit that increases the resolution of the wide-angle image using the light source information estimated by the light source information estimation unit. The image processing apparatus according to claim 1.
The imaging device information acquisition unit
The image obtained by acquiring the imaging device information using at least one of an output of an angle sensor mounted on the imaging device and an image captured by the wide-angle image capturing unit. Processing equipment. The image processing apparatus according to claim 1.
The wide-angle image capturing unit sets an exposure time of a pixel of the image capturing device to a different value for each region. The image processing apparatus according to claim 3.
The wide-angle image capturing unit sets an exposure time of a pixel in a region corresponding to a peripheral portion of a lens of the image capturing device shorter than a region corresponding to a central portion of the lens. . The image processing apparatus according to claim 1.
A shape information acquisition unit that acquires surface normal information or three-dimensional position information of the subject as shape information;
The image high-resolution part is
An image processing apparatus, wherein the resolution of the wide-angle image is increased using the shape information acquired by the shape information acquisition unit. The image processing apparatus according to claim 5.
The image high-resolution unit separates the wide-angle image into a diffuse reflection component and a specular reflection component, and individually separates the diffuse reflection component and the specular reflection component into a high resolution. An image processing apparatus. The image processing apparatus according to claim 5.
The image processing device according to claim 1, wherein the image resolution increasing unit decomposes the wide-angle image into parameters, and individually increases the resolution of the decomposed parameters. The image processing apparatus according to claim 1.
The image processing device according to claim 1, wherein the image resolution increasing unit sets the enlargement magnification for increasing the resolution to a different value for each area of the image. The image processing apparatus according to claim 8.
The image processing apparatus, wherein the image resolution increasing unit sets an enlargement magnification higher in a region corresponding to a peripheral portion of the lens of the imaging device than in a region corresponding to the central portion of the lens. The image processing apparatus according to claim 1.
A light source environment determination unit that determines whether or not the light source environment for the imaging device has changed,
The light source information estimation unit estimates light source information when the light source environment determination unit determines that the light source environment has changed. The image processing apparatus according to claim 10.
The light source environment determination unit
An image processing apparatus comprising: determining whether or not the imaging apparatus has moved from an output of an angle sensor mounted on the imaging apparatus; and determining that the light source environment has changed when the imaging apparatus has moved. The image processing apparatus according to claim 10.
The imaging environment determination unit
An image processing apparatus, wherein an image captured by the wide-angle image capturing unit is used to determine whether or not the image capturing apparatus has moved, and when the image capturing apparatus has moved, it is determined that the light source environment has changed. The image processing apparatus according to claim 1.
An estimated light source information determination unit that determines whether the light source information estimation unit accurately estimates light source information;
The light source information estimation unit estimates the light source information again when the estimated light source information determination unit determines that the light source information is not accurately estimated. The image processing apparatus according to claim 13.
The estimated light source information determination unit estimates the direction of sunlight or the position of the sun using the output of a GPS (Global Positioning System) sensor and time information, and the estimated direction or position is determined by the light source information estimation unit. An image processing apparatus that performs determination by comparing with estimated light source information. The image processing apparatus according to claim 13.
The estimated light source information determination unit estimates an image of the subject using shape information of the subject and light source information estimated by the light source information estimation unit, and the estimated image is captured by the wide-angle image capturing unit. An image processing apparatus that performs a determination by comparing with an image. The image processing apparatus according to claim 13.
The estimated light source information determination unit performs determination by comparing color information of an image captured by the wide-angle image capturing unit with a color of a light source estimated by the light source information estimation unit. Processing equipment. An image processing system for increasing the resolution of an image,
A communication terminal having the wide-angle image capturing unit and the imaging device information acquiring unit according to claim 1 and transmitting the wide-angle image captured by the wide-angle image capturing unit and the imaging device information acquired by the imaging device information acquiring unit. When,
A light source information estimation unit and an image resolution enhancement unit according to claim 1, wherein the wide angle image and the imaging device information transmitted from the communication terminal are received, and the wide angle image is converted into the light source information estimation unit and the image height. A server for providing to the resolution unit, and for providing the imaging device information to the light source information estimation unit,
The communication terminal instructs the server to estimate light source information and to increase the resolution of an image. A first step of automatically and continuously imaging a subject using an imaging device capable of imaging a wide-angle image;
A second step of acquiring imaging device information representing a state of the imaging device when imaging is performed in the first step;
A third step of estimating light source information including the position, color, and illuminance of the light source using the wide-angle image captured in the first step and the imaging device information acquired in the second step;
And a fourth step of increasing the resolution of the wide-angle image using the light source information estimated in the third step.