JP2017215851A - Image processing device, image processing method, and molding system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing device capable of precisely estimating light source color with a simple configuration, and an image processing method.SOLUTION: An image processing device acquires a plurality of images that photograph an object from different viewpoints. Corresponding points 72 to 74 to the plurality of images are detected. Color represented by specific coordinates on a line 75 obtained from color chromaticity coordinates of a pixel value of the corresponding points is estimated as light source color at the time of photographing the plurality of images. The above estimation means may estimate color that a cross point 77 of a blackbody locus 76 with the above line represents as the light source color. Alternatively, the above estimation means may estimate color that coordinates of the cross point 77 of a plurality of lines 75, 85, and 89 obtained from pixel values of the plurality of corresponding points 72 to 74, 82 to 84, and 86 to 88 represent, as the light source color.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing apparatus and an image processing method, and more particularly to a technique for estimating a light source color from an image. The invention also relates to a modeling system.

  An apparatus (3D scanner) that generates 3D shape data of an object based on a captured image of a three-dimensional (3D) object is known. For example, when making a miniature of a certain three-dimensional object with a 3D printer (3D modeling apparatus), if information about not only the 3D shape of the object but also the color can be acquired, a model with good reproducibility can be obtained.

  However, the color of the object in the captured image is affected by the color of the ambient light. For this reason, it is desired to eliminate the influence of the color of the ambient light on the captured image. As a technique for estimating the color of ambient light (light source color) from a captured image, Patent Document 1 discloses a method for specifying a light source color that irradiates a subject by calculating a difference between adjacent pixels having different brightnesses. Has been. Patent Document 2 discloses a method for estimating a light source color by estimating a region where specular reflection occurs from an input image and calculating a difference between pixel values in the region.

JP 2007-13415 A JP 2014-7611 A

  However, the methods disclosed in Patent Documents 1 and 2 estimate the light source color on the assumption that the color of the object corresponding to the position of the pixel from which the difference is obtained is the same. Therefore, for example, when the object color is continuously changing, there is a problem that the estimation accuracy is lowered when the color of the object corresponding to the position of the pixel from which the difference is acquired is different.

  An object of the present invention is to provide an image processing apparatus and an image processing method capable of accurately estimating a light source color with a simple configuration.

  The purpose described above is to acquire a plurality of images obtained by photographing an object from different viewpoints, to detect a corresponding point of a plurality of images, and to specify a straight line obtained from chromaticity coordinates of pixel values of the corresponding points. And an estimation unit that estimates a color represented by the coordinates as a light source color at the time of photographing a plurality of images.

  According to the present invention, it is possible to provide an image processing apparatus and an image processing method capable of estimating a light source color with a simple configuration.

The figure for demonstrating the estimation method of the light source color which concerns on embodiment The figure which shows the structural example of the camera system which concerns on embodiment. The figure regarding the image pick-up element which the camera system which concerns on embodiment has The figure about the reflection of the light on the object and brightness in an embodiment Flowchart relating to operation of camera system according to embodiment The figure about block matching in an embodiment The figure which shows the structural example of the 3D modeling system which concerns on embodiment

  Hereinafter, exemplary embodiments of the present invention will be described with reference to the accompanying drawings. In the following, a camera system (single-lens reflex digital camera) as an example of an image processing apparatus according to an embodiment of the present invention or as an example of an apparatus capable of performing an image processing method according to an embodiment of the present invention will be described. To do. However, the present invention can be implemented in an electronic device capable of image processing, and a photographing function is not essential. Examples of electronic devices that can implement the present invention include an imaging device, an electronic device having a photographing function, various computer devices (such as a personal computer, a tablet computer, and a wearable computer), a smartphone, a game machine, and a media player. .

  2A and 2B are diagrams illustrating a configuration example of the camera system according to the embodiment. FIG. 2A is a vertical cross-sectional view passing through the optical axis of the photographing optical system, and FIG. 2B is an electrical configuration example. It is a block diagram. Note that the camera system of the present embodiment can also acquire information on the three-dimensional shape of the subject, and thus functions as a 3D scanner. In addition, since it is possible to estimate the color of the ambient light and acquire information on the object color of the subject, it also functions as a light source color or object color estimation device.

  The camera system 100 includes a camera body 1 and a lens unit 2 that is detachably attached to the camera body 1. The camera body 1 includes a camera control unit 5, an image sensor 6, an image processing unit 7, a memory unit 8, a display unit 9, an operation detection unit 10, an electrical contact 11, and a quick return mirror 14. The lens unit 2 includes a photographing optical system 3 including a plurality of lenses having a common optical axis 4, a lens driving unit 13, and a lens control unit 12. The photographing optical system 3 includes movable members such as a diaphragm, a focus lens, a variable power lens, and a correction lens. The lens driving unit 13 changes the position of the movable member by driving an actuator or a motor according to the control of the lens control unit 12.

  In the camera system 100, the photographing optical system 3 and the image sensor 6 constitute at least a part of an imaging unit that converts a subject image into an electrical signal. The memory unit 8 and the display unit 9 constitute at least a part of a recording / reproducing unit that records or outputs image data. The camera control unit 5 includes one or more processors and a memory.

  The camera control unit 5 controls the operations of the camera body 1 and the lens unit 2 by loading the program stored in the non-volatile portion of the memory into the volatile portion of the memory and executing the program, and the operation of the entire camera system 100 Is realized. The camera control unit 5 communicates with the lens control unit 12 through the electrical contact 11 and controls the operation of the lens unit 2 through the lens control unit 12.

  The lens control unit 12 includes one or more processors and a memory. The lens control unit 12 loads the program stored in the non-volatile part of the memory into the volatile part of the memory and executes it, thereby establishing communication with the camera control unit 5 and controlling the camera control unit 5. Or perform actions based on it. Further, the lens control unit 12 calculates the driving direction and driving amount of the correction lens based on a signal from a shake detection sensor such as an acceleration sensor provided in the lens unit 2 and drives the correction lens via the lens driving unit 13. Then, shake correction is executed. Note that whether or not camera shake correction is performed is controlled by the camera control unit 5.

  The imaging optical system 3 forms an optical image of the subject on the imaging surface of the imaging element 6. As will be described later, a microlens array (hereinafter referred to as MLA) in which a plurality of microlenses are arranged in a lattice shape is disposed on the imaging surface of the image sensor 6. The image sensor 6 has a plurality of photoelectric conversion units that share one microlens, and can generate a focus detection signal. The image processing unit 7 generates a predetermined evaluation value based on a signal obtained from the image sensor 6. The camera control unit 5 can execute automatic focus detection (AF) and automatic exposure control (AE) processing based on the evaluation value.

  The image processing unit 7 executes signal reading from the image sensor 6 and various processes for the read signal. Examples of processing executed by the image processing unit 7 include the following. A / D conversion, D / A conversion, noise reduction, signal format conversion, white balance adjustment, gamma correction, demosaicing, scaling, filtering, encoding and decoding, evaluation value generation, image feature amount generation, correlation operation , Image recognition, GUI image generation, encryption and decryption. Note that these are merely examples, and other processes may be executed or some of them may not be executed.

  In addition, the image processing unit 7 generates a plurality of images (multi-view images) having different shooting viewpoints based on a signal obtained from the imaging element 6 in one shooting, and the subject distance in each pixel of the multi-view images. Can be calculated. The image processing unit 7 also executes processing for estimating the color of the light source (environment light) at the time of shooting the multi-viewpoint image based on the multi-viewpoint image.

  The image processing unit 7 can execute at least a part of these processes with a dedicated hardware circuit. Further, the image processing unit 7 can realize at least a part of these processes by executing a program with a processor. Therefore, the camera control unit 5 may also serve as at least a part of the image processing unit 7. Further, the image sensor 6 may have an A / D conversion function.

  The memory unit 8 includes a storage device and is used as a recording destination for various data, and is also used as a video memory for the display unit 9 and a work memory for the camera control unit 5 and the image processing unit 7. When the memory unit 8 includes a removable storage medium (for example, a memory card), the memory unit 8 also includes hardware necessary for reading and writing the recording medium. Further, the memory unit 8 stores information relating to black body radiation, for example, information representing a curve indicating black body radiation in the xy chromaticity space.

  The operation detection unit 10 detects a user instruction to the camera system. The operation detection unit 10 may include buttons, switches, touch panels and the like that can be operated by the user. Note that the user instruction may be performed by line-of-sight input or voice input, and the operation detection unit 10 has a configuration according to the type of input that can be accepted. The input detected by the operation detection unit 10 is notified to the camera control unit 5. The camera control unit 5 executes processing corresponding to the notified input.

  For example, when the operation detection unit 10 detects that the release button is pressed, the camera control unit 5 performs a still image shooting / recording operation. Specifically, the camera control unit 5 raises the quick return mirror 14 and controls the charge accumulation time of the diaphragm and the image sensor 6 included in the photographing optical system 3 according to the exposure condition determined by the AE operation. In addition, the camera control unit 5 reads out an image signal from the image sensor 6 and supplies it to the image processing unit 7 to generate display image data and a recording image data file. Further, the camera control unit 5 writes the display image data in the video memory of the memory unit 8 and displays it on the display unit 9, and the recording image data file is written in the storage device of the memory unit 8.

  For example, as part of the white balance adjustment process in the image processing unit 7, a light source color estimation process described later can be executed, and an automatic white balance adjustment using a white detection range corresponding to the estimated light source color can be executed. .

  FIG. 3 is a diagram illustrating a configuration example of the image sensor 6. In the present embodiment, the image sensor 6 has a pupil division function and can generate a plurality of images with different viewpoints by one exposure. FIG. 3A is a schematic top view and side view of the image sensor 6. An MLA 20 is provided on the image sensor 6, and the front principal point of the MLA 20 is disposed in the vicinity of the imaging plane of the photographing optical system 3. In FIG. 3A, the shape of each microlens constituting the MLA 20 is deformed so that it can be easily grasped.

  In FIG. 3, the z axis is an axis extending in the direction of the lens unit from the origin in the camera body 1 in parallel with the optical axis of the photographing optical system 3. Further, when the bottom surface of the camera body 1 (B in FIG. 2A) is leveled and the camera body 1 is viewed from a point on the optical axis, the axis extending vertically upward from the origin is the y axis, and the left side is horizontally from the origin. The axis extending in the direction is the x-axis.

  FIG. 3B is a diagram schematically illustrating a correspondence relationship between two horizontal and vertical microlenses and the photoelectric conversion region of the image sensor 6. Here, 5 rows × 5 columns = 25 photoelectric conversion regions are associated with one microlens 21 (25 photoelectric conversion regions share one microlens). Here, the photoelectric conversion region is a minimum unit that allows the image pickup device 6 to read signals independently, and signals from a plurality of photoelectric conversion regions can be combined (added) and read.

  FIG. 3C is a diagram schematically illustrating a correspondence relationship between the photoelectric conversion regions 20-a to 20-e in FIG. 3B and the exit pupil 31 of the photographing optical system 3. The lower part of FIG. 3C shows an aa cross section of the part corresponding to the upper left microlens 21 of FIG. The upper diagram in FIG. 3C shows the exit pupil plane of the photographic optical system 3. Here, for easy understanding, the exit pupil plane is rotated by 90 degrees and illustrated as a plane parallel to the paper (see the direction of the axis).

  As shown in FIG. 3C, in the microlens 21, each of the 25 photoelectric conversion regions associated with the microlens 21 is conjugated with different specific regions in the exit pupil 31 of the photographing optical system 3. Designed to have a relationship. That is, in the example of FIG. 3C, the photoelectric conversion regions 20-a to 20-e correspond to the regions 30-a to 30-e in the exit pupil 31 on a one-to-one basis. Due to such a conjugate relationship, only the light beam emitted from a specific area in the exit pupil 31 enters each photoelectric conversion area. The signal obtained in each photoelectric conversion region represents the intensity of the light beam incident on the microlens 21 from a specific direction. For example, only the light beam emitted from the region 30-b in the exit pupil 31 from a specific direction enters the photoelectric conversion region 20-b. In FIG. 3C, for ease of understanding, only five of the 25 photoelectric conversion regions in the horizontal direction have been described, but the same applies to the other photoelectric conversion regions.

  From a signal group obtained from a photoelectric conversion region (in this example, 20-b) corresponding to the same region (for example, region 30-b) in the exit pupil among a plurality of photoelectric conversion regions associated with each microlens. An image can be formed. Therefore, when the image sensor 6 having the configuration of FIG. 3 is used, 25 types of images can be formed by one exposure. These 25 types of images are images of the subject viewed from different areas in the exit pupil 31, and are therefore multi-viewpoint images (parallax images) obtained by photographing the same subject from different viewpoint positions.

  In the present embodiment, the configuration in which a multi-viewpoint image is obtained using an imaging device having a configuration in which a plurality of photoelectric conversion regions are associated with one microlens has been described. However, a multi-viewpoint image can be obtained by other configurations such as using a multi-view camera.

  Next, light source color estimation processing in the present embodiment will be described. In the present embodiment, dichroic reflection in which reflected light from an object is observed as the sum of a specular reflection component having the same spectral distribution as the light source and a diffuse reflection component in which a part of the wavelength is absorbed by the material of the object. Estimate the light source color based on the model.

FIG. 4A is a diagram schematically showing the reflected light (diffuse reflected light) 52 of the light beam 51 when the surface of the object 50 is a completely diffuse reflecting surface (Lambertian surface). In FIG. 4A, the reflected light 52 is represented by an intensity envelope. Diffuse reflected light on the Lambertian surface is uniformly reflected in each direction, and the brightness (intensity) of the reflected light does not depend on the observation direction. The luminance Ld (λ) of the diffuse reflected light can be expressed by the following equation.
Ld (λ) = Ie (λ) Kd (λ) cos θ (1)
Where Ie (λ) is the luminance of the light source, Kd (λ) is the diffuse reflection characteristic, θ is the incident angle, and λ is the wavelength.

FIG. 4B is a diagram schematically showing the reflected light (specular reflected light) 53 of the light beam 51 when the surface of the object 50 is a mirror surface. Reference numeral 61 denotes the regular reflection direction of the light beam 51. Similarly to FIG. 4A, the reflected light 53 is represented by an intensity envelope. Here, the phone model is used as the model of the specular reflection light, but other models may be used. The intensity of the specular reflected light 53 has a strong peak in the regular reflection direction 61 of the light beam 51 and is greatly attenuated when the observation direction is away from the regular reflection direction. The brightness Ls (λ) of the specular reflection light can be expressed by the following formula.
Ls (λ) = Ie (λ) (cosφ) ⁿ (2)
However, φ is an angle formed by the specular reflection direction and the observation direction, and n is a constant determined by the characteristics of the object.

According to the dichroic reflection model, the reflected light of the object is a combination of the specular reflection light and the diffuse reflection light. Therefore, the luminance L (λ) of the reflected light observed at an arbitrary viewpoint p is given by Can be represented.
L (λ) = Ld (λ) + Ls (λ)
= Ie (λ) Kd (λ) cos θ + Ie (λ) (cos φ _p ) ⁿ (3)
However, phi _p is the angle between the specular direction and the viewpoint p.

FIG. 4C is a diagram schematically illustrating a state in which the reflected light 54 of the object 50 is captured by the camera system 100, which is represented by Expression (3). Here, it is assumed that the optical axis 4 passes through the reflection point of the light beam 51 and the optical axis 4 and the regular reflection direction 61 are in a slightly shifted positional relationship. The photographing optical system 3 is focused on the reflection point 50a. The reflected light 54 has an intensity distribution obtained by combining the diffuse reflected light 52 and the specular reflected light 53 as indicated by the envelope. Φ _a and φ _e indicate angles formed between the reflected light incident on the pupil regions 30-a and 30-e of the photographing optical system 3 and the regular reflection direction 61, respectively.

In this example, since φ _a > φ _e , the intensity of the specular reflected light included in the light beam incident on the pupil region 30-a is greater than the intensity of the specular reflected light included in the light beam incident on the pupil region 30-e. Lower. On the other hand, since the intensity of the diffuse reflected light does not depend on the observation direction, the intensity of the diffuse reflected light included in the light flux incident on the pupil regions 30-a and 30-e is equal.

  FIG. 4D shows a state in which FIG. 4C is expanded to a plurality of parallel lights. The light rays 51a to 51d enter the object 50 and are reflected at the incident points (reflection points) 50a to 50d. 61a to 61d are regular reflection directions of the light beams 51a to 51d. Here, the direction of the optical axis 4 of the photographing optical system 3 coincides with the regular reflection direction 61b of the light beam 51b and is focused on the reflection points 50a to 50d. Therefore, the reflected light from a certain reflection point is incident on the same position on the imaging plane regardless of where in the exit pupil. Therefore, the reflected lights of the light beams 51 a to 51 d are imaged at points 6 a to 6 d on the imaging surface of the image sensor 6.

  The lower part of FIG. 4D shows the intensity distribution 52-a of incident light from the pupil region 30-a and the intensity distribution 52-e of incident light from the pupil region 30-e, which are observed by the imaging device 6. ing. Here, the intensity distribution is indicated by the magnitude of the luminance. Reference numeral 53 denotes the intensity of incident light from the pupil region 30-a observed at the position 6c on the imaging plane, and reference numeral 54 denotes the incidence from the pupil region 30-e observed at the position 6c on the imaging plane. Luminance 54 of the light and 55 represent the luminance of the diffusely reflected light. The intensity 55 of the diffuse reflected light corresponds to Ld (λ) in the equations (1) and (3).

  Thus, when the same point on the object surface is observed from different viewpoints, a difference occurs in the intensity of the specular reflection light included in the reflected light to be observed. Therefore, the color of specular reflection light is estimated using this intensity difference. Since the specular reflected light has the same spectral distribution as that of the environmental light source, the estimation of the color of the specular reflected light is equal to the estimation of the color of the environmental light.

Hereinafter, a specific light source color estimation operation will be described.
FIG. 5A is a flowchart of the shooting operation in the camera system 100. Here, it is assumed that the light source color estimation is performed in the course of the photographing operation. However, the light source color estimation operation of the present embodiment can be performed on image data already recorded in the storage device of the memory unit 8, for example. That is, as long as image data obtained by observing the same object from different viewpoints can be obtained, the data acquisition method is not limited. In addition, here, for example, it is assumed that information on a three-dimensional shape of a subject and estimation of an object color are executed assuming use in a 3D printer, but these processes are not essential.

  The process illustrated in FIG. 5A is executed, for example, when the camera system 100 is in a shooting standby state. It is assumed that the camera system 100 performs a so-called live view display operation in which moving image shooting is continuously performed in a shooting standby state and the shot moving image is displayed on the display unit 9.

  In S <b> 110, the camera control unit 5 acquires an image for one frame from the image sensor 6 and supplies the image to the image processing unit 7. The image processing unit 7 generates image data for live view display and writes it in the video memory portion of the memory unit 8. In live view display, it is not necessary to generate a multi-viewpoint image. Therefore, the camera control unit 5 supplies the image processing unit 7 with a signal obtained by adding signals obtained by each of the plurality of photoelectric conversion units sharing one microlens. The signal reading may be performed for each photoelectric conversion unit, and may be added by the image processing unit 7 so as to obtain one signal per microlens.

  In S120, the camera control unit 5 reads the image data from the memory unit 8, displays the image data on the display unit 9, and advances the processing to S130.

  In S <b> 130, the camera control unit 5 checks whether or not a shooting start instruction is detected by the operation detection unit 10. The shooting start instruction may be, for example, a full pressing operation of a release button included in the operation detection unit 10. If the shooting start instruction is detected, the camera control unit 5 advances the process to S140. If not detected, the camera control unit 5 returns the process to S110 to execute the acquisition and display operation of the next frame. Although not shown in FIG. 5A, when an instruction to start a shooting preparation operation is detected through the operation detection unit 10, the camera control unit 5 performs AF processing or the like in parallel with the operations of S110 and S120. An imaging preparation operation such as an AE process is executed.

  In S <b> 140, the camera control unit 5 performs still image shooting according to the exposure condition determined in the AE process, and supplies the still image to the image processing unit 7. Here, the camera control unit 5 supplies a signal for each photoelectric conversion unit to the image processing unit 7. The image processing unit 7 generates an image of one frame for each signal group obtained from the photoelectric conversion unit having the same pupil region having a conjugate relationship, and temporarily stores the image in the memory unit 8. When the image sensor 6 has the configuration shown in FIG. 3, a multi-viewpoint image of 25 frames can be generated by one exposure. Note that at least two frames need only be generated for the multi-viewpoint image, and not all 25 frames need to be generated.

In S150, the image processing unit 7 estimates corresponding points between the multi-viewpoint images. Details will be described later.
In S160, the camera control unit 5 estimates the light source color using the information on the corresponding points estimated in S150. Details will be described later.

  In S170, the camera control unit 5 estimates the object color based on the estimated light source color. By estimating the light source color in S160, the light source luminance Ie (λ) in the equations (1) to (3) is obtained. Since the image obtained by photographing is based on the reflected light represented by the expression (3), the diffuse reflected light Ld (λ) can be obtained by removing the component of the specular reflected light Ls (λ). . The specular reflection light component decreases as the light source luminance Ie (λ) decreases. Therefore, the signal of the pixel with the lowest luminance among the pixels corresponding to the same object in the multi-viewpoint image can be selected as the signal containing the color component (diffuse reflected light) of the object in the highest ratio. Alternatively, a function representing the intensity of diffuse reflected light is obtained (fitted) by regression analysis or the like under an appropriate reflection model (for example, the Phong model), and the diffuse reflected light component Ld (λ) in Equation (3) is estimated. May be. Extracting the diffuse reflected light component corresponds to extracting the diffuse reflected light 52 component from the reflected light 54 in FIG.

  The color represented by the diffuse reflection light component thus extracted is affected by the light source color. Therefore, by applying the white balance gain based on the light source color estimated in S160, the influence of the light source color can be reduced from the observed color, and the object color can be estimated. For example, when the image sensor 6 uses a color filter of the primary color Bayer arrangement, the signal value corresponding to the transmitted light of the red and blue filters is set with the signal value corresponding to the transmitted light of the green filter as a reference (gain 1.0). Adjust the gain.

  In S180, the camera control unit 5 outputs (at least one of display and recording) an image obtained by photographing. The images to be recorded may be all of the multi-view images, but may be a part of the multi-view images or an image obtained by synthesizing the multi-view images. For display, one of the multi-viewpoint images or an image obtained by synthesizing the multi-viewpoint images can be used. In the present embodiment, it is assumed that an image subjected to white balance adjustment based on the estimated light source color is recorded. Thereby, the recorded image shows the color of the object independent of the color of the ambient light. There is no limitation on the format of image data to be recorded and the format of a file storing image data. Of course, together with the estimated light source color data, an image that has not been subjected to white balance adjustment (for example, an image in RAW format) may be recorded. In addition, when the purpose of photographing is acquisition of three-dimensional information of a subject, a composite image, a distance image, and an estimated light source color may be recorded.

  In step S190, the camera control unit 5 determines whether to end the shooting and recording operation. If a predetermined end instruction is detected, the process ends. If not, the process returns to step S110. The detection of the end instruction may be, for example, when a switching instruction to the reproduction mode is detected or when a power-off instruction is detected, but is not limited thereto.

  Next, details of the corresponding point estimation process performed in S150 will be described with reference to the flowchart shown in FIG. Corresponding point estimation is a process of searching for a relative positional relationship for any two images among the multi-viewpoint images generated in S140. A part of one image is cut out in a rectangular (block) shape and used as a template. Then, a search area is set in the other image, the similarity (correlation) is calculated at each position while moving the template within the search area, and a position with the highest similarity is searched. For example, by estimating one corresponding point with each other using one of the multi-viewpoint images as a reference image, the corresponding point in each of the other images can be detected for each pixel of the reference image.

  In S310, the image processing unit 7 sets a template for one image. For example, the template may be a rectangular block of 8 × 8 pixels. The setting position of the template is not particularly limited, but the detection accuracy of the corresponding points can be increased by setting the template so as to include a high-contrast pattern or a characteristic pattern. Further, when searching for corresponding points for each pixel of the image, it is possible to set a template centering on the target pixel while sequentially moving the search target pixel from the left to the right and from the top to the bottom of the image. it can.

  In S320, the image processing unit 7 performs block matching. For example, assume that the images 91 and 94 shown in FIGS. 6A and 6B are targeted, and the template 93 is set to the image 91 in S310. In this case, the image processing unit 7 searches for corresponding points in the image 94.

  Specifically, the image processing unit 7 searches for an area having the highest similarity with the template 93 in the image 94. For example, the image processing unit 7 searches a region having the smallest difference absolute value sum (SAD) between the template 93 and the pixel value as a region having the highest similarity with the template 93 among the regions in the image 94. In the example of FIG. 6, the image processing unit 7 first calculates the similarity with the template 93 for the region 95 in the image 94 corresponding to the cutout position of the template 93 of the image 91. Thereafter, as indicated by an arrow 96, the image processing unit 7 calculates the similarity at each position while sequentially moving the position of the region for calculating the similarity in the image 94 in the horizontal and vertical directions. However, since the epipolar constraint is applied to the images 91 and 94, the search can be limited to the positions of the images 91 and 94 on the epipolar line.

  Here, it is assumed that the region 97 is detected as the region having the highest similarity. FIG. 6C is a diagram schematically showing the images 91 and 94, the template 93, and the region 97 superimposed. The movement direction and movement amount of the template 93 are shown as a motion vector 99.

  In S330, the image processing unit 7 stores the corresponding point information. The corresponding point information may be, for example, a movement vector from the template 93 to the region having the highest similarity to the template 93 detected in S320. Or you may preserve | save the coordinate of the pixel of the image 94 detected as a corresponding point of the object pixel in the template 93. FIG.

  In S340, the image processing unit 7 determines whether the matching process for the image 91 has been completed (matching process has been performed using all predetermined regions as templates). If it is determined that the matching process has been completed, the image processing unit 7 ends the corresponding point estimation process. If not, the image processing unit 7 returns the process to S310 and executes the matching process using another region as a template. .

  By corresponding point estimation processing, corresponding points in two images with different viewpoints, that is, stereo images, are specified. Therefore, using a known parameter such as a pixel pitch, the subject distance for each pixel position can be estimated based on the principle of triangulation. Thereby, information regarding the shape of the subject can be obtained. For example, the shape information of the entire three-dimensional object can be obtained by photographing the three-dimensional object a plurality of times from different angles and combining the shape information obtained by each photographing.

Next, the details of the light source color estimation processing performed in S160 will be described with reference to the flowchart shown in FIG.
In S410, the camera control unit 5 determines the values (colors) of a plurality of pixels corresponding to the same position (that is, the corresponding point) of the same object based on the corresponding point information obtained by the corresponding point estimation process and the pixel value. ) To the chromaticity diagram. Details of the mapping will be described later with reference to FIG. Mapping is performed for at least one corresponding point included in the region used for light source color estimation.

  In S420, the camera control unit 5 determines whether or not the mapping process of S410 has been performed for all regions used for light source color estimation. Note that if there is one region used for light source color estimation, S420 can be omitted. However, for example, it is possible to improve estimation accuracy by setting a plurality of regions having different colors as regions used for light source color estimation and estimating the final light source color based on the light source color estimated in each region. it can.

  In general, it is considered most important to correctly recognize the color of the main subject. For this reason, the main subject region in the image is identified and set as the region used for light source color estimation using information on the focus region and focus detection region, or using subject detection results such as face detection and person recognition. can do. For example, the focused area may be regarded as a main subject area and set as an area used for light source color estimation, or a plurality of areas having greatly different saturations may be set as an area used for light source color estimation in the focused area. it can.

  In S430, the camera control unit 5 determines the estimated color of the light source based on maximum likelihood estimation or the like. As in the example described later with reference to FIG. 1B, when mapping is performed for a plurality of corresponding points, individually estimated light source colors may not match. In such a case, the camera control unit 5 determines the light source color based on maximum likelihood estimation such as a least square method.

  The pixel value mapping to the chromaticity diagram and the light source color estimation principle will be described with reference to FIG. Here, the xy chromaticity diagram defined as the CIE standard color system is used as the chromaticity diagram. However, there is a condition that the chromaticity of reflected light (reflected light obtained by combining diffuse reflected light and specular reflected light) for the same object color is located on a straight line connecting the object color and the light source color on the chromaticity diagram. Any color system that satisfies can be used.

FIG. 1A is a diagram illustrating a light source color estimation method when it is assumed that the light source can be approximated by black body radiation.
71 is a line drawn on the xy chromaticity diagram by the color of the single wavelength light (the color with the highest saturation), and 72 to 74 are coordinates of the color observed when the same point on the object is viewed from different viewpoints ( 75 indicates a straight line calculated from coordinates 72-74. Reference numeral 76 denotes a black body locus, and 77 denotes chromaticity coordinates of the estimated light source color.

  A black body is an ideal object that completely absorbs and radiates the energy of electromagnetic waves incident from the outside regardless of the wavelength. If the absolute temperature [K] of the black body is determined, An object whose spectral distribution (that is, color) is uniquely determined. A low color temperature (about 1000K) has a reddish color, an intermediate color temperature (about 6000K) has a white color, and a high color temperature (about 10,000K) has a bluish color. The black body locus 76 shows the color change of the black body radiation according to the color temperature in the xy chromaticity diagram. Sunlight is known to be very well approximated by blackbody radiation, and outdoor light sources (excluding artificial light sources) can be approximated by blackbody radiation. Information representing the black body locus on the chromaticity diagram is stored in, for example, the memory unit 8.

  Coordinates 72 to 74 are examples of colors observed when the same point on the object mapped from S410 in the light source color estimation process is viewed from different viewpoints.

  As shown in FIG. 4 and Formula (3), the magnitude of the specular reflection component included in the reflected light from one point on the object varies depending on the viewpoint. Therefore, when the same position on the object is observed from different viewpoints, it looks different. As described above, since the size of the diffuse reflection component does not depend on the observation position, the color difference depending on the viewpoint position is due to the difference in the specular reflection component size. When colors that have the same diffuse reflection component and are observed in different colors depending on the size of the specular reflection component are mapped (plotted) to the chromaticity diagram, a straight line (straight line) is defined by their coordinates (in this example, coordinates 72 to 74). 75) can be specified. A straight line 75 indicates an apparent color locus according to the magnitude of the specular reflection component observed for a specific object color. Therefore, both the object color and the light source color exist on the straight line 75. Therefore, when the light source can be approximated by black body radiation, the intersection 77 of the straight line 75 and the black body locus 76 corresponds to the light source color.

  When a straight line that passes through all coordinates cannot be obtained, such as when three or more points are mapped, the camera control unit 5 can determine the straight line 75 by a method such as a least square method. And the camera control part 5 calculates | requires the intersection 77 of the black body locus | trajectory 76 and the straight line 75 as an estimated color of a light source.

  When the light source can be well approximated by black body radiation, it is easier to estimate the light source color using the black body locus, but with black body radiation, the estimation accuracy decreases for a light source with low approximation accuracy. FIG. 1B is a diagram illustrating a method for estimating a light source color based only on information obtained from an image without assuming that the light source can be approximated by black body radiation.

  In FIG. 1 (b), the same reference numerals are assigned to components common to FIG. 1 (a). Coordinates 82 to 84 indicate colors observed when a point on an object having a color different from the coordinates 72 to 74 is viewed from different viewpoints, and 85 indicates a straight line obtained from the coordinates 82 to 84. The coordinates 86 to 88 are colors observed when a point on the object having a color different from the coordinates 72 to 74 and the coordinates 82 to 84 is viewed from different viewpoints, and 89 is a straight line obtained from the coordinates 86 to 88. Indicates.

  In the example of FIG. 1B as well, the light source color is specified as coordinates on a straight line obtained from a plurality of coordinates having different apparent colors depending on the magnitude of the specular reflection component, as in the example of FIG. It is. In the example of FIG. 1B, a plurality of straight lines relating to different object colors are obtained instead of the black body locus, and the coordinates of the intersection of the plurality of straight lines are specified as the light source color. Specifically, straight lines 75, 85, and 89 are obtained for a plurality of object colors, and the intersection of the straight lines is set as the estimated light source color. If the straight lines do not intersect at one point, the coordinate 77 having the smallest sum of the distances from the three straight lines is obtained as the closest coordinate to the intersection, and is used as the estimated color of the light source. Although three straight lines are obtained here, the light source color can be estimated based on an arbitrary number of two or more straight lines.

  In the example of FIG. 1B, intersections of a plurality of straight lines obtained from the mapped pixel values are obtained. Therefore, a plurality of regions having different colors are mapped as regions used for light source color estimation.

  Further, in this embodiment, regardless of whether the light source can be approximated by black body radiation (whether the black body locus is used for estimation) or not, the light source color is obtained using a straight line obtained from the coordinates of the mapped pixel values. Is estimated. Therefore, it is easier to obtain a correct straight line if the coordinates defining the straight line are somewhat apart on the chromaticity diagram, and the estimation accuracy is improved. For example, among a plurality of parallax images, an image pair in which the magnitude of the specular reflection component is likely to change significantly, such as an image pair with the most distant viewpoint, can be used for mapping. In the corresponding point estimation process, pixels in a region where the similarity is locally low in the search range may be used for mapping.

  As described above, in this embodiment, by using a plurality of images (multi-viewpoint images) obtained by observing points on the object from different viewpoints, for example, the light source color is accurately obtained from the reflected light of the object whose color continuously changes. Can be estimated well. Note that the multi-viewpoint image may be acquired by using a multi-view camera or moving the shooting position of one camera. In addition, a recorded multi-viewpoint image can be used if information such as a viewpoint position at the time of shooting, a pixel pitch, and a subject distance is recorded. On the other hand, by using an image sensor as used in the present embodiment, there is an advantage that a multi-viewpoint image can be acquired by one shooting with one camera.

(Other embodiments)
FIG. 7 schematically illustrates a configuration example of a modeling system using the camera system described in the above-described embodiment as a 3D scanner. The modeling system includes a camera system 100 as a 3D scanner, a personal computer (PC) 200 that is an example of an image processing apparatus, and a three-dimensional modeling apparatus 300 such as a 3D printer. The camera system 100 and the PC 200 are connected by a cable 150, and the PC 200 and the three-dimensional modeling apparatus 300 are connected by a cable 250, respectively. Note that devices may be connected using wireless communication instead of cables.

  For example, the camera control unit 5 transmits the information recorded in S180 of FIG. For example, the PC 200 generates a distance image from a multi-viewpoint image, estimates the color of an environmental light source, or performs white balance adjustment based on the estimated light source color, and generates modeling data to be used by the three-dimensional modeling apparatus 300. Generate. One or more of the distance image, the light source color, and the object color may be acquired from the camera system 100. Note that the user may acquire an image of the entire circumference of the subject using the camera system 100 and transmit it to the PC 200 as necessary. The three-dimensional modeling apparatus 300 generates a three-dimensional object based on the modeling data generated by the PC 200. In such a modeling system, it is possible to estimate a light source color with a simple configuration and provide a modeled object that faithfully reproduces the color.

  The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

DESCRIPTION OF SYMBOLS 2 ... Lens unit, 3 ... Imaging optical system, 5 ... Camera control part, 7 ... Image processing part, 8 ... Memory part, 9 ... Display part, 10 ... Operation detection part, 12 ... Lens control part, 100 ... Camera system

Claims (13)

Obtaining means for obtaining a plurality of images obtained by photographing an object from different viewpoints;
Detecting means for detecting corresponding points of the plurality of images;
Estimating means for estimating a color represented by a specific coordinate on a straight line obtained from chromaticity coordinates of a pixel value of the corresponding point as a light source color at the time of photographing the plurality of images;
An image processing apparatus comprising:   The image processing apparatus according to claim 1, wherein the estimation unit estimates a color represented by a coordinate of an intersection between a black body locus and the straight line as the light source color.   The image processing apparatus according to claim 1, wherein the estimation unit estimates a color represented by coordinates of intersections of a plurality of straight lines obtained from pixel values of the plurality of corresponding points as the light source color.   The said estimation means estimates the color which the coordinate from which the distance from the said some straight line becomes the minimum represents the said light source color, when the said some straight line does not cross | intersect at one point. Image processing apparatus.   5. The image processing according to claim 1, wherein the estimation unit obtains the straight line using pixel values of corresponding points included in a focus area of the plurality of images. 6. apparatus.   5. The image processing according to claim 1, wherein the estimation unit obtains the straight line by using pixel values of corresponding points included in a main subject region of the plurality of images. 6. apparatus.   The image according to any one of claims 1 to 6, wherein the acquisition unit acquires the plurality of images using an imaging device including a plurality of photoelectric conversion regions sharing a microlens. Processing equipment.   The image processing apparatus according to claim 1, wherein the estimation unit further estimates an object color at the corresponding point based on the estimated light source color.   The image processing apparatus according to claim 1, wherein the detection unit further generates shape information of the object from the plurality of images.   The image processing apparatus according to claim 9, wherein the shape information is a distance image. The image processing apparatus according to any one of claims 1 to 10,
Generating means for generating modeling data for use by the three-dimensional modeling apparatus from the information generated by the image processing apparatus;
The three-dimensional modeling apparatus using the modeling data;
A modeling system characterized by comprising: An image processing method executed by an image processing apparatus,
Acquire multiple images of objects taken from different viewpoints,
Detecting corresponding points of the plurality of images;
Estimating a color represented by specific coordinates on a straight line obtained from chromaticity coordinates of the pixel value of the corresponding point as a light source color at the time of photographing the plurality of images;
An image processing method.   The program for functioning a computer as each means of the image processing apparatus of any one of Claims 1-10.

Images