JP2014010783A - Image processing apparatus, image processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce a blur of the area to be focused by stable processing so as to synthesize an image.SOLUTION: Masks 1003 and 1004 are applied to respective images 901 and 902 of actual imaging parts 701 and 702, and images that extract focusing areas are images 1005 and 1006 (Processing 2). The images thus obtained are images 1007 and 1008 projected onto a focusing surface and converted into the coordinates on the multi-view point image (Processing 3). If these images are superposed with each other, a virtual view-point image 1009 is obtained (Processing 4). If remained as they are, the images are shifted in the respective viewpoints due to the shift of the camera parameter. Thus, position adjustment is made so as to superpose these images with the image of one view point as a reference, and a correction parameter is calculated to describe a positional deviation such as that of a correction image 1010 (Processing (5)). This correction parameter is applied, similarly to a correction image 1011, entirely to the image obtained by the imaging part which is not the reference when synthesizing the virtual view point image, and thereby the image having no blur in the focusing area can be obtained (Processing 6).

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program, and more particularly to an image processing apparatus, an image processing method, and a program for processing images captured from a plurality of viewpoints.

  When shooting with a normal camera, focusing is performed at the time of shooting. Therefore, if the shooting is performed with incorrect focus position adjustment, re-shooting is necessary to obtain a shot image with the correct focus position.

  In recent years, a technique called light field photography has been developed in which a plurality of images are acquired from multiple viewpoints, and the focus position is adjusted by image processing after acquisition, that is, post-shooting image processing. Since this technique can adjust the focus position after shooting, there is an advantage that even if focus adjustment at the time of shooting fails, the image can be adjusted to the correct focus position by subsequent image processing. In light field photography, the direction and intensity of light passing through a certain position (light field, hereinafter abbreviated as LF) are acquired from a multi-viewpoint image at a plurality of positions in space. An image formed on a virtual sensor at an arbitrary position is calculated using the acquired LF information. Mathematical properties and mathematical foundations related to LF It is discussed by NG (for example, refer nonpatent literature 1).

  By adjusting the position of the virtual sensor as appropriate and adjusting it to a position where the desired subject is in focus, the above-described focus adjustment after photographing can be performed. Hereinafter, a process of calculating an image acquired at a virtual sensor position from a multi-viewpoint image is referred to as a refocus process. An image acquired in this way is called a virtual viewpoint image. As an image pickup apparatus for acquiring LF, a camera array in which small cameras are arranged, and a Plenoptic Camera in which a microlens array is placed behind a main lens are known. In any imaging apparatus, an image obtained when a sensor is virtually placed can be synthesized after shooting from the acquired LF. As a method of synthesizing images on a virtual sensor from LF, a method has been proposed in which a plurality of acquired images are subjected to projective transformation on a virtual sensor, and added and averaged (see, for example, Patent Document 1). ). Such an image composition method is called a synthetic aperture photographing method.

  In image synthesis using synthetic aperture photography, if camera parameters (camera position, orientation, angle of view, focal length, etc.) given as known information deviate from the actual values, you want to focus There is a problem that the area is blurred. There has been proposed a technique for correcting such a camera parameter shift causing image quality degradation by estimating a camera parameter using a camera calibration technique (see, for example, Non-Patent Document 2).

International Publication No. 2008/050904 Pamphlet

R. NG, M.M. Levoy, M.M. Bredif, G.M. Duval, M.M. Horowitz, P.M. Hanrahan, “Light Field Photographic With a Hand-Held Plenoptic Camera” (Stanford Tech Report CTSR 2005-02, 2005) Richard Szeliski, “Computer Vision: Algorithms and Applications”, Springer, New York, 2010

  However, the conventional camera calibration technique used in Non-Patent Document 2, for example, has a problem in that the user needs to perform complicated work when using a chart. Further, there is a problem in that the process becomes unstable when calibrating from a natural image.

  The image processing apparatus of the present application combines and converts each of a plurality of image data captured from different preset viewpoints based on the position of the captured viewpoint, and captures a virtual viewpoint image captured from a virtually set viewpoint Image synthesizing means for generating composite image data, focus area extracting means for extracting an area corresponding to the focus area in the composite image data from each of the plurality of image data, and each area of the plurality of image data The image data is converted so that the difference between the pixel values of the image data becomes smaller, and alignment means for calculating the difference from the image data synthesized and converted based on the position of the preset viewpoint is used, and the calculated difference is used. It is characterized in that composite image data is generated by converting the entire combined image data of a plurality of image data.

  When compositing a virtual viewpoint image from a multi-viewpoint image by refocus processing, even if there is a deviation from the actual value of a given camera parameter, the blur of the area to be focused is reduced with stable processing Thus, it is possible to synthesize an image.

It is a figure showing an example of the outline of the imaging device which can apply the present invention. 1 is a block diagram illustrating a configuration example of a multi-viewpoint imaging apparatus according to Embodiment 1. FIG. 3 is a block diagram illustrating a configuration example of a multi-viewpoint imaging unit according to Embodiment 1. FIG. 3 is a block diagram illustrating a configuration example of an imaging unit according to Embodiment 1. FIG. It is a figure showing the example of arrangement | positioning of an imaging part. It is the figure which looked at arrangement | positioning of each imaging part of one Example from the side. It is the figure which looked at arrangement | positioning of each imaging part of one Example from the upper part. It is a figure for demonstrating the relationship between the coordinate of each imaging part of one Example, and the coordinate of a multiview imaging part. It is a figure which shows the relationship of the to-be-photographed object, virtual imaging system, and multiview imaging part of one Example. It is a figure for demonstrating the property of an ideal optical system. It is a figure for demonstrating the principle of a virtual viewpoint image composition. It is a figure for demonstrating the principle of a virtual viewpoint image composition. It is a figure for demonstrating the principle of a virtual viewpoint image composition. It is a figure for demonstrating the principle of a virtual viewpoint image composition. It is a figure explaining the outline | summary of an example of the image composition process in this invention from the side of intermediate generation data. FIG. 3 is a block diagram illustrating a configuration example of an image processing unit in Embodiment 1. FIG. 3 is a diagram illustrating a flow of image composition processing in the first embodiment. It is a flowchart which shows an example of the flow of a position alignment process. FIG. 10 is a block diagram illustrating a configuration example of an image processing unit in Embodiment 2. FIG. 10 is a diagram illustrating a flow of image composition processing in the second embodiment. It is a flowchart which shows the process of an example of the method of alignment. It is a flowchart which shows an example of the effective block determination method. It is a flowchart which shows a process of an example of an effective motion vector determination method.

[Example 1]
<Overall configuration of imaging device>
FIG. 1 is a schematic diagram showing an example of a multi-viewpoint imaging apparatus to which the present invention can be applied. A member or device for capturing a multi-viewpoint image is attached to the housing 101 of the multi-viewpoint imaging device. A plurality of imaging units 105 to 113 are arranged on the front surface (a) of the multi-viewpoint imaging apparatus in FIG. 1, and thereby the multi-viewpoint imaging apparatus acquires images of a plurality of viewpoints. A shooting button 102, a display 103, and an operation button 104 are arranged on the rear surface (b) of the multi-viewpoint imaging apparatus of FIG. Various settings of the multi-viewpoint imaging apparatus are performed using the operation button 104 and the display 103, and when the shooting button 102 is pressed, the imaging units 105 to 113 perform imaging. The acquired image and the synthesized image are displayed on the display 103.

  FIG. 2 is a block diagram showing an example of the configuration of the multi-viewpoint imaging apparatus according to an embodiment of the present invention. Although the details will be described later, the multi-viewpoint imaging unit 213 includes a plurality of imaging units, and acquires images from a plurality of viewpoints. Each imaging unit constituting the multi-viewpoint imaging unit 213 converts an image formed by the optical system into image data by an image sensor as described later.

  Although details will be described later, the image processing unit 214 performs refocus processing from the multi-view image data acquired from the multi-view image capturing unit 213 to generate composite image data. The CPU 201 is involved in all the processes of each configuration, reads the instructions stored in the ROM 202 and the RAM 203 in order, interprets them, and executes the processes according to the results. The ROM 202 and RAM 203 provide the CPU 201 with programs, data, work areas, and the like necessary for the processing. The bus 210 functions as a path for exchanging data and processing instructions between the components.

  The image processing unit 214 performs image composition based on information on arrangement and characteristics of each imaging unit that is the position of a plurality of viewpoints constituting the multi-viewpoint imaging unit. It is assumed that these pieces of information are calculated in advance based on system and apparatus settings and stored in the ROM 202 or the RAM 203. The operation unit 204 is a button or a mode dial, for example, and receives a user instruction input via these buttons. The character generation 209 generates characters and graphics. Generally, a liquid crystal display is widely used as the display unit 206, and displays captured images, composite images, characters, and the like received from the character generation unit 209 and the display control unit 205. Further, a touch screen function may be provided, and in that case, a user instruction using the touch screen can be used as an input of the operation unit 204.

  The digital signal processing unit 212 performs γ adjustment of the multi-viewpoint image data acquired by the multi-viewpoint imaging unit 213, interpolation of defective pixels, and the like. These processes are performed by the image processing unit 214 before image processing, which is a feature of this embodiment. The encoder unit 211 performs an encoding process on the output multi-viewpoint image data. The media interface 207 is an interface for coupling the apparatus to a PC or other media (for example, PC, hard disk, memory card, CF card, SD card, USB memory, etc.) 208. For example, composite image data and multi-viewpoint image data are output via the media interface 207. Usually, the components of the apparatus are present in addition to the above, but the description thereof is omitted because they are not particularly necessary for explaining the present invention.

<Configuration of multi-viewpoint imaging unit>
A configuration example of the multi-viewpoint imaging unit 213 will be described with reference to the block diagram of FIG. The imaging units 301 to 309 acquire multi-viewpoint image data including a plurality of image data by capturing images from a single viewpoint that is arranged spatially different from each other. The imaging units 301 to 309 will be described as corresponding to the imaging units 105 to 113 in FIG.

  An example of the configuration of each of the imaging units 301 to 309 will be described with reference to FIG. The lens 401 and the lens 402 constitute an imaging optical system. Light rays emitted from the subject are collected by this imaging optical system, and pass through the aperture 403, the IR cut filter 404, the low pass filter 405, and the color filter 406, and then form an image on the image sensor 407. The optical system control unit 410 controls the lens 401 and the lens 402 to change the focusing distance and the focal length as the imaging optical system. Further, the aperture 403 is opened and closed to adjust the F value. In the present embodiment, the control optical system control unit 410 of each of the image capturing units 301 to 309 controls the optical system of each image capturing unit, and the control value is collectively set by the image capturing control unit 310. The control values to be set are, for example, a focusing distance, a focal length, an F value, and the like, and are calculated by a setting value input by the user through the operation unit 204, or an automatic exposure device and an autofocus device that are provided separately. It is determined based on the value.

  Note that the configuration of the optical system shown here is a simplified example for explanation, and any configuration may be used as long as it has a function of forming an image of a subject on the image sensor. Although the configuration for acquiring a color image has been described, the acquired image can be an image having black and white or four or more pixels, and can be an image with different exposure for each pixel.

  As the image sensor 407 and the A / D converter 408, an image sensor such as a CMOS image sensor can be used. In general, the image sensor 407 has pixels arranged in a two-dimensional lattice pattern, and detects and aggregates light amounts on the pixels to convert an image formed into an electrical signal. The A / D converter 408 converts the information of the image converted into an electric signal into a digital signal. The image data converted into the digital signal is stored in the buffer 409. The image sensor control unit 411 controls the start of exposure and signal readout for the image sensor 407 and the A / D converter 408. Control values such as control timing are collectively set by the imaging control unit 310.

  As described above, the configuration of the general imaging unit has been described with reference to FIG. 4, but each of the imaging units 301 to 309 does not have to have the same configuration, and forms an image of a subject and acquires image data. Any design or configuration may be used as long as it has a function to do so.

  With reference to FIGS. 5 and 6, the arrangement of the imaging units 301 to 309 of the present embodiment will be described. Each imaging unit is stored in a cylindrical casing having the optical axis direction of the imaging optical system as an axis, and is arranged in a grid on the casing 501 of the multi-viewpoint imaging apparatus. The housing 501 corresponds to the housing 101 of FIG. In this embodiment, the optical axes of the image forming optical systems of the respective image pickup units are arranged in the same direction. FIG. 5 is a front view of the multi-viewpoint imaging apparatus. FIG. 6 is a diagram of the multi-viewpoint imaging device viewed from the left in FIG. FIG. 7 is a diagram of the multi-viewpoint imaging device viewed from above in FIG. In order to clarify the relationship between each figure, right-handed coordinate axes 502, 503 and 504 are shown in each figure.

  In the present embodiment, the case of nine image pickup units has been described. However, any number of image pickup units may be used as long as image pickup from a plurality of viewpoints is possible. Further, the arrangement of FIG. 5 is merely an example, and any arrangement may be used as long as the arrangement of each viewpoint can be known from the design or the measurement result. Furthermore, in this embodiment, each imaging unit is shown as a cylindrical shape, but any shape imaging device can be used.

  With reference to FIG. 8, how to describe the arrangement information of each imaging unit used in the following description, that is, the position information of each viewpoint, in this embodiment will be described. Here, an imaging unit that acquires an image of each viewpoint is considered as a simple perspective projection camera. As the coordinate system, an imaging unit coordinate system and a multi-viewpoint imaging unit coordinate system are considered, and the arrangement of viewpoints is expressed from these relationships. The relationship between the coordinate axes is a right-handed system.

The multi-viewpoint imaging unit coordinate system is a coordinate system corresponding to the multi-viewpoint imaging apparatus, and serves as a reference for the position and orientation of the viewpoint of each imaging unit. The imaging unit coordinate system is a coordinate system corresponding to each imaging unit, the z-axis direction representing the optical axis direction of each imaging unit, and the origin representing the optical center. The origin of the imaging unit coordinate system of the n-th imaging unit is expressed as t _n using the multi-view imaging unit coordinate system. The x-axis, y-axis, and z-axis of the imaging unit coordinate system of the n-th imaging unit are represented as v _xn , v _yn , and v _zn using unit vectors of the multi-view imaging unit coordinate system, respectively.

  In this embodiment, each imaging unit is arranged at a position close to a plane such that the z component is 0 in the multi-view imaging unit coordinate system, and the optical axis of each imaging unit is substantially z in the multi-view imaging unit coordinate system. It shall be facing the axial direction. The above imaging unit arrangement information is calculated by design values or calibration, and is stored in the ROM 202 or RAM 203 in advance.

  The multi-viewpoint imaging unit coordinate system is also a reference for the placement of a virtual imaging system that is reproduced by refocus processing. The virtual imaging system is expressed using a surface that is conjugate with the virtual image sensor, that is, a surface that is in focus on the image side. Such a surface is hereinafter referred to as a virtual in-focus surface. In this embodiment, the virtual focusing plane is assumed to be perpendicular to the z-axis of the multi-view imaging unit coordinate system, and the position of the virtual focusing plane is the distance d from the origin of the multi-view imaging unit coordinate system to the virtual focusing plane. It expresses using.

  Information representing the virtual in-focus plane is calculated from parameters input by the user via the user interface, and is stored in the RAM 203 in advance. As a specific calculation method, any method known in this technical field can be used.

  Note that the above-described arrangement of the imaging units and the representation of the virtual imaging system are merely examples, and the information about each imaging unit, the multi-view imaging device, and the arrangement of the virtual imaging system can be shown. Any expression is acceptable.

<Overview of image composition processing>
An overview of image composition processing performed by the image processing unit 214, which is a characteristic configuration of the present embodiment, will be described. The image processing unit 214 synthesizes an image as if it were captured by a desired virtual imaging system by processing images obtained by capturing the same scene acquired using the multi-viewpoint imaging unit 213 from different viewpoints. Can do. FIG. 9 is a diagram for explaining the arrangement of a subject, an imaging unit, and a virtual imaging system in the image composition process. In the scene shown in FIG. 9, there are subjects 704, 705, and 706 having different distances from the multi-viewpoint imaging unit 213, and these are imaged by the actual imaging units 701 and 702 that exist at different viewpoints. The imaging units 701 and 702 acquire light information in the space as an image, and use this to synthesize an image as if it was captured by a virtual imaging system 703. Hereinafter, a composite image as if it was captured by a virtual imaging system is referred to as a virtual viewpoint image.

  With reference to FIG. 10, an outline of a method for synthesizing a virtual viewpoint image will be described. The virtual imaging system combined in the image processing unit 214 is an imaging system having a finite depth of field. That is, in the imaging system 703 shown in FIG. 10, the light beam reflected by the subject such as natural light or illumination is condensed by the optical system 804, imaged on the sensor 805, and sampled to obtain an image. Shall. In order to synthesize an image acquired by such a virtual imaging system, it is only necessary to reproduce the imaging process by the optical system 804 and the sampling process by the sensor 805 using the light beams acquired by the imaging units 701 and 702. . The pixel value of the pixel 807 on the sensor 805 is obtained by adding the light rays incident on the pixel 807. If the optical system 804 is an ideal optical system, a light beam incident on the pixel 807 on the sensor 805 always passes through a conjugate point 808 with respect to the pixel 807 on the focusing surface 806 that is a conjugate surface of the sensor 805. Accordingly, the image obtained by projecting the images acquired by the imaging units 701 and 702 on the in-focus surface is deformed according to the coordinates of the virtual viewpoint image according to the relationship between the in-focus surface 806 and the sensor 805, and this is applied to all viewpoints. Are added to obtain a virtual viewpoint image. Actually, in order to maintain the brightness of the image, averaging is performed from all viewpoints instead of simple addition.

  FIG. 11 shows an image obtained by projecting the images 901 and 902 shown in FIG. 12 acquired by the imaging units 701 and 702 by setting a focal plane 903 of a virtual imaging system at a distance where the subject 704 exists. FIG. The subject 704 is a focused picture so that it is projected at the same position in both the imaging units 701 and 702. On the other hand, since the subjects 705 and 706 are not on the focusing plane 903, the projected position is shifted and the picture is blurred. The blurred state in FIG. 11 is shown with the subjects 705 and 706 doubled. This is a reproduction of a finite depth of field, and the image processing is performed so that the subject on the in-focus plane is focused and the subject on the other plane is blurred.

  In the above-described method for synthesizing a virtual viewpoint image, it is assumed that camera parameters such as the arrangement, orientation, and angle of view of the imaging units 701 and 702 are known. However, when these camera parameters actually change for some reason, as shown in FIG. 14, a deviation occurs between the given value and the actual value, and the in-focus state as shown in FIG. When projected onto the surface, the subject 704 to which the images of all viewpoints should overlap is also a blurred picture. Therefore, it is necessary to reduce this blur by correction processing.

  The outline of the correction process of this embodiment will be described with reference to FIG. In this embodiment, the distance of the subject is calculated from the multi-viewpoint image. Images 1001 and 1002 are schematic diagrams of image data obtained based on subject distance information corresponding to the imaging units 701 and 702, respectively. From the distance information, an area on the image where the subject, in this embodiment the subject 704 exists, is extracted on the in-focus plane. Schematic diagrams of extracted region masks corresponding to the imaging units 701 and 702 are masks 1003 and 1004 (processing (1)). Here, it is assumed that the in-focus surface is on the subject 704.

  Images 1005 and 1006 are obtained by applying the masks 1003 and 1004 obtained above to the images 901 and 902 of the actual imaging units 701 and 702 as shown in FIG. Process (2)). Images 1007 and 1008 are obtained by projecting the image thus obtained onto the in-focus plane and converting it into coordinates on the multi-viewpoint image (processing (3)). When these are superimposed, a virtual viewpoint image 1009 is obtained (processing (4)). As it is, the image is shifted between the viewpoints due to the shift of the camera parameter. The positions are aligned with reference to the image of one viewpoint so that they overlap, and a correction parameter describing a positional deviation like the corrected image 1010 is calculated (process (5)). By applying this correction parameter to the entire image like the corrected image 1011 to the image obtained by the imaging unit that is not the reference at the time of virtual viewpoint image synthesis, an image without blur in the in-focus area is obtained (processing) (6)). Here, for the sake of explanation, two imaging units are used, but any number of imaging units may be used.

<Configuration of image processing unit>
A configuration example of the image processing unit 214 will be described with reference to FIG. The multi-view image acquisition unit 1101 acquires multi-view image data captured by the multi-view image capturing unit 213 via the bus 210. The viewpoint information acquisition unit 1102 acquires information about the arrangement and characteristics of each imaging unit constituting the multi-viewpoint imaging unit 213 set based on the design value and the calibration value via the bus 210. The virtual imaging system information acquisition unit 1103 acquires information about the virtual imaging system preset by the user via the bus 210.

  As will be described in detail later, the pixel position conversion parameter calculation unit 1104 sets a parameter indicating which position on the image generated by the image synthesis unit 1108 the pixel on the image captured from each viewpoint corresponds to the viewpoint. It calculates from information and virtual imaging system information. Although details will be described later, the distance calculation unit 1105 calculates the subject distance for each pixel on the image captured from each viewpoint from the multi-viewpoint image and the viewpoint information. Although the details will be described later, the focus area extraction unit 1106 is imaged from each viewpoint in the virtual imaging system from the subject distance calculated by the distance calculation unit 1105 and the virtual imaging system information acquired by the virtual imaging system information acquisition unit. The region to be focused on is determined. Image data of the specified region is extracted from image data captured from each viewpoint constituting the multi-viewpoint image.

  The alignment unit 1107 converts the focus area on the image captured from each viewpoint extracted by the focus area extraction unit 1106 into a virtual viewpoint image based on the parameter calculated by the pixel position conversion parameter calculation unit. Although the details will be described later on the basis of an image of one viewpoint among the converted images, alignment parameters representing conversion that overlaps with other images are calculated. The image composition unit 1108 converts the image of each viewpoint into a virtual viewpoint image based on the pixel position conversion parameter, and further performs correction by applying the alignment parameter. Although details will be described later, the virtual viewpoint images are synthesized by superimposing the images thus obtained over all the viewpoints. The image output unit 1109 outputs the image data of the image synthesized by the image synthesis unit 1108 to the buffer 210.

  An example of the flow of image composition processing will be described with reference to FIG. In step S1201, a plurality of image data is acquired as a multi-viewpoint image. In step S1202, information regarding each imaging unit that has acquired multi-viewpoint image data is acquired. In step S1203, information related to the virtual imaging system set by the user is acquired. In step S1204, as will be described in detail later, based on the viewpoint information acquired in step S1202 and the virtual imaging system information acquired in step S1203, at which position on the virtual viewpoint image the pixel of the image captured from each viewpoint is located. Judge whether it corresponds. In accordance with the determined correspondence, a pixel position conversion parameter for converting (synthesizing) an image captured from each viewpoint is calculated.

  In step S1205, although details will be described later, the subject distance is calculated for each pixel on the image captured from each viewpoint from the multi-viewpoint image acquired in step S1201 and the viewpoint information acquired in step S1202. In step S1206, although details will be described later, based on the virtual imaging system information acquired in step S1203 and the subject distance calculated in step S1205, an area where the virtual imaging system on the image captured from each viewpoint is in focus is determined. Extract. In step S1207, although the details will be described later, based on the pixel position conversion parameter calculated in step S1204, the in-focus area extracted in step S1206 is converted into a virtual viewpoint image. Using the image captured from one viewpoint among the converted images as a reference image, alignment parameters for conversion so as to overlap with other images are calculated.

  In step S1208, although details will be described later, the image of each viewpoint is converted into a virtual viewpoint image based on the pixel position conversion parameter calculated in step S1204, and further corrected in step S1207 by applying the alignment parameter. I do. The virtual viewpoint image data is synthesized by superimposing the images captured by all the imaging units thus obtained. In step S1209, the virtual viewpoint image data synthesized in step S1208 is output.

  In this embodiment, the subject distance is calculated from the multi-viewpoint image, but another calculated subject distance may be acquired from the outside.

<Pixel position conversion parameter calculation method>
The image processing unit 214 uses the multi-viewpoint image captured by the multi-viewpoint imaging unit 213 to perform virtual viewpoint image composition processing for compositing an image as if it was captured by a virtual imaging system. A calculation method of the pixel position conversion parameter will be described below. In general, multi-viewpoint images have scene ray information, so if you recreate the process of imaging with a virtual optical system and imaging with a virtual image sensor, is the image captured with a virtual imaging system? An image like this is obtained. For this reproduction, pixel position conversion parameters are required.

  Due to the nature of the imaging optical system, light rays that pass through a certain point on the image side always pass through a conjugate point on the object side. Therefore, by considering a conjugate plane on the object side of the virtual sensor with respect to the virtual optical system, and adding the light rays that pass through each point on the conjugate plane, an image as if it was captured by a virtual imaging system can be obtained. This conjugate plane is a plane that is in focus on the object side of the virtual imaging system, and is called a virtual focusing plane here. A composite image is obtained by sampling the light rays added on the virtual in-focus plane. The pixel position conversion parameter is a parameter for converting coordinates on each image of the multi-viewpoint image into coordinates on the virtual viewpoint image.

An example of a pixel position conversion parameter calculation method will be described. First, the horizontal field angle of the n-th imaging unit is θ _{x n} , the vertical field angle is θ _{y n} , the horizontal pixel number is s _{x n} , the vertical pixel number is s _{y n} , the horizontal pixel number is s _{x n} , and the vertical pixel is the number s _{y n,} the optical center position on the image as a _{(c x n, c y n} ), representing the internal parameter matrix a _n by the equation (1).

It defines the internal parameter matrix A _v of the virtual imaging system to reproduce as well by the virtual viewpoint image synthesis processing. The horizontal angle of view is θ _{x n} , the vertical angle of view is θ _{y v} , the number of horizontal pixels is s _{x v} , the number of vertical pixels is s _{y v} , the number of horizontal pixels is s _{x v} , the number of vertical pixels is s _yv , Assuming that the optical center position is (c _{x v} , _{cy v} ), the internal parameter matrix A _v is expressed as in equation (2).

Further, using the notation of FIG. 6, a 4 × 4 coordinate transformation matrix M _n from each imaging unit coordinate system to the multi-viewpoint imaging unit coordinate system is expressed as in Expression (3).

Also, a 3 × 3 rotation matrix R _n is expressed as shown in Equation (4).

When the coordinate of a certain pixel in a certain camera is (x, y) and the homogeneous coordinate x _h of (x, y) as shown in equation (5) is used, the corresponding ray r is as shown in equation (6). It is expressed in

Here, k _{(x, y)} is a parameter representing the trajectory of the light beam. K _{(x, y)} corresponding to the intersection of the ray and the virtual focal plane, the distance d from the origin of the multi-viewpoint imaging unit coordinate system to the virtual focal plane, and the z axis that is the normal unit vector of the virtual focal plane relation to the direction unit vector e _z is as equation (7).

A solution of this to 1 / k _{(x, y)} is as shown in equation (9) using B _n shown in equation (8).

Substituting equation (9) into equation (6), applying the A _v from the left, homogeneous coordinates representing points obtained by projecting the intersection of the ray and the virtual focus plane coordinates on the synthesized image is obtained. Since homogeneous coordinates is also multiplied by a constant it is equivalent to those obtained by dividing the both sides of the equation (6) with k _{(x, y),} substituting the equation (9), even when allowed to act A _v from the left Then, a homogeneous coordinate x ′ _h representing a point obtained by projecting the intersection of the ray and the virtual focal plane to the coordinate on the composite image is obtained as shown in Expression (10).

λ is a coefficient representing indefiniteness and is a certain scalar. Expression (10) corresponds to projective transformation of an image. In the present embodiment, a collection of matrices obtained by calculating the projection transformation matrix H _n as shown in Expression (11) for all imaging units is handled as a pixel arrangement parameter.

  It should be noted that the pixel arrangement parameter and the calculation method thereof according to the present embodiment are merely examples, and any method may be used as long as it defines the correspondence between the coordinates on the image obtained by the multi-viewpoint imaging unit and the coordinates on the composite image. It may be anything.

<Distance calculation method>
The distance calculation unit 1105 calculates the subject distance for each pixel on the image captured from each viewpoint from the multi-viewpoint image. A method for estimating the distance using images captured from a plurality of different viewpoints is described in detail in Chapter 11 of Non-Patent Document 2, and an appropriate method according to the system configuration may be used. As an example, there is a method in which the virtual camera is set to be the same as the viewpoint itself for which the subject distance is to be calculated, and the Plane Sweep method is performed. It should be noted that the reciprocal of the distance or the like is actually obtained so that the distance of the subject at an infinite distance can be handled.

<Focus area extraction method>
The in-focus area extraction unit 1106 uses the subject distance calculated by the distance calculation unit 1105 and the virtual imaging system information to extract an area in which the virtual imaging system is in focus on the image captured from each viewpoint. That is, the virtual focusing plane given as virtual imaging system information is converted into each imaging unit coordinate system, and the distance to the virtual focusing plane is obtained for each pixel on the image captured from each viewpoint, and compared with the subject distance. Thus, it is determined whether or not the pixel belongs to the in-focus area of the virtual imaging system.

If the homogeneous coordinates of a point on the virtual in-focus plane is X _v , the plane can be expressed as in equation (14) using the homogeneous coordinates L _v of the plane corresponding to the virtual in-focus plane.

Here, as described with reference to FIG. 8, if the virtual focal plane is perpendicular to the z axis and the distance from the origin is d, l ₁ = 0, l ₂ = 0, l ₃ = 1, l ₄ = −d. When L _v is converted into the homogeneous coordinates L _n of the virtual focusing plane in the imaging unit coordinate system of the n-th imaging unit, Equation (15) is obtained using Equation (3).

If the homogeneous coordinates of a certain point on the image of the n-th imaging unit are set to x _n , if the distance to the focal plane is d _n , the imaging unit coordinate system of the intersection of the focal plane and the light beam passing through the point is used. The homogeneous coordinate X _n is as shown in Equation (16).

  Expression (17) holds from the relationship between the homogeneous coordinates of the plane and the points on the plane.

Solving for 1 / d _n, it is as Equation (18), distance information to the focusing surface can be obtained.

The in-focus area is determined by using the threshold ε as an equation ( _where the distance of the subject corresponding to the point x _n on the n-th imaging system image calculated by the distance calculation unit 1105 is d ( _n )) ( 19) is processed as generation of a mask _mn based on.

  The extracted in-focus area is handled as data in the form of an image of each viewpoint and a set of in-focus area determination masks associated with each viewpoint.

<Positioning method>
The alignment unit 1107 converts the in-focus area of the imaging unit serving as each viewpoint extracted by the in-focus area extraction unit 1106 onto a virtual viewpoint image based on the parameters calculated by the pixel position conversion parameter calculation unit 1104. Using the image captured from one viewpoint among the converted images as a reference image, alignment parameters for conversion so as to overlap with other images are calculated.

  With reference to FIG. 18, a method for calculating the alignment parameter will be described. In step S1301, image data picked up from each viewpoint constituting a multi-viewpoint image and a corresponding focus area determination mask are acquired. In step S1302, a pixel position conversion parameter for converting an image captured from each viewpoint onto a virtual viewpoint image is acquired. In step S1303, the image data captured from each viewpoint acquired in step S1301 and the corresponding focus area determination mask are converted into a virtual viewpoint image using the pixel position conversion parameter acquired in step S1302. In step S1304, an imaging unit that captures an image serving as a reference for calculating the alignment parameter, that is, a reference viewpoint is selected.

  Steps S1305 to S1307 are repetitive processes, in which an imaging unit that captures an image to be corrected, that is, a process is performed a plurality of times while changing the viewpoint of the correction target, and the images captured for all imaging units are corrected. In step S1305, a correction target viewpoint is selected from unprocessed viewpoints. In step S1306, alignment is performed so that the in-focus area of the reference viewpoint image converted into the virtual viewpoint image in step S1303 overlaps the in-focus area of the correction target image. The alignment method will be described later, but the parameter representing the result is the alignment parameter. In step S1307, it is determined whether all viewpoints have been processed. If there are unprocessed viewpoints, the process returns to step S1305. If all the viewpoints have been processed, the alignment parameter is output in step S1308. Note that a parameter representing conversion that does not move the position is output as an alignment parameter for the reference viewpoint image.

  A method of performing alignment using the alignment parameters for the image captured by each imaging unit calculated as described above will be described. Here, the subject corresponding to the in-focus area of the image to be aligned exists on a single plane called the in-focus plane in the subject-side space. Therefore, such alignment is possible by a simple deformation model such as projective transformation, affine transformation, or second-order polynomial transformation. For model-based registration, sparse corresponding point search is performed, model parameters are calculated from the results, and the displacement of the entire image is described parametrically to optimize the brightness difference between images. There are methods.

  The former sparse corresponding point search is detailed in Chapter 4 of Non-Patent Document 2. A method for calculating the model parameters is detailed in Chapter 6 of Non-Patent Document 2. In the case of performing alignment with these combinations, in the present embodiment, the in-focus area is excluded from the corresponding point search target by using the in-focus area determination mask.

  Also, the latter method of optimizing so as to reduce the luminance difference between images is Parametric motion described in Chapter 8 of Non-Patent Document 2. In the case of performing alignment by this method, in this embodiment, the in-focus area is excluded from the calculation target of the luminance difference between images by using the in-focus area determination mask.

  Hereinafter, an example will be described in which a motion vector for each block is obtained by block matching using a sum of absolute differences or a sum of squares of squares as a sparse corresponding point search, and a deformation of the entire screen is obtained using affine transformation.

  FIG. 21 is a flowchart showing an example of the alignment method. When the motion vector for each block is obtained, the focus area block determination in step S1601 and the effective block determination in step S1602 are performed as preprocessing. The focused area block determination is a determination as to whether or not a block for which a motion vector is to be obtained is included in the focused area, and only blocks included in the focused area are used in the subsequent processing. Although the details will be described later, the effective block determination is a process of excluding a block that may not be able to obtain a correct motion vector. In step S1603, a motion vector of the block is calculated. Here, a general block matching method will be described.

  In the block matching method, a sum of squares of differences or a sum of absolute differences between pixels in a block is used as a matching evaluation value. If the target block for which the vector is to be calculated is sequentially moved within the search range of the reference image, the evaluation value is determined. The position having the smallest evaluation value among all the evaluation values obtained in the search range is the position having the highest correlation with the symmetric block, and the movement amount is a motion vector. A method of obtaining the search range pixel by pixel is called full search. On the other hand, a method of obtaining a minimum evaluation value while thinning out the search range and then performing a detailed search for the vicinity thereof is called a step search. Step search is well known as a method for obtaining a motion vector at high speed. In step S1605, effective motion vector determination is performed. Although details will be described later, this is a process of excluding those obtained from the motion vectors that are determined to be incorrect. In step S1604, end determination is performed. When all blocks have been processed, affine parameters are calculated from valid motion vectors in step S1606.

  Next, details of affine parameter calculation will be described. Assuming that the center coordinates of the target block are (x, y) and the center coordinates of the block in the reference image have moved to (x ′, y ′) from the motion vector calculation result, these relationships are expressed by the equation (20). ).

  Here, a 3 × 3 matrix is an affine transformation matrix. When each element of the matrix is an affine parameter and a = 1, b = 0, d = 0, and e = 1, this conversion is parallel movement, c is the horizontal movement amount, and f is the vertical movement amount. It becomes. Further, the rotational movement at the rotation angle θ can be expressed by a = cos θ, b = −sin θ, d = sin θ, e = cos θ. Expression (20) can be expressed as general expression (21) in the form of a generalized matrix.

  Here, x and x ′ are 1 × 3 matrices, and A is a 3 × 3 matrix. When there are n effective motion vectors, the coordinate value of the target image can be expressed by an n × 3 matrix as shown in Expression (22).

  Similarly, the coordinate values after movement can be expressed by an n × 3 matrix as shown in Equation (23).

  Therefore, for n motion vectors, the expression is as in Expression (24).

  That is, if the affine matrix A in the equation (24) is obtained, it becomes the amount of displacement of the entire screen. When the affine matrix is calculated as a least squares solution of the error between the coordinate value of the target image and the coordinate value after movement for n motion vectors, the affine matrix is obtained as shown in Expression (25).

  Here, the effective block determination method will be described with reference to the flowchart of FIG. When trying to obtain a correlation between blocks by block matching, an image in the block needs to have some feature amount. A correct motion vector cannot be obtained in a flat block containing almost only a DC component. On the other hand, if the edge is included in the horizontal direction or the vertical direction, it is considered that matching becomes easy. The determination process of the flowchart shown in FIG. 17 is one method for excluding such a flat portion block. Here, the processing for one block will be described.

  First, in step S1701, the difference value between the maximum value and the minimum value is calculated for one horizontal line in the block. For example, assuming that the block size is composed of 50 × 50 pixels, the maximum value and the minimum value are obtained from the 50 pixels in the horizontal direction in the block, and the difference value is calculated. This is repeated for the number of horizontal lines, that is, 50 times. In step S1703, the maximum difference value is obtained from the 50 difference values. In step S1704, the preset Tx is compared with the maximum difference value.

  If the maximum difference value is smaller than the threshold value Tx, it is regarded as a block having no feature amount in the horizontal direction, and is determined as an invalid block in step S1705. If it can be considered that the feature amount is in the horizontal direction, the same verification is performed in the vertical direction. First, in step S1706, a difference value between the maximum value and the minimum value is calculated for one vertical line in the block. That is, the maximum value and the minimum value are obtained from 50 pixels in the vertical direction in the block, and the difference value is calculated. This is repeated for the number of vertical lines, that is, 50 times. In step S1708, the maximum difference value is obtained from the 50 difference values. In step S1709, the preset Ty is compared with the maximum difference value. If the maximum difference value is smaller than the threshold value Ty, the block is regarded as a block having no feature amount in the vertical direction, and is determined as an invalid block in step S1705. If the block has features in both the horizontal and vertical directions, it can be expected that accurate block matching is performed. Therefore, in step S1710, the block is determined to be an effective block.

  Next, the effective motion vector determination method will be described with reference to the flowchart of FIG. First, a motion vector is input in step S1801, and the occurrence frequency is calculated in step S1802. In step S1803, this process is repeated until the occurrence frequencies of all the motion vectors are obtained. When the processing is completed, the motion vector having the maximum occurrence frequency is obtained in step S1804. Next, in step S1805, a motion vector is input again. In step S1806, it is determined whether this motion vector is a motion vector with the maximum occurrence frequency or a motion vector in the vicinity thereof.

  In the case where a positional deviation occurs due to an error in the camera posture or the like, since the positional deviation can be generally expressed by a simple model, it is considered that many motion vectors are generated in the vicinity of the motion vector having the maximum occurrence frequency. Therefore, a motion vector included in these values is determined as an effective motion vector in step S1807, and a motion vector deviating from these values is determined as an invalid motion vector in step S1808. In step S1809, it is determined whether or not the processing has been completed for all the motion vectors, and the processing from step S1010 is repeated until the processing is completed.

<Virtual viewpoint image composition method>
As described above, the image composition unit 1108 converts the image captured by the imaging unit arranged at the position of each viewpoint into a virtual viewpoint image based on the pixel position conversion parameter. The virtual viewpoint image data is synthesized by superimposing the image data obtained by performing the correction by further applying the alignment parameter to the image data thus converted, over all viewpoints.

The conversion from the coordinates on the image of the n-th imaging unit to the coordinates on the virtual viewpoint image is expressed using a projective transformation matrix H _n described by Expression (11). Further, the conversion from the coordinates on the virtual viewpoint image corrected by the alignment calculated by the alignment unit 1107 to the coordinates on the virtual viewpoint image before the correction corresponding to the n-th imaging unit is performed by projective conversion. This is expressed using a matrix C _n . Therefore, the conversion H ′ _n ⁻¹ from the coordinates on the virtual viewpoint image to the coordinates on the image of the n-th imaging unit is expressed by Expression (26).

By using H ′ _n ⁻¹ to identify a pixel on the image captured by each imaging unit corresponding to each pixel on the virtual viewpoint image and calculating a pixel value by interpolation such as bilinear or bicubic, An image obtained by converting an image captured by the imaging unit onto a virtual viewpoint image is obtained. A virtual viewpoint image is obtained by adding and averaging them over all viewpoints. Here, the alignment parameter is described using a projective transformation matrix, but any model such as translational transformation, affine transformation, and higher-order polynomial transformation may be used.

  As described above, according to the present embodiment, when a virtual viewpoint image having a finite depth of field is synthesized from a multi-viewpoint image, even when there is a deviation from an actual value in a given camera parameter, Combining is possible while reducing blur in the in-focus area.

[Example 2]
In the first embodiment, the alignment unit 1107 uses a single deformation model, and performs alignment so that focused regions on images taken from different viewpoints overlap on the virtual viewpoint image. Calculated. However, if the in-focus area is distributed in a very narrow area on the image, the estimation of the model parameters becomes unstable, and in particular, pixels located at a position away from the position where the in-focus area exists on the image, A phenomenon may occur in which the position is converted to an unintended position by the correction. Therefore, in the second embodiment, flexible correction can be performed by switching the deformation model used in the alignment unit 1107 according to the distribution of the focused region on the image. In other words, a deformation model that emphasizes stability is used when the in-focus area is distributed in a narrow range, and a deformation model that can be expected to have higher correction accuracy when the in-focus area is distributed in a wide area. Adopted and allows appropriate correction in any case.

  FIG. 19 is a block diagram illustrating a configuration example of the image processing unit 214 of the present embodiment. The configuration of the present exemplary embodiment is a configuration in which an alignment method selection unit 1401 is added to the image processing unit of the first exemplary embodiment. Although the details will be described later, the alignment method selection unit 1401 selects a deformation model to be used by the alignment unit 1107 according to the distribution on the image of the in-focus area extracted by the in-focus area extraction unit 1106.

  FIG. 20 is a flowchart illustrating a procedure of image composition processing in the present embodiment. In this flowchart, step S1501 is added to the flowchart of the first embodiment shown in FIG. In step S1501, a deformation model used for alignment is selected in step S1207 in accordance with the distribution on the image of the in-focus area extracted in step S1206.

<Positioning method selection method>
A method for selecting a deformation model for alignment performed by the alignment method selection unit 1401 will be described. In general, the deformation model used for alignment is more stable as the degree of freedom is higher, but can deal with complex deviations. The degree of freedom is lower as it is more stable, but can only deal with simple deviations.

  The degree of freedom of translational transformation is 2, the degree of freedom of affine transformation is 6, and the degree of freedom of projective transformation is 8, and the degree of freedom increases in order. In this embodiment, description will be made assuming that selection is made from these three deformation models. Note that the deformation model used in the present embodiment is an example, and any transformation may be used as the deformation model.

  The deformation model is selected according to the distribution of the focus area extracted by the focus area extraction unit 1106. Assume that the focus area extraction unit 1106 outputs a focus area determination mask m (x, y) as the focus area extraction result. Here, x and y are coordinates on the image of the imaging unit. m (x, y) has a value of 1 for pixels determined to be in focus, and 0 for other pixels. In general, it is sufficient to determine the deformation model if there is a focused region extraction result of an image taken from one viewpoint. Here, the viewpoint selected by the alignment unit 1107 as the reference viewpoint is used. Using the focus area determination mask m (x, y), the dispersion p of the focus area can be calculated by Expression (27).

Here, S _x and S _y are the numbers of pixels in the x direction and y direction, respectively, on the image of the imaging unit, and w _x and w _y are the x coordinate and y coordinate of the center of gravity of the focused area, respectively. Threshold values ε1 and ε2 are set for the dispersion p in the in-focus region, and translational transformation is selected as p <ε1, affine transformation is selected as ε1 ≦ p <ε2, and projective transformation is selected as ε2 ≦ p. Etc. to switch.

  As described above, according to the present embodiment, when a virtual viewpoint image having a finite depth of field is synthesized from a multi-viewpoint image, even when a given camera parameter has a deviation from an actual value, It is possible to perform composition while reducing blur in the in-focus area with stable processing.

[Other Examples]
In the first and second embodiments, the case where a multi-viewpoint imaging unit is configured using a plurality of imaging units having a single viewpoint has been described. However, a configuration in which the imaging unit divides the single imaging element 407 illustrated in FIG. 4 into a plurality of regions and provides an optical system corresponding to the divided regions may be employed. A plurality of such imaging units may be used.

  The present invention can also be realized by executing the following processing. That is, software (program) for realizing the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, etc.) of the system or apparatus reads the program. It is a process to be executed. The present invention can also be realized by a plurality of processors cooperating to perform processing.

Claims (9)

Each of a plurality of image data captured from different preset viewpoints is combined and converted based on the position of the captured viewpoint to generate composite image data as a virtual viewpoint image captured from a virtually set viewpoint. Image synthesizing means,
An in-focus area extracting means for extracting an area corresponding to an in-focus area in the composite image data from each of the plurality of image data;
Positioning means for converting the image data so as to reduce a difference in pixel values of each region of the plurality of image data, and calculating a difference with the image data synthesized and converted based on the preset viewpoint position And
An image processing apparatus that generates combined image data by converting the entire combined image data of each of the plurality of image data using the calculated difference.   The image processing apparatus according to claim 1, wherein the entire conversion of the image data is performed by projective conversion or polynomial conversion of the image data.   3. The image processing apparatus according to claim 1, wherein a method of converting the entire image data differs depending on the shape of the area extracted by the focused area extracting unit.   The plurality of pieces of image data are a plurality of pieces of image data picked up while changing the arrangement of one image pickup device, or a plurality of pieces of image data picked up by a multi-viewpoint image pickup device including a plurality of image pickup units having different arrangements. The image processing apparatus according to any one of claims 1 to 3. Each of a plurality of image data captured from different preset viewpoints is combined and converted based on the position of the captured viewpoint to generate composite image data as a virtual viewpoint image captured from a virtually set viewpoint. Image compositing hand step,
An in-focus area extracting step for extracting an area corresponding to an in-focus area in the composite image data from each of the plurality of image data;
An alignment step of converting the image data so as to reduce a difference in pixel values of each region of the plurality of image data, and calculating a difference from the image data synthesized and converted based on the position of the preset viewpoint And
An image processing method comprising: generating the composite image data by converting the entire composite image data of each of the plurality of image data using the calculated difference.   6. The image processing method according to claim 5, wherein the image processing is performed by conversion of the entire image data, projective conversion of image data, or polynomial conversion. The method of converting the entire image data is as follows:
7. The image processing method according to claim 5, wherein the image processing method differs depending on the shape of the area extracted by the focus area extracting means. The plurality of image data are:
8. A plurality of pieces of image data picked up while changing the arrangement of one image pickup apparatus, or a plurality of pieces of image data picked up by a multi-viewpoint image pickup apparatus including a plurality of image pickup units having different arrangements. The image processing method according to claim 1.   A program for causing a computer to function as the image processing apparatus according to any one of claims 1 to 4.

Images