JP2017034388A - Dictionary generation method, dictionary generation apparatus, image processing method, image processing apparatus, and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To generate a light field dictionary to be used for generating a light field image, and to generate the light field image by referring to the light field dictionary.SOLUTION: A light field dictionary is generated by using light-ray information obtained by a light field camera or a method for acquiring/generating the other light-ray information. Further, a light field information is generated by restoring light-ray information from a general image not including the light-ray information while referring to the light field dictionary based on focus blur information or the like included in the image.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a dictionary generation method, a dictionary generation device, an image processing method, an image processing device, and an image processing program.

  Light field cameras that record light information including the direction of light entering the image sensor have been put into practical use, and new image processing using light information, such as focus adjustment after image capture and restoration of three-dimensional information. Research and development of technology and applications are actively being carried out. A method of recording light ray information has been present for a long time, and a method of photographing a large number of cameras closely arranged in synchronization is well known (for example, see Non-Patent Document 1). This method requires a system in which a large number of cameras of the same type are moved in synchronization with each other, and since strict camera calibration is required, it has been considered difficult to implement in terms of cost and labor.

  However, in a light field camera (for example, see Non-Patent Document 2) that has recently been sold to the general public, a large number of cameras are arranged by arranging microlens arrays before and after the main lens inside the camera. It is possible to record the light beam information in the same way as in the case of the above (it is possible to adopt a different configuration by changing the installation position of the microlens array).

B. Wilburn, N. Joshi, V. Vaish, E.-V. Talvala, E. Antunez, A. Barth, A. Adams, M. Horowitz, and M. Levoy, “High performance imaging using large camera arrays,” ACM Transactions on Graphics, vol. 24. p. 765, 2005. R. Ng, M. Levoy, G. Duval, M. Horowitz, and P. Hanrahan, “Light Field Photography with a Hand-held Plenoptic Camera,” Stanford Tech Rep. CTSR, pp. 1-11, 2005.

  Unlike a conventional camera, a light field camera with a microlens array can record light beam information. However, in this camera recording method, the spatial resolution and the angular resolution are a trade-off, and it is difficult to achieve the resolution required for photography with a conventional camera with a practically usable imaging device. In addition, the amount of light is insufficient due to attenuation of light by the microlens array, so that it is easily affected by noise, and in order to compensate for it, it is also affected by blurring. As described above, there is a problem that it is difficult to obtain a quality that can be easily achieved by a conventional camera in imaging using a light field camera.

  The present invention has been made in view of such circumstances, and a dictionary capable of generating a light field dictionary using light information obtained by a light field camera or another method of acquiring and generating light information. An object is to provide a generation method and a dictionary generation device. Also, an image processing method and image that can generate a light field image by restoring light information from a general image that does not contain light information by referring to a light field dictionary based on out-of-focus information included in the image. It is an object to provide a processing device and an image processing program.

  One aspect of the present invention is a dictionary generation method performed by a dictionary generation apparatus that generates a light field dictionary from a light field image used for dictionary learning, and includes focusing with an arbitrary position from the light field image. A focused image generating step for generating an image; a light field feature vector generating step for generating a light field feature vector representing a feature of the light field image from the light field image; and a feature of the focused image from the focused image A dictionary generation step including: an image feature vector generation step for generating an image feature vector representing the image; and a dictionary generation step for generating the light field dictionary used when generating a light field image from the light field feature vector and the image feature vector Direction It is.

  One aspect of the present invention is the dictionary generation method, wherein the image feature vector generation step estimates a blur parameter for estimating a parameter representing a focus blur of the focused image and sets the image feature vector.

  One aspect of the present invention is an image processing method performed by an image processing apparatus that inputs an input image not including light ray information and a light field dictionary generated in advance by the dictionary generation method, and generates a light field image. An image feature vector generating step for generating an image feature vector representing the feature of the input image from the input image, and a light field feature vector representing the feature of the light field image is estimated from the image feature vector and the light field dictionary. An image processing method comprising: a light field feature vector estimation step; and a light field image generation step for generating the light field image to be generated from the light field feature vector.

  One aspect of the present invention is the image processing method, wherein the image feature vector generation step estimates a blur parameter for estimating a blur parameter representing a focal blur of the input image and sets the image feature vector as the image feature vector.

  One aspect of the present invention is the image processing method, wherein in the light field image generation step, the light field feature vector matches a light field feature vector obtained from a newly restored light field image. The light field image is generated.

  One aspect of the present invention is a dictionary generation device that generates a light field dictionary from a light field image used for dictionary learning, and a focus that generates a focused image in which a focal length is adjusted to an arbitrary position from the light field image. A matched image generating means; a light field feature vector generating means for generating a light field feature vector representing the characteristics of the light field image from the light field image; and an image feature vector representing the characteristics of the focused image from the focused image. A dictionary generation device comprising: an image feature vector generation means for generating a light field dictionary; and a dictionary generation means for generating the light field dictionary used when generating a light field image from the light field feature vector.

  One aspect of the present invention is an image processing apparatus that inputs an input image that does not include light ray information and a light field dictionary generated in advance by the dictionary generation apparatus, and generates a light field image. Image feature vector generation means for generating an image feature vector representing the feature of the input image, and light field feature vector estimation for estimating a light field feature vector representing the feature of the light field image from the image feature vector and the light field dictionary And a light field image generating means for generating the light field image to be generated from the light field feature vector.

  One aspect of the present invention is an image processing program for causing a computer to execute the image processing method.

  According to the present invention, it is possible to easily generate a light field dictionary using light ray information obtained by a light field camera or other light ray information obtaining / generating method. Further, it is possible to easily generate a light field image by restoring light information from a general image that does not include light ray information by referring to the light field dictionary based on information such as out-of-focus information included therein. Is obtained.

It is a block diagram which shows the structure of the dictionary production | generation apparatus by one Embodiment of this invention. It is a flowchart which shows the processing operation of the dictionary production | generation apparatus 100 shown in FIG. It is a block diagram which shows the structure of the light field image generation apparatus by one Embodiment of this invention. 4 is a flowchart showing a processing operation of the light field image generation device 200 shown in FIG. 3.

  Hereinafter, a dictionary generation apparatus and a light field image generation apparatus according to an embodiment of the present invention will be described with reference to the drawings. First, the dictionary generation device will be described. FIG. 1 is a block diagram showing the configuration of the dictionary generation apparatus according to the embodiment. As shown in FIG. 1, the dictionary generation apparatus 100 includes a light field image input unit 101, a focused image generation unit 102, a patch generation unit 103, a light field feature vector generation unit 104, an image feature vector generation unit 105, and a learning unit. 106 is provided.

  The light field image input unit 101 inputs a light field image used for dictionary learning from the outside. Hereinafter, this light field image is referred to as an LF image. The focused image generation unit 102 generates a focused image in which the focal length is adjusted to an arbitrary position from the input LF image. The patch generation unit 103 divides the LF image and the focused image into patches having a predetermined size, and generates a light field patch group and an image patch group. Hereinafter, the light field patch group is referred to as an LF patch group.

  The light field feature vector generation unit 104 generates a light field feature vector group from the LF patch group, and the image feature vector generation unit 105 generates an image feature vector group from the image patch group. Hereinafter, the light field feature vector group is referred to as an LF feature vector group. The learning unit 106 generates and outputs a light field dictionary (hereinafter referred to as LF dictionary) from the LF feature vector group and the image vector group.

  Next, the processing operation of the dictionary generation device 100 shown in FIG. 1 will be described with reference to FIG. FIG. 2 is a flowchart showing the processing operation of the dictionary generation apparatus 100 shown in FIG. First, the light field image input unit 101 inputs an LF image used for dictionary learning (step S101). The LF image is an image in which information of light rays incident on a certain imaging system is recorded. The LF image may be expressed in any way.

  In general, a normal image is represented by a two-dimensional array in which pixels are arranged vertically and horizontally, whereas an LF image is a four-dimensional array in which two dimensions are added to express angles in two directions. expressed. When the spatial resolution is H (height) xW (width) and the angular resolution is NxM, the LF image can be represented by a four-dimensional array of HxWxNxM (Reference 1: “M. Levoy and P. Hanrahan,“ Light Field rendering, ”Proc. 23rd Annu. Conf. Comput. Graph. Interact. Tech.-SIGGRAPH '96, pp. 31-42, 1996.”).

  When generating an image focused on an arbitrary distance from the LF image, the resolution of the generated image is HxW unless special processing for increasing the resolution such as resampling or super-resolution is performed. In addition, it can be expressed as N × M multi-viewpoint images having a resolution of HxW. In this case, each viewpoint image is a collection of ray information as an image for each direction, and is the same as an image captured by each camera when the same ray information is captured using a number of cameras.

  Hereinafter, an image at each viewpoint in this expression is referred to as a sub-aperture image. Alternatively, an image recorded on an image sensor via a main lens and a microlens array may be directly input as an image captured by a general light field camera. In addition, any type of LF image may be input. A plurality of LF images may be input.

  Next, the focused image generation unit 102 generates a focused image in which the focal length is adjusted to an arbitrary position from the input LF image (step S102). The focal length may be the same as the focal length of the main lens used to capture the LF image, or may be a different focal length. When there are a plurality of input LF images, they may be adjusted to the same distance or different distances. In addition, any method may be used to create the focused image. Well-known methods include the shift addition method and the Fourier slice method (reference 2 “R. Ng,“ Fourier slice photography, ”ACM SIGGRAPH 2005 Pap.-SIGGRAPH '05, p. 735, 2005.”) There is.

  Next, the patch generation unit 103 divides the LF image and the focused image into patches having a predetermined size, and generates an LF patch group and an image patch group (step S103). The size of the patch may be any size. A plurality of patches may be overlapped. In the following description, it is assumed that a patch having a spatial resolution of wxh and an angular resolution of nxm is used. The positions in the spatial direction are matched between the paired LF patch and the image patch.

  Next, the light field feature vector generation unit 104 generates a light field feature vector group from the LF patch group, and the image feature vector generation unit 105 generates an image feature vector group from the image patch group (step S104). Each feature vector may be generated by any method. In the simplest case, images obtained by rearranging each image as a one-dimensional vector may be used. In addition, in order to specialize in the gradient vector of each image and the restoration of the LF image, a vector that includes a feature of defocusing may be generated and used.

と推定される。 For example, blur amount estimation or blur kernel estimation may be performed on the focused image by an arbitrary method, and the estimated amount, estimation kernel, or parameter thereof may be used as a feature vector. Any method of estimating the amount of blur may be used. For example, Reference 3 “S. Zhuo and T. Sim,“ Defocus map estimation from a single image, ”Pattern Recognit., Vol. 44, no. 9, pp 1852-1858, Sep. 2011. ", there is a method of comparing the gradient amounts of images that are Gaussian filtered with two different parameters. In this method, the focal blur is assumed to be Gaussian blur, and its parameters are estimated. If the original image is I, the parameters of an arbitrary Gaussian filter are σ ₁ , σ ₂ : σ ₁ <σ ₂ , the operator of the Gaussian filter is G (x, y, σ), and the gradient operator is ∇, the position (x , Y), the blur parameter σ (x, y) is

It is estimated to be.

  It is also possible to apply the same method to different models such as average blur. In addition to the method of estimating parameters assuming a model, Reference 4 “H. Zhang, J. Yang, Y. Zhang, and TS Huang,“ Sparse Representation Based Blind Image Deblurring, ”IEEE Int. Conf. Multimed. As shown in Expo, pp. 1-6, 2011., a method of directly estimating a blur kernel may be used, and the estimated blur kernel may be used as a feature vector.

  A plurality of feature vectors and images may be combined into one feature vector. For example, an image feature vector is obtained by vectorizing and combining a gradient vector and a blur kernel. However, as for the LF feature vector, it is necessary to be able to generate an LF image from information obtained from an image patch and a light field feature vector restored using an LF dictionary in a light field image generation device described later.

  For example, for each angle image of the LF patch, the difference between the LF patch and the image patch is used as a feature vector. Any feature vector may be selected as long as the condition is satisfied. Further, when various image processing is necessary for generating the feature vector, it may be performed before the patch generation.

  Next, the learning unit 106 generates and outputs an LF dictionary from the LF feature vector group and the image feature vector group (step S105). The LF dictionary may be generated by any method. The simplest method is to use a database that directly includes a pair of vectors as an LF dictionary. As a more preferable method, there is a method in which both vector groups are coupled together to form a coupled vector group, and a finite number of bases that best represent all coupled vectors are obtained by learning.

  As a learning method, any method such as principal component analysis or independent principal component analysis may be used, but in the following, sparse coding (reference document 5 “M. Elad and M. Aharon,“ Image denoising via sparse and redundant representation ” over learned dictionaries, ”IEEE Transations Image Process., vol. 15, no. 12, pp. 3736-3745, 2006.”). In sparse coding, the LF dictionary is optimized so that any vector can be constructed from a very small number of bases in the LF dictionary.

第二項はスパース項であり、λはそのパラメータである。 In the following, it is assumed that the matrix composed of the LF feature vector group is F ₁ , the matrix composed of the image feature vector group is F _i , the matrix composed of the combined vector is F _c , the dictionary is D, and the coefficient matrix is A. When L1 regularization (lasso) is used as a method of optimizing the dictionary D, optimization is performed based on the following equation.

The second term is a sparse term and λ is its parameter.

となる係数ベクトルαを求めると、対応するＬＦ特徴ベクトルｆ_ｌはおおむね

として推定できる。 Due to the influence of the second term (sparse term), noise such as white noise becomes difficult to express sparsely if there are enough samples to be sampled, so each base expresses the essential features of the image well. Optimized for things. The created dictionary can be divided into an LF dictionary and an image dictionary by dividing at the connection points of the connection vectors. An LF feature vector and an image feature vector restored using a common coefficient and each dictionary are Corresponding to each other. That is, LF dictionary is D _l , image dictionary is D _i, and some arbitrary image feature vector f _i

When the coefficient vector α is obtained, the corresponding LF feature vector f _l is approximately

Can be estimated.

  In addition, the dictionary is learned independently for the LF feature vector group and the image feature vector group, and then a conversion operator between the dictionaries is created and converted so that the feature vectors in the correspondence relationship can be expressed by a common coefficient. It is good.

となるＡ_ｌとＡ_ｉについて、

となる変換演算子Ｔを求めることで、（６）式のαに対応するＬＦ特徴ベクトルｆ_ｌはおおむね

として推定できる。 In that case,

For A _l and A _i

LF feature vector f _l corresponding to α in equation (6) is approximately

Can be estimated.

とし、このＤ_ｉに対して

となるＡを求め、

となるＤ_ｌを求めるなどである。 As another method, a dictionary may be generated from LF feature vectors, and an arbitrary operator may be applied to the dictionary to generate an image dictionary. For example, when the LF image itself is used as the LF feature vector, a focus image corresponding to each base is generated by using the focus image generation from each base of the dictionary, and the image feature vector is obtained from the focus image. It may be generated and used as the base of an image dictionary. In such a case, both dictionaries may be optimized by repeatedly performing dictionary generation and image dictionary generation. For example, for D _l satisfying equation (8), let the focus image generation operator be R.

And for this _Di

For A to be

For example, D ₁ is obtained.

  Next, a light field image generating device according to an embodiment of the present invention will be described with reference to the drawings. FIG. 3 is a block diagram showing the configuration of the light field image generation device according to the embodiment. As shown in FIG. 3, the light field image generation device 200 includes an image input unit 201, a dictionary input unit 202, a patch generation unit 203, an image feature vector generation unit 204, a coefficient vector estimation unit 205, and a light field image generation unit 206. I have.

  The image input unit 201 inputs an image that does not include light ray information that is the basis of the LF image. Hereinafter, this image is referred to as an input image. The dictionary input unit 202 inputs an LF dictionary and an image dictionary. The patch generation unit 203 divides the input image into patches having a predetermined size, and generates an image patch group. The image feature vector generation unit 204 generates an image feature vector group from the image patch group. The coefficient vector estimation unit 205 estimates a coefficient vector group corresponding to each feature vector from the image feature vector and the image dictionary. The light field image generation unit 206 generates an LF feature vector from the coefficient vector group and the LF dictionary, generates an LF patch group from the LF feature vector, and generates and outputs an LF image from the LF patch group.

  Next, the processing operation of the light field image generating apparatus 200 shown in FIG. 3 will be described with reference to FIG. FIG. 4 is a flowchart showing the processing operation of the light field image generating apparatus 200 shown in FIG. First, the image input unit 201 inputs an image that does not include light ray information that is the source of the LF image (step S201). The input image may be any image, but it is preferable that an arbitrary point is focused, and defocusing is observed at an out-of-focus position.

  Next, the dictionary input unit 202 inputs an LF dictionary and an image dictionary (step S202). Both dictionaries may be input independently or combined. As described for the dictionary generation apparatus 100 described above, if any conversion operator is required between the dictionaries, it may be input separately.

  Next, the patch generation unit 203 divides the input image into patches having a predetermined size, and generates an image patch group (step S203). The size of the patch is determined by the input dictionary.

  Next, the image feature vector generation unit 204 generates an image feature vector group from the image patch group (step S204). The image feature vector may be generated by any method, but it is assumed that the image feature vector of the same type as the image feature vector used when generating the image dictionary can be generated. In addition, when various image processes such as a process for obtaining a gradient for generating the image feature vector are necessary, the process may be performed before the patch generation.

としてもよい。その他にどのような方法を用いてもよい。 Next, the coefficient vector estimation unit 205 estimates a coefficient vector group corresponding to each feature vector from the image feature vector and the image dictionary (step S205). Any method may be used for the estimation of the coefficient vector, and a method different from the optimization method in the dictionary generation apparatus 100 may be used. For example, using L0 regularization instead of L1 regularization

It is good. Any other method may be used.

  Next, the light field image generation unit 206 generates an LF feature vector group from the coefficient vector group and the LF dictionary, generates an LF patch group from the LF feature vector group, and generates and outputs an LF image from the LF patch group. (Step S206). The generation method of the LF feature vector is generated in accordance with the dictionary learning method in the dictionary generation device 100 by the expression (7), the expression (11), or an arbitrary method corresponding to the dictionary generation method. Any method may be used for generating the LF patch group from the LF feature vector group. For example, when the dictionary generation apparatus 100 generates an LF feature vector by a method such as using a difference between an image patch of each angle image of the LF patch as a feature vector, the restored LF feature vector The LF patch can be restored by imaging the image and adding the value of the image patch.

となるようなＬＦパッチを推定するなどの方法を使用してもよい。このほかにどのような方法を使用してもよいし、画像パッチまたは画像特徴ベクトルの情報を利用するなどしてもよい。例えば、画像パッチをｐ_ｉ、ＬＦ画像から焦点合わせ画像を生成する演算子をＲとして、

となるようなＬＦパッチを推定するなどの方法を使用してもよい。 In addition, any non-linear method may be used when an LF feature vector that cannot restore the LF patch by linear calculation is generated from the LF feature vector. For example, if you are using a gradient of LF patch as LF feature vector, the LF patch _p l, the LF feature vector and _{f l,} the gradient operator as G,

A method such as estimating an LF patch such that Any other method may be used, and information on an image patch or an image feature vector may be used. For example, an image patch p _i, operators that generate focus image from LF image as R,

A method such as estimating an LF patch such that

  Any method may be used for generating the LF image from the LF patch group. In general, a method is used in which all patches are matched to corresponding image positions, and an average of the portions where a plurality of patches overlap is obtained. The generated LF image is output and the process ends.

  In the above embodiment, a set of LF dictionaries and image dictionaries are generated and used for restoration. However, a plurality of dictionaries may be generated and used. For example, the amount of blurring of an image may be estimated by a method represented by Equations (1) and (2), and a different dictionary may be generated and used for each amount of blurring.

  In the above-described embodiment, the LF image is generated using the dictionary generated by the dictionary generation apparatus 100. However, a dictionary generated by another method may be used. For example, a pair of image dictionaries may be created using an arbitrary LF dictionary by the method shown in equation (12), or a conversion operator between an arbitrary LF dictionary and an image dictionary may be expressed by equation (11). It may be determined by the method shown in FIG.

  As described above, when generating a light field image from an input image having no light ray information, by restoring the light ray information using the image feature vector generated from the image patch obtained by dividing the input image and the LF dictionary, A light field image can be generated from the input image.

  In all the above embodiments, the order of some processes may be changed. In all the above embodiments, the order of some processes may be changed. Further, all or part of the dictionary generation device and the light field image generation device in the above-described embodiment may be realized by a computer. In that case, a program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be read into a computer system and executed. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory inside a computer system serving as a server or a client in that case may be included and a program held for a certain period of time. Further, the program may be a program for realizing a part of the above-described functions, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system. It may be realized using hardware such as PLD (Programmable Logic Device) or FPGA (Field Programmable Gate Array).

  As mentioned above, although embodiment of this invention has been described with reference to drawings, the said embodiment is only the illustration of this invention, and it is clear that this invention is not limited to the said embodiment. is there. Therefore, additions, omissions, substitutions, and other modifications of the components may be made without departing from the technical idea and scope of the present invention.

  It can be applied to applications indispensable for generating a light field image from a general image that does not include light ray information and performing focus adjustment and estimation of three-dimensional information.

  DESCRIPTION OF SYMBOLS 100 ... Dictionary generation apparatus, 101 ... Light field image input part, 102 ... Focusing image generation part, 103 ... Patch generation part, 104 ... Light field feature vector generation part, 105 ... Image feature vector generation unit 106 ... Learning unit 200 ... Light field image generation device 201 ... Image input unit 202 ... Dictionary input unit 203 ... Patch generation unit 204 ..Image feature vector generation unit, 205 ... coefficient vector estimation unit, 206 ... light field image generation unit

Claims (8)

A dictionary generation method performed by a dictionary generation device that generates a light field dictionary from a light field image used for dictionary learning,
A focused image generating step for generating a focused image in which a focal length is adjusted to an arbitrary position from the light field image;
A light field feature vector generating step for generating a light field feature vector representing the feature of the light field image from the light field image;
An image feature vector generation step for generating an image feature vector representing the feature of the focused image from the focused image;
A dictionary generation method comprising: the light field feature vector; and a dictionary generation step of generating the light field dictionary used when generating a light field image from the image feature vector.   The dictionary generation method according to claim 1, wherein the image feature vector generation step estimates a blur parameter for estimating a parameter representing a focal blur of the focused image and sets the parameter as the image feature vector. An image processing method performed by an image processing apparatus that inputs an input image not including light ray information and a light field dictionary generated in advance by the dictionary generation method according to claim 1 and generates a light field image,
An image feature vector generating step for generating an image feature vector representing the feature of the input image from the input image;
A light field feature vector estimation step for estimating a light field feature vector representing a feature of the light field image from the image feature vector and the light field dictionary;
A light field image generation step of generating the light field image to be generated from the light field feature vector.   The image processing method according to claim 3, wherein the image feature vector generation step estimates a blur parameter for estimating a blur parameter representing a focal blur of the input image and uses the blur parameter as the image feature vector.   The light field image generation step generates the light field image so that the light field feature vector matches a light field feature vector obtained from a newly restored light field image. Image processing method. A dictionary generation device that generates a light field dictionary from a light field image used for dictionary learning,
A focused image generating means for generating a focused image in which a focal length is adjusted to an arbitrary position from the light field image;
A light field feature vector generating means for generating a light field feature vector representing the feature of the light field image from the light field image;
Image feature vector generation means for generating an image feature vector representing the feature of the focused image from the focused image;
A dictionary generation device comprising: the light field feature vector; and dictionary generation means for generating the light field dictionary used when generating a light field image from the image feature vector. An image processing apparatus that inputs an input image that does not include light ray information and a light field dictionary generated in advance by the dictionary generation apparatus according to claim 6 to generate a light field image,
Image feature vector generation means for generating an image feature vector representing the feature of the input image from the input image;
A light field feature vector estimating means for estimating a light field feature vector representing the feature of the light field image from the image feature vector and the light field dictionary;
An image processing apparatus comprising: a light field image generating unit configured to generate the light field image to be generated from the light field feature vector.   An image processing program for causing a computer to execute the image processing method according to any one of claims 3 to 5.

Images