JP6263024B2 - Image processing apparatus, image processing method, program, and image reproduction apparatus - Google Patents

Description

  The present invention relates to an image total processing apparatus and an image reproduction apparatus for reproducing an image by image processing using image data having light field information.

  In recent years, a technique has been disclosed in which a multi-view parallax image (light field information) is acquired and focus position adjustment (refocus) after photographing can be performed by image reconstruction processing.

  In the refocus processing of image data having such light field information (light field image data), a subject-side focus surface in the image after reconstruction processing, or a virtual surface on which pixel addition is performed to obtain a focus surface (reconstruction) Designation). Japanese Patent Application Laid-Open No. 2004-151820 discloses a technique in which an image displayed on a playback device is touched to refocus so that the subject at the touched coordinate is in focus. In this technique, the reconstruction plane coordinates in the depth direction of the image can be designated intuitively.

  In the reconstruction process, in addition to parallel translation of the focal plane in the depth direction facing the camera, it is possible to set the focal plane obliquely with respect to the lens plane of the camera, as with a general photographic lens. Yes, it is also possible to generate an image corresponding to the lens tilt.

US2008 / 0131019

  However, the prior art such as Patent Document 1 is based on the designation of the Z coordinate (position in the depth direction) at a specific point on the display screen XY coordinates, and the reconstruction surface corresponding to the operation of tilting the lens is intuitive. It is difficult to specify. Therefore, an object of the present invention is to realize an intuitive designation configuration of a reconstruction surface inclined with respect to a lens surface in the refocus processing of light field image data.

  In order to achieve the above object, according to the present invention, an image processing apparatus that processes image data that has parallax in a subject space and is capable of image synthesis with a shift amount for each parallax on a reconstruction plane is provided. Image data input means for acquiring data, first input information for setting the first reconstruction plane, second input information for setting the rotation axis of the first reconstruction plane, and the rotation angle of the rotation axis Image composition information input means for acquiring the third input information for setting the first reconstruction plane based on the acquired first input information, the first reconstruction plane, the second input information A composite image is generated from the image data by reconstruction surface setting means for setting the second reconstruction surface based on the third input information and image composition on the reconstruction surface set by the reconstruction surface setting means. Image synthesizing means.

  In the refocus processing of light field image data, it is possible to provide an image processing device and an image reproduction device that can realize an intuitive designation configuration of a reconstruction surface inclined with respect to a lens surface.

1 is a block diagram of an image reproduction apparatus to which an image processing apparatus according to an embodiment of the present invention is applied. The figure which shows the flowchart of the image processing operation | movement which concerns on the Example of this invention. The figure which shows an example of the imaging optical system for acquiring light field image data The figure for demonstrating the image shift method and image generation method in a light field The figure for demonstrating the image shift method and image generation method with respect to an oblique reconstruction surface The figure for demonstrating the relationship between an imaging surface and a reconstruction surface, and a focus surface FIG. 6 is a diagram for explaining a user interface for designating a focus plane in the image reproduction apparatus according to the embodiment of the present invention. The figure for demonstrating the method to input depth information in the image reproduction apparatus which concerns on the Example of this invention. The figure for demonstrating the synthesized image obtained by the image processing by the image processing apparatus which concerns on the Example of this invention.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

  An image processing apparatus according to an embodiment of the present invention and an image reproduction apparatus to which the image processing apparatus is applied will be described with reference to FIGS.

  First, a composite image obtained by image processing in the image processing apparatus according to the present embodiment will be described with reference to FIG.

  9A and 9B are diagrams showing the positional relationship between the camera and the subject. 900 is a camera, 910 is a plurality of subjects having different depths from the camera to the subject, and 910a is a subject. Among them, a subject 920 that is particularly focused on in the following description indicates a focus surface. In the image processing apparatus of the present embodiment, an effect equivalent to lens tilt is obtained by image processing. However, when the lens tilt is performed, the focus surface can be changed from a state directly facing the camera to a tilted state.

  The diagrams on the right side in FIGS. 9A and 9B show examples of the reconstructed composite image 930. The dots shown in the image 930 schematically represent the degree of blur of the subject. The larger the dot, the larger the blur, and the smaller the dot, the smaller the blur. The absence of a dot indicates that the subject is in focus.

  In FIG. 9A, the focus surface 920a faces the camera 900 and is at a depth position that coincides with the subject 910a of interest. Therefore, the subject 910a is focused on the composite image, and the blur is larger as the subject is farther away from the subject 910a.

  FIG. 9B shows a case where the focus surface 920a is rotated about the rotation axis 705 on the subject 910a to the focus surface 920b to obtain the same effect as the lens tilt. As shown in the figure, it is possible to focus all the subjects 910 on the composite image by matching all the subjects 910 with the focus surface 420b.

  FIG. 9C shows a case where the focus surface 920a is rotated about the rotation axis 705 on the subject 9410a to the focus surface 920c to obtain a lens tilt effect. As shown in the figure, in this case, subjects other than the subject 910a of interest are farther away from the focus surface than in the state of FIG. Therefore, on the composite image, the subject 910a is in focus, and the other subjects 910 are uniformly out of focus from the state in FIG. 9A.

  Thus, by considering the change of the focus surface due to the lens tilt with the rotation axis and the rotation angle around it, the setting operation of the focus surface can be performed while intuitively understanding the change of the blurred state. In this embodiment, an operation in a blurred state in which the focus surface is tilted is performed by image processing to be described later. At that time, an operation for setting the focus surface by an intuitive designation method is provided. In this embodiment, an example in which the image processing apparatus according to this embodiment is applied to an image reproduction apparatus having a display unit will be described. The image processing apparatus is not limited to an apparatus specialized for image reproduction, but has an image reproduction and display function such as an imaging apparatus such as a digital camera, a mobile terminal apparatus such as a smartphone, a tablet-type information processing apparatus, and a PC. It may be a device. In this embodiment, the focus plane setting information is input using a display unit having a touch panel. However, the present invention is not limited to this. An input configuration in which a cursor is displayed on the display unit and the cursor is moved by key operation to specify a position on the image, or a user interface screen in which a rotation angle and depth can be visually set is configured to perform a key operation. Also good.

  FIG. 1 is a block diagram showing a configuration of an image reproducing apparatus to which an image processing apparatus according to an embodiment of the present invention is applied. As will be described later, the image reproducing apparatus is roughly composed of an input system A, an image processing system B, and a display system C. Each component is composed of finer elements. Each component shown in FIG. 1 can be controlled by loading and executing a program stored in the memory 110 by the CPU of the control unit 109 of the image capturing and reproducing apparatus. In this case, the image processing apparatus may have a unique control unit, and the processing may be shared with the control unit of the image reproduction apparatus.

  The configuration of the input system A will be described. The image data input unit 100 receives image data having a subject space parallax that allows image composition on the reconstruction screen. The image data having the parallax of the subject space is image data including captured image data and parallax amount information in the imaging optical system. The image composition information input unit 101 includes a point input unit 101a, an angle detection unit 101b, and a depth input unit 101c, and accepts a user's point input operation, angle input operation, and depth input operation, respectively.

The configuration of the image processing system B will be described. The input operation determination unit 102 determines whether the input from the point input unit 101a is a one-point input (first input operation) that is an instruction of only one point by the operator, or a two-point input (second Input operation). As will be described later, in this embodiment, the point input is performed by a touch operation in which an operation unit such as a touch panel provided on the display unit 108 is touched with a finger, so that one point input is performed with one finger as shown in FIG. The touching instruction and 2-point input are touching instructions with two fingers as shown in FIG. The input information by one point input is processed as the coordinate data of the focus subject that is the XY coordinates of the subject to be focused, and the input information by the two point input is processed as the data of the rotation axis setting coordinates. A focus plane (rotation axis setting focus plane) setting unit 103 for setting the rotation axis is based on one-point input information from the input operation determination unit 102 and synthesized image data from the image synthesis unit 107 described later. An axis setting surface which is a focus surface for setting is set (first reconstruction surface setting). The axis setting plane set here is a plane on which the subject in focus subject coordinates (one-point input information) on the composite image is focused. In addition, reconstruction plane information for obtaining a focus plane set in the synthesized image is set and output to the image synthesis unit 107. The focus surface rotation axis setting unit 104 determines the rotation axis of the focus surface based on the two-point input information from the input operation determination unit 102 and the axis setting surface information from the focus surface setting unit 103 that sets the rotation axis. Set. The rotation axis of the focus plane set here is on the axis setting plane and is a straight line passing through the two input XY coordinates. The focus surface rotation angle setting unit 105 sets the rotation angle information of the focus surface based on the angle input information that is the third input operation detected by the angle detection unit 101b. Here, the rotation angle information may be an angle as the angle input information as it is, or sensitivity adjustment may be performed by scaling to several times or a fraction. The focus plane setting unit 106 sets the focus plane based on the rotation axis information from the rotation axis setting unit 104 of the focus plane and the rotation angle information from the rotation angle setting unit 105 of the focus plane, and this focus plane is displayed in the composite image. A reconstruction surface for obtaining a surface is set (second reconstruction surface setting). An image composition unit 107 is an image composition unit, and is based on image data having parallax of the subject space from the image data input unit 100 that is a composition source input unit, and reconstructed surface information from the focus surface setting unit 106. Generate data. The synthesized image data is synthesized by appropriately shifting the parallax of the subject space so that an image is formed on the reconstruction plane. Details of the shift of the parallax of the subject space will be described later. Composite image data is output to the setting unit 103 and the display unit 108 of the focal plane for setting the rotational axis. If there is no input of reconstruction plane information from the focus plane setting unit 106 and reconstruction plane information is input from the focus plane setting unit 103 for setting the rotation axis (for example, two-point input waiting state). , 103, the image is synthesized on the reconstruction plane. As can be understood from the description of the flowchart in FIG. 2 (described later), the reconstruction plane information from 103 and 106 is input to the image composition unit 107 by different input operations at different timings. The image composition may be performed as appropriate. When the reconstruction plane is not input from both 103 and 106, the composite image data is output without shifting the parallax of the subject space.

  The display system C includes a display unit 108 that displays the composite image data from the image composition unit 107.

  With reference to FIG. 2, the image processing operation of the image processing apparatus according to the present embodiment will be described. FIG. 2 is a flowchart illustrating the image processing operation according to the present embodiment. This operation is realized by the control unit 109 executing a program stored in the memory 110 to control each unit of the image reproduction apparatus. The image processing apparatus of the present embodiment obtains input information and light field image data for setting the focus plane from the image playback apparatus to which the image processing apparatus is applied, performs image processing, and performs the image processing on the set focus plane. A composite image is generated and displayed on the display unit of the image reproduction apparatus.

  FIG. 2A shows a flowchart of the entire image processing operation. In the figure, image data is acquired and displayed in step S200, and one-point input in step S240 and two-point input in step S260 are received in the input waiting loop in step S220.

  FIG. 2B shows a flowchart of image data acquisition and display processing in step S200. In step S201, the image data input unit 100 acquires image data having a parallax image of the subject space in accordance with an instruction from the operator. In step S202, the control unit 109 sets initial parameters. The initial parameters referred to here indicate a reconstruction plane, a focus plane information for setting a rotation axis, and depth information, which will be described later. For example, the reconstruction plane is the same plane as the imaging plane, the focusing plane that sets the rotation axis is the focusing plane of the image generated with the imaging plane as the reconstruction plane, the depth information is the depth obtained by adding all the added synthetic pixels described later, etc. And it is sufficient. In step S203, the image composition unit 107 performs image generation processing according to the initial parameters, and obtains composite image data. In step S204, the display unit 108 displays the generated composite image data.

FIG. 2C shows a flowchart of the one-point input processing operation in step S240. In step S241, the input operation determination unit 102 performs conditional branching on whether or not there is a one-point input that is the first input operation. If YES, the process proceeds to step S242. In the case of NO, the remaining processing is skipped, the process proceeds to RETURN, and the process returns to the main routine of FIG. In step S242, the rotation axis setting focus plane setting unit 103 changes the depth Z coordinate of the focus plane so that the subject on the screen XY coordinates of the one-point input is focused based on the input one point input. In addition, only the depth Z coordinate is changed and the focus surface is moved in parallel. In step S243, the image composition unit 107 performs image generation on the changed focus plane to obtain image data. In step S244, display unit 108 updates the display image to the image data obtained in step S243. In step S245, the rotation axis setting unit 104 acquires the focus plane information for setting the rotation axis as the focus plane of the image data obtained in step S243.

  FIG. 2D shows a flowchart of the two-point input acceptance processing operation in step S260. In step S261, the input operation determination unit 102 performs conditional branching on whether there is a two-point input that is the second input operation. In the case of YES, the process proceeds to step S262. In the case of NO, the remaining processing is skipped, the process proceeds to RETURN, and the process returns to the main routine of FIG. In step S262, based on the input two-point input, the rotation axis setting unit 104 sets a straight line passing through two points where the XY coordinates are two-point input and the Z-coordinate is on the focus plane for setting the rotation axis. It is set as the rotation axis that determines the focus surface corresponding to the lens tilt. In step S263, the angle detection unit 101b performs conditional branching as to whether or not there is an angle input as the third input operation. In the case of YES, the process proceeds to step S264. In the case of NO, the remaining processing is skipped, the process proceeds to RETURN, and the process returns to the main routine of FIG. In step S264, the rotation angle setting unit 105 sets a rotation angle for determining a focus surface based on the input angle input. As described above, the focus surface is set according to the input of the set focus surface, the rotation axis, and the rotation angle. In step S265, based on the output of the depth input unit 101c, the control unit 109 performs conditional branching on whether there is a depth input. If YES, the process proceeds to step S266. In the case of NO, step S266 is skipped and the process proceeds to step S267. In step S266, the image composition unit 107 sets the number of added composite pixels to be described later based on the input depth input. The depth of the image data to be synthesized changes depending on the number of added synthesized pixels. In step S267, the image composition unit 107 performs image generation processing on the reconstruction plane for obtaining the set focus plane and the set number of added synthesized pixels (depth) to obtain synthesized image data. In step S268, the display unit 108 updates the display image to the composite image data obtained in step S267.

  The above is the processing flow of the present invention.

  In order to perform image processing by the image processing apparatus of this embodiment, it is necessary that the image data has angle information in addition to the position of the light beam. FIG. 3 is a diagram illustrating an example of a photographing optical system for obtaining image data having parallax in the subject space. In the present photographing optical system, the microlens array 301 is disposed in the vicinity of the image forming surface, and a plurality of pixels are disposed corresponding to one lens 311 constituting the microlens array 301, whereby angle information is obtained. Have acquired.

  FIG. 3A is a diagram schematically showing the relationship between the image sensor 302 and the microlens array 301. FIG. 3B is a diagram schematically illustrating the correspondence between the pixels of the image sensor 302 and the microlens array 301. FIG. 3C is a diagram illustrating that the pixels provided below the microlens array are associated with a specific pupil region by the microlens array 301.

FIG. 3A shows the microlens array 301 viewed from the front and side of the imaging apparatus. When viewed from the front, the microlens 311 of the microlens array 301 is disposed so as to cover the pixels on the image sensor 302. For ease of viewing, the size of each microlens 311 is exaggerated and large, but is actually several times as large as a pixel. FIG. 3B is an enlarged view of a part of the microlens array 301 in FIG. 3A, in which a lattice-shaped frame indicates each pixel of the image sensor 302, and a bold circle forms the microlens array 301. Each microlens 311 is shown. A plurality of pixels are assigned to one microlens. In the example shown in the figure, 25 pixels of 5 rows × 5 columns are provided for one microlens. Therefore, the size of each microlens is 5 × 5 times the size of the pixel. The output of the 25 pixels obtained by adding synthesis will be referred to as adding synthetic pixel 250.

  3C, the figure on the right side of the figure is a cross-sectional view of the image pickup device 302 cut along the optical axis of the microlens so that the short direction of the sensor is the vertical direction of the figure. In FIG. 3C, reference numerals 321, 322, 323, 324, and 325 denote each pixel (one photoelectric conversion unit) of the image sensor 302. On the other hand, the diagram on the left side of FIG. 3C shows the exit pupil plane of the photographing optical system. Actually, when the direction of the sensor on the right side of FIG. 3C is aligned, the exit pupil plane is in the direction perpendicular to the plane of FIG. 3C, but the projection direction is changed for the sake of explanation. For simplicity, FIG. 3C will describe one-dimensional projection / signal processing. In an actual device, it is expanded to two dimensions. The pixels 321, 322, 323, 324, and 325 in FIG. 3C have a positional relationship corresponding to 321 a, 322 a, 323 a, 324 a, and 325 a in FIG. As shown in FIG. 3C, each pixel is designed to be conjugate with a specific region on the exit pupil plane of the photographing optical system by the micro lens 311. In the example of FIG. 3C, the pixel 321 and the region 331 on the exit pupil plane correspond to the pixel 322 and the region 332, the pixel 3223 and the region 333, the pixel 324 and the region 334, and the pixel 325 and the region 335, respectively. Yes. That is, only the light beam that has passed through the region 331 on the exit pupil plane of the photographing optical system is incident on the pixel 221. The same applies to the other pixels. As a result, it is possible to acquire angle information from the passing area on the pupil plane and the positional relationship on the image sensor 202.

  Here, it can be considered that one microlens 311 and pixels under the microlens 311 are the same as one camera, and the microlens array 301 and the image sensor 302 are also a camera array system having parallax composed of a large number of cameras. The above-described image data having information on the position and angle of the light beam incident on each pixel is also image data having parallax in the subject space. Therefore, even in a multi-lens camera system in which a general camera having more pixels is arranged as an array, an image having parallax in the subject space can be acquired in the same manner.

  Next, an image shift method and an image generation method will be described with reference to FIG. In the figure, the same parts as those in FIG. 3 are denoted by the same reference numerals as those in FIG.

  Reference numeral 301 denotes a microlens array, 441 denotes the position of the imaging surface, and 442 denotes a column of pixels arranged on the imaging surface. Reference numeral 443 denotes the position of the reconstruction plane, which is a virtual plane created from information of the pixel column 442 on the imaging plane 441. The process of creating this virtual surface is refocusing.

One addition synthesized pixel 250 is two-dimensionally arranged as shown in FIG. For convenience of explanation, FIG. 4 shows a one-dimensional array of 5 rows and 1 column as viewed from the side, but a similar array is also arranged in the direction perpendicular to the paper surface (X-axis direction). 5 is arranged. In the case of a pixel array of 5 rows and 1 column, for example, in the addition synthesis pixel 450a on the imaging surface, integration of the incident light in the angular direction by adding the pixels 1, 2, 3, 4, and 5 under each microlens. To obtain an added composite pixel output. By performing this process sequentially for each microlens 311, an image similar to that of a normal camera is generated. In this case, the unit pixel of the generated image is an addition composite pixel. As described with reference to FIG. 3, angle information can be acquired from the positional relationship on the pupil plane and the passing region on the imaging element 302, so that the pixel information is shifted by a shift amount corresponding to the incident angle. By adding, it is possible to create an image on an arbitrary reconstruction plane from a single shot. For example, in order to reconstruct an image at the focus position indicated on the reconstruction surface 443, the following processing is performed. In FIG. 4, pixels on the imaging surface that pass through the imaging surface 441 from each microlens and intersect with the lines connected to the pixels 2, 5, 3, 1, and 4 of the addition combined pixel 250 b (shaded portion) of the reconstruction surface 443 Is added. That is, the pixels 2, 3, and 4 on the imaging surface under the micro lens, the pixel 1 under the micro lens adjacent to one side, and the output of the pixel 5 under the micro lens adjacent to one side are respectively weighted. Multiply by coefficient 1 and add. A unit pixel of a reconstructed image can be generated by sequentially performing addition processing on each microlens. It is also possible to generate different refocus images by performing similar processing on a reconstruction surface (not shown) behind the reconstruction surface 443 and a reconstruction surface (not shown) in front of the microlens. is there. In this way, shifting the image information so as to correspond to each reconstruction plane is called image shift, and generating the added composite pixel 450 by pixel addition on the imaging plane or reconstruction plane is called image generation. To do. Further, for example, if the weighting coefficient of only the output of the pixel 3 is set to 1 and the weighting coefficient of the other pixels is set to 0 among the addition synthesized synthetic pixels below the microlens at the time of pixel addition, an image with a narrowed pupil is obtained. And the depth of field of the image can be increased.

  Based on the above-described image shift method, an image shift method and an image generation method on an inclined reconstruction surface, which are characteristic of the present embodiment, will be described with reference to FIG. In the figure, the same elements as those in FIG. 4 are denoted by the same reference numerals.

  The image shift method on the inclined reconstruction plane is basically the same method as the image shift method described above. What is necessary is just to add and synthesize the pixel output on the imaging surface connected to each pixel of the addition and synthesis pixel 450c on the inclined reconstruction plane. For example, the pixel 442b in the addition combined pixel 450c corresponds to the pixel 442a on the imaging surface 441. By setting an inclined virtual plane in this way and tracing the pixels for addition synthesis, it is possible to generate an image on the tilted reconstruction plane.

  In the present embodiment, as described above, the designation is performed by the procedure of the focus surface for setting the rotation axis, the rotation axis of the focus surface, and the rotation angle of the focus surface. In the refocus processing, in order to obtain the focus surface, the corresponding reconstruction surface is set according to the procedure. In FIG. 5, the reconstruction plane that is the focus plane for setting the rotation axis is the imaging plane 441, the reconstruction plane that corresponds to the rotation axis of the focus plane is 500, and the reconstruction plane that corresponds to the rotation angle of the focus plane The rotation angle of the surface corresponds to 501. In this embodiment, these three parameters are designated to determine the reconstruction plane, that is, the focus plane.

  Next, the relationship between the imaging surface and the reconstruction surface, the focus surface, and the lens surface will be described with reference to FIG. The relationship between these surfaces follows the so-called Scheinproof principle. Although the surface is discussed using FIG. 6, for the sake of simplicity, it is assumed that each surface is parallel to the depth direction of the paper (X-axis direction).

  In a general photographic optical system, the relationship between the imaging surface, the lens surface, and the focus surface is a parallel relationship such as the imaging surface 1, the lens surface, and the focus surface 1. Now, consider tilting the imaging surface like the imaging surface 2 with respect to the lens surface. At this time, the corresponding focus surface 2, the lens surface, and the surface obtained by extending the imaging surface 2 intersect at one point on one line as shown in FIG. This relationship is called the Scheinproof principle. When the lens is tilted in the tilt lens, the parallel relationship between the lens surface and the imaging surface is broken, and an image having a focus surface oblique to the imaging surface can be obtained.

  Setting the reconstructed surface tilted with respect to the lens surface by performing the image shift described above corresponds to tilting the lens, whereby an oblique focus surface in accordance with the Scheinproof principle can be obtained. Therefore, the same can be considered by replacing the imaging surface in FIG. 6 with a reconstruction surface. Also, based on the principle of Scheinproof, it is possible to set the necessary reconstruction plane by tracing the reverse from the desired focus plane.

  In this embodiment, the focus surface is set by designating the focus surface for setting the rotation axis, the rotation axis on the focus surface, and the rotation angle of the focus surface, as described above. In FIG. 6, a desired focus surface is determined first, and a reconstruction surface necessary to obtain the focus surface is set. For example, the focus surface 1 is considered as the focus surface for setting the rotation axis, the rotation axis A2 on the focus surface 1 is set, and the rotation angle θ2 is set. And in the said procedure, the setting by the side of a reconstruction surface is performed sequentially. The reconstruction surface 1 is set as a surface corresponding to the focus surface 1 which is a focus surface for setting the rotation axis, and the rotation axis A1 corresponding to the rotation axis A2 is set. Then, the reconstruction plane 2 is set by setting the rotation angle θ1 corresponding to the rotation angle θ2. Here, the focus surface for setting the rotation axis is a plane parallel to the lens surface, but may be a surface that is originally inclined.

A method for deriving the reconstruction surface (that is, the rotation angle θ1) for setting the focus surface and obtaining the focus surface will be described. Since the relationship between the focus surfaces D1 and D2 follows a general imaging formula based on the lens focal length, the derivation of D2 from D1 is omitted. Further, the relationship between the distances (image heights) H1 and H2 between the rotation axis and the optical axis can be similarly obtained as the vertical magnification from the focal length, and is omitted here. The relationship between the rotation angles θ1 and θ2 can be considered as a distance L from the optical axis to the intersection line. The distance L can be expressed as L = D1 / tan (θ1) = D2 / tan (θ2) from the geometric relationship between the focus surface side and the imaging surface side. From this relationship, θ1 can be obtained as follows as an expression of D1, D2, and θ2.

  Next, a method for designating the focus plane on the user interface will be described with reference to FIG. In the figure, the same elements as those in FIG.

  In the figure, reference numeral 701 denotes a playback apparatus whose configuration has been described with reference to FIG. 2, and displays a composite image generated according to the above-described image processing configuration on the display unit 108. This image reproduction apparatus includes an input system A including a point input unit (so-called multi-touch screen) that can input a plurality of points on the display unit 108, an angle input unit that is an angle sensor (not shown), and a depth input unit. .

First, as shown in FIG. 7A, one coordinate 703 on the display unit 108 is input by designating one point of the image with the index finger, for example. This operation includes the point input to the point input unit 101a in FIG. 1, the one-point input determination in the input operation determination unit 102 (S241), and the output of one-point input information to the setting unit 103 of the rotation axis setting focus surface. Corresponding (S242). In FIG. 7A, the focus plane orthogonal to the optical axis in which the subject on 703 is focused is designated as the focus plane once, and the synthesized image data obtained by image synthesis is updated as a display image. This is because the output of reconstruction plane information from the rotation axis setting focus plane setting unit 103 to the image synthesis unit 107 in FIG. 1, the image synthesis in the image synthesis unit 107, and the output of synthesized image data to the display unit 108. This corresponds to the display of the composite image data (S243 to S244). In FIG. 7A, the focus plane of the image displayed on the display unit 108 is the focus plane for setting the rotation axis. This corresponds to the setting of the axis setting surface in the rotation axis setting focus surface setting unit 103 in FIG. 1 (S245).

  Next, two coordinates 704 on the display unit 108 are input by designating two points on the screen with both thumbs in FIG. 7B. This corresponds to the point input to the point input unit 101a in FIG. 1 and the two-point input determination in the input operation determination unit 102 (S261). Accordingly, a straight line 705 passing through the two points projected on the reconstruction screen designated in FIG. 7A is designated as the rotation axis of the focus plane. This corresponds to the setting of the rotation axis in the rotation axis setting unit 104 in FIG. 1 (S262).

Next, as shown in FIG. 7C, with the two coordinates 704 input, the image playback device 701 itself is tilted, for example, from the state 706a to the state 706b in the direction of the arrow 708, thereby setting the angle 707. input. Image reproducing apparatus 70 1 has an angle detection sensor (not shown) for detecting the attitude of the device, the relative angle between before and after the operation to perform the operation of rotating the device is detected as an angle 707. This corresponds to the angle detection by the angle detection unit 101b in FIG. 1 (S263). From this angle detection information, the rotation angle of the focus surface is set as described above (S264).

Then, the focus surface after the input operation is set based on the set rotation axis and rotation angle of the focus surface (S264), and a composite image is generated on the reconstruction surface that obtains this focus surface. The composite image is updated as a display image 702 (S268). Note that if an angle is input when the image reproducing device 701 is tilted without the input of the two coordinates 704, there is a possibility that an input unconscious by the user may be performed. For user performs consciously inputs, only when the input two coordinate 704 is, it is preferable that the input of the angles 707 can be accepted. In this example, a horizontal rotation axis is set for the sake of explanation using the side view, but the rotation axis may of course be vertical, or may be vertical or oblique to the vertical. I do not care. When the rotation axis is set in this way, for each case, the operation method of depth input to be described later is clearly distinguished in operation, and is set so as not to be confused with input information by other operations. .

With reference to FIG. 7D, image data generated by performing the operations of FIGS. 7A to 7C will be described. Figure on the left side of FIG. 7 (d), the image data generated is a perspective view of images reproducing apparatus 701 displayed on the display unit 108. In the figure, reference numeral 710 denotes a display image, which is an example of an image taken so that a building, a person, or the like is seen from an oblique sky. The dots shown in the display image 710 schematically represent the amount of blur of the image. An area without dots is in focus without blur, an area with dots is in blur, The larger the dot, the greater the amount of blur. Figure on the right side of FIG. 7 (d), the horizontal axis taken defocus amount, a graph plotting the coordinates Yn of Cartesian and the rotary shaft 705 on the display image on the vertical axis. Reference numeral 720 denotes a defocus amount of the ground ignoring a building or a person in the display image for simplicity.

In the operation example of FIG. 7, the coordinates 703 of the subject to be focused on and the rotation axis 705 are set in the vicinity. Since the area near the rotation axis 705 does not change the focus state before and after the reconstruction plane is changed, the area 711 near the rotation axis 705 that is originally in focus is in focus even in the updated composite image. The state (defocus amount = 0) is entered. Since a straight ground is considered, as the distance from the rotation axis 705 increases, the defocus amount increases and the amount of blur increases. Region from the vicinity area 711 without blurring 712a, begin cause blurring as becomes 712b, respectively arrows 713a, the blur amount as will if you go to 713b increases.

  Also, in the graph on the right side of FIG. 7D, the area 712a with a positive defocus amount is an area in which a subject exists behind the focus plane of the display image 710 and the defocus amount is a negative value. In 712b, a subject is present in front of the focus surface of the display image 710. Considering this, let us consider an operation when it is desired to change the focus surface from the composite image in FIG. 7D to an image that is entirely in focus. When the image playback device 701 is rotated in the direction opposite to the arrow 708 in FIG. 7C, the rotation direction of the focus surface is set so that the defocus amount decreases in the region 712a and the defocus amount increases in the region 712b. Easy to understand operation. At this time, in order to intentionally enhance an effect like a diorama photograph that focuses only on the vicinity of the rotation shaft 705, the image reproducing device 701 may be rotated in the direction of the arrow 708. Then, the defocus amount increases in the region 712a, and the defocus amount decreases in 712b, so that a desired effect can be obtained. In this way, by inputting the rotation axis and rotation angle of the focus surface by a predetermined operation of the image reproducing apparatus corresponding to the change (rotation) of the straight line in the graph, it is easy to understand the correspondence with the operation. Can be input.

  In the description using FIG. 4, pixel addition for obtaining a so-called deep image with a narrowed pupil has been described. Here, an input operation of depth information will be described using FIG. 8. In the figure, the same elements as those in FIG. 7 are denoted by the same reference numerals.

  There are several methods for depth input, but it is appropriately separated from the input method of the focus surface, the input of the focus surface rotation axis, and the input of the focus surface rotation angle described above. It is necessary to allocate an operation method.

  The method shown in FIG. 8A is a method of shaking the image reproducing device 701 itself. The image reproducing device 701 is equipped with a vibration sensor (not shown), and when the image reproducing device 701 is shaken, the display image is sequentially updated to an image having a shallower depth. When the settable shallowest depth is reached and still shaken, the image is reset to the deepest depth. By doing in this way, the operation appropriately distinguished from the input operation of the focus surface, the rotation axis, and the rotation angle described above can be distributed to the input of depth information.

The method shown in FIG. 8B is a method of allocating the rotation on the rotation axis different from the angle input in the generation of the reconstruction surface to the depth input operation. In the angle input operation for the reconstruction plane, the two points 704 for determining the rotation axis are input as described above, and the image reproducing device 701 itself is rotated about the rotation axis 705. In contrast, performing depth input by rotating about an axis of rotation 705 to a right angle, the display portion of the display image images at 108 shaft 730. An intuitive depth input operation can be performed by assigning one direction as a direction in which the depth becomes shallower and assigning the other direction as a direction in which the depth becomes deeper.

  The operations in FIGS. 8A and 8B are the input to the depth input unit 101c in FIG. 1, the output from 101c to the image synthesis unit 107, the image synthesis in 107 and the output to the display unit 108. , 108 corresponds to an operation of updating the display image once or repeatedly (S265 to S268).

  As described above, according to the present embodiment, an intuitive designation method of a focus surface tilted with respect to a lens surface is realized in the refocus processing of light field image data. In addition, the image processing apparatus of the present invention can be used in a wider range of apparatuses by appropriately setting the peripheral devices of the apparatus to which the image processing apparatus is applied as an input configuration of the setting information on the focus surface instead of the touch panel of the above embodiment. It is possible to realize refocused image reproduction.

  The object of the present invention can also be achieved by supplying a storage medium storing software program codes for realizing the functions of the above-described embodiments to a system or apparatus. That is, it goes without saying that the object of the present invention can also be achieved when the computer (or CPU or MPU) of the system or apparatus reads and executes the program code stored in the storage medium.

  In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the program code itself and the storage medium storing the program code constitute the present invention.

  As a storage medium for supplying the program code, for example, a flexible disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, a nonvolatile memory card, a ROM, or the like can be used.

  In addition, the functions of the above-described embodiment can also be realized when an OS (basic system or operating system) operating on the computer performs part or all of the actual processing based on the instruction of the program code read by the computer. Realized. Needless to say, this case is also included in the present invention.

  Furthermore, after the program code read from the storage medium is written to the memory provided in the function expansion board inserted in the computer or the function expansion unit connected to the computer, the processing based on the instruction of the program code is also performed. Included in the invention. That is, it goes without saying that the present invention also includes the case where the CPU or the like provided in the function expansion board or function expansion unit performs part or all of the actual processing based on the instruction of the program code to realize the functions of the above-described embodiment. Yes.

  The above-described embodiments are merely examples of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed as being limited thereto. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

Claims (14)

In an image processing apparatus that reconstructs light field data and generates image data on a predetermined reconstruction surface ,
Display control means for displaying an image based on the light field data on a display unit;
First input means for acquiring first input information corresponding to a rotation axis, with a touch operation from a user on a touch panel provided in the display unit as input information;
Second input means for acquiring second input information corresponding to the rotation angle;
By determining the rotation axis based on the first input information and determining the rotation angle based on the second input information, the surface rotated about the rotation axis by the rotation angle. a reconstruction plane setting means for setting the reconstruction plane,
Generating means for generating the image data on the reconstruction plane set by the reconstruction plane setting means from the light field data ,
The first input information includes information on positions of two points designated by the touch operation,
The reconstruction plane setting means, based on the first input information, the image processing apparatus characterized that you determine the rotation shaft on the line connecting the two points. The image processing apparatus according to claim 1, wherein the second input unit acquires the second input information based on an output of an angle sensor that detects an angle of the image processing apparatus. 3. The touch panel according to claim 1, wherein the touch panel is a multi-touch display, and the first input information is acquired by the first input unit in response to a two-point input made by the touch operation. An image processing apparatus according to 1. The image processing apparatus according to claim 2, wherein the second input unit accepts the second input information only when the two-point input by the touch operation is performed. Third input information for setting the reconstruction plane perpendicular to the optical axis, which is in focus on the subject at the position of the one-point input, is obtained using the one-point input by the touch operation as input information. The image processing apparatus according to claim 1, further comprising an input unit. Having fourth input means for acquiring fourth input information corresponding to the depth of field;
The image processing apparatus according to claim 1, wherein the generation unit generates a reconstructed image having a depth of field based on the fourth input information. The second input means acquires the second input information based on an output of an angle in a first direction from an angle sensor that detects an angle of the image processing device,
The said 4th input means acquires the said 4th input information based on the output of the angle of the 2nd direction different from the said 1st direction from the said angle sensor, The said 4th input information is acquired. Image processing apparatus. The image processing apparatus according to claim 1, wherein the light field data is an image captured from a plurality of different viewpoints. The generation unit generates the image data on the reconstruction plane set by the reconstruction plane setting unit by shifting and synthesizing each pixel of the light field data according to an incident angle of each pixel. The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. In a control method of an image processing apparatus that reconstructs light field data and generates image data on a predetermined reconstruction surface ,
A display control step of displaying an image based on the light field data on a display unit;
A first input step of acquiring first input information corresponding to a rotation axis using a touch operation from a user on a touch panel provided in the display unit as input information;
A second input step of acquiring second input information corresponding to the rotation angle;
By determining the rotation axis based on the first input information and determining the rotation angle based on the second input information, the surface rotated about the rotation axis by the rotation angle. a reconstruction plane setting step of setting the reconstruction plane,
Generating the image data on the reconstruction plane set by the reconstruction plane setting step from the light field data , and
The first input information includes information on positions of two points designated by the touch operation,
Wherein in the reconstruction plane setting step, based on the first input information, the control method for an image processing apparatus characterized that you determine the rotation shaft on the line connecting the two points. A program for controlling an image processing apparatus that reconstructs light field data and generates image data on a predetermined reconstruction plane ,
Computer
Display control means for displaying an image based on the light field data on a display unit;
First input means for acquiring first input information corresponding to a rotation axis, with a touch operation from a user on a touch panel provided in the display unit as input information;
Second input means for acquiring second input information corresponding to the rotation angle;
By determining the rotation axis based on the first input information and determining the rotation angle based on the second input information, the surface rotated about the rotation axis by the rotation angle. a reconstruction plane setting means for setting the reconstruction plane,
Generating means for generating the image data on the reconstruction plane set by the reconstruction plane setting means from the light field data ,
The first input information includes information on positions of two points designated by the touch operation,
The reconstruction plane setting means, on the basis of the first input information, a program to function as a respective means of the image processing device for determining the rotation shaft on the line connecting the two points. A computer-readable storage medium storing the program according to claim 11 . Computer program to function as each means of the images processing apparatus according to any one of claims 1 to 9. A storage medium storing a program that causes a computer to function as each unit of the image processing apparatus according to any one of claims 1 to 9 .

Images