JP6515978B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present technology relates to an image processing apparatus and an image processing method that can specify parameters for refocusing an image using a user interface (UI).

  In the image pickup apparatus, by recording the light field between the objective lens and the image pickup element, an image can be picked up, and then the image can be developed (refocused) freely by changing the in-focus position. The technology is open to the public. (For example, refer to patent document 1.)

  In addition, as a UI for specifying the in-focus position and the depth of field, which are necessary parameters for refocusing, touch the subject you want to focus on the screen to specify the in-focus position, and touch time There is one that specifies the depth of field by the length of. (For example, refer to patent document 2.)

Patent No. 4900723 JP, 2012-95186, A (3rd example)

  However, when the touch time length is used to specify the depth of field, the amount of time only increases as time passes, so the depth of field is increased only from the initial value or decreased. I could only specify the direction of the

  Also, the conventional UI has various other problems, and was not easy to use.

  In view of the above circumstances, an object of the present technology is to provide an image processing apparatus and an image processing method provided with an easy-to-use UI.

  In order to achieve the above object, an information processing apparatus according to an embodiment of the present technology displays a first image of a subject captured by an imaging device on a display screen, and the first image displayed on the display screen A position of an object to be focused based on at least one of the first position and the second position, each receiving a first position selected by the user and a second position different from the first position. Calculating the depth of field corresponding to the distance between the first position and the second position, focusing on the calculated position of the object, and calculating the calculated depth of field And a controller configured to display a second image having the image on the display screen.

  In the present technology, since the depth of field can be specified by the distance between the first position and the second position, an easy-to-use UI can be obtained.

  In order to achieve the above object, in the information processing apparatus according to one aspect of the present technology, the control unit may be configured to display a graphic having a size corresponding to the calculated depth of field on the display screen.

  In order to achieve the above object, in the information processing apparatus according to an embodiment of the present technology, the control unit sets the figure as the first position and the second position with the first position as a center. A configuration in which a circle whose radius is the distance may be displayed.

  In order to achieve the above object, an information processing apparatus according to an aspect of the present technology, between the first image and the second image, an imaging lens and an imaging element that acquires imaging data based on received light rays. An image processing unit that generates based on imaging data acquired in a state in which the traveling direction of a light beam is held by an imaging optical system having a microlens array in which one microlens is allocated to a plurality of pixels of the imaging device; May be further provided.

  In order to achieve the above object, in the information processing apparatus according to one aspect of the present technology, the image processing unit is configured to move from the imaging lens to the refocusing surface according to the first position received by the control unit. A distance information calculation unit that calculates distance information, a refocus coefficient setting unit that sets a refocus coefficient based on the depth of field and the distance information calculated by the distance information calculation unit, and the refocus coefficient setting unit And a reordering unit configured to generate an image at an arbitrary focal point by reordering the imaging data based on the refocusing coefficient set by the user.

  In order to achieve the above object, in the information processing apparatus according to one aspect of the present technology, the control unit displays, as the graphic, a circle whose diameter is a line connecting the first position and the second position. The configuration may be

  In order to achieve the above object, an electronic device according to an aspect of the present technology causes a display screen to display a first image of a subject captured by an imaging device, and a user displays the first image displayed on the display screen. Receiving a first position selected by the control unit and a second position different from the first position, and setting a position of an object to be focused based on at least one of the first position and the second position Calculating the depth of field corresponding to the distance between the first position and the second position, focusing on the calculated position of the subject, and calculating the calculated depth of field And a display unit having the display screen, and a position detection unit capable of detecting the first position and the second position.

  In order to achieve the above object, a server according to an aspect of the present technology transmits a first image of a subject captured by an imaging device to a terminal device, and a user at the terminal device on which the first image is displayed To receive from the terminal device a first position selected by the second position and a second position different from the first position, and focus based on at least one of the first position and the second position The position of the subject is calculated, the depth of field corresponding to the distance between the first position and the second position is calculated, the calculated position of the subject is focused, and the calculated object is calculated. A control unit is provided for transmitting a second image having a depth of field to the terminal device.

  In order to achieve the above object, an information processing program according to an aspect of the present technology displays a first image of a subject imaged by an imaging device on a display screen, and the first image displayed on the display screen A position of an object to be focused based on at least one of the first position and the second position, each receiving a first position selected by the user and a second position different from the first position. Calculating the depth of field corresponding to the distance between the first position and the second position, focusing on the calculated position of the object, and calculating the calculated depth of field And causing the computer to function as a control unit for causing the display screen to display a second image having

  In order to achieve the above object, in an information processing method according to an aspect of the present technology, a first image of a subject captured by an imaging device is displayed on a display screen, and the first image displayed on the display screen is displayed. The control unit receives a first position selected by the user and a second position different from the first position, and focuses on at least one of the first position and the second position. The control unit calculates the position of the subject to be calculated, the depth of field corresponding to the distance between the first position and the second position is calculated by the control unit, and the calculated position of the subject is focused. And a second image having the calculated depth of field is displayed on the display screen.

  As described above, according to the present technology, an easy-to-use UI can be provided.

It is a figure showing composition of an imaging device concerning this art. FIG. 2 is a functional block diagram showing an entire configuration of an image processing unit 14; FIG. 2 is a functional block diagram showing an entire configuration of a touch panel type display device 2 on which the imaging device 1 is mounted. It is a figure which shows a mode that a user touched photographic subject SUB1 on touch panel 150 on which picture IMG0 is displayed, and picture IMG1 was displayed. FIG. 7 is a diagram showing an example of a depth map DM for obtaining the distance d based on the coordinate position touched by the user on the touch panel 150. FIG. 31 is a diagram showing a state in which the user touches the circle on the object SUB2 with the index finger and displays the circle CIR0 while touching the circle with the thumb and extends the circle to a circle CIR1 with a larger radius. FIG. 31 is a diagram showing a state in which the user extends the circle CIR0 displayed by touching the subject SUB2 with the index finger to a circle CIR2 with a larger radius by dragging with the thumb. 15 is a flowchart for explaining the flow of processing of a refocusing UI. When a user touches subject SUB1, while circle CIR0 is drawn, it is a figure showing the example where the distance from imaging device 1 to subject SUB1 is displayed near touch position P0 by character string STR. FIG. 17 is a diagram showing an example in which a user touches two points P0 and P1 on touch panel 150 on which image IMG0 is displayed, and a circle having a diameter of two points touched is drawn.

Hereinafter, embodiments according to the present technology will be described with reference to the drawings.
[About device configuration]
The configuration of a display device according to the present technology will be described. The display device may include an imaging device that captures a refocused image. First, the configuration of the imaging device will be described.

[About the configuration of the imaging device]
FIG. 1 illustrates the entire configuration of an imaging device 1 according to an embodiment of the present technology. The imaging device 1 outputs image data Dout by imaging the imaging object 2 and performing image processing. The imaging device 1 includes an imaging lens 11 having an aperture stop 10, a microlens array 12, an imaging element 13, an image processing unit 14, an imaging element driving unit 15, and a control unit 16.

  The aperture stop 10 is an optical aperture stop of the imaging lens 11. An image of the object to be imaged 2 having a similar shape to the shape (for example, a circular shape) of the aperture of the aperture stop 10 is formed on the imaging element 13 for each microlens.

  The imaging lens 11 is a main lens for imaging the imaging target object 2 and is configured of, for example, a general imaging lens used in a video camera, a still camera or the like.

  The microlens array 12 is a two-dimensional array of a plurality of microlenses, and is disposed on the focal plane (imaging plane) of the imaging lens 11. Each of the microlenses has, for example, a circular planar shape, and is formed of, for example, a solid lens, a liquid crystal lens, or a diffractive lens.

  The imaging device 13 receives light rays from the microlens array 12 to acquire imaging data D0 (multi-viewpoint image) including a plurality of pixel data, and is provided on the focal plane (imaging plane) of the microlens array 12. It is arranged. The imaging device 13 is configured of a plurality of two-dimensional solid-state imaging devices such as a plurality of charge coupled devices (CCDs) or complementary metal-oxide semiconductors (CMOSs) arranged in a matrix.

  The image processing unit 14 can generate image (reconstructed image, refocused image) data Dout set to an arbitrary focus by performing predetermined image processing on the imaging data D0 obtained by the imaging device 13 It is supposed to be. At this time, the image processing unit 14 extracts predetermined distance information (disparity map or depth map) from the imaging data D0, and rearranges the imaging data based on the distance information. The detailed configuration of the image processing unit 14 will be described later. The processing of the image processing unit 14 may be realized by causing a computer to execute a program.

  The imaging device drive unit 15 drives the imaging device 13 to control the light receiving operation thereof.

  The control unit 16 controls operations of the image processing unit 14 and the imaging device driving unit 15, and is configured of, for example, a microcomputer.

  Here, the detailed configuration of the image processing unit 14 will be described with reference to FIG. FIG. 2 is a functional block diagram showing the entire configuration of the image processing unit 14. The image processing unit 14 includes a defect correction unit 141, a clamp processing unit 142, a distance information extraction unit 143, a refocus coefficient setting unit 149, a rearrangement processing unit 144, a noise reduction unit 145, an outline emphasis unit 146, and a white balance adjustment unit 147. And a gamma correction unit 148.

  The defect correction unit 141 corrects a defect (such as a defect caused by an abnormality in the element itself of the image pickup device 13) such as an overexposure included in the imaging data D0. The clamp processing unit 142 performs setting processing (clamp processing) of the black level of each pixel data in the image pickup data after the defect correction by the defect correction unit 141. In addition, color interpolation processing such as demosaicing processing may be performed on the imaging data subjected to the clamping processing.

  The distance information extraction unit 143 extracts predetermined distance information based on the imaging data D0, and includes, for example, a phase difference detection unit and a distance information calculation unit (all not shown) as follows. It is done.

  The phase difference detection unit generates a plurality of parallax images (arbitrary viewpoint images from different viewpoints) having different parallaxes based on the imaging data D1 supplied by the clamp processing unit 142, and at least 2 of the plurality of parallax images. The phase difference between the parallax images of the sheets is detected. In addition, the parallax image for detecting a phase difference extracts the pixel data acquired by the pixel P arrange | positioned in the same position between unit images light-received on the image pick-up element 13, and is synthesize | combined by combining. Can be generated. Therefore, the number of parallax images generated is the same as the number of pixels assigned to one microlens.

  The distance information calculation unit, based on the phase difference detected by the phase difference detection unit, distance information (distance d described later) between the imaging lens and the refocus plane to be focused (determine the in-focus position), That is, the focal length on the object side of the imaging lens at the time of refocusing is calculated.

  The refocusing coefficient setting unit 149 performs integration processing in the sorting processing unit 144 based on the distance information (calculated by the distance information calculating unit) extracted by the distance information extracting unit 143 and the predetermined depth of field information ( The refocusing coefficient α to be used for the refocusing calculation process is set. The imaging data D1 is input to the rearrangement processing unit 144 together with the refocusing coefficient α.

  The reordering unit 144 performs a predetermined reordering process on the imaging data D1 using the refocusing coefficient α set by the refocusing coefficient setting unit 149, for example, a method called “light field photography”. The imaging data D2 (refocused image) is generated by performing refocusing calculation processing using the above.

  In the refocused image, the depth of field is provided by a predetermined amount in the front and back directions with the designated in-focus position in between, but from the in-focus position to the front, The depth of field and the depth of field in the back side direction from the in-focus position may be equal. Alternatively, both depths of field may be distributed at a constant ratio, or may be set to any depth.

  The noise reduction unit 145 performs processing for reducing noise (for example, noise generated when imaging is performed in a dark place or a place where the sensitivity is insufficient) included in the imaging data D2 supplied by the rearrangement processing unit 144. is there. The contour emphasizing unit 146 performs contour emphasizing processing for emphasizing the contour of the image on the imaging data supplied by the noise reducing unit 145.

  The white balance adjustment unit 147 performs color balance on the imaging data supplied by the outline emphasizing unit 146 due to the influence of individual differences in devices such as the pass characteristics of the color filter and the spectral sensitivity of the imaging device 13 and illumination conditions. Adjustment processing (white balance adjustment processing) is performed.

  The gamma correction unit 148 generates imaging data Dout by performing predetermined gamma correction (correction of contrast and brightness) on the imaging data supplied by the white balance adjustment unit 147.

[Configuration of Display Device]
Next, the entire configuration of the display device will be described. FIG. 3 is a functional block diagram showing an entire configuration of a touch panel type display device 2 in which the imaging device 1 is mounted.

  The display device 2 can perform focus adjustment at an arbitrary position in the display image plane, and an image (imaging data DRout) set on a different focal plane from the displayed image (imaging data Dout) It is possible to switch to). The display device 2 includes an imaging device 1 including an image processing unit 14, a touch panel 150, a distance information calculation unit 151, a refocus coefficient setting unit 149, a rearrangement processing unit 144, a noise reduction unit 145, an outline emphasizing unit 146, and a white balance. The adjustment unit 147, the gamma correction unit 148, and a control unit (not shown) that integrally controls these functional blocks are configured. In addition, about the image processing similar to the said imaging device 1, the same code | symbol is attached and description is abbreviate | omitted suitably.

  The touch panel 150 has a display control unit that displays a reconstructed image based on the imaging data Dout output from the imaging device 1 on a display panel (display unit), and an arbitrary position of the displayed image is designated. It is comprised from the position detection part which detects a position. For example, when a finger, a pen, or the like contacts the display panel, the position where the contact is made can be detected by, for example, a pressure-sensitive type or an electrostatic type. As the display panel, a liquid crystal panel, an organic EL (Electro Luminescence) panel, or the like is used.

  The distance information calculation unit 151 is detected by the touch panel 150 from the phase difference disparity map or depth map DM of each parallax image detected by the distance information extraction unit 143 (phase difference detection unit) of the image processing unit 14 described above. The phase difference (disparity) is selected according to the position (Pixel), and the distance d is calculated based on the disparity.

  For example, when it is desired to generate an image (hereinafter referred to as image IMG1) focused (focused) at a designated position of a displayed image (hereinafter referred to as image IMG0), the distance d is It is calculated using the disparity at the detected position as the distance between the position detected by the position 150 and the refocus plane where the in-focus is achieved.

  In the display device 2 as described above, the display controller displays the image IMG0 based on the imaging data Dout on the touch panel 150. When an arbitrary position of the image IMG0 is specified, the specified position is detected, and the position is detected. Information (position information) and information about the depth of field (depth of field information) are input to the distance information calculation unit 151.

  On the other hand, the distance information calculation unit 151 receives the disparity map or depth map DM of each parallax image from the distance information extraction unit 143 (phase difference detection unit) in the image processing unit 14 of the imaging device 1.

  For example, after the disparity corresponding to the designated position is selected in the distance information calculating unit 151, the distance d is calculated based on the selected disparity, and is input to the refocusing coefficient setting unit 149 together with the imaging data Dout. Be done.

  As described above, the refocusing coefficient setting unit 149 sets the refocusing coefficient α based on the distance d and the depth of field information, and inputs the refocusing coefficient α together with the imaging data Dout to the rearrangement processing unit 144.

  The rearrangement processing unit 144 rearranges the imaging data Dout based on the refocusing coefficient α, and reconstructs an image (imaging data D3) focused at a designated position of the image IMG0, for example. The imaging data D3 thus reconstructed is subjected to predetermined image processing by the noise reducing unit 145, the outline emphasizing unit 146, the white balance adjusting unit 147, and the gamma correcting unit 148, and the touch panel 150 is used as the imaging data DRout. Is input to the display control unit of Thus, the display control unit causes the display panel to display the image IMG1 focused at the position designated in the image IMG0.

  As described above, the disparity is detected according to the designated position of the image IMG0, the predetermined distance information is calculated, the refocusing coefficient α is set, and the imaging data is rearranged. It is possible to instantaneously switch and display the image IMG1 on which focus adjustment has been performed at any position from IMG0.

<Modification 1 (Separation of Imaging Device and Display Device)>
In the above description, although the display device 2 includes the imaging device 1, the imaging device 1 and the display device 2B may be separated. In that case, imaging data Dout and DM are supplied to the display device 2B from the imaging device 1 outside, a server on the cloud, or the like.

<Modification 2 (Simplification of Display Device and Use of Cloud)>
Further, from the display device 2B of the first modification, the distance information calculation unit 151, the refocus coefficient setting unit 149, the rearrangement processing unit 144, the noise reduction unit 145, the outline emphasis unit 146, the white balance adjustment unit 147, and the gamma correction unit A configuration using the display device 2C except 148 may be employed. In that case, first, the imaging data Dout is supplied from the server on the cloud to the display device 2C, and the image IMG0 is displayed on the display panel of the touch panel 150.

  When the user touches the position of the object to be focused on to designate the position information, and further designates the depth of field information by the method described later, the display device 2C obtains the position information and the depth of field information as a server. Send to

  The server includes a distance information calculation unit 151, a refocus coefficient setting unit 149, a rearrangement processing unit 144, a noise reduction unit 145, an outline emphasis unit 146, a white balance adjustment unit 147, and a gamma correction unit 148. . The server generates imaging data DRout from these functional blocks and transmits the imaging data DRout to the display device 2C.

  The display device 2C displays the imaging data DRout supplied from the server as an image IMG1 on the display panel of the touch panel 150.

[About UI for Refocus]
Next, details of the UI for specifying the in-focus position and the depth of field at the time of refocusing will be described.

(Specifying the in-focus position)
First, a method of specifying a focusing position in the UI of the present technology will be described. The designation of the in-focus position for refocusing at a position desired by the user is designated by touching the object to be focused on on the image IMG0 displayed on the touch panel 150 by the user.

  FIG. 4 is a diagram showing a state in which the user touches the subject SUB1 on the touch panel 150 on which the image IMG0 is displayed, and the image IMG1 is displayed. As can be seen from this figure, when the user touches the subject SUB1, the distance d to the subject SUB1 is specified as the in-focus position, refocusing is performed, the subject SUB1 is in focus, and the subject SUB1 is behind An image IMG1 in which an object SUB2 or the like is blurred is displayed.

  Then, a circle CIR0 relatively representing the depth of the depth of field of the image IMG1 is displayed around the touch position P0.

  The circle CIR0 is drawn with the relative value of the specified value or the depth of field specified last time as a radius centering on the position P0 where the user touches the touch panel 150.

(Example of depth map)
Although not displayed on the display panel as a UI, an example of a depth map DM for obtaining the distance d based on the coordinate position touched by the user on the touch panel 150 is shown in FIG.

  In this figure, the distance d from the imaging device of each subject is shown by shading of color. In one subject, since the distances d are almost equal, they are expressed in approximately the same depth as the color depth. In the actual depth map DM, each pixel holds the value of the distance d.

  When the user touches the touch panel 150, XY coordinates of the position touched by the position detection unit are detected, and the distance d of the touch position can be obtained by comparing with the depth map DM.

  The depth map DM can be generated by providing the distance d calculated based on the above-described disparity map as a value for each pixel.

(Specifying the depth of field)
Next, a method of specifying the depth of field in the UI of the present technology will be described. The specification of the depth of field for refocusing at the depth of field desired by the user is performed by setting the size of the circle CIR0 drawn around the position P0 which the user first touched to specify the in-focus position. It is done by changing. The size change is performed by dragging the next touched position P1 to change the distance between the positions P0 and P1.

  FIG. 6 is a diagram showing a state in which the user touches the circle CIR0 displayed by touching the subject SUB2 with the index finger, and then dragging the circle CIR0 while touching it with the thumb to expand it to a circle CIR1 with a larger radius.

  In this figure, since the depth of field is still shallow, the subject SUB2 is in focus, but the subject SUB1 in front of the subject SUB2 is blurred. Further, the subject SUB3 behind the subject SUB2 is also blurred.

  Next, the situation when the depth of field is further changed is shown. FIG. 7 is a view showing a state in which the user extends the circle CIR0 displayed by touching the subject SUB2 with the forefinger to a circle CIR2 with a larger radius by dragging with the thumb.

In this figure, since the depth of field is further expanded, in addition to the subject SUB2, the subjects SUB1 and SUB3 located before and after the subject SUB2 are also in focus, unlike the previous figure.
So far, the UI for refocusing has been described.

[Flow of processing of UI]
Next, the flow of processing of the refocusing UI according to the present technology will be described. FIG. 8 is a flowchart for explaining the flow of processing of the refocusing UI.

(Processing when touching 2 points)
First, processing in the case of touching the first point and then the second point will be described.
First, the image processing unit 14 sets parameters of the in-focus position and the depth of field to initial values, and generates imaging data Dout using these initial values. (Step S1)

  Next, the display control unit of the touch panel 150 displays an image IMG0 based on the imaging data Dout on the display panel. (Step S2)

  Next, the position detection unit of the touch panel 150 determines whether or not the first touch on the touch panel 150 has occurred. (Step S3)

  When the first point of contact is made (Yes in step S3), the distance information calculation unit 151 acquires the coordinates of the position P0 where the contact is made from the position detection unit, and uses the depth map DM acquired from the image processing unit 14 Based on this, the distance d of the position P0 is determined. (Step S4)

  Next, the refocusing coefficient setting unit 149 changes the parameter of the in-focus position from the calculated distance d, and resets the refocusing coefficient α. (Step S5)

  Next, the reordering unit 144 reconstructs the refocused image using the reset refocusing coefficient α, and the image IMG1 is generated. The generated image IMG1 is displayed on the display panel by the display control unit of the touch panel 150. (Step S6)

  Next, the display control unit of the touch panel 150 draws a circle having a size corresponding to the set depth of field on the display panel. (Step S7)

  Next, the position detection unit of the touch panel 150 determines whether or not there is a second touch on the touch panel 150. (Step S8)

  If there is a second touch (Yes in step S8), the distance information calculation unit 151 acquires the coordinates of the position P1 where the touch occurs from the position detection unit, and based on the coordinates of the position P0 and the position P1, Calculate the distance between points. (Step S9)

  Next, the refocusing factor setting unit 149 resets the parameter of the depth of field from the calculated distance. (Step S10)

After step 10, the display device 2 returns control to step 4 to continue the process.
The process in the case of touching two points has been described above.

(Process when there is no first point of contact)
In step S3, when there is no first contact (No in step S3), next, the position detection unit of the touch panel 150 determines whether or not a predetermined time has passed without the first contact. (Step S11)

  If the predetermined time has not passed yet (No in step S11), the display device 2 returns the control to step S3.

  If the predetermined time has elapsed (Yes in step S11), the refocusing coefficient setting unit 149 resets the parameter of the depth of field to the initial value and resets it. (Step S12)

After step S12, the display device 2 returns the control to step S3.
The process in the case where there is no first contact has been described above.

(Process when there was a first point of contact but there was no second point of contact)
In step S8, when there is no second contact (No in step S8), next, the position detection unit of the touch panel 150 determines whether or not the first contact is held. (Step S13)

  If the first contact is held (Yes in step S13), the display device 2 returns the control to step 4 and continues the process.

  When the first contact is not held (No in step S13), the display device 2 returns the control to step 3 and continues the process.

  As described above, the process in the case where the first point of contact has been made but the second point of contact has not been described.

<Modification 3 (Display of Distance)>
In the above description, the circle is displayed when the first touch is performed. However, when drawing the circle, the distance to the touched subject may be displayed at the same time.

  FIG. 9 is a diagram showing an example in which the circle CIR0 is drawn when the user touches the subject SUB1, and the distance from the imaging device 1 to the subject SUB1 is displayed near the touch position P0 by the character string STR. is there.

<Modification 4 (circle with diameter of two points touched)
In the above description, a circle is drawn with the first point touched as the center and the distance between the second point touched and the first point as the radius, but a circle with the distance between the two points touched as the diameter is drawn It is also good. In this configuration, the position of the midpoint of the line segment connecting the two touched points is designated as the position to be focused. The two points of touch may be simultaneous or may be sequentially touched one by one.

  FIG. 10 is a diagram showing an example in which a user touches two points P0 and P1 on the touch panel 150 on which the image IMG0 is displayed, and a circle whose diameter is the two points touched is drawn. As shown in this figure, when drawing a circle whose diameter is the two touched points, it is not necessary to place a finger at the center point of the circle, so it is easy to see the part that will be the in-focus position without hiding with fingers. There is an advantage.

<Modification 5 (representation of depth of field by graphic)>
In the above description, the circle is drawn to indicate the relative depth of the depth of field. However, the present invention is not limited to this. For example, the depth of field may be determined according to the length of the line segment or the size of other figures. It may represent relative depth.

  Other figures include triangles, squares, heart marks, and those imitating the diaphragm blades of a camera.

<Modification 6 (Aperture Value Designation by Circle)>
In the above description, the size of the circle represents the relative depth of field depth, but the size of the circle represents not the depth of field but the aperture value of the camera lens. It may be In this case, when the circle becomes smaller, the aperture is narrowed and the depth of field becomes deeper. When the circle becomes larger, the aperture is opened and the depth of field becomes shallower.

<Modification 7 (Display of Aperture Value)>
In the third modification, the distance is displayed near the first touch position, but an aperture value corresponding to the set depth of field may be displayed.

<Modification 8 (Default Circle Size)>
An appropriate size may be selected according to the size and resolution of the display panel of the touch panel 150, as the size of the circle displayed corresponding to the initial depth of field.

  For example, in the case where the display device 2 is a television receiver having a 50-inch screen size and the case where it is a 5-inch smartphone, the size of the displayed image IMG0 is significantly different. However, since the size of the user's hand operating both is the same, the size of the circle displayed as the specified value is easy to use if it is adjusted so that the user can easily touch with a finger.

  In the case of a display device 2 having a 9-inch display panel, it is conceivable to draw a circle with a radius of 2 centimeters, for example.

  Of course, the size of the predetermined circle may be set by the user.

<Modification 9 (default blur method)>
In the above description, the depth of field when the first touch point is initially set to the initial value is used to reconstruct the refocused image, but the initial value is set by the user. It may be changed by According to this configuration, the user can change the default blur pattern when the user touches the first point to a preferred blur pattern.

<Modification 10 (Reset Area)>
In the UI of the present technology, after the user causes the display panel to display the refocused image according to an arbitrary focus position and depth of field, the user explicitly sets the focus position and depth of field as initial values. A reset area may be provided to receive an instruction to reset.

  When the user touches the reset area, the in-focus position and the depth of field can be easily reset to the initial values.

<Modification 11 (Retention of depth of field after releasing hand)>
In the UI of the present technology, after the user touches and operates the touch panel 150 to specify the in-focus position and the depth of field, the specified depth of field is obtained even if the user releases the touch panel 150. The image may be displayed for a predetermined time.
With this configuration, the user can release the finger from the touch panel 150 and can view the refocused image in a state where there is no shield on the display panel.

<Modification 12 (Reset due to no operation for a fixed time)>
In the UI of the present technology, when the user does not touch the touch panel 150 for a certain period of time, the in-focus position of the refocused image and the depth of field may be reset to initial values.

<Modification 13 (All-in-Focus Image When Specification of Depth of Field Exceeds Default Value)>
In the UI of the present technology, when the depth of field specified by the user exceeds a predetermined value, the display device 2 may be configured to generate an omnifocal image as a refocused image.

<Modification 14 (generate a refocused image in advance)>
In the above description, the refocused image is reconstructed each time the user touches the touch panel 150. However, the present invention is not limited to this configuration, and the in-focus position may be specified by the user in advance from multi-viewpoint images. A plurality of images corresponding to the depth of field may be constructed and stored in a storage unit provided in the display device or the server.

  In this configuration, it is possible to display the refocused image at high speed because time for rebuilding the refocused image after the user touches can be omitted.

<Modification 15 (Preparation of Image by Typical Camera)>
Although the modification 14 is configured to construct a plurality of images corresponding to the in-focus position and the depth of field that may be designated by the user in advance from the multi-viewpoint image, the present invention is not limited to this. A configuration may be used in which a plurality of images corresponding to the in-focus position and the depth of field that may be designated by the user are captured using a storage unit provided in the display device or the server.

  In this configuration, the user can view the refocused image on the display device without using a light field camera such as the imaging device 1.

<Modified Example 16 (Specification of Position Not Due to Touch)>
In the above description, the user has specified the in-focus position and the depth of field by touching the touch panel 150, but the invention is not limited thereto. The specification of the in-focus position and the depth of field is a noncontact method. It may be performed by the position detection apparatus of the above, or may be performed using a mouse.

<Modification 17 (Specification of depth of field not touching the circle)>
In the above description, in order to specify the depth of field, the user touches the circumference of the circle displayed by the first touch on the touch panel 150, and slides the finger while touching the subject. I changed the depth of field. However, the present invention is not limited to this configuration, and even if the user does not touch the circumference of the circle, the touch position on the touch panel 150 may be accepted as the second touch position.

  Even in this case, when the user slides the finger from the position touched as the second point, the size of the displayed circle is scaled according to the distance between the first and second touch positions.

[Supplementary matter]
In addition, the present technology is not limited to only the above-described embodiment, and it goes without saying that various modifications can be made without departing from the scope of the present technology.

Reference Signs List 1 imaging device 2 to 2C display device 10 aperture stop 11 imaging lens 12 micro lens array 13 imaging device 14 image processing unit 15 imaging device driving unit 16 control unit 141 defect correction unit 142 clamp Processing unit 143 ... Distance information extraction unit 144 ... Rearrangement processing unit 145 ... Noise reduction unit 146 ... Contour emphasis unit 147 ... White balance adjustment unit 148 ... Gamma correction unit 148
149 Refocusing factor setting unit 150 Touch panel 151 Distance information calculation unit

Claims (13)

A first display image to be displayed on a display device is generated based on imaging data, and an in-focus position is obtained by specifying an arbitrary subject in the first display image displayed on the display device, and the designation Generation of a second display image in which the depth of field is increased or decreased from the first display image based on the user's input for changing the displayed graphic in association with the displayed position of the subject Image processing apparatus having a unit.   The image processing apparatus according to claim 1, wherein the in-focus position is obtained by a distance from an imaging device of an object specified by a user on a screen of the display device.   The image processing apparatus according to claim 1, wherein the graphic simulates an aperture blade of a camera. The display device has a touch panel,
The image processing apparatus according to any one of claims 1 to 3, wherein the graphic is deformed according to a distance dragged from a position touched by the user.   The image processing apparatus according to any one of claims 1 to 4, wherein the imaging data is data obtained by a light field camera.   The imaging data holds the traveling direction of the light beam by an imaging optical system having a microlens array in which one microlens is allocated to a plurality of pixels of the imaging device between the imaging lens and the imaging device. The image processing apparatus according to any one of claims 1 to 5, which is acquired imaging data.   The image generation unit is configured to reset the depth of field to an initial value when the user's input does not occur for a certain period after the depth of field of the first display image is changed. An image processing apparatus according to any one of the preceding claims.   The image generation unit generates an omnifocal image when the depth of field of the first display image changed by the deformation of the graphic exceeds a predetermined value. Image processing apparatus as described.   At least a plurality of second display images different in in-focus position are generated and stored in the storage unit, and when the in-focus position is designated by the user, any one closer to the designated in-focus position from the storage unit The image processing apparatus according to any one of claims 1 to 8, wherein the second display image is read out and displayed on the display device.   The image processing apparatus according to any one of claims 1 to 9, wherein the imaging data is acquired via a server.   The image processing apparatus according to claim 1, further comprising a display device configured to display the display image generated by the image generation unit. An imaging device for acquiring imaging data of a subject;
A first display image to be displayed on a display device is generated based on imaging data, and an in-focus position is obtained by specifying an arbitrary subject in the first display image displayed on the display device, and the designation Generation of a second display image in which the depth of field is increased or decreased from the first display image based on the user's input for changing the displayed graphic in association with the displayed position of the subject An image processing apparatus comprising a unit. The image generation unit generates a first display image to be displayed on a display device based on imaging data, and obtains an in-focus position by specifying an arbitrary subject in the first display image displayed on the display device. And a second display image obtained by increasing or decreasing the depth of field from the first display image based on the user's input for changing the displayed shape in association with the designated position on the display of the subject Image processing method to generate.