JP6323022B2 - Image processing device - Google Patents

Description

The present invention relates to an image processing apparatus.

  There is known an electronic device that can set a focus position and the like after photographing a subject. There is a digital camera having a focus bracket function as this type of electronic device (see, for example, Patent Document 1). Focus bracketing refers to taking a plurality of images with different in-focus positions by changing the in-focus position while continuously shooting. For example, the digital camera described in Patent Document 1 appropriately selects a plurality of pieces of image data having different in-focus positions, and changes the in-focus position by combining the selected image data.

JP 2003-141506 A

  However, in the above-described electronic device, a refocus function for performing focus adjustment after shooting is realized, but the usability of such an electronic device is not sufficient, and there is still room for improvement.

  In an aspect of the present invention, it is an object to improve the operability of an electronic device by changing at least one of an in-focus position of an image and a depth of field according to a user's operation.

According to an aspect of the present invention, a display unit for displaying the video playback speed is being played at a first rate, it is found provided on the display unit, a detecting unit for detecting a contact touch position, the contact by the detection unit When the detected position is detected, a control unit that reproduces the moving image at a second speed slower than the first speed, and a detecting unit that reproduces the moving image at the second speed by the control unit . and contacting the Rifuku number of positions, the change of the contact is a plurality of positions are detected, based on different image data with depth of field that is generated the depth of field by changing parameters, the display And a processing unit that changes the depth of field of the image displayed on the unit. According to the first aspect of the present invention, the operation unit that can be operated by the user and the display that displays the image on the display unit and changes the display state of the image according to the first operation of the operation unit by the user. Provided is an electronic device comprising: a control unit; and a changing unit that changes at least one of an in-focus position and a depth of field according to a second operation of the operation unit by a user during image display Is done.

  According to the second aspect of the present invention, an image is displayed on the display unit on the control device of the electronic device having the operation unit that can be operated by the user, and the image is displayed according to the first operation of the operation unit by the user. A display control process for changing the state, and a change process for changing at least one of the in-focus position and the depth of field according to the second operation of the operation unit by the user during the display of the image. A control program to be executed is provided.

  According to the aspect of the present invention, it is possible to improve the operability of the electronic device by changing at least one of the focus position of the live view image or the moving image and the depth of field according to the operation of the user. .

It is sectional drawing which shows schematic structure of the optical system part in the light field camera which is an example of an electronic device. It is a figure explaining the pixel arrangement | sequence and incident area of an image pick-up element. It is a block diagram which shows the structure of the light field camera which concerns on 1st Embodiment. It is a figure explaining the reconstruction of an image. It is a flowchart for demonstrating the image process which the image process part of 1st Embodiment performs. It is a figure which shows the distance relationship between a light field camera and each to-be-photographed object. It is a flowchart for demonstrating the imaging | photography operation | movement which the system control part of 1st Embodiment performs. It is a figure which shows the example of a display of the live view image displayed on the display screen of a display part. It is a flowchart for demonstrating the moving image reproduction process which the system control part of 1st Embodiment performs. It is a figure which shows the 1st example of a display during the moving image reproduction | regeneration of 1st Embodiment. It is a figure which shows the 2nd example of a display during the moving image reproduction | regeneration of 1st Embodiment. It is a figure which shows the 3rd display example during the moving image reproduction | regeneration of 1st Embodiment. It is a flowchart for demonstrating the imaging | photography operation | movement which the system control part of 2nd Embodiment performs. It is a block diagram which shows the structure of the imaging device and electronic device which concern on 3rd Embodiment.

  Embodiments of the present invention will be described below with reference to the drawings. However, the present invention is not limited to this. Further, in the drawings, in order to describe the embodiment, the scale may be appropriately changed and expressed, for example, a part of the drawing may be enlarged or emphasized. In the following embodiments, a light field camera will be described as an example of an electronic device.

<First Embodiment>
FIG. 1 is a cross-sectional view illustrating a schematic configuration of an optical system portion in a light field camera which is an example of an electronic apparatus. In FIG. 1, the right direction (optical axis direction) of the paper surface is the X axis, the upper direction of the paper surface orthogonal to the X axis is the Z axis, and the direction from the back of the paper surface orthogonal to the X axis and Z axis is the direction from the back. Is the Y axis. FIG. 1 is a cross-sectional view when the optical system portion in the light field camera is cut along a plane including the optical axis.

  The light field camera of the present embodiment is an electronic device that can set a focus position, a viewpoint, and the like after photographing a subject. This light field camera acquires information on a light field (light space) that is a set of light beams in a three-dimensional space, and generates an image based on the acquired information on the light field. In the following description, the light field information is referred to as light ray information. As a typical function of the light field camera, there is a refocus function for generating an image whose focal length is changed by post-processing after photographing. Therefore, the light field camera is also called a refocus camera.

  The light field camera of this embodiment includes a photographing optical system (imaging optical system) 11 and a microlens array 12 as its optical system part. FIG. 1 also shows an image sensor 20 that is a configuration other than the optical system portion of the light field camera.

  The photographing optical system 11 is a single focus lens composed of a plurality of lens groups. The photographing optical system 11 forms an image of a light beam from the subject in the vicinity of its focal plane (imaging plane). Further, the photographing optical system 11 includes an aperture stop (not shown) that controls an aperture value (F value) during photographing. Since the aperture value can be changed by reconstructing the image after shooting, the aperture diameter of the aperture stop may be fixed. As shown in FIG. 1, a microlens array 12 and an image sensor 20 are sequentially arranged near the focal plane of the photographing optical system 11.

  The microlens array 12 has a configuration in which a plurality of microlenses (positive lenses) having positive power are two-dimensionally arranged. In the microlens array 12 shown in FIG. 1, only three microlenses are arranged in the Z-axis direction, but actually, a large number of microlenses are arranged in the Z-axis direction and the Y-axis direction.

The image pickup device 20 is disposed on the rear side of the microlens array 12 (opposite side of the photographing optical system 11) with a slight gap. The image sensor 20 receives light beams that have passed through the microlenses of the microlens array 12 by a plurality of photoelectric conversion elements, and photoelectrically converts the light beams received by the photoelectric conversion elements to generate a plurality of pixel signals. The plurality of photoelectric conversion elements are arranged in a matrix, and each photoelectric conversion element forms a pixel. In the present embodiment, the image sensor 20 is a CCD (Charge Coupled Device) or CMOS (Complementary).
It consists of a two-dimensional solid-state image sensor such as a metal-oxide semiconductor).

  In the example shown in the partially enlarged view of FIG. 1, a plurality of pixels are arranged on the rear side of each of the plurality of microlenses 12a, 12b, and 12c. Specifically, three pixels 21a, 21b, 21c are arranged on the rear side of the micro lens 12a, three pixels 22a, 22b, 22c are arranged on the rear side of the micro lens 12b, and on the rear side of the micro lens 12c. Three pixels 23a, 23b, and 23c are arranged.

  Three pixels 21a, 21b, and 21c are pixels 21 corresponding to the microlens 12a, three pixels 22a, 22b, and 22c are pixels 22 corresponding to the microlens 12b, and three pixels 23a, 23b, and 23c are microscopic. This is a pixel 23 corresponding to the lens 12c. In the partially enlarged view of FIG. 1, three pixels are shown as pixels corresponding to the microlenses 12a, 12b, and 12c, but actually, a large number of pixels are arranged in the Z-axis direction and the Y-axis direction ( (See FIG. 2 described later).

  Next, acquisition of a light field by a light field camera will be briefly described with reference to FIG. In FIG. 1, it is assumed that the subject at the position A focuses on the position of the microlens array 12 by the photographing optical system 11. The light beam from the point A1 on the subject passes through the partial region 11a on the pupil of the photographing optical system 11, passes through the microlens 12a, and is received by the pixel 21c of the image sensor 20. Further, another light ray from the point A1 on the subject passes through the partial region 11b on the pupil of the photographing optical system 11, passes through the microlens 12a, and is received by the pixel 21b of the imaging device 20. Further, another light ray from the point A1 on the subject passes through the partial region 11c on the pupil of the photographing optical system 11, passes through the microlens 12a, and is received by the pixel 21a of the imaging device 20.

  Similarly, the light beam from the point A2 on the subject passes through the partial area 11a on the pupil of the photographing optical system 11, passes through the microlens 12b, and is received by the pixel 22c of the image sensor 20. Further, another ray from the point A2 on the subject passes through the partial region 11b on the pupil of the photographing optical system 11, passes through the microlens 12b, and is received by the pixel 22b of the image sensor 20. Further, another light ray from the point A2 on the subject passes through the partial region 11c on the pupil of the photographing optical system 11, passes through the microlens 12b, and is received by the pixel 22a of the imaging device 20.

  Further, the light beam from the point A3 on the subject passes through the partial area 11a on the pupil of the photographing optical system 11, passes through the microlens 12c, and is received by the pixel 23c of the image sensor 20. Further, another light ray from the point A3 on the subject passes through the partial region 11b on the pupil of the photographing optical system 11, passes through the microlens 12c, and is received by the pixel 23b of the image sensor 20. Further, another light ray from the point A3 on the subject passes through the partial region 11c on the pupil of the photographing optical system 11, passes through the microlens 12c, and is received by the pixel 23a of the image sensor 20. As described above, the light beam from the subject at the position A is received by the pixels of the image sensor 20 located on the rear side of the microlenses 12a, 12b, and 12c, whereby the brightness and the traveling direction are acquired as light beam information. The

  Of the three pixels corresponding to the microlenses 12a, 12b, and 12c, the image generated by the pixel signals of the lower pixels 21c, 22c, and 23c is an image of light rays that have passed through the partial region 11a. In addition, the image generated by the pixel signals of the middle pixels 21b, 22b, and 23b is an image of light rays that have passed through the partial region 11b. Further, the image generated by the pixel signals of the upper pixels 21a, 22a, and 23a is an image of light rays that have passed through the partial region 11c. For example, when the number of pixels corresponding to each microlens is N, N reconstructed images configured by arranging pixel values of pixels located at the same position with respect to the microlens are configured to capture the photographing optical system 11 as N. N stereo image groups acquired by dividing into partial regions are formed.

  FIG. 2 is a diagram illustrating the pixel array and the incident area of the image sensor 20. FIG. 2 shows a state where the image sensor 20 is observed from the X-axis direction of FIG. In the image sensor 20 shown in FIG. 2, a region where pixels are arranged is referred to as a pixel region 200. In the pixel region 200, for example, 10 million or more pixels are arranged in a matrix. The circular area in the pixel area 200 is an area where light rays that have passed through one microlens are incident. Accordingly, a plurality of circular regions are formed for each of the plurality of microlenses. These regions are referred to as incident regions 201, 202. For example, 586 × 440 microlenses are arranged. However, such an arrangement number is an example, and a larger or smaller number may be used. In the example shown in FIG. 2, the diameter of one microlens is 13 pixels. However, the number of pixels is not limited to this.

  As described above, the light field camera extracts and reconstructs pixel values of pixels at the same position in the incident area (the same position in the Y-axis direction and the Z-axis direction in the incident area) from all the microlenses. The reconstructed image group obtained in this way corresponds to an image group in which a subject is observed from different partial areas of the photographing optical system 11. Therefore, these reconstructed image groups have parallax according to the position of the partial region and the distance to the subject. For example, as shown in FIG. 2, a reconstructed image composed of pixel values of pixels of each microlens located at the same position as the pixel 201 a in the incident area 201 and each micro located at the same position as the pixel 201 b in the incident area 201. It has parallax with the reconstructed image composed of pixel values of the lens pixels.

  FIG. 3 is a block diagram showing a configuration of the light field camera 1 according to the first embodiment. The light field camera 1 shown in FIG. 3 includes an imaging unit 10, an image processing unit 30, a work memory (storage unit) 40, a display unit 50, an operation unit 55, a recording unit 60, an audio output unit 65, and a system control unit 70. Prepare. The imaging unit 10 includes a photographing optical system 11, a microlens array 12, a drive control unit 13, an imaging element 20, and a drive unit 21. Since the photographing optical system 11 and the microlens array 12 have been described with reference to FIG. 1, description thereof is omitted in FIG.

  For example, a Bayer array color filter array is disposed on the imaging surface of the imaging element 20. The image sensor 20 makes the light beam that has passed through the photographing optical system 11 and the microlens array 12 enter. The image sensor 20 photoelectrically converts an incident light beam to generate a pixel signal of each pixel of an analog signal. In the imaging unit 10, an amplifier (not shown) amplifies the pixel signal of each pixel of the analog signal with a predetermined gain factor (amplification factor). Further, an A / D converter (not shown) converts the pixel signal of each pixel of the analog signal amplified by the amplifier into the pixel signal of each pixel of the digital signal. Then, the imaging unit 10 outputs RAW data including pixel signals of each pixel of the digital signal to the image processing unit 30. The light ray information acquired by the light field camera 1 is specified by RAW data output from the imaging unit 10.

  The drive control unit 13 performs aperture stop drive control based on control information transmitted from the system control unit 70 in order to adjust the aperture diameter of the aperture stop included in the photographing optical system 11. The drive unit 21 is a control circuit that controls driving of the image sensor 20 under imaging conditions set based on an instruction signal from the system control unit 70. Examples of imaging conditions include exposure time (shutter speed), frame rate, gain, and the like.

  Here, the exposure time refers to the time from when the photoelectric conversion element of the image sensor 20 starts to accumulate after the start of charge accumulation according to incident light. The frame rate is a value representing the number of frames processed (displayed or recorded) per unit time in a moving image. The unit of the frame rate is represented by fps (Frames Per Second). The gain refers to a gain factor (amplification factor) of an amplifier that amplifies the pixel signal. By changing this gain, the ISO sensitivity can be changed. This ISO sensitivity is a photographic film standard established by ISO and represents how much light the photographic film can record. However, generally, ISO sensitivity is also used when expressing the sensitivity of the image sensor 20. In this case, the ISO sensitivity is a value representing the ability of the image sensor 20 to capture light.

  The image processing unit 30 performs various kinds of image processing on the RAW data including the pixel signal of each pixel, and generates image data in a predetermined file format (for example, JPEG format). The image data generated by the image processing unit 30 includes image data of still images, moving images, and live view images. The live view image is an image displayed on the display unit 50 by sequentially outputting the image data generated by the image processing unit 30 to the display unit 50. The live view image is used for the user to confirm the image of the subject imaged by the imaging unit 10. The live view image is also called a through image or a preview image. As illustrated in FIG. 3, the image processing unit 30 includes an image generation unit 31, a first editing unit 32, a second editing unit 33, and a changing unit 34.

  The image generation unit 31 uses the work memory 40 as a work space to generate image data set at an arbitrary focal length by performing predetermined image processing on RAW data including pixel signals of each pixel. Specifically, the image generation unit 31 generates image data set at an arbitrary focal length by performing the following image processing.

  FIG. 4 is a diagram for explaining image reconstruction. As shown in FIG. 4, the image generation unit 31 has pixel values (values indicated by pixel signals) of pixels 201 a, 202 a... At the same position with respect to the respective incident areas 201, 202. To extract. The image generation unit 31 rearranges the extracted pixel values to generate reconstructed image data 500a. When the number of pixels corresponding to each of the incident areas 201, 202... Is N, N reconstructed image data are generated.

  The image generation unit 31 translates the N reconstructed image data groups by a predetermined amount in accordance with a predetermined focal length. Then, the image generation unit 31 averages the N reconstructed image data groups after the parallel movement. Thereby, image data set to a predetermined focal length is generated. Since the image of the subject existing at a depth corresponding to the amount of movement is aligned, a sharp contour image is formed. An image of a subject that does not exist at a depth corresponding to the amount of movement is blurred. Note that the above-described image processing is an example, and the image generation unit 31 may generate image data set at an arbitrary focal length by executing image processing different from the above-described image processing. For example, the image generation unit 31 calculates a light ray acquired by each pixel of the image sensor 20 by performing a predetermined calculation. Then, the image generation unit 31 generates image data set to a predetermined focal length by projecting the calculated light beam onto a virtual image plane set to a predetermined focal length.

  In the present embodiment, the image generation unit 31 has a focal position (focal point) such as image data focused on the foreground subject, image data focused on the backmost subject, and image data focused on a subject at an intermediate position. A plurality of image data having different distances are generated.

  The image generation unit 31 performs various image processing on the image data set at an arbitrary focus using the work memory 40 as a work space. For example, the image generation unit 31 generates RGB image data by performing color signal processing (tone correction) on a signal obtained by the Bayer array. The image generation unit 31 performs image processing such as white balance adjustment, sharpness adjustment, gamma correction, and gradation adjustment on the RGB image data. Further, the image generation unit 31 performs a process of compressing in a predetermined compression format (JPEG format, MPEG format, etc.) as necessary.

  The image generation unit 31 detects one or a plurality of subjects from the generated image data. In the present embodiment, the image generating unit 31 detects the subject by specifying the boundary of the subject region based on the contrast and color change between the bright and dark portions in the image data. Then, the image generation unit 31 outputs position data indicating the detected size and position of the subject in pixel units to the system control unit 70.

  Note that the image generation unit 31 may detect a subject using a known face detection function for a human subject. The image generation unit 31 may detect a moving subject by comparing a plurality of pieces of image data obtained in time series from the image generation unit 31 for a moving subject (moving subject). Further, in addition to the face detection, the image generation unit 31 may detect a human body included in the image data as a subject as described in, for example, Japanese Patent Application Laid-Open No. 2010-16621 (US2010 / 0002940). .

  In response to an instruction from the system control unit 70, the first editing unit 32 edits image data of a predetermined subject in the live view image or moving image (that is, processing for deleting the predetermined subject or orientation of the predetermined subject). To change). When the first editing unit 32 performs processing for deleting image data of a predetermined subject, the first editing unit 32 uses a plurality of reconstructed image data having parallax (see FIG. 8 described later) to delete the region of the deleted subject. Process to complement the image data.

  In response to an instruction from the system control unit 70, the second editing unit 33 edits the image data of the predetermined subject in the live view image or the moving image (that is, the process of deleting the predetermined subject or the direction of the predetermined subject). To change). Further, when the second editing unit 33 performs the process of deleting the image data of the predetermined subject, the second editing unit 33 moves the predetermined subject in the live view image or the moving image of the predetermined period stored in the work memory 40 (described later). Based on FIG. 8, a process for complementing the image data of the deleted subject area is performed.

  The changing unit 34 performs a process of changing at least one of the in-focus position and the depth of field of the live view image or moving image in accordance with an instruction from the system control unit 70. Note that the depth of field refers to the range of the focal length of a subject that appears to be in focus. For example, when the focus position and the depth of field range are designated by an instruction from the system control unit 70, the changing unit 34 acquires a plurality of image data having a focus position close to the designated focus position. To do. Then, the changing unit 34 applies an appropriate weight to the image data and then performs averaging, thereby generating image data in the range of the designated in-focus position and depth of field.

  The image processing unit 30 outputs the generated image data to the recording unit 60. In addition, the image processing unit 30 outputs the generated image data to the display unit 50.

  The work memory 40 temporarily stores RAW data, image data, and the like when image processing by the image processing unit 30 is performed. The display unit 50 displays an image captured by the imaging unit 10 and various information. The display unit 50 includes a display screen 51 configured by, for example, a liquid crystal display panel. The image displayed on the display unit 50 includes a still image, a moving image, and a live view image. A touch panel (operation unit) 52 is formed on the display screen 51 of the display unit 50. The touch panel 52 outputs a signal indicating a position touched by the user to the system control unit 70 when the user performs an operation such as selection of a subject displayed on the display screen 51.

  The operation unit 55 includes a release switch (a switch that is pressed when shooting a still image), a moving image switch (a switch that is pressed when shooting a moving image), various operation switches, and the like that are operated by the user. The operation unit 55 outputs a signal corresponding to the operation by the user to the system control unit 70. The recording unit 60 has a card slot into which a storage medium such as a memory card can be mounted. The recording unit 60 stores the image data and various data generated by the image processing unit 30 in a recording medium mounted in the card slot. The recording unit 60 has an internal memory. The recording unit 60 can also record the image data and various data generated by the image processing unit 30 in the internal memory. The audio output unit 65 performs audio output from the speaker based on the instruction signal from the system control unit 70.

  The system control unit 70 controls the overall processing and operation of the light field camera 1. The system control unit 70 has a CPU (Central Processing Unit). As shown in FIG. 3, the system control unit 70 includes a display control unit 71, a selection unit 72, and an imaging control unit 73. The display control unit 71 performs control to output the image data generated by the image processing unit 30 to the display unit 50 and display an image (live view image, still image, moving image) on the display screen 51 of the display unit 50. . In addition, the display control unit 71 performs control to read out the image data recorded in the recording unit 60 and output the image data to the display unit 50 and display an image on the display screen 51 of the display unit 50.

  In addition, the display control unit 71 performs control to change the display state of the live view image or the moving image according to the touch operation of the touch panel 52 by the user. Specifically, the display control unit 71 changes the playback speed (the number of frames to be displayed per unit time) of the live view image or video being displayed on the display unit 50 in accordance with the touch operation of the touch panel 52 by the user. Control.

  The selection unit 72 recognizes the size and position of the subject in the image data based on the position data indicating the size and position of the subject output from the image processing unit 30. In addition, the selection unit 72 causes the first editing unit 32 to execute image data editing processing based on whether the subject to be edited has moved a predetermined amount within a predetermined time, or causes the second editing unit 33 to perform image processing. Select whether to execute the data editing process. Then, the selection unit 72 outputs to the image processing unit 30 an instruction signal that instructs the selected editing unit (the first editing unit 32 or the second editing unit 33) to execute an image data editing process.

  The imaging control unit 73 provides an instruction signal for instructing predetermined imaging conditions in order to perform imaging with predetermined imaging conditions (exposure time, frame rate, gain, etc.) on the imaging surface (pixel region 200) of the image sensor 20. Output to the drive unit 21. In addition, the imaging control unit 73 sends an instruction signal instructing the focus position and depth of field of the image data generated based on the RAW data to the image processing unit 30 in response to a touch operation on the touch panel 52 by the user. Output. Further, the imaging control unit 73 instructs a control parameter to cause the image processing unit 30 to execute image processing with predetermined control parameters (control parameters such as color signal processing, white balance adjustment, gradation adjustment, and compression rate). The instruction signal is output to the image processing unit 30. In addition, the imaging control unit 73 performs control for causing the recording unit 60 to record the image data generated by the image generation unit 31.

  In addition to the above control, the system control unit 70 also performs control such as adjustment of the aperture diameter of the aperture stop provided in the photographing optical system 11.

  In the system control unit 70, the display control unit 71, the selection unit 72, and the imaging control unit 73 correspond to processing and control executed by the CPU based on a control program (not shown), respectively.

  Next, image processing executed by the image processing unit 30 will be described. FIG. 5 is a flowchart for explaining image processing executed by the image processing unit 30 according to the first embodiment. In the processing shown in FIG. 5, the image generation unit 31 acquires RAW data output from the imaging unit 10 (step S1). Then, the image generation unit 31 generates image data of a predetermined in-focus position (focal length, focal position) based on the acquired RAW data (Step S2).

  Specifically, as described above, the image generation unit 31 extracts pixel values of pixels at the same position with respect to each incident region in the pixel region 200, and reconstructs the extracted pixel values to generate a plurality of pixels. A reconstructed image data group is generated. Then, the image generating unit 31 translates the plurality of reconstructed image data groups according to a predetermined focus position, and adds and averages the plurality of reconstructed image data groups after the parallel movement. As a result, image data set at a predetermined in-focus position is generated. In step S2, the image generation unit 31 generates a plurality of image data having different in-focus positions. The image data generated in step S2 includes still image data, moving image data, and live view image data.

  FIG. 6 is a diagram illustrating a distance relationship between the light field camera 1 and each of the subjects O1 to O4. In the example shown in FIG. 6, the light field camera 1 is located at the camera position d0. In addition, the person O1 who is the subject is located at the in-focus position d1. The subject vehicle O2 is located at the in-focus position d2, the subject house O3 is located at the in-focus position d3, and the subject tree O4 is located at the in-focus position d4. In FIG. 6, the direction from the light field camera 1 toward the subjects O1 to O4 is referred to as the back side direction, and the opposite direction is referred to as the near side direction. The direction on the back side and the direction on the near side are referred to as the depth direction. In step S2, the image generation unit 31 generates, for example, five image data at each of the in-focus positions d1 to d4.

  Returning to the description of FIG. 5, the image generation unit 31 performs color signal processing (tone correction), white balance adjustment, sharpness adjustment, and four image data having different focus positions d1 to d4 generated in step S2. Various image processes such as gamma correction and gradation adjustment are executed (step S3).

  Next, the image generation unit 31 detects the subjects O1 to O4 from the image data generated by the image generation unit 31 (step S4). The image generation unit 31 outputs position data indicating the detected sizes and positions of the subjects O1 to O4 to the system control unit 70.

  In addition, the image generation unit 31 detects (calculates) the distance (focal length) to each of the subjects O1 to O4 detected in step S4 (step S5). For example, the image generation unit 31 detects and detects the parallax of each of the subjects O1 to O4 in a plurality of reconstructed image data (for example, reconstructed image data composed of pixel values of pixels at two specific positions in the incident region). Based on the parallax, the distances to the subjects O1 to O4 are calculated. At this time, the image generation unit 31 calculates the distance to the person O1 as d1, the distance to the car O2 as d2, the distance to the house O3 as d3, and the distance to the tree O4 as d4. The image generation unit 31 outputs distance data indicating the distances to the detected subjects O1 to O4 to the system control unit 70.

  Thereafter, when there is a subject editing request from the system control unit 70 (see step S18 and step S19 in FIG. 7), the image generation unit 31 edits the image data of the subject requested to be edited (step S6). . The image data edited by the image generation unit 31 is referred to as edited image data.

  Next, the photographing operation of the light field camera 1 according to the first embodiment will be described. FIG. 7 is a flowchart for explaining a photographing operation executed by the system control unit 70 of the first embodiment.

  In the processing shown in FIG. 7, after the light field camera 1 is turned on, when the user operates the operation unit 55 or the like to start shooting, the imaging control unit 73 captures a live view image. Start (step S11). In other words, the imaging control unit 73 executes drive control of the imaging element 20 by outputting an instruction signal to the driving unit 21. The display control unit 71 displays the live view image captured by the image sensor 20 on the display screen 51 of the display unit 50 (step S12).

  FIG. 8 is a diagram illustrating a display example of the live view image 500 displayed on the display screen 51 of the display unit 50. The live view images 500 shown in FIGS. 8A and 8B are based on the distance relationship between the light field camera 1 and the subjects O1, O3, and O4 shown in FIG. , O4 is a live view image. In the live view image 500 shown in FIGS. 8A and 8B, the car O2 is not displayed. Here, the image data of the live view image 500 is, for example, image data focused on the person O3 among a plurality of pieces of image data having different focus positions d1, d3, and d4 (that is, a focus position d3 that is an intermediate position). Image data). Note that the image data of the live view image 500 may be image data focused on all the subjects O1, O3, and O4. For example, the image data of the live view image 500 is image data obtained by the image generation unit 31 extracting and synthesizing the image data of the subjects O1, O3, and O4 from the image data at all in-focus positions d1, d3, and d4. May be.

  In the example shown in FIGS. 8A and 8B, in the live view image 500, the person O1 is displayed on the lower left side. House O3 is displayed on the lower left side, and tree O4 is displayed on the right side.

  As described above, the reconstructed image data group generated by the image generation unit 31 has parallax with respect to the subjects O1 to O4. The parallax increases as the subject is closer to the light field camera 1 and decreases as the subject is farther from the light field camera 1. Therefore, the parallax with respect to the person O1 is the largest and the parallax with respect to the tree O4 is the smallest. In FIG. 8A, the parallax in the person O1 is also shown. The position of the person O1 ′ in some reconstructed image data and the position of the person O1 ″ in other reconstructed image data differ depending on the parallax. Although not shown in FIG. The tree O4 also has parallax between the plurality of reconstructed image data, and Fig. 8B shows a state where the person O1 is moving to the right side.

  Next, the selection unit 72 recognizes the subjects O1, O3, and O4 included in the live view image 501 (step S13). Specifically, the selection unit 72 requests the image processing unit 30 to detect the subject. When receiving a request for detecting a subject from the system control unit 70, the image generation unit 31 detects each of the subjects O1, O3, and O4 included in the live view image 500 (see step S4 in FIG. 5). Then, the image generation unit 31 outputs position data indicating the detected size and position of each of the subjects O1, O3, and O4 in units of pixels to the system control unit 70. Further, the image generation unit 31 detects the distances to the subjects O1 to O5 (see step S5 in FIG. 5). Then, the image generation unit 31 outputs distance data indicating the distances to the detected subjects O1, O3, and O4 to the system control unit 70. The selection unit 72 recognizes the size and position of each of the subjects O1, O3, and O4 based on the position data from the image processing unit 30. The selection unit 72 recognizes the distances to the subjects O1, O3, and O4 based on the distance data from the image processing unit 30.

  Next, the selection unit 72 determines whether or not the subject to be edited has been selected by the touch operation of the touch panel 52 by the user (step S14). For example, when the user selects the person O1 shown in FIG. 8A, the user touches the person O1 displayed on the display screen 51. The touch panel 52 formed on the display screen 51 outputs a signal indicating the position of the person O1 touched by the user to the system control unit 70. The selection unit 72 determines whether a subject has been selected by the user based on a signal output from the touch panel 52. Further, when the selection unit 72 determines that the subject has been selected by the user, which subject has been selected based on the position indicated by the signal from the touch panel 52 and the position of the subject recognized in step S13. Also judge. If the selection unit 72 determines that the subject has not been selected by the user (NO in step S14), the process ends.

  If the selection unit 72 determines in step S14 that the subject to be edited has been selected by the user (YES in step S14), the selection unit 72 stores the image data of the live view image in the work memory 40 as first storage data for a predetermined period. (Step S15). The work memory 40 is a storage unit that serves as a buffer for temporarily storing image data. Therefore, even when no subject is selected by the user, the image data of the live view image is stored for a predetermined period. May be. That is, the work memory 40 may always store the image data of the live view image for a predetermined period.

  Next, the selection unit 72 determines whether or not the subject to be edited (person O1) has moved a distance equal to or greater than a threshold value (reference amount) within a predetermined time in which the work memory 40 stores the image data (Step S72). S17). Here, the threshold is a movement distance of the subject such that the subject area before the subject to be edited does not overlap with the subject area after the subject movement. This threshold is a value that can be changed depending on the size of the subject to be edited. When the selection unit 72 determines that the subject to be edited has moved a distance equal to or greater than the threshold within a predetermined time, the selection unit 72 uses the image data of the region of the subject to be edited based on the image data stored in the work memory 40. It is determined that it can be complemented. On the other hand, if the selection unit 72 determines that the subject to be edited has not moved a distance greater than or equal to the threshold within a predetermined time, the image data of the region of the subject to be edited is stored in the work memory 40. It is determined that complementation is not possible based on the image data.

  If the selection unit 72 determines that the subject to be edited has not moved a distance equal to or greater than the threshold (NO in step S17), the image processing unit 30 causes the first editing unit 32 to generate edited image data. An instruction signal for instructing the image processing unit 30 to execute the editing processing of the image data of the live view image is output to the image processing unit 30 (step S18). At this time, when the editing content (deletion or change of orientation) of the image data is selected by the user's operation of the operation unit 55 or the like, the selection unit 72 sends an instruction signal instructing the editing content to the image processing unit. 30 may be output.

  In response to the instruction signal from the system control unit 70, the first editing unit 32 uses the plurality of reconstructed image data having parallax to edit the image data of the subject to be edited in the live view image (that is, the first editing unit 32). A process for deleting the subject or a process for changing the direction of the subject is performed. In addition, when the first editing unit 32 performs the process of deleting the image data of the subject to be edited, the first editing unit 32 complements the image data of the deleted subject region using a plurality of reconstructed image data having parallax. Process. By performing the editing process by the first editing unit 32 as described above, edited image data in which the subject to be edited is deleted or the direction of the subject to be edited is changed is generated.

  If the selection unit 72 determines that the subject to be edited has moved a distance equal to or greater than the threshold (YES in step S17), the image processing unit 30 causes the second editing unit 33 to generate edited image data. An instruction signal for instructing execution of image data editing processing of the live view image is output to the image processing unit 30 (step S19). Also at this time, when the editing content (deletion or change of orientation) of the image data is selected by the user's operation of the operation unit 55 or the like, the selection unit 72 outputs an instruction signal for instructing the editing content. You may output with respect to the part 30.

  The second editing unit 33 uses the image data of the live view image stored in the work memory 40 for a predetermined time in response to the instruction signal from the system control unit 70, and the image data of the subject to be edited in the live view image. (I.e., a process for deleting a subject or a process for changing the direction of a subject). In addition, when the second editing unit 33 performs the process of deleting the image data of the subject to be edited, the second editing unit 33 uses the image data of the live view image stored in the work memory 40 for a predetermined time to determine the deleted subject. A process for complementing the image data of the region is performed. When the editing process is performed by the second editing unit 33, edited image data in which the subject to be edited is deleted or the orientation of the subject to be edited is changed is generated.

  The display control unit 71 causes the display unit 50 to output the edited image data generated by the first editing unit 32 or the second editing unit 33, so that the edited image (an image in which the editing target subject is deleted or the editing target subject) is output. Is displayed on the display screen 51 of the display unit 50 (step S20). In addition, the imaging control unit 73 outputs the edited image data generated in the first editing unit 32 or the second editing unit 33 to the recording unit 60, and records the edited image data in the recording unit 60 as second saved data ( (Save) (step S21).

  In the process described with reference to FIG. 7, the system control unit 70 causes the image processing unit 30 (the first editing unit 32 or the second editing unit 33) to edit the image data of the predetermined subject in the live view image. However, the image processing unit 30 (the first editing unit 32 or the second editing unit 33) may edit the image data of the predetermined subject in the moving image. Specifically, when the selection unit 72 determines in step S14 that the subject to be edited has been selected by the user (YES in step S14), the selection unit 72 starts shooting a moving image. Then, the selection unit 72 stores the moving image data as first storage data in the work memory 40 for a predetermined period (see step S15).

  Next, the selection unit 72 determines whether or not the subject to be edited has moved a distance equal to or greater than a threshold (reference amount) within a predetermined time when the work memory 40 stores the image data (see step S17). If the selection unit 72 determines that the subject to be edited has not moved a distance equal to or greater than the threshold value, the selection unit 72 causes the image processing unit 30 to generate a moving image image in order to cause the first editing unit 32 to generate edited image data. An instruction signal instructing execution of the data editing process is output to the image processing unit 30 (see step S18). In response to an instruction signal from the system control unit 70, the first editing unit 32 uses a plurality of reconstructed image data having parallax to edit image data of a subject to be edited in a moving image (that is, the subject is selected). A process of deleting or a process of changing the direction of the subject).

  When the selection unit 72 determines that the subject to be edited has moved a distance equal to or greater than the threshold value, the selection unit 72 causes the image processing unit 30 to generate a moving image image in order to cause the second editing unit 33 to generate edited image data. An instruction signal instructing execution of the data editing process is output to the image processing unit 30 (see step S19). The second editing unit 33 uses the moving image data stored in the work memory 40 for a predetermined time in accordance with an instruction signal from the system control unit 70 to edit the image data of the subject to be edited in the moving image. (In other words, a process for deleting the subject or a process for changing the direction of the subject).

  The display control unit 71 causes the display unit 50 to output the edited image data generated by the first editing unit 32 or the second editing unit 33, so that the edited image (an image in which the editing target subject is deleted or the editing target subject) is output. Is displayed on the display screen 51 of the display unit 50 (see step S20). In addition, the imaging control unit 73 outputs the edited image data generated in the first editing unit 32 or the second editing unit 33 to the recording unit 60, and records the edited image data in the recording unit 60 as second saved data ( (Save) (see step S21).

  Further, in the processing described with reference to FIG. 7, the system control unit 70 edits image data of a predetermined subject in the live view image or moving image to the image processing unit 30 (the first editing unit 32 or the second editing unit 33). However, the image processing unit 30 (the first editing unit 32 or the second editing unit 33) may edit image data of a predetermined subject in a plurality of still images taken continuously. In this case, when it is determined in step S14 that the subject to be edited is selected by the user (YES in step S14), the selection unit 72 starts continuous shooting (continuous shooting) of still images. Then, the selection unit 72 stores the image data of a plurality of still images in the work memory 40 as first saved data for a predetermined period (see step S15).

  If the selection unit 72 determines that the subject to be edited has not moved a distance equal to or greater than the threshold (reference amount) within a predetermined time when the work memory 40 stores the image data (see NO in step S17). ) In order to cause the first editing unit 32 to generate the edited image data, an instruction signal instructing the image processing unit 30 to execute the editing processing of the image data of the still image is output to the image processing unit 30 ( (See step S18). In response to an instruction signal from the system control unit 70, the first editing unit 32 uses a plurality of reconstructed image data having parallax to edit image data of a subject to be edited in a still image (ie, subject Or a process of changing the direction of the subject).

  If the selection unit 72 determines that the subject to be edited has moved a distance equal to or greater than the threshold (see YES in step S17), the selection unit 72 causes the second editing unit 33 to generate edited image data. An instruction signal for instructing the image processing unit 30 to execute editing processing of moving image data is output to the image processing unit 30 (see step S19). The second editing unit 33 uses the image data of a plurality of still images stored in the work memory 40 for a predetermined time in response to an instruction signal from the system control unit 70, and image data of the subject to be edited in the still image. (I.e., a process for deleting a subject or a process for changing the direction of a subject).

  The display control unit 71 causes the display unit 50 to output the edited image data generated by the first editing unit 32 or the second editing unit 33, so that the edited image (an image in which the editing target subject is deleted or the editing target subject) is output. Is displayed on the display screen 51 of the display unit 50 (see step S20). In addition, the imaging control unit 73 outputs the edited image data generated in the first editing unit 32 or the second editing unit 33 to the recording unit 60, and records the edited image data in the recording unit 60 as second saved data ( (Save) (see step S21).

  Next, processing for changing the in-focus position and the depth of field during moving image display (reproduction) will be described. FIG. 9 is a flowchart for explaining the moving image reproduction process executed by the system control unit 70 of the first embodiment. FIG. 10 is a diagram illustrating a first display example during moving image reproduction according to the first embodiment. FIG. 11 is a diagram illustrating a second display example during moving image reproduction according to the first embodiment. FIG. 12 is a diagram illustrating a third display example during moving image reproduction according to the first embodiment.

  In the processing shown in FIG. 9, the display control unit 71 performs a moving image being shot by the imaging unit 10 (that is, a moving image immediately after being generated by the image processing unit 30) or the like according to an operation of the operation unit 55 or the like by the user. The moving image data recorded in the recording unit 60 is output to the display unit 50 and displayed on the display screen 51 of the display unit 50 (step S31).

  The user touches the display screen 51 of the display unit 50 to change the in-focus position or the depth of field of the moving image being displayed (reproducing). The touch operation by the user at this time is referred to as a first operation. Based on the signal from the touch panel 52, the display control unit 71 determines whether or not there has been a touch operation on the screen by the user (step S32). If the display control unit 71 determines that there is no touch operation on the screen by the user, the display control unit 71 ends the process.

  If the display control unit 71 determines that the user has touched the screen (YES in step S32), the display control unit 71 slows down the moving image reproduction speed to execute slow motion reproduction (step S33). Specifically, the display control unit 71 performs control to reduce the playback speed of the moving image being displayed on the display unit 50. At this time, when the moving image being shot by the imaging unit 10 is played back in step S31, the display control unit 71 changes the playback speed to match the frame rate of the moving image. In addition, when the display control unit 71 executes the slow motion reproduction for the moving image being shot by the imaging unit 10, the slow motion reproduction is performed in order to absorb the difference between the moving image being shot and the displayed moving image. During this time, the moving image is recorded in the work memory 40 or the recording unit 60.

  In the example illustrated in FIG. 10, when the user touches the display screen 51 (that is, the touch panel 52) when the moving image 501 is displayed on the display screen 51 (FIG. 10 (1)), the reproduction speed of the moving image 502 is slow. (FIG. 10 (2)). In the moving image 502, the moving speed of the car O2 is slower than that of the moving image 501.

  When the user wants to change the in-focus position of the moving image that is being displayed (reproduced), the user touches the subject for which the in-focus position is to be changed among the subjects displayed on the display screen 51. The touch operation by the user at this time is referred to as a second operation. The display control unit 71 determines whether or not the user has touched the subject (step S34).

  If the display control unit 71 determines that the user has touched the subject (YES in step S34), the display control unit 71 focuses on the subject selected by the touch operation (the subject touched by the user). An instruction signal for instructing to change the in-focus position of the moving image is output to the image processing unit 30 so that the image is in focus (step S35). The changing unit 34 acquires a plurality of image data having a focus position close to the focus position specified by the instruction signal from the system control unit 70. Then, the changing unit 34 applies the appropriate weights to these image data and then averages them to generate image data at the designated in-focus position. Then, the changing unit 34 outputs the generated image data to the display unit 50. As a result, an image whose focus position has been changed is displayed on the display screen 51.

  In the example shown in FIG. 10, when the user touches the car O2 in the video 503 displayed on the display screen 51 (FIG. 10 (3)), the video 504 is displayed from the display state focused on the house O3 in the video 503. The display state is changed to focus on the car O2 (FIG. 10 (4)). In the example shown in FIG. 11, when the user touches the car O2 in the moving image 503 displayed on the display screen 51 (FIG. 11 (3)), the display state is focused on the house O3 in the moving image 503. The display state is gradually changed to the display state focused on the car O2 in the moving image 503A (FIG. 11 (3A)), and the display state is focused on the car O2 in the moving image 504 (FIG. 11 (4)). 11 (1) and 11 (2) are display examples similar to those in FIGS. 10 (1) and 10 (2), description thereof will be omitted.

  The user performs a pinch-out operation or a pinch-in operation on the display screen 51 in order to change the range of the depth of field of the moving image being displayed (playing back). The pinch-out operation is an operation in which the finger is moved away after the display screen 51 is pinched with the finger. The pinch-in operation is an operation in which the finger is brought close after the display screen 51 is pinched with the finger. These operations are also referred to as second operations. The display control unit 71 determines whether a pinch-out operation or a pinch-in operation has been performed by the user (step S36). In this embodiment, the range of the depth of field is widened by a pinch-out operation by the user, and the range of the depth of field is narrowed by a pinch-in operation by the user.

  If the display control unit 71 determines that there has been a pinch-out operation or a pinch-in operation by the user (YES in step S36), the display control unit 71 changes the range of the depth of field so as to correspond to these operations. An instruction signal instructing to change the depth of field is output to the image processing unit 30 (step S37). The changing unit 34 applies an appropriate weight to a plurality of pieces of image data having different in-focus positions so as to be within the range of the depth of field designated by the instruction signal from the system control unit 70, and then averaging. The image data in the range of the designated depth of field is generated. Then, the changing unit 34 outputs the generated image data to the display unit 50. Thereby, an image in which the range of the depth of field is changed is displayed on the display screen 51.

  In the example shown in FIG. 12, when the user performs a pinch-out operation on the display screen 51 (FIGS. 12 (3-1) and (3-2)), the range of the depth of field in the moving image 504 (for example, The range extending back and forth in the depth direction from the in-focus position of the vehicle O2 becomes wider (FIG. 12 (4)). 12 (1) and 12 (2) are display examples similar to those shown in FIGS. 10 (1) and 10 (2), and a description thereof will be omitted.

  Next, when the change of the in-focus position and the depth of field is completed, the display control unit 71 returns the moving image playback speed to the original playback speed (step S38). 10 (4), 11 (4), and 12 (4), the playback speed of the moving image 504 is returned to the original playback speed (that is, the moving speed of the car O2 is returned to the original moving speed). State).

  In the process described with reference to FIG. 9, the system control unit 70 performs the slow motion playback of the moving image, the change of the in-focus position of the moving image, and the change of the depth of field of the moving image according to the touch operation of the user. I was going. However, the system control unit 70 can also perform slow-motion playback of the live view image, change the in-focus position of the live view image, and change the depth of field of the live view image according to the user's touch operation. Is possible. In this case, the system control unit 70 does not live in the work memory 40 or the recording unit 60 for the time during which slow motion playback is performed in order to absorb the deviation between the live view image being shot and the live view image being displayed. Record the view image.

  The system control unit 70 also plays back a still image (for example, frame-by-frame playback or slide playback), changes the focus position of the still image, and changes the depth of field of the still image according to a user's touch operation. It is also possible to make changes. In this case, the system control unit 70 records a plurality of continuous still images in the work memory 40 or the recording unit 60 for the time during which the still image is being reproduced.

  Further, the system control unit 70 may pause the moving image or the live view image instead of performing the slow motion reproduction of the moving image or the live view image in response to a user's touch operation. In addition, the system control unit 70 may pause the still image instead of frame-by-frame playback or slide playback of the still image according to a user's touch operation. Further, the system control unit 70 performs slow motion reproduction, change of the focus position, and change of the depth of field according to the operation of the operation unit 55 by the user instead of the touch operation of the touch panel 52 by the user. You may go. Further, the system control unit 70 may change only one of the change of the focus position and the depth of field.

  As described above, in the first embodiment, the operation unit 52 (or the operation unit 55) that can be operated by the user and the image (live view image, moving image, or still image) are displayed on the display unit 50, A display control unit 71 that changes the display state of the image according to the first operation of the operation unit 52 by the user, and an in-focus position of the image according to the second operation of the operation unit 52 by the user during image display. And a changing unit 34 that changes at least one of the depth of field. According to such a configuration, the operability of the light field camera 1 which is an electronic device is improved. That is, in the electronic device described in Patent Document 1, the focus position of the still image is changed. However, in the light field camera 1 of the first embodiment, at least the focus position of the image and the depth of field are at least. Either one can be changed. In addition, in the electronic device described in Patent Document 1, the user changes the focus position of the image by moving the scroll bar. However, in the case of such an operation method, the live view image or the moving image is focused. It is inappropriate for changing the position. On the other hand, in the light field camera 1 of the first embodiment, the live view image or the in-focus position of the moving image is changed by the touch operation of the touch panel 52 by the user. The focus position can be changed.

  In the first embodiment, the display control unit 71 lowers the image reproduction speed or pauses the image according to the first operation of the operation unit 52 by the user, so that the user selects a subject. The operability such as changing the in-focus position is further improved.

  In the first embodiment, the changing unit 34 changes the focus position of the subject designated by the second operation of the operation unit 52 by the user. According to such a configuration, it can be easily confirmed by focusing on the subject in the image. In the first embodiment, the changing unit 34 changes the depth of field including the in-focus position of the subject specified by the second operation of the operation unit 52 by the user. Even with such a configuration, the user can easily confirm the subject in the image.

  In the first embodiment, the display control unit 71 restores the display state of the image after the change by the changing unit 34. Therefore, the image can be displayed at a normal reproduction speed after the user's subject is confirmed. .

  In the first embodiment, the image generation unit 31 generates a plurality of images having parallax with respect to one or a plurality of subjects, and the storage unit 40 stores the plurality of images generated by the image generation unit 31 for a predetermined period. A first editing unit 32 that edits the image of the predetermined subject based on the parallax of the plurality of images, and the image of the predetermined subject is edited based on the movement of the predetermined subject of the image over a predetermined period stored in the storage unit 40 A second editing unit 33, and a selection unit 72 that selects one of the first editing unit 32 and the second editing unit 33. According to such a configuration, it is possible to edit an image of a predetermined subject in accordance with the movement of the subject in a live view image or a moving image. That is, when the predetermined subject is not moving, the image processing unit 30 cannot perform editing such as deletion on the predetermined subject based on the image data stored in the storage unit 40. Further, when the size of the predetermined subject is large, the image processing unit 30 cannot perform editing such as deletion on the predetermined subject based on a plurality of image data having parallax. However, in the first embodiment described above, since the selection unit 72 selects one of the first editing unit 32 and the second editing unit 33, editing such as deletion can be performed on a predetermined subject. (Or more likely).

  In the first embodiment, the selection unit 72 detects the amount of movement of a predetermined subject during a predetermined period, and compares one of the first editing unit 32 and the second editing unit 33 by comparing the movement amount with a reference amount. Select. Accordingly, it is possible to reliably delete the predetermined subject in the image processing unit 30. In the first embodiment, the first editing unit 32 and the second editing unit 33 delete the predetermined subject or change the direction of the predetermined subject as editing the predetermined subject. Therefore, it is possible to edit the image data desired by the user. For example, it is possible to edit an image in which a subject existing in front of the main subject is deleted, or to an image in which a person's subject is facing the front.

Second Embodiment
In the first embodiment described above, the selection unit 72 detects the amount of movement of the subject in a predetermined period and compares one of the first editing unit 32 and the second editing unit 33 by comparing the movement amount with the reference amount. Was selected. On the other hand, in the second embodiment, the display control unit 71 causes the display unit 50 to display the image edited by the first editing unit 32 and the image edited by the second editing unit 33, and selects the selection unit 72. Selects one of the first editing unit 32 and the second editing unit 33 according to the touch operation of the touch panel 52 by the user.

  FIG. 13 is a flowchart for explaining a photographing operation executed by the system control unit 70 of the second embodiment. In the processing shown in FIG. 13, the same processing (steps S1 to S15) as the processing shown in FIG.

  If the selection unit 72 determines in step S14 that the subject to be edited has been selected by the user (YES in step S14), the selection unit 72 stores the image data of the live view image in the work memory 40 as first storage data for a predetermined period. At the same time (step S15), in order to cause the first editing unit 32 to generate the edited image data, an instruction signal for instructing the image processing unit 30 to execute the editing process of the image data of the live view image is sent to the image processing unit. 30 (step S41). Note that the process of step S41 is a process corresponding to the process of step S18 shown in FIG.

  In addition, the selection unit 72 performs image processing on an instruction signal that instructs the image processing unit 30 to execute image data editing processing of the live view image in order to cause the second editing unit 33 to generate edited image data. It outputs to the part 30 (step S42). The process of step S42 is a process corresponding to the process of step S19 shown in FIG.

  The display control unit 71 causes the display unit 50 to output the edited image data generated by the first editing unit 32 and the second editing unit 33 to display two edited images (an image in which the subject to be edited has been deleted or an editing target). Is displayed on the display screen 51 of the display unit 50 (step S43). The user determines which of the two edited images displayed on the display screen 51 has been edited without deteriorating the image quality. Then, the user touches one of the two edited images and selects the edited image. The imaging control unit 73 outputs the edited image data selected by the user's touch operation on the touch panel 52 to the recording unit 60, and records (saves) the edited image data as the second saved data in the recording unit 60 (step). S44).

  In the process described with reference to FIG. 13, the system control unit 70 causes the image processing unit 30 (the image generation unit 31 or the first editing unit 32) to edit image data of a predetermined subject in the live view image. The image processing unit 30 (the image generation unit 31 or the first editing unit 32) may edit image data of a predetermined subject in the moving image.

  As described above, in the second embodiment, the display control unit 71 includes the image edited by the first editing unit 32 (live view image, moving image, or still image) and the image edited by the second editing unit 33 (live). (View image, moving image, or still image) is displayed on the display unit 50, and the selection unit 72 selects one of the first editing unit 32 and the second editing unit 33 according to the operation of the operation unit 52 by the user. To do. According to such a configuration, the edited image data according to the user's judgment can be recorded in the recording unit 60.

<Third Embodiment>
In the third embodiment, the light field camera 1 in the first embodiment described above is separated into an imaging device 1A and an electronic device 1B.

  FIG. 14 is a block diagram illustrating configurations of the imaging apparatus 1A and the electronic apparatus 1B according to the third embodiment. In the configuration shown in FIG. 14, the imaging device 1 </ b> A is a device that captures an image of a subject and has a refocus function. The imaging apparatus 1A includes an imaging unit 10, an image processing unit 30, a work memory 40, an operation unit 55, a recording unit 60, an audio output unit 65, and a first system control unit 70A. In the imaging apparatus 1A, the configurations of the imaging unit 10, the image processing unit 30, the work memory 40, the operation unit 55, the recording unit 60, and the audio output unit 65 are the same as the configurations illustrated in FIG. Accordingly, the same components are denoted by the same reference numerals, and redundant description is omitted.

  The electronic device 1B is a device that displays images (still images, moving images, live view images). The electronic device 1B includes a display unit 50 and a second system control unit 70B. The configuration of the display unit 50 in the electronic apparatus 1B is the same as the configuration illustrated in FIG. Accordingly, the same components are denoted by the same reference numerals, and redundant description is omitted.

  The first system control unit 70A includes a first communication unit 75A. The second system control unit 70B has a second communication unit 75B. The first communication unit 75A and the second communication unit 75B transmit and receive signals to each other in a wired or wireless manner. In such a configuration, the first system control unit 70A receives the second image data (image data image-processed by the image processing unit 30 and image data recorded in the recording unit 60) via the first communication unit 75A. It transmits to the communication part 75B. The second system control unit 70B causes the display unit 50 to display the image data received by the second communication unit 75B.

  The configuration shown in FIG. 3 (the display control unit 71, the selection unit 72, and the imaging control unit 73) may be provided in either the first system control unit 70A or the second system control unit 70B. 3 may be provided in the first system control unit 70A or the second system control unit 70B, and a part of the configuration shown in FIG. 3 is provided in the first system control unit 70A. Configurations other than a part of the configuration shown in FIG. 3 may be provided in the second system control unit 70B.

  Note that the imaging apparatus 1A includes, for example, a digital camera, a smartphone, a mobile phone, and a personal computer having an imaging function and a communication function, and the electronic device 1B includes, for example, a smartphone, a mobile phone, and a portable personal computer that have a communication function. Consists of a portable terminal such as a computer.

  In the first system control unit 70A illustrated in FIG. 14, processing executed by the CPU based on the control program corresponds to the configuration of all or part of the display control unit 71, the selection unit 72, and the imaging control unit 73. Further, in the second system control unit 70B shown in FIG. 14, the processing executed by the CPU based on the control program corresponds to all or part of the configuration of the display control unit 71, the selection unit 72, and the imaging control unit 73.

  As described above, in the third embodiment, in addition to the effects described in the first embodiment and the second embodiment, an image captured by the imaging apparatus 1A using a mobile terminal such as a smartphone is displayed on the electronic device 1B. Can be displayed on the display unit 50.

  As mentioned above, although this invention was demonstrated using embodiment, the technical scope of this invention is not limited to the range as described in the said embodiment. Various modifications or improvements can be added to the above-described embodiment without departing from the spirit of the present invention. In addition, one or more of the requirements described in the above embodiments may be omitted. Such modifications, improvements, and omitted forms are also included in the technical scope of the present invention. In addition, the configurations of the above-described embodiments and modifications can be applied in appropriate combinations.

  For example, in each of the above-described embodiments, the light field camera 1 is cited as an example of the electronic device. However, the present invention is not limited to this. For example, a camera array in which a plurality of cameras are arranged may be used. Even in such a camera array, a plurality of images having parallax can be acquired.

  Further, the image processing unit 30 and the system control unit 70 shown in FIG. 3 may be configured integrally. In this case, the system control unit having one CPU executes the processing based on the control program, thereby taking on the functions of the image processing unit 30 and the system control unit 70. In the configuration shown in FIG. 14, the image processing unit 30 and the first system control unit 70A may be configured integrally. In this case, the system control unit having one CPU performs the processing based on the control program, thereby performing the function of the image processing unit 30 and the function of the first system control unit 70A.

  In the first embodiment described above, the arrangement of the color filters 102 is a Bayer arrangement, but an arrangement other than this arrangement may be used.

  Further, the light field camera 1 according to each of the above-described embodiments may include a zoom function for performing zooming adjustment automatically or in accordance with an operation of the operation unit 55 or the like by a user. In this case, the drive control unit 13 performs drive control of the photographing optical system 11 (for example, a zooming lens) based on control information transmitted from the system control unit 70 in order to perform zooming adjustment. The light field camera 1 can change the focal position after photographing the subject, but may include an autofocus function that automatically adjusts the focal position before photographing the subject. In this case, the drive control unit 13 performs drive control of the photographing optical system 11 (for example, a focusing lens) based on control information transmitted from the system control unit 70 in order to adjust the focus of the photographing optical system 11. .

  DESCRIPTION OF SYMBOLS 1 ... Light field camera (electronic device), 1A ... Imaging device, 1B ... Electronic device, 10 ... Imaging part, 20 ... Image sensor, 30 ... Image processing part, 31 ... Image generation part, 32 ... 1st editing part, 33 ... 2nd editing part, 34 ... Change part, 50 ... Display part, 51 ... Display screen, 52 ... Touch panel, 60 ... Recording part, 70 ... System control part, 70A ... 1st system control part, 70B ... 2nd system control 71, display control unit, 72 ... selection unit, 73 ... imaging control unit

Claims (6)

A display unit that displays a video that is being played at a first playback speed;
Wherein provided in the display portion, et al is, a detection unit for detecting a contact touch position,
When the detected position is detected by the detection unit, a control unit that reproduces the reproduction speed of the moving image at a second speed slower than the first speed;
When the video is played in the second speed by the control unit, and the contact to Rifuku number of positions by the detection unit, the change of the contact is a plurality of positions are detected, the A processing unit for changing the depth of field of the image displayed on the display unit based on image data having different depth of field generated by a parameter capable of changing the depth of field ;
An image processing apparatus comprising: Prior Symbol processor such that said subject image displayed on the display unit is focused, if generated a focus position by changing parameters to be selected based on the contacted plurality of locations The image processing apparatus according to claim 1, wherein an in- focus position of an image displayed on the display unit is changed based on image data having different focus positions . The image processing apparatus according to claim 1 , wherein the detection unit detects the contacted position by a touch panel provided on the display unit .   The image processing apparatus according to claim 3, wherein changes in the plurality of touched positions of the touch panel detected by the detection unit are caused by a pinch-out operation to the touch panel or a pinch-in operation to the touch panel.   When the detection unit detects the pinch-out operation, the processing unit widens the range of the depth of field of the image generated by image processing, and when the detection unit detects the pinch-in operation The image processing apparatus according to claim 4, wherein a range of a depth of field of the image generated by image processing is narrowed. When the detection unit detects the pinch-out operation, the processing unit widens a range of depth of field between the contacted positions of the image generated by image processing, and detects the detection The image processing apparatus according to claim 4, wherein when the pinch-in operation is detected by a unit, a range of a depth of field between the plurality of touched positions of the image generated by image processing is narrowed.