JP2017184144A - Image processing system, imaging apparatus, and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To solve a problem in that although a technology for generating a focused moving picture by tracking an arbitrary subject from moving-picture output data after photographing is known, a depth of field is not taken into consideration when the moving picture is generated from the moving-picture output data.SOLUTION: An image processing system comprises: a setting unit that sets a depth of field and the position of a subject at which the focal point of an imaging optical system meets in an optical axis direction, for first image data output from an imaging unit having a plurality of micro-lenses on which light from the subject transmitted through an imaging optical system is made incident and also having a plurality of light-receiving parts provided for the corresponding micro-lenses; a generating unit that generates, from the plurality of first image data, a plurality of second image data having the depth of field and the position of the focused subject set by the setting unit, and generates a plurality of pieces of moving-picture data having the plurality of second image data; and an alteration unit that alters the depth of field or at least the position of the focused subject of at least some second image data of the moving-picture data generated by the generating unit.SELECTED DRAWING: Figure 3

Description

  The present invention relates to an image processing device, an imaging device, and an image processing program.

  A technique is known that generates moving image data in which a focal position is adjusted following a moving subject from moving image data captured by a camera that can generate an image at an arbitrary focal position from the captured image (see Patent Document 1). . The prior art does not consider changing the focal position of the moving image data in which the generated focal position is matched.

JP 2013-145314 A

An image processing apparatus according to a first aspect of the present invention is output from an imaging unit having a plurality of microlenses on which light from a subject transmitted through an imaging optical system enters and a plurality of light receiving units provided for each of the microlenses. A setting unit that sets the position of the subject in focus in the optical axis direction of the imaging optical system and the depth of field with respect to the first image data, and the in-focus subject set by the setting unit. A generating unit that generates a plurality of second image data having a position and the depth of field from the plurality of first image data, and generates moving image data having the plurality of second image data; and the generating unit And a changing unit that changes at least one of the position of the in-focus subject and the depth of field of the second image data of at least a part of the moving image data generated in step (b).
An imaging device according to a second aspect of the present invention includes the image processing device according to the first aspect and an imaging unit.
An image processing program according to the third aspect of the present invention is output from an imaging unit having a plurality of microlenses on which light from a subject transmitted through an imaging optical system enters and a plurality of light receiving units provided for each of the microlenses. A setting step for setting the position of the subject in focus in the optical axis direction of the imaging optical system and the depth of field with respect to the first image data; and the subject in focus set in the setting step Generating a plurality of second image data having a position and the depth of field from the plurality of first image data, and generating moving image data having the plurality of second image data; and the generating step A change step for changing at least one of the position of the in-focus subject and the depth of field in the second image data of at least a part of the moving image data generated in step When causes the computer to execute. .

It is a block diagram of an imaging device. It is sectional drawing which showed typically the light beam from the light spot on the image surface of a synthetic | combination object, and an image pick-up element. It is a flowchart which shows operation | movement of an imaging device. It is a figure which shows the example of a display of a display part. It is a flowchart which shows operation | movement of 2nd Embodiment of an imaging device. It is a flowchart which shows operation | movement of 3rd Embodiment of an imaging device. It is a flowchart which shows operation | movement of 4th Embodiment of an imaging device. (A) (b) It is a display example of the refocus range. It is a figure which shows the modification of the display in a display part.

(First embodiment)
Hereinafter, a first embodiment of the present invention will be described with reference to the drawings.
FIG. 1 is a configuration diagram of the imaging apparatus 10. The imaging device 10 will be described using an example of a digital camera, but is not limited to a digital camera, and may be, for example, a smartphone terminal, a game machine, a tablet terminal, a personal computer, a clock-type or glasses-type information terminal, or the like.

  As illustrated in FIG. 1, the imaging apparatus 10 includes an imaging unit 11, an image processing unit 15, an imaging control unit 16, a control unit (setting unit, generation unit, change unit, display control unit) 17, recording interface 18, display unit 19. The operation unit 20 is provided. Furthermore, the imaging unit 11 includes an imaging optical system 12, a microlens 13, and an imaging element 14. In addition, a portable memory 21 as a storage medium is mounted via the recording interface 18.

  The imaging optical system 12 is a lens that forms an image of a light beam from the object field, and includes, for example, an objective lens, a focus lens, a diaphragm, and the like. The microlenses 13 are two-dimensionally arranged on a surface in the vicinity of the imaging surface of the imaging optical system 12.

  The image sensor 14 is formed by two-dimensionally arranging a plurality of light receiving elements (pixels) on the rear side of each microlens 13. A plurality of light receiving elements correspond to one microlens. The image sensor 14 is, for example, a CMOS (Complementary Metal Oxide Semiconductor) image sensor that converts an optical signal generated by the formed subject optical image into an electrical signal and outputs the electrical signal. The imaging device 14 has a plurality of photoelectric conversion regions, each of which is arranged two-dimensionally, and acquires a plurality of parallax images with different viewpoints from the output of the photoelectric conversion region group existing at the same position in each pixel. can do.

  The image processing unit 15 captures a series of parallax images (captured images) generated by the image sensor 14 and stores the images in the portable memory 21. In the case of moving image shooting, one frame of the moving image corresponds to a series of parallax images, and a moving image including such a plurality of frames is captured and stored in the portable memory 21. In other words, the original image data of the moving image obtained by imaging is stored in the portable memory 21.

  The imaging control unit 16 includes a circuit that controls driving of the imaging unit 11. The imaging control unit 16 drives the imaging element 14 in response to an instruction from the control unit 17 and acquires a series of parallax images as described later.

  The recording interface 18 is an interface between the portable memory 21 attached to the imaging device 10 and the control unit 17. The recording interface 18 writes a captured moving image or a refocused moving image obtained by refocusing the captured moving image to the portable memory 21 according to an instruction from the control unit 17, or reads these moving images from the portable memory 21. . The portable memory 21 is a storage medium that stores captured moving images and refocused moving images.

  The display unit 19 includes a monitor such as a liquid crystal color monitor and a touch sensor that detects a user operation on the monitor. Such a display unit 19 displays a moving image or the like according to an instruction from the control unit 17, detects the content of a user operation on the moving image or the like, and transmits the detected content to the control unit 17.

  The operation unit 20 includes various input switches such as a power button, a moving image shooting button, and a playback button (not shown). The user inputs various instructions such as a moving image shooting instruction to the control unit 17 via the operation unit 20.

  The control unit 17 is a computer that includes a ROM, a RAM, a CPU, and the like (not shown). The CPU controls each unit of the imaging apparatus 10 by executing a program stored in the ROM, and performs shooting and reproduction of moving images. Note that the RAM is a memory that temporarily stores data necessary during the operation of the CPU.

  The image composition function that the imaging apparatus 10 has will be described. The image composition function in the present embodiment is a function for creating an image of an arbitrary image plane (so-called refocus image) from original image data. An example of a known refocus image creation method will be described with reference to FIG.

  FIG. 2 is a cross-sectional view schematically showing the light flux from the light spot on the image plane to be synthesized and the image sensor 14. In FIG. 2, consider a light spot P provided on the image plane S to be synthesized. The spread angle θ of light from the light spot P toward the image sensor 14 is defined by the size of the pupil of the image pickup optical system 12 (that is, the aperture value of the image pickup optical system 12). The aperture value of the microlens 13 is configured to be the same as or smaller than the aperture value of the imaging optical system 12. Therefore, the light beam emitted from the light spot P and incident on a certain microlens 13 does not spread outside the region covered with the microlens 13.

  Here, as shown in FIG. 2, if the light beam from the light spot P is incident on the five microlenses 13 (1) to 13 (5), the microlenses 13 (1) to 13 (5) are incident on the microlenses 13 (1) to 13 (5). By limiting the amount of incident light (photoelectric conversion output of the light receiving elements 31 (1) to 31 (5)) on the light receiving surface of the incident light beams 30 (1) to 30 (5), it is limited to the pupil from the light spot P. The total incident amount is obtained. That is, the amount of light at the light spot P (the pixel to be synthesized) on the image plane S to be synthesized is obtained.

  The control unit 17 sets a plurality of light spots P on the designated image plane S, and specifies, for each light spot P, the microlens 13 on which the light beam from the light spot P is incident. The control unit 17 specifies which light receiving element 31 the light beam from the light spot P enters for each specified microlens 13. The control unit 17 calculates the pixel value of the light spot P by integrating the photoelectric conversion outputs of the identified light receiving elements 31. A refocused image can be obtained by combining images using pixel values of a plurality of light spots P.

  The control unit 17 can synthesize an image by designating an arbitrary image plane, but can also synthesize an image by setting an arbitrary depth of field. Referring to FIG. 2, the spread of the light beam from the light spot P is defined by the stop F0 of the preceding imaging optical system 12. When the diaphragm of the imaging optical system 12 is set to F1 which is narrower than F0, the spread of the light beam is narrower than the spread of FIG. Therefore, for example, by performing image composition on the assumption that the light beams 30 (1) and 30 (5) in FIG. 2 do not reach the light receiving element 31, an image having a deeper depth of field than that in FIG. 2 can be obtained. Can do. Assuming that the depth of field of image synthesis using all of the light flux extending from the light spot P is the shallowest, the image obtained by image processing without using some outer light flux has a shallow depth. Therefore, it is possible to realize a depth of field according to the number of omitted outer luminous fluxes.

Thus, the control unit 17 can generate a refocus image visualized by image processing after shooting. In addition, the control unit 17 according to the first embodiment performs refocus image processing in which the image plane position (focus position) and the depth of field are automatically set according to the distance to the main subject. When the captured image is a moving image, the control unit 17 performs the following image processing to generate a refocus image. A moving image is composed of a plurality of frames, and each frame is a parallax image. A plurality of parallax images constituting a moving image are weighted and synthesized with their positions shifted from each other, thereby creating one frame of a refocus image in which refocus conditions (subject distance, aperture value, image plane position) are set. Create a refocused image for each frame of the video to create a refocused video.
A parallax image is an image obtained by synthesizing pixel values obtained when a light beam that has passed through a plurality of light spots P passes through a plurality of microlenses 13 and enters a plurality of light receiving elements 31 with reference to FIG. is there.

  Hereinafter, the operation of the imaging apparatus 10 in the first embodiment will be described with reference to the flowchart shown in FIG. Each process shown in this flowchart is performed by the control unit 17 executing a program stored in a ROM (not shown).

  In step S11 of FIG. 3, moving image shooting is started by a user's operation of pressing a moving image shooting button. Each frame of the moving image data obtained by imaging is a frame image before the refocus image processing, which is obtained by the image processing unit 15 capturing a series of parallax images (captured images) generated by the image sensor 14. The moving image data composed of the frame image before the refocus image processing is recorded in the portable memory 21 as RAW data.

  In step S12, the moving image data recorded in the portable memory 21 is read, and a main subject is detected for each frame image included in the moving image data. Then, according to the detection result, the depth of field and the image plane position are set for the frame image. For example, if the closest subject in the field is the main subject, the image plane position and the depth of field are set based on the distance. The image plane position is set in proportion to the distance to the main subject. The farther the distance is, the farther the image plane position is from the image sensor. Also, for example, when the distance to the main subject is short, that is, when the image plane position is close to the image sensor, the depth of field is deep, and when the distance is far, that is, the image plane position is far from the image sensor. In case of, set shallow depth of field. Such refocus image processing conditions are applied to each frame of moving image data to perform refocus image processing.

  By applying refocus image processing conditions (image plane position and depth of field) that follow the subject distance for each frame, refocused images are generated with appropriate image plane position and depth of field for each of multiple frames. can do. The moving image data that has undergone the refocus image processing is recorded in the portable memory 21 as refocused moving image data.

  In step S 13, the refocus moving image data is read from the portable memory 21 and displayed on the display unit 19. The user looks at the displayed refocus video and determines whether the image plane position and depth of field of the main subject are appropriate. The display unit 19 displays adjustment buttons that are operated to start adjustment of the image plane position and the depth of field. When the user wants to adjust the image plane position or the depth of field, the adjustment button is touch-operated.

  In step S14, the touch sensor detects whether the adjustment button has been operated. If the adjustment button has been operated, the process proceeds to step S15. In step S15, each frame of the refocus moving image data is analyzed, and the image plane position and the depth of field are displayed in a graph on the display unit 19 with the shooting time as the horizontal axis according to the analysis result. FIG. 4 shows a display example of the graph.

  Analysis of each frame can be performed as follows. When the refocused moving image data is N frames of moving image data, the first frame shooting time, the second frame shooting time,..., The N-1 frame shooting time, the N frame shooting are performed by moving image data analysis. Time is detected. Further, for each frame, the image plane position and the depth of field set based on the distance to the main subject are detected. The graph of FIG. 4 is created using the image plane position and the depth of field with respect to the shooting time of each frame.

  As shown in FIG. 4, in the graph display area 19a, the horizontal axis represents the photographing time, and the vertical axis represents the image plane position and the depth of field. The graph represents changes in the image plane position and depth of field of each frame over time of the refocus moving image data. The solid line in the figure indicates the image plane position, and in the example of FIG. 4, the image plane position continuously changes from the point A to the point B from the photographing time ta to the time tb. Here, the frames at points A and B are referred to as key frames. A key frame is a frame in which the amount of change in image plane position between frames is equal to or greater than a predetermined threshold, for example. Since the display unit 19 includes a touch sensor, the user touches the position of the point A or B and slides up and down in the drawing to adjust the image plane position of the point A or B. . By this user adjustment, the image plane position and the depth of field for a plurality of frames between the key frame whose image plane position is point A and the key frame whose point is B are adjusted in conjunction with each other. It is not always necessary to adjust all the frames between the point A key frame and the point B key frame. For example, when the point A key frame is adjusted, Only a plurality of frames before and after may be adjusted in conjunction. In addition, the key frame is a frame in which the amount of change in the image plane position between frames is equal to or greater than a predetermined threshold. However, the key frame may be set at a predetermined time interval of shooting time.

Further, adjustment windows 19b and 19c are displayed corresponding to the key frames. In the adjustment windows 19b and 19c, a key frame image and a slide bar for adjusting the depth of field are displayed, respectively. By touching the slide bar to the left and right, it is possible to adjust the depth of field deeper or to make it shallower with reference to the key frame. The depth of field is displayed as a dotted line in the graph display area 19a, and indicates the depth / depth of the depth of field depending on whether the distance from the solid image plane position is large or small. In the example of FIG. 4, a frame with a close image plane position has a deep depth of field, and a frame with a far image plane position has a shallow depth of field. In this example, the case where there are two key frames is shown, but an adjustment window is displayed according to the number of key frames.
The number of adjustment windows may be smaller than the number of key frames. For example, with one adjustment window, one key frame to be adjusted by the user among a plurality of key frames may be selected, and only the adjustment window corresponding to the selected key frame may be displayed.
The graph of FIG. 4 is shown as a straight line with one axis being the time axis and the other axis being the image plane position, but the image plane position may be displayed by a curve. The graph of FIG. 4 displays the range representing the depth of field around the straight line representing the image plane position, but displays the range representing the depth of field around the curve representing the image plane position. May be.

  When the adjustment of the image plane position and the depth of field by the user is completed, the enter button 19d is touched. In step S15, in response to the touch operation of the enter button 19d, the control unit 17 performs refocus image processing based on the image plane position adjusted by the user and the depth of field for each frame of the moving image data. .

  In step S16, the adjusted refocus video is displayed on the display unit 19. In step S <b> 17, the adjusted refocus moving image is recorded in the portable memory 21. In this case, the refocus moving image before adjustment may be recorded in the portable memory 21. If the adjustment button is not operated in step 14, the refocus moving image before adjustment is recorded in the portable memory 21. Further, the refocus moving image may be recorded in the portable memory 21 as RAW data. Further, the photographing time, the image plane position, and the depth of field of the adjusted key frame may be recorded in the portable memory 21. If the refocus video before adjustment and the shooting time, image plane position, and depth of field of the adjusted key frame are recorded, the adjusted refocus video can be generated from them.

  According to the first embodiment, when a refocused image is created by refocusing image processing for a moving image obtained by imaging, the image plane position following the distance of the main subject and the depth of field are automatically set for each frame. Can be set. Therefore, it is possible to easily generate an appropriate refocus movie based on the main subject. Further, the image plane position and the depth of field of the moving image that has been subjected to the refocus image processing can be appropriately adjusted according to the user's designation. Therefore, when editing a moving image composed of a plurality of frames, it is not necessary to set the image plane position and the depth of field for each frame, and the editing operation is easy.

(Second Embodiment)
A second embodiment will be described with reference to the drawings. The configuration of the imaging device 10 is the same as that of the first embodiment shown in FIG. Furthermore, since the flowchart executed by the control unit 17 is the same as that of the first embodiment shown in FIG. 3, the description thereof is omitted, but the processing of step S <b> 12 in FIG. 3 is different in the second embodiment. . Hereinafter, a flowchart of a part replacing step S12 in FIG. 3 will be described with reference to FIG. Each process shown in this flowchart is performed by the control unit 17 executing a program stored in a ROM (not shown).

  Here, it is assumed that the original image for generating the refocus moving image is N frames of moving image data. The image plane position of each frame is set based on the distance to the main subject, with the farthest subject in the image of the frame as the main subject. The depth of field is also set according to the distance to the main subject. Since the main subject is a distant view, the depth of field is set deep, and a refocused image with a focus on the whole is obtained. For each frame, when a moving object such as a person or a car enters the main object in the distance, the moving object is recognized as the main object, and the depth of field is set based on the distance of the moving object. That is, an image in which the depth of field is set shallow and the moving object is in focus is obtained.

This will be described with reference to the flowchart of FIG.
In step S1201, for each of the N frames of the moving image, the distant subject is recognized as the main subject and the image plane position is set.
Next, in step S1202, the depth of field according to the image plane position of the distant view is set. Specifically, when the image plane position is far away, the depth of field is set deep so that the entire image is in focus.

  In step S1203, it is determined whether all the frames of the moving image have been completed. If not completed, the determination in step S1204 is performed for the next frame. That is, in step S1204, it is determined whether a moving body has entered the frame as a subject. If no moving object has entered, the process returns to step S1201, and the processing from step S1201 to step S1203 is repeated to perform refocus image processing according to the distant view.

If it is determined in step S1204 that a moving object has entered the frame, the process proceeds to step S1205. In step S1205, the image plane position is set on the moving body.
Next, in step S1206, the depth of field is set according to the image plane position of the moving object. Specifically, the depth of field is set shallow so that a moving object such as a person or a vehicle is in focus.

  In step S1207, it is determined whether all the frames of the moving image have been completed. If not completed, it is determined in step S1208 whether the moving object has left the frame. If the moving body has not left the frame, the process returns to step S1205, and the processing from step S1205 to step S1207 is repeated to perform refocus image processing in accordance with the moving body.

If it is determined in step S1208 that the moving object has left the frame, the process returns to step S1201, and the processing from step S1201 to step S1203 is repeated to perform refocus image processing in accordance with the distant view.
In step S1203 or step S1207, when all the frames of the moving image are completed, the refocus image processing is ended.
The adjustment of the refocus moving image by the user is the same as that in the first embodiment described in step 15 in FIG.

  According to the second embodiment, when a moving object enters a distant view of a moving image obtained by imaging, the image plane position and the depth of field can be set in the moving object by refocus image processing. That is, the control unit 17 (steps S1201 to S1208) according to the second embodiment sets a deep first depth of field in a frame in which a moving body that is a main subject is not detected, and the moving body is detected. A second depth of field shallower than the first depth of field is set for each frame. Therefore, it is possible to generate an appropriate moving image corresponding to the movement of the moving body. Further, the image plane position and the depth of field of the moving image that has been subjected to the refocus image processing can be appropriately adjusted according to the user's designation.

(Third embodiment)
A third embodiment will be described with reference to the drawings. The configuration of the imaging device 10 is the same as that of the first embodiment shown in FIG. Furthermore, since the flowchart executed by the control unit 17 is the same as that of the first embodiment shown in FIG. 3, the description thereof is omitted, but the processing of step S12 of FIG. 3 is different in the third embodiment. . Hereinafter, a flowchart of a part replacing step S12 in FIG. 3 will be described with reference to FIG. Each process shown in this flowchart is performed by the control unit 17 executing a program stored in a ROM (not shown).

  In the third embodiment, when a person is recognized as a main subject by face recognition and a plurality of main subjects are detected, the depth of field is set so that all the main subjects are in focus and Generate.

  In step S <b> 1211 of FIG. 6, the image plane position and the depth of field are set so as to cover all the detected main subjects, that is, all the persons in focus for the moving image frame. Specifically, the image plane position is set near the middle of the subject distance of all persons, and the depth of field is set so that all persons are in focus. For example, the depth of field can be set based on the difference between the distance to the farthest person and the distance to the nearest person.

  It should be noted that a predetermined fixed image plane position and depth of field may be set for shooting moving image data of a stage such as a play. Further, the control unit 17 acquires the focal length of the imaging optical system 12 of the imaging unit 11, and when the first focal length is acquired, the first depth of field is set and is shorter than the first focal length. When the second focal length is acquired, a second depth of field that is shallower than the first depth of field may be set. Specifically, lens information is acquired from the imaging unit 11, and when a wide-angle lens with a short focal length is attached, the depth of field is set shallow.

In step S1212, it is determined whether the image plane position and the depth of field can be set so as to cover all persons. Specifically, compare the subject distance difference between the farthest person and the nearest person and the deepest depth of field that can be set, and focus on both the farthest person and the nearest person. Determine whether you can. If it is determined that the above setting is possible, it is determined in the next step S1213 whether all the frames of the moving image have been completed. If not completed, the process returns to step S1211, and for the next frame, the processing from step S1211 to step S1212 is repeated to perform refocus image processing so as to cover all persons.
If it is determined in step S1212 that the image plane position and the depth of field that cover all persons cannot be set, the process proceeds to step S1214.

  In step S1214, the image plane position is set with priority on the closest person. Specifically, the image plane position is set near the middle, which is 80% for the nearest person and 20% for the farthest person. In step S1215, the depth of field is set according to the set image plane position. For example, the deepest depth of field is set for the closest person, and the shallowest depth of field is set for the farthest person. The depth of field of the person at the intermediate position is set to a depth corresponding to the distance to the person.

In steps S1214 and S1215, the control unit 17 registers information about a person, for example, compares the person in the frame with the person registered in advance, and if a matching person is detected, the controller 17 The depth of field may be set so that it is in focus.
That is, the control unit 17 (steps S1214 to S1215) is configured to further include a registration unit that registers information on the main subject, and the main information whose information is registered in the registration unit among the detected plurality of main subjects. The depth of field and the image plane position may be set so that the subject is included in the depth of field. Thereby, a moving image in which the registered subject is in focus can be generated.
The main subject to be registered is not limited to a person's face, but may be a specific vehicle, structure, or the like.

  In step S1216, it is determined whether all the frames of the moving image have been completed. If not completed, the process of step S1217 is performed.

  In step S1217, it is determined whether the number and position of main subjects have changed in the next frame. If it is determined that the number or position of the main subject has not changed, the process returns to step S1214, the processing from step S1214 to step S1215 is repeated, and the refocus image giving priority to the previous main subject for the next frame. Process.

  If it is determined in step S1217 that the number or position of main subjects has changed, the process returns to step S1211, and the processing from step S1211 to step S1212 is repeated to perform refocus image processing so as to cover all the main subjects. Do.

In step S1213 or step S1216, when all the frames of the moving image are completed, the refocus image processing is ended.
The adjustment of the refocus moving image by the user is the same as that in the first embodiment shown in step 15 of FIG.

  According to the third embodiment, it is possible to perform refocus image processing in which the image plane position and the depth of field are set so as to cover all main subjects with respect to a moving image obtained by imaging. That is, when a plurality of main subjects are detected, the control unit 17 (step S1211) sets the depth of field and the image plane position so that the plurality of main subjects are included in the depth of field. The control unit 17 (steps S1214 to S1215) detects the first main subject located at the first distance and the second main subject located at the second distance farther than the first distance. The depth of field and the image plane position are set so that the main subject is included within the depth of field. Accordingly, it is possible to generate an appropriate moving image in which a plurality of main subjects in the object field are in focus. In addition, when it is impossible to cover all the main subjects, refocus image processing is performed by setting the image plane position and the depth of field with priority on the front main subject. It is possible to generate an appropriate moving image in which a plurality of main subjects are in focus. Furthermore, the image plane position and the depth of field of the moving image that has undergone the refocus image processing can be appropriately adjusted according to the user's designation.

(Fourth embodiment)
A fourth embodiment will be described with reference to the drawings. The configuration of the imaging device 10 is the same as that of the first embodiment shown in FIG. Furthermore, since the flowchart executed by the control unit 17 is the same as that of the first embodiment shown in FIG. 3, the description thereof is omitted, but the process of step S <b> 12 in FIG. 3 is different in the fourth embodiment. . Hereinafter, a flowchart of a part replacing step S12 in FIG. 3 will be described with reference to FIG. Each process shown in this flowchart is performed by the control unit 17 executing a program stored in a ROM (not shown).

In the fourth embodiment, a refocus image process is performed on a moving image obtained by imaging according to an imaging scene.
In step S1221 of FIG. 7, an imaging scene at the time of moving image shooting is determined. Of the imaging scenes, portrait, pet, cooking, child, close-up, party, and candlelight are set as the first imaging scene, and scenery and night view are set as the second imaging scene. In step S1221, it is determined whether the frame is the first imaging scene or the second imaging scene.

  When it is determined that the scene is the first imaging scene, the process proceeds to step S1222, and the image plane position corresponding to the subject, that is, the imaging plane corresponding to the imaging scene is set. In step S1223, the depth of field is set shallow.

  If it is determined in step S1221 that the scene is the second imaging scene, the process proceeds to step S1224, and an image plane position corresponding to the subject is set. In step S1225, the depth of field is set deep.

After the process of step S1223 or S1225, the process proceeds to step S1226, and it is determined whether all the frames of the moving image have been completed. If not completed, the process returns to step S1221, and refocus image processing is performed for the next frame.
If all the frames of the moving image have been completed in step S1226, the refocus image processing is ended.

The image processing apparatus according to the fourth embodiment has a data table in which shooting scenes are associated with depth of field. Then, at the time of image editing, a shooting scene set by the user is recognized for each frame, and an optimum depth of field is obtained from the data table based on the recognized shooting scene, and refocus image processing is performed.
The adjustment of the refocus moving image by the user is the same as that in the first embodiment shown in step 15 of FIG.

  According to the fourth embodiment, refocus image processing in which the image plane position and the depth of field are set according to the imaging scene can be performed on the moving image obtained by imaging. That is, the control unit 17 (steps S1221 to S1226) further includes a scene recognition unit that recognizes an imaging scene and classifies the scene into one of a first imaging scene such as a portrait and a second imaging scene such as a landscape. For example, the depth of field is set shallow in the first imaging scene, and the depth of field is set deep in the second imaging scene. Therefore, it is possible to easily generate a refocus moving image according to the imaging scene. Further, the image plane position and the depth of field of the moving image that has been subjected to the refocus image processing can be appropriately adjusted according to the user's designation.

  Note that the same refocus image processing as in the fourth embodiment may be performed simultaneously with moving image shooting. In other words, the image processing apparatus according to this modification obtains the optimum depth of field from the data table based on the shooting scene set by the user during moving image shooting. Then, the depth of field and the image plane position are stored in association with the frame. The image plane position and the depth of field are read for each frame in the edit reproduction process, and the refocus image process is performed. When the refocus image processing can be performed in a short time, the refocus image processing may be performed for each frame in real time during moving image shooting.

(Example of refocus range display)
In general, when a plurality of subjects exist at a distance from each other, there is a limit to obtain a focused image by performing refocusing image processing from moving image shooting data. In the following, an example in which the possible range of refocus image processing is displayed in a through image during moving image shooting will be described.

  FIG. 8A is a display example of the refocus range, and FIG. 8B is an explanatory diagram thereof. As shown in FIG. 8A, subjects O, P, and Q are displayed on the display unit 19 as through images at the time of moving image shooting. The subjects O and P are distinguished from the subject Q and displayed in color. This indicates that the subjects O and P are within the possible range of refocus image processing. On the right end of the display unit 19, an operator 19e for adjusting the possible range of refocus image processing is displayed. The possible range of the refocus image processing is adjusted by touching the operator 19e and sliding it up and down. Note that the possible range of refocus image processing may be adjusted by rotating the ring of the lens barrel instead of the slide operation of the operation element 19e.

  FIG. 8B is a diagram illustrating a positional relationship between the imaging optical system 12 and the subjects O, P, and Q existing in front of the imaging optical system 12. In the figure, r indicates the possible range of refocus image processing. By sliding the operation element 19e up and down, the refocus image processing possible range r is moved back and forth. For example, the operator 19e can be adjusted by sliding up and down so that the subjects P and Q are within the refocus image processing possible range r. In this state, the subjects P and Q are distinguished from the subject O and displayed in color on the display unit 19. Thereby, the user can know the possible range of the refocus image processing at the time of shooting. Whether or not refocus image processing is possible is determined by the control unit 17 by performing refocus image processing on the through image in real time to determine whether or not the subjects O, P, and Q can be focused. The displayed subject is identified and displayed.

According to the embodiment described above, the following operational effects can be obtained.
(1) The imaging apparatus 10 includes an image processing unit 15 to which first moving image data captured by an imaging element 14 having a plurality of microlenses 13 and a larger number of light receiving elements than the plurality of microlenses 13 is input; For a plurality of frames included in the first moving image data, a control unit 17 (step S12) for setting a depth of field and an image plane position within a predetermined range in the optical axis direction, and each of the plurality of frames. And a control unit 17 (step S13) that outputs second moving image data converted into an image at an image plane position having a depth of field set by the control unit (setting unit) 17. Therefore, a refocus moving image in which an appropriate image plane position and depth of field are set in each of a plurality of frames of the moving image is generated. There is no need to set the image plane position and depth of field frame by frame.

(2) The control unit 17 (step S12) detects the main subject from the frame image and sets the depth of field and the image plane position so that the main subject is included in the depth of field. It is possible to generate an appropriate video based on the video.

(3) The control unit 17 (steps S1201 to S1208) sets a deep first depth of field in a frame in which the main subject is not detected, and the first subject in a frame in which the main subject is detected. Since the second depth of field shallower than the depth of field is set, for example, an appropriate moving image corresponding to the movement of the moving body that is the main subject can be generated.

(4) When a plurality of main subjects are detected, the control unit 17 (step S1211) sets the depth of field and the image plane position so that the plurality of main subjects are included in the depth of field. Therefore, it is possible to generate an appropriate moving image that covers all main subjects.

(5) When the control unit 17 (steps S1214 to S1215) detects the first main subject located at the first distance and the second main subject located at the second distance farther than the first distance, Since the depth of field and the image plane position are set so that one main subject is included in the depth of field, an appropriate moving image can be generated with priority given to the main subject in front.

(6) The control unit 17 (steps S1214 to S1215) further includes a registration unit for registering information on the main subject, and among the detected main subjects, the control unit 17 covers the main subject whose information is registered in the registration unit. Since the depth of field and the image plane position are set so as to be included in the depth of field, an appropriate moving image can be generated with priority on the registered subject.

(7) The control unit 17 (steps S1221 to S1226) recognizes an imaging scene for a frame, and classifies the scene recognition unit into either a first imaging scene such as a portrait or a second imaging scene such as a landscape. In addition. For example, the depth of field is set shallow in the first imaging scene, and the depth of field is set deep in the second imaging scene. Therefore, it is possible to generate a refocus moving image corresponding to the imaging scene.

  The following modifications are also within the scope of the present invention, and modifications can be combined with the above-described embodiment.

(Modification of display example)
In step S15 of FIG. 3, as already described, each frame of the refocus moving image data is analyzed, and according to the analysis result, the image plane position and the depth of field are displayed on the display unit 19 with the shooting time as the horizontal axis. Display a graph. FIG. 4 shows a display example in this case, and FIG. 9 shows a modification of this display example. Hereinafter, a description will be given with reference to FIG.

  As shown in FIG. 9, in the graph display area 19a, a three-dimensional graph with the shooting time, the image plane position, and the depth of field as axes is displayed. The shooting time (x-axis) represents the shooting time of each frame of the refocus moving image data. The image plane position (y-axis) and the depth of field (z-axis) are graphically displayed changes in the shooting time of each frame. Point A and point B in the figure indicate that the image plane position has changed discontinuously. Frames at points A and B are called key frames.

  Further, adjustment windows 19b and 19c are displayed corresponding to each key frame. In the adjustment windows 19b and 19c, an image of a key frame, a slide bar for adjusting the image plane position, and a slide bar for adjusting the depth of field are displayed, respectively. By touching the slide bar for image plane position adjustment to the left and right, the image plane position can be adjusted far or near with reference to the key frame. Further, the depth of field can be adjusted deeper or shallower with reference to the key frame by touching the slide bar for adjusting the depth of field left and right. In this example, the number of key frames is two, but an adjustment window may be displayed according to the number of key frames.

  When the adjustment of the image plane position and the depth of field by the user is completed, the enter button 19d is touched. In response to the touch operation of the enter button 19d, the control unit 17 performs refocus image processing based on the image plane position and the depth of field adjusted by the user for each frame of the moving image data (original image data). Do.

(Other variations)
In the image processing apparatus and the imaging apparatus according to the above-described embodiment, it has been described that the control unit 17 automatically performs the image plane position and the depth of field of each frame. However, the image plane position and the depth of field may be manually set. For example, refocus image processing is performed on all frames of an original image of a moving image according to the image plane position and depth of field set by default to generate and store a refocus image. Displays a refocused image of a specific image plane position and a specific depth of field saved, and manually sets the optimal image plane position and depth of field for each frame on the editing screen as shown in FIG. May be.

The present invention is not limited to the above-described embodiment, and other forms conceivable within the scope of the technical idea of the present invention are also included in the scope of the present invention as long as the characteristics of the present invention are not impaired. . Moreover, it is good also as a structure which combined the above-mentioned embodiment and a modification.
Therefore, the present invention includes an input step of inputting first moving image data obtained by imaging with the imaging element 14 having a plurality of microlenses 13 and a larger number of light receiving elements 31 than the plurality of microlenses 13; A setting step for setting a depth of field and an image plane position within a predetermined range in the optical axis direction for each of a plurality of frames included in the moving image data, and a setting step for setting each of the plurality of frames. And an image processing program for causing the computer to execute an output step of outputting the second moving image data converted into an image at an image plane position having a depth of field.

DESCRIPTION OF SYMBOLS 10 Imaging device 11 Imaging part 12 Imaging lens 13 Micro lens 14 Imaging element 15 Image processing part 16 Imaging control part 17 Control part 18 Recording interface 19 Display part 20 Operation part 21 Portable memory

Claims (21)

With respect to the first image data output from an imaging unit having a plurality of microlenses on which light from a subject that has passed through the imaging optical system enters and a plurality of light receiving units provided for each microlens, the imaging optical system A setting unit for setting the position of the subject in focus in the optical axis direction and the depth of field;
Generating a plurality of second image data having a position of the in-focus subject set by the setting unit and the depth of field from the plurality of first image data, and the plurality of second image data A generating unit for generating moving image data having,
An image processing apparatus comprising: a changing unit that changes at least one of the position of the focused subject and the depth of field of the second image data of at least a part of the moving image data generated by the generating unit . The image processing apparatus according to claim 1.
A detection unit for detecting a main subject from the first image data;
The image processing apparatus, wherein the setting unit sets the depth of field and the position of the subject in focus so that the main subject detected by the detection unit is included in the depth of field. The image processing apparatus according to claim 2,
The setting unit sets a first depth of field when the main subject is in a first position in the optical axis direction of the imaging optical system, and the main subject is more in the imaging optical than the first position. An image processing apparatus that sets a second depth of field deeper than the first depth of field when the second position is close to the system. The image processing apparatus according to claim 3.
The setting unit sets a first depth of field in the first image data in which the main subject is not detected by the detection unit, and the first image in which the main subject is detected by the detection unit. An image processing apparatus that sets a second depth of field shallower than the first depth of field in the data. The image processing apparatus according to claim 2,
When the detection unit detects a plurality of the main subjects, the setting unit subjects the depth of field to the focus so that the plurality of main subjects are included in the depth of field. Image processing apparatus for setting the position of The image processing apparatus according to claim 2,
The setting unit includes a first main subject at a first position in an optical axis direction of the imaging optical system and a second main subject at a second position farther from the imaging optical system than the first position. An image processing apparatus that sets a position of a subject that is in focus with the depth of field so that the first main subject is included in the depth of field when detected by a detection unit. The image processing apparatus according to claim 6.
The setting unit is an image processing apparatus that sets the position of the subject that is in focus at a position that is a predetermined distance away from the first main subject in the direction of the imaging unit. The image processing apparatus according to claim 2,
A registration unit for registering information on the main subject;
The setting unit includes the depth of field so as to include the main subject whose information is registered in the registration unit among the plurality of main subjects detected by the detection unit. And an image processing apparatus for setting the position of the subject in focus. In the image processing device according to any one of claims 2 to 8,
A focal length acquisition unit for acquiring a focal length of the imaging optical system;
The setting unit sets a first depth of field when the first focal length is acquired by the focal length acquisition unit, and a second focal length shorter than the first focal length by the focal length acquisition unit. Is acquired, the second depth of field shallower than the first depth of field is set. In the image processing device according to any one of claims 1 to 8,
A scene recognition unit for recognizing an imaging scene and classifying the first image data into one of a first imaging scene and a second imaging scene;
The setting unit sets the depth of field to be set and the subject that is in focus with the first image data classified into the first imaging scene and the first image data classified into the second imaging scene. An image processing device that changes the position of the. In the image processing device according to any one of claims 1 to 10,
An image processing apparatus comprising: a display control unit configured to display, on a display unit, the position of the subject in focus set to each of the plurality of first image data and the depth of field. The image processing apparatus according to claim 11.
The display control unit displays the position of the subject in focus on the display unit by a straight line or a curve with one axis as a time axis and the other axis as the position of the subject in focus. The image processing apparatus according to claim 12.
The display control unit is an image processing device that displays a range representing the depth of field on the display unit around a straight line or a curve representing the position of the subject in focus. The image processing apparatus according to claim 11.
The display control unit is an image processing apparatus that performs three-dimensional display on the display unit with a first axis as a time axis, a second axis as the position of the subject in focus, and a third axis as the depth of field. The image processing apparatus according to any one of claims 11 to 14,
The setting unit resets the set depth of field and the focused subject position to at least one of the focused subject position and the depth of field changed by the changing unit,
The display control unit is an image processing device that changes the display of the display unit according to the reset depth of field and the position of the subject in focus. The image processing apparatus according to claim 15, wherein
The setting unit sets a predetermined first image data of the plurality of first image data as a key frame,
The changing unit is configured to change at least one of a position of the subject in focus of the key frame set by the setting unit and a position of the subject in focus different from the depth of field and the depth of field. ,
The setting unit sets the position of the in-focus subject and the depth of field in the first image data between two key frames, and the position of the changed in-focus subject for one key frame and the subject. An image processing apparatus that changes stepwise between at least one of a depth of field and at least one of a position of the subject that is changed in focus with respect to the other key frame and a depth of field. The image processing apparatus according to claim 16.
The setting unit is an image processing apparatus that sets, as a key frame, first image data in which the depth of field and the position of a subject in focus are changed. In the image processing device according to any one of claims 2 to 9,
The setting unit is an image processing device that sets, as a key frame, first image data in which a detection result of the main subject by the detection unit has changed. The image processing apparatus according to any one of claims 1 to 18,
The imaging unit;
An imaging apparatus comprising: The imaging device according to claim 19,
An imaging apparatus comprising an informing unit for informing a range of a position of a subject in focus. With respect to the first image data output from an imaging unit having a plurality of microlenses on which light from a subject that has passed through the imaging optical system enters and a plurality of light receiving units provided for each microlens, the imaging optical system A setting step for setting the position of the subject in focus in the optical axis direction and the depth of field;
Generating a plurality of second image data having a position of the in-focus subject set in the setting step and the depth of field from the plurality of first image data; A generation step for generating video data having;
A changing step for changing at least one of the position of the focused subject and the depth of field of the second image data of at least a part of the moving image data generated in the generating step;
An image processing program for causing a computer to execute.

Images