JP6221452B2 - Image processing apparatus, image display apparatus, and imaging apparatus - Google Patents

Description

  The present invention relates to an image processing apparatus, an image display apparatus, and an imaging apparatus that process a captured image that can be set after an imaging condition.

  In recent years, cameras that acquire captured images that can be set after imaging conditions have been attracting attention. As such a camera, a light field camera (Patent Document 1), a camera capable of taking a focus bracket (Patent Document 2), a camera array (Patent Document 3), and the like are known.

  According to the light field camera (Patent Document 1), it is possible to set the shooting distance (distance of the in-focus plane in the object field), the aperture value, and the viewpoint later, and the camera capable of focus bracket shooting (Patent Document 2). According to the above, it is possible to set the shooting distance later, and according to the camera array (Patent Document 3), the shooting distance and the viewpoint (or aperture value) can be set later.

  Patent Document 2 discloses a technique for displaying a plurality of photographed images with different photographing distances as a slide show. Patent Document 3 discloses a photographed image (refocused image) focused on a subject desired by the user. The technique etc. which synthesize | combine are disclosed.

US Patent Publication No. 7936392 JP 2010-166519 A JP 2011-22796 A

  However, there is still room for improvement in order for the user to freely set the shooting conditions.

  Therefore, an object of the present invention is to further improve user operability related to the post-setting of shooting conditions.

An example of the image processing apparatus of the present invention includes a reading unit that reads a captured image that can be set after a shooting condition, a display control unit that displays the captured image on a monitor, a user touch operation on the monitor, and the touch operation in response to a series of operations consisting of a continuous slide operation, and an adjustment means for adjusting at least two photographing condition of the displayed photographic image to the monitor, the touch operation, imaging of the captured image distance is an operation for adjusting the sliding operation, Ru operation der to adjust the perspective of the captured image.

Another example of the image processing apparatus of the present invention is a reading means for reading a photographed image that can be set after photographing conditions, a display control means for displaying the photographed image on a monitor, a user touch operation on the monitor, Adjusting means for adjusting at least two shooting conditions of a shot image displayed on the monitor in accordance with a series of operations including a slide operation that is continuous with the touch operation, and the touch operation includes the shot image. The slide operation is an operation for adjusting the aperture value of the photographed image, and the adjustment means is operated from the position where the touch operation is performed on the monitor. The aperture value of the captured image is set by performing the slide operation in a direction away from or in a direction approaching a position where the touch operation is performed on the monitor. To integer.

  An example of the image display device of the present invention includes any one of the image processing devices of the present invention and the monitor that displays the captured image.

  An example of the imaging apparatus of the present invention includes an imaging unit that acquires the captured image and an example of the image display apparatus of the present invention.

  According to the present invention, it is possible to further improve the user operability regarding the post-setting of the shooting conditions.

It is a figure explaining a light field camera. It is a structural example of an image file. It is a figure explaining 1st Embodiment. It is a figure explaining 2nd Embodiment. It is a figure explaining 3rd Embodiment. This is a variation of the operation for drawing a closed curve. It is a figure explaining 5th Embodiment. It is a figure explaining the modification of 5th Embodiment.

  Hereinafter, a light field camera will be described as a first embodiment of the present invention.

  FIG. 1A is a configuration example of the light field camera 10, and FIG. 1B is an example of the light field camera 1 viewed from the back.

  As shown in FIG. 1A, the light field camera 10 includes an imaging unit 11, an image processing unit 15, a camera control unit 16, a system control unit 17, a recording interface 18, a display unit 19, an operation member 20, and the like. Among these, the imaging unit 11 includes a photographic lens 12, a microlens array 13, an imaging device 14, and the like, and a card memory 21 that is a storage medium is attached to the recording interface 18.

  The taking lens 12 is an imaging lens that forms an image of a light beam from the object scene. The actual photographing conditions of the photographing lens 12 may be adjustable, but here it is assumed that they are unchanged for simplicity. The photographic lens 12 may be interchangeable with other photographic lenses having different specifications. However, for the sake of simplicity, the photographic lens 12 whose photographing conditions are unchanged is always attached to the light field camera 10. Assuming that

  The actual photographing conditions of the photographing lens 12 include an actual photographing distance of the photographing lens 12 (a distance to the surface conjugate with the microlens array 13 with respect to the photographing lens 12), an actual aperture value of the photographing lens 12, and At least one of the actual angle of view of the photographing lens 12 is included.

  Incidentally, when the actual shooting distance of the photographing lens 12 changes, the range of the photographing distance that can be set later (refocus range) changes, and when the actual aperture value of the photographing lens 12 changes, the aperture value that can be set later is changed. When the range changes and the actual angle of view of the photographic lens 12 changes, the range of the viewpoint that can be set later changes, but each range is unambiguous from the actual shooting distance, the actual aperture value, and the actual angle of view. It is decided.

  The microlens array 13 is formed by two-dimensionally arranging a plurality of microlenses on a surface in the vicinity of the imaging surface of the photographing lens 12.

  The imaging element 14 is formed by two-dimensionally arranging a plurality of light receiving elements (pixels) behind the individual microlenses belonging to the microlens array 13. Note that the pixel arrangement pattern of the pixel group (partial imaging element 14a) located on the rear side of each microlens is common among the individual microlenses.

  Here, when attention is paid to the partial image sensor 14a located on the rear side of one microlens, between the light beams individually incident on the pixel coordinates a, b, c, d, and e of the partial image sensor 14a, The passing areas in the exit pupil of the photographing lens 12 are different from each other. This applies similarly to all the partial image sensors 14a.

  Therefore, the pixel coordinates arranged in one direction on the partial image sensor 14a are sequentially set as a, b, c, d, and e, and an image is obtained one by one with pixels having the same pixel coordinates among the plurality of partial image sensors 14a. Once generated, the image Ia generated from the pixel coordinate a, the image Ib generated from the pixel coordinate b, the image Ic generated from the pixel coordinate c, the image Id generated from the pixel coordinate d, the pixel The relationship with the image Ie generated from the coordinate e is the same as the relationship between the five parallax images taken from different viewpoints of the object scene. Therefore, hereinafter, the images Ia, Ib, Ic, Id, and Ie are referred to as “parallax images”.

  Note that the actual pixel arrangement direction of the partial image sensor 14a extends in two directions instead of one direction, and the actual number of pixels of the partial image sensor 14a is sufficiently larger than “5”, but in the following description, For the sake of simplicity, it is assumed that the pixel consists of only five pixel coordinates a, b, c, d, and e arranged in one direction.

  The image processing unit 15 takes in a light field image (consisting of a series of parallax images Ia, Ib, Ic, Id, and Ie) generated by the image sensor 14 and temporarily stores the light field image in a buffer memory (not shown). To do. Further, the image processing unit 15 performs various types of image processing on the light field image on the buffer memory in accordance with an instruction from the system control unit 17.

  For example, the image processing unit 15 shifts the positions of the five parallax images Ia, Ib, Ic, Id, and Ie that make up the light field image, and then performs weighting synthesis to obtain shooting conditions (shooting distance, aperture value, One shot image with a (viewpoint) set is created. Hereinafter, a photographed image set after photographing conditions is referred to as a “refocus image”.

  Note that the shooting distance of the refocus image is determined by the position shift amount of the parallax images Ia, Ib, Ic, Id, and Ie. For example, when the amount of position shift of the parallax images Ia, Ib, Ic, Id, and Ie is “zero”, the shooting distance of the refocus image is the same distance as the plane conjugate with the microlens array 13 with respect to the shooting lens 12. When the position shift amount is a value other than “zero”, the shooting distance of the refocus image is the same distance as the non-conjugated surface with respect to the microlens array 13 with respect to the shooting lens 12.

  The aperture value of the refocus image is determined by the weight ratio of the parallax images Ia, Ib, Ic, Id, and Ie. For example, when the weight ratio of the parallax images Ia, Ib, Ic, Id, and Ie is “0: 0: 1: 0: 0”, the aperture value of the refocus image is maximized, and the parallax images Ia, Ib, Ic, When the weight ratio of Id and Ie is “1: 1: 1: 1: 1”, the aperture value of the refocus image is minimum.

  The viewpoint of the refocus image is also determined by the weight ratio of the parallax images Ia, Ib, Ic, Id, and Ie. For example, when the weight ratio of the parallax images Ia, Ib, Ic, Id, and Ie is “1: 0: 0: 0: 0”, the viewpoint of the refocus image is one end point, and the parallax images Ia, Ib, and Ic , Id, and Ie have a weight ratio of “0: 0: 0: 0: 1”, the viewpoint of the refocus image is the other end point.

  Note that any known light field camera configuration can be applied to the configuration of the imaging unit 11 described above, and the operation of any known light field camera can be applied to the operation of the image processing unit 15 described above. Applicable. Examples of known light field cameras include those disclosed in Japanese Patent Application Laid-Open No. 2011-217411, US Pat. No. 7,936,392, and Japanese Patent Application Laid-Open No. 2011-259218.

  The camera control unit 16 includes a mechanism that drives the imaging unit 11 and a circuit that controls the mechanism. The camera control unit 16 drives the image sensor 14 in accordance with an instruction from the system control unit 17 and acquires a light field image.

  The card memory 21 can store a large number of light field image image files (described later) acquired by the imaging unit 11 and light field image image files acquired by other light field cameras. It is.

  The recording interface 18 serves as an interface between the card memory 21 attached to the light field camera 10 and the system control unit 17. The recording interface 18 writes a light field image image file to the card memory 21 and reads a light field image image file from the card memory 21 in accordance with an instruction from the system control unit 17.

  The display unit 19 includes a multi-touch panel and includes at least a monitor such as a liquid crystal color monitor and a multi-touch sensor that detects a user operation on the monitor. Among them, the multi-touch sensor has two or more multi-touch sensors in addition to user operations using one finger (touch (touch), slide (trace), tap (tap), double tap (tap twice)), etc. It is possible to detect user operations (such as pinching (pinching), pinching in (narrowing), pinching out (expanding), etc.) using the fingers at the same time. Such a display unit 19 displays various images in accordance with instructions from the system control unit 17, detects the contents of user operations on the images, and transmits them to the system control unit 17.

  In addition to a power button (not shown), the operation member 20 includes various input switches such as a release button 20a, a reproduction button 20b, and a cross button 20c shown in FIG. The user inputs various instructions such as a shooting instruction and a mode switching instruction to the system control unit 17 via the operation member 20. In the present embodiment, since the display unit 19 configured by a touch panel is used, the operation members 20 other than the power button can be omitted.

  The system control unit 17 is a computer that includes a ROM, a RAM, a CPU, and the like (not shown). The CPU controls each part of the light field camera 10 by executing a program stored in the ROM, and operates the light field camera 10 in various modes such as a photographing mode and a reproduction mode. Note that the RAM is a memory that temporarily stores data necessary during the operation of the CPU.

  Incidentally, when the light field camera 10 is in the shooting mode, the system control unit 17 drives the imaging unit 11 in accordance with the shooting instruction from the user to acquire the light field image, and creates an image file of the light field image. And stored in the card memory 21.

  When the light field camera 10 is in the reproduction mode, the system control unit 18 displays the image file on the card memory 21 as a thumbnail, and reproduces and displays the image file on the card memory 21.

  FIG. 2 is a configuration example of an image file. As shown in FIG. 2, this image file has an area for storing a light field image and an area for storing accompanying information of the light field image. The accompanying information includes, for example, subject information, thumbnail images, history information, and the like.

  First, the subject information is information for explaining each of one or a plurality of subjects shown in the light field image. The subject information of each subject includes attribute information of the subject and an area indicating the existence area of the subject. Information and distance information indicating the subject distance of the subject are included.

  Among these, the attribute information includes subject type information (human, building, tree, mountain, sea,...) Indicating the type of subject, and subject name information (Mr. A, Mr. B, etc.) for distinguishing subjects of the same type from each other. Building 1, Building 2, Tree 1, Tree 2, ...).

  Next, the thumbnail image is a size-reduced version of the refocus image created from the light field image. Here, it is assumed that a plurality of thumbnail images having different shooting conditions (shooting distance, aperture value, viewpoint combination) are stored in the image file. Among the plurality of thumbnail images, a thumbnail image of a thumbnail image of a pan focus image (= a thumbnail image of a refocus image having a maximum depth of field, that is, a maximum aperture value) is also included.

  Next, the history information indicates a history of shooting conditions adopted at the time of image file reproduction or thumbnail display, and includes at least information of recently adopted shooting conditions (previous shooting conditions).

  Of the accompanying information described above, the subject information and the thumbnail image are extracted from the light field image by the image processing unit 15 when the image file is created or necessary. Alternatively, the image file is extracted from the light field image by another light field camera or a personal computer at the time when the image file is created or necessary.

  Of the accompanying information, the history information is written by the system control unit 17 at the timing when the image file is displayed as a thumbnail or reproduced. Alternatively, the image file is written by another light field camera or personal computer at the timing when the image file is displayed as a thumbnail or reproduced.

  Incidentally, in order to extract subject area information and attribute information from a light field image, a known subject extraction process such as template matching may be performed on the light field image. However, in order to extract area information and attribute information of all subjects, a plurality of refocus images with different shooting distances are created from the light field image, and subject extraction is performed for each of the plurality of refocus images. It is necessary to perform processing. Alternatively, it is necessary to create a pan focus image from the light field image and subject extraction processing to the pan focus image. Note that the pan-focus image corresponds to a refocus image having the maximum depth of field (maximum aperture value). In addition, in order to extract subject distance information from a light field image, it is necessary to perform distance calculation processing according to the coordinates of the subject on at least two parallax images constituting the light field image.

  In the above description, the storage location of the accompanying information of each image file is each image file. However, the storage location of some or all of the accompanying information of each image file may be a database for managing each image file. . In this database, the accompanying information of each image file is associated with the file name of each image file. The storage location of this database is, for example, the card memory 21.

  Hereinafter, the operation of the light field camera 10 in the reproduction mode will be described.

  At the beginning of the reproduction mode, the system control unit 17 refers to the plurality of image files stored in the card memory 21 and displays the thumbnail images of the plurality of image files side by side on the display unit 19 (thumbnail display). At this time, a thumbnail image having the same shooting condition as the previous shooting condition is selected as the thumbnail image of each image file. However, a thumbnail image having a default shooting condition is selected as a thumbnail image of an image file having no history information. If a necessary thumbnail image is not found, it is created at this timing.

  When the user taps a desired thumbnail image on the display unit 19, the system control unit 17 reproduces and displays an image file corresponding to the thumbnail image. Specifically, the system control unit 17 causes the image processing unit 15 to combine the refocus image of the image file, and the combined refocus image is displayed on the entire surface of the display unit 19 as illustrated in FIG. 3A, for example. indicate. The shooting conditions for this refocus image are also set to be the same as the previous shooting conditions. However, a refocus image of an image file that does not have history information is set to a default shooting condition.

  Incidentally, the shooting distance of the refocus image shown in FIG. 3A is relatively short, and the aperture value is also relatively small. For this reason, each of the five subjects (human, building, car, and tree) shown in the refocus image is blurred.

  Next, as shown in FIG. 3B, when the user touches (touches) the refocused image with one finger, the system control unit 17 sets the subject distance of the subject (automobile) located at the touch point. , The shooting distance of the refocused image being displayed is set to be the same as the subject distance. As a result, the refocus image being displayed is focused on the subject (automobile), and shooting conditions other than the shooting distance of the refocus image are maintained.

  Note that the system control unit 17 operates as follows to switch the shooting distance of the refocused image being displayed. That is, first, the system control unit 17 reads a light field image from an image file corresponding to the refocus image being displayed. Next, the system control unit 17 instructs the image processing unit 15 to synthesize a refocus image at a desired shooting distance based on the read light field image. Next, the system control unit 17 displays the refocus image synthesized by the image processing unit 15 instead of the currently displayed refocus image.

  Subsequently, when the user touches the finger touched without sliding, the system control unit 17 changes the aperture value of the refocused image being displayed in accordance with the slide amount, so that a value other than the aperture value is obtained. Maintain shooting conditions.

  At this time, as shown in FIG. 3C, the system control unit 17 increases the aperture value of the refocus image when the finger slides away from the first touched point. In the example of FIG. 3C, the depth of field of the refocus image is deep, and not only the focus of the automobile but also the focus of the building located far from the automobile is shown.

  Further, as shown in FIG. 3D, the system control unit 17 decreases the aperture value of the refocus image when the finger slides in the direction approaching the point touched first. In the example of FIG. 3 (D), the depth of field of the refocus image becomes shallow again, and the building located far from the automobile is blurred again.

  Therefore, the user of the present embodiment can intuitively adjust the shooting distance and aperture value of the refocus image by a one-action operation called “touch and slide”.

  Thereafter, when the user's finger leaves the monitor, the system control unit 17 writes the refocus image shooting condition at that time in the history information of the corresponding image file. Thereby, the history information of the image file is updated.

  Thereafter, when an instruction to save the refocus image being displayed is input from the user, the system control unit 17 newly creates an image file of the refocus image and writes the image file to the card memory 21. In addition, it is desirable that the image file information (file name, etc.) of the light field image that is the creation source of the refocus image is written in this image file.

[Modification of First Embodiment]
In the first embodiment, the aperture value is increased when the finger slide direction is away from the first touch point, and the aperture value is decreased when the finger slide direction is closer to the first touch point. However, the aperture value may be decreased when the finger slide direction is away from the first touch point, and the aperture value may be increased when the finger slide direction is closer to the first touch point.

  In the first embodiment, the aperture value is increased when the finger slide direction is away from the first touch point, and the aperture value is decreased when the finger slide direction is closer to the first touch point. However, the aperture value may be increased when the finger slide direction is a predetermined direction, and the aperture value may be decreased when the finger slide direction is the opposite direction.

  In the first embodiment, the shooting condition of the first refocus image displayed is basically set to be the same as the previous shooting condition. However, the default shooting condition (for example, the aperture) is set regardless of the previous shooting condition. (Minimum value) may be set.

  In the first embodiment, the refocus image is focused on the subject located at the touch point. However, the refocus image may be focused on the touch point.

  However, in this case, it is desirable that distance information of each point in the object field is written in advance in the image file of the light field image. Otherwise, it is necessary to extract distance information of necessary points from the light field image.

  Incidentally, in order to calculate the distance information of a certain point in the object scene, for example, a distance calculation process corresponding to the position of the point is performed on at least two parallax images constituting the light field image. That's fine.

  In the first embodiment, a predetermined mark (dot mark, cross mark, cross mark, etc.) may be displayed on the point so that the user's first touch point becomes clear.

  In the first embodiment, in order to distinguish between an object located within the depth of field (focused object) and an object located outside the depth of field (blurred object) on the refocus image, A difference may be provided between the display color (luminance or saturation) of an object located within the depth of field and the display color (luminance or again) of an object located outside the depth of field.

  Alternatively, in order to distinguish between a subject located within the depth of field (focused subject) and a subject located outside the depth of field (blurred subject), a subject located within the depth of field; You may give the mark for distinguishing both with respect to at least one with the to-be-photographed object located outside a depth of field. For example, a rectangular frame mark may be given to a subject located within the depth of field, and a rectangular frame mark may not be given to a subject located outside the depth of field.

  In the first embodiment, the shooting condition that can be adjusted by one action is a combination of the shooting distance and the aperture value, but may be a combination of the shooting distance and the viewpoint or a combination of the aperture value and the viewpoint. .

[Second Embodiment]
Hereinafter, a light field camera will be described as a second embodiment of the present invention.

  Here, only differences from the first embodiment will be described. The difference is that the user's initial operation for adjusting the shooting distance and aperture value is “two-point touch” instead of “one-point touch”.

  Here, it is assumed that a refocus image as shown in FIG. 4A is displayed on the entire surface of the display unit 19. This refocus image is the same as that shown in FIG.

  As shown in FIG. 4B, when the user touches (pinch) the refocused image with two fingers, the system control unit 17 subjects (automobile) located at the midpoint between the two touched points. Referring to the subject distance, the shooting distance of the refocused image being displayed is set to be the same as the subject distance. As a result, the refocus image being displayed is focused on the subject (automobile), and shooting conditions other than the shooting distance of the refocus image are maintained.

  Subsequently, when the interval between the two touch points is enlarged / reduced without releasing the two fingers touched by the user, the system control unit 17 changes the aperture value of the displayed refocus image according to the enlargement / reduction amount. Keep shooting conditions other than aperture value.

  At this time, the system control unit 17 increases the aperture value of the refocus image when the interval between the two touch points increases as shown in FIG. In the example of FIG. 4C, the depth of field of the refocus image is deep, and not only the focus of the automobile but also the focus of the building located far from the automobile is shown.

  Further, as shown in FIG. 4D, the system control unit 17 decreases the aperture value of the refocus image when the interval between the two touch points is narrowed. In the example of FIG. 4D, the depth of field of the refocus image is shallow again, and the building located farther from the car is blurred again.

  Therefore, the user of the present embodiment can intuitively adjust the shooting distance and aperture value of the refocus image with a one-action operation of “pinch out” or “pinch in”.

[Modification of Second Embodiment]
In the second embodiment, “two-point touch” is adopted as the initial operation for adjusting the shooting distance and the aperture value, but “three or more multi-point touch” may be adopted.

  In this case, for example, the system control unit 17 may regard the centers of the plurality of touch points as points designated by the user.

  Or the system control part 17 should just consider the point (gravity center etc.) calculated | required by performing the predetermined arithmetic processing to the coordinate of a some touch point as a point designated by the user.

  In the second embodiment, the aperture value is increased when the interval between the two touch points is enlarged, and the aperture value is decreased when the interval between the two touch points is reduced. The aperture value may be reduced when enlarged, and the aperture value may be increased when the interval between two touch points is reduced.

  In the second embodiment, the refocus image is focused on the subject located at the midpoint between the two touch points. However, the refocus image may be focused on the midpoint.

  In the second embodiment, a predetermined mark (dot mark, cross mark, x mark, etc.) is displayed at the midpoint of the two touch points so that the point designated first by the user is clear. Also good.

  In addition, the second embodiment can be modified similarly to the first embodiment.

[Third Embodiment]
Hereinafter, a light field camera will be described as a third embodiment of the present invention.

  Here, only differences from the first embodiment will be described. The difference is that the light field camera 10 is equipped with an area confirmation mode.

  This area confirmation mode is a mode that the user can call as necessary during the playback mode, or a mode that is mounted on the light field camera 10 instead of the playback mode.

  Here, it is assumed that a refocus image as shown in FIG. 5A is displayed on the entire surface of the display unit 19 at the beginning of the area confirmation mode. This refocus image is the same as that shown in FIG.

  In addition, at the beginning of the area confirmation mode, a pan focus image (FIG. 5 (A ′)) corresponding to the refocus image being displayed is stored in a buffer memory (not shown).

  This pan-focus image (FIG. 5A ') is synthesized by the image processing unit 15 from the light field image from which the refocus image (FIG. 5A) is created. In this pan-focus image, almost all subjects (people, cars, trees, buildings) are in focus. The viewpoint of this pan-focus image (FIG. 5 (A ')) is the same as the viewpoint of the refocus image (FIG. 5 (A)).

  The storage location of the pan focus image may be an image file of a refocus image instead of a buffer memory (not shown). However, if the pan focus image is stored, the pan focus image is stored in the area confirmation mode. It can be used repeatedly.

  As shown in FIG. 5A, when the user taps (slaps) the refocused image with one finger, the system control unit 17 is positioned at the tap point as shown by the dotted line in FIG. A circumscribed rectangular area of a subject (tree) to be viewed is regarded as a user designated area.

  Subsequently, the system control unit 17 refers to the image (in-focus partial image) in the area corresponding to the designated area in the pan-focus image (FIG. 5 (A ′)) (FIG. 5 (B ′)). The in-focus partial image is superimposed and displayed on the designated area of the refocus image as shown in FIG. As a result, only the designated area of the refocus image is switched to the focused image.

  Subsequently, when the user performs a pinch-out operation on the focused partial image displayed in a superimposed manner, the system control unit 17 enlarges the display size of the focused partial image as illustrated in FIG.

  When the user performs a pinch-in operation on the focused partial image displayed in a superimposed manner, the system display unit 17 reduces the display size of the focused partial image.

  Therefore, the user of this embodiment can focus only the designated area with a single tap operation while maintaining the shooting conditions of the non-designated area on the refocus image.

  In addition, the user of the present embodiment can adjust the display size of the focused partial image simply by performing a pinch out or pinch in operation on the focused partial image.

  Further, in this embodiment, since a pan-focus image focused on each subject is prepared in advance, the calculation load related to the display of the focused partial image can be suppressed, and the waiting time of the user can be suppressed. For this reason, switching of designated areas (that is, repeated area designation) can be performed smoothly.

  In this embodiment, since the display size of the focused partial image can be adjusted, the user can also check the designated area in detail as necessary.

[Modification of Third Embodiment]
In the third embodiment, the shooting condition of the refocus image that is displayed first is basically set to the same as the previous shooting condition. However, the default shooting condition (for example, the aperture) is set regardless of the previous shooting condition. (Minimum value) may be set.

  In the third embodiment, the shape of the designated area is a circumscribed rectangle of the subject, but may be a circumscribed ellipse of the subject or the same as the contour shape of the subject.

  In the third embodiment, in order to allow the user to specify an arbitrary subject, the subject is tapped. However, other user operations such as “an operation for drawing a closed curve surrounding the subject”, “a closed curve applied to the subject” are performed. The “drawing operation” can also be applied. In addition, as operation for a user to draw a closed curve, any operation of FIG. 6 (A)-(E) is applicable, for example.

  In the third embodiment, the user designates an arbitrary subject and the area uniquely determined from the contour of the subject is the designated area. However, the user can draw an arbitrary closed curve and leave the area in the closed curve as it is. It may be a designated area. In addition, as operation for a user to draw a closed curve, operation in any one of FIG. 6 (A)-(E) is applicable, for example.

  Alternatively, in the third embodiment, an area having a predetermined size and a predetermined shape that is uniquely determined by the user tapping an arbitrary point and uniquely determined from the tap point may be set as the designated area.

  In the third embodiment, in order to create a focused partial image, a pan-focus image in which each subject is focused is used. However, refocus that is focused only on a designated area (or a subject in the designated area). An image may be used. In this case, however, it is necessary to synthesize a refocus image every time the designated area is switched, or to prepare a plurality of refocus images in focus on different subjects.

  In the third embodiment, after the focused partial image is displayed, the display magnification of the focused partial image is adjusted by the user. However, instead of the display magnification or in addition to the display magnification, the focused partial image is displayed. The user may adjust at least one of the aperture value and the viewpoint.

  In the third embodiment, the user operation for adjusting the display magnification, the aperture value, or the viewpoint of the focused partial image is a specific operation on the display unit 19 (for example, any one described in the first embodiment or the second embodiment). May be a user operation) or a specific operation on the operation member 20.

  For example, a user operation for designating an area is a 1-point or 2-point touch operation, and a user operation for adjusting the display magnification or aperture value or viewpoint of a focused partial image is a slide operation that is continuous with the touch operation. It is good. In this case, it is possible to specify the area and adjust the focused partial image with one action.

  In the third embodiment, the size of the designated area may be adjusted by the user before displaying the focused partial image. The user operation for adjusting the size of the area may be a specific operation on the display unit 19 or a specific operation on the operation member 20.

  For example, a user operation for designating an area may be a 1-point or 2-point touch operation, and a user operation for adjusting the size of the designated area may be a slide operation that is continuous with the touch operation. In this case, it is possible to specify the area and adjust the size of the area with one action.

  Incidentally, in the fourth embodiment described below, area designation and area size adjustment are performed with one action.

[Fourth Embodiment]
Hereinafter, a light field camera will be described as a fourth embodiment of the present invention.

  Here, only differences from the third embodiment will be described. The difference is that the size of the specified area can be adjusted before the focused partial image is displayed, and the area specification and area size adjustment can be performed with a single action. Alternatively, it is performed by “pinch out”.

  When the user touches the refocused image with two fingers, the system control unit 17 according to the present embodiment regards the circumscribed rectangular area of the subject located at the midpoint between the two touch points as the user designated area.

  Subsequently, when the interval between the two touch points is enlarged / reduced without releasing the two fingers touched by the user, the system control unit 17 changes the size of the designated area according to the enlargement / reduction amount.

  At this time, the system control unit 17 increases the size of the designated area when the interval between the two touch points increases, and decreases the size of the specified area when the interval between the two touch points decreases.

  Thereafter, when the two fingers touched by the user are separated from the display unit 19, the superimposed display of the focused partial image (see the third embodiment) is started.

  Therefore, the user of the present embodiment can perform area designation and area size adjustment by a one-action operation of “pinch out” or “pinch in”.

[Modification of Fourth Embodiment]
In the fourth embodiment, “two-point touch” is adopted as the initial operation for specifying the area and adjusting the size of the area. However, “three or more multi-point touch” may be adopted.

  In that case, for example, the system control unit 17 may regard the circumscribed rectangular area of the subject located at the center of the plurality of touch points as the designated area.

  Alternatively, the system control unit 17 may regard a circumscribed rectangular area of the subject located at a point (such as the center of gravity) determined by performing predetermined arithmetic processing on the coordinates of a plurality of touch points as a designated area.

  Alternatively, the system control unit 17 may set a rectangular area uniquely determined from a plurality of touch points as the designated area.

  In addition, the shooting conditions for the refocused image that is displayed first, the shape of the specified area, the type of the refocused image that is the source of the focused partial image, the adjustment parameters for the focused partial image, the user operation of the focused partial image, etc. The fourth embodiment can be modified in the same manner as the third embodiment.

[Fifth Embodiment]
Hereinafter, a light field camera will be described as a fifth embodiment of the present invention.

  Here, only differences from the first embodiment will be described. The difference is that the details of the thumbnail display can be adjusted.

  At the beginning of thumbnail display, the system control unit 17 refers to a plurality of image files stored in the card memory 21 and displays the thumbnail images of the plurality of image files side by side on the display unit 19. At this time, a thumbnail image having the same shooting condition as the previous shooting condition is selected as the thumbnail image of each image file. However, a thumbnail image having a default shooting condition is selected as a thumbnail image of an image file having no history information.

  When the user taps a desired thumbnail image on the display unit 19, the system control unit 17 reproduces and displays an image file corresponding to the thumbnail image. Specifically, the system control unit 17 causes the image processing unit 15 to combine the refocused image of the image file, and displays the combined refocused image on the display unit 19 as shown in FIG. Display in a large area. The shooting conditions for this refocus image are also set to be the same as the previous shooting conditions. However, a refocus image of an image file that does not have history information is set to a default shooting condition.

  Since the number of images that can be displayed side by side on the display unit 19 is limited, here, the image file of the refocused image and the image file with the file creation date and time (shooting date and time) are preferentially selected as thumbnail display targets. Assume that In this case, since the shooting date and time are close to each other between the refocus image being displayed and the plurality of thumbnail images being displayed, there is a high possibility that the subject includes a common subject.

  Next, as shown in FIG. 7B, when the user touches (touches) the refocused image with one finger, the system control unit 17 moves the subject (Mr. A) located at the touch point. With reference to the subject distance, the shooting distance of the refocus image being displayed is set to be the same as the referenced subject distance.

  At the same time, the system control unit 17 finds a thumbnail image in which the same subject (Mr. A) appears in the displayed thumbnail image, and refers to the subject distance of the subject (Mr. A) in the thumbnail image. Set the shooting distance of the image to be the same as the referenced subject distance.

  As a result, the refocus image and the thumbnail image in which the same subject (Mr. A) is photographed are collectively focused on the subject (Mr. A), and shooting conditions other than the shooting distance of the refocus image and the thumbnail image are Maintained.

  Note that the method of switching the shooting distance of the refocused image being displayed is as described in the first embodiment. The method for switching the shooting distance of the currently displayed thumbnail image is as follows. That is, the system control unit 17 reads a thumbnail image having a desired shooting distance from the image file corresponding to the thumbnail image, and displays the read thumbnail image instead of the currently displayed thumbnail image.

  Thereafter, when the user's finger leaves the refocus image, the system control unit 17 updates the history information of the refocus image and the thumbnail image whose shooting conditions have been changed.

  In the example of FIG. 7A, since Mr. A is reflected in each of the three thumbnail images being displayed, when Mr. A is touched as shown in FIG. 7B, the refocus image and the three thumbnails are displayed. Each of the images is collectively focused on Mr. A, and the history information of each of the refocus image and the three thumbnail images is updated in a batch.

  In the example of FIG. 7A, since the building 1 is shown in the left thumbnail image and the center thumbnail image, when the building 1 is touched as shown in FIG. 7C, the refocus image and the left The thumbnail image and the central thumbnail image are collectively focused on the building 1, and the history information of each of the refocus image, the left thumbnail image, and the central thumbnail image is updated collectively.

  Here, if a thumbnail image that does not include the subject (building 1) specified by the user exists in the currently displayed thumbnail image, the system control unit 17 of the present embodiment displays the thumbnail image. In order to cancel, the thumbnail image is close to the shooting date and the image file including the subject (building 1) designated by the user is searched from the card memory 21, and the thumbnail image of the detected image file is the target of cancellation. The thumbnail image is displayed on the display unit 19 instead of the thumbnail image. At this time, the system control unit 17 matches the shooting distance of the thumbnail image to be newly displayed with the subject distance of the subject (building 1).

  Therefore, in this embodiment, when the building 1 is touched as shown in FIG. 7C, another thumbnail image in which the building 1 is captured is automatically replaced with the right thumbnail image in which the building 1 is not captured. Are read and displayed (see the lower right of FIG. 7C).

  Therefore, the user of the present embodiment can focus each of a plurality of displayed images (refocus image and thumbnail image) collectively on the same subject only by specifying the subject on one refocus image. Can do.

  In addition, the user of the present embodiment can call up another image (thumbnail image) including the subject from the card memory 21 only by specifying the subject on one refocus image. That is, the user of this embodiment can also search for an image file from the card memory 21 using an arbitrary subject as a search key.

[Modification of Fifth Embodiment]
In the fifth embodiment, when there is an image that does not include the subject specified by the user in the displayed image group, the display of the image is stopped, but the display of the image is continued. Also good. In that case, the shooting condition of the image is maintained regardless of the user-specified content.

  In the fifth embodiment, when there is an image that does not include the subject specified by the user in the displayed image group, the display of the image is canceled. May be suppressed. In order to suppress the display of the image, for example, at least one of the luminance of the image and the saturation of the image may be set low.

  In the fifth embodiment, the displayed images are collectively focused on the subject designated by the user. However, the shooting distance of each displayed image is collectively set to be the same as the subject distance of the subject designated by the user. May be set (see FIG. 8A).

  FIG. 8A shows an example in which the shooting distance of each image is set to be the same as the subject distance of the subject designated by the user. Each of the right thumbnail image and the center thumbnail image includes the same subject (Mr. A) as the subject (Mr. A) specified by the user, but the subject distance of the subject (Mr. A) in these thumbnail images is Since the subject distance of the subject (Mr. A) in the refocus image is different, the subject (Mr. A) remains blurred in the thumbnail images.

  In the fifth embodiment, only the shooting distance is set as a parameter that can be collectively adjusted between a plurality of images. However, other shooting conditions such as an aperture value and a viewpoint may be used. Other parameters such as information content may be added to the parameters that can be collectively adjusted.

  For example, in the fifth embodiment, a parameter that can be collectively adjusted between a plurality of images may be the depth of field. The depth of field is a parameter determined by a combination of the shooting distance and the aperture value.

  Thus, for example, as shown in FIG. 8B, when the user designates two subjects with different distances on one image, the distance range from the subject distance of one subject to the subject distance of the other subject matches. The shooting distance and aperture value of each image may be collectively adjusted so that the image is in focus (contains within the depth of field).

  In the fifth embodiment, the editing contents (combination of parameters to be collectively adjusted and user operations necessary for batch adjustment) to be performed on one image by the user are described in the first embodiment and the second embodiment. Any of the edit contents described in the embodiment, the third embodiment, and the fourth embodiment may be adopted.

  In the fifth embodiment, the image to be edited by the user is a refocus image displayed larger than the other thumbnail images, but may be a thumbnail image displayed larger than the other thumbnail images.

  In the fifth embodiment, the image to be edited by the user is displayed in a larger size than the other images. However, the image to be edited by the user and the other image are displayed in the same size. Needless to say.

  In the fifth embodiment, the image group to be collectively adjusted is the image group being displayed on the display unit 19, but may be all the image groups in the card memory 21 or all the images in the specific folder. It is good also as an image group. Further, the user may specify in advance the storage destination (folder name, etc.) of the image group to be subjected to batch adjustment.

  In the fifth embodiment, the display method of the image group on the display unit 19 is the thumbnail display (simultaneous display), but the slide show display (sequential display) may be used.

  In that case, the user may perform batch adjustment of the image group before the slide show display, or may perform batch adjustment of the image group during the slide show display. Further, the batch adjustment may be performed at each of a plurality of timings during the slide show display.

  Incidentally, when displaying an image group as a slide show, if an image that does not include the subject specified by the user is excluded from the display target, only the image of the subject specified by the user can be selectively displayed as a slide show, which is effective. .

[Modification of each embodiment]
In each of the embodiments described above, the shooting distance, the aperture, and the viewpoint are described as the shooting conditions that can be set later. However, in the shooting conditions that can be set later, the “posture or shape of the focal plane”, “Shooting distance distribution in the refocus image” may be added.

  Further, the function of the light field camera 10 of the first embodiment, the function of the light field camera 10 of the second embodiment, the function of the light field camera 10 of the third embodiment, and the light field camera 10 of the fourth embodiment. A light field camera having both the above function and the function of the light field camera 10 of the fifth embodiment may be configured.

  Further, in each of the above-described embodiments, the light field camera 10 has been described. However, the present invention is also applied to other cameras that can acquire a photographed image that can be set after photographing conditions, for example, a camera that can perform focus bracket photographing, a camera array, and the like. The invention is applicable.

  Further, in each of the above-described embodiments, the apparatus having both the imaging function and the monitor function has been described. However, the present invention can also be applied to an apparatus that does not have at least one of the imaging function and the monitor function. For example, the present invention can be applied to a digital photo frame, a tablet PC, a smartphone, a desktop PC, a notebook PC, and the like.

  Moreover, you may provide the function of at least one part of the apparatus of any one of embodiment mentioned above with respect to the computer executable program or the storage medium which memorize | stored it.

[Operational effects of the first embodiment or the second embodiment]
As described above, the image processing apparatus (the image processing unit 15, the system control unit 17, and the recording interface 18) according to the first embodiment or the second embodiment can perform shooting after setting of shooting conditions (shooting distance, aperture value, viewpoint). Reading means (system control unit 17, recording interface 18) for reading an image (light field image), display control means (system control unit 17) for displaying the photographed image (refocused image) on a monitor, and user for the monitor Adjusting means (system control) that adjusts at least two shooting conditions of a shot image (refocused image) displayed on the monitor in accordance with a series of operations including a touch operation and a slide operation continuous with the touch operation. Unit 17 and image processing unit 15).

  Therefore, the user can adjust at least two photographing conditions of the photographed image (refocus image) with one action.

  The touch operation is an operation for adjusting the shooting distance of the shot image (refocus image), and the slide operation is an operation for adjusting the aperture value of the shot image (refocus image). is there.

  Therefore, the user can adjust the shooting distance and aperture value of the captured image (refocus image) with one action.

  The series of operations is a pinch-in operation or a pinch-out operation.

  Therefore, the user can intuitively adjust the shooting distance and the aperture value of the shot image (refocus image).

[Operational effects of the third embodiment or the fourth embodiment]
As described above, the image processing apparatus (the image processing unit 15, the system control unit 17, and the recording interface 1) according to the third embodiment or the fourth embodiment can perform shooting after setting shooting conditions (shooting distance, aperture value, viewpoint). Reading means (system control unit 17, recording interface 18) for reading an image (light field image), display control means (system control unit 17) for displaying the captured image (refocused image) on a monitor, and the captured image ( When the user inputs an instruction to adjust the shooting condition for a desired area in the refocused image), adjustment means (system control unit 17, image processing unit) that adjusts the shooting condition for only the area in the captured image (refocused image) 15).

  Therefore, the user can adjust the shooting conditions of the designated area while maintaining the shooting conditions of the area (non-designated area) other than the area (designated area) on the photographed image (refocus image). .

  The adjustment instruction is an instruction for adjusting at least one of the shooting distance of the designated area, the aperture value of the designated area, and the viewpoint of the designated area.

  Therefore, the user maintains the shooting distance, the aperture value, and the viewpoint of the designated area other than the area (designated area) on the photographed image (refocus image) while maintaining the photographing distance, Aperture value and viewpoint can be adjusted.

  Further, the adjustment means (system control unit 17, image processing unit 15) prepares the captured image (pan focus image) having the maximum depth of field in advance, and designates the captured image (pan focus image) from the captured image (pan focus image). A partial image (in-focus partial image) corresponding to the region is cut out, and the partial image (in-focus partial image) is superimposed and displayed on the designated region of the captured image (refocus image).

  Therefore, the calculation load of the adjusting means (image processing unit 15) can be kept low, and the waiting time of the user can be kept short. For this reason, the designated area is smoothly switched.

  The adjustment means (system control unit 17 and image processing unit 15) adjusts at least one of the display size, aperture value, and viewpoint of the partial image (focused partial image) as necessary.

  Therefore, the user can confirm the designated area in detail as necessary.

[Effects of Fifth Embodiment]
As described above, the image processing apparatus (the image processing unit 15, the system control unit 17, the recording interface 18) according to the fifth embodiment reads the captured image that can be set after the shooting conditions (shooting distance, aperture value, viewpoint). (System control unit 17, recording interface 18), display control means (system control unit 17) for displaying a plurality of the captured images (refocused images and thumbnail images) having different imaging timings simultaneously or sequentially on the monitor, and When the user inputs a shooting condition adjustment instruction for one of a plurality of photographed images (refocused images and thumbnail images), all the photographing conditions of the plurality of photographed images (refocused images and thumbnail images) are collectively displayed. Adjustment means (system control unit 17 and image processing unit 15) for adjustment.

  Therefore, the user can collectively adjust all the shooting conditions of the plurality of shot images only by adjusting the shooting conditions of the one image.

  The adjustment instruction is an instruction for adjusting the shooting distance of the shot image.

  Therefore, the user can collectively adjust the shooting distances of all of the plurality of shot images.

  The adjustment instruction is an instruction for designating a focus target object of the captured image.

  Therefore, the user can collectively focus all of the plurality of captured images on the same subject.

  The adjusting means (the system control unit 17 and the image processing unit 15) cancels or suppresses display of the captured image when there is a captured image that does not include the subject in the plurality of captured images. .

  Therefore, the user can easily distinguish between a captured image including the subject and a captured image not including the subject.

  The display control means displays a plurality of the captured images as thumbnails, and reflects the contents of the adjustment in the thumbnail display.

  Therefore, the user can list a plurality of the captured images and the contents of the adjustment.

  The display control means displays a plurality of the captured images as a slide show and reflects the contents of the adjustment in the slide show display.

  Therefore, the user can sequentially confirm the plurality of captured images and the content of the adjustment.

DESCRIPTION OF SYMBOLS 11 ... Imaging part, 15 ... Image processing part, 16 ... Camera control part, 17 ... System control part, 18 ... Recording interface, 19 ... Display part, 20 ... Operation member, 12 ... Shooting lens, 13 ... Micro lens array, 14 ... Image sensor, 10 ... Card memory

Claims (5)

Reading means for reading captured images that can be set after shooting conditions;
Display control means for displaying the captured image on a monitor;
An adjusting unit that adjusts at least two shooting conditions of a shot image displayed on the monitor according to a series of operations including a user's touch operation on the monitor and a slide operation that is continuous with the touch operation;
Equipped with a,
The touch operation is an operation for adjusting the shooting distance of the shot image,
The slide operation is an operation for adjusting the viewpoint of the captured image.
An image processing apparatus. The image processing apparatus according to claim 1 .
The series of operations includes
An image processing apparatus characterized by a pinch-in operation or a pinch-out operation.   Reading means for reading captured images that can be set after shooting conditions;
  Display control means for displaying the captured image on a monitor;
  An adjusting unit that adjusts at least two shooting conditions of a shot image displayed on the monitor according to a series of operations including a user's touch operation on the monitor and a slide operation that is continuous with the touch operation;
  With
  The touch operation is an operation for adjusting the shooting distance of the shot image,
  The slide operation is an operation for adjusting the aperture value of the captured image,
  The adjustment unit is configured to perform the slide operation in a direction away from a position where the touch operation is performed on the monitor or a direction approaching a position where the touch operation is performed on the monitor. Adjusting the aperture value
  An image processing apparatus. The image processing apparatus according to any one of claims 1 to 3 ,
The monitor for displaying the captured image;
An image display device comprising: Imaging means for acquiring the captured image;
An image display device according to claim 4 ,
An imaging apparatus comprising: