JP5967172B2 - Image processing apparatus, image processing method, and image processing program - Google Patents

Description

The present invention relates to an image processing apparatus or the like, an image processing apparatus that performs image processing of the blurring process and the like, relates to an image processing method and an image processing program.

  In recent years, with the widespread use of digital cameras, various digital image processing has been performed on captured still images after the still images are captured by the digital cameras. The blur processing is one of them. The blur process is a process for changing the blur amount of a still image.

  As a blur processing technique, when a pixel to be focused on a still image is selected, the blur amount of the portion corresponding to each pixel of the still image is uniquely changed so that the portion corresponding to the pixel is focused. Then, there is a technique of generating a blurred image. This technology changes the amount of blur by preparing a plurality of still images of the same scene according to the focal length and calculating a three-dimensional distance based on these still images. For example, see Patent Document 1).

  However, in order to perform blur processing, still images acquired at a plurality of focal lengths are necessary, and blur processing cannot be performed with only one still image.

  Therefore, when a finger touches the display unit on which one still image is displayed, the depth / depth of the still image is focused so that the portion corresponding to the pixel at the touched position (hereinafter referred to as “target pixel”) is focused. A technique for performing blur processing according to a map (Depth Map) has been proposed (see, for example, Patent Document 2). A depth map is a plurality of pixels constituting a subject in which distance data indicating a distance between a lens and a subject at the time of shooting (hereinafter referred to as “lens-subject distance”) is reflected in a shot still image. A set of objects corresponding to each lump or pixel. Accordingly, the distance between the lens and the subject can be determined by referring to the distance data of the depth map.

FIG. 14 is an explanatory diagram showing a still image displayed on the display unit.
The still image 702 is an image including a dog subject 703 and a cat subject 704 close to the subject 703.

FIG. 15 is an explanatory diagram showing the positional relationship between the imaging lens 706 and the subjects 703 and 704 at the time of shooting (when the still image 702 is acquired).
As shown in FIG. 15, the subject 703 is closer to the imaging lens 706 than the subject 704. On the other hand, the subject 704 is located farther from the imaging lens 706 than the subject 703. That is, in the still image 702, the lens-to-subject distance of the portion corresponding to each pixel constituting the subject 703 is closer than the lens-subject distance of the portion corresponding to each pixel constituting the subject 704.

FIG. 16 is a distribution diagram showing a depth map of a portion corresponding to each pixel at the position of the horizontal broken line in the still image 702 shown in FIG.
The vertical axis represents the lens-subject distance indicated by the distance data of the portion corresponding to each pixel, and the horizontal axis represents the pixel number of each pixel.
The lens-to-subject distance of pixel number 711 corresponding to the nose position 707 at the left end of the subject 703 is the shortest. On the other hand, the lens-to-subject distance of pixel number 712 corresponding to the right end position 708 of the floating ring of the subject 704 is the longest. It should be noted that the pixel number 711 (hereinafter referred to as “nose pixel”) and the pixel number 712 (hereinafter referred to as “floating ring pixel”) are neighboring pixels, but the lens-subject distance between the respective pixels. Are significantly different.

FIG. 17 is an explanatory diagram illustrating a state in which the user selects a target pixel for the still image 702 described with reference to FIGS. 14 to 16.
When the user wants to select the nose pixel as the target pixel, the user touches the vicinity of the nose of the still image 702 with the finger 705. At this time, if the nose pixel is touched correctly, the nose pixel is selected as the target pixel, and blur processing is performed so as to focus on the portion corresponding to the nose pixel.

JP 2003-141506 A JP 2009-15828 A

  However, depending on the thickness of the user's finger 705 and the contact area, a floating ring pixel that is a neighboring pixel of the nose pixel may be selected as the target pixel. In this case, the blurring process is performed so as to focus on a portion corresponding to a floating ring pixel having a lens-subject distance greatly different from a portion corresponding to a nose pixel to be focused.

  As described above, in the technique described in Patent Document 2, the user may not be able to select a desired pixel, and the position of the pixel instructed by the user to focus (hereinafter, “focus point”). However, there is no means for confirming where the in-focus point is. Therefore, there arises a problem that blur processing unintended by the user is performed on a still image in which the distance between the lens and the subject is significantly different between neighboring pixels.

  In the technique described in Patent Document 1, there is no means for confirming where the in-focus point is. Therefore, it is described in Patent Document 2 that unintentional blur processing is performed without noticing even if the focus point is shifted in a still image in which the lens-subject distance is greatly different between adjacent pixels. There are similar problems with other technologies.

The present invention has been made in view of such a point, and an object of the present invention is to obtain a user-desired blur image based on a focus point selected by the user.

In order to solve the above-described problems, the present invention performs image processing on an image focused on a target pixel , which is a portion corresponding to the position of an indicator with respect to the image , based on distance information corresponding to the image displayed on the display unit. To generate. Then, in response to an instruction to the blur amount adjusting unit , the image processing unit is caused to generate an image in which the degree of focusing of the image is changed .

According to the above-described configuration of the present invention, in response to a user instruction to the blur amount adjustment unit, each pixel is focused on an image (blurred image) focused on a target pixel that is a portion corresponding to the position of the pointer. The degree (blur amount) is changed.

According to the present invention, Yu chromatography The is by instructing the blur amount adjustment unit, based on the pixel of interest of your choice, it is possible to obtain he wanted blurred image of the user, there is an effect that.

1 is an external configuration diagram illustrating an image processing apparatus according to an embodiment of the present invention. 1 is a functional block diagram illustrating a configuration of an image processing apparatus according to an embodiment of the present invention. 4 is a flowchart illustrating an operation of the image processing apparatus according to the embodiment of the present invention. It is explanatory drawing which shows the screen displayed on a display panel at the time of the process of step S6 of FIG. It is explanatory drawing which shows the screen displayed on a display panel at the time of the process of step S8 of FIG. It is explanatory drawing which shows the screen displayed on a display panel at the time of the process of step S9 of FIG. It is explanatory drawing which shows the screen displayed on a display panel at the time of the process of step S10 of FIG. It is explanatory drawing which shows the screen displayed on a display panel at the time of the process of step S11 of FIG. It is explanatory drawing which shows the screen displayed on a display panel at the time of the process of step S12 of FIG. It is explanatory drawing which shows the screen displayed on a display panel at the time of the process of step S14 of FIG. It is explanatory drawing which shows the screen displayed on a display panel at the time of the process of step S15 of FIG. It is explanatory drawing which shows the attention outline detection process by the image processing apparatus which concerns on one Embodiment of this invention. It is explanatory drawing which shows the image processing apparatus which concerns on a modification. It is explanatory drawing which shows the still image displayed on the display part of the conventional image processing apparatus. It is explanatory drawing which shows the positional relationship of an imaging lens and a to-be-photographed object at the time of the still image acquisition shown in FIG. It is a distribution map which shows the depth map of the part corresponding to each pixel in the position of the dotted line shown in FIG. It is explanatory drawing which shows a mode that a user selects an attention pixel with respect to the still image shown in FIG.

  Embodiments for carrying out the present invention will be described below. The embodiments described below are preferable specific examples of the present invention. Therefore, various technically preferable limitations are attached. However, the present invention is not limited to these forms unless otherwise specified in the following description. For example, the numerical conditions of each parameter given in the following description are only suitable examples, and the dimensions, shapes, and arrangement relationships in the drawings used for the description are also schematic.

The description will be made in the following procedure.
<One Embodiment>
1. 1. External configuration of image processing apparatus 2. Functional configuration of image processing apparatus Operation of the image processing device

<One Embodiment of the Present Invention>
An example of an embodiment of the present invention will be described with reference to FIGS.
[1. Appearance configuration of image processing apparatus]

FIG. 1 is an external configuration diagram showing an image processing apparatus according to an embodiment of the present invention.
The image processing apparatus 100 is, for example, a personal computer, a digital still camera, a digital video camera, a mobile phone, a television broadcast receiver, a digital photo frame, or the like. The image processing apparatus 100 includes a housing 102 containing an electronic device including a CPU (Central Processing Unit), a memory, and the like, and a display unit 103 incorporated on the surface of the housing 102. The display unit 103 includes a display panel 104 that displays an image, and a position detection panel 105 that detects the position on the detection surface of the indicator that is stacked on the display panel 104 and touches or is placed on the detection surface. Has been. The indicator is, for example, a user's finger or a touch pen.

  Such an image processing apparatus 100 performs a process of changing the amount of blur in each pixel constituting an image being displayed on the display panel 104 (hereinafter, “blurring process”) based on a user operation on the position detection panel 105. And generate a blurred image. Note that the calculation related to the blur processing is executed by an electronic device built in the housing 102.

[2. Functional configuration of image processing apparatus]
Next, a functional configuration of the image processing apparatus 100 for realizing blur processing will be described with reference to FIG.
FIG. 2 is a functional block diagram showing the configuration of the image processing apparatus according to the embodiment of the present invention.

  As described above, the image processing apparatus 100 includes the display panel 104 and the position detection panel 105 corresponding to the display unit 103. Furthermore, a control unit 201, a first memory unit 204, a recording medium 205, an image processing unit 206, and a second memory unit 207 corresponding to an electronic device built in the housing 102 (see FIG. 1) are provided.

  The recording medium 205 is a non-volatile recording medium such as a hard disk or a flash memory, for example. Various programs for performing blur processing and other processing, pre-captured images, and depth maps (Depth) as meter data related to the images. Map) etc. are recorded. The recording medium 205 also records a blurred image generated by blurring processing. The recording medium 205 may be either an internal recording medium or an external recording medium.

Here, the de TOPS map, the distance between the shooting lens and the object (hereinafter, - referred to as "lens distance between the subject") the distance data indicating, constitute a subject of elaborate-through to the still pictures This is a group of a plurality of pixels or a set of those corresponding to each pixel, which is a so-called distance data distribution. The distance data can also be said to be information indicating the depth of the portion corresponding to each pixel of the still image. By referring to the depth map distance data, the distance between the lens and the subject can be determined.

  As a method for obtaining the distance data, for example, a TOF (Time of Flight) method, a blur amount analysis (Depth from Defocus), or the like can be employed. Here, the TOF method is a method for obtaining the distance to the subject from the light delay time until the light emitted from the light source is reflected by the object and reaches the sensor and the speed of the light. The blur amount analysis is to calculate the distance to the subject from the analysis result of the blur amount (focus degree) in the image. A specific configuration for realizing these methods is described in, for example, Japanese Patent Application Laid-Open No. 2009-15828.

  The control unit 201 performs control for causing each block in the control unit 201 to function based on a program acquired from the recording medium 205 and expanded in the first memory unit 204. As will be described in detail later, the control unit 201 acquires a selected image from the recording medium 205, displays the acquired image (hereinafter referred to as “selected image”) on the display panel 104, and performs various operations related to the selected image. An image is generated and displayed on the display panel 104. This selected image is also output to the image processing unit 206. The first memory unit 204 is composed of, for example, a RAM (Random Access Memory) or the like, and temporarily records data necessary for the control unit 201 to operate.

  The image processing unit 206 performs blurring processing on the selected image input from the control unit 201 so as to focus on a portion corresponding to a target pixel described later. The blur processing is executed based on the depth map related to the selected image.

  The image processing unit 206 temporarily records in the second memory unit 207 data (depth map, etc.) necessary for executing the blur processing. The generated blurred image is recorded on the recording medium 205 via the control unit 201 and is output to the display panel 104. The second memory unit 207 is the same RAM or the like as the first memory unit 204, and may be shared with the first memory unit 204.

Next, a more detailed configuration of the control unit 201 will be described.
The control unit 201 includes a GUI display control unit 208, an enlarged display control unit 209, an icon display control unit 210, and a contour display control unit 211 in order to generate various display screens to be displayed on the display panel 104. .

  The GUI display control unit 208 displays a predetermined operation screen, setting screen, and the like on the display panel 104 based on a program recorded in the first memory unit 204, and a graphical user interface (GUI). It is for realizing. Further, in response to a user operation on the position detection panel 105, an operation input signal for causing the control unit 201 to execute a process desired by the user is generated.

  The enlarged display control unit 209 is for enlarging and displaying a part of the selected image. When the position of the indicator is detected by the position detection panel 105 while the selected image is displayed on the display panel 104, a pixel (hereinafter referred to as “indicator detection position”) corresponding to the position of the indicator (hereinafter referred to as “indicator detection position”). Hereinafter, “a pixel of interest”) is detected from each pixel constituting the selected image. Then, a partial area in the display panel 104 is assigned to a small window (an example of a small screen), and an enlarged image obtained by enlarging an image of a predetermined area including the target pixel is displayed in the small window. Note that the enlargement ratio of the enlarged image can be changed according to the resolution of the selected image. For example, when the resolution of the selected image is larger than the screen resolution of the display panel, if the selected image is not subjected to compression processing and the partial area of the selected image is output to a small window without changing the resolution, the most enlarged image is enlarged. Appears in a small window.

  The icon display control unit 210 is for displaying an icon indicating the position of a target pixel that is a reference when performing blur processing. When the enlarged image is displayed on the display panel 104, a pixel corresponding to the pixel of interest in the selected image (hereinafter, “corresponding pixel”) is specified from among the pixels constituting the enlarged image. Then, an icon for indicating the corresponding pixel is displayed in a small window of the display panel 104. Moreover, the icon display control part 210 moves an icon based on operation of the indicator with respect to a user's detection panel. Since the corresponding pixel is changed when the icon is moved, the target pixel is also changed as the icon is moved.

  The contour display control unit 211 is for clearly displaying the contour of the subject. When the enlarged image is displayed on the display panel 104, the contour of the subject including the target pixel identified by the icon display control unit 210 (hereinafter referred to as “target contour”) is detected. Then, the display panel 104 is displayed so that the noticeable outline becomes clear. For example, the noticeable outline is clarified by changing the color of the noticeable outline to an emphasized color or changing the color of the region surrounded by the noticeable outline.

[3. Operation of image processing apparatus]
Next, the operation of the image processing apparatus will be described with reference to FIGS.
FIG. 3 is a flowchart showing an operation flow of the image processing apparatus according to the embodiment of the present invention. The processing of the characteristic part of the present invention realized by the configuration shown in FIG. 2 is particularly processing of steps S9 to S16 described below.

  First, the user turns on the image processing apparatus 100 (step S1). Then, the control unit 201 confirms whether or not the current editing mode is set to a still image editing mode (hereinafter, still image editing mode) (step S2).

  If the still image editing mode is not set (NO in step S2), the process ends. If the still image editing mode is set (YES in step S2), the control unit 201 sets the current editing mode to a mode for editing the blur amount of the still image (hereinafter referred to as “blur amount editing mode”). It is further confirmed whether or not it has been performed (step S3).

  If the blur amount edit mode is not set (NO in step S3), the process is terminated. If the blur amount editing mode is set (YES in step S3), the control unit 201 displays a list of still images recorded on the recording medium 205 on the display panel 104 (step S4).

  The user uses the position detection panel 105 to select an image to be edited from the still images displayed on the display panel 104 (YES in step S5). Then, as shown in FIG. 4, the control unit 201 displays the selected still image on the display panel 104 as the selected image 402 (step S6).

  Subsequently, the GUI display control unit 208 generates a GUI including a sentence “Please touch the point you want to focus on” and temporarily displays it on the display panel 104 (step S7). After the GUI is erased from the display panel 104, the control unit 201 confirms whether or not the position detection panel 105 has been touched with the indicator (position detection of the indicator has been performed) (step S8). If the position detection panel 105 has not been touched for a certain period of time (NO in step S8), the process returns to step S7.

  For example, as shown in FIG. 5, when the position detection panel 105 is touched with the indicator 403 (YES in step S8), the enlarged display control unit 209 instructs the pixel from among the pixels constituting the selected image 402. A pixel of interest corresponding to the position of the body 403 is detected. Then, as shown in FIG. 6, for example, a lower right partial region of the display panel 104 is allocated to the small window 404, and an enlarged image 405 in which the selected image 402 of a predetermined region including the target pixel is enlarged in the small window 404. Is displayed (step S9).

  Subsequently, the icon display control unit 210 identifies a corresponding pixel corresponding to the target pixel identified in the process of step S <b> 9 from each pixel constituting the enlarged image 405. Then, as shown in FIG. 7, a “◯ (circle)” icon 406 for indicating the position of the identified corresponding pixel is displayed in the small window 404 (step S10).

  When the corresponding pixel is specified in the process of step S <b> 10, the contour display control unit 211 detects the contour of the “floating ring” that is the subject including the corresponding pixel as the target contour 407 as illustrated in FIG. 7. Then, the attention outline 407 is displayed in the small window 404 with a predetermined color. Note that a method of detecting the target contour 407 will be described later with reference to FIG.

  Here, the user is confirmed whether or not the current pixel of interest is set as a focusing point. That is, the GUI display control unit 208 generates a confirmation GUI 408 including a sentence “Do you want to focus at the current point?”, “YES” button, and “NO” button, as shown in FIG. Then, the confirmation GUI 408 is displayed in an area other than the small window 404 of the display panel 104 (step S11).

If the “YES” button is touched with the indicator 403 (YES in step S11), the enlarged display control unit 209 stops displaying the small window 404 (step S17). Then, the image processing unit 206 uses the depth map of the selected image 402 for each pixel constituting the selected image 402 so as to be focused on a portion corresponding to the current target pixel (hereinafter referred to as a blur amount). , “Referred to as a focused pixel focusing blur amount”) (step S18). Then, to change the blurring amount of each Kakue element constituting the selected image 402 to the pixel of interest focus blur (step S19). Then, the selected image whose amount of blur has been changed (hereinafter referred to as “edited image”) is input and displayed on the display panel 104 (step S20).

  After the above processing is completed, the user is confirmed whether or not the current blur amount is acceptable. That is, the GUI display control unit 208 generates a confirmation GUI including the sentence “Is it OK to set the current blur amount?”, “YES” button, and “NO” button, as in the process of step S10. . Then, the generated confirmation GUI is displayed on the display panel 104 (step S21). If the “YES” button is touched with the indicator 403 (YES in step S21), the GUI display control unit 208 stops displaying the confirmation GUI on the display panel 104. Then, a confirmation GUI including a sentence “Do you want to save the edited image?”, A “YES” button and a “NO” button is generated and displayed on the display panel 104 (step S26). If the “NO” button is touched with the indicator 403 (NO in step S26), the process returns to step S3.

  If the “YES” button is touched with the indicator 403 (YES in step S26), the image processing unit 206 compresses the edited image data by JPEG (Joint Photographic Experts Group). Then, an image file according to the Jpeg method is generated (step S27). The control unit 201 increments the file name of the image file generated by the process of step S27 and records it on the recording medium 205 (step S28), and ends the process. The process in step S28 is a process for changing the file name of the image file recorded on the recording medium 205 to a file name different from the file name of the previously generated image file.

  If the “NO” button of the confirmation GUI 408 (see FIG. 8) is touched with the indicator 403 (NO in step S11), the GUI display control unit 208 stops displaying the confirmation GUI 408. Then, as shown in FIG. 9, a confirmation GUI 409 including a sentence “Move the circle mark up and down” and an “OK” button is newly generated. Subsequently, the confirmation GUI 409 is displayed in an area other than the small window 404 of the display panel 104 (step S12).

  If the “OK” button of the confirmation GUI 409 is touched with the indicator 403 (YES in step S13), the GUI display control unit 208 stops the display of the confirmation GUI 409. Then, a position adjustment GUI 410 for changing the position of the “circle (circle)” icon 406, and a confirmation GUI 411 including a sentence “Please press determination when the position is determined.” And a “decision” button are generated. To do. The generated position adjustment GUI 410 and confirmation GUI 411 are displayed in an area other than the small window 404 of the display panel 104 as shown in FIG. 10 (step S14).

  The user moves the position of the icon on the small window 404 by operating the cursor key displayed as an icon as the position adjustment GUI 410 with the indicator 403 (step S15). When the subject including the corresponding pixel is changed from “Float” to “Dog” as the icon 406 moves (see FIG. 11), the contour display control unit 211 changes the contour of “Dog” to the target contour 412. Detect as. Then, the attention outline 412 is displayed in the small window 404 with a predetermined color.

  After the above processing is completed, the control unit 201 confirms whether or not the “OK” button of the confirmation GUI 411 is touched with the indicator 403 (step S16). If the “OK” button is not touched (NO in step S16), the process returns to step S15 to further change the position of the icon 406 of “◯ (circle)”. If the “OK” button is touched (YES in step S16), the process proceeds to step S11 described above.

  On the other hand, if the “NO” button of the confirmation GUI generated in the process of step S21 is touched (NO in step S21), the GUI display control unit 208 displays the edited image generated in the process of step S20. A blur amount adjustment GUI for adjusting the blur amount is generated. Then, the generated blur amount adjustment GUI is displayed on the display panel 104 (step S22). This blur amount adjustment GUI is composed of a “+” button and a “−” button. When the “+” button is pressed, the amount of blur of the edited image is increased. When the “−” button is pressed, the blur of the edited image is increased. The amount is reduced.

  At this time, the GUI display control unit 208 further generates a confirmation GUI including a document “Please press the decision button when the amount of blur is changed to an optimum value” and a “decision” button. Then, the generated confirmation GUI is displayed on the display panel 104 (step S23).

  The user operates the “+” button and the “−” button of the blur amount adjustment GUI. Then, the image processing unit changes the current blur amount in each pixel constituting the edited image to a blur amount corresponding to the user's operation on the “+” button and the “−” button, and generates a new edited image. (Step S24).

  After the above process is completed, the control unit 201 confirms whether or not the “determine” button of the confirmation GUI generated in the process of step S23 is touched with the indicator 403 (step S25). If the “OK” button has not been touched (NO in step S25), the process returns to step S23. If the “OK” button is touched (YES in step S25), the GUI display control unit 208 stops displaying the confirmation GUI and the blur adjustment GUI on the display panel 104, and the process proceeds to step S26. .

Next, processing in which the contour display control unit 211 detects the attention contours 407 and 412 in the processing of step S10 and step S15 (hereinafter referred to as “target contour detection processing”) will be described with reference to FIG.
FIG. 12 is an explanatory diagram showing attention contour detection processing by the image processing apparatus according to the embodiment of the present invention.
FIG. 12A is an explanatory diagram showing an enlarged image displayed in the small window.
FIG. 12B is a distribution diagram showing a depth map of a portion corresponding to each pixel of the broken line portion shown in FIG. The vertical axis represents the lens-to-subject distance indicated by the distance data, and the horizontal axis represents the pixel number of each pixel.

  When the target contour detection process is started, the contour display control unit 211 in the enlarged image 405 includes the depth / number at each pixel in the row direction including the current corresponding pixel 502 (hereinafter referred to as “each pixel in the nth row”). A map (distance data of a portion corresponding to each pixel in the nth row) is acquired. Note that each pixel in the n-th row corresponds to a broken line portion shown in FIG. Then, in order to determine which pixel number the contour pixel 503 forming the contour of the n-th row in the enlarged image 405, that is, the lens-subject distance changes rapidly, the obtained depth map is used as a pixel. A differential depth map is calculated by differentiating with a number (processing 1).

  Subsequently, the contour display control unit 211 detects a pixel number in which the absolute value of the calculated differential depth map is larger than a predetermined threshold value. The pixel with the detected pixel number corresponds to the contour pixel 503 (processing 2).

  Next, the contour display control unit 211 recognizes the two contour pixels 503a and 503b that are closest to the corresponding pixel 502 as a part of the pixels that form the target contour (hereinafter referred to as “corresponding pixels”) ( Process 3).

  Here, the contour display control unit 211 acquires a depth map (distance data of a portion corresponding to each pixel in the n + 1 row) in each pixel in the n + 1 row. Then, the obtained depth map is differentiated by the pixel number to calculate a differential field intensity distribution (processing 4). Then, the same processing as the processing 2 is performed on the differential field intensity distribution calculated in the processing 4 (processing 5).

  Subsequently, the contour display control unit 211 recognizes a pixel adjacent to the corresponding pixel in the nth row (contour pixels 503a and 503b) among the contour pixels in the n + 1th row as a corresponding pixel in the n + 1th row (processing). 6).

  After the above processing is completed, the contour display control unit 211 repeats processing 4 to processing 6 (processing 7). Then, by performing the processing 1 to processing 7 also in the column direction of the enlarged image, the target contour is detected two-dimensionally (processing 8).

  As described above, in one embodiment of the present invention, an enlarged image including the target pixel and an icon indicating the position of the corresponding pixel corresponding to the target pixel can be displayed in the small window. As a result, the user can clearly grasp the target pixel selected by the user in the enlarged image, and can prevent selection errors of the target pixel. As a result, there is an effect that a blurred image as desired by the user can be obtained.

  In one embodiment of the present invention, the pixel of interest corresponding to the corresponding pixel can be changed by moving the icon on the small window. Thereby, the target pixel after the change can be grasped, and a selection error of the target image can be prevented more reliably.

  In one embodiment of the present invention, an enlarged image in which the contour of the subject including the target pixel is emphasized can be displayed in a small window. As a result, it is possible to inform the user in an easy-to-understand manner to which subject the target pixel is included. As a result, it is possible to more reliably prevent an erroneous selection of the target image.

  In the above-described embodiment, the target pixel is selected by moving the icon 406 indicating the corresponding pixel on the enlarged image 405. However, as shown in FIG. 13, an icon 602 indicating the target pixel may be displayed on the selected image 402, and the target pixel may be directly selected by moving the icon 602 on the selected image 402. In this case, the amount of data to be processed can be reduced, so that the processing speed can be increased. Needless to say, in the selected image, the outline of the subject including the target pixel may be highlighted.

  In the above-described embodiment, the boundary pixel is detected using the differential of the depth map when detecting the target contour. However, boundary pixels can also be detected using double differentiation of the depth map. Specifically, a pixel in which the double differential value of the depth map at each pixel constituting the enlarged image becomes zero corresponds to a boundary pixel.

  In the above-described embodiment, the distance data is used to detect the target contour. However, the target contour can also be detected using luminance. For example, the brightness of an arbitrary pixel of an enlarged image is compared with the brightness of an adjacent pixel, and a target contour is detected from a pixel having a brightness difference between the adjacent pixel and a predetermined value. In addition, it goes without saying that the contour of interest may be detected by detecting an object based on a grayscale image.

  In the above-described embodiment, the position of the icon 406 is changed using the position adjustment GUI 410. However, the icon 406 may be moved to the position touched by the indicator in the small window 404.

  In the above-described embodiment, the “O (circle)” icon is used, but the icon is not limited thereto. For example, “× (cross mark)” may be used. In this case, the “x” icon has fewer portions where the background image is hidden compared to the “◯” icon, and the user can obtain more information.

  In the above-described embodiment, the GUI is operated in the processes of steps S14, S16, S22, S23, and S25. However, a hard key is provided in the apparatus body so that the hard key can be operated. May be.

  In the above-described embodiment, the edited image data is compressed by the JPEG method. However, the present invention is not limited to this. The compression method may be, for example, a GIF (Graphics Interchange Format) method, a PNG (Portable Network Graphics) method, a BMP (Bitmap) method, or a TIFF (Tagged Image File Format) method.

  In the above-described embodiment, the file name of the newly generated image file is incremented. However, a folder with the folder name incremented is generated, and the newly generated file is stored in the folder. It may be.

  In the above-described embodiment, a partial area at the lower right of the display panel is assigned to a small window and an enlarged image is displayed in the small window. However, the present invention is not limited to this. For example, on the display panel, a partial area centered on the pixel of interest selected by the user may be assigned to the small window, and the enlarged image may be superimposed and displayed on the small window. As a result, the user can check the corresponding pixel without moving the line of sight from the position corresponding to the pixel of interest selected by himself / herself, which is compared with the case where a small window is displayed at the lower right of the display panel. There is an effect that the selection of pixels becomes easier.

  In the above-described embodiment, when a blurred image is generated, the amount of blur may be determined based on a shooting history, scene confirmation, subject detection, or a user's blurred image editing learning history.

  In the embodiment described above, the blur amount of the still image is adjusted so as to focus on the position corresponding to the target pixel. However, the focus is set on the position corresponding to the target pixel for each frame. You can also adjust the amount of motion blur. The procedure for adjusting the amount of blur of a moving image is as follows. Adjustment of the blur amount of the moving image can be executed by the image processing unit 206 and the control unit 201 as in the above-described embodiment.

  First, the user selects a moving image editing mode and starts editing a moving image. Then, a moving image to be edited is selected. Next, the selected moving image is reproduced, and the image (selected image) of the frame that the user wants to edit is selected. Then, in the selected image, the target pixel is determined in the same manner as in the above-described embodiment.

  When the reproduction of the moving image is resumed after determining the pixel of interest, the image processing unit 206 performs a blurring process on the moving image so as to focus on the position corresponding to the pixel of interest in each frame. Specifically, an image composed of pixels in a predetermined range (peripheral region) including the current pixel of interest (hereinafter referred to as “template image”) is cut out from the selected image. This template image is compared with each frame image constituting the moving image to perform template matching. For comparison between the template image and each frame image, the lens-to-subject distance or color indicated by the luminance and distance data is used.

  Next, a corresponding portion that is similar to the template image, that is, the closest value indicating the lens-to-subject distance or color indicated by the brightness and distance data of the template image is detected for each frame. Then, the pixel of interest in each frame is tracked by identifying the pixel having the closest lens-to-subject distance to the portion corresponding to the pixel of interest for each detected portion of each frame. Subsequently, the blur amount of the image of each frame is adjusted based on a depth map so as to focus on the position of the target pixel of each frame.

  As another method for adjusting the blur amount of the moving image, the moving object detection (motion detection) of the subject image including the target pixel is performed in each frame, and when performing template matching, each moving subject detected subject image is a template image. It is good also as a comparison object. In addition, when a moving image is encoded by, for example, an MPEG (Moving Picture Experts Group) method, the above-described moving object detection processing can be performed in common with MPEG signal processing. In this case, it is possible to simultaneously detect the moving object as described above while decoding the moving image by the MPEG method.

  After the above blur processing is completed, the control unit 201 records the moving image whose blur amount is adjusted as a moving image file on the recording medium 205 and increments the file name of the moving image file. In this way, the moving image generated by the blurring process can be used more effectively by performing the process of dividing and joining the moving image files stored in the recording medium 205.

  As mentioned above, although the example of each embodiment of the present invention was explained, the present invention is not limited to the above-mentioned each embodiment, and it is not limited to the scope of the present invention described in the claims. Needless to say, modifications and application examples are included.

  DESCRIPTION OF SYMBOLS 100 ... Image processing apparatus, 102 ... Housing | casing, 103 ... Display part, 104 ... Display panel, 105 ... Position detection panel, 201 ... Control part, 204 ... 1st memory part, 205 ... Recording medium, 206 ... Image processing part, 207 ... second memory unit, 208 ... GUI display control unit, 209 ... enlarged display control unit, 210 ... icon display control unit, 211 ... contour display control unit, 402 ... selected image, 403 ... indicator, 404 ... small window, 405: enlarged image, 406: icon, 407: attention outline, 408 ... confirmation GUI, 409 ... confirmation GUI, 410: position adjustment GUI, 411 ... confirmation GUI, 412 ... attention outline, 702: still image, 703 , 704 ... Subject, 705 ... Finger, 707, 708 ... Position, 709, 710 ... Pixel

Claims (20)

Based on the distance information corresponding to the image displayed on the display unit, an image focused on the target pixel, which is a portion corresponding to the position corresponding to the user's instruction for the image, is generated,
The image obtained by changing the blur amount calculated based on the distance information of the image focused on the target pixel in accordance with a user operation on each of the operation unit for adding the blur amount and the operation unit for reducing the blur amount. Is generated,
An image processing apparatus comprising: an image processing control unit configured to perform control so as to execute a process of determining the image with the amount of blur based on an instruction of the user after the image with the amount of blur being changed is generated. The image processing apparatus according to claim 1, wherein the blur amount is changed for each pixel constituting the image focused on the target pixel. The image processing apparatus according to claim 1, wherein the image processing control unit controls the operation unit that adds the blur amount and the operation unit that reduces the blur amount to be displayed on the display unit as a GUI. The image processing apparatus according to claim 1, further comprising: a display control unit that displays an icon on a pixel corresponding to a position of an indicator according to the user's instruction. The image processing apparatus according to claim 4, wherein the display of the icon is switched according to an instruction from the indicator. The image processing apparatus according to claim 4 or 5 , wherein display of a target pixel that is a pixel corresponding to the position of the indicator or a target contour including a pixel corresponding to the target pixel is switched according to an instruction from the indicator. . The distance information is represented by a function indicating a relationship between a distance based on a photographing lens and a subject and a pixel,
The image processing device according to claim 6, wherein the target contour is formed of pixels having a differential value obtained by differentiating the function of the distance information with the pixels and larger than a predetermined threshold value. The distance information is a function indicating a relationship between a distance based on a photographing lens and a subject and a pixel,
The image processing device according to claim 6, wherein the target contour is formed from a pixel in which a double differential value obtained by double differentiation of the function of the distance information with the pixel is zero. An enlarged display control unit that displays a child screen including an enlarged image of a portion corresponding to the target pixel of the image on the display unit;
When the enlarged image is displayed on the display unit, the display control unit specifies a pixel corresponding to the pixel of interest from each pixel constituting the enlarged image, and selects a pixel corresponding to the pixel of interest. The image processing apparatus according to claim 4, wherein the icon for indicating is displayed on the sub-screen. The image processing apparatus according to claim 9, wherein the enlarged display control unit displays the child screen on the display unit so as to overlap with a region including a portion corresponding to the target pixel of the image. The distance information is a function indicating a relationship between a distance based on a photographing lens and a subject and a pixel,
The display control unit moves from the pixel whose differential value obtained by differentiating the function of the distance information of the enlarged image displayed on the child screen to a position of the indicator is larger than a predetermined threshold value. The image processing apparatus according to claim 9, wherein a target contour that is a corresponding pixel or a target contour including a pixel corresponding to the target pixel is detected. The distance information is a function indicating a relationship between a distance based on a photographing lens and a subject and a pixel,
The display control unit corresponds to a position of the indicator from a pixel in which a double differential value obtained by double differentiation of the function of the distance information of the enlarged image displayed on the sub-screen is zero. The image processing apparatus according to claim 9, wherein a target contour including a target pixel that is a pixel or a pixel corresponding to the target pixel is detected. The image processing apparatus according to claim 1, wherein the image is a still image recorded in a recording unit. The image processing device according to claim 13, wherein the still image is recorded in the recording unit together with the distance information. The image processing device according to claim 13, wherein the still image is a still image constituting a moving image recorded in the recording unit. The image processing apparatus according to claim 15, wherein the degree of focusing of another still image including the target pixel constituting the moving image is also changed based on the change of the degree of focusing on the still image. The image processing apparatus according to claim 1, wherein the display unit includes a display panel that displays the image and a position detection panel that detects a position on a detection surface of an indicator according to an instruction from the user. The image processing apparatus according to claim 17, wherein the position of the indicator with respect to the image is determined based on contact of the indicator with the position detection panel. Based on the distance information corresponding to the image displayed on the display unit, the image processing control unit generates an image focused on the target pixel, which is a portion corresponding to the position corresponding to the user's instruction for the image. Step to control,
A user operation on each of an operation unit for adding the blur amount and an operation unit for reducing the blur amount calculated by the image processing control unit based on the distance information of the image focused on the target pixel. Controlling to generate the image modified according to:
And a step of controlling the image processing control unit to execute a process of determining the image with the blur amount based on an instruction from the user after the image with the blur amount changed is generated. Processing method. Based on distance information corresponding to the image displayed on the display unit, a process of generating an image focused on the target pixel, which is a portion corresponding to a position corresponding to the user's instruction for the image,
The image obtained by changing the blur amount calculated based on the distance information of the image focused on the target pixel in accordance with a user operation on each of the operation unit for adding the blur amount and the operation unit for reducing the blur amount. Processing to generate
After the image with the blur amount changed is generated, a process of determining the image with the blur amount based on an instruction from the user.
An image processing program to be executed by a computer.

Images