JP2018029270A - Image processing apparatus, control method thereof, imaging apparatus, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To utilize depth information of a subject within a captured image, for image processing on the captured image.SOLUTION: An imaging part 101 acquires a captured image, and a distance information acquisition part 104 acquires distance information indicating the depth in the depth direction of each subject within the captured image. A divided image generation part 108 generates plural divided images from the captured image by using the distance information. Each of the divided images is an image corresponding to the range of an individual subject distance and can be selected arbitrarily by a user. A system control part 103 outputs image data of the generated divided images to a display device and sets one of the divided images in an initial selection state where the image can be selected by the first display. An image processing part 107 performs image processing on a selected image among the images.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing technique using subject distance information related to a captured image.

  There is an imaging device that can detect individual subject areas existing in an image of an imaging scene and obtain subject distance information. The imaging apparatus disclosed in Patent Literature 1 uses the distance information of each subject to present a bird's-eye view map representing the positional relationship in the depth direction of the subject in the imaging scene and the current focus distance. At the time of imaging, the photographer can easily grasp which subject is in focus. In addition, there is an apparatus that can be connected to an external display device as well as a liquid crystal display provided in a main body of a digital camera. A technique for converting the resolution of image data in accordance with a display device connectable with a digital camera has been proposed (Patent Document 2).

JP 2010-177741 A Japanese Patent No. 4652210 JP 2015-186210 A

In the conventional apparatus, the subject distance information (depth information) is exclusively used for image processing performed at the time of imaging, and is not assumed to be used by the user after imaging. For example, a method for allowing a user to easily perform image processing using distance information of a subject during image processing on a captured image has not been proposed.
An object of the present invention is to make it possible to use depth information of a subject in a captured image for image processing on the captured image.

  An apparatus according to an embodiment of the present invention uses a captured image, acquisition means for acquiring depth information indicating the depth of each subject in the captured image, and data of the captured image using the depth information. Generating means for generating image data of a plurality of images each corresponding to a range in a different depth direction, selecting means for selecting one of the plurality of images, and displaying the image data generated by the generating means A control unit that outputs to the device and sets one of the plurality of images to an initial selection state that can be selected in the initial display by the selection unit, and image processing for the image selected by the selection unit Image processing means for performing

  According to the present invention, depth information of a subject in a captured image can be used for image processing on the captured image.

It is a block diagram which shows the structural example of the imaging device in Example 1 of this invention. It is a block diagram which shows the structural example of the divided image generation part in the Example of this invention. 3 is a flowchart for describing overall processing during image editing according to the first exemplary embodiment. It is a flowchart explaining the process following FIG. It is a flowchart explaining the divided image generation process in the Example of this invention. It is a figure which illustrates the captured image and depth division image in Example 1. FIG. FIG. 6 is a diagram for describing first display image data generation processing according to the first embodiment. 6 is a diagram illustrating an example of an image for subject detection in Embodiment 1. FIG. FIG. 6 is a diagram for describing second display image data generation processing according to the first embodiment. It is a figure explaining a divided image generation process. 6 is a diagram illustrating a display example on a display device in Embodiment 1. FIG. It is a block diagram which shows the structural example of the imaging device in Example 2 of this invention. 12 is a flowchart illustrating processing during image editing according to the second exemplary embodiment. 10 is a flowchart illustrating subject extraction processing according to the second exemplary embodiment. FIG. 10 is an explanatory diagram of subject extraction processing according to the second embodiment. 10 is a diagram illustrating a display example on a display device in Embodiment 2. FIG.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the following embodiments, in an image processing apparatus that can use distance information of a subject in a captured image for image processing, a user can select a processing target in a short time or accurately select an image when editing the image. It is.

[Example 1]
FIG. 1 is a block diagram illustrating a configuration example of an imaging apparatus to which the image processing apparatus according to the first embodiment of the present invention is applied. The imaging unit 101 of the imaging apparatus 100 includes a lens, a shutter, a diaphragm, and an imaging element. The imaging unit 101 performs a shooting operation according to an instruction from the system control unit 103, acquires captured image data, and stores the captured image data in the memory 105. Each unit in the imaging apparatus 100 is connected to the bus 102, and transmits and receives data and control signals.

  A system control unit 103 controls the entire imaging apparatus 100. The system control unit 103 includes a CPU (Central Processing Unit), and implements various processes to be described later by reading and executing a program stored in the nonvolatile memory 106. The system control unit 103 performs display control on a display device connected to the imaging apparatus 100. For example, the system control unit 103 instructs a display switching process for the display unit 110 and the communication unit 111 described later, and a display image data generation process for the image processing unit 107 according to the resolution of the display device.

  The distance information acquisition unit 104 acquires distance information corresponding to the captured image data acquired by the imaging unit 101. The distance information corresponding to the captured image data is stored in the memory 105. Regarding the distance information, there are various embodiments as information corresponding to the depth in the depth direction (depth direction) of each subject in the captured image. That is, the information (depth information) indicated by the data corresponding to the depth of the subject directly represents the subject distance from the imaging device 100 to the subject in the image, or the subject distance (subject distance) in the image, Any information may be used as long as it represents the relative relationship of depth. For example, control for changing the in-focus position is performed on the imaging unit 101, and a plurality of captured image data captured is acquired. The distance information can be acquired from the focus area of each captured image data and the focus position information of the captured image data. In addition, when the imaging device of the imaging unit 101 has a pupil-divided pixel configuration, distance information for each pixel can be acquired from the phase difference between a pair of image signals. Specifically, the imaging device converts a pair of light beams that pass through different pupil partial regions of the imaging optical system into optical signals, and converts the paired image data into a plurality of photoelectric conversion units. Output from. The image shift amount of each region is calculated by the correlation calculation between the paired image data, and an image shift map representing the distribution of the image shift amount is calculated. Alternatively, the image shift amount is further converted into a defocus amount, and a defocus map representing the distribution of the defocus amount (distribution on the two-dimensional plane of the captured image) is generated. When this defocus amount is converted into the subject distance based on the conditions of the imaging optical system and the imaging element, distance map data representing the distribution of the subject distance is obtained. Image shift map data, defocus map data, or distance map data of the subject distance converted from the defocus amount can be acquired.

  In the present embodiment, a plurality of captured image data captured by changing the in-focus position is acquired for the imaging unit 101. The distance information is acquired from the focus area of each captured image data and the focus position information of the captured image data. The distance information corresponding to the captured image data is stored in the memory 105. The distance information has a format of a distance image in which each pixel value represents a distance, but the data format is not limited to this. For example, the distance information may be information representing the position and distance of a region of a specific subject (for example, a person) existing in the image of the shooting scene. There is no particular limitation on the method of generating distance information by the distance information acquisition unit 104, and various methods can be applied.

  The memory 105 stores various types of data such as captured image data acquired by the imaging unit 101 and distance information related to the captured image data acquired by the distance information acquisition unit 104. The memory 105 has a storage capacity sufficient for storing a predetermined number of captured image data. The nonvolatile memory 106 is an electrically erasable and recordable memory. For example, an EEPROM (Electrically Erasable Programmable Read-Only Memory) is used. The nonvolatile memory 106 stores constants, setting values, programs, and the like for operation of the system control unit 103. The program here is a program for executing processing of various flowcharts described later in the embodiment.

  The image processing unit 107 performs predetermined image processing on the captured image data read from the imaging unit 101 or the memory 105 in accordance with an instruction from the system control unit 103. The predetermined image processing includes pixel interpolation, resizing processing, color conversion processing, blurring processing for individual distance ranges calculated using distance information, sharpness processing, color conversion processing, and the like.

  The divided image generation unit 108 acquires captured image data from the memory 105, distance information from the distance information acquisition unit 104, and display device information described later, and subject images included in a predetermined distance range in the captured image data. The process which extracts is performed. A subject image included in a predetermined distance range (depth direction range) in captured image data is referred to as a depth-divided image or simply a divided image. The divided image generation unit 108 generates image data of the depth divided image and stores it in the memory 105. For example, it is assumed that the distance range is divided into three distance ranges in the depth direction (the direction along the shooting direction). The first distance range is the distance range closest to the imaging device, and the third distance range is the distance range farthest from the imaging device. The first divided image is an image including a subject image within the first distance range (an image of a foreground subject). The second divided image is an image including a subject image within the second distance range (an image of a subject at an intermediate distance). The third divided image is an image including a subject image within the third distance range (an image of a distant subject). Details of the divided image generation unit 108 will be described later.

  The operation unit 109 is a user interface unit that receives a user operation instruction, converts it into control information of the imaging apparatus 100, and notifies the system control unit 103 of the control information. The display unit 110 includes a display device such as an LCD (Liquid Crystal Display), and displays image data read from the memory 105 on the screen. The communication unit 111 is a communication interface unit for transmitting / receiving control commands and data to / from an external apparatus including a display device. For example, the imaging device 100 communicates with an external device through a wired connection based on a standard such as USB (Universal Serial Bus) or HDMI (registered trademark). Alternatively, the imaging apparatus 100 communicates with an external apparatus through wireless connection based on a standard such as a wireless LAN (Local Area Network) or BlueTooth (registered trademark). The imaging device and the external device may be directly connected, or may be connected via a server or a network such as the Internet. In the present embodiment, it is assumed that display device information can be received and display image data related to a captured image can be transmitted to at least a display device of an external device to which the communication unit 111 is connected.

  The recording medium I / F unit 112 is an interface unit with the recording medium 113 such as a memory card or a hard disk. The recording medium 113 is composed of a semiconductor memory, a magnetic disk, or the like, and records captured image data and distance information related to the captured image data.

  The subject detection unit 114 detects a subject area and its attribute in the captured image acquired from the memory 105, associates the detected attribute information with the subject area, and stores the detected attribute information in the memory 105 as a subject detection result. The attribute information is information for identifying a subject area, and it is sufficient that individual subject images can be distinguished. For example, the attribute information is information indicating that the subject is a person, an animal, a plant, a building, a person's face, or the like. The subject detection unit 114 detects a feature quantity such as a feature vector from an input image using a known technique (Patent Document 3), and compares the detected feature quantity with a feature quantity held in a recognition dictionary. The recognition dictionary is a database holding feature quantities of detection targets such as people, faces, animals, mountains, buildings, and the like. The subject detection unit 114 calculates a likelihood called a similarity or an evaluation value, and determines that a detection target has been detected when the likelihood is equal to or greater than a predetermined threshold.

  The object image generation unit 115 generates object image data corresponding to the detected subject attribute information from the detection result of the subject by the subject detection unit 114 acquired from the memory 105 and stores the object image data in the memory 105. The personal authentication unit 116 compares the subject that the subject detection unit 114 determines to be a face with personal authentication data registered in advance in the nonvolatile memory 106, and the detected face image of the person is registered. It is determined whether or not it matches the human face image. The personal authentication unit 116 stores the personal authentication determination result in the memory 105. The in-focus degree calculation unit 117 calculates an in-focus degree indicating the degree of focus on each subject detected by the subject detection unit 114, and stores the in-focus degree calculation result in the memory 105.

  A divided image initial selection determination unit (hereinafter referred to as an initial selection determination unit) 118 acquires the determination result of the personal authentication unit 116 and the calculation result of the focus degree calculation unit 117 from the memory 105. When the depth division image is displayed on the screen of the display device, the initial selection determination unit 118 first determines the depth division image to be selected, and stores the selection result in the memory 105. In the present embodiment, an example in which the initial selection image of the depth-divided image is determined based on the results of the personal authentication unit 116 and the focus degree calculation unit 117 will be described, but the present invention is not limited to this. For example, the initial selection determination unit 118 may determine the initial selection image from the result of the personal authentication unit 116 or the result of the in-focus degree calculation unit 117, or may be determined by another method.

  With reference to FIG. 2, an internal configuration of the divided image generation unit 108 will be described. FIG. 2 is a block diagram illustrating a configuration example of the divided image generation unit 108. The divided image generation unit 108 includes a distance range division processing unit 201, a divided image generation mask data generation processing unit 202, and a divided image data generation processing unit 203.

A distance range division processing unit (hereinafter, referred to as a division processing unit) 201 divides a subject distance range (a range in the depth direction) into a plurality of pieces using the distance information acquired from the memory 105 and the display device information. By dividing the subject distance range, subject distance range division information is generated. A divided image generation mask data generation processing unit (hereinafter referred to as a mask generation processing unit) 202 acquires distance range division information of a subject from the division processing unit 201 and generates mask image data for generating a depth division image. . A divided image data generation processing unit (hereinafter referred to as a data generation processing unit) 203 generates a plurality of divided image data using captured image data acquired from the memory 105 and mask image data for generating a depth divided image. To do. The generated divided image data is stored in the memory 105 in association with the captured image data.
Note that the configuration of the imaging apparatus 100 is not limited to the configuration illustrated in FIGS. 1 and 2 as long as the processing described below is possible, not limited to the configuration example illustrated in FIGS. 1 and 2.

The processing of this embodiment will be described with reference to FIGS. 3 and 4 are flowcharts for explaining the overall processing flow. FIG. 5 is a flowchart for explaining the divided image generation processing. FIG. 6 is an explanatory diagram of captured image data, display image data, and the like. In this embodiment, image processing using distance information is performed on captured image data. Image processing includes, for example, refocus processing for a subject at a specific distance, color space conversion, and the like. An example in which the user performs image editing on a captured image using the imaging apparatus 100 will be described below. The imaging apparatus 100 can be connected to display devices having different display resolutions. For example, it is as follows.
-Display device A capable of high-resolution display
Vise with a display that can secure a sufficient display resolution for detailed image editing, such as a PC (personal computer) display.
-Display device B with lower display resolution than display device A
A display device, such as a display panel attached to the main body of an imaging device, that cannot ensure a sufficient display resolution for performing detailed image editing.

  With reference to the flowcharts of FIGS. 3 and 4, the overall processing at the time of image editing will be described. The following processing is realized by the CPU of the system control unit 103 reading and executing a program from the memory.

  In S <b> 301, the system control unit 103 acquires captured image data and distance information corresponding to the captured image data from the recording medium 113, acquires the maximum number of divided images N from the nonvolatile memory 106, and stores these information to the memory 105. Store. Note that the maximum number of divided images N is a fixed value or a variable value, and in the latter case, it is determined, for example, by designation from the user.

  In step S302, the system control unit 103 acquires display device information of a display destination. The display device information is information related to a display device that is a display destination, and is unique to the display device A, the display device B, and the like. The display device information in the present embodiment is display resolution information of the display device. In addition, the display device information may include the size of the display device (screen size information), or both the display resolution and the size of the display device. The acquisition form of the display device information is as follows.

A form in which the imaging apparatus 100 stores display device information For example, the nonvolatile memory 106 stores display device information related to a display device that can be connected to the imaging apparatus 100. In this case, the system control unit 103 acquires display device information from the nonvolatile memory 106.
Form in which the imaging apparatus 100 acquires display device information from the display device that is the display destination The system control unit 103 receives the display device information via the communication unit 111 for the display device in which the connection with the imaging apparatus 100 is detected. Request and get display device information.

  In S303, the divided image generation unit 108 uses the captured image data, the distance information, the display device information, and the maximum number N of divided images stored in the memory 105 according to an instruction from the system control unit 103. Depth division image data is generated. The divided image generation unit 108 stores the generated N pieces of depth divided image data in the memory 105. In the present embodiment, the maximum number of divided images N is N = 5. A specific example is shown in FIG. FIG. 6A illustrates captured image data. From the captured image data, each data of the depth-divided images 1 to 5 shown in FIG. 6B is generated. Details of the divided image generation processing in S303 will be described later with reference to the flowchart of FIG.

  In S <b> 304 of FIG. 3, the subject detection unit 114 detects a subject from the captured image data in accordance with an instruction from the system control unit 103, and stores a detection result of the detected subject in the memory 105. The detection result of the subject is composed of subject attribute information and region information in the captured image. With respect to the types of subject attributes to be detected in this process, for example, all subject attributes that can be detected by the subject detection unit 114 may be targeted, or only some attributes may be targeted. Furthermore, the embodiment may be such that the user is prompted to select, a predetermined input screen is presented, and the user selects. In the present embodiment, in consideration of the time of image editing, for example, only attributes that can be subjected to image processing corresponding to subject attributes such as color correction suitable for the face and blurring processing are detected. . A specific example is shown in FIG. FIG. 8A shows an example in which a person, a person's face, an animal, and a tree are detected as subjects in the captured image data of FIG. 6A. In this case, the detection target is a subject having attributes of a person, a person's face, and an animal.

  In S305 of FIG. 3, the personal authentication unit 116 collates the face attribute detected in S304 with the face registration data stored in the nonvolatile memory 106 in accordance with an instruction from the system control unit 103. The personal authentication unit 116 stores the result of authentication processing by verification in the memory 105. In step S <b> 306, the focus degree calculation unit 117 calculates the focus degree for each subject detected in step S <b> 304 according to an instruction from the system control unit 103, and stores the calculation result in the memory 105.

  In step S <b> 307, the initial selection determination unit 118 determines a depth division image that can be selected in the initial state when displaying the depth division image in accordance with an instruction from the system control unit 103, and stores the result in the memory 105. The divided images that can be selected in the initial state are determined based on the processing results performed in S305 and S306. For example, the initial selection determination unit 118 determines, as an initial selection state, a depth division image in which a large number of people registered in advance in the apparatus are present among a plurality of depth division images based on the processing result of S305. If there are the same number of registered persons in each depth-divided image, the initial selection determination unit 118 determines the depth division with a high degree of focus among the plurality of depth-divided images from the processing result of S306. The image is determined as the initial selection state. The initial selection based on the face authentication result and the degree of focus is an example, and is not limited to this. The depth division image may be determined only from the face authentication result, or the depth division image may be determined only from the degree of focus. Or you may determine from a depth division image by the method of other than that.

  In S308 of FIG. 4, the system control unit 103 compares the display resolution of the display device with a predetermined threshold based on the display device information. The predetermined threshold value is stored in advance in the nonvolatile memory 106. When the display resolution of the display device is equal to or greater than the predetermined threshold, the process proceeds to S309. If the display resolution of the display device is less than the predetermined threshold value, the process proceeds to S316. In the present embodiment, it is assumed that the display device A has a display resolution equal to or higher than the threshold, and the display resolution of the display device B is less than the threshold. When the display device information is screen size information indicating the size of the display device, the screen size information is compared with the threshold value. In this case, the determination condition of S308 is that the screen size information of the display device is greater than or equal to a predetermined threshold value.

  In step S <b> 309, the image processing unit 107 generates display image data from N pieces of depth-division image data stored in the memory 105 according to an instruction from the system control unit 103. Hereinafter, the display image data generated in S309 is referred to as first display image data. The image processing unit 107 stores the generated first display image data in the memory 105.

  In the present embodiment, the system control unit 103 controls processing for generating first display image data representing the distance relationship between the divided images and the shooting angle of view from the depth divided image data illustrated in FIG. To do. Specifically, the image processing unit 107 performs resizing processing on each piece of depth-division image data so that the closer the distance from the imaging device is, the smaller the image size becomes, and then the first display image data To represent the shooting angle of view.

  FIG. 7A is a diagram illustrating first display image data expressed by five layers. The depth division image 1 corresponding to the short distance is arranged in the frontmost layer, and the depth division image 5 corresponding to the long distance is arranged in the innermost layer. The depth division images 2 to 4 are respectively arranged in layers set according to the distance information. Image data that is displayed by arranging all the depth-division image data so that at least a part of the image data can be seen is defined as first display image data. FIG. 7A illustrates a case where the person's face image arranged in the divided image 2 in FIG. 6B is a person's face image registered in the imaging device 100. Based on the processing result of the personal authentication unit 116, the initial selection determination unit 118 determines that there is a high possibility of performing image processing on a registered person. As a result, as shown in FIG. 7A, the divided image 2 is selected as the initial state, and is presented in an emphasized display that is easy for the user to visually recognize.

  In addition to the example of the first display image data shown in the present embodiment, the user can also select and select all the depth-division image data. For the depth-divided image data, it is only necessary that the divided images are arranged and displayed in a direction in which the subject distance (the distance from the imaging device to the subject) increases or decreases. For example, the depth-division image data may be arranged so that the subject distance increases from the right to the left instead of from the front to the back.

  FIG. 7B is a schematic diagram for explaining the association between the first display image data and the depth-division image data. It is assumed that the origin (0, 0) is set and the coordinate axes are set in the vertical axis direction and the horizontal axis direction. Selection information in which the coordinates on the image of the first display image data designated by the user operation are associated with the selection destination of the depth-division image is stored in the memory 105. In the example of FIG. 7B, the position coordinates designated by the user operation with respect to the first display image data are in the divided image 2, and the selection destination of the depth divided image is the divided image 2.

  In S <b> 310, the display unit 110 displays the display image data acquired from the memory 105 on the display device according to an instruction from the system control unit 103. A specific display example will be described later. In this embodiment, when connection with a plurality of display devices is detected at the same time by the imaging apparatus 100, only the display device with the highest resolution is displayed. In the case of an embodiment in which display is performed on a plurality of display devices at the same time, the processing of S308 to S310 is repeated for the number of display devices that are display destinations, and display image data corresponding to each display device is generated. Is done.

  In step S311, the system control unit 103 determines whether the operation unit 109 has detected a user operation. If a user operation is detected, the process proceeds to S312. If no user operation is detected, the process returns to S310. S312 is determination processing as to whether or not the user operation detected in S311 is an image processing operation for a depth-division image. Regarding the image processing operation for the depth-divided image, it is sufficient that at least the depth-divided image is selected by the user operation. In the depth division image selection operation, the user can select a desired depth division image by, for example, key input of the operation unit 109. The depth-division image corresponding to the key input is highlighted. At this time, the depth division image highlighted in the initial state is the depth division image determined in S307. In addition to this, when the display device is in a touch panel format, the user can select desired depth-division image data by designating coordinates within the surface of the display image.

  If the user operation detected in S311 is an image processing operation for a depth-division image, the process proceeds to S313. If the user operation is not an image processing operation for a depth-division image, the process proceeds to S314. In step S <b> 313, the image processing unit 107 executes predetermined image processing on the selected depth division image and subject image in accordance with an instruction from the system control unit 103. The image processing unit 107 stores the data after image editing that has undergone predetermined image processing in the memory 105. The predetermined image processing is a blurring process, a sharpness process, a color conversion process, or the like performed on a distance range corresponding to the depth division image selected by the user operation. In addition, image processing such as blurring processing, sharpness processing, and color conversion processing on a subject area in a depth-division image selected by the user may be performed. The image processing in S313 may be image processing for a divided image that can be performed by the user selecting the divided image. Thereafter, the process proceeds to S303.

  S314 is a determination process regarding whether or not the user operation is an image editing mode end operation. If the user operation is an operation for ending the image editing mode, the process proceeds to S315. If the user operation is not an operation for ending the image editing mode, the process proceeds to S310. In step S <b> 315, the system control unit 103 performs processing for storing the image edited data subjected to desired image processing in the recording medium 113 via the recording medium I / F unit 112, and ends a series of processing.

  Next, the processing from S316 to S318 will be described. If it is determined in S308 that the display resolution of the display device is less than the predetermined threshold, second display image data is generated. The second display image data generation process will be described with reference to FIG. FIG. 8 is a diagram for explaining the second display image data generation process.

  In step S <b> 316, the object image generation unit 115 acquires the detection result of the subject and the captured image data from the memory 105 according to an instruction from the system control unit 103, and generates object image data corresponding to the attribute of the subject. The generated object image data is stored in the memory 105. Specific examples are shown in FIGS. 8A to 8C. A group of object image data corresponding to the subject attribute is stored in the nonvolatile memory 106 in advance. Corresponding object image data is acquired from the non-volatile memory 106 for attributes such as people and animals. FIG. 8C shows object image data for face attributes. For the face attribute, image data obtained by cutting out the face area of the subject from the captured image using the captured image data and the subject area information, that is, the extracted image data is used as object image data. In addition to this, for example, color information of a subject area in a captured image may be acquired and used for color information of object image data.

  In S317 of FIG. 4, the system control unit 103 calculates the detected subject distance information using the subject region information acquired from the memory 105 and the distance information corresponding to the captured image. Specifically, when the distance information is information in the form of a distance image, the system control unit 103 treats the area information in the captured image of the subject as the area of the subject in the distance image, and the pixel value in the area Is calculated as the distance information of the subject.

  In step S <b> 318, the image processing unit 107 generates second display image data using the object image data, the subject area information, and the distance information corresponding to the captured image in response to an instruction from the system control unit 103. The generated second display image data is stored in the memory 105. A specific example is shown in FIG. FIG. 9A shows second display image data composed of object image data for the attributes of face, person, and animal.

  FIG. 9B is a schematic diagram for explaining the association between the second display image data and the depth-division image data. Using object distance information, object image data corresponding to a short distance among object image data corresponding to each object is arranged on the near side, and object image data corresponding to a long distance is arranged on the back side. The image data arranged and displayed so that at least a part of all the object image data can be seen is defined as second display image data. At this time, the object image data for displaying a person's face in FIG. 9A is set as an initial selection state and presented in a highlighted manner to the user. However, the present invention is not limited to the example of the second display image data shown in the present embodiment, and all object image data can be sorted and selected by the user. As for the object image data, it is only necessary that each image is arranged and displayed in a direction in which the subject distance corresponding to the object increases or decreases. For example, the object image data may be arranged so that the subject distance increases from the right to the left instead of from the front to the back.

  As shown in FIG. 9B, the system control unit 103 stores, in the memory 105, selection information that associates the coordinates designated on the image of the second display image data with the selection destination of the divided image data. To do. It is assumed that the origin (0, 0) is set and the coordinate axes are set in the vertical axis direction and the horizontal axis direction. The selection information in which the coordinates of the second display image data designated by the user operation and the selection destination of the depth division image are associated with each other is stored in the memory 105. In the example of FIG. 9B, the position coordinates designated by the user operation with respect to the second display image data are in the divided image 2, and the selection destination of the depth divided image is the divided image 2.

  If no subject to be detected is detected in the process of S304 in FIG. 3, the system control unit 103 stores the captured image data in the memory 105 as second display image data. In this case, the system control unit 103 does not generate selection information that associates the coordinates of the display image data with the selection destination of the depth-division image data.

Next, the divided image generation process will be described with reference to the flowchart of FIG. 5 and FIGS. 6 and 10. FIG. 10 is an explanatory diagram of the divided image generation process. It is assumed that the maximum number of divided images N = 5.
In S401 of FIG. 5, the value of the reference variable (denoted i) of the depth-division image data is initialized to zero (i = 0). In step S <b> 402, the division processing unit 201 obtains a distribution of distances in which the subject exists in the depth direction (histogram by subject distance) based on the distance information. The distance information is in the form of a distance image in which the pixel value indicates the distance of the subject (subject distance) that appears in the captured image. Therefore, the subject distance histogram shows the frequency of pixel values of the distance image, that is, the distribution frequency of the subject distance. By the processing of S402, the distribution frequency of the subject distance corresponding to the captured image data (see FIG. 6A) is obtained. FIG. 10A shows a histogram by distance of the subject. The horizontal axis in FIG. 10A represents the distance in the depth direction, and the vertical axis represents the distribution frequency.

  In S403 of FIG. 5, the division processing unit 201 calculates distance range information for the i-th divided section from the distance histogram of the subject. The distance range divided by the maximum number of divided images N is stored in advance in the nonvolatile memory 106 as a constant. The division processing unit 201 may determine N division ranges (distance ranges) in the order of distances where the appearance frequency of the subject is low, using the distance-specific histogram of FIG. As a result, the distance range information shown in FIG. 10B is obtained. In the case of N = 5, the division processing unit 201 divides the subject distance histogram shown in FIG. 10A into five in the horizontal axis direction, thereby dividing 5 from division 1 to division 5 shown in FIG. Determine the distance range divided into two.

  In S404 of FIG. 5, the mask generation processing unit 202 generates divided image generation mask data from the i-th distance range information. Specifically, using the distance range information for the i-th divided image calculated in S403, a process of binarizing the pixel value is executed for the distance image that is the distance information. That is, the process of setting the pixel value within the distance range to 1 and the pixel value outside the distance range to 0 is performed, and divided image generation mask data is generated.

  In step S405, the data generation processing unit 203 generates i-th divided image data using the captured image data and the i-th divided image generation mask data. Specifically, a process of multiplying each pixel of the captured image data by the pixel value of the divided image generation mask data is executed. Thereby, only the area | region where the pixel value of the mask data for division | segmentation image generation is 1 is each extracted from captured image data, and the data of the depth division | segmentation images 1-5 of FIG. 10 (B) are produced | generated.

  Next, in S406, the reference variable i is incremented and the i value is increased by one. In S407, the value of the reference variable i is compared with the maximum number of divided images N, and it is determined whether or not “i> N”. If the value of the reference variable i is larger than the maximum number N of divided images, the divided image generation process ends. If the value of the reference variable i is equal to or less than the maximum number of divided images N, the process returns to S403 and continues. By performing the above processing, the divided image generation unit 108 generates, for the captured image data, the data of the depth-divided image that is divided in the distance range of the maximum number of divided images N according to the display device that is the display destination.

  A display example will be described with reference to FIG. FIG. 11 is a diagram illustrating a display example on different display devices. As shown in FIG. 11A, on the screen 1101 of the display device A, first display image image data (see FIG. 7A) generated from the depth-division image data is displayed. The user can visually recognize a subject included in all the distance ranges in the captured image data and select a desired subject image. In the example of FIG. 11A, the display device A displays a plurality of depth-divided images for each different layer corresponding to the depth information of the subject, and one image (nearest) is displayed as indicated by the bold line frame. (Division image of distance) is set to the initial selection state.

  As shown in FIG. 11B, the screen 1102 of the display device B has second display image data generated from the object image corresponding to the detected subject in the captured image (see FIG. 9A). Is displayed. The user can easily visually recognize the main subject in the captured image and the subject that can be processed by the image processing unit 107, and can select a desired subject image. In the example of FIG. 11B, the display device B displays a plurality of object images for each different layer corresponding to the depth information of the subject, and one of the images (a subject image with a face attribute) as indicated by a bold frame. ) Is set to the initial selection state. Note that, for a display device that displays the second display image data, such as the display device B, various display switching is possible depending on the display purpose during image editing. You may make it compatible by switching the display of 2nd display image data, and the display at the time of normal image reproduction, or combining these two displays in arbitrary ratios. For example, when image processing is performed on a selected subject image, switching to display of image editing data is performed in an arbitrary period. The user can perform an operation of selecting an image in the initial selection state on the first display screen of the display device A or B, and perform a predetermined operation (such as a slide bar operation) to display a divided image other than the image. It is also possible to select.

  In the present embodiment, the distance information related to the captured image data is used to divide the captured image data into a plurality of images according to the distance range of the subject, and each divided image is presented to be selectable by the user. Also, by switching the display image generation process according to the display device information of the display destination, it becomes possible to perform display corresponding to the display device. That is, it is possible to provide the user with image data for a subject existing at a specific distance in accordance with the resolution and screen size of the display device. When the user edits an image, it is easy to select an image processing target, and convenience is improved.

[Example 2]
Next, a second embodiment of the present invention will be described. In this embodiment, a process of classifying subjects in a captured image using distance information related to the captured image and extracting a predetermined number of subject regions from a plurality of detected subject regions will be described. In the present embodiment, the same reference numerals as those used in Embodiment 1 are used for the same components as those in Embodiment 1, and detailed descriptions thereof are omitted, and differences are mainly described.

  FIG. 12 is a block diagram illustrating a configuration example of the imaging apparatus 100. The personal authentication unit 1216 performs personal authentication processing on the subject whose attribute is determined to be a human face by the subject detection unit 114. At that time, the personal authentication unit 1216 refers to dictionary data for personal authentication recorded in the nonvolatile memory 106. In the non-volatile memory 106, dictionary data obtained by extracting the feature amount of the face of the subject based on an image obtained by the user in advance of the face of the subject to be personally authenticated is registered. The personal authentication unit 1216 determines whether or not the detected face image of the person is the same person face image as the registered dictionary data, and stores the determination result in the memory 105.

  The focus level calculation unit 1217 calculates the focus level for each subject detected by the subject detection unit 114 and stores the calculation result in the memory 105. The degree of focus of the present embodiment is calculated from the contrast information of the image detected from the captured image data, and it is determined that the degree of focus is higher as the image contrast is higher.

  The subject extraction unit 1218 extracts a subject region to be displayed in each depth division image in accordance with an instruction from the system control unit 103 and stores the extraction result in the memory 105. The subject extraction unit 1218 extracts a subject region in order to select an intended subject accurately when the user selects a subject by touching the screen of the display device with a finger to select an image processing target. For example, it is assumed that a large number of subject images are densely included in the captured image. In this case, it is necessary to avoid selecting a surrounding subject unintended by the user. For this purpose, the subject extraction unit 1218 classifies a plurality of subjects based on attributes, and estimates and extracts subject regions that are likely to be selected during image processing for each depth-divided image, thereby making it easier to select subjects. . The extracted subject area is determined from the processing results of the personal authentication unit 1216 and the focus degree calculation unit 1217 and is stored in the memory 105 as subject area information. Note that the present invention is not limited to this embodiment, and a method of preferentially extracting a subject area in the captured image from the processing result of the personal authentication unit 1216 or a priority of the subject area in the captured image from the processing result of the focusing degree calculation unit 1217. There is a method to extract automatically. Alternatively, the process may be performed by another method based on the personal authentication result or the degree of focus. In addition, when the number of subjects to be extracted is larger than a predetermined number, an upper limit may be provided for the number of subjects to be extracted for each depth-divided image.

A processing example in the present embodiment will be described with reference to FIGS. 13 and 4. The processing different from the first embodiment is the processing from S1305 to S1307 and S312.
In S1305 of FIG. 13, the personal authentication unit 1216 collates the subject with the face attribute detected in S304 with dictionary data for personal authentication stored in the nonvolatile memory 106, and stores the processing result in the memory 105. . In step S <b> 1306, the degree-of-focus calculation unit 1217 calculates the degree of focus from the image contrast information for each subject detected in step S <b> 304 according to an instruction from the system control unit 103, and stores the calculation result in the memory 105. In step S <b> 1307, the subject extraction unit 1218 extracts some subjects from the subjects detected in step S <b> 304 according to instructions from the system control unit 103, and stores the extraction results in the memory 105. The subject to be extracted is determined based on the processing results performed in S1305 and S1306. Details of the subject extraction method will be described later.

  In S <b> 312 of FIG. 4, a determination process is performed as to whether or not the user operation is an image processing operation for a depth-division image. In the present embodiment, when the display device is in the touch panel format, the depth division image is selected by specifying the coordinates of the display image on the screen. When a touch operation by the user is detected, the system control unit 103 selects depth-division image data from the coordinates touched on the display image using the selection information generated in S309 or S318. In addition, the divided image corresponding to the key input by the operation unit 109 may be highlighted and presented to the user at the time of selecting the depth divided image data. If the user operation is an image processing operation for a depth-division image, the process proceeds to S313. If the user operation is not an image processing operation for a depth-division image, the process proceeds to S314.

  Next, the subject extraction method shown in S1307 in FIG. 13 will be described with reference to FIGS. In step S901, the subject extraction unit 1218 determines one depth-divided image to be processed. At the first execution, the foremost depth division image is selected. When S901 is executed again after executing the processing of S902 to S909, which will be described later, among the depth division images that have not yet been processed, the foremost depth division image is determined as the processing target. In the present embodiment, description will be made assuming that processing is performed in order from the front, but the processing order is not limited to this and can be arbitrarily determined.

  In step S <b> 902, the subject extraction unit 1218 compares the number of detected subjects with the upper limit (hereinafter referred to as the upper limit subject number, referred to as “M”) for one determined depth-divided image. In the present embodiment, the upper limit subject number M is determined for each depth division image. For example, the upper limit subject number M may be determined automatically by the imaging device in consideration of the distance, the angle of view, or the like, or may be configured to be designated by a user operation. Hereinafter, the upper limit value for the divided image J (J = 1 to 5) will be described as “upper limit value J”, and an example in which each upper limit value is determined in advance will be described. Note that the upper limit value J is not determined for each depth-divided image, but the upper-limit subject number of each depth-divided image is set to the same number, or in consideration of the ease of selecting the subject, depending on the display resolution. A mode in which the upper limit number of subjects is set or changed may be used.

  If the number of detected subjects is larger than the upper limit subject number M in S902, the process proceeds to S903, and if the upper limit subject number M is not exceeded, the process proceeds to S908. For example, in the processing of S902, the case where the depth division image of FIG. 6B is generated for the captured image data of FIG. 6A and the detection result of the subject shown in FIG. 8A is obtained. Suppose. In this case, since the number of detected subjects does not exceed the upper limit subject number M, that is, the upper limit values 1 to 5 in each of the depth division images 1 to 5 in FIG. The process is executed. In step S <b> 908, the subject extraction unit 1218 determines all subjects in the depth-divided image as extraction targets, and stores the extracted results in the memory 105.

  FIG. 15 illustrates a case where the number of subjects detected in a certain depth-divided image exceeds the upper limit subject number M. For the captured image data shown in FIG. 15A, depth-division images 1 to 5 shown in FIG. 15B are generated. FIG. 15C shows a subject detection result for a captured image. In this case, in the depth division image 2 of FIG. 15B, the number of detected subjects exceeds the upper limit subject number M, and in the other depth division images 1, 3 to 5, the number of detected subjects is the upper limit. The number of subjects M is not exceeded. Therefore, the processing after S903 is executed for the depth division image 2, and the processing of S908 is executed for the other depth division images.

  In step S <b> 903, the subject extraction unit 1218 extracts the subject image authenticated by the personal authentication processing in step S <b> 1305 from the subject images in the depth-division image that is the processing target, and stores the extracted result in the memory 105. In step S <b> 904, the subject extraction unit 1218 ranks each subject in the depth-division image that is the processing target in order from the subject with the highest focus degree calculated by the focus degree calculation unit 1217, and the ranking result. Is stored in the memory 105. In step S905, the subject extraction unit 1218 determines whether the number of subjects extracted in step S903 exceeds the upper limit subject number M. When the number of extracted subjects is larger than the upper limit subject number M, the process proceeds to S906, and when the number of extracted subjects does not exceed the upper limit subject number M, the process proceeds to S907.

  In step S906, the subject extraction unit 1218 extracts a predetermined number of subject images from the subject image extracted in step S903. The predetermined number is, for example, the upper limit subject number M, and M subject images are extracted in descending order of focus based on the processing result of S904, and the extraction results are stored in the memory 105.

  On the other hand, in step S907, the subject extraction unit 1218 extracts subject images with a high degree of focus from subjects other than the personally authenticated subject up to the upper limit subject number M together with the number of subject images extracted in step S903. Specifically, the upper limit subject number M is set to 3, and the number of subject images not extracted in S903 is set to 3. When the number of subject images extracted in S903 is 1, two subject images are extracted in descending order of focus from the processing result in S904. That is, subject images are extracted such that M = 3 together with the number of subject images extracted in S903, and the extraction result is stored in the memory 105.

After the processing of S906, S907, and S908, the subject extraction unit 1218 determines in S909 whether extraction processing has been completed for all the depth-divided images. When subject extraction processing is completed for all depth-divided images, a series of processing ends. If there is an unprocessed depth division image, the process returns to S901 to continue the process.
In the present embodiment, the subject image extraction process based on the personal authentication result in S903 and the in-focus degree in S904 is exemplified, but the corresponding subject image may be determined from the individual authentication result or the in-focus degree.

  FIG. 16 shows a screen display example of the display device. A case will be described in which, from the captured image data in FIG. 15A, two subject images in which a face is detected among the four subjects in the divided image 2 are extracted. 16A shows a display example of the display device A, and FIG. 16B shows a display example of the display device B. In any display example, the extracted subject is displayed in an easy-to-understand manner for the user.

As shown in FIG. 16A, when the display resolution of the display device is equal to or higher than the threshold value, the depth-divided images respectively assigned to the plurality of layers are displayed. Referring to the subject area information acquired at the time of the subject detection result in S303 in FIG. 13, the mask generation processing unit 202 mask data for separating the extracted subject and the subject that has not been extracted. Is generated. The data generation processing unit 203 generates depth division image data using the generated mask data. Examples of divided image generation mask data are as follows.
The value in the subject area corresponding to the extracted subject is set to 1.
The value in the subject area that has not been extracted is set to 0.5.
-The values of other areas are set to 0.

  A process of multiplying each pixel value of the captured image data by the pixel value of the divided image generation mask data is executed. Thereby, the image of the subject that has not been extracted is displayed darkly, and the extracted image of the subject is displayed with a predetermined brightness. Therefore, the user can easily visually recognize the image of the subject that has been personally authenticated.

  Further, as shown in FIG. 16B, when the display resolution of the display device is less than the threshold value, the subject image is displayed according to the attribute information of each subject. In this case, when the object image data is displayed, the image of the subject that has not been extracted is displayed darkly, and the size thereof is displayed smaller than that of the extracted image of the subject. The user can clearly distinguish between the extracted subject image and the unextracted subject image.

  In this embodiment, it is possible to provide an image processing apparatus capable of accurately selecting a processing target by extracting a selectable subject image at the time of image editing. FIGS. 16A and 16B show an example of display. The display method is not limited to this, and other images such as a semi-transparent image of an object that has not been extracted are displayed. You may carry out by the method. If there is a subject that the user wants to specify among the extracted subjects, image processing is performed after the target subject is selected by a user operation. On the other hand, if the extracted subject does not include the subject that the user wants to specify, the user interface is configured so that the user can switch the desired display form. For example, the user can arbitrarily switch the display screen of FIG. 16A or FIG. 16B regardless of the type of display device. When the user performs a display switching operation, for example, another subject image can be extracted by executing S1307 in FIG. 13 again on the subject image other than the subject extracted first.

  In the subject extraction process, the upper limit subject number M may be determined according to the display resolution (or screen size) of the display device connected to the imaging apparatus. For example, the lower the display resolution, the smaller the upper limit subject number M is set for the display device. Even when the display resolution is low, the user can accurately select a desired subject.

[Other Examples]
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

DESCRIPTION OF SYMBOLS 101 Image pick-up part 103 System control part 104 Distance information acquisition part 107 Image processing part 108 Divided image generation part 109 Operation part 114 Subject detection part 115 Object image generation part 116,1216 Personal authentication part 117,1217 Focus degree calculation part 118 Divided image Initial selection decision unit 1218 Subject extraction unit

Claims (17)

An acquisition unit configured to acquire a captured image and depth information indicating a depth of each subject in the captured image;
Generating means for generating image data of a plurality of images each corresponding to a range in a different depth direction from the data of the captured image using the depth information;
Selecting means for selecting any of the plurality of images;
Control means for outputting the image data generated by the generation means to a display device and setting one of the plurality of images to an initial selection state selectable by the selection means in an initial display;
An image processing apparatus comprising: an image processing unit that performs image processing on the image selected by the selection unit. Detecting means for detecting the attribute and position of the subject in the captured image;
The image processing apparatus according to claim 1, further comprising: an authentication unit that authenticates an individual from the face image of the subject detected by the detection unit. The control unit sets the image including the face image of the subject authenticated by the authentication unit to the initial selection state with respect to the subject region in the image generated by the generation unit. The image processing apparatus described. Detecting means for detecting the attribute and position of the subject in the captured image;
Calculating means for calculating the degree of focus of the image of the subject detected by the detecting means;
The control means includes a first image whose degree of focus calculated by the calculation means is a first focus degree on a subject existing in the image generated by the generation means, and the first focus. The image processing apparatus according to claim 1, wherein the second image is set to the initial selection state among the second images having a second in-focus degree higher than the in-focus degree. The initial selection state is a state in which one of the plurality of images displayed for each different layer corresponding to the depth information by the display device can be selected by the selection unit. The image processing apparatus according to claim 1. An acquisition unit configured to acquire a captured image and depth information indicating a depth of each subject in the captured image;
Detecting means for detecting the attribute and position of the subject in the captured image;
Extracting means for classifying the objects detected by the detecting means to extract a subject area;
Generating means for generating image data of a plurality of images each corresponding to a range in a different depth direction from the data of the captured image using the information of the subject classified by the extracting means and the depth information; ,
Selecting means for selecting any of the plurality of images;
Control means for outputting the image data generated by the generating means to a display device and making any of the plurality of images selectable by the selecting means;
An image processing apparatus comprising: an image processing unit that performs image processing on the image selected by the selection unit. Comprising authentication means for authenticating an individual from the face image of the subject detected by the detection means;
The image processing apparatus according to claim 6, wherein the extraction unit preferentially extracts an image of the subject authenticated by the authentication unit. Calculating means for calculating the degree of focus of the image of the subject detected by the detecting means;
The image processing apparatus according to claim 6, wherein the extraction unit preferentially extracts a subject image having a high degree of focus from the plurality of subject images. The image processing apparatus according to claim 6, wherein the extraction unit extracts a subject area equal to or less than a predetermined number for each of the image data generated by the generation unit. The first generation unit that generates image data corresponding to the attribute of the subject and the second generation unit that generates image data of the plurality of images. 10. The image processing apparatus according to any one of items 9. The generation unit generates display image data for displaying the plurality of images on different layers corresponding to the depth information,
The image processing apparatus according to claim 1, wherein the control unit performs control to output the display image data to the display device. The control means outputs a first display image data generated by the generating means to a display device, and a second control outputs the second display image data generated by the generating means to the display device. 2 control,
The first display image data is image data arranged in the depth direction so that the plurality of images are displayed in the front in the order of images including a subject having near depth information.
The image processing apparatus according to claim 11, wherein the second display image data is image data in which an image of a detected subject is arranged according to depth information of the subject. The control means acquires information about the display device, performs the first control when the display resolution or screen size of the display device is equal to or greater than a threshold, and displays the display resolution or screen size of the display device. The image processing apparatus according to claim 12, wherein the second control is performed when the value is less than a threshold value.   An imaging apparatus comprising: the image processing apparatus according to claim 1; and an imaging unit. An acquisition step of acquiring a captured image and depth information indicating a depth in the depth direction of each subject in the captured image;
Using the depth information, a generation step in which a generation unit generates image data of a plurality of images corresponding to different ranges in the depth direction from the captured image data,
A control step, wherein the control means outputs the image data generated by the generation means to a display device, and sets one of the plurality of images to an initial selection state selectable by the selection means in an initial display; ,
An image processing unit comprising: an image processing step of performing image processing on an image selected by the selection unit. An acquisition step of acquiring a captured image and depth information indicating a depth in the depth direction of each subject in the captured image;
A detection step of detecting the attribute and position of the subject in the captured image by a detection means;
An extraction step in which the extraction means classifies the subject detected by the detection means to extract a subject area;
Using the subject information classified by the extracting unit and the depth information, the generating unit generates image data of a plurality of images corresponding to different ranges in the depth direction from the captured image data. Generation process;
A control step, wherein the control means outputs the image data generated by the generation means to a display device, and allows the selection means to select any of the plurality of images; and
An image processing unit comprising: an image processing step of performing image processing on an image selected by the selection unit. A program that causes a computer to function as each unit included in the image processing apparatus according to any one of claims 1 to 13.

Images