JP6300547B2 - Imaging device - Google Patents

Description

The present invention relates to an imaging apparatus .

  When image data obtained by photographing a landscape is reviewed at a later date by an image pickup apparatus (for example, a digital camera) that acquires an image using an image pickup device such as a CCD or CMOS, the atmosphere at the time of shooting may differ. One of the factors is that the sense of saturation, vividness, and depth differ due to water vapor and dust in the atmosphere, as compared with the case where the photographed image is viewed with the naked eye. CCD is an abbreviation for Charge Coupled Device. CMOS is an abbreviation for Complementary Metal Oxide Semiconductor.

  Here, in order to improve the feeling of depth, it is conceivable to estimate the depth and change the image processing in accordance with the estimated depth. Patent Document 1 discloses a pseudo stereoscopic image generation device that generates depth estimation data for generating a pseudo stereoscopic moving image.

JP 2006-185033 A

  The pseudo-stereoscopic image generation device disclosed in Patent Literature 1 generates depth estimation data using three types of basic depth models indicating depth information. However, with this pseudo-stereoscopic image generation device, it is possible to obtain distance information (far distance information) to a distant view portion in the image, but it is not possible to obtain distance information (short distance information) of a foreground portion of the image. Therefore, in this pseudo-stereoscopic image generation device, an unnatural image is generated in the foreground portion.

  An object of the present invention is to provide an imaging apparatus that creates accurate distance information about a captured image and executes image processing according to the created distance information.

An imaging apparatus according to an embodiment of the present invention includes a first distance that generates first distance information related to a distance between a photographing unit that photographs a subject and outputs an image signal, and a subject in a pixel of an image related to the image signal. A creation means; and a composition pattern storage means in which a composition pattern having distance information for each pixel or a plurality of pixels is stored, and depth information creation means for creating depth information related to the depth of the image , The depth information creation means extracts a feature amount for a predetermined feature amount extraction region of the image, and among the feature amount extraction regions, the feature amount extraction region for which the first distance information is created Determining the feature amount extraction region that needs to compensate for the missing part, estimating the missing part from the surrounding image and supplementing the missing part to extract the feature amount, based on the extracted feature amount, Serial composition pattern storage means of the stored composition patterns, said determining the corresponding composition pattern to the image, using the composition pattern described above determined to create the depth information, the depth information and the first distance information based on the bets comprises a second distance generating means for generating a second distance information about the distance to the object, and an image processing means for processing the image based on the second distance information.

  According to the present invention, distance information from a short distance to a long distance can be created, and human memory colors can be reproduced and a sense of perspective can be improved by performing image processing according to the distance.

It is a figure which shows the structural example of an imaging device. It is a figure explaining the structure of the pixel of an image sensor. It is a top view explaining arrangement | positioning of the pixel of an image pick-up element, and a photoelectric conversion part. It is a figure explaining the image formation state of an object image. It is a figure explaining short distance information. It is a figure which shows an example of a composition pattern. It is a figure which shows an example of a composition pattern. It is a figure which shows an example of a composition pattern. It is a flowchart explaining the operation | movement process of the imaging device at the time of imaging | photography. It is a flowchart explaining the creation process of depth information. It is a figure explaining the feature-value extraction area | region. It is a flowchart explaining the creation process of depth information. It is a figure which shows the image data by which the feature point extraction area | region was interpolated. It is a figure which shows the structure of an imaging device. It is a flowchart explaining the creation process of depth information. It is a flowchart explaining the creation process of depth information.

Example 1
FIG. 1 is a diagram illustrating a configuration of the imaging apparatus according to the first embodiment.
The imaging apparatus of the present embodiment is a digital camera. The application range of the present invention is not limited to a digital camera. The present invention can be applied to any imaging apparatus such as a digital video camera and a camera-equipped mobile phone.

FIG. 1 is a block diagram illustrating the configuration of the digital camera according to the first embodiment.
The imaging element 100 is a light receiving element such as a CCD or CMOS sensor that converts a received subject image into an electrical signal and outputs an image signal. The A / D converter 101 converts the analog output signal of the image sensor 100 into a digital signal. The short distance information creating unit 102 creates short distance information from the imaging device to the subject (first distance creating means). The short distance information is first distance information related to the distance between the captured image, that is, the pixel of the image related to the image signal output from the image sensor 100 and the subject. Specifically, the short distance information is distance information to the near view portion of the image. Details of the short distance information creation processing will be described later.

  The composition pattern 103 is data having distance information for each rectangular block composed of each pixel or a plurality of pixels, and is held in the memory 111 or the non-volatile memory 113 functioning as a composition pattern storage unit. The depth information creation unit 104 creates depth information using the composition pattern. The depth information is distance information to a distant view portion of the photographed image. Details of the composition pattern will be described later.

  The depth information creation unit 104 creates depth information. Specifically, the depth information creation unit 104 creates depth information for a location where the short distance information creation unit 102 could not create the short distance information. Details of the depth information creation processing will be described later.

  The distance information creation unit 105 merges the depth information created by the depth information creation unit 104 with respect to the location where the short distance information has not been created, for each rectangular block including each pixel or a plurality of pixels in the image data. Create distance information. That is, the distance information creation unit 105 functions as a second distance creation unit that creates second distance information related to the distance to the subject based on the short distance information and the depth information.

  The image processing unit 106 executes image processing such as filter processing, contrast, brightness, and color change for each image area according to the distance indicated by the distance information created by the distance information creation unit 105. For this purpose, the image processing unit 106 includes a plurality of processing units that execute image processing such as white balance, black level correction, shading correction, scratch correction, noise reduction, gamma correction, reduction / enlargement, and distortion correction. The image processing unit 106 includes an image buffer memory (not shown), a compression / decompression unit that performs compression processing and decompression processing of image data, a display control unit that displays on a monitor, a storage processing unit that stores data in a recording medium, and the like.

  The data transfer unit 107 includes WRDMAC and RDDMAC of a plurality of Direct Memory Access controllers (not shown) that perform data transfer. The image data is output to the bus 109 by WRDMAC and temporarily stored in the memory 111 via the memory controller 110. The image data temporarily stored in the memory 111 is output from the memory 111 to the bus 109 by the RDDMAC and output to the image processing unit 106.

  A bus 108 is a system bus, and a bus 109 is an image data bus. In this embodiment, the system bus and the image data bus are separated, but the same bus may be used.

  The memory controller 110 writes data to the memory 111 or reads data from the memory 111 in response to an instruction from the CPU 114 or the data transfer unit 107 which is a system control unit. Note that output data from the A / D converter 101 may be directly written into the memory 111. The memory 111 is a storage device having a storage capacity sufficient to store a predetermined number of still images, moving images for a predetermined time, data such as sound, constants for operating the system control unit 114, programs, and the like.

  The non-volatile memory controller 112 writes data to the non-volatile memory 113 or reads data from the non-volatile memory 113 in accordance with instructions from the system control unit 114. The nonvolatile memory 113 is an electrically erasable / recordable memory. As the nonvolatile memory 113, for example, an EEPROM or the like is used. The nonvolatile memory 113 stores constants, programs, and the like for operating the system control unit 114.

  The system control unit 114 includes a microcomputer that controls the operation of the digital camera. The system control unit 114 gives various instructions to each processing unit constituting the digital camera, and executes various control processes. The system control unit 114 controls the short distance information creation unit 102, the depth information creation unit 104, and the distance information creation unit 105 connected via the bus 108. In addition, the system control unit 114 controls the image processing unit 106, the data transfer unit 107, the memory controller 110, the nonvolatile memory controller 112, the operation unit 115, and the image sensor 100. The microcomputer included in the system control unit 114 executes a program recorded in the nonvolatile memory 113.

  The operation unit 115 includes a switch (SW), a button, and the like operated by the user. The operation unit 115 is used for operations such as power ON / OFF and shutter ON / OFF. In the present embodiment, when the SW is half-pressed, the system control unit 114 executes AF control. When the SW is fully pressed, the system control unit 114 executes a depth information acquisition process.

  The subject recognizing unit 116 divides the region of the image data according to the local feature amount of the image data, and recognizes the subject based on the feature amount for each region. The subject recognition unit 116 groups subjects recognized for each region. The subject recognition unit 116 notifies the depth information creation unit 104 as subject recognition information for each region, for example, mountains, sky, sea, buildings, people, trees, animals, etc. as groups.

  Next, short distance information creation processing will be described. The short distance information creation unit 102 calculates the distance from the subject in each pixel by a phase difference detection method using an image sensor having a plurality of photoelectric conversion regions. That is, the short distance information creation unit 102 calculates the short distance information by the phase difference method based on the amount of deviation of the image formed by the light flux that has passed through different pupil regions of the photographing optical system.

FIG. 2 is a diagram illustrating a pixel structure of the image sensor.
The image sensor 100 has a pixel structure shown in FIG. FIG. 2A is a front view of a pixel. FIG. 2B is a cross-sectional view of the pixel. One pixel includes a microlens 201 and photoelectric conversion units 202 and 203. As described above, pupil division is performed by arranging two photoelectric conversion units in one pixel. The left and right pixels are referred to as A pixel and B pixel, and the images related to the signals from the left and right pixels are respectively referred to as A image and B image.

  FIG. 3 is a plan view for explaining the arrangement of the pixels of the image sensor, the photoelectric conversion unit, and the color filter. A circle in each pixel represents a microlens. Here, only 4 columns × 4 columns are extracted and shown. This image pickup element forms a general Bayer array by regularly arranging color filters of R (red), G (green), and B (blue) on each pixel. One pixel has two photoelectric conversion units that output image signals corresponding to the A image and the B image, respectively. For example, in the example illustrated in FIG. 3, the upper left G pixel includes G1 that is the left photoelectric conversion unit 300 and G2 that is the right photoelectric conversion unit 301. Therefore, the two photoelectric conversion units in each pixel in FIG. 3 are photoelectrically converted through different exit pupils by pupil division. In the imaging device, R (red), G (green), and B (blue) are divided into left and right parts for each pixel, but a dedicated pixel for focus detection of a colorless filter may be arranged.

FIG. 4 is a diagram for explaining the imaging state of the object image.
With reference to FIG. 4, a method of calculating the subject and distance using the image sensor will be described. Reference numeral 400 denotes a lens, and 401 denotes a cross section of the image sensor. In the optical system using the image pickup device as shown in FIG. 3, for example, when an object image is formed in front of the image pickup device as shown in FIG. 4, the upper side of the exit pupil shown by A in FIG. The half-light flux passing through is shifted downward as A ′ on the image sensor. Further, the half light flux passing through the lower side of the exit pupil indicated by B shifts upward as shown by B ′.

That is, a pair of image signals formed by a light beam that passes through half of the pupil of the imaging optical system in each pixel has a phase shifted in the vertical direction in FIG. 4 in accordance with the imaging state of the object image. . Accordingly, the system control unit 114 calculates a defocus amount by obtaining a deviation amount, which is a relative positional relationship between two subject images, from the correlation thereof, and uses each information of the lens and the diaphragm to determine the distance from the subject. Is calculated. For example, if the defocus amount and direction are calculated by dividing the rectangular block of 8 pixels in the X (row) direction and 6 pixels in the Y (column) direction, the lens extension amount of the taking lens is determined. Assuming that the subject distance in the rectangular block is D, the taking-out amount of the photographing lens, and the focal length are P and f, respectively, approximately D≈f ^ 2 / P
The relationship is established. Based on this relationship, it is possible to obtain the short distance information of the captured image corresponding to the rectangular block.

FIG. 5 is a diagram for explaining the short distance information.
An image image 500 shows image data obtained by photographing a person, and a short distance information image 501 is obtained by adding short distance information to the image image 500. The value displayed in each rectangular block of the short distance information image 501 is a distance to the subject (unit: meters) simplified for convenience. However, a location where no value is displayed means infinity. Also, another calculation method may be used as the short distance information calculation method. Further, in this example, the short distance information is calculated using the imaging surface phase difference detection method, but the distance information may be obtained by a phase difference detection method, a triangular distance measurement, or the like using a separate distance measuring sensor. In this example, distance information is calculated for each rectangular block, but may be calculated for each pixel.

6 to 8 are diagrams illustrating examples of composition patterns.
6 in FIG. 6, 700 in FIG. 7, and 800 in FIG. 8 are image images obtained as a result of photographing a landscape. Composition patterns corresponding to the respective image images are composition patterns 601, 701, and 801.

  The composition pattern is divided for each rectangular block in which a plurality of pixels in the X (row) direction and Y (column) direction are collected. In the examples shown in FIGS. 6 to 8, the larger the numerical value described in the rectangular block, the farther the distance is. Specifically, the image 600 has a horizontal line at the center of the screen, a road extends from the bottom of the screen toward the center of the screen, a vanishing point near the center, and an image image diagram showing the sky above the center of the screen. It is.

  The composition pattern 601 indicates that the upper side of the screen is a distant view and the distance is uniformly distant, and the lower side of the screen indicates that the distance is different between the center and the left and right. The image 700 has a path extending from the bottom of the screen toward the center of the screen, and has a vanishing point near the center. Moreover, although it is empty above the center of the screen, there is a roadside tree on the side of the road, so that a part of the sky is hidden by the roadside tree. A composition pattern 701 shows a pattern whose distance increases toward the center of the screen. The image image 800 is an image image diagram having a horizontal line at the center of the screen, showing the sea below the horizontal line, and showing the sky above the horizontal line. In the composition pattern 801, the upper side of the screen is a distant view, the pattern is uniformly distant from the distance, and the lower side of the screen is a pattern that is uniformly far from the left and right.

FIG. 9 is a flowchart for explaining an operation process of the imaging apparatus at the time of shooting.
First, the system control unit 114 determines whether the SW of the operation unit 115 is half-pressed (step S900). If SW is not half pressed, the process returns to step S900. When the SW is half-pressed, the system control unit 114 starts shooting and takes in the image data from the image sensor 100 to each processing unit through the A / D converter 101. Then, the process proceeds to step 901.

  Next, the system control unit 114 drives the photographic lens to execute autofocus (AF) control (step S901). Specifically, the system control unit 114 calculates a defocus amount and direction by applying a phase difference detection method using an image plane or a dedicated distance sensor, and based on the calculated defocus amount and direction. The lens control unit (not shown) is controlled.

  Next, the system control unit 114 acquires short distance information (step S902). Specifically, the short distance information creation unit 102 creates the short distance information, and the system control unit 114 acquires the created short distance information. The system control unit 114 may control the memory controller 110 to temporarily record the short distance information in the memory 111.

  Next, the system control unit 114 determines whether the SW included in the operation unit 115 has been fully pressed (step S903). If SW is fully pressed, the process proceeds to step S904. If SW is not fully pressed, the process returns to step S901, and AF control is executed again. The AF control may be operated only once when the SW is half-pressed according to the camera mode.

  In step 904, the system control unit 114 acquires depth information (step S904). Specifically, the depth information creation unit 104 creates depth information, and the system control unit 114 acquires the created depth information. The system control unit 114 may temporarily record the depth information in the memory 111 by controlling the memory controller 110. Depth information creation processing will be described later with reference to FIGS. 10 and 11.

  Next, the distance information creation unit 105 creates distance information based on the short distance information and the depth information (step S905). The system control unit 114 may control the memory controller 110 to temporarily record the created distance information in the memory 111.

  Next, the system control unit 114 controls the image processing unit 106 to perform filter processing, contrast, brightness, and color for each region for each pixel or rectangular block according to the distance indicated by the distance information created in step 905. Image processing such as change is performed (step S906). Further, the system control unit 114 controls the image processing unit 106 to perform other image processing (white balance, black level correction, shading correction, scratch correction, noise reduction, gamma correction, reduction / enlargement, distortion correction, and the like). . Further, the system control unit 114 controls the image processing unit 106 to compress the image data, store it in a recording medium, display it on a monitor, etc., and complete photographing.

  FIG. 10 is a flowchart illustrating the depth information creation process according to the first embodiment. The system control unit 114 controls the depth information creation unit 104 to execute depth information creation processing.

  First, the system control unit 114 determines whether short distance information is present in all rectangular blocks of image data (step S1000). In step S1000, the system control unit 114 may determine whether all pixels of the image data have short distance information.

  When all the rectangular blocks of the image data have short distance information, it is not necessary to create depth information. Therefore, in this case, the process ends. When there is a rectangular block with no short distance information in the image data, it is necessary to create depth information for the rectangular block. Therefore, in this case, the process proceeds to step S1001.

  In step 1001, the system control unit 114 determines whether to extract a feature amount from the feature amount extraction region.

FIG. 11 is a diagram for explaining the feature amount extraction region.
The feature amount extraction region is a region from which a feature amount is extracted. FIG. 11 shows a feature amount extraction area in the image data 1100. The image data 1100 corresponds to the image data 500 in FIG. Areas A to I surrounded by a dotted line frame (feature quantity extraction area frame) in FIG. 11 are feature quantity extraction areas.

  If the system control unit 114 determines in the determination process in step S1001 of FIG. 10 that a feature amount is extracted from the feature amount extraction region, the process proceeds to step S1002. Then, the system control unit 114 extracts (calculates), for example, a color histogram and edge (contour) information as the feature amount, and the process proceeds to step S1003. If the system control unit 114 determines that no feature amount is extracted from the feature amount extraction region, the process proceeds to step S1003.

  Two specific examples of the determination process in step S1001 will be described. A first specific example of the determination process in step S1001 is as follows. The system control unit 114 determines whether there is short-distance information for the feature amount extraction region. When there is short distance information for the feature amount extraction region, the system control unit 114 determines that the feature amount is not calculated for the feature amount extraction region. When there is no short-distance information for the feature quantity extraction region, the system control unit 114 determines to calculate a feature quantity for the feature quantity extraction region.

  Assume that short-distance information for regions E, G, H, and I in FIG. 11 can be acquired, but short-distance information for regions A, B, C, D, and F cannot be acquired. In this case, for regions E, G, H, and I, it is determined in step S1001 in FIG. 10 that no feature value is extracted, and the process proceeds to step S1003. For regions A, B, C, D, and F, it is determined in step S1001 in FIG. 10 that feature amounts are extracted, and the process proceeds to step S1002.

  A second specific example of the determination process in step S1001 is as follows. The system control unit 114 controls the subject recognition unit 116 to execute subject recognition processing for recognizing the subject from the image data. The system control unit 114 determines whether the feature amount extraction region is a background region based on the subject recognition processing result. If the feature amount extraction region is not the background region, the system control unit 114 determines that the feature amount is not extracted. If the feature amount extraction region is a background region, the system control unit 114 determines to extract a feature amount.

  A subject is recognized from areas E and H in the image data 1100 shown in FIG. Therefore, since the areas E and H are not background areas, the system control unit 114 determines not to extract feature amounts. On the other hand, no subject is recognized from the regions A, B, C, D, F, G, and I. Therefore, since the areas A, B, C, D, F, G, and I are background areas, the system control unit 114 determines to extract feature amounts.

  In step S1003, the system control unit 114 determines whether or not all the feature amount extraction regions of the image data have been processed (steps S1001 and S1002) (step S1003). If processing has been performed for all the feature amount extraction regions of the image data, the processing proceeds to step S1004. If there is a feature amount extraction region that has not been processed, the process returns to step S1001.

  In step 1004, the system control unit 114 controls the depth information creation unit 104 to compare and analyze the feature amounts of the regions from which the feature amounts have been extracted from the regions A to I in FIG. Subsequently, the system control unit 114 controls the depth information creation unit 104 to determine a composition pattern to be used for creation of depth information based on the comparison / analysis processing result in step 1004 and the short-range information (step S1005).

  The determination of the composition pattern will be described below. A case is assumed in which it is determined whether or not to extract a feature amount based on the presence / absence of short-range information (S1001). Since areas A to C in FIG. 11 are empty, the system control unit 114 calculates the same histogram for these areas. Regions D and F include the sky and the ground, but substantially the same histogram is calculated. By comparing the calculated histograms, the system control unit 114 determines that the similarity between the left and right sides is high, but the similarity between the screen and the center of the screen is low.

  Further, the system control unit 114 detects a horizontal line in the image data 1100 based on the feature amounts (edge information) of the regions D and F. The E region has short-range information and there is a possibility that there is an object in the foreground, so it is not used for the horizontal line detection process.

  The system control unit 114 determines the composition pattern 601 in FIG. 6 or the composition in FIG. 8 based on the analysis result of the histogram that the similarity between the left and right is high and the similarity between the screen and the center of the screen is low, and the horizontal line detection processing result. The pattern 801 is used as a composition pattern candidate used for creating depth information. Furthermore, it can be seen that, as a narrowing-down condition, the short distance information in the center and the short distance information on the left and right differ depending on the short distance information. Therefore, the system control unit 114 determines the composition pattern 601 as a composition pattern used for creating depth information.

  On the other hand, a case is assumed in which it is determined whether to extract a feature amount (S1001) based on the subject recognition processing result. Since areas A to C in FIG. 11 are empty, the system control unit 114 calculates the same histogram for these areas. Similarly, regions G and I are roads and show substantially the same histogram. Regions D and F include the sky and the ground, but show substantially the same histogram.

  Further, since a person is shown in the foreground in the region E, the histogram of the region E is different from the histograms of the regions D and F. Similarly, since a person is shown in the foreground in the region H, the histogram of the region H is different from the histograms of the regions G and I. Further, since the regions A, D, and G are different, the histograms are different. The system control unit 114 compares the histograms as described above, and determines that the left and right similarities are high, but the upper and lower similarities are low.

  Further, the system control unit 114 detects a horizontal line in the image data 1100 based on the feature amounts (edge information) of the regions D and F. The system control unit 114 sets the composition pattern 601 in FIG. 6 or the composition pattern 801 in FIG. 8 to the depth based on the histogram comparison result that the left and right similarity is high and the vertical similarity is low and the horizontal line detection processing result. The composition pattern is used as a candidate for composition. Furthermore, it can be seen that, as a narrowing-down condition, the short distance information in the center and the short distance information on the left and right differ depending on the short distance information. Therefore, the system control unit 114 determines the composition pattern 601 as a composition pattern used for creating depth information.

  Returning to the description of FIG. In step 1006, the system control unit 114 controls the depth information creation unit 104 to create depth information from the composition pattern, and the depth information creation processing ends.

  According to the present embodiment, it is possible to create distance information from a short distance to a depth regarding a captured image. Therefore, it is possible to perform image processing according to distance, and it is possible to reproduce human memory colors and improve perspective.

(Example 2)
Example 2 will be described below. In the second embodiment, only portions different from the first embodiment will be described, and the same portions will be denoted by the same reference numerals and detailed description thereof will be omitted.

FIG. 12 is a flowchart illustrating the depth information creation process according to the second embodiment.
In step 1200, the system control unit 114 determines whether to interpolate the feature amount extraction region. If the system control unit 114 determines to interpolate with respect to the feature amount extraction region, the process proceeds to step S1002 . If the system control unit 114 determines not to interpolate the feature amount extraction region, the process proceeds to step S1201 .

  Two specific examples of the determination process in step S1200 will be described. A first specific example of the determination process in step S1200 is as follows. The system control unit 114 determines whether there is short-distance information for the feature amount extraction region. When there is short-distance information for the feature amount extraction region, the system control unit 114 determines to interpolate for the feature amount extraction region. When there is no short-distance information for the feature quantity extraction region, the system control unit 114 determines not to interpolate for the feature quantity extraction region.

  A second specific example of the determination process in step S1200 is as follows. The system control unit 114 controls the subject recognition unit 116 to execute subject recognition processing for recognizing the subject from the image data. The system control unit 114 determines whether the feature amount extraction region is a background region based on the subject recognition processing result. If the feature amount extraction region is not a background region, the system control unit 114 determines to interpolate with respect to the feature amount extraction region. For example, in the region E in FIG. 11, the background is hidden by the face of the person. That is, since the area E is not a background area, the system control unit 114 determines to interpolate with respect to the feature amount extraction area. When the feature amount extraction region is a background region, the system control unit 114 determines not to interpolate with respect to the feature amount extraction region.

  In step 1201, the system control unit 114 controls the depth information creation unit to interpolate from the surrounding pixels to interpolate the feature amount extraction region, and then extract the feature amount (step S1201). When the distance information of the peripheral pixel region is approximately the same distance as the feature amount extraction region, no interpolation is performed and no feature amount is extracted.

  FIG. 13 is a diagram illustrating image data in which a feature point extraction region is interpolated. Specifically, FIG. 13 shows the image data 1300 after the feature amount extraction region is interpolated with respect to the image data 1100.

  For the region E, the system control unit 114 first detects a horizontal line from surrounding pixels and interpolates the horizontal line. Further, the system control unit 114 interpolates from the sky region for pixels above the horizontal line, and interpolates from the road region for pixels below the horizontal line. The system control unit 114 interpolates the area H from the road area.

  In the second embodiment, a feature quantity is calculated after interpolating from surrounding pixels with respect to a feature quantity extraction area whose background is hidden, and the feature quantities are compared and analyzed with respect to a larger number of feature quantity extraction areas. As a result, the composition pattern selection accuracy can be improved.

(Example 3)
Example 3 will be described below. In the third embodiment, only portions different from the first embodiment will be described, and the same portions will be denoted by the same reference numerals and detailed description thereof will be omitted.

FIG. 14 is a diagram illustrating the configuration of the imaging apparatus according to the third embodiment.
The position information acquisition unit 1400 acquires position information at the time of shooting obtained by GPS or the like. The posture determination unit 1401 determines the posture of the imaging device obtained by a gyro or an inertial navigation sensor. The lens information acquisition unit 1402 acquires information such as zoom magnification and aperture value at the time of shooting.

  A communication unit 1403 performs communication by connecting an imaging device and another device 1405 (for example, a mobile phone or an external server). As communication standards, wireless communication typified by Wi-Fi, 3G (third generation mobile communication system), cellular phone communication typified by LTE (Long Term Evolution), etc. are used, but are not limited to these. Alternatively, another communication standard may be used.

The depth map 1404 is data having distance information for creating depth information for each rectangular block composed of each pixel or a plurality of pixels. In this example, the depth map 1404 holds distance information associated with each shooting location / direction. For example, the depth map 1404 holds distance information for the X direction at point A, the Y direction at point A, the X direction at point B, and the Y direction at point B. The depth map 1404 is held in the memory 111 or the nonvolatile memory 113.

FIG. 15 is a flowchart illustrating the depth information creation process according to the third embodiment.
First, the system control unit 114 controls the lens information acquisition unit 1402 to acquire lens information (step S1500). Subsequently, the system control unit 114 controls the position information acquisition unit 1400 to acquire position information (step S1501).

  Next, the system control unit 114 controls the posture determination unit 1401 to determine the posture of the imaging device (step S1502). Note that the order of processing from step S1500 to step S1502 is in no particular order.

Next, the system control unit 114 controls the depth information creation unit 104 to create shooting direction information from the position information and orientation information, and create field angle information from the lens information (step S1503). Then, the system control unit 114 controls the depth information creation unit 104 to identify the shooting location from the shooting direction information and the angle-of-view information created in step S1503. The system control unit 114 acquires distance information associated with the shooting direction information and the shooting location from the depth map 1404, and creates depth information based on the acquired distance information (step S1504).

Example 4
The imaging apparatus according to the fourth embodiment acquires depth information from the other apparatus 1405 by communicating with the other apparatus 1405.

FIG. 16 is a flowchart for describing depth information creation processing according to the fourth embodiment. The description of the same parts as those in FIG. 15 is omitted.
In step 1600, the system control unit 114 controls the depth information creation unit 104 to transmit shooting direction information and field angle information to the other device 1405 via the communication unit 1403. It is assumed that the other apparatus 1405 stores in advance a depth map that holds distance information associated with a shooting location and a shooting direction. The other device 1405 identifies the shooting location from the shooting direction information and the angle-of-view information transmitted from the imaging device. The other apparatus 1405 obtains distance information associated with the information on the shooting direction and the shooting location from the depth map, and creates depth information. Then, the other apparatus 1405 transmits the created depth information to the imaging apparatus. In the imaging device, the system control unit 114 receives (acquires) depth information transmitted from the other device 1405 via the communication unit 1403.

  The present invention has been specifically described above based on the preferred embodiments. However, the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the scope of the present invention. Needless to say.

(Other examples)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed. In this case, the program and the storage medium storing the program constitute the present invention.

102 Short distance information creation unit 104 Depth information creation unit 105 Distance information creation unit

Claims (12)

Photographing means for photographing a subject and outputting an image signal;
First distance creating means for creating first distance information related to a distance from a subject in a pixel of an image related to the image signal;
Depth information creating means comprising composition pattern storage means for storing a composition pattern having distance information for each pixel or a plurality of pixels, and creating depth information related to the depth of the image;
With
The depth information creating means includes
Extracting a feature amount for a predetermined feature amount extraction region of the image;
Among the feature amount extraction regions, for the feature amount extraction region in which the first distance information is created,
Judge as the feature extraction area that needs to compensate for the missing part,
Estimating the missing part from the surrounding image and supplementing the missing part to extract the feature amount,
Based on the extracted feature amount, among the composition patterns stored in the composition pattern storage means, a composition pattern corresponding to the image is determined,
Creating the depth information using the determined composition pattern;
Second distance creating means for creating second distance information related to the distance to the subject based on the first distance information and the depth information;
Image processing means for processing the image based on the second distance information ;
An imaging apparatus comprising: The first distance creating means creates the first distance information based on a shift amount between two images obtained by forming an image of a light beam that has passed through different pupil regions of the photographing optical system. The imaging device according to claim 1. Recognizing means for executing recognition processing of a subject from an image related to the image signal;
The depth information creation means needs to compensate for a defective portion in the feature quantity extraction area in which the first distance information is created among the feature quantity extraction areas based on the recognition processing result of the subject. Judge as a certain feature amount extraction area
The imaging apparatus according to claim 1 or 2, wherein The depth information creating means determines a composition pattern for the image using the extracted feature amount and the first distance information in the pixels of the image.
The image pickup apparatus according to claim 1, wherein the image pickup apparatus is an image pickup apparatus. Photographing means for photographing a subject and outputting an image signal;
First distance creating means for creating first distance information related to a distance from a subject in a pixel of an image related to the image signal;
Depth information creating means for creating depth information related to the depth of the image , comprising depth storage means for storing depth information associated with a shooting location and a shooting direction ;
With
The depth information creating means includes
Create angle-of-view information from lens information when shooting the subject,
Based on the depth information stored in the depth storage means, creates depth information corresponding to the angle of view information and the information of the shooting direction at the time of shooting ,
Second distance creating means for creating second distance information related to the distance to the subject based on the first distance information and the depth information;
Image processing means for processing the image based on the second distance information;
With
An imaging apparatus characterized by that. The first distance creating means creates the first distance information based on a deviation amount between two images obtained by forming an image of a light beam that has passed through different pupil regions of the photographing optical system.
The imaging apparatus according to claim 5. The depth information creation means creates the shooting direction information from position information and posture information at the time of shooting of the subject.
The imaging apparatus according to claim 5 or 6. The depth information creating means identifies a shooting location from the angle of view information and the shooting direction information, and acquires distance information associated with the shooting direction information and the shooting location based on a depth map. The depth information is created based on the acquired distance information
The image pickup apparatus according to claim 5, wherein the image pickup apparatus is an image pickup apparatus. Comprising communication means for communicating with an external device;
Photographing means for photographing a subject and outputting an image signal;
First distance creating means for creating first distance information related to a distance from a subject in a pixel of an image related to the image signal;
Depth information creating means for creating depth information related to the depth of the image;
With
The depth information creating means includes
Create angle-of-view information from lens information when shooting the subject,
By transmitting the angle of view information and the information of the shooting direction at the time of shooting to the external device via the communication unit, the angle of view information and the information of the shooting direction at the time of shooting are transmitted from the external device. Get the corresponding depth information,
Second distance creating means for creating second distance information related to the distance to the subject based on the first distance information and the depth information;
Image processing means for processing the image based on the second distance information;
With
An imaging apparatus characterized by that. The first distance creating means creates the first distance information based on a deviation amount between two images obtained by forming an image of a light beam that has passed through different pupil regions of the photographing optical system.
The imaging apparatus according to claim 9. The depth information creation means creates the shooting direction information from position information and posture information at the time of shooting of the subject.
The imaging apparatus according to claim 9 or 10, wherein The external device specifies a shooting location from the angle-of-view information and the shooting direction information, acquires distance information associated with the shooting direction information and the shooting location based on a depth map, and Create the depth information based on the acquired distance information
The image pickup apparatus according to claim 9, wherein the image pickup apparatus is an image pickup apparatus.

Images