JP6245974B2 - Image processing apparatus, control method, and program - Google Patents

Description

  The present invention relates to an image processing apparatus, a control method, and a method for calculating a correlation between a plurality of signals in a program, and more particularly to phase difference calculation (so-called correlation calculation) between a plurality of parallax images.

  Some imaging devices such as a digital camera have an automatic focus adjustment function in order to focus on a specific subject. One of the methods by which the imaging apparatus detects the focus state in focus adjustment is two images obtained by photographing with different optical axes, or two or more images obtained with respect to light beams that have passed through different pupil regions. There is a so-called phase difference detection method for calculating a phase difference (Patent Documents 1 and 2). Specifically, the position of the image of the same subject included in the phase difference detection target image or signal is specified by the correlation calculation of the pair of images, and the focus state is detected based on the image shift amount.

JP-A-7-199052 JP 2007-11314 A

  However, the correlation calculation performed on a pair of images as shown in Patent Documents 1 and 2 described above tends to increase the amount of calculation and prolong the processing time required for focus detection. In addition, when there is a large amount of noise generated in a pair of images for which correlation calculation is performed, for example, when the luminance of the shooting environment is low, the reliability of the correlation calculation may be reduced due to noise, and a good focus detection result may not be obtained. there were.

  The present invention has been made in view of the above-described problems, and an object thereof is to provide an image processing apparatus, a control method, and a program for detecting a focus state at high speed with good accuracy or detecting a subject distance. To do.

In order to achieve the above object, an image processing apparatus of the present invention is characterized by having the following configuration. Specifically, the image processing apparatus acquires, from an image group obtained by shooting at the same time, an acquisition unit that acquires a plurality of images having a parallax only in a predetermined direction, and the acquisition unit Extraction means for extracting a pixel row that is a pixel row extending in a predetermined direction from each of the plurality of images and photographing the corresponding spatial position, and the plurality of pixel rows extracted by the extraction means are A generating unit that generates a two-dimensional image that is captured according to the positional relationship in a predetermined direction and a pixel that corresponds to the same subject included in the two-dimensional image generated by the generating unit. detecting means for detecting a straight line, have a, specifying means for specifying an object distance of the same subject from the slope of the detected straight line by the detecting means, the predetermined direction, you at least one image of the image group That is determined according to the distribution of the high-frequency component, characterized in Rukoto.

  With such a configuration, according to the present invention, it is possible to detect the focus state at high speed with good accuracy.

1 is a block diagram showing functional configurations of a digital camera 100 and a lens 200 included in a camera system according to an embodiment of the present invention. The figure for demonstrating the detailed structure of the imaging part 102 which concerns on embodiment of this invention. The figure for demonstrating the production | generation of the two-dimensional image which concerns on embodiment of this invention The figure for demonstrating the aspect for every focus state of the two-dimensional image which concerns on embodiment of this invention. The flowchart which illustrated the focus state detection process performed with the digital camera 100 which concerns on embodiment of this invention. The figure for demonstrating the straight line and inclination detection which are contained in the two-dimensional image which concerns on embodiment of this invention The figure for demonstrating the straight line and inclination detection which are contained in the two-dimensional image when perspective conflict exists based on embodiment of this invention The figure which showed the aspect of the system which concerns on the modification of this invention

[Embodiment]
Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings. Note that an embodiment described below is an example of an image processing apparatus that includes a camera system having an imaging device that can acquire a plurality of image signals obtained by imaging the same subject at different positions at the same time. An example to which the present invention is applied will be described. However, the present invention can be applied to any device capable of acquiring a plurality of image signals obtained by imaging the same subject at different positions at the same time.

  In this specification, the following terms are defined and described.

・ "LF Data (Light Field Data)"
An image signal output from the imaging unit 102 included in the digital camera 100 according to the present embodiment and obtained by applying image processing related to predetermined development. Each pixel of the LF data indicates a signal intensity corresponding to a light beam having a different combination of the divided pupil region and the incident direction of the imaging optical system 202 that has passed. The LF data is also called light irradiation field information or light space information.

・ "Reconstructed image"
An image generated from LF data and focused on an arbitrary focal plane position. Specifically, with respect to a plurality of (pupil division number) divided pupil images generated by pixels that have passed through the same divided pupil region from LF data, the alignment is performed so that the images of the subject existing at the generated focal length match. An image obtained by performing and summing (superimposing) pixel values of corresponding pixels.

《Camera system configuration》
FIG. 1 is a block diagram showing functional configurations of a digital camera 100 and a lens 200 included in a camera system according to an embodiment of the present invention.

<Configuration of digital camera 100>
The camera control unit 101 is a CPU, for example, and includes a ROM and a RAM (not shown). The camera control unit 101 reads out an operation program for each block of the digital camera 100 from the ROM, develops the program in the RAM, and executes the program, thereby controlling the operation of each block of the digital camera 100. In addition, the camera control unit 101 sends information on the focal position determined by analysis of the image signal output from the imaging unit 102 and information on the aperture value corresponding to the determined exposure setting to the lens 200 via the electrical contact 107. Send.

  The imaging unit 102 is an imaging element such as a CCD or a CMOS sensor. The imaging unit 102 performs exposure and readout of each photoelectric conversion element (pixel) included in the imaging device based on the timing signal generated by the camera control unit 101 based on the set exposure setting. Then, the imaging unit 102 outputs an analog image signal of the obtained LF data to the image processing unit 103. Specifically, the imaging unit 102 photoelectrically converts an optical image formed on the light receiving surface of the imaging element by the imaging optical system 202 of the lens 200, and outputs an analog image signal.

  Further, in the digital camera 100 of the present embodiment, a microlens array 108 in which microlenses 20 are arranged in a lattice pattern as shown in FIG. One microlens 20 of the microlens array 108 is associated with a plurality of photoelectric conversion elements (pixels) of the image sensor 109 as shown in FIG. The light beam that has entered through the imaging optical system 202 of the lens 200 is divided into pupils by being imaged by the microlenses 20 on the pixels of the associated image sensor 109. That is, a light beam that has passed through the divided pupil region of the imaging optical system 202 corresponding to the associated pixel position is imaged on each pixel of the associated image sensor 109. In the example of FIG. 2B, 5 × 5 = 25 pixels are associated with one microlens 20, and the number of pupil divisions of the imaging optical system 202 is 25.

  FIG. 2C is a diagram illustrating a correspondence relationship between pixels associated with one microlens and divided pupil regions of the exit pupil of the imaging optical system 202 through which a light beam formed on each pixel passes. . 2C, for the sake of simplicity, it is assumed that five pixels 21a to 21e in the horizontal direction with respect to one microlens 20 are arranged side by side on a horizontal line passing through the center of the microlens 20. . At this time, each pixel is designed by the microlens 20 so as to have a conjugate relationship with the divided pupil regions 31a to 31e on the exit pupil plane. In the example of FIG. 2C, the pixel 21a is the divided pupil region 31a, the pixel 21b is the divided pupil region 31b, the pixel 21c is the divided pupil region 31c, the pixel 21d is the divided pupil region 31d, and the pixel 21e is the divided pupil region 31e. It is in a conjugate relationship.

  As described above, since the LF data corresponds to the light fluxes through which the pixel groups associated with the respective microlenses 20 have passed through different divided pupil regions, the pixel values are summed up for each pixel group to constitute one pixel. Thus, an image equivalent to the case where the pupil is not divided can be generated. The reconstructed image generated at this time is an image focused on a subject existing at a distance corresponding to the focal length of the focus lens at the time of shooting.

  In addition, an image corresponding to the position of the divided pupil region is generated by extracting an image corresponding to the light beam that has passed through the same divided pupil region from the pixel group associated with each microlens 20 to form an image. You can also. That is, as shown in FIG. 2B, when the microlens 20 and the photoelectric conversion element are associated with each other, 25 types of images can be generated. At this time, the image corresponding to the position of each divided pupil region can be regarded as an image taken with respect to different optical axes, for example, in a relationship having parallax taken using a multi-view camera having an adjacent optical system. It can be said that it is equivalent to a certain image group. Therefore, in this embodiment, a method for detecting a focus state using LF data will be described. However, the present invention can also be applied to a group of images photographed using the above-described multi-lens camera. The distance of the subject appearing in the image can be detected.

  The image processing unit 103 performs various image processes including a focus detection process on the analog image signal of the LF data output from the imaging unit 102. Specifically, the image processing unit 103 performs A / D conversion processing, white balance adjustment processing, gamma correction processing, and the like on the input analog image signal in addition to focus detection processing. In the present embodiment, the image processing unit 103 also performs processing for generating a reconstructed image from the obtained LF data. Further, the image processing unit 103 performs compression processing on the LF data and reconstructed image data generated in each step described above according to a predetermined encoding method. Further, the image processing unit 103 records the LF data and reconstructed image data generated in each process described above in a memory or a recording medium according to a mode or a user instruction, or converts the data into a predetermined format and externally Output. Various image processes may be realized by a dedicated circuit or the like.

  The memory 104 includes a memory element and a processing circuit that reads from and writes to the memory element. The memory 104 outputs to the storage element and stores an image to be output to the display unit 105. The memory 104 stores encoded images, moving images, audio data, and the like.

  The display unit 105 is a display device provided in the digital camera 100 such as an LCD. The display unit 105 displays a reconstructed image or the like generated from LF data obtained by imaging.

  The operation detection unit 106 detects an operation performed on a user interface such as a release button of the digital camera 100. Specifically, when the operation detection unit 106 detects that the user has operated, for example, a release button, the operation detection unit 106 outputs a control signal corresponding to the operation to the camera control unit 101.

<Configuration of lens 200>
The lens control unit 201 is a CPU, for example, and incorporates a ROM and a RAM (not shown). The lens control unit 201 controls the operation of each block by reading the operation program of each block included in the lens 200 stored in the ROM, developing the program in the RAM, and executing the program. When the lens control unit 201 receives information on the focal position and the aperture value from the camera control unit 101 via the electrical contact 107, the lens control unit 201 transmits the information to the lens driving unit 203 and drives the corresponding optical member of the imaging optical system 202. Let

  The imaging optical system 202 includes a lens group, a diaphragm, and the like that the lens 200 has. In the present embodiment, the imaging optical system 202 includes at least a focus lens, a shift lens, and a diaphragm. The lens driving unit 203 performs drive control of the focus lens, the shift lens, and the diaphragm of the imaging optical system 202 according to the information input from the lens control unit 201. Note that a camera shake detection sensor (not shown) is connected to the lens control unit 201 of this embodiment, and the lens driving unit 203 drives the shift lens according to the output of the sensor input from the lens control unit 201.

<Overview of how to detect the focus state>
Hereinafter, an outline of a method for detecting a focus state performed by the image processing unit 103 of the digital camera 100 according to the present embodiment will be described.

  In the present embodiment, the image processing unit 103 detects a focus state by paying attention to a linear component such as an edge in a subject to be photographed. This is based on the fact that the edge portion contains a high-frequency component and the contrast changes sensitively to the focus state (blurring occurs), but in the implementation of the present invention, the object used for the evaluation of the focus state Is not limited to the edge portion. For example, the image processing unit 103 extracts a linear component in each of the vertical direction and the horizontal direction for one reconstructed image generated from LF data, and determines the direction in which the focus state is detected depending on which direction has more linear components. decide. For example, when there are many vertical line components in the image, the image processing unit 103 detects the focus state described below in the horizontal direction. For example, when there are many linear components in the horizontal direction in the image, the image processing unit 103 detects the focus state in the vertical direction. In other words, the image processing unit 103 identifies a direction in which a high frequency component appears in a subject (shooting range) that is a shooting target, and performs focus state detection processing in the direction. Therefore, the embodiment of the present invention is not limited to the vertical direction and the horizontal direction as described above. For example, when there are many linear components in the oblique direction in the shooting range, the focus state detection process is performed in the direction orthogonal to the direction. Just do it. In the present embodiment, the focus state detection process is performed in a direction orthogonal to the direction in which there are many linear components in the imaging range so that the focus state of more subjects (linear components) can be evaluated. Implementation of the invention is not limited to this. For example, when it is desired to detect only the focus state of the main subject, the direction in which the focus state detection process is performed may be determined based on the linear component in the region where the image of the main subject appears. Further, when it is desired to reduce the amount of calculation related to the focus state detection process, the focus state detection process may be performed in a direction in which there are many linear components. Of course, the direction in which the focus state detection process is performed may not be limited to one direction.

  When the direction (detection direction) in which the focus state detection process is performed in this way is determined, the image processing unit 103 extracts a signal sequence (pixel sequence) arranged in the detection direction from the LF data. For example, as shown in FIG. 2B, when 25 photoelectric conversion elements are assigned to one microlens 20, pixels that are outputs of five photoelectric conversion elements in the detection direction are arranged in the direction of the microlens array 108. Pixel rows connected by the number of microlenses 20 arranged are extracted. Specifically, for example, when the detection direction is the horizontal direction (x direction in FIG. 2), the light flux that has passed through the divided pupil regions 31a, 31b, 31c, 31d, and 31e in the pixel row as shown in FIG. Corresponding pixels appear repeatedly. In FIG. 3, a reference number assigned to a pixel is a lens number, and indicates a pixel associated with the microlens 20 in the x direction of the microlens array 108. The letters a through e attached to the reference numbers indicate which of the divided pupil areas 31a through e through which the light flux corresponding to the pixel has passed. In the following description, since the reference number and additional characters are used, the detection direction is assumed to be the horizontal direction.

  The image processing unit 103 rearranges the pixel rows in the detection direction extracted from the LF data in this way so that the pixels that are the outputs of the light beams that have passed through the same divided pupil region are arranged in a row, as shown in FIG. A dimensional image 300 is generated. In the two-dimensional image 300 of FIG. 3, pixels corresponding to the light beam that has passed through the microlens 20 are arranged in one direction in the order of the microlens 20 appearing in the detection direction in the microlens array 108. Also, in a direction orthogonal to this, pixels corresponding to the light beams that have passed through each microlens 20 are arranged so that the divided pupil regions through which the corresponding light beams have passed are in the order of the detection direction (and the opposite direction). That is, as shown in FIG. 3, in the two-dimensional image 300, pixels with different divided pupil regions through which the corresponding light flux has passed are arranged in the row direction and correspond to light beams that have passed through the same divided pupil region in the column direction. The pixels are arranged so that the microlenses 20 through which the luminous flux has passed are in the order of the detection direction.

  In the following description, the rightward direction of the two-dimensional image 300 in FIG. 3, that is, the direction of the divided pupil region a → e through which the corresponding light flux has passed is referred to as a divided direction. Also, in the vertical direction of the two-dimensional image 300, pixels are arranged so that the microlenses through which the corresponding light flux has passed are in the order of detection directions on the microlens array 108. In the following description, the detection direction of LF data In order to distinguish them from each other, the upward direction in FIG. 3 is referred to as a lens position direction. That is, in the two-dimensional image 300, pixels are arranged in the order of pupil division regions a, b, c, d, and e in the division direction, and lens numbers n, n−1,. Pixels are arranged in the order of 2 and 1. In the two-dimensional image 300 of the present embodiment, the positive direction of the lens position direction is opposite to the increase of the lens number, but this depends on the definition of the angle positive / negative direction in angle detection described later.

  The image processing unit 103 of the present embodiment detects the focus state using the two-dimensional image generated in this way. Here, how the two-dimensional image changes depending on the focus state will be further described with reference to FIG.

  When focused on a subject showing a linear edge extending in a direction orthogonal to the detection direction, the light beam corresponding to the reflected light of the subject passes through any path, and the focus of the focus lens It converges on the imaging surface located at a distance. For example, when a focused edge appears as an image having a width of one pixel in the detection direction in a reconstructed image generated from LF data after shooting, the light flux corresponding to the edge is detected on the imaging surface. It converges to one of the microlenses arranged in the direction. Then, since the light beam is separated and received by the photoelectric conversion elements arranged in the detection direction by the microlens, when a two-dimensional image is generated from the pixel row extracted from the LF data, as shown in FIG. The edges are aligned in the dividing direction. In FIG. 4B, in order to make the luminance difference due to the edge (contrast between a bright pixel and a dark pixel above a predetermined level) easy to understand, the bright pixel is shown in white and the dark pixel is shown in black. As shown in FIG. 4B, when the focus is on the edge, the luminance switching in the lens position direction appears as a line (straight line) parallel to the dividing direction in the two-dimensional image.

  On the other hand, when the focal length for focusing on the subject indicating the edge is shorter than the focal length of the focus lens at the time of shooting the LF data, the luminous flux corresponding to the reflected light of the subject is on the focal plane in front of the imaging plane. It will converge. For example, as shown in FIG. 4D, a focal plane 400 corresponding to a focal length focused on a subject indicating an edge is a lens with respect to an imaging plane 410 that is a microlens array plane when LF data is captured. Consider the case where it is close to 200. At this time, the light flux corresponding to the edge converges on the focal plane 400, but then spreads further to the imaging plane 410. If the virtual microlens 401 is defined at the position of the focal plane 400 for easy understanding, the light beams separated by the virtual microlens 401 in the example of FIG. It will enter into 411 thru | or 415. That is, even if the light beam corresponding to the edge converges on the virtual microlens 401, the imaging surface 410 does not receive the light on the photoelectric conversion element associated with only one microlens. It appears as shown in FIG. That is, when the focus is not on the edge and the lens is in the so-called rear pin state, the luminance switching in the lens position direction is not a line in the division direction in the two-dimensional image, but the division direction and the angle (the declination angle is minus ( 0> θ> −π / 2)).

  Similarly, when the focal length for focusing on a subject indicating an edge is longer than the focal length of the focus lens at the time of photographing LF data, the light beam corresponding to the reflected light of the subject is a focal plane behind the imaging plane. Will converge to. For example, as shown in FIG. 4D, a focal plane 420 corresponding to a focal length focused on a subject indicating an edge is captured with respect to an imaging plane 410 that is a microlens array surface when LF data is captured. Consider the case where it exists on the element side. At this time, the light flux corresponding to the edge is converged on the focal plane 420 but not yet converged on the imaging plane 410. If the virtual microlens 421 is defined at the position of the focal plane 420 for easy understanding, the light beams separated by the virtual microlens 421 in the example of FIG. It will enter into 411 thru | or 415. That is, the light beam corresponding to the edge is received by the photoelectric conversion elements associated with the plurality of microlenses on the imaging surface 410 before the light flux corresponding to the edge converges on the virtual microlens 421. It appears as 4 (c). That is, when the focus is not on the edge and the lens is in the so-called front pin state, the luminance switching in the lens position direction in the two-dimensional image is not a line in the division direction, but the division direction and angle (a declination is positive in the polar coordinate system 0 <θ <π / 2)) appears as a line in the direction.

  Therefore, the image processing unit 103 according to the present embodiment generates a two-dimensional image using the pixel row in the detection direction extracted from the LF data, and displays straight lines (an array of pixels having the same attribute) that appear in the image by the pixels corresponding to the edges. ) To detect the focus state of the edge. That is, the image processing unit 103 can determine whether the edge is in focus when the LF data is captured, or whether the edge is in the front pin state or the back pin state based on the slope of the straight line appearing in the two-dimensional image. Further, since the inclination of the straight line appearing in the two-dimensional image indicates the degree of focusing, the image processing unit 103 can detect the focus state without performing the correlation calculation as long as the straight line can be detected. This is because the light beam that should converge when the distance between the imaging surface and the focal plane is increased is more dispersed (diffused) in the detection direction and enters the microlens.

<Focus state detection processing>
Hereinafter, specific processing of the focus state detection processing executed in the image processing unit 103 of the present embodiment will be described with reference to the flowchart of FIG. The focus state detection process will be described as being started when a shooting preparation instruction is received, for example, when the digital camera 100 is in the shooting mode and the release button is pressed halfway.

  In step S501, the image processing unit 103 determines a detection direction related to detection of a focus state. Specifically, the image processing unit 103 acquires LF data that is captured and output with the current setting of the focus lens, and generates a reconstructed image. Then, the image processing unit 103 grasps the amount of the linear component in the horizontal direction and the vertical direction, for example, by applying an appropriate filter to the generated reconstructed image, and determines the direction in which the amount of the linear component is small as the detection direction. To do. That is, the direction orthogonal to the direction in which the amount of the linear component is large is determined as the detection direction.

  In step S502, the image processing unit 103 extracts a pixel row in the detection direction from the LF data, and generates a two-dimensional image. As described above, a two-dimensional image is generated by dividing pixel rows arranged in the detection direction of LF data in units of microlenses and connecting them in a direction orthogonal to the detection direction. Note that the pixel column used for generating the two-dimensional image may be acquired from any position of the LF data.

  In step S503, the image processing unit 103 applies a process related to luminance correction to the two-dimensional image. This is not a problem when the relationship between the imaging optical system 202 and the imaging element 109 is ideally configured. However, in general, a lens frame is used for a light beam that passes through a pupil region with a high image height (distance from the optical axis). So-called vignetting occurs in which the luminance decreases due to a shielding material such as a diaphragm (excluding the stop). When vignetting occurs, the luminance decreases, and it is difficult to determine whether a plurality of pixels in the two-dimensional image are the same subject image. Therefore, the image processing unit 103 performs luminance correction. Reduces brightness reduction due to vignetting.

  In step S504, the image processing unit 103 applies, for example, a filter process that extracts pixels having a predetermined spatial frequency in order to extract pixels configured by the same edge image in the two-dimensional image. The filter is applied to the lens position direction in the two-dimensional image. For example, when the two-dimensional image shows a distribution of bright pixels and dark pixels as shown in FIG. 6A, a filter that extracts only the spatial frequency indicating the switching between the bright pixels and the dark pixels is applied, so that FIG. Only pixels (edges) indicating switching as in b) can be extracted.

  In step S <b> 505, the image processing unit 103 detects a straight line composed of pixels indicating a predetermined spatial frequency and the inclination of the straight line from the image after the filter application. Note that the straight line detected here does not indicate the straight line represented by the edge component in the reconstructed image, but is formed at different positions by the light beams that have passed through different divided pupil regions when configured as a two-dimensional image. A pixel at the edge indicates a straight line constituting the two-dimensional image. In addition, the slope of the straight line indicates how far away in the lens position direction a pixel showing the same edge image appears when moving one pixel in the dividing direction in a two-dimensional image.

  The processing of this step is due to the appearance of a light / dark pattern of the subject that is basically common for each divided pupil region in the vertical direction (corresponding to the pixels of the microlens in the detection direction) in the two-dimensional image. In other words, the position of the microlens on which the subject image is formed may differ depending on the focus state at the time of shooting and the divided pupil region that has passed, but if the subject image pattern that appears in the detection direction is viewed, The same is true for each of the areas as long as there is no change in the shielding relationship. Therefore, when the two-dimensional image of this embodiment is generated, the same pattern is arranged in the lens position direction for each divided pupil region, so even if the patterns are shifted from each other depending on the focus state at the time of shooting, the same edge is formed. A straight line appears in the two-dimensional image with the corresponding image.

<Linear and tilt detection>
Hereinafter, the linear component detection method performed in this step will be described in detail with reference to the drawings. In the present embodiment, the image processing unit 103 detects the detection of the linear component in the two-dimensional image using the Hough transform. The Hough transform is one of conversion methods used for image feature extraction, and can be used, for example, to detect a straight line that most closely matches the image feature.

  Since only the pixels corresponding to the edge including the high frequency component are extracted as shown in FIG. 6B by the filter processing performed in S504, the image processing unit 103 first selects the extracted pixels in order and performs straight lines. Perform detection. In the following description, the inclination of the straight line appearing in the two-dimensional image is considered based on the division direction, and has a positive inclination when the detected straight line is inclined counterclockwise from the division direction. If it is tilted in the clockwise direction, the slope is negative.

  Here, the straight line 600 appearing in the two-dimensional image will be exemplified by its straight line and inclination detection method. First, when focusing on the pixel 601 existing at the coordinates (division direction, lens position direction) = (a, 7) in FIG. 6B, the image processing unit 103 considers all straight lines passing through the pixel. Then, voting is performed on the coordinates in the lens position direction through which each straight line passes at the coordinates in a specific division direction. Specifically, the image processing unit 103 defines a straight line that passes through the pixel 601 while sequentially changing the inclination. For example, the lens position through which the straight line passes at the coordinates in the division direction c corresponding to the central divided pupil region. Vote for the direction coordinates. Thereby, it is possible to specify a distribution of straight lines that can pass through the pixel 601 at the coordinates in the dividing direction c. For example, when the slope of the straight line passing through the pixel 601 is 0, it passes (c, 9) in the coordinates in the dividing direction c, and when the slope is +1, the coordinates in the dividing direction c are (c, 5 ), And when the inclination is −1, it passes (c, 9) in the coordinates in the dividing direction c. Therefore, when the distribution of straight lines that can pass through the pixel 601 is shown in a distribution diagram as shown in FIG. 6C showing the relationship between the tilt and the coordinates in the lens position direction, (tilt, lens position direction) = (0, 7 ), (1, 5), and (-1, 9). Therefore, the image processing unit 103 votes for the bin (each square in the distribution) through which the straight line 611 has passed.

  Next, when focusing on the pixel 602 existing at the coordinates (b, 8) in FIG. 6B, the image processing unit 103 similarly applies all the straight lines passing through the pixel 602 in the coordinates in the division direction c. Voting is performed on the coordinates in the lens position direction through which the straight line passes. For example, when the slope of the straight line passing through the pixel 602 is 0, it passes (c, 8) in the coordinates in the dividing direction c, and when the slope is +1, it is (c, 7) in the coordinates in the dividing direction c. ), And when the inclination is −1, it passes (c, 9) in the coordinates in the dividing direction c. Accordingly, in the distribution diagram showing the relationship between the tilt and the coordinates in the lens position direction in FIG. 6C, the distribution of straight lines that can pass through the pixel 602 is shown as (tilt, lens position direction) = (0, 8). , (1, 7), (−1, 9), the straight line 612 is obtained. Accordingly, the image processing unit 103 similarly votes for the bin through which the straight line 612 has passed.

  For the pixels 603, 604, and 605 corresponding to the edges that appear as the straight line 600, straight lines 613, 614, and 615 are respectively defined in the distribution chart showing the relationship between the inclination and the coordinates in the lens position direction. Vote on the bin. Accordingly, as a result of the voting by the image processing unit 103 for all the pixels appearing as the straight line 600, the straight lines 611 to 615 overlap (tilt, lens position direction) = (− 1, 9) is the most votes (peak). This is because, among the straight lines having various inclinations passing through the pixels on the straight line 600, only the straight line having the inclination corresponding to the straight line 600 is a straight line passing through all pixels. Therefore, based on such a voting result, the image processing unit 103 can detect that the straight line 600 is a straight line having an inclination of −1 passing through (c, 9).

  The image processing unit 103 also votes in the same manner for the straight lines 620 and 630 in FIG. Then, from the most voted (tilt, lens position direction) = (− 1, 3), (−1, 12), a straight line with a slope of −1 passing through (c, 3), (c, 12), respectively. It can be detected that the straight line has a slope of −1.

After detecting the straight line and the inclination of the straight line in this way, the image processing unit 103 converts the information of the inclination of each straight line into a defocus amount of the image of the edge corresponding to each straight line and outputs it in S506, The focus state detection process is completed. When the defocus amount is output after being converted into a phase difference as used in a focus detection system based on a conventional phase difference, Δp = tan θ using a projection angle θ corresponding to the inclination
Can be calculated as Here, Δp indicates the phase difference of the image between adjacent divided pupil regions, and the unit is a pixel. In addition, the pixel has shown the pixel in the image produced | generated from LF data, and is different from the pixel pitch of an image pick-up element. The defocus amount is not limited to the configuration calculated from the inclination value based on such a predetermined conversion formula. For example, the straight line inclination and the defocus amount prepared in advance in the imaging optical system 202 are also provided. May be obtained by referring to a table indicating the relationship between

  Information on the defocus amount output by the focus state detection process is transmitted to, for example, the camera control unit 101, and the camera control unit 101 uses the information to focus on a specific subject corresponding to the edge. The lens driving amount is transmitted to the lens driving unit 203.

  As described above, in the digital camera 100 of the present embodiment, by using the LF data, it is possible to easily detect the focus state of the subject without performing a correlation calculation with a large calculation amount. In addition, when adopting a method for obtaining the defocus amount using voting in the Hough transform as shown in this embodiment, even if the edge is not properly captured due to the influence of noise at the time of capturing, it is stable. A suitable defocus amount can be acquired. In other words, this method can increase the number of samplings when detecting the slope of a straight line that appears in the two-dimensional image due to an edge, compared with the case of performing a correlation operation between two images. Therefore, even if there is a pixel whose inclination is erroneously detected due to the influence of noise, such inclination can be eliminated because the number of votes decreases, and as a result, it is determined by a pixel that is not affected by noise. In addition, a highly accurate defocus amount can be acquired.

《Example of application when there are multiple subjects in shielding relationship》
In the description of the focus state detection process described above, an example in which the subject to be detected is not shielded by another subject and the subject to be detected in the focus state in the depth direction of the shooting environment is one is described. The invention is not limited to this. Hereinafter, an example of processing when the present invention is applied will be described with reference to FIG. 7 when a plurality of subjects are present at different positions in the depth direction of the photographing environment (so-called perspective conflict) and shielding occurs.

  As shown in FIG. 7D, a situation is considered in which a subject 701, a subject 702, and a subject 703 (existing outside the illustrated range) are arranged in the depth direction viewed from the imaging optical system 202. At this time, as described above, for example, when the horizontal direction is set as the detection direction and the pixel row extracted from the LF data is deformed, a two-dimensional image as shown in FIG. 7A is obtained. In FIG. 7A, the pixels corresponding to the subject image are shown as bright pixels and dark pixels as in FIG. 6A, but the bright pixels are shown with different hatching according to the luminance of each subject. It was. The negative direction of the lens position direction (the direction in which the number of the microlens increases) corresponds to the direction from the right to the left of the imaging range shown in FIG. The object 702 arranged in the area corresponds to the pixel group 712 in FIG. Similarly, the subject 701 that appears to the left in FIG. 7D and appears next corresponds to the pixel group 711 in FIG. 7A, and is the leftmost pixel 701 arranged in FIG. 7D. A distant subject 703 corresponds to the pixel group 713 in FIG.

  When the above-described filter processing is applied to the two-dimensional image obtained in this way, only pixels (edges) indicating switching between bright pixels and dark pixels in the lens position direction are extracted as shown in FIG. 7B. . In FIG. 7B, the edge component of each subject appears, but it will be understood that the slopes of the straight lines composed of pixels corresponding to the same edge are different. In the example of FIG. 7B, LF data is captured in a state where the subject 702 is focused, and a pixel group 722 corresponding to the edge of the subject 702 appears as a straight line parallel to the dividing direction in the two-dimensional image. On the other hand, since the subject 701 is in a so-called rear pin state, the pixel groups 721a and b corresponding to the edge of the subject 701 appear as straight lines having a negative slope in the two-dimensional image. Since the subject 703 is in a so-called front pin state, the pixel groups 723a and b corresponding to the edge of the subject 703 appear as straight lines having a positive inclination in the two-dimensional image.

  The image processing unit 103 performs voting in the Hough transform while selecting each pixel of the pixel groups 721 to 723 thus detected, and generates a distribution diagram as illustrated in FIG. The inclination of the straight line corresponding to the group can be acquired in the same manner as in the case of a single subject. In FIG. 7C, the bins 731, 732, 733, 734, and 735 are the most frequently voted as a result of voting for each pixel group by the edges that appear in the two-dimensional image. The bins are hatched for the purpose of comparison with other bins. The shades of hatching depend on the fact that bins 731, 734, and 735 correspond to edges with low contrast of the subject.

  Therefore, since the image processing unit 103 can generate and output a defocus amount for each of the subjects included in the shooting range, the camera control unit 101 focuses on the subject to be focused based on the information. Thus, a control signal to the lens driving unit 203 may be output.

  Note that it may be determined, for example, as follows whether the subject corresponding to which of the detected straight lines is focused. As one method, a method of selecting the subject in the bin having the maximum value in FIG. In this case, the subject having the edge with the highest contrast can be focused. That is, in the example of FIG. 7C, the bin 732 or 733 is selected and focused on the subject 701.

  As another method, after selecting the bins with the highest voting value in FIG. 7C and adding in the lens position direction (converting to a value for each inclination), the inclination having the maximum value is obtained. A method of selecting is conceivable. In this way, it is possible to focus on a subject with high contrast. In the example of FIG. 7C, the inclination −1 obtained by adding the bin 732 and the bin 733 is selected, and the subject 701 is focused.

  As another method, after selecting the top few bins in FIG. 7C and adding them in the direction of the lens position (converting them to values for each inclination), the extreme value is obtained from the side where the subject is in front. A search method is conceivable. This is effective when it is desired to focus on the subject 701 in front. In the example of FIG. 7C, the angle +1 obtained by adding the bin 734 and the bin 735 is selected, and the farthest subject 703 is focused.

  In the present embodiment, a method for generating a two-dimensional image using a pixel column extracted from LF data and detecting a defocus amount of an image of a high-frequency component appearing in the pixel column has been described. It is not limited to the pixel column extracted from the LF data. As described above, the LF data is equivalent to recording a group of images with different optical axes formed by the light beams that have passed through each of the divided pupil regions. Therefore, the present invention can also be applied to an output image group of a multi-view camera that captures the same subject. That is, a pixel row extracted from each of the images having a parallax only in the determined detection direction in the output image group of the multi-view camera is developed in the lens position direction, and is combined with the positional relationship of the optical system. By arranging them in the dividing direction, the above-described two-dimensional image can be generated. At this time, the pixel row extracted from each image is an arrangement of pixels on the same straight line extending in the detection direction when each image is arranged in accordance with the positional relationship of the optical system. As a result, for the same subject, the amount of image shift between images taken by the optical system aligned in the detection direction can be grasped without performing a correlation operation, and the subject distance of the subject together with the positional relationship of the optical system Can be specified. Note that the method of grasping the correlation between images using the two-dimensional image described in the present embodiment is a two-dimensional image from three images having a parallax with respect to at least a straight line to be detected in order to maintain the reliability. Is preferably generated.

  As described above, the present invention generates a two-dimensional image using a pixel array acquired from a group of images having different optical axes and having a parallax, thereby enabling simple straight line detection without performing complex correlation calculation. By processing, the defocus amount of the edge or the subject distance can be specified. That is, the method of the present invention can handle more signals as a sample at a time than the conventional method of obtaining a correlation from a pair of signals (two signals having different parallaxes) by a correlation calculation, and therefore an error (according to a Gaussian distribution) For example, a more favorable result can be obtained for thermal noise). In other words, since the number of samples is large, the influence of the pixels can be reduced even if there are pixels affected by noise or the like, so that the calculation result is more robust than the conventional method. . Therefore, even when the shooting environment is dark and the S / N ratio of the sensor output is not good, a suitable focus state detection result can be obtained.

  As described above, the image processing apparatus according to the present embodiment can detect the focus state at high speed with good accuracy. Specifically, the image processing apparatus acquires a plurality of images having a parallax relationship in a predetermined direction from the image group, and is a pixel row extending in a predetermined direction from each of the plurality of images, and correspondingly A pixel row that captures the spatial position to be extracted is extracted. Then, a two-dimensional image in which each of the plurality of extracted pixels is photographed for each of the plurality of images and arranged according to the positional relationship in a predetermined direction is generated, and the same subject included in the two-dimensional image is generated. A straight line constituted by pixels is detected. The image processing apparatus specifies the subject distance of the same subject from the detected inclination of the straight line.

[Modification]
In the above-described embodiment, the example in which the present invention is applied to the digital camera 100 capable of photographing and outputting LF data has been described. However, the application of the present invention is not limited to this. For example, as shown in FIG. 8A, the present invention is applied to an information processing apparatus 800 such as a PC that can acquire LF data from the digital camera 100, generate a reconstructed image, and display the reconstructed image on the display apparatus 801. Applicable.

<< Display Mode of Display Device 801 >>
In this modification, the CPU of the information processing apparatus 800 generates a reconstructed image from the LF data acquired from the digital camera 100 and outputs the reconstructed image to the display device 801 for display. FIGS. 8B and 8C show images (windows) displayed on the display device 801, each showing an image focused on a subject present at a different position in the depth direction.

  When the CPU accepts the user's instruction input for the area on the reconstructed image when the reconstructed image is displayed on the display device 801, the CPU refocuses the reconstructed image on the subject appearing in the area. Is newly generated and displayed on the display device 801.

  For example, when shooting LF data, the user wanted to shoot with the subject 820 in focus, but unexpectedly focused on the subject 830 and the LF data was shot. Think about the case. At this time, when a reconstructed image corresponding to the focal length at the time of shooting is generated from the LF data, the image is as shown in FIG. 8B, and the image of the subject 820 is a low-quality image with low contrast. When such a reconstructed image is displayed on the display device 801, when the user performs an operation for designating an area where the image of the subject 820 is present, the CPU performs, for example, the process of the flowchart illustrated in FIG. To generate a reconstructed image focused on the subject 820. In the following description, it is assumed that the detection direction is the horizontal direction of the reconstructed image.

  In S841, the CPU reads a pixel column of an image including a designated area having a relationship having at least a parallax only in the detection direction from the LF data, and generates a two-dimensional image. Then, the CPU performs filter processing, straight line detection processing, and angle detection processing on the two-dimensional image, and acquires information on the inclination of the straight line by the image corresponding to the subject 820. Further, the CPU specifies the focal distance at which the correlation is maximum, that is, the focal distance position at which the inclination becomes 0 (a line parallel to the dividing direction) from the inclination information.

  In S842, the CPU generates a reconstructed image from the LF data for the surface corresponding to the identified focal length, and transmits the image generated in S843 to the display device 801 for display.

  By doing so, the reconstructed image as shown in FIG. 8C focused on the subject 820 is displayed on the display device 801. As described above, according to the method of the present invention, correlation between images such as the amount of defocus can be acquired with simple processing without performing complicated correlation calculation.

[Other Embodiments]
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (11)

Acquisition means for acquiring a plurality of images having a parallax in a predetermined direction from a group of images obtained by photographing;
An extraction means for extracting a pixel row extending in the predetermined direction and capturing a corresponding spatial position from each of the plurality of images obtained by the obtaining portion;
Generating means for generating a two-dimensional image in which the plurality of pixel columns extracted by the extracting means are taken according to the positional relationship in the predetermined direction, each of which is taken of the plurality of images;
Detecting means for detecting a straight line composed of pixels corresponding to the same subject included in the two-dimensional image generated by the generating means;
Have a, specifying means for specifying an object distance of the same subject from the slope of the detected straight line by the detection means,
The predetermined direction, the image processing apparatus according to claim Rukoto determined according to the distribution of the high frequency component in at least one of an image of the image group. An image processing apparatus that generates and outputs an image focused on a predetermined subject from a plurality of images having a parallax in a predetermined direction obtained by photographing,
Obtaining means for obtaining the plurality of images from an image group;
An extraction means for extracting a pixel row extending in the predetermined direction and capturing a corresponding spatial position from each of the plurality of images obtained by the obtaining portion;
Generating means for generating a two-dimensional image in which the plurality of pixel columns extracted by the extracting means are taken according to the positional relationship in the predetermined direction, each of which is taken of the plurality of images;
Detecting means for detecting a straight line composed of pixels corresponding to the same subject included in the two-dimensional image generated by the generating means;
Output means for generating and outputting an image focused on the same subject by moving the positions of the plurality of images in the predetermined direction according to the inclination of the straight line detected by the detection means; An image processing apparatus comprising: Acquisition means for acquiring a plurality of images having a parallax in a predetermined direction from image groups respectively corresponding to pupil-divided light fluxes;
An extraction means for extracting a pixel row extending in the predetermined direction and capturing a corresponding spatial position from each of the plurality of images obtained by the obtaining portion;
Generating means for generating a two-dimensional image in which a plurality of the pixel columns extracted by the extracting means are connected according to a positional relationship in a pupil region of a light beam corresponding to each of the plurality of images;
Detecting means for detecting a straight line composed of pixels corresponding to the same subject included in the two-dimensional image generated by the generating means;
Have a, specifying means for specifying a focus state of the same object at the time of shooting the images from the slope of the detected straight line by the detection means,
The predetermined direction, the image processing apparatus according to claim Rukoto determined according to the distribution of the high frequency component in at least one of an image of the image group. The image processing apparatus according to claim 2 , wherein the predetermined direction is determined according to a distribution of high-frequency components in at least one image of the image group.   The predetermined direction is determined in a direction orthogonal to a direction having the largest number of linear components appearing in at least one of the images in the image group. The image processing apparatus according to item.   The said detection means extracts the pixel which shows the brightness | luminance difference more than the predetermined level which appears in the direction of the said several pixel row | line | column with respect to the said two-dimensional image, and detects a straight line based on this pixel. The image processing apparatus according to any one of 1 to 5.   The image processing apparatus according to claim 1, wherein the detection unit detects a straight line by a Hough transform of the two-dimensional image. An acquisition step in which an acquisition unit of the image processing apparatus acquires a plurality of images having a parallax in a predetermined direction from an image group obtained by photographing;
An extraction step in which the extraction unit of the image processing apparatus extracts a pixel row extending in the predetermined direction and capturing a corresponding spatial position from each of the plurality of images acquired in the acquisition step. When,
The generation unit of the image processing apparatus generates a two-dimensional image in which the plurality of pixel columns extracted in the extraction step are arranged according to the positional relationship in the predetermined direction, in which each of the plurality of images is captured. Generating process to
A detecting step in which the detecting means of the image processing apparatus detects a straight line constituted by pixels corresponding to the same subject included in the two-dimensional image generated in the generating step;
Specifying means of the image processing apparatus, have a, a specifying step of specifying an object distance of the same subject from the slope of the detected straight lines in the detection step,
The predetermined direction is determined according to the distribution of the high frequency component in at least one of an image of the image group control method for an image processing apparatus according to claim Rukoto. An image processing apparatus that generates and outputs an image focused on a predetermined subject from a plurality of images having a parallax in a predetermined direction obtained by photographing,
An acquisition step in which the acquisition unit of the image processing apparatus acquires the plurality of images from an image group;
An extraction step in which the extraction unit of the image processing apparatus extracts a pixel row extending in the predetermined direction and capturing a corresponding spatial position from each of the plurality of images acquired in the acquisition step. When,
The generation unit of the image processing apparatus generates a two-dimensional image in which the plurality of pixel columns extracted in the extraction step are arranged according to the positional relationship in the predetermined direction, in which each of the plurality of images is captured. Generating process to
A detecting step in which the detecting means of the image processing apparatus detects a straight line constituted by pixels corresponding to the same subject included in the two-dimensional image generated in the generating step;
The output unit of the image processing device generates an image focused on the same subject by moving the positions of the plurality of images in the predetermined direction according to the inclination of the straight line detected in the detection step. And an output process for outputting the image processing apparatus. An obtaining step in which an extraction unit of the image processing apparatus obtains a plurality of images having a parallax in a predetermined direction from an image group corresponding to each of the pupil-divided light beams;
An extraction step in which the extraction unit of the image processing apparatus extracts a pixel row extending in the predetermined direction and capturing a corresponding spatial position from each of the plurality of images acquired in the acquisition step. When,
The generation unit of the image processing apparatus generates a two-dimensional image obtained by connecting the plurality of pixel columns extracted in the extraction step according to a positional relationship in a pupil region of a light beam corresponding to each of the plurality of images. Generation process;
A detecting step in which the detecting means of the image processing apparatus detects a straight line constituted by pixels corresponding to the same subject included in the two-dimensional image generated in the generating step;
Specifying means of the image processing apparatus, have a, a specifying step of specifying a focus state of the same object at the time of shooting the images from the slope of the detected straight lines in the detection step,
The predetermined direction is determined according to the distribution of the high frequency component in at least one of an image of the image group control method for an image processing apparatus according to claim Rukoto.   The program for functioning a computer as each means of the image processing apparatus of any one of Claims 1 thru | or 7.