JP6137941B2 - Image processing system, image processing method and program, and imaging apparatus - Google Patents

Description

  The present invention relates to an image processing system that processes an imaging signal imaged by an imaging device using a microlens array, and more particularly to an image processing system for correcting defective pixels of the imaging device.

  In recent years, there has been proposed a camera capable of acquiring imaging data so as to include information on the light traveling direction in addition to the light intensity distribution and reconstructing an image at an arbitrary focal position designated by the user. For example, Non-Patent Document 1 proposes a light field camera using a technique called “Light Field Photography”. In Non-Patent Document 1, a microlens array is arranged between a photographing lens and an image sensor, and light that has passed through different pupil regions of the photographing lens is configured to converge on each microlens. A plurality of pixels are assigned to one microlens, and light converged on the microlens is acquired by the plurality of pixels for each incident direction. An image at an arbitrary focal position can be obtained by collecting the signals of the pixels that are formed on the virtual image plane (refocus plane) and received by the light incident on each microlens from the captured image thus acquired. Can be reconfigured.

  By the way, in a solid-state imaging device such as a CCD image sensor or a CMOS image sensor, defective pixels may occur due to local crystal defects existing on a semiconductor substrate. Since a correct output cannot be obtained from the pixel signal of such a defective pixel, it is necessary to correct the pixel signal of the defective pixel. For example, in Patent Document 1, in the manufacturing process of an imaging device or an imaging element, an address of a defective pixel is recorded based on an image captured in advance and stored in a memory of the imaging device, and the defective pixel in the captured image is recorded according to the address. A method for correcting the signal is disclosed.

  In addition, as a method of correcting the pixel signal from the defective pixel, a method of replacing the pixel value of the defective pixel with a pixel value of a pixel adjacent to the defective pixel, or a method of replacing the pixel value of the defective pixel with a plurality of pixels adjacent to the defective pixel. A method of replacing with an average value of pixel values of pixels is common.

Ren.Ng and 7 others, “Light Field Photography with a Hand-Held Plenoptic Camera”, Stanford Tech Report CTSR 2005-02

JP 2005-26794 A

  In a light field camera as described in Non-Patent Document 1, a plurality of light beams that form an image on a refocus plane are separately received by a plurality of pixels. Further, as the position of the refocus plane is further away from the position of the microlens array, the plurality of light beams are received by the pixels at further away positions.

  Therefore, in the method of correcting the pixel value of the defective pixel using the pixel value of the adjacent pixel based on the address of the defective pixel acquired in advance as in Patent Document 1, a suitable correction is performed depending on the position of the refocus plane. There was a problem that I could not.

  Accordingly, an object of the present invention is to provide an image processing system capable of performing appropriate defect correction processing on imaging data acquired so as to include information on the traveling direction of light.

  In order to achieve the above object, according to the present invention, an image signal generated by an imaging unit having a pupil division unit of a photographic lens and a pixel array for imaging an optical image formed by the photographic lens and the pupil division unit is obtained. An image processing system to process includes a holding unit that holds defective pixel information about a defective pixel included in a pixel array, a setting unit that sets a refocus position for generating a reconstructed image, and a reconstructed image at a refocus position. A refocusing unit that generates using an image signal and a correcting unit that corrects the generated reconstructed image based on defective pixel information are provided. The correcting unit corresponds to the reconstructed image at the set refocus position. The reconstructed image generated using the defective pixel position information and output level information is corrected.

  According to the present invention, an image processing system capable of appropriately correcting defective pixels of an image sensor in an image processing system of a light field camera is provided.

1 is an overall block diagram of an imaging apparatus to which an image processing system according to a first embodiment of the present invention is applied. The figure for demonstrating the structure of the image pick-up element and micro lens array of the imaging device of 1st Example of this invention. The figure for demonstrating the structure of the imaging lens of the imaging device of 1st Example of this invention, a micro lens array, and an image pick-up element. The figure for demonstrating the correspondence of the pupil region of a taking lens, and a light reception pixel in the imaging device of 1st Example of this invention. The figure for demonstrating the correspondence of the pupil area | region of the imaging lens in the imaging device of 1st Example of this invention, the pixel on a refocus surface, and a micro lens. The figure for demonstrating the structure of the image process part which comprises the image processing system which concerns on 1st Example of this invention. The figure which shows the data structural example of the defective pixel information which concerns on 1st Example of this invention. The figure which shows the flowchart of the production | generation operation | movement of the defective pixel information based on 1st Example of this invention. The figure which shows the example of the crack level table of 1st Example of this invention. The figure which shows the example of conversion of the defective pixel information which concerns on 1st Example of this invention. The figure which shows the flowchart of imaging | photography operation | movement of the imaging device in 1st Example of this invention. The figure which shows the flowchart of the defect correction operation | movement of 1st Example of this invention. The figure which shows the flowchart of imaging | photography operation | movement of the imaging device to which the image processing system which concerns on 2nd Example of this invention is applied.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is an overall block diagram of an imaging apparatus to which an image processing system according to a first embodiment of the present invention is applied. In the figure, reference numeral 101 denotes a photographic lens composed of a plurality of lenses, which is an optical system that can adjust the focus state of a subject. The light that has passed through the photographing lens 101 forms an optical image near the focal position of the photographing lens 101. Reference numeral 102 denotes a microlens array (hereinafter referred to as MLA), which is composed of a plurality of microlenses 120. The MLA 102 is disposed in the vicinity of the focal position of the taking lens 101. Light that has passed through different pupil regions of the taking lens 101 enters the MLA 102 and is emitted separately for each pupil region (pupil dividing means).

  Reference numeral 103 denotes an image sensor, which picks up an optical image by receiving light emitted by being separated by the MLA 102. The image sensor 103 uses, for example, a CCD image sensor or a CMOS image sensor, and is arranged near the focal position of the MLA 102. A timing generator (TG) 112 generates various timings for driving the image sensor 103.

  An analog front end (AFE) 104 performs reference level adjustment (clamp processing), analog-digital conversion processing, and the like on the image signal from the image sensor 103. Reference numeral 105 denotes a digital front end (DFE), which performs digital correction processing such as a minute reference level shift on the digital output image from the AFE 104. An image processing unit 106 performs predetermined image processing on the digital image from the DFE 105 to generate and output image data. The image processing unit 106 also performs image reconstruction and defective pixel correction processing by refocus processing described later. The memory unit 107 is composed of a non-volatile memory for holding defective pixel information (address, scratch level) and the like. Reference numeral 108 denotes a control unit that comprehensively controls the entire imaging apparatus, incorporates a known CPU, and executes a program for controlling each unit. Reference numeral 109 denotes an operation unit such as a button or a touch panel, which electrically receives an operation member in the digital camera.

  The user can set an arbitrary refocus position for generating a refocus image using the operation unit 109. The control unit 108 may automatically set the refocus position. Reference numeral 110 denotes a display unit for displaying an image or the like, which includes a liquid crystal display or the like. Reference numeral 111 denotes a recording unit such as a memory card or hard disk.

  The image processing unit 106 according to the present exemplary embodiment includes a reconstruction unit 601 and a defect correction unit 602 as illustrated in FIG. The reconstructing unit 601 can reconstruct an image at a set arbitrary focal position (refocus position) from imaging data by performing arithmetic processing using a technique called “Light Field Photography”. is there.

  The defect correction unit 602 performs defect pixel correction processing on the image generated by the reconstruction unit 601 based on the defective pixel information recorded in the memory unit 107. Details of the reconstruction unit 601 and the defect correction unit 602 will be described later.

  Next, the configuration of the taking lens 101, the MLA 102, and the image sensor 103 in the image pickup apparatus of the present embodiment will be described with reference to FIGS. 1 to 4 are denoted by the same reference numerals.

  FIG. 2 is a diagram illustrating the configuration of the image sensor 103 and the MLA 102. This figure is a diagram of the image sensor 103 and the MLA 102 observed from the optical axis Z direction in FIG. One microlens 120 is arranged to correspond to a plurality of unit pixels 201. A plurality of unit pixels 201 behind one microlens are collectively defined as a pixel array 200. In this embodiment, it is assumed that the pixel array 200 includes a total of 25 unit pixels 201 of 5 rows and 5 columns.

  FIG. 3 is a diagram in which a state in which the light beam emitted from the photographing lens 101 passes through one micro lens 120 and is received by the image sensor 103 is observed from a direction perpendicular to the optical axis Z. Light emitted from each of the pupil regions a1 to a5 of the photographing lens 101 and passed through the microlens 120 forms an image at the corresponding unit pixels p1 to p5 at the rear.

  FIG. 4A is a view of the aperture of the photographing lens viewed from the optical axis Z direction. FIG. 4B is a view of one microlens 120 and the pixel array 200 disposed behind the microlens 120 as viewed from the optical axis Z direction. As shown in FIG. 4A, when the pupil region of the photographing lens 101 is divided into the same number of regions as the pixels under one microlens, one pixel includes light from one pupil division region of the photographing lens 101. Will be imaged. However, here, it is assumed that the F-numbers of the photographing lens and the microlens 120 are substantially the same. The correspondence between the pupil division areas a11 to a55 of the photographing lens 101 shown in FIG. 4A and the pixels p11 to p55 shown in FIG. 4B is point-symmetric when viewed from the optical axis Z direction. Therefore, the light emitted from the pupil division area a11 of the photographing lens forms an image on the pixel p11 in the pixel array 20 behind the microlens. Similarly, the light emitted from the pupil division area a11 and passing through another microlens forms an image on the pixel p11 in the pixel array 200 behind the microlens.

  Next, image reconstruction processing at an arbitrarily set focal position (refocus plane) with respect to imaging data acquired by the configuration of the photographing lens 101, the MLA 102, and the imaging element 103 will be described. The reconstruction process is performed in the reconstruction unit 601 illustrated in FIG. 6 using a technique called “Light Field Photography”.

  FIG. 5 shows, from the vertical direction of the optical axis Z, the light beam passing through a certain pixel on the refocus plane that is arbitrarily set, which pupil division region of the photographing lens 101 is emitted, and which microlens is incident. FIG. In the figure, the position of the pupil division area of the photographing lens 101 is coordinates (u, v), the pixel position on the refocus plane is coordinates (x, y), and the position of the microlens on the MLA 102 is coordinates (x ′, y). '). Further, the distance from the taking lens 101 to the MLA 102 is F, and the distance from the taking lens 101 to the refocus plane is αF. Here, α is a refocus coefficient for determining the position of the refocus plane, and can be arbitrarily set by the user. In FIG. 5, only the directions of u, x, and x ′ are shown, and v, y, and y ′ are omitted.

As shown in FIG. 5, the light beam 500 that has passed the coordinates (u, v) and the coordinates (x, y) reaches the coordinates (x ′, y ′) on the MLA. These coordinates (x ′, y ′) can be expressed as in equation (1).

When the output of the pixel that receives the light beam 500 is L (x ′, y ′, u, v), the output E (x, y) obtained with the coordinates (x, y) on the refocus plane is , L (x ′, y ′, u, v) is integrated with respect to the pupil region of the taking lens 101, and therefore, Equation (2) is obtained.

  In equation (2), since the refocus coefficient α is determined by the user, if (x, y) and (u, v) are given, the position (x ′, y ′) of the microlens where the light beam enters can be known. . Then, the pixel corresponding to the position (u, v) is known from the pixel array 200 corresponding to the microlens. The output of this pixel is L (x ', y', u, v). E (x, y) can be calculated by performing this for all pupil division regions and summing (integrating) the obtained pixel outputs. If (u, v) is the representative coordinates of the pupil division area of the photographing lens 101, the integral of equation (2) can be calculated by simple addition.

  As described above, an image at an arbitrary focal position can be reconstructed by performing the calculation process of Expression (2).

Next, defective pixel information recorded in the memory unit 107 will be described with reference to FIG.
FIG. 7 is a diagram illustrating an example of a data configuration of defective pixel information recorded in the memory unit 107. The X address and the Y address are position information indicating the coordinates of the defective pixel in the image reconstructed by the refocus processing. The scratch level is a value set according to the magnitude of the abnormal output signal from the defective pixel (output level information), and whether or not to correct the target defective pixel depending on the shooting conditions (ISO sensitivity, exposure time, etc.) Used for judgment. The refocus position indicates at which refocus position the defective pixel information is data, and corresponds to the refocus coefficient α described in FIG. In this embodiment, a data length of 14 bits is assigned to the X address and Y address, 4 bits are assigned to the scratch level, and 8 bits are assigned to the refocus position.

  Next, an operation of generating defective pixel information for recording in the memory unit 107 will be described with reference to a flowchart shown in FIG. The operation according to this flowchart is performed using an information processing apparatus such as a PC in factory adjustment of the imaging apparatus.

  In step S <b> 801, an address of a defective pixel on the image sensor is detected from an image acquired in an adjustment process or the like of the imaging device. Further, the scratch level is set according to the magnitude of the abnormal output signal from the defective pixel. The scratch level may be set according to the table shown in FIG. 9 according to the magnitude of the abnormal output signal, for example.

In S802, the address on the image sensor of the defective pixel acquired in S801 is converted into an address on the reconstructed image at a plurality of refocus positions. This conversion is performed by Expression (3) obtained by transforming Expression (1) with respect to coordinates (x, y) on the refocus plane.

  An example of defective pixel information converted by the equation (3) is shown in FIG. As shown in FIG. 10, the defective pixel information of the image sensor is converted into defective pixel information at each of a plurality of refocus positions, and information on the refocus position is also associated and added at the same time. Further, in the coordinate conversion of Expression (3), a plurality of defective pixels of the image sensor may have the same coordinates after conversion. The defective pixel information whose coordinates are duplicated after the coordinate conversion is collected into one. In FIG. 10, coordinates X3 and Y3 are defective pixel information combined into one. At this time, the defect level of each defective pixel is converted into a pixel output (for example, the minimum pixel output for each table) according to the table of FIG. Thereafter, the scratch level is reset according to the table of FIG. Note that the processing of S802 is performed by an external PC or the like in the factory adjustment process of the imaging apparatus, but may be performed by the image processing unit 106 in the imaging apparatus.

  Finally, in S803 of FIG. 8, the defective pixel information converted in S802 is recorded in the memory unit 117, and the defective pixel information generation operation is terminated.

  The shooting control operation of the imaging apparatus according to the present invention will be described with reference to the flowchart shown in FIG. This control operation is executed by the control unit 108 controlling each unit of the imaging apparatus.

  In FIG. 11, first, in step S <b> 1101, shooting conditions such as ISO sensitivity and exposure time are set according to the operation of the operation unit 109 by the user. Note that this setting may be automatically set by the control unit 108.

  In step S1102, the control unit 108 controls the TG 112 to supply a drive signal to the image sensor 103, and drives the image sensor 103 to perform shooting to generate image data.

  Next, in step S1103, an arbitrary refocus position (refocus coefficient α) is set by the operation unit 109 or the like, and in step S1104, the reconstruction unit 602 performs refocusing by the refocus processing described with reference to FIG. Generate a composition image. In step S1105, the defect correction unit 602 performs defect correction processing on the reconstructed image generated in step S1104. Details of this defect correction processing will be described later.

  Next, in step S1106, various signal processes such as white balance correction and gamma correction are performed on the reconstructed image that has been subjected to the defect correction process in step S1105. In step S1107, the recording unit 111 performs a final image. Record. The shooting control operation according to the flowchart shown in FIG.

  Next, the operation of defect correction processing by the defect correction unit 602 in step S1106 will be described using the flowchart shown in FIG. This operation is performed by the image processing unit 106 under the control of the control unit 108.

  In step S1201, defective pixel information is read from the memory unit 107 for each pixel. The defective pixel information read out here is defective pixel information on the reconstructed image shown in FIG.

  In step S1202, the refocus position information of the read out defective pixel information is referenced to determine whether or not the refocus position of the reconstructed image matches, and the result is retained. Next, in step S1203, the defect level of the read out defective pixel information is referred to, and it is determined whether or not a correction target level determined for each photographing condition (ISO sensitivity, exposure time, etc.) is satisfied, and the result is retained. To do.

  In step S1204, it is determined based on the determination results held in steps S1202 and S1203 whether the target pixel is a correction target. Specifically, the correction process in step S1205 is performed when the refocus position of the defective pixel information matches the refocus position of the reconstructed image and the scratch level satisfies the level to be corrected. In other cases, the correction process of step S1205 is not performed, and the process proceeds to step S1206.

  In step S1205, the image processing unit 106 performs correction processing on the correction target pixel. As an example of the correction process, in this embodiment, a method of interpolating a pixel value to be corrected from pixel values of the same color adjacent to the correction target pixel at the position of the defective pixel information at the top and bottom or the left and right. It is arbitrary whether to use the same color pixels adjacent vertically or horizontally, but may be adaptively determined depending on the subject, for example.

  In step S1206, it is determined whether or not the processing from steps S1201 to S1205 has been executed for all defective pixel information recorded in the memory unit 107. As a result of the determination, when this operation is executed for all recorded defective pixel information, the defect correction processing operation is terminated. If unprocessed defective pixel information remains, the unprocessed defective pixel information is read and the processing from step S1201 to S1205 is executed. Thus, the operation of the defect correction process in FIG. 12 is finished.

  As described above, according to the present embodiment, position information of defective pixels is converted into coordinates after refocusing, and defective pixel correction processing is performed on the reconstructed image after refocusing. Therefore, it is possible to correct appropriately regardless of the refocus position.

  Next, a second embodiment of the present invention will be described with reference to FIG. Since the configuration of the image pickup apparatus, the configuration of defective pixel information, the configuration of the scratch level table, and the defect correction process in this embodiment are the same as those in the first embodiment, description thereof is omitted here.

  In the first embodiment, defective pixel information of the image sensor is converted into defective pixel information at a plurality of refocus positions in advance for factory adjustment or the like and recorded in the memory unit 107. In contrast, in this embodiment, defective pixel information of the image sensor before conversion is recorded in the memory unit 107, and the conversion process is performed after the refocus process.

  FIG. 13 is a flowchart of the shooting control operation of the imaging apparatus to which the image processing system according to the second embodiment of the present invention is applied. FIG. 13 corresponds to FIG. 11 in the first embodiment, and parts that perform the same processing as in the first embodiment are denoted by the same reference numerals and description thereof is omitted. Hereinafter, processing operations different from those of the first embodiment will be described.

  In step S1301, the image processing unit 106 converts the defective pixel information of the image sensor recorded in the memory unit 107 into coordinates after refocusing. This process is the same as the process described in step S802 of FIG. 8, but in the first embodiment, defective pixel information is converted into coordinates at a plurality of refocus positions, whereas in this embodiment, step S1103 is performed. Convert only to the coordinates at the refocus position set in. Thereafter, defect correction processing is performed in step S1105 using the converted defective pixel information. Therefore, it is possible to correct the defect appropriately regardless of the refocus position as in the first embodiment.

  In this embodiment, since defective pixel information is converted after refocusing, it takes time to output the final image, but only the defective pixel information of the image sensor is recorded in the memory unit. Therefore, the required memory capacity is smaller than that of the first embodiment.

  As mentioned above, although each embodiment of the present invention was described, the present invention is not limited to these embodiments. For example, in the above-described embodiments, the present invention has been described by taking the imaging apparatus as an example. However, the present invention may be applied to an image processing system for image signals provided from a recording medium or the like in an information processing apparatus such as a PC.

  In the above-described embodiment, each process shown in FIGS. 11 to 13 is realized by reading a program for realizing the function of each process from the memory and executing it by the CPU of the control unit 108.

  Note that the present invention is not limited to the above-described configuration, and all or some of the functions shown in FIGS. 11 to 13 may be realized by dedicated hardware. The memory described above may be a non-volatile memory such as a magneto-optical disk device or a flash memory, a storage medium that can only be read such as a CD-ROM, or a volatile memory other than a RAM. Moreover, you may comprise from the computer-readable / writable storage medium by those combination.

  Further, a program for realizing the functions of the processes shown in FIGS. 11 to 13 is recorded on a computer-readable storage medium, and the program recorded on the storage medium is read into a computer system and executed. Each process may be performed. Here, the “computer system” includes an OS and hardware such as peripheral devices. Specifically, the program read from the storage medium is written into a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer. After that, the CPU of the function expansion board or function expansion unit performs part or all of the actual processing based on the instructions of the program, and the functions of the above-described embodiments are realized by the processing.

  The “computer-readable storage medium” refers to a storage device such as a flexible disk, a magneto-optical disk, a portable medium such as a ROM or a CD-ROM, and a hard disk built in a computer system. Furthermore, the “computer-readable storage medium” includes a medium that holds a program for a certain period of time. For example, it includes a volatile memory (RAM) inside a computer system that becomes a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line.

  The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line.

  The program may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, and what is called a difference file (difference program) may be sufficient.

A program product such as a computer-readable recording medium in which the above program is recorded can also be applied as an embodiment of the present invention. The above program, recording medium, transmission medium, and program product are included in the scope of the present invention.
The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes designs and the like that do not depart from the gist of the present invention.

Claims (13)

In an image processing system for processing an image signal generated by a pupil division unit of a photographing lens and an image pickup unit having a pixel array for picking up an optical image formed by the photographing lens and the pupil division unit,
Holding means for holding defective pixel information about defective pixels of the pixel array;
Setting means for setting a refocus position for generating a reconstructed image;
Refocusing means for generating a reconstructed image of the refocus position using the image signal;
Correction means for correcting the generated reconstructed image based on the defective pixel information, and
The image processing system, wherein the correcting unit corrects the generated reconstructed image using position information and output level information of a defective pixel corresponding to the reconstructed image at the set refocus position.   The holding means holds, as the defective pixel information, position information and output level information of defective pixels corresponding to a reconstructed image at each of a plurality of different refocus positions in association with information on the corresponding refocus position. The image processing system according to claim 1, wherein:   The holding means holds, as the defective pixel information, position information and output level information of defective pixels corresponding to the pixel arrangement, and the correction means uses the position information and output level information held in the holding means. In association with the information on the set refocus position, the position information and the output level information corresponding to the reconstructed image at the set refocus position are converted, and the converted position information and output level information are used. The image processing system according to claim 1, wherein the generated reconstructed image is corrected.   When the position information of different defective pixels corresponding to the pixel array is converted into the same position information corresponding to the reconstructed image at the set refocus position, the correction unit is different corresponding to the pixel array. 4. The image processing system according to claim 3, wherein the output level information of the defective pixel is converted into one output level information associated with the same position information corresponding to the reconstructed image. The correction unit is configured to detect the defective pixel based on the refocus position information, output level information corresponding to the refocused position reconstructed image indicated by the refocus position information, and the set refocus position. 5. The image processing according to claim 2, wherein whether or not to correct the generated reconstructed image is determined based on information, and the correction is performed according to a determination result. system.   6. The correction unit according to claim 1, wherein the correction unit corrects the reconstructed image using a pixel adjacent to a position indicated by position information of a defective pixel corresponding to the generated reconstructed image. The image processing system described in any one of Claims. An optical system having a photographing lens for forming an optical image of a subject;
An imaging unit having a pixel array for capturing an optical image formed by the imaging lens and the pupil dividing unit, and generating an image signal;
Holding means for holding defective pixel information about defective pixels of the pixel array;
Setting means for setting a refocus position for generating a reconstructed image;
Image processing means for processing the image signal;
The image processing means includes
Refocusing means for generating a reconstructed image of the refocus position using the image signal;
Correction means for correcting the generated reconstructed image based on the defective pixel information, and
The image pickup apparatus, wherein the correction unit corrects the generated reconstructed image using position information and output level information of a defective pixel corresponding to the reconstructed image at the set refocus position.   The position information and the output level information of the defective pixel corresponding to the pixel array are acquired, and the acquired position information and output level information are used for the defective pixel corresponding to the reconstructed image at each of a plurality of different refocus positions. Conversion means for converting into position information and output level information, and holding the converted position information and output level information in the holding means as defective pixel information in association with the corresponding refocus position information. The imaging apparatus according to claim 7, wherein the imaging apparatus is characterized. In a control method of an image processing system for processing an image signal generated by a pupil dividing unit of a photographing lens and an image pickup unit having a pixel array for picking up an optical image formed by the photographing lens and the pupil dividing unit,
A holding step of holding in the memory means defective pixel information about defective pixels of the pixel array;
A setting step for setting a refocus position for generating a reconstructed image;
A refocusing step of generating a reconstructed image of the refocus position using the image signal;
A correction step of correcting the generated reconstructed image based on the defective pixel information, and
The control method is characterized in that the correcting step corrects the generated reconstructed image using position information and output level information of a defective pixel corresponding to the reconstructed image at the set refocus position. A program for controlling an image processing system that processes an image signal generated by a pupil division unit of a photographing lens and an image pickup unit that has a pixel array that captures an optical image formed by the photographing lens and the pupil division unit,
Computer
Holding means for holding defective pixel information about defective pixels of the pixel array;
Setting means for setting a refocus position for generating a reconstructed image;
Refocusing means for generating a reconstructed image of the refocus position using the image signal;
Correcting means for correcting the generated reconstructed image based on the defective pixel information, and using the position information and output level information of the defective pixel corresponding to the reconstructed image at the set refocus position, A program that functions as a correction unit that corrects a generated reconstructed image.   A computer-readable storage medium storing the program according to claim 8.   A program that causes a computer to function as each unit of the image processing system according to any one of claims 1 to 6.   A storage medium on which the program according to claim 12 is recorded.

Images