JP2019186763A - Image pickup apparatus and control method thereof - Google Patents

Abstract

To suppress deterioration in image quality and data compression ratio caused by a sensitivity difference between pixels of the same color regardless of an inclination state of a light receiving surface of an image pickup device.SOLUTION: The inclination angle of the light receiving surface 401 of an image pickup device 200 is variable. A control unit 310 determines a replacement range on the basis of an inclination angle θ of the light receiving surface 401 formed with respect to a lens main surface 402. When a pixel signal from the image pickup device 200 is converted into an image signal, an output signal of at least a part of pixels of a replacement range is replaced with a value based on an output signal of at least one same-color pixel adjacent to the periphery of the pixel.SELECTED DRAWING: Figure 5

Description

  The present invention relates to an imaging apparatus capable of capturing an image by tilting a light receiving surface of an imaging element with respect to a main surface of an imaging optical system, and a control method therefor.

  Conventionally, when a plurality of subjects are photographed simultaneously with a general camera, the in-focus positions are different when the distances to the subjects are different. In that case, a plurality of subjects can be accommodated within the depth of field by wide-angle shooting with a narrowed aperture to increase the depth of field. On the other hand, using the Scheimpflug principle, imaging is performed so that the light receiving surface of the image sensor, which is normally parallel, the main surface (lens main surface) of the imaging optical system, and the plane on which a plurality of subjects exist intersect. It is known to adjust the focal position of each subject by tilting the element.

  For example, Patent Document 1 discloses an imaging device that can easily focus on each subject even when the distances to the plurality of subjects are different using the above principle. By using this principle, it is possible to obtain an image of a subject focused on a wide range within the angle of view when shooting a fixed angle of view such as a surveillance camera.

  On the other hand, an image sensor usually has some structural difference such as a difference in arrangement position with respect to electrodes shared by adjacent pixels. Due to this structural difference, a sensitivity difference between pixels occurs particularly with respect to incident light from an oblique direction.

Japanese Patent Laying-Open No. 2015-219754

  Considering a periodic pixel array, particularly a Bayer array, in the image sensor, a high frequency pattern noise is generated due to a sensitivity difference between the G pixel in the same row as the R pixel and the G pixel in the same row as the B pixel. There is a case. When high-frequency pattern noise occurs, there is a risk that image quality and data compression rate will decrease. In particular, in a surveillance camera, when photographing a plurality of subjects at different distances, it is desired to avoid as much as possible that the detection accuracy of the subject decreases or the amount of communication / stored data increases due to the inclination angle of the light receiving surface of the image sensor.

  In order to correct the sensitivity difference between the pixels, it is conceivable to apply a gain based on the correction table to some pixels. However, since the sensitivity difference that occurs varies depending on the incident angle of light with respect to the image sensor, in an imaging device in which the angle formed by the light receiving surface of the image sensor and the main surface of the imaging optical system is variable, in order to perform appropriate correction It was necessary to have multiple correction tables for each angle.

  An object of the present invention is to suppress a decrease in image quality and data compression rate caused by a difference in sensitivity between pixels of the same color regardless of the inclination state of a light receiving surface of an image sensor.

  In order to achieve the above object, the present invention provides an imaging optical system and an imaging device having a light receiving surface, in which pixels are periodically arranged, and the light receiving with respect to a plane orthogonal to the optical axis of the imaging optical system. An image sensor having a variable surface tilt angle; a determination unit that determines a range to be replaced in the image sensor based on the tilt angle of the light receiving surface of the image sensor; and an output signal of each pixel of the image sensor A conversion means for converting the signal into an image signal and an output signal of at least a part of a pixel to be replaced determined by the determination means adjacent to the periphery of the pixel when the conversion means converts the image signal into an image signal. Replacement means for replacing with a value based on an output signal of at least one same-color pixel.

  According to the present invention, it is possible to suppress a decrease in image quality and data compression ratio due to a difference in sensitivity between pixels of the same color regardless of the inclination state of the light receiving surface of the image sensor.

It is a typical block diagram of an imaging device. It is a block diagram of a signal processing part. It is a figure explaining the condition where a sensitivity difference generate | occur | produces between pixels. It is a figure which shows a mode that two to-be-photographed objects from which distance differs are image | photographed simultaneously. It is a flowchart of an image signal generation process. It is a figure which shows the example of an aspect of pixel replacement. It is a flowchart of an image signal generation process. It is a typical block diagram of an imaging device.

  Embodiments of the present invention will be described below with reference to the drawings.

(First embodiment)
FIG. 1 is a schematic configuration diagram of an imaging apparatus according to the first embodiment of the present invention. The imaging device is configured as a digital camera, for example. The subject image collected through the lens unit 100 as the imaging optical system is converted into an electrical signal by the imaging device 200, and then converted into image data (video signal) by the signal processing unit 300 and output. The lens unit 100 includes a focus lens, and may include a zoom lens, a diaphragm unit, and the like. The configuration of the lens unit 100 is not limited to the shape and configuration illustrated in FIG.

  The image pickup device 200 includes a semiconductor image pickup device such as a CCD sensor or a CMOS sensor and its peripheral circuits. The pixels constituting the image sensor 200 have high sensitivity to visible light. The imaging element 200 has a configuration in which pixels having high sensitivity are regularly arranged (periodically arranged) in any one of red (R), green (G), and blue (B) in the visible light range. The light from the subject can be acquired as a color image. For the image sensor 200, for example, a Bayer array used in a general camera or the like is employed (see FIGS. 3, 6A, and 6B). In this Bayer array, a square pattern configuration is regularly arranged with 4 pixels as one unit, with two G pixels, one R pixel, and one B pixel. Since the G pixel having the highest sensitivity is arranged on the diagonal line, the subject image can be captured clearly.

  The inclination angle of the light receiving surface 401 of the image sensor 200 is variably configured. The motor 201 has an inclination angle θ, which is an acute angle formed by the light receiving surface 401 of the image sensor 200 and a surface (lens main surface 402) orthogonal to the optical axis of the lens unit 100, based on a drive signal from the signal processing unit 300 (FIG. 4). (B)) can be changed. Although only one motor is illustrated in FIG. 1, a configuration in which the inclination angle θ can be changed two-dimensionally using a plurality of motors may be employed.

  FIG. 2 is a block diagram of the signal processing unit 300. A pixel signal (output signal) from the image sensor 200 is input to the signal processing unit 300, and is first converted into a signal suitable for image processing by the data conversion unit 301. The signal is converted by the image processing unit 302 as conversion means by correction processing, development processing, or the like, and an image signal is obtained. The developed image signal is output to the outside as image data (video signal) by the communication control unit 303. The control unit 310 controls the data conversion unit 301 and the image processing unit 302, and controls the lens unit 100, the image sensor 200, and the like so that shooting can be performed with settings according to the subject using pixel signals and image signals. . The determined settings and operation programs are stored in the non-volatile storage unit 304, and the control unit 310 reads them when necessary. The image processing unit 302 may also include a subject detection unit that detects a range where the subject exists from the obtained image signal. The control unit 310 includes a lens control unit 311 that controls the lens unit 100 and an image sensor control unit 312 that controls the image sensor 200.

  A situation in which a sensitivity difference occurs between each pixel in the Bayer array will be described with reference to FIG. FIG. 3A is a schematic view of a part of the image sensor 200 as viewed from the front of the light receiving surface 401. 3B and 3C are side views of a part of the image sensor 200. FIG. The electrode wiring 202 is arranged at the boundary between the row with the R pixel and the row with the B pixel so as to be symmetric with respect to one unit of four pixels in the Bayer array. The G1 pixel and the G2 pixel are adjacently arranged diagonally. Accordingly, in one unit, the G pixels are arranged on both sides of the electrode wiring 202.

  Consider a case where light from an oblique direction enters each pixel. As shown in FIG. 3B, a part of the light incident from above in FIG. 3B is blocked by the electrode wiring 202. As a result, the amount of incident light on the G2 pixel is smaller than that on the G1 pixel. On the other hand, as shown in FIG. 3C, a part of the light incident from below in FIG. 3C is blocked by the electrode wiring 202, and as a result, the incident light quantity of the G1 pixel is reduced. Thus, one G pixel is darker than the other G pixel.

  Thus, when there is a structural difference in the Bayer array, a sensitivity difference may occur with respect to light from an oblique direction. In the configuration of FIG. 3, a case where a difference in sensitivity occurs between the pixels on both sides sandwiching the electrode wiring 202 due to the electrode wiring 202 in the horizontal direction is illustrated. In the case where the vertical electrode wiring 202 is present, a sensitivity difference between the G pixels may occur in the same manner with respect to light from an oblique horizontal direction. Note that the situation in which the light from the lens is incident obliquely may be a case where a lens having a large F value is used, or a case where the imaging element is tilted.

  If there is a sensitivity difference between adjacent pixels, particularly G pixels, a high-frequency pattern of about one pixel interval may be generated in the luminance value of each pixel after demosaicing. JPG and H.C. In a compression technique such as H.264, the higher the correlation with adjacent pixels, the higher the compression ratio. Therefore, an image having such a high-frequency pattern has a lower compression ratio, resulting in an increase in the amount of communication / stored data.

  Next, consider a situation in which shooting is performed with a focus on a wide range within the angle of view using the principle of Scheinproof. FIG. 4A is a diagram illustrating a situation in which two subjects having different distances (here, a human face is assumed) are photographed simultaneously. FIG. 4B is a side view illustrating a state where the imaging device captures an image of a subject. FIG. 4C is a diagram illustrating an example of a captured image. Here, as an example, it is assumed that a subject surface 403 that is a plane on which an imaging target exists is the ground surface or the floor surface. It is not essential that the subject surface 403 is a surface parallel to the ground. The lens main surface 402 is a surface perpendicular to the optical axis of the lens unit 100.

  In order to photograph the subjects A and B at the same time, it is necessary to keep the subjects A and B within the angle of view of the lens unit 100 (a range indicated by a dotted line). Here, it is assumed that the plane including the lens principal surface 402 and the plane including the subject surface 403 intersect with one straight line CL. That is, the straight line CL is an imaginary straight line formed by the intersection of the plane including the lens principal surface 402 and the plane including the subject surface 403. In order to obtain a focused image on all the subject surfaces 403 within the angle of view, the inclination angle θ of the light receiving surface 401 is set so that the plane including the light receiving surface 401 of the image sensor 200 includes the straight line CL.

  A dotted line in FIG. 4B indicates a position where the subject images of the subjects A and B are formed on the image sensor 200. As shown in FIG. 4C, the imaging position on the light receiving surface 401 is determined by the distance to the subject. Here, the distances from the image sensor 200 to the subjects A and B are defined as a first distance L1 and a second distance L2, respectively. The installation height of the image sensor 200 (the shortest distance between the subject surface 403 and the image sensor 200) is defined as a height H. It is assumed that the position of the image sensor 200 is defined by the position of the center of gravity or the center position of the light receiving surface 401. The first distance L1 and the second distance L2 are respectively the distance from the image sensor 200 to the nearest end and the distance to the farthest end in the imaging range. Assume that the focal length of the lens unit 100 is 100 mm, the size of the image sensor 200 is 24 × 36 mm, and the height H is 2.5 m. When the inclination angle θ of the image sensor 200 is set to about 2.3 °, the first distance L1 is 10 m and the second distance L2 is 66 m. That is, an image focused on a range of approximately 10 to 66 m from the image sensor 200 can be obtained on the light receiving surface.

  In general, when the light receiving surface, the lens main surface, and the subject surface are inclined, the light incident on the light receiving surface is incident with different inclinations depending on the position in the light receiving surface. In particular, as shown in FIG. 4B, the incident light from the subject A in the vicinity of the imaging device 200 becomes incident light that is tilted more than the incident light from the subject B located far away. For this reason, the influence of the sensitivity difference between the G pixels is more pronounced in the vicinity of the subject, and the influence of the high frequency pattern noise is increased. Therefore, in the present embodiment, the signal processing unit 300 determines the third distance Lx between the first distance L1 and the second distance L2, and the first distance L1 from the third distance Lx. The pixel range of the image sensor 200 corresponding to the range up to is determined as the “replacement range”. That is, the signal processing unit 300 determines the pixel range of the image sensor 200 on which subject light closer to the third distance Lx is incident as the pixel range to be replaced with the pixel signal to be replaced. A mode of pixel replacement by the data conversion unit 301 will be described later.

  Here, if the size of the subject is taken into consideration, the subject in the vicinity appears larger in size on the screen than the subject in the distance, and therefore the recognition accuracy of the subject is less affected even if the resolution is lowered. Therefore, for example, if the value of one G pixel in one unit in the Bayer array is replaced with the value of the other G pixel only for a pixel range corresponding to a nearby subject, the distance between adjacent G pixels The sensitivity difference disappears. Therefore, it is possible to suppress a high-frequency pattern caused by a sensitivity difference between G pixels.

  The replacement range for performing the replacement process using the adjacent pixels is preferably determined based on the resolution (required resolution) and size necessary for object recognition, but it is not essential to be based on these. By performing the replacement process, the resolution is about 1 / √2, but when the subject size is doubled, the influence on the recognition of the subject is almost eliminated. This is because the subject size on the screen increases as the distance to the subject decreases. For example, by executing pixel replacement, the subject size about 47 m ahead (third distance Lx) becomes about √2 times the size of the subject 66 meters ahead (second distance L2). It is desirable to perform a replacement process on a range where a close subject forms an image. Specifically, although it varies slightly depending on the type of lens, even if replacement processing is performed on pixels in the range of about 90% from the vicinity of the entire pixel, the resolution is equal to or higher than that of the farthest subject. It is possible to obtain.

  FIG. 5 is a flowchart of the image signal generation process. This process is realized by the CPU provided in the control unit 310 reading the control program stored in the storage unit 304 into the RAM provided in the control unit 310 and executing it.

  Considering the case of shooting with an imaging device fixed like a surveillance camera, the shooting range and the distance to the subject are determined when the imaging device is fixed. Therefore, the user (installer) stores the focal length of the lens unit 100 and the size of the imaging element 200 in the storage unit 304 of the imaging apparatus in advance. The process of FIG. 5 corresponds to an operation sequence of the imaging apparatus when the imaging apparatus is installed and fixed.

  In step S501, the control unit 310 causes the storage unit 304 to store conditions input by the user when the imaging apparatus is fixed. This condition includes fixed state information of the imaging device (height H from the subject surface, distances L1 and L2 in the photographing range) and subject information (size and required resolution). The subject information is not essential for determining the replacement range.

  In step S <b> 502, the control unit 310 obtains an angle at which the image sensor 200 is tilted with respect to the lens principal surface 402 based on the conditions stored in the storage unit 304, and sets this as the tilt angle θ. As described with reference to FIG. 4B, the control unit 310 sets the inclination angle θ based on the height H and the distances L1 and L2 so that the plane including the light receiving surface 401 of the image sensor 200 includes the straight line CL. Calculate and set. When the height H is constant and the plane including the light receiving surface 401 includes the straight line CL, the distances L1 and L2 are determined if the inclination angle θ is determined, and the inclination angle θ is determined if the distances L1 and L2 are determined. There is a relationship.

  In step S503, the control unit 310 controls the motor 201 using the motor control unit 313, and tilts the image sensor 200 until the set tilt angle θ is reached. In step S504, the control unit 310 serving as a determination unit obtains the third distance Lx based on the above conditions, and the pixels of the image sensor 200 corresponding to the range from the third distance Lx to the first distance L1. The range is determined as the replacement range. As an example of the third distance Lx determination method, if L2 = 10 m and L3 = 66 m, the control unit 310 calculates a distance 1 / √2 times the second distance L2 as the third distance Lx. To do. That is, the third distance Lx is calculated to be about 47 m by Lx = L2 × (1 / √2). In this case, the pixel range of the image sensor 200 corresponding to the range of 10 m to 47 m is determined as the replacement range. When L2 = 10 m and L3 = 38 m, the third distance Lx is calculated to be about 27 m by Lx = L2 × (1 / √2). In this case, the pixel range of the image sensor 200 corresponding to the range of 10 m to 27 m is determined as the replacement range. Note that it is not essential to use an arithmetic expression to obtain the third distance Lx. For example, a table or the like may be used.

  Here, the maximum designable resolution at the farthest end is referred to as “maximum possible resolution”. The required resolution is defined by the resolution at the farthest end. The necessary resolution is set based on designation from the user. When the necessary resolution to be specified by the user exceeds the maximum possible resolution, the control unit 310 sets the maximum possible resolution as the necessary resolution. When calculating the third distance Lx, if the required resolution is not set or if the required resolution is set to the maximum possible resolution, the control unit 310 sets Lx = L2 × ( The third distance Lx is calculated by 1 / √2). On the other hand, when the required resolution is set to a value lower than the maximum possible resolution, the control unit 310 calculates the third distance Lx based on the required resolution (so as to be in a range satisfying the required resolution). For example, assume that the required resolution is set to 90% of the maximum possible resolution. In this case, assuming that L2 = 10 m and L3 = 66 m, the third distance Lx is calculated to be about 52 m from Lx = L2 × (1 / √2) × 0.9. That is, the control unit 310 calculates the third distance Lx by multiplying the distance 1 / √2 times the second distance L2 by the ratio of the necessary resolution to the maximum possible resolution. In this case, the pixel range of the image sensor 200 corresponding to the range of 10 m to 52 m is determined as the replacement range. By determining the replacement range based on the necessary resolution, the desired minimum resolution is ensured regardless of the subject at any position.

  Next, in step S505, the control unit 310 stores in the storage unit 304 information indicating the tilt angle θ set in step S502 and the replacement range calculated in step S504. Step S506 and subsequent steps are an operation sequence at the time of shooting. It should be noted that after step S505, the processing may be temporarily terminated, and thereafter, the processing from step S506 may be started when the timing for starting shooting comes. The information stored in step S505 can be used when shooting is started later. Note that the tilt of the image sensor 200 may be temporarily returned to the initial position, and when the photographing is started in that case, the tilt operation of the image sensor 200 may be performed based on the tilt angle θ stored in the storage unit 304.

  In step S506, the control unit 310 reads the information stored in step S505, and executes replacement processing on the pixels in the replacement range using the data conversion unit 301 (described later in FIGS. 6A and 6B). ). In step S507, the control unit 310 performs image processing on the pixel signal obtained from the image sensor 200 (the replaced pixel signal is a signal after replacement) by the image processing unit 302, and converts the image signal into an image signal. . Then, the control part 310 complete | finishes the process of FIG.

  6A and 6B are diagrams illustrating an example of pixel replacement. First, as a first example, the control unit 310 replaces the output signal of at least some pixels in the replacement range with a value based on the output signal of at least one same-color pixel adjacent to the periphery of the pixel. For example, as illustrated in FIG. 6A, the control unit 310 replaces the pixel signal of the G2 pixel 601 with the pixel signal of the G1 pixel 602 adjacent to the upper left, and does not replace the pixel signal of the G1 pixel 602. Use as is. The control unit 310 performs the same replacement process on all the G2 pixels 601 in the replacement range. Alternatively, the control unit 310 replaces the output signal of the pixel to be replaced with an average of the output signals of a plurality of pixels adjacent to the pixel to be replaced. For example, as shown in FIG. 6B, the pixel signal of the G2 pixel 601 is replaced with a value obtained by averaging the pixel signals of the G1 pixel 602 adjacent to the upper and lower sides thereof. In addition, the mode of substitution is not limited to these examples.

  As a configuration of the data conversion unit 301, a flexible programmable grid array (FPGA) is adopted, and only a signal corresponding to a specified range is replaced.

  According to the present embodiment, in the imaging apparatus including the imaging device 200 in which the inclination angle of the light receiving surface is variable, the control unit 310 converts the pixel signal from the imaging device 200 into an image signal. Replace the output signals of at least some of the pixels in the range. That is, the control unit 310 determines a replacement range based on the inclination angle θ of the light receiving surface 401 formed with respect to the lens main surface 402, and adjoins the output signal of at least some pixels in the replacement range around the pixel. Is replaced with a value based on the output signal of at least one same color pixel. Thereby, regardless of the inclination state of the light receiving surface of the image sensor, it is possible to suppress a decrease in image quality and data compression rate due to a sensitivity difference between pixels of the same color. Accordingly, it is possible to photograph a plurality of subjects with different distances at an appropriate focus without causing a decrease in subject detection accuracy and an increase in the amount of communication / stored data. In addition, since it is not necessary to have a plurality of correction tables for each inclination angle, the complexity of the configuration can be avoided.

(Second Embodiment)
If the subject has a high-frequency pattern in the first place, the resolution may be impaired by the replacement process. Therefore, in the second embodiment of the present invention, pixel signal values before and after replacement are compared, and if the comparison result exceeds a certain threshold, it is determined that a high-frequency pattern of the subject exists and the replacement process is not reflected. To do.

  FIG. 7 is a flowchart of image signal generation processing in the second embodiment. This process is realized by the CPU provided in the control unit 310 reading the control program stored in the storage unit 304 into the RAM provided in the control unit 310 and executing it. The configuration other than the image signal generation processing is the same as that of the first embodiment. The storage unit 304 stores a threshold value in advance.

  The processing in steps S501 to S505 is the same as that described in FIG. After step S505, in step S702, the control unit 310 reads information indicating a replacement range and a threshold value from the storage unit 304, and determines a replacement range based on the information. In step S702, the control unit 310 uses the data conversion unit 301 to compare the value when the output signal is replaced with the value when the output signal is not replaced for each pixel to be replaced in the replacement range, The difference or ratio between the two is obtained as a comparison result. The replacement in this case is a provisional replacement process. Furthermore, the control unit 310 determines whether or not the obtained difference (or ratio) is larger than a threshold value. If the difference (or ratio) is larger than the threshold value, control unit 310 determines that there is a high-frequency pattern of the subject, and proceeds to step S507 without executing the replacement process in step S703. In this case, the replacement process is not executed when the pixel signal is converted into the image signal. On the other hand, when the difference (or ratio) is not greater than the threshold value, the control unit 310 advances the process to step S506. In this case, as in the first embodiment, replacement processing is performed when the pixel signal is converted into an image signal. The processing in step S507 is the same as that described in FIG.

  According to the present embodiment, regardless of the inclination state of the light receiving surface of the image sensor, with respect to suppressing a decrease in image quality and data compression rate due to sensitivity difference between pixels of the same color, the first embodiment and The same effect is produced. In addition, when the subject has a high-frequency pattern, it is possible to shoot without damaging it.

(Third embodiment)
FIG. 8 is a schematic configuration diagram of an imaging apparatus according to the third embodiment of the present invention. In the first and second embodiments, the data conversion unit 301 as a replacement unit is included in the signal processing unit 300. On the other hand, in the present embodiment, the data conversion unit 301 is not included in the signal processing unit 300 but is included in the imaging element 200. Other configurations are the same as those of the first embodiment or the second embodiment. Therefore, the image signal generation processing (FIGS. 5 and 7) is realized by the cooperation of the data conversion unit 301 and the signal processing unit 300 in the image sensor 200. The pixel signal in the replacement target range obtained from the image sensor 200 is subjected to replacement processing before being transferred to the signal processing unit 300.

  According to this embodiment, the same effects as those of the first or second embodiment can be obtained.

  In each of the above embodiments, the data conversion unit 301 has a pixel replacement function. However, the image processing unit 302 of the signal processing unit 300 is configured to execute pixel replacement processing by software processing. Also good. In the case of such a configuration, the data conversion unit 301 in the signal processing unit 300 may be eliminated.

  Although the present invention has been described in detail based on the preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the present invention are also included in the present invention. included. A part of the above-described embodiments may be appropriately combined.

DESCRIPTION OF SYMBOLS 100 Lens unit 200 Image pick-up element 301 Data conversion part 302 Image processing part 310 Control part 401 Light-receiving surface 402 Lens main surface

Claims (13)

An imaging optical system;
An imaging device having a light receiving surface, in which pixels are periodically arranged, wherein an inclination angle of the light receiving surface with respect to a plane orthogonal to the optical axis of the imaging optical system is variable;
Determining means for determining a range to be replaced in the image sensor based on an inclination angle of the light receiving surface of the image sensor;
Conversion means for converting an output signal of each pixel of the image sensor into an image signal;
In the conversion to the image signal by the conversion means, the output signal of at least a part of the pixel in the range to be replaced determined by the determination means is converted into the output signal of at least one same color pixel adjacent to the periphery of the pixel. An imaging apparatus comprising: a replacement unit that replaces with a based value.   The angle of inclination of the light receiving surface is set so that the plane on which the light receiving surface exists includes a straight line formed by the intersection of the plane on which the imaging target exists and the plane orthogonal to the optical axis. The imaging apparatus according to claim 1, wherein the imaging apparatus is characterized. When a range from a first distance from the image sensor to a second distance that is longer than the first distance is set as an image range, the tilt angle is calculated by using a plane on which the imaging target exists and the image sensor. Is set based on the shortest distance to the first distance, the first distance, and the second distance,
The determining unit determines a third distance between the first distance and the second distance, and also determines the range of the imaging element corresponding to a range from the third distance to the first distance. The imaging apparatus according to claim 2, wherein a pixel range is determined as the range to be replaced.   The imaging apparatus according to claim 3, wherein the determining unit determines a distance that is 1 / √2 times the second distance as the third distance.   The imaging apparatus according to claim 3, wherein the determination unit sets a required resolution at the second distance and determines the third distance based on the required resolution.   The replacement means compares a value when the output signal of the pixel in the replacement target range is replaced with a value when the replacement is not performed, and converts the output signal into an image signal based on the comparison result. 6. The imaging apparatus according to claim 1, wherein whether or not to perform replacement is determined when performing the inspection.   The replacing unit converts the output signal of the pixel in the replacement target range into an image signal by the converting unit when the difference or ratio between the value when the pixel is replaced and the value when the pixel is not replaced is not larger than a threshold value. The imaging apparatus according to claim 6, wherein replacement is performed when the conversion is performed, and if the difference or ratio is greater than the threshold value, replacement is not performed when the conversion unit converts the image into an image signal.   The imaging device according to claim 1, wherein the replacement unit replaces an output signal of one adjacent pixel in the range to be replaced with an output signal of the other pixel. .   The replacement means replaces an output signal of the pixel to be replaced by an average of output signals of a plurality of pixels adjacent to the pixel to be replaced. Imaging device. It has a signal processing unit that acquires the output signal of each pixel of the image sensor and includes the conversion unit,
The imaging apparatus according to claim 1, wherein the replacement unit is included in the signal processing unit.   The imaging apparatus according to claim 1, wherein the replacement unit is included in the imaging element. A signal processing unit for obtaining an output signal of each pixel of the image sensor;
The imaging apparatus according to claim 1, wherein the function of the replacement unit is realized by software processing in the signal processing unit. An imaging optical system, and an imaging device having a light receiving surface, in which pixels are periodically arranged, wherein an inclination angle of the light receiving surface with respect to a plane perpendicular to the optical axis of the imaging optical system is variable A method of controlling an imaging apparatus comprising:
Based on the inclination angle of the light receiving surface of the image sensor, determine the range of the replacement target in the image sensor,
In converting an output signal of each pixel of the image sensor into an image signal, an output signal of at least a part of the determined pixel to be replaced is output from at least one same color pixel adjacent to the periphery of the pixel. A method for controlling an imaging apparatus, wherein the value is replaced with a value based on

Images