WO2021145158A1 - Imaging device, and method for controlling imaging device - Google Patents

Abstract

An imaging device according to an embodiment is provided with: a lens unit for collecting incident light from a subject; an imaging unit for imaging the incident light collected by the lens unit in a non-focused state; and an image analyzing unit which analyzes an image captured by the imaging unit and which, on the basis of the pixel values of a plurality of predetermined surrounding pixels positioned around a pixel of interest, set a pixel value for the pixel of interest and generates a result image for identifying the position of the subject.

Description

Imaging device and control method of imaging device

The present disclosure relates to an imaging device and a control method for the imaging device.

The image sensor may have defects in the pixels (hereinafter referred to as defective pixels) due to manufacturing variations or the influence of radiation, which may appear as random noise in the imaging results. Most of these defective pixels are fixed as "white" or "black" dots.

In addition, when imaging in a dark place, generally, a method such as long-time exposure and amplification of a minute signal (increasing the gain) by an amplifier has been adopted.

When exposing for a long time, if the subject is moving or the shooting device side is not fixed properly, subject blurring and camera shake will occur, and there is a problem that the intended shooting result will not be obtained. This problem can be solved by shortening the exposure time sufficiently and using an amplifier or the like to amplify a minute signal.

However, there is a problem that random noise is recorded in the shooting result because the noise component of the amplifier itself and the noise from the power supply circuit are amplified at the same time.

In addition, when shooting in outer space, defective pixels may increase due to the influence of cosmic rays (radiation), or the pixels may malfunction temporarily, resulting in imaging as bright spot noise, for example. When a star is photographed from an artificial satellite, there is a problem that the star and noise cannot be distinguished.
Further, various techniques have been proposed for correcting noise caused by defective pixels (see, for example, Patent Documents 1 to 3).

Japanese Unexamined Patent Publication No. 2015-305717 Japanese Unexamined Patent Publication No. 2001-128068 Special Table 2019-500731, Gazette

However, since the purpose of the above-mentioned prior art is to correct "defective pixels" generated on the image sensor, it is suddenly generated due to random noise generated at the time of shooting in a dark place or irradiation with cosmic rays. We could not deal with bright spots.
In particular, when shooting a starry sky, which is necessary for surveying purposes, or shooting markers in a dark place, there is a problem that accurate surveying cannot be performed due to the effects of random noise and cosmic rays.

This technology was made in view of such a situation, and it is possible to easily identify the position of the subject by removing defective pixels, random noise, the influence of cosmic rays, etc., and by extension, perform accurate surveying. It is an object of the present invention to provide a possible image pickup apparatus and a control method for the image pickup apparatus.

The image pickup apparatus of the embodiment analyzes a lens unit that collects incident light from a subject, an image pickup unit that captures the incident light collected by the lens unit in an unfocused state, and an image captured by the image pickup unit. , An image analysis unit that sets the pixel value of the pixel of interest based on the pixel values of a plurality of predetermined peripheral pixels located around one pixel of interest and generates a result image for identifying the position of the subject. , Equipped with.

It is a schematic block diagram of the image pickup apparatus of 1st Embodiment. It is explanatory drawing of the arrangement relation of the lens which constitutes a lens part, and the photoelectric conversion part. It is an outline processing flowchart of an image analysis unit. It is explanatory drawing of an example of the captured image. It is explanatory drawing of the image scanning and the obtained result image. It is explanatory drawing of the image scanning and the obtained result image. It is explanatory drawing of the noise removal processing of the modification of 1st Embodiment. It is explanatory drawing of the result image of the modification of 1st Embodiment. It is explanatory drawing in the case of scanning the image origin in a window of 3 × 3 pixels. It is a summary processing flowchart in the modification of 1st Embodiment. It is a schematic block diagram of the image pickup apparatus of 2nd Embodiment. It is operation explanatory drawing of 2nd Embodiment. It is a schematic block diagram of the image pickup apparatus of 3rd Embodiment. It is explanatory drawing of 4th Embodiment. It is explanatory drawing of 4th Embodiment. It is a flowchart of the outline processing of 4th Embodiment. It is explanatory drawing of the window of 4th Embodiment. It is a schematic block diagram of the image pickup apparatus of 5th Embodiment. It is a block diagram of the 1st aspect of 5th Embodiment. It is a block diagram of the 2nd aspect of 5th Embodiment. It is explanatory drawing of an example of the captured image of the 2nd aspect of 5th Embodiment. It is explanatory drawing of the window in 2nd aspect of 5th Embodiment. It is a summary processing flowchart of the image analysis part in the 2nd aspect of 5th Embodiment. It is explanatory drawing of the result image in 2nd aspect of 5th Embodiment. It is explanatory drawing in the case of scanning the image origin in a cross-shaped window having a 5-pixel configuration. It is a schematic block diagram of the image pickup apparatus of the modification of 5th Embodiment.

Hereinafter, embodiments will be described in detail with reference to the drawings.
[1] First Embodiment FIG. 1 is a schematic block diagram of an image pickup apparatus of the first embodiment.
The image pickup apparatus 10 includes a lens unit 11, an image pickup unit 12, an image storage unit 13, an image analysis unit 14, and a result storage unit 15.

The lens unit 11 collects the light from the subject and guides it to the imaging unit 12.
The image pickup unit 12 performs photoelectric conversion and analog / digital conversion of the light collected by the lens unit 11 and outputs the light to the image storage unit 13. In this case, the light receiving surface of the imaging unit is fixed at a position in front of the image point determined by the lenses constituting the lens unit 11. This is because the received image is intentionally blurred for shooting.

In this case, the imaging unit 12 includes a photoelectric conversion unit 21, an analog amplifier unit 22, and an AD conversion unit 23.
The photoelectric conversion unit 21 includes an image sensor such as a CCD (Charged-coupled devices) or CMOS (Complementary metal-oxide-semiconductor), performs photoelectric conversion of the light incident from the lens unit 11, and uses an analog amplifier as an original imaging signal. Output to unit 22.

The analog amplifier unit 22 amplifies the input original imaging signal and outputs it as an imaging signal to the AD conversion unit 23.
The AD conversion unit 23 performs analog / digital conversion of the imaging signal and outputs it as imaging data to the image storage unit 13.

The image storage unit 13 stores the input imaging data in units of captured images.
The image analysis unit 14 reads the image capture data of each image captured image from the image storage unit 13, analyzes the image, and outputs the result image to the result storage unit 15 after removing the effects of random noise, cosmic rays, defective pixels, and the like. do.
The result storage unit 15 stores the result image output by the image analysis unit 14.

FIG. 2 is an explanatory diagram of the arrangement relationship between the lens constituting the lens unit and the photoelectric conversion unit.
As shown in FIG. 2, the first focal point FP1 is located on the optical axis of the lens 11A between the subject OBJ such as a star and the lens 11A which is a biconvex lens constituting the lens unit 11. ..

Similarly, the second focal point FP2 is located on the optical axis of the lens 11A between the image point 21F0 of the lens 11A and the lens 11A.
Then, in the case of the example of FIG. 2, the light receiving surface 21F of the photoelectric conversion unit 21 is positioned on the lens 11A side from the image point. Therefore, the image formed on the light receiving surface 21F is in a more blurred state than the image formed on the image point 21F0.

Therefore, the change in the light receiving intensity on the light receiving surface 21F is slower than the change in the light receiving intensity at the image point 21F0. Here, "gentle" means a state in which the change in light receiving intensity is not pulse-like, and for example, the change in light receiving intensity changes from the maximum value like a normal distribution curve.
On the other hand, the change in light receiving intensity with respect to adjacent pixels due to the incident of cosmic rays or pixel defects becomes a steep pulse.

Here, the image analysis process will be described in detail.
FIG. 3 is an outline processing flowchart of the image analysis unit.
In the following description, the pixel of interest (pixel of interest) in the photoelectric conversion unit 21 is scanned by using a window WD of 3 × 3 pixels (pixels) including the surrounding eight pixels (pixels) to be noticed. An example of performing image analysis on pixels will be described.

First, the image analysis unit 14 sets the parameters X and Y for specifying the pixels Px (X, Y) as the pixel of interest to 0, which is the initial value (step S11). In the present embodiment, the parameter X is a parameter in the row direction, and the parameter Y is a parameter in the column direction (see FIG. 4 described later).

Next, the image analysis unit 14 determines whether or not the parameter Y exceeds Ymax, which is the maximum value of the parameter Y (step S12). That is, the image analysis unit 14 determines whether or not the processing of all the pixels has been completed.
In the determination of step S12, if the parameter Y exceeds the maximum value Ymax of the parameter Y (step S12; Yes), the image analysis process is terminated.

In the determination of step S12, if the parameter Y has not yet exceeded the maximum value Ymax of the parameter Y (step S12; No), it is determined whether or not the parameter X exceeds the maximum value of the parameter X, Xmax. (Step S13). That is, the image analysis unit 14 determines whether or not the processing of the pixels for one line is completed.

In the determination of step S13, if the parameter X exceeds Xmax, which is the maximum value of the parameter X (step S13; Yes), the image analysis unit 14 sets Y = Y + 1 (step S14) and steps the process again. Move to S12.
In the determination of step S13, if the parameter X has not yet exceeded Xmax, which is the maximum value of the parameter X (step S13; No), the image analysis unit 14 sequentially scans the window WD of 3 × 3 pixels. , Acquire the values of the imaging data corresponding to each of the nine pixels (step S15).
Subsequently, the image analysis unit 14 acquires the minimum value of the imaging data among the nine imaging data corresponding to the 3 × 3 pixel window (step S16).

Subsequently, the image analysis unit 14 compares the minimum value of the imaging data acquired in step S16 with the predetermined threshold value data D (step S17).
Here, the threshold data D is data for treating the value of the imaging data as noise if the value of the imaging data is smaller than this, and treating it as imaging data of a black spot (the lowest light receiving intensity (luminance) level). ..

As a result, in the comparison of step S17, if the minimum value of the imaging data acquired in step S16 is equal to or less than the threshold data D (step S16; minimum value of imaging data ≤ D), the value of the imaging data of the pixel of interest. Is treated as noise and clamped to be treated as imaging data of black spots (light receiving intensity (luminance) level lowest) (step S18).

Then, the value of the imaging data after clamping is output as the pixel value of the pixel of interest, and is stored in the result storage unit 15 (step S19).

On the other hand, in the comparison of step S13, when the minimum value of the imaging data acquired in step S12 exceeds the threshold data D (step S17; minimum value of imaging data> D), the imaging data of the pixel of interest. Is output as the pixel value of the pixel of interest, and is stored in the result storage unit 15 (step S19).

Subsequently, the image analysis unit 14 sets X = X + 1 (step S20), and shifts the process to step S13 again.

As a result, the value of the imaging data corresponding to the pixel of interest changes sharply with respect to any of the values of the imaging data of the eight adjacent pixels, that is, due to the incident of cosmic rays or pixel defects. When it is considered that the value is changed, the value of the imaging data corresponding to the pixel of interest is treated as noise.

On the other hand, the values of the imaging data corresponding to the pixel of interest are considered to be gradual changes with respect to all the imaging data values of the eight adjacent pixels, that is, the imaging data corresponding to the actual captured image. If this is the case, the value of the imaging data corresponding to the pixel of interest will be saved as it is.

As a result, the result image stored in the result storage unit 15 is an image in which the effects of cosmic ray incidents, pixel defects, etc. are removed.

Next, acquisition of a more specific result image will be described with reference to the drawings.
FIG. 4 is an explanatory diagram of an example of a captured image.
In FIG. 4, for the sake of easy understanding, an image G1 which is an image of an image-imposed image data corresponding to 9 pixels in length × 20 pixels in width will be described as an example.

The captured image G1 includes the imaging data PE-N caused by the incident of cosmic rays, pixel defects, etc., and the imaging data PE-1 corresponding to the actual subject.

As shown in FIG. 4, it can be seen that the value of the imaging data PE-N (brightness of the pixels) has a sharp change in the light receiving intensity with respect to the surrounding pixels.

On the other hand, since the imaging data PE1 corresponding to the actual subject captures a blurred image due to the arrangement relationship between the lens 11A and the photoelectric conversion unit 21, the change in the light receiving intensity with respect to the surrounding pixels is gradual. It turns out that it is a good thing.

5A and 5B are explanatory views of image scanning and the resulting image obtained.
First, the image analysis unit 14 scans the entire captured image by sequentially changing the attention pixels from the left side to the right side and from the upper side to the lower side in the window WD of 3 × 3 pixels, and for each of the one attention pixel. The values of the imaging data corresponding to the nine pixels are acquired.

More specifically, when the pixel of interest is a pixel corresponding to the imaging data PE-N, as shown in FIG. 5A, the light receiving intensity of the imaging data PE-N is steep with respect to the eight surrounding pixels. Of the nine imaging data corresponding to the 3 × 3 pixel window WD, the minimum value of the imaging data is any of the eight peripheral pixels.

In this case, assuming that the threshold data D is set between the second light-receiving intensity range from the bottom and the third light-receiving intensity range from the bottom in the five-step light-receiving intensity range shown in FIG. Since the minimum value value (first light receiving intensity range from the bottom) of the eight surrounding pixels is equal to or less than the value of the threshold data D, the light receiving intensity of the pixel whose attention pixel corresponds to the imaging data PE-N is , Clamped to the lowest light receiving intensity.

On the other hand, when the pixel of interest is the imaging data PE-1, the light receiving intensity of the imaging data PE-1 gradually changes with respect to the eight surrounding pixels, as shown in FIG. 5A. Among the nine imaging data corresponding to the 3 × 3 pixel window WD, the minimum value of the imaging data is any of the four peripheral pixels located diagonally with respect to the imaging data PE-1.

In this case, it is assumed that the threshold data D is set between the second light receiving intensity range from the bottom and the third light receiving intensity range from the bottom in the five-step light receiving intensity range shown in FIG. Since the minimum value value (third light receiving intensity range from the bottom) of the four diagonally located peripheral pixels exceeds the value of the threshold data D, the pixel of interest is in the imaging data PE-1. The light receiving intensity of the corresponding pixel is the third light receiving intensity range from the bottom.

The same processing is also performed for the eight pixels around the pixel in which the pixel of interest corresponds to the imaging data PE-1, and the final result image is as shown in FIG. 5B.
From this, it can be seen that a star or the like as a subject exists in the pixel whose attention pixel corresponds to the imaging data PE-1.

Similarly, by scanning and processing the pixel of interest for the entire captured image, cosmic ray irradiation and noise caused by defective pixels are removed by processing only the captured image, and imaging corresponding to the subject is performed. You can get the data.

In other words, as for the small light spots randomly generated on the photoelectric conversion unit 21, only the blurred point light source obtained by passing through the lens 11A of the lens unit 11 is stored in the result storage unit 15. ..
In this case, since it is not necessary to store the location of the defective pixel or the like, the storage capacity can be reduced, and it is not necessary to perform exception processing, so that a desired image can be obtained by simple processing.

In the above description, the size of the window is 3 × 3 pixels, but it can be adjusted to a size of 5 × 5 pixels, 7 × 7 pixels, or the like according to the image resolution.
On the contrary, when the size of the subject is sufficiently larger than the window size, the image can be reduced and resized according to the window size. As a result, even if the image has a large number of pixels, the number of scans can be the same as that of an image having a small number of pixels, and the processing can be simplified or speeded up.

FIG. 6 is an explanatory diagram of noise removal processing of a modified example of the first embodiment.
In the noise removal processing of the first embodiment, a grayscale image is used as the image to be captured, and the light receiving intensity of the pixel of interest is set by comparing the minimum value of the peripheral pixels of the pixel of interest with a predetermined value D. In this modification, a binarized image is generated from the captured image using a predetermined threshold value, and the peripheral pixels of the pixel of interest (in the case of the above-mentioned 3 × 3 pixel window, the peripheral 8 pixels). When even one pixel having a low light receiving intensity (= 0) is included, the value of the pixel of interest is set to the low light receiving intensity (= 0).

For example, when the captured image G1 of FIG. 4 is binarized, it is obtained as a binarized image G3 as shown in FIG.
With respect to the binarized image G3, the image analysis unit 14 scans the entire captured image by sequentially changing the attention pixels from the left side to the right side and the upper side to the lower side in the window WD of 3 × 3 pixels. , Acquire the values of the imaging data corresponding to each of nine pixels for one pixel of interest.

Then, it is determined whether or not even one pixel having a low light receiving intensity (= 0) is included among the eight surrounding pixels with respect to the pixel of interest.

In the case of the example of FIG. 6, when the pixel of interest is a pixel corresponding to the imaging data PE-N, all of the surrounding eight pixels have low light receiving intensity (= 0), so attention is paid. The value of the pixel corresponding to the imaging data PE-N, which is a pixel, is set to low light reception intensity (= 0).

On the other hand, when the pixel of interest is a pixel corresponding to the imaging data PE-1, all of the surrounding eight pixels have high light receiving intensity (= 1), so that the imaging data of interest is pixel. The value of the pixel corresponding to PE-1 is a high light receiving intensity (= 1).

FIG. 7 is an explanatory diagram of a result image of a modified example of the first embodiment.
As a result of the above processing, in the result image G4 obtained by the captured image G3 of FIG. 6, as shown in FIG. 7, only the pixel whose attention pixel corresponds to the imaging data PE-1 has a high light receiving intensity (= 1). Therefore, it can be easily determined that a star or the like, which is a subject, is located at the position without being affected by noise.

FIG. 8 is an explanatory diagram when the origin of the image is scanned in a window of 3 × 3 pixels.
In the above description, the case where the target pixel exists in all of the 3 × 3 pixel window WD has been described, but when scanning the peripheral portion of the captured image, it is included in a part of the 3 × 3 pixel window WD. The processing target pixel may not exist.
In the case of the example of FIG. 8, among the window WDs of 3 × 3 pixels, the processing target pixels do not exist in 5 pixels.
In such a case, for example, the process is performed by any of the following methods.
(1) The portion where the processing target pixel does not exist is ignored, and the processing is performed only on the portion where the processing target pixel exists. More specifically, in the case of the example of FIG. 8, the evaluation is performed only with the four pixels in which the processing target pixel exists.
(2) For the portion where the processing target pixel does not exist, the value is set to 0 and processing is performed as usual.
(3) Copy the pixel values of pixels adjacent in the column direction or row direction as they are (Note that the upper left pixel does not have pixels adjacent in the column direction or row direction, so as in the case of (1). , The processing is performed with the remaining 8 pixels, or the value is set to 0 as in the case of (2).
By adopting the above configuration, processing can be reliably performed even when scanning the peripheral portion of the captured image.

Here, the processing of the modified example of the first embodiment will be described.
FIG. 9 is an outline processing flowchart in a modified example of the first embodiment.
Also in the following description, the pixel of interest (pixel of interest) in the photoelectric conversion unit 21 is scanned by using a window of 3 × 3 pixels (pixels) including the surrounding eight pixels (pixels) of interest. An example of performing image analysis on the above will be described.

First, the image pickup apparatus 10 takes a picture in a state of being out of focus (step S21).
Next, the image analysis unit 14 binarizes the captured data of the obtained captured image to generate a binarized image (step S22).
In this case, in the binarization process, for example, when the pixel value is less than a predetermined threshold value (= dark), it is set to "0" corresponding to black, and is equal to or higher than the predetermined threshold value (= = Bright), it is set to "1" corresponding to white.

Next, the image analysis unit 14 sets the parameters X and Y for specifying the pixel Px (X, Y) as the pixel of interest to 0, which is the initial value (step S23).

Next, the image analysis unit 14 determines whether or not the parameter Y exceeds Ymax, which is the maximum value of the parameter Y (step S24). That is, the image analysis unit 14 determines whether or not the processing of all the pixels has been completed.

In the determination of step S24, when the parameter Y exceeds the maximum value Ymax of the parameter Y (step S24; Yes), the processing is completed for all the pixels constituting the captured image, so that the obtained image is obtained. Is stored in the result storage unit 15 (step S30), and the image analysis process is completed.

In the determination of step S24, if the parameter Y has not yet exceeded the maximum value Ymax of the parameter Y (step S24; No), it is determined whether or not the parameter X exceeds the maximum value of the parameter X, Xmax. (Step S25). That is, the image analysis unit 14 determines whether or not the processing of the pixels for one line is completed.

In the determination of step S25, if the parameter X exceeds Xmax, which is the maximum value of the parameter X (step S25; Yes), the image analysis unit 14 sets Y = Y + 1 (step S26) and steps the process again. Move to S24.
In the determination of step S25, if the parameter X has not yet exceeded Xmax, which is the maximum value of the parameter X (step S25; No), the image analysis unit 14 sequentially scans the 3 × 3 pixel window (step S25; No). Step S27), it is determined whether or not there is at least one pixel having a low light receiving intensity (= 0) among the eight peripheral pixels corresponding to the pixel of interest, and one of the eight peripheral pixels corresponding to the pixel of interest is selected. When there is a pixel having a low light receiving intensity (= 0) at any time, the value of the imaging data of the pixel of interest is set to the low light receiving intensity (= 0) (step S28).

Subsequently, the image analysis unit 14 sets X = X + 1 (step S29), and shifts the process to step S25 again.

As a result of the above processing being performed and the obtained image being stored in the result storage unit 15, the value of the imaging data corresponding to the pixel of interest is steep with respect to any of the values of the imaging data of the eight adjacent pixels. That is, when it is considered that the value is changed due to the incident of cosmic rays or pixel defects, the value of the imaging data corresponding to the pixel of interest is set to low light receiving intensity (= 0). Therefore, it will be treated as noise.

On the other hand, when the imaging data of the eight pixels adjacent to the pixel of interest all have high light receiving intensity, the value of the imaging data corresponding to the pixel of interest remains at high light receiving intensity (= 1) and is stored. Will be done.

As a result, the result image saved in the result storage unit 15 is an image showing the position of the original target subject (star, etc.) by removing the influences of cosmic ray incidents, pixel defects, and the like.

[2] Second Embodiment FIG. 10 is a schematic block diagram of the image pickup apparatus of the second embodiment.
In FIG. 10, the same parts as those in the first embodiment of FIG. 1 are designated by the same reference numerals.
The difference between the second embodiment and the first embodiment is that the imaging unit 12 is driven along the optical axis of the lens 11A of the lens unit 11 and the light receiving surface 21F of the photoelectric conversion unit 21 constituting the imaging unit 12 is driven. This is a point in which an imaging control unit 31 capable of varying the distance from the image point 21F0 is provided.

In other words, in the second embodiment, the imaging surface is configured to be able to be shifted, the focus is shifted as necessary, and the edges of the obtained image are blurred. That is, it is configured so that the value of the imaged data can be smoothed, and eventually the smoothing of the image can be achieved as needed.
Here, smoothing the value [image] of the imaging data means, for example, changing the value of the imaging data that changes in a pulse manner in a Gaussian distribution so that the image changes gently (hereinafter, the image). Similarly).

In this case, the imaging control unit 31 makes the light receiving surface 21F of the photoelectric conversion unit 21 coincide with the image point 21F0, so that the imaging device 10A acquires a focused image as an captured image in the same manner as a normal camera. It becomes possible to do.

FIG. 11 is an operation explanatory view of the second embodiment.
Further, as shown in FIG. 10, the distance of the light receiving surface 21F of the photoelectric conversion unit 21 from the image point 21F1 is changed by the imaging control unit 31, so that the degree of blurring of the image on the light receiving surface 21F can be changed. , Optimal captured images can be obtained under various imaging conditions.

For example, even if the subject is small and all of it fits within one pixel, the distance from the image point 21F0 of the light receiving surface 21F of the photoelectric conversion unit 21 can be increased so that the subject has a plurality of pixels. It is also possible to remove the noise by performing the same processing in the state of receiving the light of. Further, when imaging is performed for the purpose of viewing or the like, an image in which the focal position (focus) is matched can be obtained by controlling the image in the same manner as in the conventional imaging device.

[3] Third Embodiment FIG. 12 is a schematic block diagram of the image pickup apparatus of the third embodiment.
In FIG. 12, the same parts as those in the first embodiment of FIG. 1 are designated by the same reference numerals.

The difference between the third embodiment and the first embodiment is that the lens unit 11 is driven along the optical axis of the lens 11A of the lens unit 11 and the light receiving surface of the photoelectric conversion unit 21 constituting the imaging unit 12 is formed. This is a point where a lens control unit 32 capable of changing the distance from the image point 21F0 of the 21F is provided.

In other words, in the third embodiment, the lens portion is configured to be capable of shifting along the optical axis, the focus is shifted as necessary, and the edges of the obtained image are blurred, that is, imaging. It is configured so that the data value [image] can be smoothed.

In this case, the lens control unit 32 makes the light receiving surface 21F of the photoelectric conversion unit 21 coincide with the image point 21F0, so that the imaging device 10B acquires a focused image as an captured image in the same manner as a normal camera. It becomes possible to do.

Further, by changing the distance of the light receiving surface 21F of the photoelectric conversion unit 21 from the image point 21F0 by the lens control unit 32, the degree of blurring of the image on the light receiving surface 21F can be changed, and various imaging conditions can be met. The optimum captured image can be obtained.
Further, when imaging is performed for the purpose of viewing or the like, an image in which the focal position (focus) is matched can be obtained by controlling the image in the same manner as in the conventional imaging device.

[4] Fourth Embodiment FIGS. 13A and 13B are explanatory views of the fourth embodiment.
Further, FIG. 14 is a flowchart of the outline processing of the fourth embodiment.
Further, FIG. 15 is an explanatory view of the window of the fourth embodiment.
The fourth embodiment differs from each of the above embodiments in that the light receiving intensity of the pixel of interest is set to the light receiving intensity of the pixel having the minimum light receiving intensity among the peripheral pixels.

In the following description, with reference to FIG. 1 again, with respect to the pixel of interest (pixel of interest) in the photoelectric conversion unit 21, a window of 3 × 3 pixels (pixels) including eight surrounding pixels (pixels) is displayed. An example will be described in which scanning is performed and image analysis is performed on the pixel of interest.

First, the image pickup apparatus 10 takes a picture in a state of being out of focus (step S31).
Next, as shown in FIG. 13A, the image analysis unit 14 sequentially scans the window of 3 × 3 pixels (step S32), and the smallest of the imaging data corresponding to the eight peripheral pixels corresponding to the pixel of interest. The value (minimum value of light receiving intensity) is set as the value of the imaging data of the pixel of interest (step S33).

Specifically, as shown in FIG. 15, when the values of the imaging data corresponding to the eight peripheral pixels corresponding to the pixel of interest in the imaging data of 3 × 3 pixels (pixels) are a1 to a8, The value C of the imaging data corresponding to the pixel of interest is defined by the following equation using the function MIN that calculates the minimum value.
C = MIN (a1 to a8)

As a result, the value C of the imaging data corresponding to the pixel of interest is set to the minimum value among the imaging data corresponding to the eight peripheral pixels corresponding to the pixel of interest.
Then, the image analysis unit 14 repeats the processes of steps S32 and S33, with all the pixels constituting the captured image as the pixels of interest one by one.

Then, when the processing for all the pixels constituting the captured image is completed, the obtained image is saved in the result storage unit 15 (step S34).

As a result, the value of the imaging data corresponding to the pixel of interest changes sharply with respect to any of the values of the imaging data of the eight adjacent pixels, that is, for example, the incident of a cosmic ray or the pixel. When it is considered that the value is changed due to a defect, the value of the imaging data corresponding to the pixel of interest is regarded as having a low light receiving intensity and is treated as noise.

On the other hand, when the value of the imaging data corresponding to the pixel of interest changes gently with respect to all the values of the imaging data of the eight adjacent pixels, among the imaging data of those eight pixels , The minimum value is saved as the value of the imaging data corresponding to the pixel of interest.

As described above, also in the fourth embodiment, the result image saved in the result storage unit 15 by simple arithmetic processing removes the influence of cosmic ray incidents, pixel defects, etc., and is the original image. It is an image showing the position of the target subject (star, etc.).

In the above description, in the fourth embodiment, as in the first embodiment, the light receiving surface 21F of the photoelectric conversion unit 21 is set as a position shifted from the image point (specifically, a position on the lens 11A side). The edges of the obtained image were blurred (the value [image] of the imaging data was smoothed).

However, as in the second embodiment, the image plane is edged so as to be able to be shifted, the focus is shifted as necessary, and the edge of the obtained image is blurred (smoothing of the value [image] of the imaged data]. The edge of the obtained image is configured so that the lens portion can be shifted along the optical axis, and the focus is shifted as necessary, as in the third embodiment. It is also possible to make it blurry (to smooth the value [image] of the imaging data).

[5] Fifth Embodiment FIG. 16 is a schematic block diagram of the image pickup apparatus of the fifth embodiment.
In FIG. 16, the same parts as those in the first embodiment of FIG. 1 are designated by the same reference numerals.

The difference between the fifth embodiment and the first embodiment is that the light receiving surface of the photoelectric conversion unit 21 constituting the image pickup unit 12 is arranged at the image point of the lens 11A, as in the normal image pickup apparatus. In addition, a filter unit 41 is provided in front of the lens unit 11 to blur the edges of the obtained image by diffusing or refracting the incident light (to smooth the value [image] of the imaging data). be.

FIG. 17 is a configuration explanatory view of the first aspect of the fifth embodiment.
As shown in FIG. 17, an optical low-pass filter LPF that functions as a filter unit 41 is located between the lens 11A and the first focal point FP1 on the optical axis of the lens 11A between the lens 11A and the lens 11A. is doing.

The light receiving surface 21F1 of the photoelectric conversion unit 21 is arranged so as to coincide with the image point.
In the above configuration, in the state where the optical low-pass filter LPF is removed, the light receiving surface 21F1 of the photoelectric conversion unit 21 coincides with the image point of the lens 11A and is in a focused state.

However, when the optical low-pass filter LPF is inserted into the optical path, the incident light is diffused, which has the effect of blurring the edge of the subject.
In this case, as the optical low-pass filter LPF, a known material such as frosted glass or anomalous refraction glass may be used.

Then, by performing the noise removal processing as in each of the above-described embodiments, the result image stored in the result storage unit 15 can be removed from the influences of cosmic ray incidents, pixel defects, etc. by a simple calculation process. Therefore, it becomes an image showing the position of the original target subject (star, etc.).

In the above description, the case where the edge of the subject is blurred by using the optical low-pass filter LPF to smooth the image has been described. However, as in the fourth embodiment, the imaging data corresponding to the pixel of interest has been described. By setting the value to the minimum value among the imaging data corresponding to the peripheral pixels corresponding to the pixel of interest, the edge of the subject can be blurred to smooth the image.

FIG. 18 is a configuration explanatory view of the second aspect of the fifth embodiment.
As shown in FIG. 18, a cross filter CF that functions as a filter unit 41 is located between the lens 11A and the first focal point FP1 on the optical axis of the lens 11A between the lens 11A and the lens 11A. ..

Here, the cross filter CF can be realized by a known technique such as carving a groove on a thin line on the glass surface.

The light receiving surface 21F1 of the photoelectric conversion unit 21 is arranged so as to coincide with the image point.
Even in the above configuration, in the state where the cross filter CF is removed, the light receiving surface 21F1 of the photoelectric conversion unit 21 coincides with the image point of the lens 11A and is in a focused state.

However, when the cross filter CF is inserted into the optical path, for example, incident light has streaks generated horizontally and vertically from the center of the point light source.

FIG. 19 is an explanatory diagram of an example of the captured image of the second aspect of the fifth embodiment.
Similar to FIG. 4, the captured image G21 also includes the imaging data PE-N caused by the incident of cosmic rays, pixel defects, and the imaging data PE-1 corresponding to the actual subject.

As shown in FIG. 19, striations are generated horizontally and vertically from the center of the point light source corresponding to the imaging data PE-1. On the other hand, since the imaging data PE-N is not affected by the cross filter CF, only one pixel has a high light receiving intensity.

Here, the image analysis process according to the second aspect of the fifth embodiment will be described in detail.
FIG. 20 is an explanatory view of the window in the second aspect of the fifth embodiment.
As a window used for scanning in the second aspect of the fifth embodiment, as shown in FIG. 20, a cross-shaped window WD2 is used.

Next, the operation in the second aspect of the fifth embodiment will be described.
FIG. 21 is an outline processing flowchart of the image analysis unit according to the second aspect of the fifth embodiment.
First, the image analysis unit 14 sets the parameters X and Y for specifying the pixel Px (X, Y) as the pixel of interest to 0, which is an initial value (step S41). In the present embodiment, the parameter X is a parameter in the row direction, and the parameter Y is a parameter in the column direction (see FIG. 4 described later).

Next, the image analysis unit 14 determines whether or not the parameter Y exceeds Ymax, which is the maximum value of the parameter Y (step S42). That is, the image analysis unit 14 determines whether or not the processing of all the pixels has been completed.
In the determination of step S42, if the parameter Y exceeds the maximum value Ymax of the parameter Y (step S42; Yes), the image analysis process is terminated.

In the determination of step S42, if the parameter Y has not yet exceeded the maximum value Ymax of the parameter Y (step S42; No), it is determined whether or not the parameter X exceeds the maximum value of the parameter X, Xmax. (Step S43). That is, the image analysis unit 14 determines whether or not the processing of the pixels for one line is completed.

In the determination of step S13, if the parameter X exceeds Xmax, which is the maximum value of the parameter X (step S43; Yes), the image analysis unit 14 sets Y = Y + 1 (step S44) and steps the process again. Move to S42.
In the determination of step S43, if the parameter X has not yet exceeded Xmax, which is the maximum value of the parameter X (step S43; No), the image analysis unit 14 sequentially scans the cross-shaped window WD2, respectively. The value of the image pickup data corresponding to the five pixels is acquired (step S41).
Subsequently, the image analysis unit 14 searches for and acquires the minimum value of the imaging data among the five imaging data corresponding to the window WD2 (step S46).

Subsequently, the image analysis unit 14 compares the minimum value of the imaging data acquired in step S46 with the predetermined threshold value data D (step S47).

As a result, in the comparison of step S47, if the minimum value of the imaging data acquired in step S46 is equal to or less than the threshold data D (step S47; No: minimum value of imaging data ≤ D), the imaging data of the pixel of interest. Is treated as noise and clamped to be treated as imaging data of black spots (light receiving intensity (luminance) level lowest) (step S48).

Then, the value of the imaging data after clamping is output as the pixel value of the pixel of interest, and is stored in the result storage unit 15 (step S49).

On the other hand, in the comparison of step S47, when the minimum value of the imaging data acquired in step S46 exceeds the threshold data D (step S47; Yes: minimum value of imaging data> D), the pixel of interest The value of the imaging data is output as the pixel value of the pixel of interest and stored in the result storage unit 15 (step S49).

As a result, the value of the imaging data corresponding to the pixel of interest changes sharply with respect to any of the values of the imaging data of the four adjacent pixels, that is, due to the incident of cosmic rays or pixel defects. When it is considered that the value is changed, the value of the imaging data corresponding to the pixel of interest is treated as noise.

On the other hand, the values of the imaging data corresponding to the pixel of interest are considered to be gradual changes with respect to all the imaging data values of the four adjacent pixels, that is, the imaging data corresponding to the actual captured image. If this is the case, the value of the imaging data corresponding to the pixel of interest will be saved as it is.

FIG. 22 is an explanatory diagram of the result image in the second aspect of the fifth embodiment.
As a result, even in the second aspect of the fifth embodiment, the result image stored in the result storage unit 15 is an image obtained by removing the effects of cosmic ray incidents, pixel defects, etc., as shown in FIG. Become.

That is, by effectively detecting the striations generated by the cross filter CF, the image analysis unit 14 can easily distinguish between the captured image of the subject and the bright spot due to noise.

In the above description of the second aspect of the fifth embodiment, a case similar to that of the first embodiment has been described as a method of determining the center pixel value using the cross filter CF, but the first aspect shown in FIG. 9 has been described. It is also possible to apply the process in the modified example of the embodiment or the process in the fourth embodiment shown in FIG. 14 to determine the center pixel value.

FIG. 23 is an explanatory diagram when the origin of the image is scanned by a cross-shaped window having a 5-pixel configuration.
In the above description, the case where the target pixels exist in all of the cross-shaped window WD2 has been described, but when scanning the peripheral portion of the captured image, the processing target pixels are present in a part of the cross-shaped window WD2. It may not exist.

In the case of the example of FIG. 23, the processing target pixel does not exist in two pixels of the cross-shaped window WD2.
In such a case, for example, the process is performed by any of the following methods.
(1) The portion where the processing target pixel does not exist is ignored, and the processing is performed only on the portion where the processing target pixel exists. More specifically, in the case of the example of FIG. 23, the evaluation is performed only with the three pixels in which the processing target pixel exists.
(2) For the portion where the processing target pixel does not exist, the value is set to 0 and processing is performed as usual.
(3) Copying and processing the pixel values of pixels adjacent to each other in the column direction or the row direction More specifically, in the case of the example of FIG. 23, the pixel values of the pixels Px (0,0) are copied respectively. It will be evaluated.

By adopting the above configuration, processing can be reliably performed even when scanning the peripheral portion of the captured image.

In the above description, the size of the cross-shaped window WD2 has a 5-pixel configuration, but it is similar to a 9-pixel configuration, a 13-pixel configuration, or a 5-pixel configuration in which the vertical and horizontal lengths are increased according to the image resolution. It is also possible to adjust the size to a 20-pixel configuration of the shape. As a modified example of the cross shape, it is also possible to have a square-shaped 9-pixel configuration at the center and a 13-pixel configuration in which one pixel is added vertically and horizontally.

On the contrary, if the size of the subject is sufficiently larger than the window size, it is possible to reduce the size of the image and resize it according to the window size. As a result, even if the image has a large number of pixels, the number of scans can be the same as that of an image having a small number of pixels, and the processing can be simplified or speeded up.

FIG. 24 is a schematic block diagram of an image pickup apparatus according to a modified example of the fifth embodiment.
In FIG. 24, the same parts as those in the fifth embodiment of FIG. 16 are designated by the same reference numerals.

The modified example of the fifth embodiment is different from the fifth embodiment in that the filter unit 41 can be inserted into and removed from the optical path of the incident light so that the optical axis of the lens 11A of the lens unit 11 passes through the filter unit 41. The point is that the filter control unit 42 that drives the lens is provided.

In this case, the filter control unit 42 sets the inside of the filter unit 41 to the outside of the optical path of the lens 11A, so that the image pickup apparatus 40A can acquire a focused image as a captured image as in the case of a normal camera. It will be possible.

In the above description, the filter unit 41 is inserted and removed from the optical path of the incident light by the filter control unit 42, but it is for measuring the optical low-pass filter LPF, the cross filter CF, etc. (for position measurement). It is also possible to switch between the filter and a normal optical filter (for example, an infrared filter, an ND filter, etc.) and insert the filter into the optical path.

As described above, according to each embodiment, the incident of cosmic rays from the outside and the influence of defective pixels of the image sensor can be removed by a simple process, and an image of the subject to be imaged can be obtained. Is possible.

[6] Modifications of the Embodiment The embodiment of the present technology is not limited to the above-described embodiment, and various changes can be made without departing from the gist of the present technology.

For example, in the second aspect of the first embodiment and the fifth embodiment, the adjustment of the size of the window WD or the window WD2 and the adjustment of the image size have been described, but the modified example of the first embodiment and the second embodiment have been described. , The third embodiment, the fourth embodiment, and the first aspect of the fifth embodiment can be similarly applied.

Further, in the second aspect of the first embodiment and the fifth embodiment, the case where the target pixel exists in all the regions of the window WD or the window WD2 has been described. Also in the first embodiment of the embodiment, the third embodiment, the fourth embodiment, and the fifth embodiment, when scanning the peripheral portion of the captured image, the window WD or a part of the window WD2 having 3 × 3 pixels is used. It is also possible to configure the above-described methods (1) to (3) to be similarly applied to the case where the processing target pixel does not exist. By adopting such a configuration, processing can be reliably performed even when scanning the peripheral portion of the captured image.

Further, in the second embodiment, the image pickup surface is configured to be able to be shifted, and by shifting the focus as necessary, in the third embodiment, the lens portion is moved along the optical axis. It is configured so that it can be shifted, and by shifting the focus as necessary, the edges of the obtained image can be blurred (the image is smoothed).

However, instead of these, as in the fourth embodiment, the value of the imaging data corresponding to the pixel of interest is set to the minimum value among the imaging data corresponding to the peripheral pixels corresponding to the pixel of interest, thereby setting the edge of the subject. It is also possible to blur the image so as to smooth the image.

Further, the present technology can also have the following configurations.
(1)
A lens unit that collects incident light from the subject,
An imaging unit that captures the incident light collected by the lens unit in an unfocused state, and an imaging unit.
The image captured by the imaging unit is analyzed, and the pixel value of the pixel of interest is set based on the pixel values of a plurality of predetermined peripheral pixels located around one pixel of interest to specify the position of the subject. An image analysis unit that generates a result image for
Imaging device equipped with.
(2)
When the minimum value of the pixel value of a plurality of predetermined peripheral pixels exceeds a predetermined threshold value, the image analysis unit sets the pixel value of the pixel of interest to the minimum value.
When the minimum value is equal to or less than the threshold value, the pixel value of the pixel of interest is set to a predetermined low light receiving intensity value.
(1) The imaging device according to the above.
(3)
The image analysis unit sets the minimum value of the pixel values of a plurality of predetermined peripheral pixels as the pixel values of the pixel of interest.
(1) The imaging device according to the above.
(4)
The image analysis unit performs binarization processing for classifying the pixel values constituting the captured image into high light receiving intensity and low light receiving intensity, and classifies the pixel values of a plurality of predetermined surrounding pixels into the low light receiving intensity. When the pixel value to be used is included, the pixel value of the pixel of interest is set to the low light receiving intensity value.
(1) The imaging device according to the above.
(5)
The relative arrangement position of the lens unit and the imaging unit is set to a position in the imaging unit that allows imaging in an out-of-focus state.
The imaging device according to any one of (1) to (4).
(6)
A lens control unit that drives the lens unit in the optical axis direction and sets the relative arrangement position of the lens unit and the imaging unit to a position in the imaging unit that enables imaging in an out-of-focus state. With,
(5) The imaging device according to the above.
(7)
An imaging control unit that drives the imaging unit in the optical axis direction and sets the relative arrangement position of the lens unit and the imaging unit to a position in the imaging unit that enables imaging in an out-of-focus state. With,
(5) The imaging device according to the above.
(8)
A filter unit provided with the lens unit and a filter unit inserted into the optical path between the subjects to smooth the obtained image.
The imaging device according to any one of (1) to (4).
(9)
The filter unit is configured as an optical low-pass filter.
(8) The imaging device according to the above.
(10)
A filter unit configured as a cross filter that is inserted into an optical path between the lens unit and the subject to generate striations is provided.
The imaging device according to any one of (1) to (4).
(11)
A filter unit is provided which is inserted into the optical path between the lens unit and the subject so that the incident light after passing through the lens unit is out of focus.
The imaging device according to any one of (1) to (4).
(12)
The filter unit is provided with a filter control unit that can be inserted into and removed from the optical path to enable imaging in the out-of-focus state.
The imaging device according to any one of (8) to (11).
(13)
In a control method of an imaging device having a lens unit that collects incident light from a subject and an imaging unit that captures the incident light collected by the lens unit in an out-of-focus state.
A process of analyzing a captured image and repeatedly performing a process of setting a pixel value of the pixel of interest based on the pixel values of a plurality of predetermined peripheral pixels located around one pixel of interest while updating the pixel of interest. When,
A process of generating a result image for specifying the position of the subject based on the set pixel values corresponding to a plurality of pixels of interest, and a process of generating a result image.
A control method for an image pickup device equipped with.
(14)
In the process of setting the pixel value of the pixel of interest, when the minimum value of the pixel value of a plurality of predetermined peripheral pixels exceeds a predetermined threshold value, the pixel value of the pixel of interest is set to the minimum value. The process and
A process of setting the pixel value of the pixel of interest to a predetermined low light receiving intensity value when the minimum value is equal to or less than the threshold value.
The control method of the image pickup apparatus according to (13).
(15)
The process of setting the pixel value of the pixel of interest is the process of setting the minimum value of the pixel value of a plurality of predetermined peripheral pixels as the pixel value of the pixel of interest.
The control method of the image pickup apparatus according to (13).
(16)
The process of setting the pixel value of the pixel of interest includes a process of binarizing the pixel values constituting the captured image into high light receiving intensity and low light receiving intensity.
A process of setting the pixel value of the pixel of interest to the low light-receiving intensity value when the pixel values of a plurality of predetermined peripheral pixels include pixel values classified into the low light-receiving intensity.
The control method of the image pickup apparatus according to (13).

10, 10A, 10B, 40A Imaging device 11 Lens unit 11A Lens 12 Imaging unit 13 Image storage unit 14 Image analysis unit 15 Result storage unit 21 Photoelectric conversion unit 21F0 Image point 21F, 21F1 Light receiving surface 22 Analog amplifier unit 23 AD conversion unit 31 Imaging control unit 32 Lens control unit 41 Filter unit 42 Filter control unit CF Cross filter D Threshold data LPF Optical low-pass filter OBJ Subject PE-1 Imaging data PE-N Imaging data (noise)
WD, WD2 window

Claims (12)

1. A lens unit that collects incident light from the subject,
   An imaging unit that captures the incident light collected by the lens unit in an unfocused state, and an imaging unit.
   The image captured by the imaging unit is analyzed, and the pixel value of the pixel of interest is set based on the pixel values of a plurality of predetermined peripheral pixels located around one pixel of interest to specify the position of the subject. An image analysis unit that generates a result image for
   Imaging device equipped with.
2. When the minimum value of the pixel value of a plurality of predetermined peripheral pixels exceeds a predetermined threshold value, the image analysis unit sets the pixel value of the pixel of interest to the minimum value.
   When the minimum value is equal to or less than the threshold value, the pixel value of the pixel of interest is set to a predetermined low light receiving intensity value.
   The imaging device according to claim 1.
3. The image analysis unit sets the minimum value of the pixel values of a plurality of predetermined peripheral pixels as the pixel values of the pixel of interest.
   The imaging device according to claim 1.
4. The image analysis unit performs binarization processing for classifying the pixel values constituting the captured image into high light receiving intensity and low light receiving intensity, and classifies the pixel values of a plurality of predetermined surrounding pixels into the low light receiving intensity. When the pixel value to be used is included, the pixel value of the pixel of interest is set to a value corresponding to the low light receiving intensity.
   The imaging device according to claim 1.
5. The relative arrangement position of the lens unit and the imaging unit is set to a position in the imaging unit that allows imaging in an out-of-focus state.
   The imaging device according to any one of claims 1 to 4.
6. A lens control unit that drives the lens unit in the optical axis direction and sets the relative arrangement position of the lens unit and the imaging unit to a position in the imaging unit that enables imaging in an out-of-focus state. With,
   The imaging device according to claim 5.
7. An imaging control unit that drives the imaging unit in the optical axis direction and sets the relative arrangement position of the lens unit and the imaging unit to a position in the imaging unit that enables imaging in an out-of-focus state. With,
   The imaging device according to claim 5.
8. A filter unit provided with the lens unit and a filter unit inserted into the optical path between the subjects to smooth the obtained image.
   The imaging device according to any one of claims 1 to 4.
9. The filter unit is configured as an optical low-pass filter.
   The imaging device according to claim 8.
10. A filter unit configured as a cross filter that is inserted into an optical path between the lens unit and the subject to generate striations is provided.
    The imaging device according to any one of claims 1 to 4.
11. The filter unit is provided with a filter control unit that can be inserted into and removed from the optical path to enable imaging in the out-of-focus state.
    The imaging device according to any one of claims 8 to 10.
12. In a control method of an imaging device having a lens unit that collects incident light from a subject and an imaging unit that captures the incident light collected by the lens unit in an out-of-focus state.
    A process of analyzing a captured image and repeatedly performing a process of setting a pixel value of the pixel of interest based on the pixel values of a plurality of predetermined peripheral pixels located around one pixel of interest while updating the pixel of interest. When,
    A process of generating a result image for specifying the position of the subject based on the set pixel values corresponding to a plurality of pixels of interest, and a process of generating a result image.
    A control method for an image pickup device equipped with.

Images