JP7123384B2 - Image signal processor - Google Patents

Description

The present invention relates to an image signal processing device for light incident on a photoelectric conversion unit in an imaging device having a plurality of photoelectric conversion units in one pixel.

Imaging devices such as CCD image sensors and CMOS image sensors are used in general digital cameras and digital video cameras. These imaging elements are composed of a plurality of pixels. Among the plurality of pixels, a unit pixel is mainly composed of a microlens, a photodiode as a photoelectric conversion section, and wiring, and the photoelectric conversion section may have a plurality of photoelectric conversion elements.

Therefore, when the photoelectric conversion unit has a plurality of photoelectric conversion elements, there is a difference in geometrical positional relationship between the microlens and each photoelectric conversion element in the unit pixel.

FIG. 1 is a cross-sectional view of an imaging device. As shown in FIG. 1, there is a difference in the optical path of the light passing through the microlens located on the subject side in the upper part of FIG. Occur. Specifically, the positional relationship with the wiring layer changes between the photoelectric conversion section on the right side and the photoelectric conversion section on the left side. Therefore, in the case of an oblique ray having an incident angle, part of the luminous flux may be blocked by the wiring layer and may not reach the photoelectric conversion section.

Since the photoelectric conversion unit converts only the light that reaches it into an image signal, part of the light flux blocked by the wiring layer is not converted into an image signal. In other words, in the photoelectric conversion units of the same color located nearby, all the light that has passed through the microlens reaches the photoelectric conversion units without being blocked, and part of the light flux reaches the photoelectric conversion units because it is blocked by the wiring layer. A difference occurs in the image signal after conversion between the photoelectric conversion unit and the photoelectric conversion unit that did not exist.

As a result, among photoelectric conversion units of the same color located nearby, the image signals converted by the photoelectric conversion units, which are shielded by the wiring layer and part of the light flux does not reach, are reduced, and the image signals of the adjacent image signals of the same color are reduced. appears as a signal difference of In addition, since the amount of light flux blocked by the wiring layer also changes depending on the incident angle of the light incident on the photoelectric conversion section, the signal difference (ratio) between adjacent pixels of the same color is not constant. If this signal difference is converted into an image without being processed, it will be recognized as an edge and will not be processed as noise, but will appear as noise.

Japanese Patent Application Laid-Open No. 2002-200000 uses an image pickup device in which color filters are arranged in a Bayer array in a single-plate type image pickup device. This imaging element has a problem that the sensitivity of each pixel changes due to minute differences in the pixel structure as the pixels become finer. Furthermore, due to this problem, if the demosaic processing is performed as it is, there is a problem that a flat portion with little luminance change is erroneously determined to be an edge. As a method for dealing with these problems, Japanese Patent Application Laid-Open No. 2002-200000 discloses a method of using a correction term for a pixel.

Japanese Patent No. 5672776

However, in the image processing apparatus disclosed in Japanese Patent Laid-Open No. 2002-200010, since the correction term used for pixel correction is weighted according to the distance of the surrounding pixel values, the average difference data is sought. Therefore, when detailed information with different characteristics is included in the surrounding area, the detailed information may disappear or become different due to the addition of an incorrect additional correction value because the additional correction value may not be calculated correctly. There is a problem that there is a case that it becomes.

The present invention has been made in view of such a situation, and the signal difference between the image signals converted by the photoelectric conversion element that receives the incident light, part of the luminous flux of which is blocked by the structure of the imaging device such as the wiring layer, is taken as noise. It is an object of the present invention to provide an image signal processing apparatus for discriminating and reducing noise components.

A first invention, which is a means for solving the above problems, has a plurality of photoelectric conversion units in one pixel for converting a light beam that has passed through an optical system into an image signal, and further has a structure on the path of the light beam of the optical system. In an image signal processing device for processing an image signal obtained from an imaging device having an object, for the image signal, a filter unit for smoothing the image signal of the pixel of interest according to the surrounding image signal, and the same position as the structure. an extraction unit for extracting the image signal for each of the image signals having a relationship; a comparison unit that compares the characteristics to obtain a degree of relevance; and a mixing unit for mixing.

A second invention, which is a means for solving the above-described problems, is the image signal processing method according to the first invention, wherein the filter unit has a strength to suppress a pattern formed by the structure. An image signal processing device characterized by being a low-pass filter or noise reduction of .

A third aspect of the invention, which is means for solving the above-described problems, is the image signal processing method according to the first aspect or the second aspect, wherein the mixing unit comprises an output result of the comparing unit An image signal processing device , comprising a filter for averaging.

According to the present invention, the signal difference between the image signals converted by the photoelectric conversion element into which the light with a part of the luminous flux is blocked by the structure of the imaging element such as the wiring layer is discriminated as noise, and the noise component is reduced. An image signal processing device can be provided.

1 is a cross-sectional view of a solid-state imaging device according to an embodiment of the present invention; 1 is a block diagram showing the configuration of a solid-state imaging device according to an embodiment of the present invention; FIG. 4 is a flow chart when performing noise reduction according to an embodiment of the present invention; FIG. 4 is a conceptual diagram for extracting an image signal converted by the photoelectric conversion unit according to the embodiment of the present invention; Image signal to which each signal processing according to the embodiment of the present invention is applied FIG. 4 is a cross-sectional view of a solid-state imaging device different from the embodiment of the present invention;

BEST MODE FOR CARRYING OUT THE INVENTION The best mode for carrying out the present invention will be described below with reference to the accompanying drawings. It should be noted that the present invention is not limited by this embodiment.

101, a solid-state imaging device; 102, a CPU; 1011, a microlens; 1012, a photoelectric conversion section; 1013, a wiring layer; An output section, 1023 a comparison section, and 1024 a mixing section.

FIG. 1 is a cross-sectional view of a solid-state imaging device 101 of this embodiment. It should be noted that FIG. 1 is simplified for ease of explanation of the present invention. In FIG. 1, the upper side is the object side, and the lower side is the image plane side.

In the solid-state imaging device 101 used in the embodiment of the present application, four photoelectric conversion units 1012 are arranged under one microlens 1011 when viewed from the object side. FIG. 1 is a cross-sectional view of a microlens 1011 divided at a position where two photoelectric conversion units 1012 are included.

Let us consider the incidence of light rays (a) and (b) as shown in FIG. In the case of the light ray (a), there is no difference between the light beams incident on the right (R in the drawing) and the left (L in the drawing) photoelectric conversion units 1012 . However, in the case of a light ray having an angle with respect to a line perpendicular to the photoelectric conversion unit 1012, such as the light ray (b), a difference occurs between the light beams incident on the right photoelectric conversion unit 1012R and the left photoelectric conversion unit 1012L. This is because the positional relationship between the wiring layer 1013 and the photoelectric conversion section 1012 is a problem in the case of a light ray incident from the left side like the light ray (b). In the photoelectric conversion section 1012L where the wiring layer 1013 is located on the left side, the light beam is blocked by the wiring layer 1013. FIG. On the other hand, the light beam reaches the photoelectric conversion unit 1012R, in which the wiring layer 1013 is located on the right side, without any problem. An image signal converted into an image signal by the photoelectric conversion unit 1012L whose light is blocked by the wiring layer 1013 is different from the original image signal.

Next, FIG. 2 is a configuration diagram showing the configuration of the solid-state imaging device of this embodiment.

The lens barrel 100 is detachable from the imaging device 10, and the mount portion of the lens barrel 100 and the mount portion of the camera body are connected by a bayonet mechanism or the like. In addition, it may be fixed to the camera body.

The solid-state imaging device 101 includes microlenses 1011, photoelectric conversion units 1012, wiring layers 1013, and the like.

The microlens 1011 is a lens that collects the light flux incident on the lens barrel 100 so that each photoelectric conversion unit 1012 is suitable for receiving light.

A photoelectric conversion unit 1012 converts the light flux that has passed through the microlens 1011 into an image signal.

The embodiment of the present application uses the solid-state imaging device 101 having four photoelectric conversion units 1012 in one pixel.

The wiring unit 1013 transfers the image signal converted by the photoelectric conversion unit 1012 from each photoelectric conversion unit 1012 to each unit such as the CPU 102 .

The CPU 102 performs various processes on the image signal converted by the photoelectric conversion unit 1012 and transferred through the wiring layer 1013 .

The filter unit 1021 applies a low-pass filter to the image signal converted by the photoelectric conversion unit 1012 and transferred through the wiring layer 1013 . The image signal to which the low-pass filter has been applied is sent to the mixing section 1024 . It should be noted that there is no problem with noise reduction or the like as long as a specific image signal is smoothed according to surrounding image signals.

The extraction unit 1022 organizes image signals converted by the photoelectric conversion units 1012 and transferred through the wiring layer 1013 by image signals converted by the photoelectric conversion units 1012 having the same positional relationship with the wiring layer 1013 .

For each image signal extracted from the extraction unit 1022, the comparison unit 1023 compares the pixel of interest with its surrounding pixels, obtains the relevance of the pixel of interest to its surroundings, and calculates the relevance of each extracted pixel. Calculate the degree of relevance based on A specific comparison method will be described later.

The mixing unit 1024 mixes the image signal converted by the photoelectric conversion unit 1012 and the image signal to which the low-pass filter is applied by the filter unit 1021 based on the degree of association obtained by the comparison unit 1023 .

Next, an embodiment of the image signal processing method according to the present invention will be described with reference to the flow chart of FIG.

In step #1, as shown in FIG. 1, the light flux passing through the lens barrel 1 reaches the solid-state imaging device 101, is guided to the photoelectric conversion section 1012 by the microlens 1011, and is then converted into an image signal.

At step # 2 , the image signal converted from the light flux by the photoelectric conversion unit 1012 is transferred to the filter unit 1021 through the wiring unit 1013 . A low-pass filter is applied to the transferred image signal. The image signal to which the low-pass filter has been applied is temporarily stored in a memory unit (not shown).

In step #3, the extraction unit 1022 extracts image signals having the same structure as the photoelectric conversion unit 1012 and its surroundings from the image signals transferred from the photoelectric conversion unit 1012 through the wiring unit 1013, and organizes the image signals. A specific arrangement method will be described later.

At step #4, the comparison unit 1023 compares the image signals extracted and organized by the extraction unit 1022 at step #3. A specific comparison method will be described later.

In step #5, the image signal converted by the photoelectric conversion unit 1012 and the filter unit 1021 stored in the memory unit (not shown) in step #2 are combined based on the degree of association obtained by the comparison unit 1023 in step #4. Mix the image signal to which the low-pass filter is applied at .

Next, the extracting method of the extracting unit 1022 in step #3 and the subsequent method of arranging the image signals will be described with reference to FIG.

FIG. 4 is a conceptual diagram of an image signal converted by the photoelectric conversion unit 1012. As shown in FIG. Specifically, FIG. 4(a) shows an image signal in which a light flux that has passed through the lens barrel 100 reaches the solid-state imaging device 101 and is converted by each photoelectric conversion unit 1012, and FIG. It is an image signal extracted by an extraction unit 1022 .

The photoelectric conversion unit 1012 in FIG. 4A is one square in the figure. Frames around the squares are structures and represent wiring layers 1013 for transferring image signals converted by the photoelectric conversion unit 1012 to the CPU 102 for various image signal processing.

Here, in FIG. 4A, the embodiment of the present application uses a solid-state imaging device 101 having four photoelectric conversion units 1012 in one pixel. Furthermore, a wiring layer surrounds one pixel. Characters such as “LU” and “RD” are entered for each photoelectric conversion unit 1012 . This shows the positional relationship between each photoelectric conversion unit 1012 and the wiring layer 1013 . The photoelectric conversion unit 1012 having the upper and left wiring layers is indicated as LU, and the photoelectric conversion unit 1012 having lower and right wiring layers is indicated as RD. Similarly, the photoelectric conversion unit 1012 whose wiring layers are located on the lower side and left side is denoted by LD, and the photoelectric conversion unit 1012 whose wiring layers are located on the upper side and on the right side is denoted by RU.

A wiring layer 1013 is positioned around each pixel in the same manner.

In this case, the photoelectric conversion unit 1012 in one pixel has the same positional relationship with the wiring layer 1013 as long as the arrangement in the pixel is the same. Therefore, if the above-described characters of the photoelectric conversion unit 1012 in FIG. 4A are the same, the layout of the wiring layer 1013 in each pixel is also the same.

Next, FIG. 4B will be described.

In FIG. 4(b), at step #1, each photoelectric conversion unit 1012 converts into an image signal as shown in FIG. 4(a). After that, image signals are extracted for each photoelectric conversion unit 1012 having the same positional relationship with the wiring layer 1013 . The photoelectric conversion units 1012 having the wiring layer 1013 in the same positional relationship with the photoelectric conversion units 1012 are collected (hereinafter referred to as a frame). In the embodiment of the present application, a solid-state imaging device having four photoelectric conversion units 1012 per pixel is used, so frames are divided into four patterns of LU, LD, RU, and RD.

Step #4 will be explained.

In step #3, the extraction unit 1022 extracts image signals from the image signals converted from light by the photoelectric conversion units 1012 in FIG. A target pixel is determined, and the comparison unit 1023 compares the target pixel with surrounding pixels.

Specifically, the comparison method of the comparison unit 1023 will be described below with the colored pixel positioned second from the left and second from the top in FIG. 4A as a pixel of interest. First, attention is focused on the LU of the target pixel, and the difference from the LU of the pixel positioned above in the frame of the LU in the state of FIG. 4B from which the LU is extracted is calculated. Next, the difference between the pixel of interest and the LU of the pixel positioned below is calculated. Furthermore, the difference between the LU of the pixel of interest and the LU of the pixel of interest positioned to the left and right of the pixel of interest is similarly calculated. As a result, the signal difference between the pixel of interest and the pixels located in each direction is compared, and the direction in which the difference between the LU of the pixel of interest and the LU of the peripheral pixels is the largest is defined as Ea.

Next, focus on the RU of the pixel of interest. In the frame of the RU in the state of FIG. 4(b) where the RU is extracted, the difference between the RU of the pixel of interest and the RU of the pixel positioned above is calculated in the same way as the previous LU. Calculate the difference between the pixel and the RU of the pixel located below. After that, similarly, the difference between the pixels positioned to the left and right of the pixel of interest and the RU is calculated, and the direction in which the difference between the RU of the pixel of interest and the surrounding pixels is the largest is defined as Eb.

LD and RD of the target pixel are similarly calculated for the upward, downward, leftward, and rightward directions, and the directions in which the difference between the target pixel and the surrounding pixels is the largest are defined as Ec and Ed, respectively.

For each frame, Ea to Ed, which are directions in which there is a large difference between the target pixel and the surrounding pixels, are related. Based on the relevance of these Ea to Ed, the degree of relevance, which is the degree of matching, is obtained.

While changing the target pixel, the comparison unit 1023 performs the above-described determination of the relationship between the target pixel and the peripheral pixels for all the pixels of the solid-state imaging device.

In this embodiment, the pixel of interest and the surrounding pixels are compared with each other in the four directions of the pixel of interest, namely, the four directions of the pixel of interest. The comparison may be made with respect to the target pixel in the upper right, lower right, upper left, and lower left directions.

Advantages of comparing oblique directions of the pixel of interest will be described later.

Further, in this embodiment, as a result of comparing the signal differences between the pixel of interest and the peripheral pixels, only the directions in which the difference between the pixel of interest and the peripheral pixels was the largest were set as Ea, Eb, . . . There is no problem if the signal difference, which is the difference between the signal values, is also stored. A specific usage of the signal difference will be described later.

Step #5 will be described below.

In step #5, the image signal generated by the photoelectric conversion unit 1012 and the low-pass filter of the filter unit 1021 in step #2 are obtained based on the degree of association determined based on the relationship obtained by the comparison unit 1023 in step #4. is mixed with the image signal to which is applied in the mixing unit 1024 .

Specifically, in step #5 of this embodiment, when the degree of association obtained in step #4 is high, that is, when Ea=Eb=Ec=Ed, a low-pass filter is applied in step #2. The ratio of the image signal and the image signal converted by the photoelectric conversion unit 1012 is assumed to be 100:0. That is, when the degree of association is high, the image signal to which the low-pass filter is applied is applied.

Here, the reason for replacing with an image signal to which a low-pass filter is applied when the degree of association is high will be described. In the image signal converted by the photoelectric conversion unit 1012, when the direction of the largest difference when comparing the target pixel and the surrounding pixels is the same, the possibility of gradation is extremely high. Therefore, in this embodiment, the image signal is replaced with an image signal to which a low-pass filter is applied for smoothing.

Further, when the degree of association is low, that is, when Ea=Eb=Ec=Ed is not satisfied, the ratio of the image signal to which the low-pass filter is applied in step #2 and the image signal converted by the photoelectric conversion unit 1012 is 0: 100 mixed image signal. That is, when the degree of association is low, the image signal converted by the photoelectric conversion unit 1012 is applied.

This is because when the degree of association is low, the image signal converted by the photoelectric conversion unit 1012 is highly likely to be detailed information because it can be determined that the edges are in different directions. Therefore, the image signal is used as it is without applying a low-pass filter for smoothing.

Furthermore, when the diagonal direction is compared in step #4, the direction of the edge can be determined more accurately, so that detailed information and noise can be discriminated more accurately.

Further, by using the signal difference obtained in step #4, when the signal difference in all directions is large, the pixel of interest is highly likely to be an edge. Therefore, in step #5, when the image signal to which the low-pass filter of the filter unit 1021 is applied and the image signal converted by the photoelectric conversion unit 1012 are mixed, the ratio of the image signal to which the low-pass filter is applied is reduced. If the signal differences in all directions are extremely small, there is a high possibility that the pixel of interest is a plane. Therefore, in step #5, when the image signal to which the low-pass filter is applied by the filter unit 1021 and the image signal converted by the photoelectric conversion unit 1012 are mixed, the proportion of the image signal to which the low-pass filter is applied is increased. Also, if there is variation in the signal difference in each direction, there is a possibility that the edge cannot be correctly recognized.

In the embodiment of the present application, the mixing unit 1024 employs a ratio of either 0 or 100 image signals, but it is needless to say that the ratio may be changed as appropriate.

Further, in the embodiment of the present application, since a solid-state imaging device having four photoelectric conversion units 1012 per pixel is used, four differences Ea to Ed between each pixel of interest and peripheral pixels are obtained. In the above-described embodiment, the ratio in the mixing section 1024 was changed depending on whether or not all of Ea to Ed were present. However, when three of the four directions of Ea to Ed are aligned, an image signal obtained by mixing the image signal to which the low-pass filter is applied and the image signal converted by the photoelectric conversion unit 1012 at a ratio of 75:25, When the two directions are aligned, the image signal may be similarly mixed at 50:50. In this manner, the mixing ratio may be changed according to the degree of association between the target pixel and the peripheral pixels of each photoelectric conversion unit 1012 of one pixel.

Furthermore, a low-pass filter may be applied to the degree of association obtained in step #4. By applying a low-pass filter to the mixing ratio of each pixel, the mixing ratio of the image signal to which the low-pass filter has been applied by the filter unit 1021 and the image signal converted by the photoelectric conversion unit 1012 becomes gentle, thereby rapidly converting the image signal. avoid.

By using a low-pass filter to moderate the mixing ratio of image signals, an image in which noise is eliminated while maintaining a sense of resolution is obtained. This is because, for adjacent pixels, when the mixture ratio of the image signal to which the low-pass filter is applied by the filter unit 1021 and the image signal converted by the photoelectric conversion unit 1012 repeats 100 and 0, the final image contains noise. is undesirable because it appears as an artifact similar to

Furthermore, by incorporating a Sobel filter or a Prewitt filter in the comparison unit 1023 and applying the filter to the image signal of each frame, the direction and strength of the edge in each frame are detected. It is also possible to detect the direction and strength of each edge and calculate the relevance based on the direction and edge strength.

Based on the degree of association, as described above, the image signal to which the low-pass filter is applied by the filter unit 1021 and the image signal converted by the photoelectric conversion unit 1012 are mixed to obtain the image signal of each pixel.

The low-pass filter used in the filter unit 1021 to be used in the mixing by the mixing unit 1024 is a filter for moderating the characteristics of the image signal converted by the photoelectric conversion unit 1012 due to structural factors. Therefore, since the purpose is to equalize the signal value of a specific pixel by using the signal values of surrounding pixels, there is no problem even if noise reduction or the like is used in addition to the low-pass filter.

Each figure of FIG. 5 is demonstrated. Figures 5-1 and 5-2 show image signals at different points, respectively. Figure 5-1 is the gradation part where noise should be reduced, and Figure 5-2 has many structures. This is an example of a subject whose resolution should be maintained. Note that the description of the processing method is common to both figures. FIG. 5A shows an image signal converted by the photoelectric conversion unit 1012. FIG. FIG. 5B shows an image signal to which a low-pass filter has been applied by the filter section 1021. FIG. FIG. 5(c) shows the comparison result of step #4. FIG. 5D shows the image signal of FIG. 5A converted by the photoelectric conversion unit 1012 based on the comparison result of step #4 and the image signal of FIG. The image signal mixed by the mixing unit 1024 is shown. FIG. 5E shows an image signal obtained by applying a low-pass filter adjusted to match the resolution of the output image, unlike the filter unit 1021, to the image signal converted by the photoelectric conversion unit 1012. FIG.

There is little noise in FIG. 5-1(b), but there is no sense of resolution in FIG. 5-2(b). On the other hand, there is a sense of resolution in FIG. 5-2(e), but there is much noise in FIG. 5-1(e). On the other hand, the noise is reduced in FIG. 5-1(d), and the sense of resolution remains in FIG. 5-2(d).

As a result of comparison in step #4 in FIG. 5(c), white pixels are judged to have a high degree of relevance. Then, the image signal to which the low-pass filter is applied is used. In addition, black pixels are portions where the degree of association is determined to be low, that is, when the direction in which the difference between the pixel of interest and the surrounding pixels is large in each frame is not aligned. Use As a result, the image shown in FIG. 5(d) is obtained.

By mixing the image signals as in the present invention, the output image in FIG. 5D is equivalent to the image signal in FIG. It is possible to obtain an image with less noise while maintaining the sense of resolution. This is because, as described in the embodiment of the present application, the present invention compares the image signal extracted in step #3 with the pixel of interest and the surrounding pixels for each frame in step #4. The mixture of image signals in step #5 is changed according to the degree of association with the pixel. At this time, if the degree of association is high, there is a high possibility that the target pixel is a gradation. Therefore, the image signal of the portion with high degree of relevance is replaced with the image signal to which the low-pass filter is applied. Also, when the degree of association is low, the pixel of interest often has detailed information, so it is possible to leave the image signal converted by the photoelectric conversion unit 1012 . As a result, an image that maintains a sense of resolution while reducing noise is obtained.

In the case of gradation, the change in the incident light flux is constant, but the light flux converted by the photoelectric conversion unit 1012 is affected by differences in structures. In this case, if the adjacent structures are different, the image signal converted by the photoelectric conversion unit 1012 will be different, and the gradation relationship will be lost. On the other hand, since photoelectric conversion units 1012 having the same structure can obtain the same characteristics, the gradation relationship is maintained. FIG. 5-1 shows an object having a gradation portion. From the image signal extracted in step #3, by obtaining the relationship in step #4, a portion with a matching relationship is selected as a gradation portion, and step In #5, the image is obtained by applying a low-pass filter to the gradation portion. Therefore, the amount of noise in FIG. 5-1(d) is close to that in FIG. 5-2(b).

When there is a large amount of detailed information in different directions, the incident light flux is not constant, so it is not known how much the structure will affect the light flux converted by the photoelectric conversion unit 1012 . When there is a fine subject, if this portion is smoothed out, information about the subject will be lost. FIG. 5-2 shows a subject with a lot of detailed information. From the image signal extracted in step #3, the relevance is obtained in step #4, and the parts with no matching relation are selected as the detailed information part. Thus, in step #5, the image is such that the low-pass filter is not applied to the detailed information portion. Therefore, the sense of resolution in FIG. 5-2(d) is close to that in FIG. 5-2(e).

Further, the solid-state imaging device 101 of the embodiment of the present application uses the solid-state imaging device 1013 having four photoelectric conversion units in one pixel. However, it is also possible to use a solid-state imaging device having only two photoelectric conversion units in one pixel. In that case, although it depends on the wiring layer 1013, if the wiring layer 1013 is arranged around one pixel as in the embodiment described in this specification, the positional relationship with respect to the wiring layer 1013 is such that the photoelectric conversion units 1012 on the right and left sides different in Then, when the extraction unit 1022 extracts an image signal with the same image signal from the photoelectric conversion unit 1012 and its surrounding structure in step #3, the image signal has two patterns.

When using the solid-state imaging device 101 having two photoelectric conversion units 1012 in one pixel, step #4 compares the pixel of interest and the peripheral pixels in two frames. In step #5, the result of step #4 is two patterns, one in which the direction in which the difference between the pixel of interest and the surrounding pixels in each frame is the largest matches and the other in which they do not match. The image signal is replaced with the image signal to which the low-pass filter is applied.

Further, in the embodiments of the present application, the solid-state imaging device 101 in which one microlens 1011 is arranged for one pixel and has a plurality of photoelectric conversion units 1012 therein has been described. However, as shown in FIG. 6, the present invention can be applied even to a solid-state imaging device 101 having one photoelectric conversion unit 1012 in one microlens 1011 . In this case, due to the positional relationship with the wiring layer 1013, the adjacent left L and right R photoelectric conversion units 1012 have different incident light beams. This is considered to be the same problem as in the case of having two photoelectric conversion units 1012 in one microlens 1011 described above.

In the present invention, one unit in which the positional relationship between the photoelectric conversion unit 1012 and the wiring layer 1013 is repeated is defined as one pixel. Therefore, one pixel may include a plurality of microlenses 1011 . For example, in FIGS. 1 and 4, one pixel includes one microlens 1011 and four photoelectric conversion units 1012, and in FIG. 6, one pixel includes two microlenses 1011 and two photoelectric conversion units 1012. to be included.

10 imaging device 100 lens barrel 101 solid-state imaging device 1011 microlens 1012 photoelectric conversion section 1013 wiring layer 102 CPU
1021 filter unit 1022 extraction unit 1023 comparison unit 1024 mixing unit

Claims (3)

Each pixel has a plurality of photoelectric conversion units that convert a light flux that has passed through the optical system into an image signal ,
Further , in an image signal processing device for processing an image signal obtained from an imaging device having a structure on the path of the light flux of the optical system,
a filter unit for smoothing the image signal of the pixel of interest according to the surrounding image signals;
an extraction unit for extracting the image signal for each image signal having the same positional relationship as the structure;
a comparing unit that obtains relevance from a signal difference between a specific pixel and its surrounding pixels for each of the image signal planes extracted by the extracting unit, compares the relevance, and obtains a degree of relevance;
a mixing unit that mixes the image signal processed by the filter unit and the image signal converted by the photoelectric conversion unit according to the degree of association calculated by the comparison unit;
An image signal processing device comprising: 2. The image signal processing apparatus according to claim 1, wherein the filter unit is a low-pass filter or noise reduction filter having a strength that suppresses a pattern formed by the structure. 3. The image signal processing apparatus according to claim 1, wherein said mixing section has a filter for averaging output results of said comparing section.

Images