JP2017055281A - Solid state image sensor and camera module - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce memory usage by a deconvolution matrix.SOLUTION: A solid state image sensor includes an image sensor, a memory, and a signal processing circuit. The image sensor captures a subject image. A deconvolution matrix including a first region having a filter factor, and a second region not having a filter factor, is stored in the memory. The second region is provided to correspond with the first region. The signal processing circuit divides the subject image into a plurality of blocks, complements the deconvolution matrix so that the second region has a filter factor corresponding to that of the first region, and executes filtering of the plurality of blocks, by using the complemented deconvolution matrix.SELECTED DRAWING: Figure 1

Description

  The present embodiment relates to a solid-state imaging device and a camera module.

  Conventionally, a technique for restoring the resolution of a subject image by filter processing using a deconvolution matrix is known. In such a technique, it is desired to reduce the amount of memory used by the deconvolution matrix.

JP 2011-135359 A

  One embodiment aims to reduce the amount of memory used by the deconvolution matrix.

  According to one embodiment, the solid-state imaging device includes an image sensor, a memory, and a signal processing circuit. The image sensor captures a subject image. The memory stores a deconvolution matrix including a first region having filter coefficients and a second region having no filter coefficients. The second area is provided so as to correspond to the first area. The signal processing circuit divides the subject image into a plurality of blocks, complements the deconvolution matrix on the assumption that the second area has a filter coefficient corresponding to the filter coefficient of the first area, and the deconvolution matrix after the complement Is used to perform filtering on a plurality of blocks.

FIG. 1 is an exemplary block diagram illustrating a schematic configuration of a camera system including a solid-state imaging device according to the embodiment. FIG. 2 is an exemplary block diagram illustrating a specific configuration of the solid-state imaging device according to the embodiment. FIG. 3 is an image diagram showing an example of a subject image divided into a plurality of blocks in the embodiment. FIG. 4 is an image diagram illustrating an example of a deconvolution matrix according to the embodiment. FIG. 5 is an image diagram illustrating another example of the deconvolution matrix according to the embodiment, which is different from FIG. 4. FIG. 6 is an image diagram illustrating an example of a deconvolution matrix according to a modification.

  Exemplary embodiments of a solid-state imaging device and a camera module will be described below in detail with reference to the accompanying drawings. In addition, this invention is not limited by this embodiment.

(Embodiment)
FIG. 1 is an exemplary block diagram illustrating a schematic configuration of a camera system including a solid-state imaging device according to the embodiment. The camera system 1 is an electronic device including a camera module 2, and is a mobile terminal with a camera, for example. The camera system 1 may be an electronic device such as a digital camera.

  The camera system 1 includes a camera module 2 and a post-processing unit 3. The camera module 2 includes an imaging optical system 4 and a solid-state imaging device 5.

  The imaging optical system 4 is configured to capture light from the subject. Specifically, the imaging optical system 4 includes a lens that forms an object image IM (see FIG. 3 described later).

  The solid-state imaging device 5 includes an image sensor 6, a signal processing circuit 7, and a memory 8.

  The image sensor 6 is a sensor device that captures the subject image IM. For example, the image sensor 6 is a backside illumination type CMOS (Complementary Metal Oxide Semiconductor) image sensor. In the back-illuminated CMOS image sensor, a wiring layer is provided on the opposite side of the semiconductor layer including the photoelectric conversion portion from the side on which incident light is incident. Note that the image sensor 6 is not limited to a back-illuminated CMOS image sensor, but may be a front-illuminated CMOS image sensor, a CCD (Charge Coupled Device), or the like.

  The signal processing circuit 7 performs signal processing on the image signal of the subject image IM from the image sensor 6. For example, the signal processing circuit 7 performs resolution restoration processing according to the characteristics of the lens provided in the imaging optical system 4. As a characteristic of the lens, for example, a point spread function (PSF) is used. The signal processing circuit 7 restores an image with reduced blur by multiplying the value of the image signal of the subject image IM from the image sensor 6 by a deconvolution matrix corresponding to PSF. The deconvolution matrix is stored in the memory 8.

  The post-processing unit 3 includes an ISP (Image Signal Processor) 9, a storage unit 10, and a display unit 11. The ISP 9 performs signal processing of the image signal output from the solid-state imaging device 5. The storage unit 10 stores an image that has undergone signal processing in the ISP 9. The storage unit 10 outputs an image signal to the display unit 11 according to a user operation or the like.

  The display unit 11 displays an image according to an image signal input from the ISP 9 or the storage unit 10. The display unit 11 is, for example, a liquid crystal display. The camera system 1 performs feedback control of the solid-state imaging device 5 based on data that has undergone signal processing in the ISP 9.

  FIG. 2 is an exemplary block diagram illustrating a specific configuration of the solid-state imaging device according to the embodiment. The image sensor 6 of the solid-state imaging device 5 includes a pixel array 20, a control circuit 21, a row scanning circuit 22, a column scanning circuit 23, and a column processing circuit 24.

  The pixel array 20 is provided in the imaging region of the image sensor 6. The control circuit 21, the row scanning circuit 22, the column scanning circuit 23, and the column processing circuit 24 constitute a peripheral circuit portion of the pixel array 20.

  The pixel array 20 includes pixels arranged in a matrix. Each pixel includes a photodiode which is a photoelectric conversion unit. The photoelectric conversion unit generates signal charges corresponding to the amount of incident light. The pixel accumulates signal charges generated according to the amount of incident light.

  Various data and clock signals for driving the solid-state imaging device 5 are supplied to the control circuit 21 from the outside of the solid-state imaging device 5. The control circuit 21 generates various pulse signals for controlling driving of the circuit unit in accordance with the clock signal. The control circuit 21 supplies a pulse signal instructing the drive timing to the row scanning circuit 22, the column scanning circuit 23, the column processing circuit 24 and the signal processing circuit 7.

  The row scanning circuit 22 includes a shift register and an address decoder. The row scanning circuit 22 which is a pixel driving circuit supplies a driving signal to the pixels of the pixel array 20. The control circuit 21 supplies a pulse signal corresponding to the vertical synchronization signal to the row scanning circuit 22. The row scanning circuit 22 sequentially selects pixel rows from which pixel signals are read according to the pulse signal from the control circuit 21. The row scanning circuit 22 performs readout scanning by sequentially supplying a readout signal for each pixel in the selected pixel row. The read signal is a drive signal for reading a pixel signal generated according to the amount of incident light from the pixel.

  The row scanning circuit 22 performs sweep-out scanning by supplying a reset signal to each pixel prior to supplying a readout signal to each pixel. The reset signal is a drive signal for discharging the charge remaining in the pixel. Each pixel accumulates signal charges generated according to the amount of incident light from when the reset signal is supplied to when the readout signal is supplied.

  The drive signal is transmitted from the row scanning circuit 22 to each pixel through the pixel drive line 25. The pixel drive line 25 is provided for each pixel row of the pixel array 20. A pixel row consists of pixels arranged in the row direction.

  The pixel signal is transmitted from each pixel to the column processing circuit 24 through the vertical signal line 26. The vertical signal line 26 is provided for each pixel column of the pixel array 20. The pixel column is composed of pixels arranged in the column direction.

  The column processing circuit 24 processes the pixel signal transmitted through the vertical signal line 26 by a unit circuit (not shown) provided for each pixel column. The column processing circuit 24 performs correlated double sampling processing (CDS) for reducing fixed pattern noise on the pixel signal. The column processing circuit 24 performs AD conversion, which is conversion from an analog signal to a digital signal, on the pixel signal. Note that the column processing circuit 24 may perform processing other than CDS and AD conversion. The column processing circuit 24 holds the pixel signal that has undergone CDS and AD conversion for each unit circuit.

  The column scanning circuit 23 includes a shift register, an address decoder, and the like. The control circuit 21 supplies a pulse signal corresponding to the horizontal synchronization signal to the column scanning circuit 23. The column scanning circuit 23 sequentially selects pixel columns from which pixel signals are read according to the pulse signal from the control circuit 21. The column processing circuit 24 sequentially outputs pixel signals held in each unit circuit in accordance with the selective scanning by the column scanning circuit 23. The image sensor 6 outputs an image signal including the pixel signal from the column processing circuit 24 as a subject image IM.

  The signal processing circuit 7 divides the subject image IM output from the image sensor 6 into a plurality of blocks, and executes resolution restoration processing for each block. FIG. 3 is an image diagram showing an example of the subject image IM divided into a plurality of blocks. In the example of FIG. 3, the subject image IM is divided into 7 × 5 = 35 blocks. In the example of FIG. 3, each block is composed of 9 × 9 = 81 pixels. In the following, for convenience of explanation, 26 blocks located in the outer edge side area A1 of the subject image IM are referred to as first blocks B1, and nine blocks located in the center side area A2 of the subject image IM are referred to as first blocks B1. Let it be 2 blocks B2.

  The memory 8 stores a deconvolution matrix corresponding to each of the plurality of blocks. The signal processing circuit 7 reads out the deconvolution matrix from the memory 8, and multiplies the value of each pixel of the corresponding block by the filter coefficient set as the matrix element of the deconvolution matrix, etc. Processing is executed to restore the resolution of the entire subject image IM.

  Here, in the deconvolution matrix according to the embodiment, filter coefficients as matrix elements are set only in some areas. That is, the deconvolution matrix according to the embodiment includes a first region having a filter coefficient and a second region having no filter coefficient. The second area is provided so as to correspond to the first area. The signal processing circuit 7 complements the deconvolution matrix, assuming that the second region has a filter coefficient corresponding to the filter coefficient of the first region, and filters the corresponding block using the complemented deconvolution matrix. Execute the process.

  For example, FIG. 4 is an image diagram illustrating an example of a deconvolution matrix according to the embodiment.

  FIG. 4 illustrates two types of deconvolution matrices, a first deconvolution matrix DM1 and a second deconvolution matrix DM2. The first deconvolution matrix DM1 is used for filtering the first block B1 (see FIG. 3) on the outer edge side of the subject image IM, and the second deconvolution matrix DM2 is the second deconvolution matrix DM2 on the center side of the subject image IM. It is used for the filter processing of block B2 (see FIG. 3).

  By the way, general lenses tend to change (decrease) in characteristics such as resolving power and contrast from the center toward the outer edge. Here, it is assumed that the lens of the imaging optical system 4 according to the embodiment is a lens whose characteristics hardly change in the vicinity of the center and whose characteristics greatly decrease at the outer edge. In this case, the degree of blur tends to be greater in the first block B1 on the outer edge side of the subject image IM than in the second block B2 on the center side of the subject image IM. For this reason, in the embodiment, the resolution of the first block B1 and the second block B2 is appropriately restored by properly using two different types of deconvolution matrices in the first block B1 and the second block B2, and the subject image Effectively reduces the blur of the entire IM.

  The first deconvolution matrix DM1 of FIG. 4 includes a first region R11 having a filter coefficient (see the hatched region) and a second region R12 having no filter coefficient (see the region surrounded by a thick line). including. The first deconvolution matrix DM1 includes a third region R13 having no filter coefficient around the first region R11 and the second region R12. The first region R11 and the second region R12 are provided line-symmetrically with respect to the symmetry axis Ax. The symmetry axis Ax corresponds to a straight line that bisects the first block B1 along the tangential direction of the subject image IM (radiation direction with respect to the center of the subject image IM). In the example of FIG. 4, the boundary region between the first region R11 and the second region R12 also has a filter coefficient.

  Further, the second deconvolution matrix DM2 in FIG. 4 includes a first region R21 (see hatched region) having filter coefficients and three second regions R22 having no filter coefficients. The first region R21 and the second region R22 correspond to the respective regions obtained by dividing the second deconvolution matrix DM2 into four equal parts. That is, the first region R21 and the second region R22 are provided in rotational symmetry with respect to the symmetry center O. The symmetry center O corresponds to the center of the second block B2. In the example of FIG. 4, the boundary region between the first region R21 and the second region R22 also has a filter coefficient.

  Thus, in the example of FIG. 4, in both the first deconvolution matrix DM1 and the second deconvolution matrix DM2, the first regions R11 and R21 and the second regions R12 and R22 are symmetrical to each other. Provided at various positions. Therefore, when the signal processing circuit 7 executes the resolution restoration processing of the first block B1, the first region R12 is assumed to have filter coefficients arranged symmetrically with the filter coefficients of the first region R11. Complement the deconvolution matrix DM1. Similarly, when the signal processing circuit 7 executes the resolution restoration processing of the second block B2, the second region R22 is assumed to have filter coefficients arranged symmetrically with the filter coefficients of the first region R21. Complement the 2 deconvolution matrix DM2.

  Specifically, in the example of FIG. 4, when the signal processing circuit 7 executes the resolution restoration process of the first block B1, the second region R12 is a line with respect to the filter coefficient of the first region R11 and the symmetry axis Ax. The first deconvolution matrix DM1 is complemented as having symmetrically arranged filter coefficients.

  For example, a filter coefficient located at the center of the first deconvolution matrix DM1 is expressed as k1 (0, 0), and a filter coefficient located at the i-th row and the j-th column is k1 (i, The case where it is expressed as j) will be described. In this case, the signal processing circuit 7 performs k1 (i, j) = k1 (−j, −) for the eight first deconvolution matrices DM1 in the upper left corner region X1 and the lower right corner region X2 in FIG. The first deconvolution matrix DM1 is complemented on the assumption that the formula i) holds. Further, the signal processing circuit 7 sets k1 (i, j) = k1 (j, i) for the eight first deconvolution matrices DM1 in the lower left corner region X3 and the upper right corner region X4 of FIG. As a result, the first deconvolution matrix DM1 is complemented. Further, the signal processing circuit 7 calculates k1 (i, j) = k1 (−i, j) for the four first deconvolution matrices DM1 in the central regions X5 and X6 in the vertical direction of FIG. As a result, the first deconvolution matrix DM1 is complemented. Further, the signal processing circuit 7 calculates k1 (i, j) = k1 (i, −j) for the six first deconvolution matrices DM1 in the center regions X7 and X8 in the horizontal direction of FIG. As a result, the first deconvolution matrix DM1 is complemented.

  In the example of FIG. 4, when the signal processing circuit 7 executes the resolution restoration processing of the second block B2, the second region R22 is rotationally symmetric with respect to the filter coefficient of the first region R21 and the symmetry center O. The second deconvolution matrix DM2 is complemented as having the arranged filter coefficients. For example, a filter coefficient located at the center of the second deconvolution matrix DM2 is expressed as k2 (0, 0), and a filter coefficient located at the i-th row and j-th column with respect to the center is represented by k2 (i, j), the signal processing circuit 7 satisfies the relational expression k2 (i, j) = k2 (−j, i) = k2 (−j, −i) = k2 (j, −i). As a complement, the second deconvolution matrix DM2.

  FIG. 5 is an image diagram illustrating another example of the deconvolution matrix according to the embodiment, which is different from FIG. 4. The example of FIG. 5 is different from the example of FIG. 4 in the configuration of the first deconvolution filter DM1a used for the filter processing of the first block B1 on the outer edge side of the subject image IM.

  Specifically, in the deconvolution matrix DM1a of FIG. 5, a first region R11a having a filter coefficient and a second region R12a having no filter coefficient are provided in line symmetry with respect to the symmetry axis Ax. This is the same as the first deconvolution matrix DM1 in FIG. However, in the deconvolution matrix DM1a of FIG. 5, unlike the first deconvolution matrix DM1 of FIG. 4, the first region R11a and the second region R12a extend to the end of the deconvolution matrix DM1a. That is, the deconvolution matrix DM1a in FIG. 5 is not provided with a region having no filter coefficient, like the third region R13 in FIG. 4, around the first region R11a and the second region R12a. Therefore, according to the deconvolution matrix DM1a of FIG. 5, it is possible to execute a filter process considering all the pixels in the first block B1.

  Other configurations of the example of FIG. 5 are the same as those of the example of FIG. 4 described above.

  In the embodiment, only one of the deconvolution matrix illustrated in FIG. 4 and the deconvolution matrix illustrated in FIG. 5 according to the lens characteristics is stored in the memory 8 in advance. May be. In the embodiment, both the deconvolution matrix illustrated in FIG. 4 and the deconvolution matrix illustrated in FIG. 5 are stored in the memory 8 in advance and selectively used according to a predetermined condition. May be.

  As described above, according to the embodiment, the memory 8 of the solid-state imaging device 5 includes the first region R11 (R21, R11a) having a filter coefficient and the second region R12 (R22, R12a) having no filter coefficient. Are stored in a deconvolution matrix DM1 (DM2, DM1a). The signal processing circuit 7 complements the deconvolution matrix DM1 (DM2, DM1a) on the assumption that the second region R12 (R22, R12a) has a filter coefficient corresponding to the filter coefficient of the first region R11 (R21, R11a). Then, the filtering process is executed using the complemented deconvolution matrix DM1 (DM2, DM1a). Thereby, since it is not necessary to set a filter coefficient in advance in the second region R12 (R22, R12a), the amount of memory 8 used by the deconvolution matrix DM1 (DM2, DM1a) can be reduced.

  Further, as described above, in the embodiment, a lens is assumed in which the characteristics hardly deteriorate at the center side and the characteristics greatly decrease at the outer edge side. Therefore, in the embodiment, the first deconvolution matrix DM1 (the first block B1 that can be complemented so as to have a line-symmetric filter coefficient in the tangential direction for the first block B1 that is imaged on the outer edge side of the lens whose characteristics are greatly degraded. By using DM1a), an appropriate resolution restoration process corresponding to the characteristics on the outer edge side of the lens is realized. In addition, for the second block B2 imaged on the center side of the lens whose characteristics are hardly deteriorated, the second deconvolution matrix DM2 that can be complemented so as to have a rotationally symmetric filter coefficient around the center is used. An appropriate resolution restoration process is realized according to the characteristics of the center side.

(Modification)
The setting of the first area and the second area shown in the above embodiment is merely an example, and the setting of the first area and the second area may be variously changed according to the characteristics of the lens.

  In other words, in the above-described embodiment, it is assumed that a lens whose characteristics are not significantly deteriorated on the center side and whose characteristics are greatly deteriorated on the outer edge side, and for the first block on the outer edge side of the subject image, the first area and the second area. Was used for the second block on the center side, and a deconvolution matrix in which the first region and the second region were set to be rotationally symmetric was used. However, for example, assuming a lens in which a characteristic deterioration appears immediately by moving slightly outward from the center, a part of the second block located slightly outside the center is also similar to the first block. It is also conceivable to use a deconvolution matrix in which the first region and the second region are set symmetrically.

  For example, when a lens having characteristics equivalent to those of the center side is assumed on the outer edge side, the first block on the outer edge side is the same as the second block on the center side as in the modified example of FIG. 6 described below. It is also conceivable to use a deconvolution matrix in which the first region and the second region are set to be rotationally symmetric.

  For example, FIG. 6 is an image diagram showing an example of a deconvolution matrix according to a modification. In the modification of FIG. 6, the same deconvolution matrix DM2a is used for all of a plurality of blocks obtained by dividing the subject image IM. The deconvolution matrix DM2a has the same symmetry as the second deconvolution matrix DM2 of the above-described embodiment. That is, the deconvolution matrix DM2a has a first region R21a having a filter coefficient and a second region R22a having no filter coefficient, and the first region R21a and the second region R22a are rotationally symmetrical around the center. Has been placed.

  Also in the modified example of FIG. 6, similarly to the above-described embodiment, it is not necessary to set a filter coefficient in the second region R22a in advance, so that the usage amount of the memory 8 can be reduced.

  As mentioned above, although embodiment and the modification of this invention were described, these embodiment and the modification are shown as an example and are not intending limiting the range of invention. The novel embodiments and modifications can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the spirit of the invention. These embodiments and modifications are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

  2 camera module, 5 solid-state imaging device, 6 image sensor, 7 signal processing circuit, 8 memory, B1 first block, B2 second block, DM1, DM1a, DM2, DM2a deconvolution matrix, IM subject image, R11, R11a , R21, R21a first region, R12, R12a, R22, R22a second region.

Claims (7)

An image sensor for capturing a subject image;
A memory for storing a deconvolution matrix including a first region having a filter coefficient and a second region provided so as to correspond to the first region and having no filter coefficient;
The subject image is divided into a plurality of blocks, the deconvolution matrix is complemented on the assumption that the second area has a filter coefficient corresponding to the filter coefficient of the first area, and the deconvolution matrix after complementation And a signal processing circuit that performs filter processing on the plurality of blocks using a solid-state imaging device. The first region and the second region are provided at symmetrical positions in the deconvolution matrix,
2. The solid-state imaging device according to claim 1, wherein the signal processing circuit complements the deconvolution matrix on the assumption that the second region has a filter coefficient arranged symmetrically with the filter coefficient of the first region. . The deconvolution matrix includes a first deconvolution matrix used for the filtering process on the first block located on the outer edge side of the subject image, and the filtering process on the second block located on the center side of the subject image. A second deconvolution matrix used in
The symmetry of the first region and the second region in the first deconvolution matrix and the symmetry of the first region and the second region in the second deconvolution matrix are different from each other. 2. The solid-state imaging device according to 2. The first region and the second region of the first deconvolution matrix are provided in line symmetry with respect to a symmetry axis corresponding to a tangential direction in the subject image;
The signal processing circuit complements the first deconvolution matrix by assuming that the second region has filter coefficients arranged in line symmetry with respect to the filter coefficients of the first region and the symmetry axis. The solid-state imaging device according to claim 3, wherein the filtering process is executed on the first block using the first deconvolution matrix later.   5. The solid-state imaging device according to claim 3, wherein the first deconvolution matrix further includes a third region having no filter coefficient around the first region and the second region. The first region and the second region of the second deconvolution matrix are provided in rotational symmetry with respect to a symmetry center corresponding to a center of the second block;
The signal processing circuit complements the second deconvolution matrix by assuming that the second region has filter coefficients arranged in rotational symmetry with respect to the filter coefficient of the first region and the symmetry center, and complements the second deconvolution matrix The solid-state imaging device according to claim 3, wherein the second block is used to perform the filtering process on the second block. An imaging optical system that captures light from the subject and forms a subject image;
A solid-state imaging device,
The solid-state imaging device
An image sensor for capturing the subject image;
A memory for storing a deconvolution matrix including a first region having a filter coefficient and a second region provided so as to correspond to the first region and having no filter coefficient;
The subject image is divided into a plurality of blocks, the deconvolution matrix is complemented on the assumption that the second area has a filter coefficient corresponding to the filter coefficient of the first area, and the deconvolution matrix after complementation And a signal processing circuit that performs filtering on the plurality of blocks using the camera module.

Images