JP6012555B2 - Interpolation signal generation method for image signal - Google Patents

Description

  The present invention relates to a method for generating an interpolation signal for an image signal. More particularly, the present invention relates to a method for generating an interpolation signal for an image signal, which is suitable for increasing the number of pixels of an image signal generated using a solid-state imaging device.

Currently, the image signal with the highest resolution among the image signals obtained by general television broadcasting is called full high vision, and is composed of horizontal 1920 pixels and vertical 1080 pixels. On the other hand, the development of a method called 4K in which the number of pixels is doubled in both the horizontal direction and the vertical direction has been developed, and a liquid crystal display corresponding to this has already been available.
Therefore, a method of generating an image signal corresponding to 4K by interpolating a full high-definition image signal so that the number of pixels is doubled in both the horizontal direction and the vertical direction has been proposed and implemented.

As interpolation signal generation methods that can be used for such pixel interpolation, linear interpolation methods and bicubic interpolation methods are known. Discrete cosine transform (hereinafter “DCT”) and inverse discrete cosine transform (hereinafter “IDCT”). ) Describes a method of generating an interpolation signal in US Pat. No. 5,168,375 or Japanese Patent Laid-Open No. 11-261807.
In such a method, when the DCT process is performed on the pixels of the original image signal and the IDCT process is performed using the obtained frequency coefficient, the signal is reproduced at a half interval compared to the pixel interval used in the DCT. By doing so, a signal of the interpolation pixel is generated. According to this, since the interpolation signal is generated using the frequency coefficient obtained from the original pixel, the interpolation of the pixel can be achieved without deteriorating the frequency characteristic.

  Furthermore, the above-mentioned US Pat. No. 5,168,375 describes a method for achieving frequency correction of a reproduction signal obtained by IDCT by multiplying a frequency coefficient obtained by DCT by a coefficient corresponding to a frequency characteristic. Yes. According to this method, assuming that pixel interpolation is performed by DCT and IDCT processing, frequency correction processing that is normally realized by product-sum operation of pixel signals and filter matrix coefficients is obtained by DCT. Since this can be achieved by multiplying the frequency coefficient by the correction coefficient, the entire process is simplified.

  In the prior art disclosed in Japanese Patent Application Laid-Open No. 2004-23384, when pixel interpolation is performed by DCT and IDCT processing, the interpolation signal is obtained as a linear combination of the original pixel signals. It is shown that a two-dimensional filtering process may be performed on the pixels of the block to be subjected to the above. According to this method, the process for generating the interpolation signal is further simplified.

On the other hand, in recent years, solid-state imaging devices have been used in almost all imaging devices, from general video cameras and digital still cameras to broadcast television cameras. In such an imaging device, an image signal is generated by converting a signal obtained from a plurality of pixels arranged two-dimensionally on the imaging surface of the solid-state imaging device into a format corresponding to the display device. For example, when a full-high-definition image signal is obtained with a broadcast TV camera, if a solid-state image sensor with horizontal 1920 pixels and vertical 1080 pixels is used, the pixel of the solid-state image sensor and the pixel of the full-high-definition display device have a one-to-one correspondence. To do.
For this reason, interpolation so that the number of pixels in the horizontal direction and the vertical direction is doubled in order to make a full high-definition image signal compatible with a 4K display device divides the pixels of the solid-state image sensor into the horizontal direction. -Supports double processing in the vertical direction. Therefore, when converting a full high-definition image signal obtained by a solid-state imaging device into a 4K image signal by interpolation, the frequency characteristic corresponding to the division of the pixel should be corrected. However, the interpolation signal generation method of the prior art does not show an interpolation signal generation method capable of correction suitable for the process of dividing the pixels of the solid-state imaging device.

Japanese Patent Application Laid-Open No. 2007-281720, which is one of the prior arts, shows a method for restoring the resolution of an image obtained by pixel mixture readout in which a solid-state imaging device reads out a plurality of pixel signals. In the method shown here, an image obtained by pixel mixture readout is subjected to DFT processing, and the obtained frequency coefficient is multiplied by a coefficient relating to a photographing condition such as a lens or a coefficient relating to how to mix pixel mixture. By performing inverse DFT processing, the resolution of an image obtained by pixel mixture readout is restored.
However, this shows a method of restoring the resolution when signals between non-adjacent pixels are mixed, but does not show a correction method suitable for processing to divide and double the pixels of the solid-state imaging device.

Japanese Patent Laid-Open No. 11-261807 JP 2004-23384 A JP 2007-281720 A

U.S. Pat. 5,168,375

As described above, in the interpolation signal generation method for the image signal according to the prior art, the image signal obtained by the solid-state imaging device is interpolated by the DCT and IDCT processing, and the frequency coefficient obtained by the DCT is calculated. A method is shown in which frequency correction is performed on an image signal obtained by IDCT processing by multiplying coefficients. Further, in the interpolation signal generation method according to the conventional technique, the restoration of the resolution of the image signal obtained by the pixel mixture readout is realized by the IDFT processing after multiplying the frequency coefficient obtained by DFT by the correction coefficient. It is shown.
However, there is no frequency correction method suitable for the process of dividing the pixels of the solid-state imaging device and doubling the number of pixels, and the image signal after doubling the number of pixels with sufficient accuracy by the DCT and IDCT processes is not shown. Couldn't get.

  In view of the above-described problems of the prior art, the present invention can perform frequency correction suitable for the process of dividing the pixels of the solid-state image pickup device and doubling the number of pixels, and has sufficient accuracy for the image signal obtained by the solid-state image pickup device. It is an object of the present invention to provide an interpolation signal generation method for an image signal that is suitable for interpolation.

  In order to solve the above-described problem, in the interpolation signal generation method for an image signal according to the present invention, N (N is an integer) continuous in the horizontal direction from an image signal composed of a plurality of pixels arranged two-dimensionally. A first pixel group forming an N × N pixel block formed by pixels and N pixels continuous in the vertical direction is extracted, and a signal obtained from the first pixel group is subjected to orthogonal transformation to be subjected to a first frequency. A coefficient group is generated, a frequency correction is performed on the first frequency coefficient group to generate a second frequency coefficient group, and the second frequency coefficient group is compared with the pixel interval in the first pixel group. The first signal corresponding to at least a first position where no pixel is present in the first pixel group is generated by performing inverse orthogonal transformation to generate a signal at a half interval, and the first signal in the image signal is generated. The first signal as a signal corresponding to the position of 1 It is used. At this time, the frequency correction may be based on a correction coefficient corresponding to a process of dividing the pixels constituting the image signal.

  In the interpolation signal generation method for an image signal according to the present invention, N (N is an integer) pixels that are continuous in the horizontal direction and N pixels that are continuous in the vertical direction from the image signal that is composed of a plurality of pixels arranged two-dimensionally. A first pixel group that forms a block of N × N pixels formed by N pixels to be extracted is extracted, and further, a first frequency coefficient group is generated by performing orthogonal transformation on the signal obtained from the first pixel group A process of generating a second frequency coefficient group by performing frequency correction on the first frequency coefficient group, and comparing the pixel frequency in the first pixel group with the second frequency coefficient group. A filter operation corresponding to a process for generating a first signal corresponding to a first position at which no pixel exists in at least the first pixel group by performing an inverse orthogonal transform for generating a signal at a half interval Applied to the first pixel group, and As a signal corresponding to the first position in the image signal may be used the first signal. At this time, the frequency correction may be based on a correction coefficient corresponding to a process of dividing the pixels constituting the image signal.

In the method for generating an interpolated signal of an image signal according to the present invention, the orthogonal transform may be a discrete cosine transform, and the inverse orthogonal transform may be an inverse discrete cosine transform.
Further, the correction coefficient includes an interval of the pixels in the image signal of 2dx, a ratio in one direction in which light incident on the pixels is converted into a signal, a (1/2 <a <1), and a space of the image signal. The frequency can be set to correspond to 2a / {1+ (2a-1) · cos (2π · f · dx)} where f is the frequency.

  According to the interpolation signal generation method for an image signal of the present invention, frequency correction suitable for the process of dividing the pixels of the solid-state imaging device and doubling the number of pixels becomes possible. As a result, even when an image signal obtained by the solid-state image sensor is displayed on a display device having a doubled number of pixels, an image signal close to a signal to be obtained from the solid-state image sensor having twice the number of pixels can be obtained. .

FIG. 1 is a diagram showing an embodiment of an interpolation signal generation method for image signals according to the present invention. Example 1 FIG. 2 is a diagram showing an example of the configuration of a block extraction circuit used in the interpolation signal generation method for image signals of the present invention. FIG. 3 is a diagram showing a block of pixels used for DCT processing. FIG. 4 is a diagram illustrating a method of dividing the pixels of the solid-state imaging device. FIG. 5 is a diagram showing a frequency response when an image signal having a pixel size of 1/2 and a pixel count of 2 is converted into an image signal of a pixel of 1 time. FIG. 6 is a diagram showing a frequency correction coefficient when restoring an image signal having a pixel size of 1/2 and a pixel count of 2 by DCT and IDCT processing. FIG. 7 is a diagram showing another embodiment of an interpolation signal generation method for image signals according to the present invention. (Example 2) FIG. 8 is a diagram illustrating a situation when the pixels are divided when the aperture ratio of the pixels of the solid-state imaging device is smaller than 100%.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram showing a first embodiment of an image apparatus to which an interpolation signal generating method for image signals according to the present invention is applied. In FIG. 1, reference numeral 1 denotes an imaging device. In the imaging apparatus 1, an image captured by an optical system 101 including a lens is converted into an electrical signal by a solid-state imaging device 102, and this is converted by a color signal processor 103 into R (red) and G (green) signals that are the three primary colors. , B (blue) signal to 3 channel image signal. Here, it is assumed that the R signal, the G signal, and the B signal are generated so as to have signals at all sampling positions corresponding to the respective pixels of the solid-state imaging device 102.

In FIG. 1, reference numeral 2 denotes an interpolation signal generator. In the interpolation signal generation device 2, the R signal, G signal, and B signal obtained from the imaging device 1 are respectively added to the block extraction circuit 201. The block extraction circuit 201 extracts, for example, signals of a total of 64 pixels constituting a block of 8 pixels in the horizontal direction and 8 pixels in the vertical direction necessary for DCT from the input signal and outputs them simultaneously. The 64-pixel signal obtained from the block extraction circuit 201 is applied to the DCT processing circuit 202 and subjected to DCT processing, and converted into a frequency coefficient group that forms a matrix of 8 frequencies in the horizontal direction and 8 frequencies in the vertical direction.
The frequency coefficient group obtained by the DCT processing circuit 202 is added to the frequency correction circuit 203, and each frequency coefficient is multiplied by a correction coefficient suitable for the process of dividing the pixels of the solid-state imaging device. The frequency coefficient group after frequency correction obtained from the frequency correction circuit 203 is added to the IDCT processing circuit 204, and a signal is generated at a half interval compared to the original pixel interval added to the DCT processing circuit 202. IDCT processing is performed. As a result, an interpolation signal is generated between the original pixels, and a signal in which the number of pixels is doubled in both the horizontal and vertical directions is obtained. Further, a signal obtained from the IDCT processing circuit 204 is converted into an image signal corresponding to a display device in which the number of pixels in the horizontal direction and the vertical direction is twice that of the solid-state imaging device 102 by a rearrangement circuit 205 using a memory.
The image signal obtained from the rearrangement circuit 205 is sent to a subsequent display device or transmission device not shown in the figure connected to the interpolation signal generation device 2.

  Here, the operation of the interpolation signal generation device 2 will be described in detail with reference to the drawings, with particular emphasis on the correction coefficient in the frequency correction circuit 203. Here, assuming that the same processing is performed on the G signal, R signal, and B signal output from the color signal processor 103 of the imaging apparatus 1, only one channel for processing the G signal will be described.

In the interpolation signal generation device 2, the G signal added from the color signal processor 103 of the imaging device 1 is added to the block extraction circuit 201, and a signal of 64 pixels forming a block of 8 pixels in the horizontal direction and 8 pixels in the vertical direction is obtained simultaneously. To be extracted. An example of the block extraction circuit 201 is to obtain a signal of 8 pixels in the vertical direction simultaneously by connecting in series 7 stages of 1H delay circuits 301 that delay the input signal by one horizontal scanning period as shown in FIG. It is a well-known technique that an input signal and an output of each 1H delay circuit are delayed by seven stages by a one-pixel delay circuit 302 that delays by one pixel so that signals of eight pixels in the horizontal direction can be obtained simultaneously.
FIG. 3 shows an example of 64 pixels constituting the extracted block. In the figure, the vertical position is represented by i, the horizontal position is represented by j, and the upper left corner of the figure is a pixel X (0, 0) corresponding to i = 0, j = 0. In addition, x (0,0), x (0,1), etc. written at the position of each pixel in the block indicate the magnitude of a signal obtained from each corresponding pixel.

  A pixel signal for one block obtained from the block extraction circuit 201 is added to the DCT processing circuit 202, and is subjected to a well-known discrete cosine transform expressed by the following equation (1) to obtain eight vertical frequencies (u = 0). , 1, 2,..., 7) and 8 horizontal frequencies (v = 0, 1, 2,..., 7) are converted into a total of 64 frequency coefficient groups. Here, the frequency coefficient corresponding to the vertical frequency u and the horizontal frequency v is F (u, v). Here, N = 8.

  The frequency coefficient group obtained by the DCT processing circuit 202 is added to the frequency correction circuit 203, and frequency correction suitable for the process of dividing the pixels of the solid-state imaging device is performed. Next, frequency correction performed here will be described.

FIG. 4A shows an example of four pixels adjacent to each other in the horizontal direction and the vertical direction in the solid-state imaging device. Here, it is assumed that most of the light incident on the imaging surface is used for generation of photoelectric charges in the pixel, and the aperture ratio of the pixel is close to 100%. An aperture ratio close to 100% can be realized with a back-illuminated solid-state image pickup device in which the image pickup surface is placed on the back side without wiring, or a solid-state image pickup device in which each pixel is combined with a microlens for condensing light.
In FIG. 4A, the pixel X (n, m), pixel X (n, m + 1), pixel X (n + 1, m), and pixel X (n + 1, m + 1) are accumulated. The photocharge signals are signals x (n, m), x (n, m + 1), x (n + 1, m) obtained at the positions of A, B, C, and D, which are the centers, respectively. x (n + 1, m + 1) is output from the solid-state imaging device.

  When the pixel shown in FIG. 4A is divided so that the number of pixels is doubled in the horizontal direction and the vertical direction, signals obtained at positions A, B, C, and D are obtained at intermediate positions. A method of dividing the signal so as to generate a signal to be generated is as shown in FIG. That is, the pixel is divided into a pixel half the size of the original pixel and centered on A, B, C, and D, and a pixel located in the middle and centered on the boundary of the original pixel.

  In FIG. 4B, the pixels centered at A, B, C, and D are X2 (2n, 2m), X2 (2n, 2 (m + 1)), X2 (2 (n + 1), 2m, respectively. ), X2 (2 (n + 1), 2 (m + 1)), for example, the pixel between A and B is X2 (2n, 2m + 1), and the pixel between A and C is X2 (2n + 1,2m). Similarly, assuming a pixel corresponding to the center of the original pixel and a pixel corresponding to the peripheral position, for example, the pixel X2 (2n, 2m) centered on A as shown in FIG. The upper left pixel is X2 (2n-1,2m-1), the upper pixel is X2 (2n-1,2m), the upper right pixel is X2 (2n-1,2m + 1), and the left pixel is X2 (2n, 2m-1), right pixel is X2 (2n, 2m + 1), lower left pixel is X2 (2n + 1,2m-1), lower pixel is X2 (2n + 1,2m ), And the lower right pixel is X2 (2n + 1,2m + 1).

From the comparison between FIG. 4A and FIG. 4B, the pixel X (n, m) includes all the pixels X2 (2n, 2m), and the adjacent pixels X2 (2n-1,2m) vertically and horizontally X2 (2n, 2m-1), X2 (2n, 2m + 1), X2 (2n + 1,2m), half of the pixel X2 (2n-1,2m-1), X2 (2n) -1,2m + 1), X2 (2n + 1,2m-1), and X2 (2n + 1,2m + 1) are included by 1/4. As a result, signals x2 (2n, 2m), x2 (2n-1,2m-1), x2 (2n-1,2m), x2 (2n-1,2m + 1), x2 ( 2n, 2m-1), x2 (2n-1,2m + 1), x2 (2n + 1,2m-1), x2 (2n + 1,2m), and x2 (2n + 1,2m + 1) When used, the signal x (n, m) obtained from the pixel X (n, m) is expressed by the following equation (2).

x (n, m) = x2 (2n, 2m)

+ {x2 (2n-1,2m) + x2 (2n, 2m-1) + x2 (2n, 2m + 1) + x2 (2n + 1,2m)} × 1/2

+ {x2 (2n-1,2m-1) + x2 (2n-1,2m + 1) + x2 (2n + 1,2m-1) + x2 (2n + 1,2m + 1)} × 1/4

(2)

Here, when the pixel signal x2 (2n, 2m) has a relationship of the following equation (3) that changes in a sine wave shape only in the horizontal direction, the equation (2) becomes the following equation (4). Here, f is an arbitrary frequency, and dx is a pixel interval after interpolation.

x2 (2n, 2m) = exp (j2π ・ f ・ 2m ・ dx) (3)

x (n, m) = exp (j2π ・ f ・ 2m ・ dx)

+ {exp (j2π ・ f ・ 2m ・ dx) + exp (j2π ・ f ・ (2m-1) ・ dx)

+ exp (j2π ・ f ・ (2m + 1) ・ dx) + exp (j2π ・ f ・ 2m ・ dx)} × 1/2

+ {exp (j2π ・ f ・ (2m-1) ・ dx) + exp (j2π ・ f ・ (2m + 1) ・ dx)

+ exp (j2π ・ f ・ (2m-1) ・ dx) + exp (j2π ・ f ・ (2m + 1) ・ dx)} × 1/4

= 2 ・ exp (j2π ・ f ・ 2m ・ dx) ・ {1 + exp (-j2π ・ f ・ dx) / 2 + exp (j2π ・ f ・ dx) / 2}

(Four)

Further, when exp (jx) = cos (x) + jsin (x) is applied to the expression (4), the following expression (5) is obtained.

x (n, m) = 4 ・ exp (j2π ・ f ・ 2m ・ dx) ・ 1/2 ・ {1 + cos (2π ・ f ・ dx)} (5)

The exp (j2π · f · 2m · dx) in the first half of the right side of the equation (5) is the portion representing the original signal that has a constant amplitude and changes sinusoidally, and the latter half 1/2 · {1 + cos (2π · f・ Dx)} is the frequency of the signal by the original pixel that is halved by doubling the pixel size relative to the signal when the horizontal pixel size is 1/2 and the pixel number is doubled This is the part that shows the response. In other words, the reciprocal number 2 / {1 + cos (2π · f · dx)} in the latter half is divided into pixels by halving the size of the signal obtained from the original pixel and doubling the number of pixels. Represents the frequency response of the signal.
The frequency response represented by 1/2 · {1 + cos (2π · f · dx)} is as shown in FIG. 5 and becomes zero at 1 / 2dx which is the sampling frequency of the original pixel, and is 1 which is the Nyquist frequency. It becomes 1/2 at / 4dx. Note that 1 / 4dx, which is the Nyquist frequency, corresponds to v = 8 in the horizontal frequency v of equation (1). Therefore, the frequency response R (v) represented by the horizontal frequency v is expressed by the following equation (6).

R (v) = 1/2 ・ {1 + cos (2π ・ v / 32dx ・ dx)}

= 1/2 ・ {1 + cos (π ・ v / 16)} (6)

At this time, if the frequency correction circuit 203 multiplies each of the frequency coefficient groups obtained by the DCT processing circuit 202 by the inverse 2 / {1 + cos (π · v / 16)} of the frequency response R (v). Thus, it is possible to correct the deterioration of the frequency characteristics caused in the process of replacing the image signal in which the size of the pixel in the horizontal direction is ½ and the number of pixels is twice with the image signal of one time.
In the above description, processing in the horizontal direction is taken as an example, but the same applies to the vertical direction. FIG. 6 shows the frequency correction coefficient obtained as the reciprocal of R (u, v) obtained by multiplying R (v) in equation (6) by R (u) when expanded in the vertical direction. . In FIG. 6, the horizontal direction v and the vertical direction u represent frequencies, and the values described at each frequency position are correction coefficients to be multiplied by the frequency coefficient at the frequency. That is, in the interpolation signal generation device 2 shown in FIG. 1, the frequency correction circuit 203 is shown in FIG. 6 for each of the frequency coefficient groups that form blocks of 8 horizontal frequencies and 8 vertical frequencies obtained from the DCT processing circuit 202. What is necessary is just to multiply a correction coefficient for every corresponding frequency.
After that, the frequency coefficient group F2 (u, v) after frequency correction obtained from the frequency correction circuit 203 is added to the IDCT processing circuit 204, and a signal whose pixel interval is ½ compared to the original image signal is generated. To do. In other words, if x2 (i, j) is obtained by putting 0 to 2N-1 in i and j in the following equation (7), the pixel size is 1/2 and the number of pixels is doubled. A signal close to the signal that should be originally obtained from the solid-state imaging device is obtained.

  Since the signals obtained from the IDCT processing circuit 204 have twice the number of pixels in the horizontal and vertical directions as compared with the original image signal, the rearrangement circuit 205 using the memory performs the interpolation of the scanning lines and the like. It is output from the signal generation device 2.

FIG. 7 is a block diagram showing a second embodiment of an image apparatus using an interpolation signal generation method for image signals according to the present invention. In FIG. 7, the imaging apparatus 1 is the same as that of the first embodiment shown in FIG. 1 and performs the same operation.
Further, in the interpolation signal generation device 4 in FIG. 7, the R signal, the G signal, and the B signal obtained from the color signal processor 103 of the imaging device 1 are respectively extracted from the block extraction circuit 201 as in the interpolation signal generation device 2 in FIG. 1. Add to. Similarly to the interpolation signal generation apparatus 2 in FIG. 1, the block extraction circuit 201 extracts signals using a memory so that signals of a total of 64 pixels forming a block of 8 pixels in the horizontal direction and 8 pixels in the vertical direction can be obtained simultaneously. It is. On the other hand, in the interpolation signal generation device 4, a signal of 64 pixels obtained from the block extraction circuit 201 is added to the filter processing circuit 206.

As described in Japanese Patent Laid-Open No. 2004-23384, when interpolation of pixels is performed by DCT and IDCT processing, the interpolation signal is obtained by linearly combining the original pixel signals. What is necessary is just to perform a two-dimensional filter process with respect to each pixel of the block used as object. The frequency correction circuit 203 in the interpolation signal generation device 2 shown in FIG. 1 adds the correction shown in FIG. 6 to each of the frequency coefficient groups that form blocks of 8 horizontal frequencies and 8 vertical frequencies obtained from the DCT processing circuit 202. Since the coefficients are multiplied for each corresponding frequency, the interpolation signal is still obtained by linearly combining the original pixel signals. Therefore, the filter processing circuit 206 has one filter corresponding to one signal generated by the processing of the DCT processing circuit 202, the frequency correction circuit 203, and the IDCT processing circuit 204 in the interpolation signal generation apparatus 2 shown in FIG. A processing function may be provided. As a result, the filter processing circuit 206 can obtain the same signal as that obtained from the IDCT processing circuit 204 in FIG.
At this time, the filter coefficient for the filter processing function in the filter processing circuit 206 can be obtained by a known method described in JP-A-2004-23384. That is, with respect to a signal at a certain position obtained by IDCT processing, only a signal of a certain pixel among pixels forming a block used for processing is set to “1”, and a DCT processing circuit 202, a frequency correction circuit 203, and an IDCT processing circuit 204 are placed. To obtain the output. The output at this time may be a filter coefficient corresponding to the pixel in which a signal among pixels forming a block for a signal at a certain position is set to “1”.

  The signal obtained by the filter processing circuit 206 is similar to the output of the IDCT processing circuit 204 in the interpolation signal generating apparatus 2 in FIG. It is converted into an image signal corresponding to a display device that is twice as large as 102. The image signal obtained from the rearrangement circuit 205 is sent to a subsequent display device or transmission device not shown in the figure connected to the interpolation signal generation device 4 as in the interpolation signal generation device 2 in FIG.

In the above description, the case where the same processing is applied to the G signal, the R signal, and the B signal has been described as an example. However, when the color signal processor outputs the luminance signal and the color difference signal, the above description is applied only to the luminance signal. The color difference signal may be subjected to general interpolation processing such as linear interpolation.
Moreover, although the case where the interpolation signal generation device is directly connected to the imaging device is taken as an example in the above description, a transmission device (not shown) may be inserted between the imaging device and the interpolation signal generation device. It is clear.

In the above description, the orthogonal transform by the DCT process is described. However, it can be easily analogized that the same effect can be obtained by other orthogonal transforms such as discrete sine transform and Hadamard transform.
Further, the above description has been given by taking as an example the case where the number of pixels is doubled in the horizontal direction and the vertical direction. However, depending on the correspondence with the display device or the like, only one of the horizontal direction and the vertical direction is selected in the IDCT processing circuit. A pixel signal may be generated so as to be doubled.
In the above description, the processing of the present invention is realized by hardware. However, it is obvious that the processing may be performed by software.

In the above description, the case where the aperture ratio of the pixel in the solid-state imaging device is close to 100% has been described as an example. However, even when the pixel aperture ratio is less than 100%, it is adjacent to the top, bottom, left, and right in the equation (4). The frequency response R (v) is obtained in a direction in which the ratio of the pixel signal is smaller than 1/2 and the ratio of the pixel signal in the oblique direction is smaller than 1/4. It can be easily inferred that it should be obtained.
For example, in the original image signal, when the aperture ratio when the light incident on 90% of the horizontal and vertical width 2dx of the pixel is converted into a signal is 81%, FIG. As shown in FIG. 4, light that enters the range of dx / 10 from the periphery of the pixel does not generate a signal. At this time, when the pixel is divided into halves as in FIG. 4B, it becomes as shown in FIG. 8B, whereas the aperture ratio of the pixel X4 (2n, 2m) can be regarded as 100%. In the adjacent pixel X4 (2n + 1, 2m), the light incident on the portion dx / 5 with respect to the pixel width dx in the vertical direction is not converted into a signal, so the aperture ratio is 80%. In addition, the pixel X4 (2n + 1, 2m + 1) in the oblique direction has an aperture ratio of light that is not converted into a signal because light incident on the portion dx / 5 with respect to the width dx of the pixel in the horizontal direction and the vertical direction is not converted. 64%.
At this time, the following equation (8) corresponds to the equation (2), and the frequency response R2 (v) corresponding to the equation (6) becomes the following equation (9).

x3 (n, m) = x4 (2n, 2m)

+ {x4 (2n-1,2m) + x4 (2n, 2m-1) + x4 (2n, 2m + 1) + x4 (2n + 1,2m)} × 2/5

+ {x4 (2n-1,2m-1) + x4 (2n-1,2m + 1) + x4 (2n + 1,2m-1) + x4 (2n + 1,2m + 1)} × 4/25

(8)

R2 (v) = 5/9 ・ {1 + 4/5 ・ cos (2π ・ v / 32dx ・ dx)}

= 5/9 ・ {1 + 4/5 ・ cos (π ・ v / 16)} (9)

The above explanation will be expanded when the ratio of the portion where the incident light with respect to the horizontal and vertical width 2dx of the pixel is converted into a signal is a (a is 1 or less, 1/2 or more). Then, the frequency response R3 (v) is expressed by the following equation (10).

R3 (v) = 1 / 2a ・ {1+ (2a-1) ・ cos (2π ・ v / 32dx ・ dx)}

= 1 / 2a ・ {1+ (2a-1) ・ cos (π ・ v / 16)} (10)

At this time, the correction coefficient of the frequency correction circuit may be set to the reciprocal of the frequency response thus obtained.

  As described above, according to the interpolation signal generation method for image signals of the present invention, frequency correction suitable for the process of dividing the pixels of the solid-state imaging device and doubling the number of pixels becomes possible. As a result, the number of pixels of the image signal obtained by the solid-state imaging device can be doubled with sufficient accuracy by DCT processing and IDCT processing for the frequency coefficient after frequency correction. As a result, even when a full high-definition image signal is displayed on a display device that supports 4K, a good image signal close to an image signal that should originally be obtained by a solid-state imaging device corresponding to a 4K image can be obtained.

DESCRIPTION OF SYMBOLS 1 Imaging device 2, 4 Interpolation signal production | generation apparatus 101 Optical system 102 Solid-state image sensor 103 Color signal processor 201 Block extraction circuit 202 DCT processing circuit 203 Frequency correction circuit 204 IDCT processing circuit 205 Rearrangement circuit 206 Filter processing circuit 301 1H delay circuit 302 1 Pixel delay circuit

Claims (4)

A first pixel group comprising a block of N × N pixels formed by N pixels continuous in the horizontal direction and N pixels continuous in the vertical direction from an image signal composed of a plurality of pixels arranged two-dimensionally. Extract (N is an integer),
The signal obtained from the first pixel group is subjected to orthogonal transformation to generate a first frequency coefficient group,
Applying a frequency correction to the first frequency coefficient group to generate a second frequency coefficient group;
The first frequency coefficient group is subjected to inverse orthogonal transformation for generating a signal at an interval smaller than the pixel interval in the first pixel group, and at least a first position where no pixel exists in the first pixel group Generate a first signal corresponding to
In an interpolation signal generation method for an image signal using the first signal as a signal corresponding to the first position in the image signal,
In the frequency correction, the interval between the pixels in the image signal is 2dx, the ratio in one direction in which light incident on the pixel is converted into a signal is a (1/2 <a <1), and the spatial frequency of the image signal A method for generating an interpolation signal for an image signal, which is based on a correction coefficient corresponding to 2a / {1+ (2a-1) · cos (2π · f · dx)} where is a.   2. The method for generating an interpolated signal for an image signal according to claim 1, wherein the orthogonal transform is a discrete cosine transform, and the inverse orthogonal transform is an inverse discrete cosine transform. A first pixel group comprising a block of N × N pixels formed by N pixels continuous in the horizontal direction and N pixels continuous in the vertical direction from an image signal composed of a plurality of pixels arranged two-dimensionally. Extract (N is an integer),
A process of performing orthogonal transformation on a signal obtained from the first pixel group to generate a first frequency coefficient group;
A process of generating a second frequency coefficient group by performing frequency correction on the first frequency coefficient group;
The first frequency coefficient group is subjected to inverse orthogonal transformation for generating a signal at an interval smaller than the pixel interval in the first pixel group, and at least a first position where no pixel exists in the first pixel group A filter operation corresponding to the process of generating the first signal corresponding to is performed on the first pixel group,
In an interpolation signal generation method for an image signal using the first signal as a signal corresponding to the first position in the image signal,
In the frequency correction, the interval between the pixels in the image signal is 2dx, the ratio in one direction in which light incident on the pixel is converted into a signal is a (1/2 <a <1), and the spatial frequency of the image signal A method for generating an interpolation signal for an image signal, which is based on a correction coefficient corresponding to 2a / {1+ (2a-1) · cos (2π · f · dx)} where is a.   4. The method for generating an interpolated signal for an image signal according to claim 3, wherein the orthogonal transform is a discrete cosine transform, and the inverse orthogonal transform is an inverse discrete cosine transform.

Images