JPH05167920A - Image expanding device and image signal recording and reproducing device - Google Patents

Abstract

PURPOSE:To obtain an image expanding device capable of obtaining a clear expanded image for which the degradation of the picture quality is less and a compact image signal recording and reproducing device provided with an image expanding function. CONSTITUTION:An image expanding device is provided with an (MXM) point DCT circuit 2 for applying orthogonal transformation to the image signal of a blocked original image, a '0' data adding circuit 3 for adding '0' data to an obtained original image frequency component and forming an expanded image frequency component and an (NXN) point IDCT circuit 5 for applying inverse orthogonal transformation to an expanded image frequency component multiplied by a constant N/M. An image expanding device is built in an image signal coding/decoding circuit or a sub-band dividing/composing circuit in the image signal recording and reproducing device.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image enlarging device and an image signal recording / reproducing device, and more particularly to an image signal recording / reproducing device with an image pickup device, a so-called video movie.

[0002]

2. Description of the Related Art A video movie is required to be equipped with a function that requires image enlargement processing such as electronic automatic image stabilization and electronic zoom, and image enlargement processing with higher image quality and smaller hardware scale is desired. The demand for methods is increasing.

As a conventional image enlarging device, for example, there is one disclosed as a part of a video camera device in Japanese Patent Laid-Open No. 2-250471. Hereinafter, this Japanese Patent Application Laid-Open No. 2-25
Image enlargement processing in a conventional image enlargement apparatus according to the technique disclosed in Japanese Patent No. 0471 will be described with reference to FIG.

FIG. 1 shows an example of the positional relationship of pixels between an original image and an enlarged image. In the figure, circles (◯) indicate pixels of the original image and triangles (Δ) indicate pixels of the enlarged image. ing. The pixel value of the enlarged image is obtained by linearly interpolating the pixel values of the surrounding original image. In the example shown in FIG. 1, the grayscale value Z of the pixel z of the enlarged image is the grayscale values A, B, and 4 of the four pixel points a, b, c, d of the surrounding original image.
Calculate from C and D.

In FIG. 1, s and t are numerical values representing the positional relationship between the pixel z and the four pixels a, b, c and d of the surrounding original image. The value Z of the pixel z of the enlarged image is expressed by the following equation (1).
Required by.

Z = (1-s) (1-t) * A + (1-s) t * B + s (1-t) * C + st * D (1)

FIG. 2 is a block diagram showing a configuration of an interpolation operation circuit for performing the operation of the equation (1). The interpolation calculation circuit includes multipliers 101, 102, 103, 104 for multiplying the grayscale values A, B, C, D by a predetermined constant, an adder 105 for adding the outputs of the multipliers 101, 102, and an adder 106 for adding the outputs of the multipliers 103, 104. And a multiplier 107 that multiplies the output of the adder 105 by a predetermined constant, a multiplier 108 that multiplies the output of the adder 106 by a predetermined constant, and the outputs of the multipliers 107 and 108, and outputs a gray value Z. Adder 109 for

Each of the multipliers 101, 102, 103 and 104 has the values A, B, C and D of the four pixels a, b, c and d of the original image, respectively.
Is input to the multiplier 101, the values A and (1-t) of the pixel a, the multiplier 102 and the values B and t of the pixel b, and the multiplier 103 and the values C and (1-t of the pixel c. ) And the multiplier 104 multiplies the values D and t of the pixel d, respectively. The multiplication results of the multipliers 101 and 102 are added by the adder 105, and the multiplication results of the multipliers 103 and 104 are added by the adder 106. The addition result of the adder 105 is multiplied by (1-s) by the multiplier 107, and the addition result of the adder 106 is multiplied by s by the multiplier 108. Finally,
The multiplication results of the multiplier 107 and the multiplier 108 are added by the adder 109 to calculate the value Z of the pixel z of the enlarged image.

[0009]

The conventional image enlarging apparatus and the conventional image signal recording / reproducing apparatus having the image enlarging function perform the interpolation operation as shown in FIG. 2 in order to perform the operation of the equation (1). Using a circuit. However, since this interpolation calculation circuit functions as a kind of low-pass filter, there is a problem that the image quality of the enlarged image output as the value Z deteriorates and edges and the like are blurred.

The present invention has been made to solve such a problem, and an image enlarging apparatus capable of obtaining an enlarged image in which the quality of an enlarged image is extremely small and edges are sharp. The main purpose is to provide.

Another object of the present invention is to provide an image signal recording / reproducing apparatus in which a high-quality image enlarging apparatus can be introduced while avoiding a great increase in hardware.

[0012]

The image enlarging device according to the first invention of the present application is an orthogonal transform means for performing orthogonal transform on data of an original image to obtain a frequency component, and to the frequency component.
Add "0" data to obtain an enlarged image frequency component whose data size is larger than the frequency component of the original image "0"
The data adding means and the inverse orthogonal transforming means for performing the inverse orthogonal transform corresponding to the size of the enlarged image frequency component are provided.

An image enlarging apparatus according to the second invention of the present application comprises a blocking means for dividing an image signal into blocks, and an orthogonal transform means for performing an orthogonal transform on each original image block to obtain an original image frequency component block. "0" data addition means for adding "0" data to this frequency component block to obtain an enlarged image frequency component block having a larger data size than the original image frequency component block, and the size of this enlarged image frequency component block It is characterized by comprising inverse orthogonal transformation means for correspondingly performing inverse orthogonal transformation to obtain an enlarged image block, and reconstruction means for arranging the obtained enlarged image blocks to form an enlarged image. The orthogonal transform and the inverse orthogonal transform in the second invention may be a one-dimensional orthogonal transform and a one-dimensional inverse orthogonal transform that are independent in the horizontal direction and the vertical direction, respectively.
Two-dimensional orthogonal transform and two-dimensional inverse orthogonal transform in block units may be used.

An image enlarging apparatus according to the third invention of the present application is an orthogonal transform means for performing an orthogonal transform on an image signal to obtain a frequency component and a frequency component of an original image by adding "0" data to the frequency component. "0" data adding means for obtaining a magnified image frequency component having a larger data size, and multiplication means for multiplying the magnified image frequency component by a constant according to the size ratio between the frequency component of the original image and the magnified image frequency component, And an inverse orthogonal transform means for performing an inverse orthogonal transform according to the size of a magnified image frequency component multiplied by a constant.

The image enlarging apparatus according to the fourth aspect of the present invention is an orthogonal transform means for performing a orthogonal transform using the reciprocal of the size of the image signal as a coefficient to obtain a frequency component, and the frequency component
"0" data adding means for adding data of "0" to obtain an enlarged image frequency component having a larger data size than the frequency component of the original image, and corresponding to the size of this enlarged image frequency component for this enlarged image frequency component In the fourth aspect of the present invention, the orthogonal transformation and the inverse orthogonal transformation are performed by the following equations, for example. 2D DCT (discrete cosine transform) and 2D IDCT (inverse discrete cosine transform), or 1D DC
T and one-dimensional IDCT. .

[0016]

[Equation 1]

[0017]

[Equation 2]

The image enlarging apparatus according to the fifth aspect of the present invention interpolates "0" data by interpolating "0" data between pixels of an image signal to obtain an integer multiple of the number of pixels. And a low-pass filter for extracting only low-frequency components from the subsequent image signal.

The image signal recording / reproducing apparatus according to the sixth invention of the present application uses the orthogonal transforming means and the inverse orthogonal transforming means for compressing / decompressing the image signal at the time of recording / reproducing to obtain the first, second, third, and third means.
4 is a feature of realizing the image enlarging device shown in the invention. According to the sixth aspect of the invention, as the orthogonal transforming means used when compressing the image signal, at least one orthogonal transforming means for processing smaller size data is provided in addition to the orthogonal transforming means for processing normal size data. As the inverse orthogonal transforming means used when the image signal is restored, at least one inverse orthogonal transforming means for processing larger size data is provided in addition to the inverse orthogonal transforming means for processing normal size data.

An image signal recording / reproducing apparatus according to the seventh invention of the present application includes a cutting-out means for cutting out a part of an image signal, a first state for inputting each sub-band component to all sub-band channels, and a DC component. And a switching selection means for switching between two states, that is, a second state in which the output of the cutout means is input to the subband channel for the subband component and "0" data is input to the other subband channels. To do.

An image signal recording / reproducing apparatus according to an eighth invention of the present application is the image signal recording / reproducing apparatus according to the sixth and seventh inventions, wherein a detecting means for detecting a blur amount of a captured image, a memory for storing an image signal corresponding to the image, And a memory control means for controlling a read area of the memory.

The image signal recording / reproducing apparatus according to the ninth invention of the present application is the image signal recording / reproducing apparatus according to the sixth invention, wherein a plurality of image enlarging means having different enlargement ratios, a memory for storing an image signal corresponding to a picked-up image signal, and an enlargement ratio. And a control unit for controlling selection of the read area of the memory and the image enlarging unit.

The image signal recording / reproducing apparatus according to the tenth invention of the present application is, in the eighth and ninth inventions, a display means for displaying the read area from the memory, that is, the area to be enlarged, by superimposing it on the image signal before being enlarged. Is further provided.

[0024]

In the first aspect of the invention, "0" data is added to the frequency component obtained by orthogonally transforming the image signal of the original image to create an expanded frequency component having a large data size, and the created expanded frequency component is Inverse orthogonal transformation is performed to obtain enlarged image data. Since a magnified image is obtained by orthogonal transformation, a natural magnified image is obtained with little deterioration in image quality.

In the second invention, "0" data is added to the frequency component block obtained by orthogonally transforming the image signal in block units, an enlarged frequency component block having a large data size is produced, and the produced enlargement is made. The frequency component block is subjected to inverse orthogonal transform to obtain a magnified image block forming a magnified image. Similar to the first aspect of the present invention, a natural enlarged image can be obtained, and processing suitable for each block can be performed.

In the third invention, in addition to the operation of the first invention,
Since the coefficient of the frequency component obtained by the orthogonal transformation is adjusted, an enlarged image with the same signal level as the original image can be obtained.

In the fourth invention, in addition to the operation of the first invention,
Since the orthogonal transformation is performed using an equation having the inverse of the size of the frequency component obtained by the orthogonal transformation as a coefficient, and the inverse orthogonal transformation is performed using the equation not having the inverse of the size of the expanded frequency component as the coefficient, Without any adjustment
An enlarged image with the same signal level as the original image is obtained.

In the fifth aspect of the invention, the digital image signal is
An enlarged image is obtained by interpolating "0" data and passing through a low-pass filter. In the fifth invention, interpolation processing of "0" data in both horizontal and vertical directions of the digital image signal and low-pass filter pass processing Then, a magnified image magnified in the horizontal and vertical directions is obtained.

In the sixth invention, in addition to the operations of the first to fourth inventions, the image enlargement is performed by using the orthogonal transformation and the inverse orthogonal transformation used for the encoding / decoding of the image signal, so that the encoding of the image signal is performed. -Decoding processing and image enlargement processing are performed simultaneously.

In the seventh invention, the image is enlarged in the sub-band dividing / combining apparatus. In the seventh invention, if the low pass filter, the high pass filter, the sub-sampling means, and the "0" data interpolating means are provided for both vertical and horizontal directions, the sub-band dividing / synthesizing device can make the image horizontal, Expand in each direction vertically. In addition, the ratio of the cutoff frequency of the vertical low-pass filter and the high-pass filter to the highest vertical frequency of the image signal, and the horizontal low-pass frequency of the highest horizontal frequency of the image signal. If the ratios of the cutoff frequencies of the pass filter and the high pass filter are set to be the same, and the operations of the sub-sampling means and the "0" data interpolating means in the horizontal and vertical directions are the same. , The horizontal and vertical magnifications are the same. Further, even in an image signal recording / reproducing apparatus that performs subband division only in the horizontal direction or the vertical direction, image enlargement processing is performed in the direction in which subband division is not performed before subband division, and then the subband division is performed. When the image is enlarged in the remaining direction by the band dividing / combining circuit, an enlarged image having the same enlargement ratio in the horizontal direction and the vertical direction can be obtained.

In the eighth aspect of the invention, in addition to the actions of the sixth and seventh aspects of the invention, the image enlarging process for automatic camera shake correction and the image signal encoding / decoding process are simultaneously performed.

In the ninth invention, in addition to the operation of the sixth invention,
The image enlargement process of the electronic zoom and the encoding / decoding process of the image signal are simultaneously performed.

In the tenth invention, in addition to the operations of the eighth and ninth inventions, an enlarged and reproduced area is displayed during recording.

[0034]

DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described below in detail with reference to the drawings showing the embodiments thereof.

(First Embodiment) FIG. 3 is a block diagram showing the arrangement of an embodiment of the image enlarging apparatus of the present invention. As shown in FIG. 3, in this embodiment, a block division circuit 1 for dividing an image signal into blocks, an M × M point DCT circuit 2 for performing DCT (discrete cosine transform) on each block, and data of “0” are predetermined. "0" data adding circuit 3 for adding to frequency component, multiplier 4 for multiplying the output of "0" data adding circuit 3 by a constant, IDCT (inverse discrete cosine transform) for each block
An N × N point IDCT circuit 5 for performing the process and an image reconstruction circuit 6 for reconstructing the data of each block are connected in series in this order.

In this embodiment, the line segment ratio is enlarged by N / M times. Here, M and N are integers, and N is larger than M. In addition, as the data of the original image, it is assumed that the luminance level of the monochrome image is converted into a digital signal. However, if the original image is a color image, the original image data is
For example, three digital data indicating the brightness of each of red (R), green (G), and blue (B) components, or a luminance component Y
And the color difference components U and V are composed of three pieces of digital data and the like, the circuit shown in FIG. 3 processes each component.

Next, the operation will be described. The original image data is input to the block division circuit 1 and divided into data for each block composed of M × M pixels. The image enlarging process is performed for each block by sequentially performing the following process on the original image block data. Less than,
The original image block data of M × M pixels is {x (m, n); m, n =
0, 1, ..., M-1} will be described.

The original image block data x (m, n) divided by the block dividing circuit 1 is then input to the M × M point DCT circuit 2. The M × M point DCT circuit 2 performs two-dimensional DCT of size M × M on the original image block data, and the original image frequency component {y (k, l); k, l = 0,1, ..., M-1} is generated.

Here, M × M point DCT and N × N point ID
Two-dimensional DCT and two-dimensional IDCT shown below are used as CT.

[0040]

[Equation 3]

The original image frequency component y (k, l) generated by the M × M point DCT circuit 2 is input to the “0” data adding circuit 3. The size of the “0” data addition circuit 3 is M × M
By adding "0" data to y (k, l) which is the data of N, data of N × N is generated.

Next, the output of the "0" data adding circuit 3 is multiplied by the constant N / M by the multiplier 4 to obtain N / M.
M times the magnified image frequency component {y '(k, l); k, l = 0,1,
, N-1} is generated. Figure 4 shows the original image frequency component y (k, l)
FIG. 9 is a schematic diagram showing the above-described procedure for generating an enlarged image frequency component y ′ (k, l) from the image.

When the frequency component y '(k, l) of the enlarged image is represented by the original image frequency component y (k, l), the following equation is obtained.

Y '(k, l) = N / M y (k, l) (0 ≤ k, l ≤ M-1) y' (k, l) = 0 (others)

The reason why "0" data is added to the high-order frequency component by the "0" data adding circuit 3 is as follows.

When the enlarged image frequency component of size N × N is generated from the original image frequency component of size M × M, since the original image does not have the information of higher-order frequency components, some corresponding It is necessary to add a value to make up for it.
Here, when data having a strong correlation such as natural image data is converted by the DCT, the high-order frequency components are averaged due to the DCT characteristic that the degree of energy concentration on the low-order frequency components is large. There is a property that the dispersion is very small at 0 ″. Therefore, when giving the same value to all blocks as high-order frequency components, the value "0" gives the enlarged image the closest state to the original image, and deterioration due to the loss of high-order frequency components. Also less. Further, when this method is used, the high-frequency component of the magnified image is set to "0", and therefore a magnified image that is smooth and has no natural discomfort can be obtained.

Further, the frequency component of the multiplier 4 is N /
The reason for multiplying by M is as follows.

The orthogonal transform equalizes the energy of the signal to be transformed and the energy of the transform coefficient (frequency component). That is, if the magnitude of the frequency component is not adjusted,
The original image and the enlarged image have the same energy. Since the enlarged image has more pixels than the original image, the energy per pixel is small. As a result, the brightness level and the contrast decrease. Therefore, the frequency component is N /
By multiplying by M times, the energies per pixel of the original image and the enlarged image are made equal, and an image having the same brightness level and contrast is obtained.

Magnified image frequency component obtained as described above
y ′ (k, l) is input to the N × N point IDCT circuit 5.

The N × N point IDCT circuit 5 is a two-dimensional IDC whose size is N × N with respect to the enlarged image frequency component y ′ (k, l).
Enlarged image block data {z (m, n);
m, n = 0,1, ..., N-1} is generated.

The enlarged image block data z (m, n) generated by the N × N point IDCT circuit 5 is input to the image reconstruction circuit 6. The image reconstruction circuit 6 includes the block division circuit 1
By performing the process reverse to that, the enlarged image block data is collected and an enlarged image is generated.

Next, a specific operation of the image enlarging device as described above will be described. The size of the original image is 720 x 480
In the above embodiment, M =
An example will be described in which the line segment ratio is doubled with 4, N = 8 to obtain an enlarged image of the same size as the original image. The number of pixels of the original image is based on the assumption that the television signal is sampled at 4 fsc. If it is a color image, R
Enlarge each of the three signals such as GB or YUV,
Alternatively, there is a method such as enlarging a signal in which a luminance signal and a color difference signal are combined, but this time, consider the case of enlarging one of these signals.

FIG. 5 is a block diagram showing the construction of an embodiment of an image enlarging device for enlarging the above-mentioned line segment ratio by a factor of two. In the figure, the parts with the same numbers as in FIG. 3 indicate the same members. Reference numeral 7 in the drawing denotes an image cutout circuit which cuts out a part of the image according to the enlargement ratio. Further, since M = 4 and N = 8, a 4 × 4 point DCT circuit 12 and an 8 × 8 point IDCT circuit
Use 15.

Next, the operation will be described. First, the signal of the original image is input to the image cutout circuit 7. The image cutout circuit 7 cuts out a part of the image according to the enlargement ratio. When the line segment ratio is doubled, an image with a line segment ratio of 1/2, that is, an area ratio of 1/4 is cut out from the original image. In this embodiment, since the size of the original image is 720 × 480 pixels, an image of 360 × 240 pixels is cut out. The cut-out position may be fixed or variable. If it is variable, a signal indicating the cutout position is input to the image cutout circuit 7.

The output of the image cutout circuit 7 is input to the block division circuit 1. In block division circuit 1, 90 × 60
Blocks, that is, 5400 total 4 × 4 pixel original image block data {x (0,0), x (0,1), ..., X (3,3)}. Each original image block data is processed by the 4 × 4 point DCT circuit 12 into the original image frequency components {y (0,0), y (0,1),
..., y (3,3)}. The original image frequency component is "
"0" data is added by the 0 "data adding circuit 3 to form 8x8 data.
It is quadrupled, that is doubled, and the enlarged image frequency component {y '(0,0), y'
(0,1), ..., y '(7,7)}.

FIG. 6 is a schematic diagram showing an example of a state in which the enlarged image frequency component is generated from the original image frequency component when the enlargement processing is performed at the enlargement ratio of 2 as described above. Figure 6
The original image frequency component shown in (a) is input to the "0" data adding circuit 3 to add 8x8 data as shown in FIG. 6 (b). become.
Further, as shown in FIG. 6C, the multiplier 4 multiplies N / M (in this case, double) to generate an enlarged image frequency component.

The magnified image frequency component thus obtained is magnified image block data {z (0,0), z (0,1), ..., Z (7,7) by the 8 × 8 point IDCT circuit 15. )}.

The enlarged image block data converted by the 8 × 8-point IDCT circuit 15 is then input to the image reconstruction circuit 6 to generate enlarged image data having a size of 720 × 480 pixels.

By the above processing, it is possible to obtain an enlarged image of 720 × 480 pixels, which is a partial enlargement of the image of 360 × 240 pixels from the original image of 720 × 480 pixels.

Next, the result of actual image enlargement according to the embodiment of the present invention will be described with reference to FIGS. 7, 8 and 9.

FIG. 7 is a schematic diagram showing an example in which a two-dimensional spectrum is obtained from the image data used as the original image and is displayed in contour lines. FIG. 8 is a schematic view showing an example in which a two-dimensional spectrum is obtained from a magnified image obtained by magnifying an original image with a line segment ratio of 4/3 by the bilinear interpolation method shown as the conventional technique and contour lines are displayed. is there. FIG. 9 shows contour lines obtained by obtaining a two-dimensional spectrum from an enlarged image obtained by enlarging an original image with a line segment ratio of 4/3 with M = 3 and N = 4 in the method shown in the embodiment of the present invention. It is a schematic diagram which shows an example.

It can be seen from FIG. 8 that the high-frequency component is missing from the enlarged image obtained by the conventional image enlarging device. This is because, as described above, the image enlarging apparatus of the conventional example has an effect of a kind of low-pass filter. On the other hand, FIG.
From the results, it can be seen that the spectrum of the enlarged image obtained by the embodiment of the present invention is very close to the spectrum of the original image shown in FIG. 7 as compared with FIG. That is, the enlarged image obtained by the embodiment of the present invention has almost no loss of information of the original image in the frequency domain. The spectrum is reduced because the image is enlarged.

As described above, according to the image enlarging apparatus of the present invention, it is possible to enlarge the image with almost no loss of information of the original image, and it is possible to obtain an enlarged image with very little deterioration in image quality.

The present invention is not limited to the above-mentioned embodiments, and various modifications can be made. For example, although the two-dimensional DCT and the two-dimensional IDCT are used in the above embodiment, it is possible to use the one-dimensional DCT and the one-dimensional IDCT instead of them. In this case, one-dimensional DCT and 1
The dimension IDCT is represented by the following equation.

[0065]

[Equation 4]

In this case, one-dimensional DCT and one-dimensional IDCT
By performing the enlarging processing by and in each of the vertical direction and the horizontal direction, an enlarged image equivalent to that in the above embodiment can be obtained. Further, when the one-dimensional DCT and the one-dimensional IDCT are used to perform expansion in only one direction in the vertical direction or the horizontal direction, the multiplying factor is (N / M) ^0.5 .

The orthogonal transform is not limited to the cosine transform, and the same effect can be obtained by using the orthogonal transform such as the sine transform.

Furthermore, in the embodiment, although the block division is performed and then the enlargement processing is performed, it is also possible to collectively orthogonally transform the entire image without dividing the block.

(Second Embodiment) Next, a second embodiment in which the image enlarging device shown in the first embodiment is improved to an image enlarging device which does not require a multiplier will be described.

FIG. 10 is a block diagram showing the structure of the second embodiment. Comparing FIG. 10 with FIG. 3 of the first embodiment, there is no multiplier, and there is no M × M point DCT circuit 22 and N × N point IDCT.
It is the same as FIG. 3 except that the circuit 25 is an improved version.

The processing performed by the improved M × M point DCT circuit 22 and the improved N × N point IDCT circuit 25 will now be described. First, general one-dimensional DCT and one-dimensional IDCT
And will be described.

[0072]

[Equation 5]

The above equations (2) to (4) can be expressed in matrix as follows. Note that the following formula []
^T represents a transposed matrix.

[0074]

[Equation 6]

Here, [C] _{k, m} = c _k (m) means that the matrix C has c _k (m) as its (k, m) element, and this matrix C is It is called the cosine transformation matrix. The fact that the cosine transform is an orthogonal transform means that
That is, the cosine transform matrix is an orthogonal matrix that satisfies the following conditions.

C ^-1 = C ^T

From this, the matrix C becomes as follows.

| C | = 1 (6)

Here, the following energy conservation law is established from equations (5) and (6).

[0080]

[Equation 7]

This means that when considering the case of converting an image signal, the energy of the image signal is equal to the conversion coefficient, that is, the energy of the frequency component.

Next, the equations of the one-dimensional DCT and the one-dimensional IDCT shown in the first embodiment will be considered. The formula is shown below.

[0083]

[Equation 8]

This changes the size of data processed by DCT and IDCT. In formulas (7) to (9), N =
Let M be a general one-dimensional DCT. for that reason,
When the original image signal is converted into a frequency component by the above equation, its energy is preserved, and when the frequency component is inversely converted into a magnified image by the above equation, its energy is preserved. That is, the original image signal {x (0), x (1), ..., x (M-1)} and the enlarged image signal {y (0), y (1), ..., y (N-1)} That energy is the same. However, since the size N of the enlarged image signal is larger than the size M of the original image signal, if the inverse transformation is performed without adjusting the magnitude of the transformation coefficient, the enlarged image will have a small signal level. That is, the brightness level is lowered and the contrast is lowered. Therefore, in the first embodiment, the multiplier 4 multiplies the conversion coefficient by a constant in accordance with the enlargement ratio, that is, (N / M) ^0.5 times to increase the energy, and in the enlarged image, the same brightness level as that of the original image and I tried to get the contrast.

What has been described above is the same for a two-dimensional DCT. However, the multiplier 4 multiplies (N / M) times.

In the second embodiment, this problem is solved by changing the expressions of DCT and IDCT. The formulas of the one-dimensional DCT and the one-dimensional IDCT used in the second embodiment are shown below.

[0087]

[Equation 9]

When conversion / inverse conversion is performed with M = N in these equations, the original data is restored, and there is no difference from general discrete cosine conversion. However, the energy conservation law is not established, and this transformation is not an orthogonal transformation. The energy of the transform coefficient becomes smaller than the energy of the transform coefficient obtained by performing general DCT.

However, in these equations (10) to (13), N>
When the image signal is converted as M to obtain an enlarged image, the luminance level and the contrast of the enlarged image become the same as the luminance level and the contrast of the original image. This is equation (13)
This is because there is no coefficient (1 / N) ^0.5 or (2 / N) ^0.5 having the reciprocal of N that represents the size of data in f _k (m) represented by.

As described above, in the second embodiment, an enlarged image having the same luminance level and contrast as the original image can be obtained without providing a multiplier as a coefficient adjusting means.

Although the discrete cosine transform has been described above, the present invention is not limited to the above-described embodiment, but can be applied to the case where the enlargement process is performed using another orthogonal transform such as the discrete sine transform. ..

(Third Embodiment) Next, the first and second embodiments
A third embodiment of the present invention will be described in which the image enlarging apparatus shown in the embodiment is used to simultaneously perform electronic image stabilization image enlargement and image signal compression / decompression. FIG. 11 is a block diagram showing the structure of the image signal recording / reproducing apparatus in the third embodiment.

In FIG. 11, reference numeral 31 is an image pickup element for converting an image formed on the image pickup surface into an electric signal.
Is an A / D that converts analog signals to digital signals
A converter 32, a process circuit 33 for converting into an image signal in consideration of visual characteristics, and an encoder 34 for converting into a standardized image signal such as NTSC are connected in series in this order.
The encoder 34 includes a memory 35 for writing / reading an image signal, a motion vector detecting circuit 36 for detecting the direction and magnitude of image movement from the image signal, and a D / A converter for converting a digital signal into an analog signal. 39 and are connected. The motion vector detection circuit 36 is connected to a memory control circuit 37 that controls the read address of the memory 35 so as to cancel the motion according to the output of the motion vector detection circuit 36. To the D / A converter 39, a superimposing circuit 40 for superimposing on-screen information on an image signal and an EVF 41 for monitoring a signal to be photographed are connected in series in this order. In memory 35, switch 45, image data 8 ×
8 × 8 DCT circuit 42 for converting to frequency components by 8 DCT, switch 46, variable length coding circuit 47 for performing variable length coding by Huffman code, etc., error correction code for correcting errors when errors occur during reproduction A conversion circuit 48 and a recording processing circuit 49 such as a recording amplifier are connected in series in this order. Between the switch 45 and the switch 46, image data 4
4x4 DCT that is converted to frequency components by x4 DCT
A series path with the "0" data adding circuit 44 which adds "0" data to the output of the circuit 43 and the 4x4 DCT circuit 43 to make 8x8 data is provided in parallel with the 8x8 DCT circuit 42. .. Depending on the output of the standard / enlargement switching circuit 38, which switches between "standard processing" for performing normal processing and "enlargement processing" for enlarging the image to correct camera shake, both switches 4
5, 46 can be switched. That is, the switch 45 inputs the output of the memory 35 into the 8 × 8 DCT circuit 42 or the 4 × 4 DCT circuit 43 according to the output of the standard / enlargement switching circuit 38, and the switch 46 outputs the output of the standard / enlargement switching circuit 38. In response, the output of the 8 × 8 DCT circuit 42 or the “0” data adding circuit 44 is input to the next processing circuit. Further, 50 indicates a recording medium such as a magnetic tape, and 51 to 55 indicate constituent members on the reproducing (decoding) side. That is, a reproduction processing circuit 51 such as a reproduction amplifier, an error correction decoding circuit 52 for correcting an error, a variable length decoding circuit 53 for performing decoding for encoding by the variable length coding circuit 47, and frequency component data of 8 × 8 × 8 IDCT circuit 54 for converting image data by 8 IDCT, D / A converter 55 for converting digital signal to analog signal
Are connected in series in this order.

The 8 × 8 DCT circuit 42, 4 × 4 DCT
The circuit 43 and the 8 × 8 IDCT circuit 54 are DCs used in the image enlarging apparatus which does not require the multiplier shown in the second embodiment.
A T circuit and an IDCT circuit.

Next, the operation of such an image signal recording / reproducing apparatus when the image enlarging apparatus is used for camera shake correction will be described.

First, when "standard processing" is selected in the standard / enlargement switching circuit 38, the memory 35 divides the input image signal into 8 × 8 blocks. The output of the memory 35 is input to the 8 × 8 DCT circuit 42 through the switch 45 and converted into an 8 × 8 frequency component. The output of the 8 × 8 DCT circuit 42 passes through the switch 46 and the variable length coding circuit 47.
To the error correction coding circuit 48 and the recording processing circuit 49, and recorded on the recording medium 50.

On the reproducing side, the signal obtained from the recording medium 50 is
The data is processed by the reproduction processing circuit 51, the error correction decoding circuit 52, and the variable length decoding circuit 53, inversely converted into image data by the 8 × 8 IDCT circuit 54, and then analogized by the D / A converter 55. A reproduced image signal is obtained.

Next, if "enlargement processing" is selected in the standard / enlargement switching circuit 38, the motion vector detection circuit
When the motion detection data 36 is obtained and motion correction data is obtained, the memory control circuit 37 controls the read area of the memory 35 based on this motion correction data to obtain a stable image signal.

The process of motion correction at this time will be described with reference to FIG. In FIG. 12, the effective area of the output signal of the encoder 34, that is, the writing area of the memory 35 is V0 in the vertical direction × H0 in the horizontal direction. On the other hand, the area 1 (V1 × H1) or the area 2 (V2 × H2) smaller than the effective area (V0 × H0) of the original signal is read from the memory 35. Now, when the subject moves from the state 1 to the state 2, the movement amount is detected, the area 1 is read out from the memory 35 in the state 1, and the area 2 is read out from the memory 35 in the state 2, and the original size is expanded. It is possible to realize camera shake correction.

In the embodiment shown in FIG. 11, 8/4 times, that is, 2
Since the expansion is performed twice, the area 1 and the area 2 are each half in the vertical and horizontal directions as compared with the effective area of the original signal, that is, the area ratio is 1
The area is / 4. That is, V1 = V2 = 1 / 2.V0 H1 = H2 = 1 / 2.H0.

The memory 35 divides the image data in the area thus determined into 4 × 4 data and outputs it. The output of the memory 35 passes through the switch 45 and 4 × 4DC
It is input to the T circuit 43. The output of the 4x4 DCT circuit 43 is "
It is input to the 0 "data adding circuit 44 and becomes 8x8 data. The output of the" 0 "data adding circuit 44 is input to the variable length coding circuit 47 through the switch 46 and the error correction coding circuit 48. Then, after being processed by the recording processing circuit 49, it is recorded in the recording medium 50.

The reproducing side performs exactly the same processing as in the case of the "standard processing".

(Fourth Embodiment) Next, the first and second embodiments
A fourth embodiment of the present invention will be described in which the image enlarging device shown in the embodiment is used to simultaneously perform electronic zoom image enlargement and image signal compression / decompression. FIG. 14 is a block diagram showing the arrangement of an image signal recording / reproducing apparatus according to the fourth embodiment.

In FIG. 14, parts given the same numbers as in FIG. 11 indicate the same or corresponding parts, and therefore their explanations are omitted. In the figure, 57 and 58 are both switches, both switches
Between 57 and 58, 8 × 8 DCT circuit 42 and 6 image data
6 × 6 DCT for converting to frequency components by × 6 DCT
The circuit 59 and the serial path of the "0" data adding circuit 60 for adding "0" data to the frequency component of 6x6 to make the size of 8x8, the 4x4 DCT circuit 43 and the "0" data adding circuit 44
And the serial path of are arranged in parallel. Both switches 57, 58
Are switched by the enlargement control circuit 56 which controls the magnification of the electronic zoom. The switch 57 inputs the output of the memory 35 to the 8 × 8 DCT circuit 42 or the 6 × 6 DCT circuit 59 or the 4 × 4 DCT circuit 43 according to the output of the enlargement ratio control circuit 56, and the switch 58 outputs the output of the enlargement ratio control circuit 56. 8 according to output
The output of the x8 DCT circuit 42 or the "0" data adding circuit 60 or the "0" data adding circuit 44 is converted to the variable length coding circuit 47.
To enter.

The 6 × 6 DCT circuit 59 is a DCT circuit used in the image enlarging apparatus which does not require the multiplier shown in the second embodiment.

The zoom operation of the image signal recording / reproducing apparatus when the image enlarging apparatus is used for electronic zooming will be described with reference to FIG. In FIG. 15, the effective area of the output signal of the encoder 34, that is, the writing area of the memory 35 is V0 in the vertical direction × H0 in the horizontal direction. On the other hand, an area smaller than the effective area (V0 × H0) of the original signal is read from the memory 35. The read area from the memory 35 is changed from the effective area (V0 × H0) of the original signal to the area 3 (V3 × H3) and the area 4
The zoom-up can be realized by gradually reducing the value to (V4 × H4) and increasing the enlargement ratio of the image enlargement accordingly.

In the case of the embodiment shown in FIG. 14, the enlargement ratio of the image can be selected from three stages of 1 ×, 4/3 × and 2 ×.

When the standard processing, that is, the 1-time enlargement is selected by the switch 57, the read area of the memory 35 becomes the original effective area (V0 × H0) and is read by dividing it into 8 × 8 blocks. Be done. The output of the memory 35 is a switch
It is input to the 8 × 8 DCT circuit 42 through 57. 8x8D
The output of the CT circuit 42 is input to the variable length coding circuit 47 through the switch 58, and the error correction coding circuit 48 and the recording processing circuit.
After being processed at 49, it is recorded on the recording medium 50.

On the reproducing side, after being processed in the reproducing processing circuit 51, the error correction decoding circuit 52, and the variable length decoding circuit 53, it is inversely converted into an image signal by the 8 × 8 IDCT circuit 54 and the D / A converter 55. Thus, a reproduced image signal can be obtained.

When the enlargement of 4/3 times is selected by the switch 57, the read area of the memory 35 becomes the area 3 and V
The values of 3, H3 are as follows. V3 = 3 / 4.V0 H3 = 3 / 4.H0 Further, the data in the region 3 is read by being divided into 6 × 6 blocks and passed through the switch 57 to the 6 × 6 DCT circuit 59.
Entered in. The output of the 6 × 6 DCT circuit 59 is added with “0” by the “0” data adding circuit 60 to become 8 × 8 data, which is input to the variable length coding circuit 47 through the switch 58. The following process is exactly the same as the standard process, but a 4/3 times enlarged image can be obtained as a reproduced image.

When the double enlargement is selected by the switch 57, the read area of the memory 35 becomes the area 4, which is V4, H.
The value of 4 is as follows. V4 = 1 / 2.multidot.V0 H4 = 1 / 2.multidot.H0 Further, the data in the area 4 is divided into 4 × 4 blocks and read out, and the 4 × 4 DCT circuit 43 is passed through the switch 57.
Entered in. The output of the 4 × 4 DCT circuit 43 is added with “0” by the “0” data adding circuit 44 to become 8 × 8 data, which is input to the variable length coding circuit 47 through the switch 58. The following process is exactly the same as the standard process, but a double-enlarged image can be obtained as a reproduced image.

The feature of the third and fourth embodiments described above is that the image enlargement processing is also used as the information source encoding / decoding. The image information coding algorithm includes predictive coding,
Various algorithms such as transform coding and vector quantization have been proposed and put into practical use. Further, it is often used as a hybrid coding combining these. Since normal images have a high correlation, the effect of transform coding by orthogonal transform such as DCT is high, and when this transform coding is used in a recording / reproducing apparatus, an image that is less deteriorated over a long time even with a small storage capacity. Can be played.

Regarding the image enlargement processing using the orthogonal transformation / inverse orthogonal transformation, it is possible to obtain an enlarged image with very little deterioration in image quality as already described in the first and second embodiments. .. Further, the image enlargement processing can be realized by only slightly increasing the amount of hardware by using the image enlargement processing also as the orthogonal transformation / inverse orthogonal transformation of the information source encoding / decoding.

However, the enlarged image is obtained after the decoding process, that is, at the time of reproduction. Therefore, when recording E
The image displayed on the VF 41 is based on the image signal that has not been enlarged. Therefore, as shown in FIG. 13, by providing the display area conversion circuit 10 and the superimposition circuit 40 that superimposes the area to be enlarged, that is, the read area of the memory 35, on the image signal that has not been enlarged, Can monitor replay images.

The enlargement at the time of reproduction is realized, for example, by providing a "0" data addition circuit, 16 × 16 IDCT, etc. in the reproduction system. When expanding at the time of recording, compatible reproduction can be performed by another reproducing apparatus according to the format, and when expanding at the time of reproduction, expansion and reproducing can be performed on a medium recorded by another apparatus according to the format.

(Fifth Embodiment) The fifth embodiment of the present invention, which is an image enlarging device using subband division for recording and reproducing an image signal, will be described below.

FIG. 16 is a block diagram showing the configuration of the subband division coding apparatus according to the fifth embodiment. The sub-band division encoding device shown in FIG. 16 has a switch 61 whose output can be switched and an image signal which is divided into four parts according to its frequency band.
Sub-band division circuit 62 for division, switch 75 whose input can be switched, and encoding circuit for encoding an image signal
It comprises an enlargement area extraction circuit 77 for extracting an enlargement area during enlargement processing, and a standard / enlargement switching circuit 78 for selecting standard processing and enlargement processing. Further, the sub-band division circuit 62 includes a horizontal low pass filter (horizontal LPF) 63.
, Horizontal high pass filter (horizontal HPF) 64, horizontal 2: 1 sub-sampling circuits 65 and 66, vertical low pass filter (vertical LPF) 67 and 69, and vertical high pass filter (vertical HPF) 68. , 70 and vertical 2: 1 sub-sampling circuits 71, 72, 73, 74.

FIG. 17 is a block diagram showing the structure of the subband synthesis decoding apparatus according to the fifth embodiment. Figure 17
The sub-band synthesizing / decoding device shown in (1) is composed of a decoding circuit 79 for decoding encoded data and a sub-band synthesizing circuit 80 for synthesizing four sub-bands. Further, the sub-band synthesis circuit 80 includes a vertical 1: 2 interpolation circuit 81, 82, 83,
84 and vertical low pass filter (vertical LPF) 85, 87
, Vertical high pass filter (vertical HPF) 86, 88, subtractor 89, 90, horizontal 1: 2 interpolation circuit 91, 92, horizontal low pass filter (horizontal LPF) 93, horizontal high pass It is composed of a filter (horizontal HPF) 94 and a subtractor 95.

In this embodiment, sub-band division is made into two divisions in both vertical and horizontal directions, for a total of four divisions. The magnifying power of the image magnifying function is twice the line segment ratio (four times the area ratio). The original image data is NT
4f of SC system color image signal by A / D converter
It will be described as one signal in a digital color signal in which one sample is quantized into 8 bits at a sampling frequency of sc (fsc: color subcarrier frequency), for example, a luminance signal of an image. However, when the original image is a color image, the original image data is composed of three pieces of digital data indicating the magnitudes of the luminance component Y and the color difference components U and V. The circuit is processed for each component. In addition, this embodiment is not limited to either a still image or a moving image, and an encoding circuit performs encoding corresponding to each signal.

Next, the operation will be described. First, Fig. 16
In the sub-band division encoding circuit of, the processing when the "standard processing" is selected by the standard / enlargement switching circuit 78 will be described.

First, the input image signal is input to the horizontal LPF 63 through the switch 61. Image signal is horizontal L
After the band is limited by the PF 63, the number of pixels in the horizontal direction is thinned to ½ by the horizontal 2: 1 sub-sampling circuit 65.

The horizontal LPF 63 used here has a cutoff frequency which is half of the highest frequency of the image signal, and if the original image signal is the NTSC signal sampled at 4 fsc, then the horizontal LPF 63 shown in FIG. It becomes a filter with frequency characteristics such as. Also, other horizontal LPs shown below
The F93 also has exactly the same frequency characteristics as the horizontal LPF63.

The output of the horizontal 2: 1 sub-sampling circuit 65 is passed through the vertical LPF 67, and the number of pixels in the vertical direction is thinned to 1/2 in the vertical 2: 1 sub-sampling circuit 71. The output of the vertical 2: 1 sub-sampling circuit 71 is called the LL band of the image signal, and the number of pixels is 1 / the input signal.
It is 4.

The vertical LPF 67 used here has a cutoff frequency which is half of the highest frequency of the image signal, and if the original image signal is a sampled NTSC signal, the vertical LPF 67 shown in FIG. It becomes a filter with such frequency characteristics. In addition, other vertical LPFs 69, 8 shown below
The 5, 87 also have exactly the same frequency characteristics as the vertical LPF 67.

On the other hand, the horizontal 2: 1 sub-sampling circuit 65
Is also input to the vertical HPF 68, and pixels in the vertical direction are thinned to 1/2 in the vertical 2: 1 sub-sampling circuit 72. The output of the vertical 2: 1 sub-sampling circuit 72 is called the LH band, and the number of pixels is 1/4 of the input signal.
Has become.

The vertical HPF 68 used here has a cutoff frequency of half of the highest frequency of the image signal, and if the original image signal is a sampled NTSC signal, it will be as shown in FIG. 18 (d). It becomes a filter with such frequency characteristics. In addition, other vertical HPF70, 8 shown below
The 6 and 88 also have exactly the same frequency characteristics as the vertical HPF 68.

By the way, the input image signal passed through the switch 61 is also input to the horizontal HPF 64. In the horizontal HPF 64 used here, the cutoff frequency is half the maximum frequency of the image signal, and the original image signal is NTSC.
If the signal is sampled by 4 fsc, FIG.
The filter has a frequency characteristic as shown in (b). Further, the other horizontal HPF 94 shown below also has exactly the same frequency characteristic as the horizontal HPF 64.

The output of this horizontal HPF 64 is such that in the horizontal 2: 1 sub-sampling circuit 66, the number of pixels in the horizontal direction is 1 /
Thinned out to 2.

The output of the horizontal 2: 1 sub-sampling circuit 66 is band-limited by the vertical LPF 69, and then the vertical 2: 1.
In one sub-sampling circuit 73, the number of pixels in the vertical direction is thinned out to 1/2. The output of the vertical 2: 1 sub-sampling circuit 73 is called the HL band, and the number of pixels is 1/4 of the input signal. On the other hand, the output of the horizontal 2: 1 sub-sampling circuit 66 is also input to the vertical HPF 70, and the vertical 2: 1 sub-sampling circuit 74 thins out the number of pixels in the vertical direction to 1/2. The output of the vertical 2: 1 sub-sampling circuit 74 is called the HH band, and the number of pixels is 1/4 of the input signal.

Thus, the input image signal is divided into four bands LL, LH, HL and HH. The appearance of these subbands is shown in FIG. These components are input to a coding circuit 76 through a switch 75, coded by a method suitable for the characteristics of each subband signal by orthogonal transform coding, predictive coding, etc., and further error correction coding, etc. It is applied and becomes encoded data.

Next, the operation of the subband synthesis decoding device shown in FIG. 17 will be described. The encoded data is restored by the decoding circuit 79 into data of four bands LL, LH, HL, and HH. The LL band output from the decoding circuit 79 is input to the vertical LPF 85 after the vertical 1: 2 interpolation circuit 81 doubles the number of pixels in the vertical direction by interpolating 0s between pixels in the vertical direction. It Also, the decoding circuit 79
The output LH band is input to the vertical HPF 86 after the vertical 1: 2 interpolation circuit 82 interpolates 0 between pixels in the vertical direction to double the number of pixels in the vertical direction. The subtracter 89 outputs the vertical HPF 85 from the output of the vertical LPF 85.
Subtract the output of 86. The output of the subtractor 89 doubles the number of pixels in the horizontal direction by interpolating 0 between pixels in the horizontal direction in the horizontal 1: 2 interpolation circuit 91, and then the horizontal LPF.
33 is input. On the other hand, the HL band output from the decoding circuit 79 doubles the number of pixels in the vertical direction by interpolating 0s between pixels in the vertical direction in the vertical 1: 2 interpolation circuit 83, and then the vertical LPF 87. Is entered. Further, the HH band output from the decoding circuit 79 doubles the number of pixels in the vertical direction by interpolating 0s between pixels in the vertical direction in the vertical 1: 2 interpolation circuit 84, and then the vertical HPF.
Entered at 88. The subtractor 90 subtracts the output of the vertical HPF 88 from the output of the vertical LPF 87. The output of the subtractor 90 doubles the number of pixels in the horizontal direction by interpolating 0 between pixels in the horizontal direction in the horizontal 1: 2 interpolation circuit 92, and then,
Input to the horizontal HPF 94. The subtractor 95 is a horizontal LPF93.
The output of the horizontal HPF 94 is subtracted from the output of the above, and the restored image data is output.

The above is the procedure of the "standard processing". In addition,
The interpolating circuit used in the subband synthesizing circuit 80 is necessary for synthesizing the subbands and is not provided for the image enlargement processing. Next, the processing when "enlargement processing" is selected in the standard / enlargement switching circuit 78 in FIG. 16 will be described with reference to FIG.

When performing the "enlargement processing", the input image signal is input to the enlarged area extraction circuit 77 through the switch 61. The enlarged area extraction circuit 77 extracts a signal of an area in which the number of pixels is 1/2 in both horizontal and vertical directions, that is, an area in which the total number of pixels is 1/4, that is, an enlarged area image signal from the image signal. The output of the enlarged area extraction circuit 77 is the switch 75.
Is input to the encoding circuit 76 as a LL band signal. At this time, 0 is input to each of LH, HL, and HH. The encoding circuit 76 performs exactly the same processing as the "standard processing" and outputs encoded data.

In the sub-band synthesis decoding device shown in FIG. 17, even when "enlargement processing" is selected in the standard / enlargement switching circuit 78, exactly the same processing as when "standard processing" is selected is performed. .. As a result, compatible reproduction can be performed by another reproducing device according to the format.

A signal flow when the data of the enlarged image is reproduced will be described. The enlarged area image signal restored as the LL band component is input to the vertical LPF 85 after the vertical 1: 2 interpolation circuit 81 doubles the number of pixels in the vertical direction. The output of the vertical LPF 85 is input to the subtractor 89, but L
Since the H band component is 0, the signal does not change. The output of the subtractor 89 is input to the horizontal LPF 93 after the number of pixels in the horizontal direction is doubled by the horizontal 1: 2 interpolation circuit 91.
The output of the horizontal LPF 93 is input to the subtractor 95,
Since both components of the L and HH bands are 0, the signal does not change. By the above processing, the enlarged area image signal is doubled in both the vertical and horizontal directions. In this way, a part of the image signal is enlarged to form an enlarged image having the same number of pixels. The quality of the enlarged image at this time is determined by the performance of the LPF,
The closer the LPF performance is to the ideal, the better the image quality.

Here, the 0 interpolation circuit and the low pass filter T
The image enlargement processing using O will be described. First, the one-dimensional signal used as the original signal is {x (m): m = 0,1, ..., M-1
}, And its Fourier transform coefficient is {y (k): k = 0,1, ...
, M-1},

[0137]

[Equation 10]

It becomes: The inverse transformation is as follows.

[0139]

[Equation 11]

Here, between each data of the signal x (m)
By interpolating 0 ”, a signal with double the number of data {x '(m): m =
Make 0,1, ..., 2M-1}. x '(m) is as follows. x '(m) = x (m / 2) (m: even number) x' (m) = 0 (m: odd number)

The result of Fourier transform of x '(m) is {y' (k)
If: k = 0,1, ..., 2M-1}, it becomes as follows.

[0142]

[Equation 12]

This is the signal x'where "0" is interpolated in x (m).
The Fourier transform coefficient y '(k) of (m) represents that the Fourier transform coefficient of x (m) is joined together in two. This is shown in FIG. FIG. 20A shows an example of y (k). When y (k) is as shown in Fig. 20 (a), y '
(k) is shown in Fig. 20 (b).

Here, the signal x '(m) obtained by interpolating "0" in the original signal x (m) is passed through an LPF having a cutoff frequency that is half the maximum frequency of the original signal, and the signal {x ''
(m): m = 0,1, ..., 2M-1} is obtained. When M is an even number, the Fourier transform coefficient of x ″ (m) {y ″ (k): k = 0,1, ...,
2M-1} is as follows.

[0145]

[Equation 13]

FIG. 20 (c) shows an example of obtaining a spectrum from the Fourier transform coefficient obtained above. From this, it can be seen that y ″ (k) has the same low frequency component as y (k), and has the value “0” as the high frequency component. In this form, the value of x '' (m) interpolates x (m) by the sinc function,
It shows that it has been magnified twice. That is, it indicates that it is an ideal expansion interpolation that satisfies the sampling theorem. However, since the spectrum size is the same as y '' (0) = y (k), the signal level of x '' (m) becomes x (m) when y '' (k) is inversely transformed. It will be halved compared to. Therefore, in order to make the signal level the same, it is necessary to double the size of the signal when performing "0" interpolation.

When actually realizing an LPF by digital signal processing, it is impossible to pass all components lower than the set cutoff frequency and not pass components higher than the cutoff frequency at all. Therefore, "
It is impossible to perform ideal enlargement by the 0 "interpolation and the LPF. Therefore, the image quality of the enlarged image depends on the performance of the LPF, and the image quality is improved by using a closer image to the ideal LPF. Images can be obtained.

The above description is the same even if the number of "0" s to be interpolated increases. For example, when N "0" s are interpolated between signal data, the Fourier transform coefficient of the "0" interpolated data is a form in which N + 1 Fourier transform coefficients of the original signal are connected. Therefore, if the "0" interpolated data is passed through the LPF having the cutoff frequency of 1 / (N + 1) of the highest frequency of the image, the original signal can be obtained by expanding the data by N + 1 times. I can. At this time, since the signal level is 1 / (N + 1), it is necessary to make the signal N + 1 times larger when performing "0" interpolation.

In the case of performing the 4-division subband coding shown this time, QMF (Quadrature-Mir) is used as LPF and HPF.
ror Filter) is often used. With the device of this embodiment, QM
When image enlargement is performed using F, a clear enlarged image can be obtained as compared with the linear interpolation method used in the conventional image enlargement processing. This is because when the processing by the linear interpolation method is considered as a kind of LPF, the LPF by QMF is closer to an ideal filter.

In this embodiment, the sub-band is divided into four in each of the horizontal and vertical directions, but it is also possible to use the sub-band division only in the vertical direction or only in the horizontal direction. .. At this time, the enlargement is made only in one direction in the vertical direction or the horizontal direction. When the sub-band division in only one direction is used, it is expanded in advance in the direction in which the sub-band is not divided by the 0 interpolation circuit and the LPF, or is expanded by another expansion method typified by linear interpolation or the like, so that ， It is also possible to make the vertical enlargement ratio equal.

The number of divisions in each of the vertical and horizontal directions may be an integer, and the enlarged area image signal may be input to the subband containing the DC component of the image signal. At this time, the expansion rate is the number of subband divisions in both the vertical and horizontal directions.

It is also possible to use hierarchical subband division. For example, the sub-band division of 7 is
The LL band obtained by dividing into four is further divided into four. At this time, the enlarged area image signal cut out from the image signal is input to the subband including the DC component of the image signal. In addition, the enlargement ratio becomes 4 times in both the horizontal and vertical directions.

(Sixth Embodiment) Next, a sixth embodiment of the present invention will be described in which the image enlargement of the electronic image stabilization and the compression / decompression of the image signal are simultaneously performed by using the image enlargement apparatus shown in the fifth embodiment. Will be described. FIG. 21 is a block diagram showing the configuration of an image signal recording / reproducing apparatus having an electronic image stabilization function in the sixth embodiment.

In FIG. 21, 31-37, 39-41 and 48-52
55 is the same as that shown in FIG. 11, and 61, 62 and 75 ~
Since 80 is the same as that shown in FIGS. 16 and 17, description thereof will be omitted. In FIG. 21, the enlarged area extraction circuit 77 in FIG. 16 is composed of the memory 35, the motion vector detection circuit 36, and the memory control circuit 37. Further, the coding circuit 76 in FIG. 16 includes an information source coding circuit 96 and an error correction coding circuit 48 for reducing redundancy by performing orthogonal transform coding, predictive coding, etc. on the output signal of the switch 75. It is composed of and. The decoding circuit 79 in FIG. 17 is composed of an error correction decoding circuit 52 and an information source decoding circuit 97 for restoring compressed information.

Next, the operation will be described. In the image signal recording / reproducing apparatus configured as described above, when “standard processing” is selected in the standard / enlargement switching circuit 78, the output of the encoder 34 is input to the subband division circuit 62 through the switch 61. , Divided into subbands. The signal divided into the sub-bands is input to the information source coding circuit 96 through the switch 75, and the error correction coding circuit is input.
The data is recorded on the recording medium 50 through the recording processing circuit 49. When the recorded signal is processed by the circuit on the playback side,
The image signal is reproduced.

When “enlargement processing” is selected in the standard / enlargement switching circuit 78, the output of the encoder 34 is input to the motion vector detection circuit 36 through the switch 61. When the motion vector detection circuit 36 detects the motion and obtains the motion correction data, the memory control circuit 37 controls the read area of the memory 35 based on the motion correction data to obtain an image signal without shaking. At this time, the size of the read area of the memory 35 is 1/2 in both the horizontal and vertical directions with respect to the area of the input signal.

The motion correction process at this time will be described with reference to FIG. In FIG. 12, the effective area of the output signal of the encoder 34, that is, the writing area of the memory 35 is the vertical direction V0 ×.
It is H0 in the horizontal direction. On the other hand, the area 1 (V1 × H1) or the area 2 (V2 × H2) smaller than the effective area (V0 × H0) of the original signal is read from the memory 35 and enlarged to the original size by the enlargement processing. For example, when the subject moves from the state 1 to the state 2, the movement amount is detected by the motion vector detection circuit 36, the area 1 is read out in the state 1 and the area 2 is read out from the memory 35 in the state 2, Image stabilization can be realized by expanding to the original size.

In the embodiment shown in FIG. 21, the small area 1 or area 2 is half the size of the effective area of the original signal in both the horizontal and vertical directions. That is, it becomes as follows. V1 = V2 = 1 / 2.V0 H1 = H2 = 1 / 2.H0

The image signal with the hand shake corrected thus obtained is input to the information source encoding circuit 96 as the LL component of the sub-band by the switch 75. At this time, L
A value of "0" is input to the information source coding circuit 96 as each component of H, HL, and HH. The following signal processing is exactly the same as in the case of "standard processing". When the signal recorded on the recording medium 50 is processed by the circuit on the reproducing side, the image is doubled in both the horizontal and vertical directions, and an image in which the blurring of the image due to camera shake is suppressed can be obtained.

Further, in the circuit of this embodiment, it is possible to make the enlarged area constant by making the read area of the memory 35 constant. Further, by providing a memory reading range designating means or the like, the photographer can designate the reading area of the memory 35 and enlarge the portion desired by the photographer.

A feature of the fifth and sixth embodiments is that image enlargement processing is performed by a subband division coding circuit and a subband synthesis decoding circuit. Various algorithms such as subband division coding, predictive coding, transform coding, and vector quantization have been proposed and put into practical use as algorithms for coding image information. Further, it is often used as a hybrid coding combining these. Subband coding, in which an image signal is divided into a plurality of bands for coding, is very effective when different processing is performed for each divided band in consideration of human visual characteristics and the like. When is used in a recording / playback apparatus, an image with little visual deterioration can be played back for a long time even with a small storage capacity.

Interpolation circuit used for sub-band synthesis and L
Regarding the image enlargement processing using the PF, it is possible to obtain an enlarged image with little deterioration in image quality, as already described in the fifth embodiment. Further, by performing the image enlarging processing by the sub-band dividing / synthesizing circuit, the image enlarging processing can be realized by slightly increasing the amount of hardware.

However, the enlarged image is obtained after the sub-band synthesis processing, that is, at the time of reproduction. Therefore,
What is displayed on the EVF 41 during recording is an image signal that has not been enlarged. Therefore, as shown in FIG.
Similar to the fourth embodiment, by providing the superimposing circuit 40 that superimposes the area to be enlarged, that is, the read area of the memory 35, on the image signal that has not been enlarged, the photographer can monitor the reproduced image.

As described above, the enlarged area extraction circuit 77,
By passing through the information source coding circuit 96, the error correction coding circuit 48, and the recording processing circuit 49, it is possible to digitally record on the recording medium 50, and the reproduction processing circuit 51, the error correction decoding circuit 52, and the information source decoding circuit. 97, By passing through the sub-band synthesizing circuit 80, it is possible to reproduce an image signal that has been subjected to image stabilization and enlarged processing and has little deterioration.

Further, in the image signal recording / reproducing apparatus having the electronic image stabilization function described above, since recording is performed in accordance with the format, compatible reproduction can be performed by another reproducing apparatus in accordance with the format.

[0166]

As described above, according to the first aspect of the present invention, since the enlargement process is performed by using the orthogonal transformation, the deterioration of the image quality is very little in principle, a natural enlarged image can be obtained, and the hardware scale is increased. Can be made smaller. If a two-dimensional DCT is used as the orthogonal transformation, it is possible to obtain a natural enlarged image with less deterioration. Further, the hardware scale can be further reduced by using the one-dimensional DCT as the orthogonal transformation.

In the second invention, since the enlargement processing is performed by using the orthogonal transformation, the deterioration of the image quality is extremely small in principle,
It is possible to obtain a naturally enlarged image, reduce the hardware scale, and perform fine adaptive processing for each block.

In the third invention, as in the first invention, a natural enlarged image can be obtained, the hardware scale can be reduced, and the signal level of the obtained enlarged image is the same as that of the original image. be able to.

In the fourth invention, as in the first invention, a natural enlarged image can be obtained, the hardware scale can be reduced, and an enlarged image obtained without adjusting the frequency component can be obtained. The signal level can be the same as the original image.

According to the fifth aspect of the invention, the image can be enlarged with a simple structure of "0" interpolating means and low pass filter.

In the sixth invention, in addition to the effects of the above-mentioned first to fourth inventions, the image signal encoding process and the image enlarging process can be performed simultaneously.

In the seventh invention, since the sub-band dividing / synthesizing device fulfills the image enlarging function, the hardware scale can be reduced when the image enlarging function is added to the image signal recording / reproducing device. Further, by making the enlargement ratios in the horizontal and vertical directions equal, it is possible to obtain a natural enlarged image having the same aspect ratio as the original image. Furthermore, even in an image signal recording / reproducing apparatus that only performs subband division in either the horizontal direction or the vertical direction, a natural enlarged image whose aspect ratio does not change from that of the original image can be obtained without significantly increasing the hardware scale. Obtainable.

In the eighth invention, in addition to the effects of the sixth and seventh inventions described above, automatic camera shake correction can be realized with a small hardware scale.

In the ninth invention, in addition to the effects of the sixth invention, the electronic zoom can be realized with a small hardware scale.

In the tenth invention, in addition to the effects of the eighth and ninth inventions described above, the photographer can confirm the enlarged and reproduced area during recording.

[Brief description of drawings]

FIG. 1 is a diagram for explaining processing of a bilinear interpolation method in a conventional example.

FIG. 2 is a diagram showing a main configuration of an interpolation calculation circuit of a bilinear interpolation method in a conventional example.

FIG. 3 is a block diagram showing a configuration of an image enlarging device according to a first embodiment of the present invention.

FIG. 4 is a diagram showing an example of how an enlarged image frequency component is created from an original image frequency component.

FIG. 5 is a block diagram showing a configuration of an image enlarging device for performing enlarging processing at an enlargement ratio of 2 according to the first embodiment of the present invention.

FIG. 6 is a diagram showing an example of how an enlarged image frequency component is created from an original image frequency component when performing enlargement processing at an enlargement ratio of 2.

FIG. 7 is a diagram showing an example of a two-dimensional spectrum of an image used as an original image.

8 is a diagram showing a two-dimensional spectrum of an image obtained by enlarging the original image having the two-dimensional spectrum shown in FIG. 7 by using a bilinear interpolation method which is a conventional image enlarging method.

9 is a diagram showing a two-dimensional spectrum of an image obtained by enlarging the original image having the two-dimensional spectrum shown in FIG. 7 using the image enlarging device shown in the embodiment of the present invention.

FIG. 10 is a block diagram showing the configuration of an image enlarging device that does not require adjustment of frequency components according to the second embodiment of the present invention.

FIG. 11 is a block diagram showing a configuration of an image signal recording / reproducing apparatus having an electronic image stabilization function according to a third embodiment of the present invention.

FIG. 12 is a diagram for explaining a motion correction process.

FIG. 13 is a block diagram showing a configuration of an image signal recording / reproducing apparatus having an electronic zoom function according to a fourth example of the present invention.

FIG. 14 is a diagram for explaining the operation of the electronic zoom.

FIG. 15 is a diagram for explaining the configuration and operation of the superimposing circuit in the third, fourth, and sixth embodiments.

FIG. 16 is a block diagram showing a configuration of a subband division encoding device in an image enlarging device according to a fifth embodiment of the present invention.

FIG. 17 is a block diagram showing a configuration of a subband synthesis decoding device in the image enlarging device of the fifth embodiment.

FIG. 18 is a diagram showing frequency characteristics of a low-pass filter and a high-pass filter used in the subband division circuit and the subband synthesis circuit in the fifth embodiment.

FIG. 19 is a diagram showing a state of subband division according to the fifth embodiment.

FIG. 20 is a diagram for explaining an image enlarging operation according to the fifth embodiment.

FIG. 21 is a block diagram showing a configuration of an image signal recording / reproducing apparatus having an automatic image stabilization function according to a sixth example of the present invention.

[Explanation of symbols]

 1 block division circuit 2,22 M × M point DCT circuit 3 “0” data addition circuit 4 multiplier 5,25 N × N point IDCT circuit 6 image reconstruction circuit 7 image cutout circuit 31 image sensor 32 D / A converter 33 Process circuit 34 Encoder 35 Memory 36 Motion vector detection circuit 37 Memory control circuit 38, 78 Standard / enlargement switching circuit 40 Superposition circuit 41 EVF 61, 75 switch 62 Sub-band division circuit 63, 93 Horizontal LPF 64, 94 Horizontal HPF 67, 69, 85, 87 Vertical LPF 68, 70, 86, 88 Vertical HPF 76 Encoding circuit 77 Enlarged area extraction circuit 79 Decoding circuit 80 Subband synthesis circuit

─────────────────────────────────────────────────── ───

[Procedure amendment]

[Submission date] May 7, 1992

[Procedure Amendment 1]

[Document name to be amended] Statement

[Correction target item name] 0012

[Correction method] Change

[Correction content]

[0012]

The image enlarging device according to the first invention of the present application is an orthogonal transform means for performing orthogonal transform on data of an original image to obtain a frequency component, and to the frequency component.
"0" data adding means for adding data of "0" to obtain an enlarged image frequency component having a larger data size than the frequency component of the original image, and an inverse orthogonal transform corresponding to the size of the enlarged image frequency component are executed. And an inverse orthogonal transformation means.

[Procedure Amendment 2]

[Document name to be amended] Statement

[Name of item to be corrected] 0019

[Correction method] Change

[Correction content]

The image signal recording / reproducing apparatus according to the sixth invention of the present application uses the orthogonal transforming means and the inverse orthogonal transforming means for compressing / decompressing the image signal at the time of recording / reproducing to obtain the first, second, third, and third means.
4 is a feature of realizing the image enlarging device shown in the invention. According to the sixth aspect of the invention, as the orthogonal transforming means used when compressing the image signal, at least one orthogonal transforming means for processing smaller size data is provided in addition to the orthogonal transforming means for processing normal size data. .. Ah
Or, as the inverse orthogonal transforming means used when the image signal is restored, at least one inverse orthogonal transforming means for processing larger size data is provided in addition to the inverse orthogonal transforming means for processing normal size data. ..

[Procedure 3]

[Document name to be amended] Statement

[Correction target item name] 0020

[Correction method] Change

[Correction content]

An image signal recording / reproducing apparatus according to the seventh invention of the present application includes a cutting-out means for cutting out a part of an image signal, a first state for inputting each sub-band component to all sub-band channels, and a DC component. Switching means for switching between two states, a second state in which the output of the clipping means is input to the subband channel for the subband component and "0" data is input to the other subband channel. ..

[Procedure amendment 4]

[Document name to be amended] Statement

[Name of item to be corrected] 0030

[Correction method] Change

[Correction content]

In the seventh invention, the image is enlarged in the sub-band dividing / combining apparatus. In the seventh invention, if the low pass filter, the high pass filter, the sub-sampling means, and the "0" data interpolating means are provided for both vertical and horizontal directions, the sub-band dividing / synthesizing device can make the image horizontal, Expand in each direction vertically. Also, subsample
Of the horizontal and vertical directions of the encoding means and the "0" data interpolation means.
Keeping the same operation in the direction, the ratio of the cutoff frequency of the vertical low-pass filter and the high-pass filter to the highest frequency in the vertical direction that the image signal has,
When the ratio of the cut-off frequency of the horizontal for the low-pass filter and a high pass filter with respect to the horizontal direction of the highest frequency image signal has the keep to the same, horizontal,
The magnification in the vertical direction is the same. Further, even in an image signal recording / reproducing apparatus that performs subband division only in the horizontal direction or the vertical direction, image enlargement processing is performed in the direction in which subband division is not performed before subband division, and then the subband division is performed. When the image is enlarged in the remaining direction by the band dividing / combining circuit, an enlarged image having the same enlargement ratio in the horizontal direction and the vertical direction can be obtained.

[Procedure Amendment 5]

[Document name to be amended] Statement

[Name of item to be corrected] 0136

[Correction method] Change

[Correction content]

Here, the 0 interpolation circuit and the low pass filter
The image enlarging process using is described. First, the one-dimensional signal used as the original signal is {x (m): m = 0,1, ..., M-1}
And the Fourier transform coefficient is {y (k): k = 0,1, ..., M-
1}

Claims (10)

[Claims] 1. An image enlarging device for enlarging an input digital image signal of an original image to obtain enlarged image data, and an orthogonal transform means for subjecting the image signal to orthogonal transform to create an original image frequency component. , "0" data is added to the created original image frequency component to create an enlarged image frequency component larger in size than the original image frequency component "
An image enlarging device comprising: 0 "data adding means; and inverse orthogonal transforming means for performing inverse orthogonal transformation corresponding to the size of the created enlarged image frequency component on the enlarged image frequency component to obtain enlarged image data. .. 2. An image enlarging device for enlarging an input digital image signal of an original image to obtain enlarged image data, wherein the image signal is divided into blocks for a plurality of pixels to form an original image block. Means, an orthogonal transform means for performing an orthogonal transform on the constructed original image block to create an original image frequency component block, and the original image frequency component block by adding "0" data to the created original image frequency component block "0" data adding means for creating a larger size image frequency component block and inverse orthogonal transform corresponding to the size of the created size image frequency component block are applied to the size image frequency component block to obtain the size image block data. An inverse orthogonal transforming means for obtaining the enlarged image data, and a reconstructing means for reconstructing the obtained enlarged image block data to obtain the enlarged image data. An image enlarging device, comprising: 3. An image enlarging device for enlarging an input digital image signal of an original image to obtain enlarged image data, and an orthogonal transform means for subjecting the image signal to orthogonal transform to create an original image frequency component. , "0" data is added to the created original image frequency component to create an enlarged image frequency component larger in size than the original image frequency component "
0 "data adding means, multiplication means for multiplying the created enlarged image frequency component by a coefficient corresponding to the enlargement ratio, and inverse orthogonal transform corresponding to the size of the multiplied enlarged image frequency component to the enlarged image frequency component. An image enlarging device, comprising: an inverse orthogonal transformation unit that obtains enlarged image data. 4. An image enlarging apparatus for enlarging a digital image signal of an input original image to obtain enlarged image data, wherein the image signal is orthogonal using an equation having a reciprocal of the size of the image signal as a coefficient. Orthogonal transform means for performing conversion to create an original image frequency component, and adding "0" data to the created original image frequency component to create an enlarged image frequency component having a size larger than the original image frequency component "
0 "data adding means and an inverse orthogonal transform using an equation corresponding to the size of the created enlarged image frequency component and not having the reciprocal of the size of the enlarged image frequency component as a coefficient. An image enlarging device, comprising: an inverse orthogonal transformation unit for obtaining data. 5. An image enlarging apparatus for enlarging an input digital image image signal of an original image to obtain enlarged image data, wherein N (N: integer) number of pixels are included between pixels of the image signal.
0 ”Interpolate the data to obtain (N + 1) times the number of pixels”
An image enlarging apparatus comprising: a 0 "interpolation means and a low-pass filter having a cutoff frequency of 1 / (N + 1) times the highest frequency of the image signal. 6. An image signal recording / reproducing apparatus for orthogonally transforming a digital image signal and then encoding the same to obtain encoded data, performing inverse orthogonal transform on the encoded data, and then decoding the encoded signal. And the image enlarging device according to claim 1 or claim 2 or claim 3 or claim 4 using the orthogonal transform and the inverse orthogonal transform in the decoding. Image signal recording / reproducing apparatus. 7. A subband dividing / combining apparatus for subdividing a digital image signal, transmitting the obtained plurality of subband components to a plurality of subband channels separately, and synthesizing the transmitted subband components. In an image signal recording / reproducing apparatus having: a cutting-out means for cutting out a part of the digital image signal, a first state in which a subband component is separately input to the subband channel, and a subband component including a DC component is transmitted. An image signal recording / reproducing apparatus comprising switching means for switching between a second state in which an output of the cutout means is input to a subband channel and "0" data is input to another subband channel. 8. A conversion means for converting the obtained image into a digital image signal, a detection means for detecting a blur amount of the obtained image based on the converted image signal, and the converted image signal stored therein. And a memory control unit that controls a read area from the memory based on a detection result of the detection unit.
Alternatively, the image signal recording / reproducing apparatus according to claim 7. 9. A conversion means for converting the obtained image into a digital image signal, a memory for storing the converted image signal, a plurality of image enlarging means having different enlargement ratios, and the memory based on the enlargement ratios. 7. An image signal recording / reproducing apparatus according to claim 6, further comprising a control unit for controlling selection of a read area from the image reading unit and the image enlarging unit. 10. The image signal recording / reproducing apparatus according to claim 8, further comprising display means for superimposing and displaying the area read from the memory on the image signal before being enlarged. ..