JP2009094593A - Image signal processor, image signal processing method, and image display device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To preferably make an input image signal high resolved. <P>SOLUTION: An image signal processor has an input section to which a plurality of image frames are inputted, and a resolution conversion section for increasing the number of pixels for composing the image frames to obtain an output image frame by compositing a plurality of inputted image frames. The resolution conversion section uses image data on a reference input image frame and each corresponding image data on other input image frames to estimate a sampling phase difference, converts the sampling phase difference, outputs the converted sampling phase difference, uses information on the converted sampling phase difference to compensate the motion of the image data of each input image frame and increase the number of pixels, shifts the phase of the image data of each image frame of which the number of pixels is increased by prescribed quantity, and multiplies each image data before and after phase shift by a coefficient determined by the information of the sampling phase difference for addition, thus removing folding components for output thereof. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a technique for increasing the resolution of an image signal, and in particular, by combining a plurality of image frames, the number of pixels constituting the image frame is increased and unnecessary aliasing components are removed to increase the resolution. Regarding technology.

  Recent television receivers have become larger in screen size and do not display image signals input from broadcasting, communication, storage media, etc., as they are, but display them by increasing the number of pixels in the horizontal and vertical directions by digital signal processing. It is generally done. At this time, the resolution cannot be increased only by increasing the number of pixels by an interpolation low-pass filter using a generally known sinc function, a spline function, or the like.

  Therefore, as described in Patent Document 1, Patent Document 2, and Non-Patent Document 1, a plurality of input image frames (hereinafter abbreviated as “frames”) are combined into a single frame. A technique for increasing the number of pixels while increasing the resolution (hereinafter referred to as a conventional technique) has been proposed.

JP-A-8-336046 JP-A-9-69755 Shin Aoki “Super-resolution processing using multiple digital image data”, Ricoh Technical Report pp.19-25, No.24, NOVEMBER, 1998

  In these conventional techniques, high resolution is achieved by three processes: (1) position estimation, (2) wideband interpolation, and (3) weighted sum. Here, (1) position estimation is to estimate the difference in sampling phase (sampling position) of each image data using each image data of a plurality of input image frames. (2) Wideband interpolation increases the image data density by interpolating and increasing the number of pixels (sampling points) using a wide-band low-pass filter that transmits all high-frequency components of the original signal, including aliasing components. It is to become. (3) The weighted sum is a weighted sum corresponding to the sampling phase of each densified data, canceling out aliasing components generated during pixel sampling and simultaneously removing the high-frequency components of the original signal. Is to restore.

  FIG. 2 shows an outline of these high resolution techniques. As shown in (a) in the figure, frame # 1 (201), frame # 2 (202), and frame # 3 (203) on different time axes are input, and these are combined to form an output frame (206). Assume that you get. For simplicity, first consider the case where the subject has moved (204) in the horizontal direction, and consider increasing the resolution by one-dimensional signal processing on the horizontal line (205). At this time, as shown in (b) and (d) in the figure, the position of the signal waveform is shifted in accordance with the amount of movement (204) of the subject in frame # 2 (202) and frame # 1 (201). Occurs. The amount of displacement is obtained by the above (1) position estimation, and as shown in FIG. 5C, the frame # 2 (202) is motion-compensated (207) so that the displacement is eliminated, and the pixels ( The phase difference θ (211) between the sampling phases (209) and (210) of 208) is obtained. Based on this phase difference θ (211), by performing the above (2) wideband interpolation and (3) weighted sum, as shown in FIG. High resolution is realized by generating a new pixel (212) at the position of = π). (3) The weighted sum will be described later.

  Actually, the movement of the subject may be accompanied by movements such as rotation and enlargement / reduction as well as parallel movement, but if the time interval between frames is very small or the movement of the subject is slow, These movements can also be considered by approximating local translation.

  As shown in FIG. 3, when performing the weighted sum of (3) above when the resolution is doubled in the one-dimensional direction by the conventional techniques described in Patent Document 1, Patent Document 2, and Non-Patent Document 1. It is necessary to use signals of at least three frame images. Here, FIG. 3 is a diagram showing a frequency spectrum of each component in a one-dimensional frequency region. In the figure, the distance from the frequency axis represents the signal intensity, and the rotation angle around the frequency axis represents the phase. The weighted sum of (3) above will be described in detail below.

  When the pixel interpolation is performed by the broadband low-pass filter that transmits twice the Nyquist frequency band (frequency band 0 to sampling frequency fs) in the broadband interpolation of (2) above, the same component as the original signal (hereinafter referred to as the original component) And the sum of the aliasing components according to the sampling phase is obtained. At this time, when the above-described (2) wideband interpolation processing is performed on the signals of the three frame images, as shown in FIG. 3 (a), the original components (301), (302), (303) of each frame are displayed. It is well known that the phases are all in agreement, and the phase of the aliasing components (304), (305) and (306) rotates in accordance with the difference in sampling phase of each frame. In order to facilitate understanding of the respective phase relationships, the phase relationship of the original components of each frame is shown in FIG. 5B, and the phase relationship of the folded components of each frame is shown in FIG.

  Here, with respect to the signals of the three frame images, by appropriately selecting the coefficients to be multiplied and performing the above (3) weighted sum, the folded components (304), (305), and (306) of each frame are mutually connected. It can be canceled out and only the original components can be extracted. At this time, the vector sum of the folded components (304) (305) (306) of each frame is set to 0, that is, both the Re axis (real axis) component and the Im axis (imaginary axis) component are set to 0. For this purpose, at least three folding components are required. Therefore, it is necessary to use signals of at least three frame images in order to realize double resolution enhancement, that is, to remove one aliasing component.

  Similarly, as described in Patent Document 1, Patent Document 2, and Non-Patent Document 1, when the resolution is increased with respect to a horizontal / vertical two-dimensional input signal, the aliasing comes from the vertical and horizontal directions. When the band of the original signal doubled both vertically and horizontally, the three folded components overlap, and 2M + 1 = 7 digital data (= 7 frame image signals) were required to cancel them.

  Therefore, the prior art is not economical because the scale of the frame memory and the signal processing circuit is increased. In addition, since it is necessary to accurately estimate the positions of many frame images that are separated in time, the configuration becomes complicated. That is, it is difficult for the conventional technology to increase the resolution of a moving image frame such as a television broadcast signal.

  In addition, interlace scanning is mainly used in current television broadcast signals. However, Patent Document 1, Patent Document 2, and Non-Patent Document 1 disclose that interlace scanning signals themselves have higher resolution, interlace scanning, and so on. There is no disclosure or suggestion about progressive scan conversion (IP conversion).

  In addition, in the current digital television broadcasting using terrestrial and satellite (BS, CS), programs are broadcast with HD (High Definition) image signals in addition to conventional SD (Standard Definition) image signals. . However, not all programs have been replaced with image signals shot with an HD camera.Image signals shot with an SD camera are converted into signals with the same number of pixels as HD using an SD-to-HD converter. It is well known that it is broadcast by switching between programs or scenes.

  With a conventional receiver, if the received signal is an image signal taken with an HD camera, a high-resolution image is played back, and if it is an SD-HD converted (up-con) image signal, a low-resolution image is played back. Therefore, there is a problem that the resolution is frequently switched for each program or for each scene, which is unsightly.

  Further, in the above conventional technique, since the resolution enhancement processing is performed using the difference in the sampling phase (sampling position), a signal that does not cause a difference in the sampling phase, that is, a region where the subject is stationary, a subject There is a problem that the effect of increasing the resolution cannot be obtained in a region where the movement of the image becomes an integer multiple of the pixel interval.

The present invention has been made in view of the above-described problems of the prior art, and an object thereof is to provide a technique for suitably increasing the resolution of an image signal.

  In order to achieve the above object, an embodiment of the present invention may be configured as described in the claims, for example.

  According to the present invention, it is possible to increase the resolution of an image signal more suitably.

  Embodiments of the present invention will be described below with reference to the drawings.

Moreover, in each drawing, the component to which the same code | symbol was attached | subjected shall have the same function.

  Further, the expression “phase” in each description and each drawing of this specification includes the meaning of “position” on a two-dimensional image when used in a two-dimensional image space. The said position means the position of decimal pixel precision.

  In addition, the expression “up rate” in each description and each drawing of this specification also includes the meaning of “up rate processing”. Further, the expression “upcon” in each description and each drawing of this specification indicates “upconversion processing”. Both of them mean conversion processing for increasing the number of pixels of an image (pixel number increase processing) or conversion processing for enlarging an image (image enlargement conversion processing).

  In addition, the expression “down rate” in each description and each drawing of this specification also includes the meaning of “down rate processing”. In addition, the expression “downconversion” in each description and each drawing of the present specification indicates “downconversion processing”. Both of them mean conversion processing for reducing the number of pixels in the image (pixel number reduction processing) or conversion processing for reducing the image (image reduction conversion processing).

  Further, the expression “motion compensation” in each description and each drawing of the present specification includes a meaning of performing alignment by calculating a phase difference or a sampling phase difference, that is, a spatial position difference.

In the description of each embodiment below, the method described in Reference Document 1 or Reference Document 2 may be used for the above-described (1) position estimation. As for the above-mentioned (2) wideband interpolation, a general low-pass filter having a pass band twice as high as the Nyquist frequency as described in Non-Patent Document 1 may be used.
[Reference 1] Shigeru Ando “Velocity vector distribution measurement system using spatio-temporal differential calculation of images”, Transactions of the Society of Instrument and Control Engineers, pp. 1330-1336, Vol. 22, No. 12, 1986
[Reference 2] Hiroyuki Kobayashi et al. “Phase-Only Correlation of Images Based on DCT Transform”, IEICE Technical Report ITS2005-92, IE2005-299 (2006-02), pp.73-78
In the following embodiments, the notation “SR signal” is an abbreviation of “Super Resolution signal (super-resolution signal)”.

  Embodiments of the present invention will be described below with reference to the drawings.

  FIG. 1 shows an image signal processing apparatus according to Embodiment 1 of the present invention, and features thereof will be described. The image signal processing apparatus according to this embodiment is applied to an image display apparatus such as a television receiver. In the following description of the present embodiment, an image display device will be described as an example of the image signal processing device.

  In FIG. 1, an image signal processing apparatus according to the present embodiment includes an input unit (1) to which a frame sequence of a moving image such as a television broadcast signal is input, and a frame input from the input unit (1). A resolution conversion unit (2) for increasing the resolution and a display unit (3) for displaying an image based on the frame whose resolution is increased by the resolution conversion unit (2) are further provided. As the display unit (3), for example, a plasma display panel, a liquid crystal display panel, or an electron / electrolytic emission display panel is used. Details of the resolution conversion unit (2) will be described below.

  In FIG. 1, first, the position estimation unit (101) performs a corresponding operation on the frame # 2 based on the sampling phase (sampling position) of the pixel to be processed on the frame # 1 input to the input unit (1). The pixel position is estimated, and the sampling phase difference θ (102) is obtained for each pixel.

  Next, frame # 2 is motion-compensated using the information of phase difference θ (102) by the up-raters (103) and (104) of the motion compensation / up-rate unit (115) and aligned with frame # 1. At the same time, the number of pixels in frame # 1 and frame # 2 is doubled to increase the density. The phase shift unit (116) shifts the phase of the densified data by a certain amount. Here, π / 2 phase shifters (106) and (108) can be used as means for shifting the phase of data by a certain amount. Further, in order to compensate for the delay caused by the π / 2 phase shifters (106) and (108), the signals of the frame # 1 and the frame # 2 which have been densified by the delay devices (105) and (107) are delayed.

  In the aliasing component removal unit (117), the phase difference θ (102) is calculated by the coefficient determiner (109) for each output signal of the delay units (105) (107) and the Hilbert transformers (106) (108). The generated coefficients C0, C2, C1, and C3 are respectively multiplied by multipliers (110), (112), (111), and (113), and these signals are added by an adder (114) and output. obtain. This output is supplied to the display unit 3. The position estimation unit (101) can be realized using the above-described conventional technique as it is. Details of the up-raters (103) and (104), the π / 2 phase shifters (106) and (108), and the aliasing component removal unit (117) will be described later.

  FIG. 4 shows the operation of the first embodiment of the present invention. This figure shows the outputs of the delay units (105) and (107) and π / 2 phase shifters (106) and (108) shown in FIG. 1 in a one-dimensional frequency domain. In FIG. 8A, the signals of frame # 1 and frame # 2 after the up-rate output from the delay units (105) and (107) are respectively the original components (401) and (402) and the original sampling frequency ( This is a signal obtained by adding folded components (405) and (406) folded from fs). At this time, the aliasing component (406) is rotated in phase by the above-described phase difference θ (102).

  On the other hand, the frame # 1 and frame # 2 signals output from the π / 2 phase shifters (106) and (108) are the original components (403) and (404) after the π / 2 phase shift, respectively. , A signal obtained by adding the folded components (407) and (408) after the π / 2 phase shift. (B) and (c) show the original component and the aliasing component extracted for easy understanding of the phase relationship between the components shown in (a).

  Here, when the vector sum of the four components shown in FIG. 4B is taken, the Re-axis component is set to 1, the Im-axis component is set to 0, and the four components shown in FIG. When the vector sum is taken, the coefficients to be multiplied by each component are determined so that both the Re-axis and Im-axis components are set to 0, and the weighted sum is taken. Only the components can be extracted. In other words, it is possible to realize an image signal processing apparatus that uses only two frame images to increase the resolution twice in the one-dimensional direction. Details of this coefficient determination method will be described later.

  FIG. 5 shows the operation of the up-raters (103) (104) used in the first embodiment of the present invention. In this figure, the horizontal axis represents frequency, the vertical axis represents gain (value of the ratio of output signal amplitude to input signal amplitude), and shows the “frequency-gain” characteristics of the up-raters (103) and (104). . Here, in the up-rater (103) (104), the new sampling frequency is set to a frequency (2fs) that is twice the sampling frequency (fs) of the original signal, and the position is exactly in the middle of the original pixel interval. Insert a new pixel sanding point (= zero point) into the filter to double the number of pixels and increase the density. Call. At this time, as shown in the figure, due to the symmetry of the digital signal, the characteristic repeats every frequency that is an integral multiple of 2fs.

  FIG. 6 shows a specific example of the up-raters (103) (104) used in the first embodiment of the present invention. This figure shows the tap coefficients of the filter obtained by inverse Fourier transform of the frequency characteristics shown in FIG. At this time, each tap coefficient Ck (where k is an integer) is a generally known sinc function, and is shifted by (−θ) to compensate for the sampling phase difference θ (102) for each pixel, Ck = 2sin (πk + θ) / (πk + θ) may be set. In the up-rate device (103), the phase difference θ (102) is set to 0 and Ck = 2sin (πk) / (πk). In addition, by expressing the phase difference θ (102) by the phase difference in integer pixel units (2π) + the phase difference in decimal pixel units, the phase difference compensation in integer pixel units is realized by a simple pixel shift, and decimal For compensation of the phase difference in units of pixels, the filters of the up-raters (103) and (104) may be used.

  FIG. 7 shows an operation example of the π / 2 phase shifters (106) and (108) used in the first embodiment of the present invention. A generally known Hilbert transformer can be used as the π / 2 phase shifters (106) and (108).

  In FIG. 5A, the horizontal axis represents frequency, and the vertical axis represents gain (value of the ratio of output signal amplitude to input signal amplitude), indicating the “frequency-gain” characteristics of the Hilbert transformer. Here, in the Hilbert transformer, the frequency (2fs) that is twice the sampling frequency (fs) of the original signal is used as a new sampling frequency, and all frequency components except -0 between -fs and + fs are gained. A pass band of 1.0.

  In FIG. 2B, the horizontal axis represents frequency, and the vertical axis represents phase difference (difference in output signal phase with respect to input signal phase), indicating the “frequency-phase difference” characteristic of the Hilbert transformer. Here, the phase of the frequency component between 0 and fs is delayed by π / 2, and the phase of the frequency component between 0 and −fs is advanced by π / 2. At this time, as shown in the figure, due to the symmetry of the digital signal, the characteristic repeats every frequency that is an integral multiple of 2fs.

  FIG. 8 shows an example in which the π / 2 phase shifters (106) and (108) used in the first embodiment of the present invention are configured by Hilbert transformers. This figure shows the filter tap coefficients obtained by inverse Fourier transform of the frequency characteristics shown in FIG. At this time, each tap coefficient Ck may be Ck = 0 when k = 2m (where m is an integer), and Ck = −2 / (πk) when k = 2m + 1.

  The π / 2 phase shifters (106) and (108) used in Embodiment 1 of the present invention can also use differentiators. In this case, if the general expression cos (ωt + α) representing a sine wave is differentiated by t and multiplied by 1 / ω, d (cos (ωt + α)) / dt * (1 / ω) =-sin (ωt + α) = cos (ωt + α + π / 2), and the function of π / 2 phase shift can be realized. In other words, after taking the difference between the value of the target pixel and the value of the adjacent pixel, a π / 2 phase shift function is realized by applying a filter with a frequency / amplitude characteristic of 1 / ω. May be.

  FIG. 9 shows an operation and a specific example of the coefficient determiner (109) used in the first embodiment of the present invention. As shown in FIG. 4A, when the vector sum of the four components shown in FIG. 4B is taken, the Re-axis component is set to 1, the Im-axis component is set to 0, and FIG. If the coefficients to be multiplied by each component are determined so that both the Re-axis and Im-axis components are 0 when the vector sum of the four components shown in (c) is taken, two frame images Can be used to realize an image signal processing apparatus that achieves a resolution twice as high as that in the one-dimensional direction.

  Here, as shown in FIG. 1, the coefficient for the output of the delay unit (105) (the sum of the original component and the folded component of the frame # 1 after the up-rate) is C0, and the output of the π / 2 phase shifter (106) The coefficient for (the sum of the π / 2 phase shift results of the original component and the folded component of frame # 1 after the up-rate) is C1, and the output of the delay unit (107) (the original component of frame # 2 after the up- The coefficient for the sum of the aliasing components) is C2, and the factor for the output of the Hilbert transformer (106) (the sum of the π / 2 phase shift results of the original and aliasing components of frame # 2 after the update) is C3. .

  At this time, if the conditions of FIG. 9 (a) are satisfied, the simultaneous equations shown in FIG. 9 (b) can be obtained from the phase relationships of the components shown in FIGS. 4 (b) and 4 (c). If this is solved, the result shown in FIG. 9 (c) can be derived.

  The coefficient determiner (109) according to the present embodiment outputs coefficients C0, C1, C2, and C3 that satisfy any one of FIGS. 9 (a), 9 (b), and 9 (c).

  As an example, FIG. 9 (d) shows values of the coefficients C0, C1, C2, and C3 when the phase difference θ (102) is changed from 0 to 2π every π / 8. This corresponds to a case where the position of the signal of the original frame # 2 is estimated with an accuracy of 1/16 pixel and motion compensation is performed on the frame # 1. When the value of phase difference θ (102) is less than 0 or 2π or more, use the periodicity of sin function or cos function and add the value of integer multiple of 2π to the value of phase difference θ (102) Alternatively, subtraction may be performed so that the phase difference θ (102) falls within the range of 0 to 2π.

  The up-raters (103) (104) and π / 2 phase shifters (106) (107) require an infinite number of taps to obtain ideal characteristics, but the number of taps is finite. There is no practical problem even if it is cut off and simplified. At this time, a general window function (such as a Hanning window function or a Hamming window function) may be used. If the coefficient of each tap of the simplified Hilbert transformer is the value of the left and right points centered on C0, that is, C (-k) = -Ck (k is an integer), the phase can be shifted by a certain amount. it can.

  Next, a difference in operation between the image signal processing apparatus according to the first embodiment and the above-described conventional technique will be described with reference to FIG. In FIG. 9A, an input image is prepared so that the subject moves in the right direction between frame # 1 (1701) and frame # 5 (1705). At this time, as shown in FIG. 4B, when the sampling phase in each frame is viewed, the corresponding pixel position is 1/4 pixel (= π) between frame # 1 (1701) and frame # 2 (1702). / 2) The position of the corresponding pixel is shifted by 1 pixel (= 2π) between frame # 1 (1701) and frame # 3 (1703), and frame # 1 (1701) and frame # 4 (1704 ), The corresponding pixel position is shifted by 5/4 pixels (= 5π / 2), and the corresponding pixel position is 2 pixels (= 4π) between frame # 1 (1701) and frame # 5 (1705). The subject is intentionally moved so that it is displaced. At this time, the phase of each folded component included in the signal on each frame is expressed as shown in FIG. 7C with reference to the phase of the folded component included in the signal on frame # 1 (1701). Can do. When the resolution of the input image (a) is increased by a factor of 2, in the above-described conventional technique, the aliasing component of any of the three frames from frame # 1 (1701) to frame # 5 (1705) is used. Since the vector sum cannot be reduced to 0, high resolution cannot be realized. On the other hand, if this embodiment is used, for example, the vector sum of the aliasing component can be set to 0 using two adjacent frames (eg, frame # 1 (1701) and frame # 2 (1702)). realizable. That is, the operation status of the present embodiment can be confirmed by using the input image of FIG.

  In the above description of the first embodiment, the description has been given by taking the high resolution in the horizontal direction as an example. However, the embodiments of the present invention are not limited to this, and the vertical and oblique height increases. It can be applied to resolution.

  According to the image signal processing apparatus according to the first embodiment described above, phase shift is performed on each image signal of two input image frames, which is smaller than in the conventional example, and two signals are generated from each image signal. . Thereby, four signals can be generated from the image signals of the two input image frames. Here, based on the phase difference between the two input image frames, for each of the four signals, a coefficient for canceling and combining the aliasing components of the four signals is calculated for each pixel. For each pixel of the image to be generated, a sum is calculated by multiplying the pixel value of the corresponding pixel included in each of the four signals by each coefficient to generate a pixel value of a new high-resolution image. By performing this for each pixel of the generated image, a new high-resolution image can be generated.

  Thereby, the image signal processing apparatus according to the first embodiment can generate an image with less aliasing components and higher resolution than the input image by using two input image frames that are smaller than those in the conventional example.

  In addition, since the image signal processing apparatus according to the first embodiment uses two input image frames that are fewer than the conventional example, the amount of necessary image processing can be reduced as compared with the conventional example. As a result, an image signal processing apparatus that generates an image with less aliasing components and higher resolution than the input image can be realized at a lower cost than the conventional example.

  Next, Embodiment 2 of the present invention will be described with reference to FIGS.

  The second embodiment relates to an image signal processing method in which processing equivalent to the image signal processing in the image signal processing apparatus according to the first embodiment is realized by a control unit that cooperates with software.

  First, an image processing apparatus for realizing the image signal processing method according to the present embodiment will be described with reference to FIG. The image signal processing apparatus shown in FIG. 18 stores, for example, an input unit (1) to which an image signal such as a television broadcast signal is input, and software for processing the signal input from the input unit (1). A control unit (10) that performs image signal processing on a signal input from the input unit (1) in cooperation with software stored in the unit (11), the storage unit (11), and a control unit (10) Frame buffer # 1 (21) used for data buffer in the image signal processing, frame buffer # 2 (22), and the signal after image signal processing output from the control unit (10) to the output unit (3), And buffer # 3 (23) for frame buffering.

  Here, the number of input units (1) included in the image signal processing apparatus shown in FIG. 18 is the same as two, which is the number of frames used for image processing, but only one input unit (1) is provided. One frame may be input continuously.

  In addition, the frame buffer # 1 (21), the frame buffer # 2 (22), and the storage unit (11) for storing software used for the data buffer may each be configured using individual memory chips. Alternatively, one or a plurality of memory chips may be used and each data address may be divided and used.

  In this embodiment, for the image signal input from the input unit (1), the control unit (10) performs image signal processing in cooperation with software stored in the storage unit (11), and the display unit (3) Output to. Details of the image signal processing will be described with reference to FIG.

  FIG. 14 shows an example of a flowchart of the image signal processing method according to the present embodiment. The flowchart of FIG. 14 starts from step (1401), and in step (1418), the image data of each frame is doubled. That is, in step (1402), the image data of frame # 1 is updated and written to frame buffer # 1, and in step (1403), the image data of frame # 2 is updated and written to frame buffer # 2. Here, the up-rate can be realized by clearing the value of each frame buffer to 0 and then writing the data every other pixel.

  Next, in step (1404), the first pixel (for example, the upper left pixel) of frame buffer # 1 is set as a processing target, and processing is performed until all pixel data processing for frame buffer # 1 is completed. Loop.

  In step (1405), the position of the corresponding pixel in the frame buffer # 2 is estimated with reference to the target pixel in the frame buffer # 1, and the phase difference θ is output. At this time, the above-described conventional technique can be used as it is as a method of estimating the position of the corresponding pixel.

  In step (1406), based on the phase difference θ obtained in step (1405), motion compensation is performed on pixels near the corresponding pixel in frame buffer # 2. At this time, only the pixel data used in the π / 2 phase shift processing in step (1408), that is, pixel data in a range where a finite number of taps acts, need to be motion-compensated as neighboring pixels. The motion compensation operation is the same as the operation described with reference to FIGS.

  Subsequently, in step (1419), the phase is shifted by a certain amount with respect to the frame buffer # 1 and the motion compensated frame buffer # 2. That is, in steps (1407) and (1408), the pixel data in each frame buffer is phase-shifted by π / 2.

  Subsequently, in step (1420), the coefficients C0, C1, C2, and C3 set so as to satisfy the conditions of FIGS. 9A, 9B, and 9C based on the phase difference θ are used. ) Are multiplied and added to each other, thereby removing the aliasing component from the pixel data of the frame buffers # 1 and # 2, and outputting the result to the frame buffer # 3. That is, in step (1409), coefficients C0, C1, C2, and C3 are determined based on the phase difference θ, and each coefficient and frame buffer # 1 are determined in steps (1410), (1411), (1412), and (1413). , The pixel data of # 2 and the data after the π / 2 phase shift are respectively multiplied, and then all are added in step (1414) and output to frame buffer # 3. The operation for removing the aliasing component is the same as that described with reference to FIG.

  Subsequently, in step (1415), it is determined whether or not the processing of all the pixels in the frame buffer # 1 has been completed.If the processing has not been completed, the next pixel (for example, the right adjacent pixel) is determined in step (1416). The process is set as a process target and the process returns to step (1405) and the subsequent steps. If completed, the process ends at step (1417).

  After the image signal processing of the flowchart shown in FIG. 14, the signal buffered in the frame buffer # 3 shown in FIG. 18 can be output to the display unit (3) in frame units or pixel units.

  By performing the processing as described above, it is possible to output a high resolution signal to the frame buffer # 3 using the pixel data of the frame buffer # 1 and the frame buffer # 2. When applied to a moving image, the process from step (1401) to step (1417) may be repeated for each frame.

  As for the image signal processing method according to the second embodiment, as in the description of FIG. 17, the difference in the operation of the above prior art can be confirmed. Is omitted.

  According to the image signal processing method according to the second embodiment described above, phase shift is performed on each image signal of two input image frames, which is smaller than in the conventional example, and two signals are generated from each image signal. . Thereby, four signals can be generated from the image signals of the two input image frames. Here, based on the phase difference between the two input image frames, for each of the four signals, a coefficient for canceling and combining the aliasing components of the four signals is calculated for each pixel. For each pixel of the image to be generated, a sum is calculated by multiplying the pixel value of the corresponding pixel included in each of the four signals by each coefficient to generate a pixel value of a new high-resolution image. By performing this for each pixel of the generated image, a new high-resolution image can be generated.

  As a result, the image signal processing method according to the second embodiment can generate an image with less aliasing components and higher resolution than the input image using two input image frames that are fewer than those in the conventional example.

  In addition, since the image signal processing method according to the second embodiment uses two smaller input image frames than the conventional example, the amount of necessary image processing can be reduced compared to the conventional example.

  FIG. 10 shows a third embodiment of the present invention. The configuration shown in the figure is a simplified version of the configuration shown in FIG. 1 by utilizing the relationship between the coefficients C0, C1, C2, and C3 shown in FIG. 9C. That is, since C0 = C2 = 1/2 and C1 = -C3 =-(1 + cosθ) / (2sinθ), frame # 1 after the up-rate and frame # 2 after the motion compensation / up-rate From these signals, an adder (1001) and a subtracter (1004) generate sum and difference signals. The sum signal passes through the fs cutoff filter (1002), is then multiplied by C0 (= 0.5) by the multiplier (1003), and is input to the adder (1008). Here, the fs cutoff filter (1002) is a filter that cuts off the component of the sampling frequency (fs) before the up-rate as a zero point, and can be realized by using, for example, a tap coefficient indicated by (1011) in FIG. This fs cutoff filter (1002) cannot remove the aliasing component because the gain of the frequency fs is zero due to the `` frequency-gain '' characteristic of the Hilbert transformer (1005) as shown in FIG. The purpose is to prevent an unnecessary component having a frequency fs from remaining. Therefore, if a means capable of π / 2 phase shift including the frequency fs component is used instead of the Hilbert transformer (1005), the fs cutoff filter (1002) becomes unnecessary.

  On the other hand, for the difference signal, the coefficient determined by the coefficient determiner (1007) based on the phase difference (102) after shifting the phase by a certain amount (= π / 2) by the Hilbert transformer (1005) C1 is multiplied by a multiplier (1006) and added by an adder (1008) to obtain an output. Here, the phase shift unit (1009) including the delay unit (1002) and the Hilbert transformer (1005) can be realized with a circuit scale that is half that of the phase shift unit (116) shown in FIG. The coefficient determiner (1007) only needs to output the coefficient C1 shown in FIG. 9 (c). The adder (1001), the subtracter (1004), the multipliers (1003) (1006), the adder ( 1008) and the aliasing component removal unit (1010) including the coefficient determiner (1007) can reduce the number of multipliers, and thus can be realized with a smaller circuit scale than the aliasing component removal unit (117) shown in FIG. .

  As for the image signal processing method according to the third embodiment, as in the description of FIG. 17, the difference in the operation of the above prior art can be confirmed. Is omitted.

  The image signal processing apparatus and the image signal processing method according to the third embodiment can also be applied to increase the resolution in the vertical direction and the oblique direction.

  In addition to the effects of the image signal processing apparatus according to the first embodiment, the image signal processing apparatus according to the third embodiment described above can be realized with a smaller circuit scale than the image signal processing apparatus according to the first embodiment, and can be realized at a lower cost. Can be realized.

  An image signal processing method according to the fourth embodiment of the present invention will be described with reference to FIG.

  The fourth embodiment relates to an image signal processing method in which processing equivalent to image signal processing in the image signal processing apparatus according to the third embodiment is realized by a control unit that cooperates with software. The image processing apparatus that performs the image signal processing method of the present embodiment is the image processing apparatus shown in FIG.

  FIG. 15 shows an example of a flowchart of the operation of the present embodiment. The flowchart of FIG. 15 starts from step (1501), and in step (1518), the image data of each frame is updated. That is, in step (1502), the image data of frame # 1 is updated and written to frame buffer # 1, and in step (1503), the image data of frame # 2 is updated and written to frame buffer # 2. Here, the up-rate can be realized by clearing the value of each frame buffer to 0 and then writing the data every other pixel.

  Next, in step (1504), the first pixel (for example, the upper left pixel) of frame buffer # 1 is set as the processing target, and processing is performed until all pixel data in frame buffer # 1 is processed. Loop.

  In step (1505), the position of the corresponding pixel in the frame buffer # 2 is estimated with reference to the target pixel in the frame buffer # 1, and the phase difference θ is output. At this time, the above-described conventional technique can be used as it is as a method of estimating the position of the corresponding pixel.

  In step (1506), based on the phase difference θ obtained in step (1505), pixels near the corresponding pixel in the frame buffer # 2 are motion compensated. At this time, as the “neighboring pixels”, only the pixel data used in the Hilbert transform processing in step (1510), that is, the pixel data in the range where the finite number of taps acts is compensated for motion. The motion compensation operation is the same as the operation described with reference to FIGS.

  Subsequently, in step (1520), the aliasing component is removed from the pixel data of the frame buffers # 1 and # 2 based on the phase difference θ and output to the frame buffer # 3. First, in step (1507), the value of pixel data in frame buffer # 1 and the value of pixel data in frame buffer # 2 subjected to motion compensation are added. In step (1509), the frequency fs component is cut off. The operation of this fs cutoff filter (1509) is the same as the operation of (1002) shown in FIG.

  In step (1508), the pixel data value of the frame buffer # 2 subjected to motion compensation is subtracted from the pixel data value of the frame buffer # 1. Here, with respect to the subtraction result, the phase is shifted by a certain amount in step (1519). That is, the Hilbert transform is performed in step (1510) using the neighboring data similarly subtracted. The phase shift operation is the same as the operation described with reference to FIGS.

  Subsequently, in step (1511), the data after the addition is multiplied by a coefficient C0 (= 0.5), and in step (1512), a coefficient C1 is determined based on the phase difference θ, and then in step (1513). After multiplying the coefficient C1 by the Hilbert transformed data, the two data are added in step (1514) and output to the frame buffer # 3. The operation of removing the aliasing component is the same as that described with reference to FIG.

  Subsequently, in step (1515), it is determined whether or not the processing of all the pixels in the frame buffer # 1 has been completed.If the processing has not been completed, the next pixel (for example, the right adjacent pixel) is determined in step (1516). The process is set as a process target, and the process returns to step (1505) and subsequent steps. If completed, the process ends at step (1517).

  After the image signal processing of the flowchart shown in FIG. 15, the signal buffered in the frame buffer # 3 shown in FIG. 18 can be output to the display unit (3) in frame units or pixel units.

  By performing the processing as described above, it is possible to output a high resolution signal to the frame buffer # 3 using the pixel data of the frame buffer # 1 and the frame buffer # 2. When applied to a moving image, the process from step (1501) to step (1517) may be repeated for each frame.

  The image signal processing method according to the fourth embodiment can also be confirmed by using FIG. 17 to confirm the difference in operation of the above prior art, but the result is the same as that of the first embodiment, and the description thereof is omitted. To do.

  The image signal processing method according to the fourth embodiment can also be applied to increase the resolution in the vertical direction and the oblique direction.

  The image signal processing method according to the fourth embodiment described above has the same effect of increasing the resolution of the image signal as the image signal processing method according to the second embodiment. Furthermore, the image signal processing method according to the fourth embodiment is less than the image signal processing method according to the second embodiment by sharing the contents of some processing steps as compared with the image signal processing method according to the second embodiment. There is an effect that it is possible to realize the same signal processing with a processing amount (the number of operations).

  FIG. 11 shows a fifth embodiment of the present invention. In the configuration shown in FIG. 9, the coefficients C1 and C3 become indefinite when the phase difference θ is 0 as shown in FIG. 9D, and the coefficients C1 and C3 increase as the phase difference θ approaches 0. Therefore, based on the configuration shown in FIG. 10, when the phase difference θ becomes close to 0, the output from the auxiliary pixel interpolation unit (1105) is switched. It is composed. In other words, a general interpolation low-pass filter (1101) is prepared as a bypass path, and in addition to the above-described coefficients C0 and C1 in the coefficient determiner (1103), C4 is newly generated, and the multiplier (1102) The output of the interpolation low-pass filter (1101) is multiplied by the coefficient C4, and the result is output in addition to the signal whose resolution has been increased by the adder (1104).

  The configuration other than the interpolation low-pass filter (1101), multiplier (1102), coefficient determiner (1103), adder (1104), and auxiliary pixel interpolation unit (1105) is the same as that of the third embodiment shown in FIG. Since it is the same, description is abbreviate | omitted.

  FIG. 12 shows a specific example of the interpolation low-pass filter (1101) used in the fifth embodiment of the present invention. This figure shows the tap coefficient of the filter obtained by inverse Fourier transform of the frequency characteristic with the cutoff frequency being 1/2 of the original sampling frequency fs. At this time, each tap coefficient Ck (where k is an integer) becomes a general sinc function, and may be Ck = sin (πk / 2) / (πk / 2).

  FIG. 13 shows a specific example of the coefficient determiner (1103) used in the fifth embodiment of the present invention. This figure is based on the coefficients C0 and C1 shown in Fig. 9 (d), and normally the new coefficient C4 is 0, but when the phase difference θ is close to 0, the value of the coefficient C1 is forced. The operation of setting the coefficient C4 to 1.0 and setting the coefficient C4 to 1.0 is shown. This operation automatically switches the output of the adder (1104) to the output of the interpolation low-pass filter (1101) when the phase difference θ (102) becomes 0 or close to 0 in the configuration shown in FIG. Will be able to. Note that, as the phase difference θ approaches 0, the coefficient shown in FIG. 12 may be gradually and gradually approximated to the coefficient shown in FIG. Also, when the position estimation unit (101) in FIG. 1 determines that the pixel corresponding to the pixel to be processed on the frame # 1 is not on the frame # 2, the phase difference θ (102) is close to zero. As in the case of the above, each coefficient may be controlled to automatically switch the output of the adder (1104) to the output of the interpolation low-pass filter (1101).

  Note that the image signal processing apparatus according to the fifth embodiment can also confirm the difference in operation of the above-described conventional technique with reference to FIG. 17, but the result is the same as that of the first embodiment, and thus the description thereof is omitted. To do.

  The image signal processing apparatus according to the fifth embodiment can also be applied to increase the resolution in the vertical direction and the oblique direction.

  In addition to the effects of the image signal processing device according to the third embodiment, the image signal processing device according to the fifth embodiment described above has a phase difference θ (102) of 0 or 0 as compared with the image signal processing device according to the third embodiment. The processing result will not be undefined even when it is close to 0 (i.e., still or almost stationary) or when it is determined that the pixel corresponding to the pixel to be processed on frame # 1 is not on frame # 2. This has the effect that a stable output image can be obtained.

  An image signal processing method according to the sixth embodiment of the present invention will be described with reference to FIG.

  Example 6 relates to an image signal processing method in which processing equivalent to image signal processing in the image signal processing apparatus according to Example 5 is realized by a control unit that cooperates with software. The image processing apparatus that performs the image signal processing method of the present embodiment is the image processing apparatus shown in FIG.

  FIG. 16 shows an example of a flowchart of the operation of the present embodiment. As shown in FIG. 9 (d), the processing steps shown in FIG. 9 are performed when the coefficients C1 and C3 become indefinite when the phase difference θ is 0, and as the phase difference θ approaches 0, the coefficient C1 In order to prevent the noise from being weakened by increasing C3, the step (1606) is performed when the phase difference θ becomes 0 or near 0 based on the steps of FIG. 15 described in the fourth embodiment. This processing result is output to the frame buffer # 3. That is, coefficients C0, C1, and C4 are determined based on the phase difference θ in step (1601), and the target pixel data in frame buffer # 1 and its neighboring pixel data are used in step (1602). After performing general interpolation low-pass filter processing, the coefficient C4 is multiplied in step (1603), and the result is added to the outputs of steps (1511) and (1513) in step (1604) to obtain frame buffer # 3. Output to.

  Steps other than these are the same as the processing steps of FIG. 15 described in the fourth embodiment, and a description thereof will be omitted. The coefficient determination operation in step (1601) is the same as the operation shown in FIG. The operation of the interpolation low-pass filter in step (1602) is the same as that shown in FIG.

  Note that the image signal processing method according to the sixth embodiment can also be confirmed by using FIG. 17 to confirm the difference in operation of the above-described conventional technique, but the result is the same as that of the first embodiment, and thus the description thereof is omitted. To do.

  Note that the image signal processing method according to the sixth embodiment can also be applied to increase the resolution in the vertical and oblique directions.

  In the image signal processing method according to the sixth embodiment described above, in addition to the effect of the image signal processing method according to the fourth embodiment, the phase difference θ (102) is 0 or 0 as compared with the image processing method according to the fourth embodiment. Even when it is determined that the pixel corresponding to the pixel to be processed on frame # 1 is not on frame # 2 when it is close (i.e., still or almost still), the processing result is not uncertain, This has the effect that a stable output image can be obtained.

  FIG. 20 shows an image signal processing apparatus according to Embodiment 7 of the present invention. The image processing apparatus according to the present embodiment includes, for example, an input unit (1) to which a frame sequence of a moving image such as a television broadcast signal is input, and a frame input from the input unit (1) in the horizontal and vertical directions. A resolution conversion unit (4) for increasing the combined two-dimensional resolution and a display unit (3) for displaying an image based on the frame whose resolution has been increased by the resolution conversion unit (4) are provided. .

  In this resolution conversion unit (4), by performing resolution conversion processing in each of the horizontal direction and the vertical direction, a component having a large resolution improvement effect is selectively or mixedly output from each result, thereby two-dimensionally. Realize higher resolution. Details of the resolution conversion unit (4) will be described below.

  In FIG. 20, based on frame # 1 (2010) and frame # 2 (2013) input to the input unit (1), a horizontal resolution conversion unit (2001) and a vertical resolution conversion unit (2005) are used. A frame (2011) with an increased number of pixels in the horizontal direction and a frame (2014) with an increased number of pixels in the vertical direction are generated.

  Here, each resolution conversion unit (2001) (2005) uses the configuration of the resolution conversion unit (2) of the image signal processing apparatus according to the first embodiment of the present invention shown in FIG. Each signal processing is performed. At this time, in the horizontal resolution converter (2001), the up-raters (103) (104), delay units (105) (107), and π / 2 phase shifters (106) (108) shown in FIG. Each is configured to perform horizontal up-rate, delay, and π / 2 phase shift.

  Similarly, in the vertical resolution converter (2005), the up-raters (103) (104), delay units (105) (107), and π / 2 phase shifters (106) (108) shown in FIG. Each is configured to perform vertical uprate, delay, and π / 2 phase shift. These can be implemented using the operations shown in FIGS. 5 to 8 and the prior art.

  Each resolution conversion unit (2001) (2005) includes, in place of the configuration of the resolution conversion unit of the image signal processing apparatus according to the first embodiment of the present invention, the image signal processing apparatus according to the third embodiment of the present invention. The present invention can also be realized by using the resolution conversion unit and the resolution conversion unit of the image signal processing apparatus according to the fifth embodiment of the present invention. The following description will be made assuming that the configuration of the resolution conversion unit of the image signal processing apparatus according to the first embodiment of the present invention is used.

  In the present embodiment, assuming that the subject has moved two-dimensionally in the horizontal and vertical directions, the operations shown in FIGS. 1 and 2 are extended two-dimensionally. That is, the position estimation unit ((101) in FIG. 1) and the motion compensation / uprate unit ((115) in FIG. 1) in the horizontal resolution conversion unit (2001) use the subject on the frame # 1 as a reference. As described above, the subject on the frame # 2 is compensated for two-dimensional motion, and the horizontal phase difference θH among the sampling phase differences of the pixels of each frame is determined for the coefficient of the aliasing component removal unit ((117) in FIG. 1). Use.

  Similarly, in the position estimation unit ((101) in FIG. 1) and the motion compensation / uprate unit ((115) in FIG. 1) in the vertical resolution conversion unit (2005), the subject on frame # 1 ( 2016) as a reference, the subject (2017) on frame # 2 is two-dimensionally motion compensated, and among the sampling phase differences of the pixels of each frame, the vertical phase difference θV is the aliasing component removal unit (in FIG. Used to determine the coefficient of 117)). To determine the coefficient of the aliasing component removal unit ((117) in FIG. 1), the operation shown in FIG. 9 may be used as it is.

  Assuming that the subject has moved in an oblique direction, the frame (2011) in which the number of pixels in the horizontal direction has been increased by the horizontal resolution converter (2001) will contain diagonal distortion, but the original input In a component with a low vertical frequency of the signal (such as a vertical line), this distortion is small enough to be ignored. Similarly, the frame (2014) in which the number of pixels in the vertical direction has been increased by the vertical resolution converter (2005) will include diagonal distortion, but the original input signal has a low horizontal frequency component (horizontal line). Etc.), this distortion is small enough to be ignored.

  Using this characteristic, the frame (2011) in which the number of pixels in the horizontal direction is increased according to the signal processing described above is converted into a frame by the vertical interpolation unit (2004) composed of the vertical uprater (2002) and the pixel interpolator (2003). (2012) is generated and used as the SR (horizontal) signal. Here, the pixel interpolator (2003) may use a general vertical low-pass filter that outputs an average value of upper and lower pixel data of a pixel to be interpolated. Similarly, a frame (2014) in which the number of pixels in the vertical direction is increased is generated by a horizontal interpolation unit (2008) including a horizontal up-rate unit (2006) and a pixel interpolator (2007), and a frame (2015) is generated. (Vertical) signal. Here, the pixel interpolator (2007) may use a general horizontal low-pass filter that outputs an average value of the left and right pixel data of the pixel to be interpolated.

  In this way, if the pixel interpolator (2003) (2007) is used to remove the high-frequency component in the direction orthogonal to the direction of the processing target and extract only the low-frequency component, when moving in the oblique direction described above, The influence of the generated distortion can be made small enough to be ignored. The SR (horizontal) signal and the SR (vertical) signal generated by the above processing are mixed by the mixer (2009) to be an output signal and displayed on the display unit (3).

  Here, the detailed configuration and operation of the mixer (2009) will be described. The mixer (2009) may use any of the following three configuration examples.

  FIG. 22 shows a first configuration example of the mixer (2009). In the figure, an adder (2201) and a multiplier (2202) are used to generate and output an average value of SR (horizontal) and SR (vertical) signals input to the mixer (2009). In the configuration shown in the figure, the horizontal and vertical resolution improvement effects are also halved, but the mixer (2009) is the simplest configuration and can be realized at low cost.

  FIG. 23 shows a second configuration example of the mixer (2009). In the figure, for the SR (horizontal) and SR (vertical) signals input to the mixer (2009), a multiplier (2303) and a multiplier (2304) are used to calculate a coefficient K (horizontal) and a coefficient K. Multiply each by (vertical) and add both by the adder (2305) to obtain the output. The coefficient K (horizontal) and the coefficient K (vertical) are generated by coefficient determiners (2301) and (2302), respectively. The operation of the coefficient determiners (2301) and (2302) will be described below.

  The aliasing component removal units (2108) and (2109) shown in FIG. 21 are connected to the coefficient determiner (109) shown in FIG. 1 based on the phase difference θH (2102) and the phase difference θV (2103) shown in FIG. The coefficients C0 to C3 shown in FIG. 9 are generated to perform the aliasing component removal calculation. At this time, the coefficients C1 and C3 become indefinite when the phase differences θH (2102) and θV (2103) are 0, and the coefficients C1 and C3 are increased as the phase differences θH (2102) and θV (2103) approach 0. In order to prevent it from becoming vulnerable to noise and the like by increasing the coefficient, the coefficient C4 (0 ≦ C4 ≦ 1) shown in FIG. 13 is introduced, and auxiliary pixel interpolation is performed as in the configuration shown in FIG. preferable. In other words, the effect of improving the resolution is obtained when the value of the coefficient C4 is 0.0, but the effect of improving the resolution becomes smaller as the value of the coefficient C4 approaches 1.0.

  Using this property, when the horizontal phase difference θH (2102) is near 0 (that is, the coefficient C4 (horizontal) is near 1.0), the SR (vertical) of the vertical resolution conversion result is strongly reflected, and the vertical phase difference θV ( 2103) is close to 0 (i.e., the coefficient C4 (vertical) is close to 1.0), so that the horizontal resolution conversion result SR (horizontal) is strongly reflected, using the values of the coefficient C4 in the horizontal and vertical directions respectively. A coefficient K (horizontal) and a coefficient K (vertical) are determined. In order to realize this operation, for example, the coefficient determiner (2301) shown in FIG. 23 performs an operation of K (horizontal) = C4 (horizontal) + (1-C4 (vertical)) / 2 to determine K (horizontal). The coefficient determiner (2303) determines K (vertical) by performing an operation of K (vertical) = C4 (vertical) + (1-C4 (horizontal)) / 2.

  FIG. 24 summarizes an example of the outputs (coefficient K (horizontal) and coefficient K (vertical)) of the coefficient determiners (2301) and (2302) when the coefficient C4 (horizontal) and the coefficient C4 (vertical) are changed. Show. As shown in the figure, when the coefficient C4 (horizontal) increases, the coefficient K (horizontal) decreases and the coefficient K (vertical) increases, and when the coefficient C4 (vertical) increases, the coefficient K (horizontal) increases. It operates so that the coefficient K (vertical) becomes small.

  When the values of the coefficient C4 (horizontal) and the coefficient C4 (vertical) are equal, the coefficient K (horizontal) and the coefficient K (vertical) are each 0.5. In this way, for the coefficient C4 that changes independently in horizontal and vertical, the coefficient K is determined to be just 1.0 by adding the coefficient K (horizontal) and the coefficient K (vertical), and SR (horizontal) and Mix SR (vertical).

  The third operation and configuration example of the mixer (2009) will be described using FIG. 25 and FIG. 26, respectively. FIG. 25 shows a two-dimensional frequency region in which the horizontal frequency is represented by μ and the vertical frequency is represented by ν. Assuming that the horizontal sampling frequency of the original input image is μs and the vertical sampling frequency is νs, the output of the resolution conversion unit (4) shown in FIGS. 20 and 21 is a vertical frequency in the range of −μs to μs. The frequency ν is a signal in the range of −νs to νs.

  High-frequency components are reproduced by horizontal and vertical resolution conversions, but the high-frequency components originally have a low signal level, so the effect of horizontal resolution conversion is large (μ, ν) = (± μs / 2, 0) component in the frequency region (2501) in the vicinity (particularly (μ, ν) = (+ μs / 2, 0)), a region of a frequency where μ> 0, and (μ, ν) = (− μs / 2, 0), and the component of the frequency region where μ <0), and the effect of vertical resolution conversion is large in the frequency region in the vicinity of (μ, ν) = (0, ± νs / 2) ( 2502) components (particularly including (μ, ν) = (0, + νs / 2), including a frequency region where ν> 0, and (μ, ν) = (0, -νs / 2), component in a frequency region where ν <0).

  Therefore, when these frequency components (2501) and (2502) are extracted and mixed by a two-dimensional filter, components having a large resolution improvement effect can be selectively output.

  FIG. 26 shows a configuration example of a mixer (2009) that extracts components having a large effect by horizontal and vertical resolution conversion. In the figure, a two-dimensional filter (2601) is used to extract a component in the frequency region (2501) having a large resolution improvement effect of SR (horizontal) input to the mixer (2009). Similarly, using the two-dimensional filter (2602), a component in the frequency region (2502) having a large resolution improvement effect of SR (vertical) input to the mixer (2009) is extracted.

  As a component other than the frequency domain (2501) (2502), an adder (2603) and a multiplier (2604) are used to create an average signal of SR (horizontal) and SR (vertical), and a two-dimensional filter (2605) The components other than the passbands of the two-dimensional filters (2601) and (2602) (that is, the remaining components) are extracted. The output signals of the two-dimensional filters (2601), (2602), and (2605) are added by the adder (2606) to obtain the output of the mixer (2009).

Note that the numbers in circles in the two-dimensional filters (2601), (2602), and (2605) shown in the figure indicate examples of the tap coefficients of the respective filters. (The coefficients of each filter are expressed as integers for the sake of simplicity. The original coefficient values are indicated by a circled number and “× 1/16” on the right). For example, in the two-dimensional filter (2601), the original coefficient value is obtained by multiplying each circled number by 1/16. Same for coefficient.)
The two-dimensional filter (2601) is the product of a horizontal bandpass filter and a vertical lowpass filter with ± μs / 2 as the center frequency of the passband, and the two-dimensional filter (2602) has ± νs / 2 as the center frequency of the passband. The product of the vertical bandpass filter and the horizontal lowpass filter may be used, and the two-dimensional filter (2605) may have a characteristic obtained by subtracting the passbands of the two-dimensional filter (2601) and the two-dimensional filter (2602) from the entire band.

  Next, the difference between the processing in the image signal processing apparatus according to the seventh embodiment and the operation of the conventional technique will be described with reference to FIG. The figure (a) shows frame # 1 (3401), frame # 2 (3402), frame # 3 (3403), frame # 4 (3404), frame # 5 (3405) input to the resolution converter (4). (B) shows each frame output from the resolution converter (4). In each frame, the subject moves clockwise by 1/4 pixel, and the subject is intentionally moved so as to make one round in four frames. This movement is continued in the same manner after frame # 6.

  In the prior art described in Patent Document 1, Patent Document 2, and Non-Patent Document 1, as described above, when the resolution is increased with respect to a horizontal / vertical two-dimensional input signal, the aliasing comes from two directions. So, if the bandwidth of the original signal is doubled both vertically and horizontally, the three aliasing components overlap, and 2M + 1 = 7 digital data (= 7 frame image signals) is required to cancel them. It was. Therefore, when a signal that circulates in 4 frames as shown in FIG. 34 (a) is input, independent data cannot be obtained no matter which 7 frames are selected, and the solution by the high resolution processing becomes indefinite. It is not required.

  On the other hand, if this embodiment is used, for example, two adjacent frames (for example, frame # 1 (3401) and frame # 2 (3402), (or frame # 2 (3402) and frame # 3 (3403))) are used. Thus, as shown in FIG. 5B, the horizontal (or vertical) aliasing component can be removed to achieve high resolution. That is, the operation status of the present embodiment can be confirmed by using the input image of FIG. If a generally well-known circular zone plate (CZP: Circular Zone Plate) is used as the pattern of this test pattern, the effect of resolution conversion can be directly viewed on the display unit (3). That is, if the circular zone plate is moved left and right for each frame, an image with improved horizontal resolution is displayed, and if it is moved up and down (or diagonally), an image with improved vertical (or diagonally) resolution is displayed. The effect of improving the resolution according to the moving direction of the test pattern can be confirmed.

  According to the image signal processing apparatus according to the seventh embodiment described above, phase shift is performed on each image signal of two input image frames, and two signals are generated from each image signal. As a result, four signals are generated from the image signals of the two input image frames. Here, based on the phase difference between the two input image frames, for each of the four signals, a coefficient for canceling and combining the aliasing components of the four signals is calculated for each pixel. For each pixel of the image to be generated, a sum is calculated by multiplying the pixel value of the corresponding pixel included in each of the four signals by each coefficient to generate a pixel value of a new high-resolution image. By performing this for each pixel of the generated image, an image having a higher resolution in the one-dimensional direction than the input image frame is generated.

  This is performed in each of the horizontal direction and the vertical direction, and an image with high resolution in the horizontal direction and an image with high resolution in the vertical direction are generated. The image having the high resolution in the horizontal direction and the image having the high resolution in the vertical direction are subjected to up-rate processing in the vertical direction and the horizontal direction, respectively, and then both are mixed.

  As a result, it is possible to generate a high-resolution image in which the resolution is increased in both the vertical direction and the horizontal direction from the image signals of two input image frames that are smaller than in the conventional example. That is, a two-dimensional high resolution image can be generated.

  In addition, since the image signal processing apparatus according to the seventh embodiment uses two input image frames that are fewer than those in the conventional example, the amount of necessary image processing can be reduced as compared with the conventional example. Accordingly, it is possible to realize an image signal processing apparatus that generates a high-resolution image with less aliasing components and higher resolution than the input image in both the vertical and horizontal directions at a lower cost than the conventional example. .

  In addition, according to the prior art described in Patent Document 1, Patent Document 2, and Non-Patent Document 1, one-dimensional high resolution such as a horizontal direction and a vertical direction is performed in a plurality of directions using three frames. These results may be input to the mixer (2009) shown in FIG. 20 and mixed, and output as a two-dimensional resolution conversion result. In this case, as shown in FIG. 20, the scale of the signal processing circuit such as the frame memory or the motion estimation unit is larger than the configuration in which the two-dimensional resolution conversion is performed using only two frames. 2. As described in Non-Patent Document 1, it is possible to reduce the scale of a signal processing circuit such as a frame memory or a motion estimation unit, compared to using a signal of at least 7 frames.

  Further, the present invention is not limited to the conventional techniques described in Patent Document 1, Patent Document 2, and Non-Patent Document 1, and other one-dimensional high resolution such as a horizontal direction and a vertical direction is applied by applying a conventional high resolution technology. It is also possible to perform conversion into a plurality of directions, input each result into a mixer (2009) shown in FIG. 20, and mix and output the result as a two-dimensional resolution conversion result.

  In FIG. 20, the case where the resolution of the frame # 1 is converted using the set of the input signals of the frame # 1 and the frame # 2 has been described as an example. # 3, Frame # 1 and Frame # 4, etc. are used to convert the resolution of frame # 1 and mix the results to obtain the final resolution conversion result of frame # 1. .

  As a mixing method at this time, an average value of the results may be taken, or mixing may be performed according to the value of the coefficient C4 (frame) for each frame as shown in FIGS. In this case, as the coefficient C4 (frame), the MAX value (the smaller value) of the coefficient C4 (horizontal) and the coefficient C4 (vertical) for each frame may be used. Also, all the coefficients C4 (horizontal) and C4 (vertical) are compared for each pixel, and the resolution conversion result obtained from the group with the smallest coefficient C4 (i.e., the group with the greatest resolution improvement effect) is obtained for each pixel. The final resolution conversion result of frame # 1 may be selected.

  Thus, for example, with frame # 1 as a reference, if frame # 2 is a frame earlier than frame # 1 and frame # 3 is a future frame than frame # 1, the subject is When changing from `` motion '' to `` still '' (end of movement), resolution conversion processing is performed with frame # 1 and frame # 2, and when the subject is converted from `` still '' to `` motion '' at the time of frame # 1 ( (Beginning of motion) Since each processing result is mixed for each pixel so that resolution conversion processing is performed by frame # 1 and frame # 3, the motion of the subject can be used effectively, and the resolution improvement effect is the largest can do.

  FIG. 21 shows an image signal processing apparatus according to Embodiment 8 of the present invention. The image processing apparatus according to the present embodiment is a modification of the configuration of the seventh embodiment described above, and the processing order of the resolution conversion units (2001) (2005) and the interpolation units (2004) (2008) shown in FIG. On the other hand, the resolution conversion is performed after the interpolation process. As a result, the up-rater ((103) (104) in FIG. 1) in the resolution conversion unit (2001) (2005) and the up-rater in the interpolation unit (2004) (2008) (FIG. 20 (2002) (2006)) and the horizontal resolution conversion unit (2001) and the vertical resolution conversion unit (2005), each position estimation unit ((101) in FIG. 1) Since they can be shared, similar signal processing can be realized with a smaller circuit scale and operation amount.

  In FIG. 21, first, the position estimation unit (2101) performs a corresponding operation on the frame # 2 with reference to the sampling phase (sampling position) of the pixel to be processed on the frame # 1 input to the input unit (1). The pixel position is estimated, and the sampling phase differences θH (2102) and θV (2103) in the horizontal and vertical directions are obtained.

  Next, the motion compensation / uprate unit (2110) uses the uprater (2104) (2105) to perform motion compensation on the frame # 2 using the information on the phase differences θH (2102) and θV (2103) to In addition to matching the position with 1, the number of pixels in frame # 1 and frame # 2 is doubled both horizontally and vertically (4 times in total) to increase the density. The up-raters (2104) and (2105) are obtained by extending the operation / configuration shown in FIGS. 5 and 6 to two dimensions in the horizontal and vertical directions. The phase shift unit (2111) shifts the phase of the densified data by a certain amount.

  At this time, the horizontal phase shifter (2106) performs a horizontal phase shift, and the vertical phase shifter (2107) performs a vertical phase shift. The delay units (105) and (107) shown in FIG. ) And π / 2 phase shifter (108) and the operations and configurations shown in FIGS.

  For each phase-shifted signal, horizontal and vertical folding components are removed by the horizontal folding component removal unit (2108) and vertical folding component removal unit (2109) in the folding component removal unit (2112), respectively. . Next, the output of the horizontal aliasing component removal unit (2108) is pixel-interpolated using the pixel interpolator (2003) to obtain an SR (horizontal) signal, and the output of the vertical aliasing component removal unit (2109) is the pixel. Pixel interpolation is performed using an interpolator (2007) to form an SR (vertical) signal, which is mixed by an mixer (2009) and output.

  The folded component removing units (2108) and (2109) can use the configuration of the folded component removing unit (117) shown in FIG. 1 as it is. As the phase difference θ (102), the folded component removing unit (2108) uses the horizontal phase difference θH (2102), and the folded component removing unit (2109) uses the horizontal phase difference θH (2103) to perform the operation shown in FIG. By performing the above, folding components in the respective directions can be removed.

  In the above description, the phase shift unit (2111) includes the delay units (105) and (107) and the π / 2 phase shifter (108) shown in FIG. 1 and the operations and configurations shown in FIGS. The aliasing component removal units (2108) and (2109) are performed in the same manner as described above, and the configuration of the aliasing component removal unit (117) shown in FIG. 1 is used as it is, but instead, the phase shift unit (2111) The phase shift unit (1009) of FIG. 10 is used for the vertical direction and the horizontal direction, respectively, and the aliasing component removal units (2108) and (2109) may also use the aliasing component removal means (1010) of FIG. Good. Further, at this time, each of the aliasing component removal units (2108) and (2109) may be provided with the auxiliary pixel interpolation unit (1105) of FIG. 11 as in FIG.

  Note that the mixer (2009) is the same as that of the seventh embodiment, and a description thereof will be omitted.

  Also, the operation for the input frame shown in FIG. 34 is the same as that of the seventh embodiment, and thus the description thereof is omitted.

  The image signal processing apparatus according to the eighth embodiment described above has the same effects as the image signal processing apparatus according to the seventh embodiment, but a part of the processing units is shared as compared with the image signal processing apparatus according to the seventh embodiment. By doing so, it is possible to realize the same signal processing with a circuit scale and a calculation amount smaller than those of the image signal processing apparatus according to the seventh embodiment.

  In addition, according to the prior art described in Patent Document 1, Patent Document 2, and Non-Patent Document 1, one-dimensional high resolution such as a horizontal direction and a vertical direction is performed in a plurality of directions using three frames. These results may be input to the mixer (2009) shown in FIG. 21 and mixed, and output as a two-dimensional resolution conversion result. In this case, as shown in FIG. 21, the scale of the signal processing circuit such as the frame memory and the motion estimation unit is larger than the configuration in which the two-dimensional resolution conversion is performed using only two frames. 2. As described in Non-Patent Document 1, it is possible to reduce the scale of a signal processing circuit such as a frame memory or a motion estimation unit, compared to using a signal of at least 7 frames.

  In addition, the present invention is not limited to the conventional techniques described in Patent Document 1, Patent Document 2, and Non-Patent Document 1, and other conventional high resolution techniques are applied to apply a one-dimensional height such as a horizontal direction or a vertical direction. Resolution may be performed in a plurality of directions, and each result may be input to the mixer (2009) shown in FIG. 21 and mixed, and output as a two-dimensional resolution conversion result.

  In FIG. 21, the case where the resolution of the frame # 1 is converted using the set of the input signals of the frame # 1 and the frame # 2 has been described as an example. # 3, Frame # 1 and Frame # 4, etc. are used to convert the resolution of frame # 1 and mix the results to obtain the final resolution conversion result of frame # 1. .

  As a mixing method at this time, an average value of the results may be taken, or mixing may be performed according to the value of the coefficient C4 (frame) for each frame as shown in FIGS. In this case, as the coefficient C4 (frame), the MAX value (the smaller value) of the coefficient C4 (horizontal) and the coefficient C4 (vertical) for each frame may be used. Also, all the coefficients C4 (horizontal) and C4 (vertical) are compared for each pixel, and the resolution conversion result obtained from the group with the smallest coefficient C4 (i.e., the group with the greatest resolution improvement effect) is obtained for each pixel. The final resolution conversion result of frame # 1 may be selected.

  Thus, for example, with frame # 1 as a reference, if frame # 2 is a frame earlier than frame # 1 and frame # 3 is a future frame than frame # 1, the subject is When changing from `` motion '' to `` still '' (end of movement), resolution conversion processing is performed with frame # 1 and frame # 2, and when the subject is converted from `` still '' to `` motion '' at the time of frame # 1 ( (Beginning of motion) Since each processing result is mixed for each pixel so that resolution conversion processing is performed by frame # 1 and frame # 3, the motion of the subject can be used effectively, and the resolution improvement effect is the largest can do.

  FIG. 27 shows an image signal processing apparatus according to Embodiment 9 of the present invention. The image processing apparatus according to the present embodiment has a configuration in which a high-resolution conversion unit for diagonal components in the lower right and upper right directions is added to the configuration example shown in FIG. That is, an oblique (lower right) phase shift unit (2701) and an oblique (upper right) phase shift unit (2702) are added to the phase shift unit (2708), and the aliasing component removal unit (2705) is added to the aliasing component removal unit (2709). ) (2706) and after passing through the pixel interpolators (2710) and (2711), the SR (horizontal), SR (vertical), SR (upper right) and SR (lower right) signals are mixed. Mix at (2707) and use as output. Here, the pixel interpolators (2710) and (2711) may use a general two-dimensional low-pass filter that outputs an average value of upper, lower, left, and right pixel data of a pixel to be interpolated.

  Phase difference information in the oblique direction is required as the phase difference θ, and the phase difference (θH + θV) obtained by adding the horizontal phase difference θH (2102) and vertical phase difference θV (2103) with the adder (2703) is removed. The phase difference (−θH + θV) generated by the subtracter (2704) may be input to the aliasing component removal unit (2706). The configurations and operations of the aliasing component removal units (2106), (2109), (2705), and (2706) are all the same.

  FIGS. 28A to 28D show a horizontal phase shift unit (2106), a vertical phase shift unit (2107), an oblique (lower right) phase shift unit (2701), and an oblique (upper right) phase shift in the two-dimensional frequency domain. Each operation of the unit (2702) is shown. FIGS. 28A to 28D are two-dimensional frequency regions in which the horizontal frequency is represented by μ and the vertical frequency is represented by ν, as in FIG. These phase shift units (2106) (2107) (2701) (2702) have the same configuration as the phase shift unit (116) shown in FIG. 1, and the π / 2 phase shifters (106) (108) therein ) “Frequency-phase difference” characteristics are changed according to each direction.

  That is, in the figure (a), in the horizontal phase shift unit (2106), when the horizontal frequency sampling frequency of the input signal is μs, the frequency in the range of −μs to 0 is the same as the operation shown in FIG. The phase of the component is shifted by π / 2, and the phase of the frequency component in the range of 0 to μs is shifted by -π / 2. Similarly, in the vertical phase shift unit (2107), when the vertical frequency sampling frequency of the input signal is νs, the phase of the frequency component in the range of −νs to 0 is shifted by π / 2, and the range of 0 to νs. The phase of the frequency component of is shifted by -π / 2.

  Similarly, in the oblique (lower right) phase shift unit (2701) and the oblique (upper right) phase shift unit (2702), as shown in FIG. Shift by / 2 or π / 2. These “frequency-phase difference” characteristics are arranged in the horizontal, vertical, diagonal (lower right) and diagonal (upper right) directions according to the two-dimensional sampling points. Can be easily realized.

  FIG. 29 shows a first configuration example of the mixer (2707). In the same figure, each of SR (horizontal), SR (vertical), SR (lower right), SR (upper right) input to the mixer (2707) using an adder (2901) and a multiplier (2902) Generate and output the average value of the signal. The configuration shown in the figure is an example of the simplest configuration of the mixer (2707), but the horizontal, vertical, lower right, and upper right resolution improvement effects are also each reduced to 1/4.

  FIG. 30 shows a second configuration example of the mixer (2707). In the figure, for each signal of SR (horizontal), SR (vertical), SR (bottom right), SR (top right) input to the mixer (2707), a multiplier (3005), a multiplier (3006) Using the multiplier (3007) and multiplier (3008), multiply by the coefficient K (horizontal), coefficient K (vertical), coefficient K (lower right) and coefficient K (upper right), respectively, and adder (3009) These signals are added to produce an output. The coefficient K (horizontal), the coefficient K (vertical), the coefficient K (lower right), and the coefficient K (upper right) are generated by coefficient determiners (3001) (3002) (3003) (3004), respectively. The operation of the coefficient determiners (3001) (3002) (3003) (3004) will be described below.

  The aliasing component removal units (2108), (2109), (2705), and (2706) shown in FIG. 27 have the phase difference θH (2102), phase difference θV (2103), phase difference (θH + θV), position shown in FIG. Based on the phase difference (−θH + θV), the coefficient determiner (109) shown in FIG. 1 generates coefficients C0 to C3 shown in FIG. At this time, when the phase differences θH (2102), θV (2103), (θH + θV), (−θH + θV) are 0, the coefficients C1 and C3 become indefinite, the phase difference θH (2102), In order to prevent the coefficients C1 and C3 from becoming large as θV (2103), (θH + θV), (−θH + θV) approaches 0, the coefficient C4 (0 ≦ C4 ≦ 1) is introduced, and auxiliary pixel interpolation is preferably performed as in the configuration shown in FIG. In other words, the effect of improving the resolution is obtained when the value of the coefficient C4 is 0.0, but the effect of improving the resolution becomes smaller as the value of the coefficient C4 approaches 1.0. Using this property, when the horizontal phase difference θH (2102) is close to 0 (that is, when the coefficient C4 (horizontal) is close to 1.0), the horizontal resolution conversion result SR (horizontal) becomes weak and the horizontal phase difference θH (2102 ) Is not near 0 (i.e., when the coefficient C4 (horizontal) is near 0.0), the coefficient determiner (3001) sets the coefficient K (horizontal) so that the SR (horizontal) of the horizontal resolution conversion result becomes strong. decide.

  As an example of this, the coefficient K (horizontal) = (1 + C4 (horizontal) * 3-C4 (vertical) −C4 (lower right) −C4 (upper right)) / 4 may be set. Similarly, coefficients K (vertical), K (lower right), and K (upper right) are determined by coefficient determiners (3002), (3003), and (3004), respectively. At this time, for coefficient C4 (horizontal), coefficient C4 (vertical), coefficient C4 (lower right), and coefficient C4 (upper right) that change independently, coefficient K (horizontal) + coefficient K (vertical) + coefficient K The coefficient K is determined so that (lower right) + coefficient K (upper right) = 1.0, and SR (horizontal), SR (vertical), SR (lower right), SR (upper right) are mixed.

  31 and 32 show a third operation and a configuration example of the mixer (2707), respectively. FIG. 31 shows a two-dimensional frequency region in which the horizontal frequency is represented by μ and the vertical frequency is represented by ν, as in FIG. In FIG. 31, when the horizontal sampling frequency of the original input image is μs and the vertical sampling frequency is νs, the output of the resolution converter (4) shown in FIG. 27 is the horizontal frequency μ in the range of −μs to μs, The vertical frequency ν is a signal in the range of −νs to νs.

  The effect of the diagonal (upper right) resolution conversion is large, as shown in FIG. 31, in the vicinity of (μ, ν) = (+ μs / 2, + νs / 2) and (μ, ν) = (− μs / 2, -νs / 2) in the vicinity of the frequency region (3101) (particularly (μ, ν) = (+ μs / 2, + νs / 2), with a frequency such that μ> 0 and ν> 0. Region, and (μ, ν) = (− μs / 2, −νs / 2), and a component of a frequency region where μ <0 and ν <0).

  The effect of the diagonal (lower right) resolution conversion is large because (μ, ν) = (+ μs / 2, −νs / 2) as shown in FIG. 31 and (μ, ν) = (− μs). / 2, + νs / 2) in the frequency region (3102) (particularly (μ, ν) = (+ μs / 2, -νs / 2)), and frequencies where μ> 0 and ν <0 And (μ, ν) = (− μs / 2, + νs / 2), and components in a frequency region where μ <0, ν> 0).

  Therefore, when these frequency components (3101) and (3102) are extracted by a two-dimensional filter and mixed together with the frequency components (2501) and (2502) shown in FIG. 25, components having a large resolution improvement effect are selectively output. can do.

  FIG. 32 shows an example of the configuration of a mixer (2707) that extracts components that have a large effect of horizontal, vertical, diagonal (lower right), and diagonal (upper right) resolution conversion. In the figure, a two-dimensional filter (3201) is used to extract a component in the frequency region (3102) having a large resolution improvement effect of SR (lower right) input to the mixer (2707). Similarly, using the two-dimensional filter (3202), the component of the frequency region (3101) having a large resolution improvement effect of SR (upper right) input to the mixer (2707) is extracted. In addition, the two-dimensional filters (2601) and (2602) shown in FIG. 26 extract components in the frequency regions (2501) and (2502) that have a large resolution improvement effect for SR (horizontal) and SR (vertical), respectively. As components other than frequency domain (2501) (2502) (3101) (3102), SR (horizontal), SR (vertical), SR (bottom right), SR using adder (3203) and multiplier (3204) An average signal (upper right) is created, and components other than the passbands of the two-dimensional filters (2601, 2602, 3201, and 3202) are extracted using the two-dimensional filter (3205). The output signals of the two-dimensional filters (2601), (2602), (3201), (3202), and (3205) are added by the adder (3206) to obtain the output of the mixer (2707).

  The numbers in circles in the two-dimensional filters (2601), (2602), (3201), (3202), and (3205) shown in the figure indicate examples of tap coefficients of the respective filters.

  According to the image signal processing apparatus according to the ninth embodiment described above, it is possible to generate a high-resolution image in which the resolution in the oblique direction is increased in addition to the horizontal direction and the vertical direction.

  In addition, according to the prior art described in Patent Document 1, Patent Document 2, and Non-Patent Document 1, one-dimensional (horizontal, vertical, diagonal (lower right), diagonal (upper right)) high resolution using three frames May be performed in a plurality of directions, and each result thereof may be input to the mixer (2707) shown in FIG. 27 and mixed, and output as a two-dimensional resolution conversion result. In this case, as shown in FIG. 27, the scale of the signal processing circuit such as the frame memory and the motion estimation unit is larger than the configuration in which the two-dimensional resolution conversion is performed using only two frames. 2. As described in Non-Patent Document 1, it is possible to reduce the scale of a signal processing circuit such as a frame memory or a motion estimation unit, compared to using a signal of at least 7 frames.

  In addition, the present invention is not limited to the conventional techniques described in Patent Document 1, Patent Document 2, and Non-Patent Document 1, and other one-dimensional (horizontal / vertical / diagonal (lower right)) by applying other conventional high resolution technologies. Even if the resolution (diagonal (upper right)) is increased in a plurality of directions, the respective results are input to the mixer (2707) shown in FIG. 27, mixed, and output as a two-dimensional resolution conversion result. Good.

  Also, in FIG. 27, the case where the resolution of frame # 1 is converted using a set of input signals of frame # 1 and frame # 2 has been described as an example, but other than this, for example, frame # 1 and frame # 1 # 3, Frame # 1 and Frame # 4, etc. are used to convert the resolution of frame # 1 and mix the results to obtain the final resolution conversion result of frame # 1. .

  As a mixing method at this time, an average value of the results may be taken, or mixing may be performed according to the value of the coefficient C4 (frame) for each frame as shown in FIGS. In this case, as the coefficient C4 (frame), the MAX value (the smaller value) of the coefficient C4 (horizontal) and the coefficient C4 (vertical) for each frame may be used. Also, all the coefficients C4 (horizontal) and C4 (vertical) are compared for each pixel, and the resolution conversion result obtained from the group with the smallest coefficient C4 (i.e., the group with the greatest resolution improvement effect) is obtained for each pixel. The final resolution conversion result of frame # 1 may be selected.

  Thus, for example, with frame # 1 as a reference, if frame # 2 is a frame earlier than frame # 1 and frame # 3 is a future frame than frame # 1, the subject is When changing from `` motion '' to `` still '' (end of movement), resolution conversion processing is performed with frame # 1 and frame # 2, and when the subject is converted from `` still '' to `` motion '' at the time of frame # 1 ( (Beginning of motion) Since each processing result is mixed for each pixel so that resolution conversion processing is performed by frame # 1 and frame # 3, the motion of the subject can be used effectively, and the resolution improvement effect is the largest can do.

  An image signal processing method according to Embodiment 10 of the present invention will be described with reference to FIG.

  The tenth embodiment relates to an image signal processing method for realizing a process equivalent to the image signal processing in the image signal processing apparatus according to the ninth embodiment by a control unit that cooperates with software. The image processing apparatus that performs the image signal processing method of the present embodiment is the image processing apparatus shown in FIG.

  FIG. 33 shows an example of a flowchart of the operation of the present embodiment. The flowchart of FIG. 33 starts from step (3301), and in steps (5-1), (5-2), (5-3), and (5-4), horizontal, vertical, diagonal (lower right), diagonal ( Increase the resolution in the upper right). Here, in each step (5-1) (5-2) (5-3) (5-4), the processing step (5) shown in FIG. 14 to FIG. 16 or FIG. 42 to FIG. Any one of the processing steps (5) shown may be executed in the horizontal, vertical, diagonal (lower right), and diagonal (upper right) directions. In other words, the `` frequency-phase '' characteristics such as π / 2 phase shift (1407) (1408) and Hilbert transform (1510) are changed according to the respective directions as shown in FIG. 28, and the phase difference θ is changed to θH , ΘV, (θH + θV), and (−θH + θV). The processing results of steps (5-1), (5-2), (5-3), and (5-4) are written into the respective frame buffers # 3 as described with reference to FIGS. In the following steps (3302-1), (3302-2), (3302-3) and (3302-4), vertical, horizontal, and diagonal pixel interpolation is performed, respectively, to the same number of horizontal and vertical pixels as the output frame. Thus, all the pixels of the two-dimensional frame buffer # 3 are generated. In the subsequent step (3303), the data of each frame buffer # 3 is mixed for each pixel according to the method described with reference to FIGS. 29, 30, and 32, and is output to the output frame buffer # 4. When the operations of the eighth to ninth embodiments are realized by a software program, steps (5-3) and (5-4) for performing oblique processing, and steps for performing pixel interpolation on these results (3302-3) and (3302-4) are not required. In addition, as a mixing method in step (3303), data is mixed according to the method described with reference to FIGS.

  According to the image signal processing method according to the tenth embodiment described above, it is possible to generate a high-resolution image in which the resolution in the oblique direction is increased in addition to the horizontal direction and the vertical direction.

  FIG. 38 shows an image processing apparatus according to Embodiment 11 of the present invention. The image processing apparatus according to the present embodiment uses, for example, an input unit (1) to which a frame sequence of a moving image such as a television broadcast signal is input, and four frames input from the input unit (1). A resolution converter (8) for increasing the resolution twice in the horizontal direction and twice in the vertical direction, and further displaying an image based on the frame that has been increased in resolution by the resolution converter (8). And a display unit (3). This resolution conversion unit (8) performs a phase shift in the horizontal direction, vertical direction, and horizontal / vertical direction for each of the input image signals of the four frames, thereby folding back in the two-dimensional frequency domain. Removes components and achieves two-dimensional high resolution. Details of the resolution conversion unit (8) will be described below.

In FIG. 38, first, the position estimation units (3806-2), (3806-3), and (3806-4) perform two-dimensional sampling positions of pixels to be processed on frame # 1 input from the input unit (1). Using the (sampling position) as a reference, the two-dimensional position of each corresponding image on frame # 2, frame # 3, and frame # 4 is estimated, and the horizontal phase difference θH2 (3807-2), θH3 ( 3807-3), θH4 (3807-4), and vertical phase differences θV2 (3808-2), θV3 (3808-3), and θV4 (3808-4). Next, the phase difference θH2 (3807-2) is obtained by the horizontal / vertical uprater (3801-1) (3801-2) (3801-3) (3801-4) of the motion compensation / uprate unit (3810). , ΘH3 (3807-3), θH4 (3807-4), θV2 (3808-2), θV3 (3808-3), θV4 (3808-4) information, frame # 2, frame # 3, The frame # 4 is compensated for motion and aligned with the frame # 1, and the number of pixels in each frame is doubled horizontally and doubled, respectively, for a total density of 4 times. In the phase shift unit (3811), the phase of this high-density data is converted into horizontal phase shifters (3803-1) (3803-2) (3803-3) (3803-4), vertical phase shifters (3804- 1) Horizontal direction using (3804-2) (3804-3) (3804-4), horizontal / vertical phase shifter (3805-1) (3805-2) (3805-3) (3805-4) Shift by a fixed amount in the vertical, horizontal and vertical directions. Here, a π / 2 phase shifter such as the above-mentioned Hilbert transformer can be used as means for shifting the phase of data by a certain amount. The aliasing component removal unit (3809) uses a total of 16 signals from the above-described phase shift unit (3811) and a total of 6 phase difference signals from the phase estimation unit (3812), respectively, horizontally and vertically. The output is obtained by removing the aliasing component in the direction of. This output is supplied to the display unit 3. Note that the position estimation units (3806-2), (3806-3), and (3806-4) can be realized using the above-described conventional technique as it is. The horizontal / vertical up-rater (3801-1) (3801-2) (3801-3) (3801-4) extends the operation and configuration shown in Figs. 5 and 6 to two dimensions in the horizontal and vertical directions. Is. Details of the phase shift unit (3811) and the aliasing component removal unit (3809) will be described later.

  FIG. 39 shows a configuration example of the horizontal / vertical phase shifters (3805-1), (3805-2), (3805-3), and (3805-4). Since the horizontal and vertical phases of the image signal are independent of each other, the horizontal / vertical phase shifter (3805) has the vertical phase shifter (3804) and horizontal phase shifter (3803) as shown in the figure. Can be realized by combining in series. Also, it is clear that the same operation can be achieved even if the connection order is reversed and the horizontal phase shifter (3803) is arranged in front of the vertical phase shifter (3804).

FIG. 40 shows detailed operations of the phase shift unit (3811) and the aliasing component removal unit (3809) described above. FIG. 4A shows a two-dimensional frequency region in which the horizontal frequency is μ and the vertical frequency is ν. When the horizontal sampling frequency of the original input image is μs and the vertical sampling frequency is νs, the signal near the origin (i.e., (μ, ν) = (0, 0)) in FIG. It is well known that a folding component is generated at the position of (μ, ν) = (μs, 0), (μ, ν) = (0, νs), (μ, ν) = (μs, νs). Note that these origin symmetrical positions (i.e., (μ, ν) = (-μs, 0), (μ, ν) = (0, -νs), (μ, ν) = (-μs, -νs) ) Also produces aliasing components, but these are due to the symmetry of the frequency (μ, ν) = (μs, 0), (μ, ν) = (0, νs), (μ, ν) = (μs, It is equivalent to the folded component at the position of νs). In the resolution conversion unit (8) shown in FIG. 38, in order to increase the resolution twice in the horizontal direction and twice in the vertical direction, the motion compensation / uprate unit (3810) can perform horizontal and vertical After double the up rate (0 insertion) in each direction to quadruple the number of pixels, (μ, ν) = (μs, 0), (μ, ν) shown in FIG. ) = (0, νs), (μ, ν) = (μs, νs) may be removed. The operation will be described below.

  In FIG. 40 (b), (μ, ν) = (0, 0), (μ, ν) = (μs, 0), (μ, ν) = (0, νs), (μ, ν) = ( The state of horizontal phase rotation and vertical phase rotation of each component at the position of μs, νs) is shown. As shown in FIG. 4, the phase rotation of the original component does not occur between a plurality of frames having different sampling phases, and only the aliasing component rotates in phase according to the sampling phase difference. Therefore, using the phase of the original component as a reference (Re axis), it is considered that the phase shift unit (3811) shown in FIG. 38 generates components of the phase orthogonal axis (Im axis) in the horizontal, vertical, horizontal and vertical directions. 40 (b), the original component (μ, ν) = (0, 0) (that is, # 1) horizontal Re axis (= no horizontal phase rotation) and vertical Re axis Only set the value of the component (no phase rotation in the vertical direction) (the total value of the signals after each phase shift) to `` 1 '' and the values of the other components (i.e., # 2 to # 16) to `` 0 ''. For example, the folded component can be canceled and only the original component can be extracted.

  FIG. 40 (c) shows a matrix arithmetic expression for realizing the phase relationship shown in FIG. 40 (b). In the figure, M is a matrix having 16 × 16 elements, and is an operation indicating horizontal, vertical, horizontal / vertical phase rotation. Details of the matrix M will be described later. In addition, the left side of the figure shows the “value” in FIG. 40B, and C1ReRe to C4ImIm on the right side are output to the output signals of the phase shift unit (3811) by the aliasing component removal unit (3809) shown in FIG. Indicates the coefficient to be multiplied. That is, for frame # 1 shown in FIG. 38, the output signal of the delay unit (3802-1) is multiplied by the coefficient C1ReRe, the output signal of the horizontal phase shifter (3803-1) is multiplied by the coefficient C1ImRe, and the vertical phase shifter The output signal of (3804-1) is multiplied by a coefficient C1ReIm, and the output signal of the horizontal / vertical phase shifter (3805-1) is multiplied by a coefficient C1ImIm. Similarly, for frame # 2, the output signal of the delay unit (3802-2) is multiplied by the coefficient C2ReRe, the output signal of the horizontal phase shifter (3803-2) is multiplied by the coefficient C2ImRe, and the vertical phase shifter (3804 The output signal of -2) is multiplied by the coefficient C2ReIm, and the output signal of the horizontal / vertical phase shifter (3805-2) is multiplied by the coefficient C2ImIm. For frame # 3, the output signal of the delay unit (3802-3) is multiplied by the coefficient C1ReRe, the output signal of the horizontal phase shifter (3803-3) is multiplied by the coefficient C3ImRe, and the vertical phase shifter (3804-3) The output signal is multiplied by a coefficient C3ReIm, and the output signal of the horizontal / vertical phase shifter (3805-3) is multiplied by a coefficient C3ImIm. For frame # 4, the output signal of the delay unit (3802-4) is multiplied by the coefficient C4ReRe, the output signal of the horizontal phase shifter (3803-4) is multiplied by the coefficient C4ImRe, and the vertical phase shifter (3804-4) The output signal is multiplied by a coefficient C4ReIm, and the output signal of the horizontal / vertical phase shifter (3805-4) is multiplied by a coefficient C4ImIm. If the above-described coefficients C1ReRe to C4ImIm are determined so that the relationship shown in FIG. 40 (c) always holds when all the 16 signals multiplied by the above coefficients are added by the aliasing component removal unit (3809) described later. It is possible to cancel the folded component and extract only the original component.

  FIG. 40 (d) shows details of the matrix M. The matrix M is a matrix having 16 × 16 elements as described above, and is represented by mij (where row number i and column number j are integers satisfying 1 ≦ i ≦ 4 and 1 ≦ j ≦ 4). It consists of a partial matrix with 4x4 elements. This partial matrix mij is classified as shown in (e), (f), (g), and (h) in FIG.

  FIG. 40 (e) shows each element of the partial matrix m1j (that is, m11, m12, m13, m14) when the row number i = 1. This partial matrix m1j is an element acting on the component of the frequency (μ, ν) = (0, 0), and no horizontal / vertical phase rotation occurs regardless of the sampling phase difference between frames. That is, a matrix in which all the elements on the diagonal to the lower right are 1 and the remaining elements are all 0).

  FIG. 40 (f) shows each element of the partial matrix m2j (that is, m21, m22, m23, m24) when the row number i = 2. This partial matrix m2j is an element acting on the component of (μ, ν) = (μs, 0), and according to the horizontal phase difference θHj of sampling (where j is an integer satisfying 1 ≦ j ≦ 4), This is a rotation matrix that rotates the phase in the horizontal direction. That is, a rotation matrix is obtained by rotating the phase by θHj around the horizontal frequency axis with # 5 and # 6 and # 7 and # 8 shown in FIG. 40 (b) having a common vertical phase axis as a pair. Note that the horizontal phase difference θH1 when j = 1 is not shown in FIG. 38, but this is the phase difference between the frame # 1 (reference) and the frame # 1 (processing target = same as the reference) (= 0) and treated as θH1 = 0. Hereinafter, the vertical phase difference θV1 is similarly treated as θV1 = 0.

  FIG. 40 (g) shows each element of the partial matrix m3j (that is, m31, m32, m33, m34) when the row number i = 3. This partial matrix m3j is an element acting on the component of (μ, ν) = (0, νs), and according to the sampling vertical phase difference θVj (where j is an integer satisfying 1 ≦ j ≦ 4), This is a rotation matrix that rotates the phase in the vertical direction. That is, a rotation matrix is obtained by rotating the phase by θVj around the vertical frequency axis with # 9 and # 11 and # 10 and # 12 shown in FIG.

  FIG. 40 (h) shows each element of the partial matrix m4j (that is, m41, m42, m43, m44) when the row number i = 4. This partial matrix m4j is an element acting on the component of (μ, ν) = (μs, νs), and the sampling horizontal phase difference θHj and vertical phase difference θVj (where j is an integer satisfying 1 ≦ j ≦ 4) ), The rotation matrix rotates the phase in both the horizontal and vertical directions. That is, it is the product of m2j and m3j.

  From another viewpoint, m1j, m2j, and m3j are also rotation matrices that rotate the phase in both the horizontal and vertical directions as in m4j. For m1j, θHj = θVj = 0, for m2j, θVj = 0 In the case of m3j, even if it is assumed that θHj = 0, the partial matrix is the same as described above.

In this way, the matrix M is determined based on each sampling phase difference (θHj, θVj), and a total of 16 coefficients (C1ReRe to C4ImIm) are determined so that the equation shown in FIG. 40 (c) always holds. To do. At this time, an inverse matrix M ⁻¹ with respect to the matrix M is obtained in advance, and the coefficients (C1ReRe to C4ImIm) may be determined by the calculation shown in FIG. As a method for obtaining the inverse matrix M ⁻¹ , a method using a cofactor matrix, a method using a Gauss-Jordan sweep-out method, a method of calculating by dividing into a triangular matrix, etc. are well known. .

  FIG. 41 shows a detailed configuration example of the aliasing component removal unit (3809) shown in FIG. In the figure, the coefficient determination unit (4101) is based on the horizontal phase differences (θH2, θH3, θH4) and vertical phase differences (θV2, θV3, θV4) output from the position estimation unit (3812) shown in FIG. Each coefficient (C1ReRe to C4ImIm) is generated by the inverse matrix operation shown in FIG. These coefficients are multiplied by the signal of each frame output from the phase shift unit (3811) by the multiplier (4102), fully added by the adder (4103), and output from the aliasing component removal unit (3809). (That is, the output signal of the resolution converter (8)). The horizontal phase difference (θH2, θH3, θH4) and vertical phase difference (θV2, θV3, θV4) are generally different for each pixel on the input frame, so the above inverse matrix calculation is performed for each pixel. There is a need to do. At this time, the horizontal phase difference (θH2, θH3, θH4) and the vertical phase difference (θV2, θV3, θV4) are representative phase differences (for example, an integral multiple of π / 8 as shown in FIG. 9D). ), The coefficients (C1ReRe to C4ImIm) may be generated in advance and tabulated using a ROM (Read Only Memory) or the like. Since this is well known as a general table reference method, illustration is omitted.

FIG. 42 shows another configuration example of the aliasing component removal unit (3809) shown in FIG. In the above description, when determining a total of 16 coefficients (C1ReRe to C4ImIm) so that the equation shown in FIG. 40 (c) always holds, an inverse matrix M ⁻¹ for the matrix M is obtained in advance. The coefficients (C1ReRe to C4ImIm) are to be determined by the calculation shown in FIG. 40 (i), but depending on the values of the horizontal phase differences (θH2, θH3, θH4) and vertical phase differences (θV2, θV3, θV4), the inverse matrix M ^In some cases, ⁻¹ does not exist and the coefficients (C1ReRe to C4ImIm) cannot be determined. Whether the inverse matrix M ⁻¹ exists is determined by calculating the inverse matrix M ⁻¹ in the coefficient determination unit (4101), using a cofactor matrix, using a Gauss-Jordan sweep-out method, and a triangular matrix If the inverse matrix M ⁻¹ does not exist, it can be easily determined in the calculation process such as the calculation method, and the frame # 1 and the frame # are processed by the resolution conversion unit (4) shown in FIG. The output signal may be switched so that the output signal is obtained using 2. That is, the horizontal folding component removal unit (2108), the vertical folding component removal unit (2109), the pixel interpolator (2003) (2007), and the mixer (2009) shown in FIG. 3811) and the horizontal phase difference θH2 (3807-2) and vertical phase difference θV2 (3808-2) output from the position estimation unit. And the result of the adder (4103) described above using the switch (4201) to switch to the output signal. Instead of binary switching using the switch (4201), the output of the adder (4103) and the output of the mixer (2009) are continuously mixed (that is, weighted addition). For example, in the vicinity of a pixel where the inverse matrix M ⁻¹ does not exist, the mixing ratio of the output of the mixer (2009) may be increased.

  With the aliasing component removal processing described above, in the two-dimensional frequency region shown in FIG. 40A, the effect of improving the resolution can be achieved from the center to (μ, ν) = (μs, 0) in the horizontal direction. In the vertical direction, the effect of improving the resolution can be achieved from the center to (μ, ν) = (0, νs). In the oblique direction, the effect of improving the resolution can be achieved from the center to (μ, ν) = (μs, νs).

  Here, also in the image signal processing apparatus and the image signal processing method according to the seventh embodiment, the resolution is increased in the oblique direction in addition to the horizontal direction and the vertical direction. As shown in FIG. 31, it does not reach (μ, ν) = (μs, νs).

  Therefore, the image signal processing device shown in FIG. 38 has an effect that the resolution can be improved to a high frequency component in an oblique direction as compared with the image signal processing device according to the seventh embodiment.

  Next, the difference in operation between the image signal processing apparatus according to the eleventh embodiment of the present invention and the prior art will be described with reference to FIG. The figure (a) shows frame # 1 (4401), frame # 2 (4402), frame # 3 (4403), frame # 4 (4404), frame # 4 input to the resolution converter (8) shown in FIG. A frame # 5 (4405) is shown, and FIG. 5B shows each frame output from the resolution conversion unit (8). In each frame, the subject moves clockwise by 1/4 pixel, and the subject is intentionally moved so as to make one round in four frames. This movement is continued in the same manner from frame # 5.

In the conventional techniques described in Patent Document 1, Patent Document 2, and Non-Patent Document 1, as described above, when the resolution is increased with respect to a horizontal / vertical two-dimensional input signal, folding is performed from two vertical and horizontal directions. Therefore, if the bandwidth of the original signal doubles both vertically and horizontally, the three aliasing components overlap, and 2M + 1 = 7 digital data (= 7 frame image signals) is required to cancel them. I was trying. Therefore, when a signal that circulates in 4 frames as shown in FIG. 34 (a) is input, independent data cannot be obtained no matter which 7 frames are selected, and the solution by the high resolution processing becomes indefinite. It is not required.

  On the other hand, according to the eleventh embodiment, for example, four adjacent frames (frame # 1 (4401) and frame # 2 (4402), frame # 3 (4403), frame # 4 (4404)) are used as shown in FIG. As shown in b), it is possible to realize high resolution by removing aliasing components in the horizontal direction, vertical direction, and horizontal / vertical direction. That is, the operation status of the present embodiment can be confirmed by using the input image of FIG. If a generally well-known circular zone plate (CZP: Circular Zone Plate) is used as the pattern of this test pattern, the effect of resolution conversion can be directly viewed on the display unit (3). That is, if the circular zone plate is moved so as to make one round in four frames as shown in FIG. 44 (a), an image with improved horizontal resolution and vertical resolution is always displayed. The effect can be confirmed.

  As described above, the image signal processing apparatus according to the eleventh embodiment has a plurality of types of phase shifts (horizontal direction, vertical direction, horizontal / vertical direction) with respect to each image signal of four input image frames. Is performed to generate four signals from each of the image signals. As a result, 16 signals are generated from the image signals of the four input image frames. Here, based on the phase difference between the four input image frames, for each of the 16 signals, a coefficient for canceling the aliasing component of the 16 signals and combining them is calculated for each pixel. For each pixel of the generated image, the sum of the corresponding pixel value of each of the 16 signals multiplied by each coefficient is calculated to generate a new pixel value of the high-resolution image.

  Thus, the image signal processing apparatus according to the eleventh embodiment can generate a high-resolution image in which the diagonal components in the lower right direction and the upper right direction are increased in addition to the horizontal direction and the vertical direction.

  Further, the effect of improving the resolution by the image signal processing apparatus according to the eleventh embodiment is that the resolution can be improved to a higher frequency component in the oblique direction than the image signal processing apparatus according to the seventh embodiment, and a higher-resolution high-resolution image can be obtained. Can be generated.

  An image signal processing method according to Embodiment 12 of the present invention will be described with reference to FIGS.

  The twelfth embodiment relates to an image signal processing method in which processing equivalent to the image signal processing in the image signal processing apparatus according to the eleventh embodiment is realized by a control unit that cooperates with software.

  Here, an image processing apparatus for realizing the image signal processing method according to the present embodiment will be described with reference to FIG. The image signal processing apparatus shown in FIG. 19 stores, for example, an input unit (1) to which an image signal such as a television broadcast signal is input, and software for processing the signal input from the input unit (1). A control unit (10) that performs image signal processing on a signal input from the input unit (1) in cooperation with software stored in the unit (11), the storage unit (11), and a control unit (10) Output from the frame buffer # 1 (31), frame buffer # 2 (32), frame buffer # 3 (33), frame buffer # 4 (34), and the control unit (10) used as a data buffer in the image signal processing A buffer # 5 (35) for frame-buffering the image signal processed signal output to the unit (3).

  Here, the number of input units (1) included in the image signal processing apparatus shown in FIG. 19 is the same as four, which is the number of frames used for image processing. One frame may be input continuously.

  In addition, the frame buffer # 1 (31), the frame buffer # 2 (32), the frame buffer # 3 (33), the frame buffer # 4 (34), and the storage unit (11) for storing software are used for the data buffer. Each may be configured using individual memory chips, or one or a plurality of memory chips may be used to divide and use each data address.

  In this embodiment, for the image signal input from the input unit (1), the control unit (10) performs image signal processing in cooperation with software stored in the storage unit (11), and the display unit (3) Output to. Details of the image signal processing will be described with reference to FIG.

  The flowchart in FIG. 43 starts from step (4301), and in steps (4302-1), (4302-2), (4302-3), and (4302-4), the image data of each frame is doubled both horizontally and vertically. Update to That is, the image data of frame # 1 is updated at step (4302-1) and written to frame buffer # 1, and the image data of frame # 2 is updated at step (4302-2) to frame buffer # 1. 2 is written, the image data of frame # 3 is updated at step (4302-3) and written to frame buffer # 3, and the image data of frame # 4 is updated at step (4302-4) to frame. Write to buffer # 4. Here, the up-rate can be realized by writing the data every other horizontal pixel and every other vertical pixel after once clearing the value of each frame buffer to 0.

Next, in step (4303), the first pixel (for example, the upper left pixel) of frame buffer # 1 is set as a processing target, and processing is performed until processing of all pixel data for frame buffer # 1 is completed. Loop.

  In step (4304-2), the position of the corresponding pixel in the frame buffer # 2 is estimated based on the target pixel in the frame buffer # 1, and the horizontal phase difference θH2 and the vertical phase difference θV2 are output. Similarly, in step (4304-3), the position of the corresponding pixel in the frame buffer # 3 is estimated based on the target pixel in the frame buffer # 1, and the horizontal phase difference θH3 and the vertical phase difference θV3 are output. . In step (4304-4), the position of the corresponding pixel in the frame buffer # 4 is estimated based on the target pixel in the frame buffer # 1, and the horizontal phase difference θH4 and the vertical phase difference θV4 are output. At this time, the above-described conventional technique can be used as it is as a method of estimating the position of the corresponding pixel.

  In step (4305-2), based on the horizontal phase difference θH2 and vertical phase difference θV2 obtained in step (4304-2), pixels near the corresponding pixel in the frame buffer # 2 are motion compensated. In this motion compensation operation, the operation described with reference to FIGS. 5 and 6 may be performed in the same manner for each of the horizontal direction and the vertical direction. Similarly, in step (4305-3), based on the horizontal phase difference θH3 and vertical phase difference θV3 obtained in step (4304-3), a pixel in the vicinity of the corresponding pixel in frame buffer # 3 is moved. To compensate. In step (4305-4), based on the horizontal phase difference θH4 and vertical phase difference θV4 obtained in step (4304-4), motion compensation is performed on pixels near the corresponding pixel in frame buffer # 4. To do.

  Subsequently, in step (4313), steps (4306-1), (4306-2), (4306-) are applied to frame buffer # 1, frame buffer # 2, frame buffer # 3, and frame buffer # 4 subjected to motion compensation. 3) The horizontal phase is shifted by a certain amount by (4306-4), and the vertical phase is shifted by a certain amount by steps (4307-1), (4307-2), (4307-3) and (4307-4). Further, for the results of steps (4307-1) (4307-2) (4307-3) (4307-4), further steps (4308-1) (4308-2) (4308-3) (4308-4) ) Shifts the horizontal phase by a certain amount, thereby shifting both the horizontal and vertical phases by a certain amount. That is, the pixel data in each frame buffer is phase shifted by π / 2 in the horizontal direction and the vertical direction.

  Subsequently, in step (4309), a total of 16 coefficients (C1ReRe to C1ReRe through the method shown in FIG. 40 are used based on the horizontal phase differences (θH2, θH3, θH4) and the vertical phase differences (θV2, θV3, θV4). C4ImIm) and multiplying each output of step (4313) by the above coefficients and adding (weighted addition), frame buffer # 1, frame buffer # 2, frame buffer # 3, frame buffer # 4 The aliasing component is removed from the pixel data and output to frame buffer # 5. The operation of removing the aliasing component is the same as the operation described with reference to FIG. 41 or FIG.

Subsequently, in step (4310), it is determined whether or not the processing of all the pixels in the frame buffer # 1 has been completed.If the processing has not been completed, the next pixel (for example, the right adjacent pixel) is determined in step (4311). The process is set as a process target, and the process returns to steps (4304-2), (4304-3), and (4304-4). If completed, the process ends in step (4312).

  By performing the processing as described above, using the pixel data of frame buffer # 1, frame buffer # 2, frame buffer # 3, and frame buffer # 4, a high-resolution signal is output to frame buffer # 5. Can do. When applied to a moving image, the processing from step (4301) to step (4312) may be repeated for each frame.

In FIG. 38, FIG. 41, FIG. 42, and FIG. 43, the number of input frames has been described as four. However, the present invention is not limited to this, and n frames (where n is 4 or more). (Integer integer) may be input, and four frames suitable for the resolution conversion process described above may be selected and used. For example, when outputting the inverse matrix operation shown in FIG. 40 (i), four frames used for resolution conversion processing from n frames so that the number of pixels in which the inverse matrix M ⁻¹ does not exist is minimized. This frame may be selected and switched for each pixel or for each region composed of a plurality of pixels.

  Therefore, the image signal processing method according to the twelfth embodiment has an effect that the resolution can be improved to a higher frequency component in the oblique direction than the image signal processing method according to the tenth embodiment. Details of the effect are the same as the effect of the image signal processing apparatus shown in FIG.

  As described above, the image signal processing method according to the twelfth embodiment has a plurality of types of phase shifts (horizontal direction, vertical direction, horizontal / vertical direction) with different directions for each image signal of four input image frames. Is performed to generate four signals from each of the image signals. As a result, 16 signals are generated from the image signals of the four input image frames. Here, based on the phase difference between the four input image frames, for each of the 16 signals, a coefficient for canceling the aliasing component of the 16 signals and combining them is calculated for each pixel. For each pixel of the generated image, the sum of the corresponding pixel value of each of the 16 signals multiplied by each coefficient is calculated to generate a new pixel value of the high-resolution image.

  Accordingly, the image signal processing method according to the twelfth embodiment can generate a high-resolution image in which the diagonal components in the lower right direction and the upper right direction are increased in addition to the horizontal direction and the vertical direction.

  Further, the resolution improvement effect by the image signal processing method according to the twelfth embodiment can improve the resolution to a high frequency component in an oblique direction as compared with the image signal processing method according to the tenth embodiment, and generates a high-resolution image with higher image quality. be able to.

  In the image signal processing apparatus or the image signal processing method according to the first to twelfth embodiments described above, the case where the number of pixels is doubled while improving the resolution of the image has been described as an example. For example, the number of pixels can be increased to a power of 2 (= 2 times, 4 times, 8 times, etc.) by operating the image signal processing apparatus or the image signal processing method multiple times or in multiple stages. is there. In other words, by performing signal processing using two input image frames, the number of pixels is doubled to form an intermediate image frame, and then, two additional intermediate image frames are used to create a new input image frame. As a result, an output image frame in which the number of pixels is further doubled can be obtained. In this case, an output image frame having four times the number of pixels can be obtained as compared with the input image frame. Similarly, if the signal processing is repeated a total of three times, the number of pixels of the output image frame is eight times that of the input image frame. At this time, the number of input image frames necessary to obtain one output image frame is also a power of 2 (= 2, 4, 8,...).

  The final output image is output with the number of pixels other than the power of 2 (= 2 times, 4 times, 8 times, etc.) by performing a general resolution conversion process after the image processing. It is also possible.

  FIG. 35 shows an image display apparatus according to Embodiment 13 of the present invention. The image display apparatus according to the present embodiment is an image display apparatus configured to perform the image signal processing described in any one of the seventh embodiment and the eighth embodiment.

  In the figure, an image display device 3500 includes, for example, an input unit 3501 for inputting a broadcast signal, video content, image content, and the like via a broadcast wave including a television signal, a network, and the like, and an input unit 3501 A recording / playback unit 3502 for recording or playing back content, a content storage unit 3503 for recording content by the recording / playback unit 3502, and a video signal or an image signal played back by the recording / playback unit 3502 are either of Embodiment 7 or Embodiment 8. An image signal processing unit 3504 that is an image signal processing device described in one embodiment, a display unit 3505 that displays a video signal or an image signal processed by the image signal processing unit 3504, and a recording / playback unit 3502 for playback An audio output unit 3506 that outputs the audio signal, a control unit 3507 that controls each component of the image display device 3500, and a user interface that allows the user to operate the image display device 3500. It provided with a such as the face portion 3508.

  The detailed operation of the image signal processing unit 3504 is as described in the seventh embodiment or the eighth embodiment, and thus the description thereof is omitted.

  Since the image display device 3500 includes the image signal processing unit 3504 that is the image signal processing device described in any one of the seventh embodiment and the eighth embodiment, the video signal or the image input to the input unit 3501 The signal can be displayed on the display portion 3505 as a video signal or image signal with higher resolution and higher image quality. Therefore, even when a signal having a resolution lower than that of the display device of the display unit 3505 is input from the input unit 3501, it is possible to perform high-quality and high-definition display while increasing the resolution of the reproduction signal.

  In addition, when playing back video content or image content stored in the content storage unit 3503, it can be converted into a video signal or image signal with higher resolution and higher image quality and displayed on the display unit 3505.

  In addition, by performing image processing of the image signal processing unit 3504 after reproduction of video content or image content stored in the content storage unit 3503, data stored in the content storage unit 3503 is displayed at a resolution displayed on the display unit 3505. Relatively lower resolution than Therefore, there is an effect that the content data amount can be relatively reduced and stored.

  Further, the image signal processing unit 3504 may be included in the recording / playback unit 3502, and the above-described image signal processing may be performed during recording. In this case, since it is not necessary to perform the above-described image signal processing during reproduction, the processing load during reproduction can be reduced.

  Here, it has been described that the above-described image signal processing is performed by the image signal processing unit 3504, but may be realized by a control unit 3507 and software. In this case, the image signal processing may be performed by the method described in any one of the seventh embodiment and the eighth embodiment.

  In this embodiment, the recording / playback unit 3502 may perform encoding according to the state of content such as video input from the input unit 3501 during recording, and then record it in the content storage unit 3503.

  In the present embodiment, the recording / reproducing unit 3502 may perform decoding and decoding if content such as video input from the input unit 3501 is encoded during recording.

  In the image display apparatus according to the present embodiment, the content storage unit 3503 is not necessarily required. In this case, the recording / playback unit 3503 may perform playback of content such as video input from the input unit 3501 without recording.

  Also in this case, the video signal or image signal input to the input unit 3501 can be displayed on the display unit 3505 as a video signal or image signal with higher resolution and higher image quality.

  The image display apparatus 3500 may be, for example, a plasma television, a liquid crystal television, a cathode ray tube, a projector, or an apparatus using other devices. Similarly, the display unit 3505 may be, for example, a plasma panel module, an LCD module, or a projector device. The content storage unit 3503 may be, for example, a hard disk drive, a flash memory, or a removable media disk drive. The audio output unit 3506 may be a speaker, for example. The input unit 3501 may be provided with a tuner that receives broadcast waves, may be provided with a LAN connector for connecting to a network, or may be provided with a USB connector. Furthermore, a terminal that digitally inputs a video signal or an audio signal may be provided, or an analog input terminal such as a composite terminal or a component terminal may be provided. Further, it may be a receiver that transfers data wirelessly.

  According to the image display apparatus according to the thirteenth embodiment described above, each image signal of the two input image frames included in the input video signal or the input image signal is phase-shifted, and each image signal is detected. Two signals are generated. As a result, four signals are generated from the image signals of the two input image frames. Here, based on the phase difference between the two input image frames, for each of the four signals, a coefficient for canceling and combining the aliasing components of the four signals is calculated for each pixel. For each pixel of the image to be generated, a sum is calculated by multiplying the pixel value of the corresponding pixel included in each of the four signals by each coefficient to generate a pixel value of a new high-resolution image. By performing this for each pixel of the generated image, an image having a higher resolution in the one-dimensional direction than the input image frame is generated.

  This is performed in each of the horizontal direction and the vertical direction, and an image with high resolution in the horizontal direction and an image with high resolution in the vertical direction are generated. The image having the high resolution in the horizontal direction and the image having the high resolution in the vertical direction are subjected to up-rate processing in the vertical direction and the horizontal direction, respectively, and then both are mixed.

  As a result, a high-resolution image with high resolution in both the vertical direction and the horizontal direction can be generated from each image signal of two input image frames included in the input video signal or the input image signal. That is, a two-dimensional high resolution image can be generated and displayed on the display unit.

  In addition, since the image display apparatus according to the thirteenth embodiment uses two input image frames, high-resolution display can be realized with a small image processing amount. Accordingly, it is possible to realize an image display apparatus that displays a high-resolution video or image on the display unit in both the vertical direction and the horizontal direction with few aliasing components.

  An image display device according to Embodiment 14 of the present invention is the same as the image display device according to Embodiment 13, except that the image signal processing unit 3504 shown in FIG. 35 is replaced with the image signal processing device described in Embodiment 9. . Since other configurations are the same as those of the image display apparatus according to the thirteenth embodiment, description thereof is omitted.

  Further, the detailed operation of the image signal processing unit 3504 is as described in the ninth embodiment, and thus the description thereof is omitted.

  With the image display device according to the fourteenth embodiment, the input video signal or two input image frames included in the input image signal are used, and the resolution is higher in the horizontal, vertical, and diagonal directions than the input video or input image. It is possible to generate a high resolution image. Further, it is possible to realize an image display device that displays this on the display unit.

  An image display device according to Embodiment 15 of the present invention is obtained by replacing the image signal processing unit 3504 shown in FIG. 35 with the image signal processing device described in Embodiment 11 in the image display device according to Embodiment 13. . Since other configurations are the same as those of the image display apparatus according to the thirteenth embodiment, description thereof is omitted.

  The detailed operation of the image signal processing unit 3504 is the same as that described in the eleventh embodiment, and a description thereof will be omitted.

  With the image display device according to the fifteenth embodiment, the input video signal or the four input image frames included in the input image signal are used, and the resolution is higher in the horizontal, vertical, and diagonal directions than the input video or input image. An image display device that generates a high-resolution image and displays it on the display unit can be realized.

  Further, the resolution improvement effect by the image display device according to the fifteenth embodiment is possible to improve the resolution to a high frequency component in an oblique direction as compared with the image display device according to the fourteenth embodiment, and display a higher resolution image with higher image quality. Can do.

  The image display device according to the sixteenth embodiment of the present invention is the same as the image display device according to the thirteenth embodiment, except that the image signal processing unit 3504 shown in FIG. 35 is one of the first, third, or fifth embodiments. The image signal processing apparatus described in the embodiment is replaced. Since other configurations are the same as those of the image display apparatus according to the thirteenth embodiment, description thereof is omitted.

  The detailed operation of the image signal processing unit 3504 is as described in the first embodiment, the third embodiment, or the fifth embodiment, and thus the description thereof is omitted.

  With the image display device according to the sixteenth embodiment, a high resolution in which the input video signal or two input image frames included in the input image signal are used and the resolution is increased in a one-dimensional direction as compared to the input video or the input image. An image display device that generates an image and displays the image on a display unit can be realized.

  FIG. 36 shows a recording / playback apparatus according to Embodiment 17 of the present invention. The recording / reproducing apparatus according to the present embodiment is a recording / reproducing apparatus configured to perform the image signal processing described in any one of the seventh embodiment and the eighth embodiment.

  In the figure, for example, a recording / playback apparatus 3600 includes, for example, an input unit 3501 for inputting a broadcast signal, video content, image content, and the like via a broadcast wave including a television signal, a network, and the like, and an input from the input unit 3501 A recording / playback unit 3502 for recording or playing back the recorded content, a content storage unit 3503 for recording the content by the recording / playback unit 3502, and a video signal or an image signal played back by the recording / playback unit 3502 according to the seventh or eighth embodiment. An image signal processing unit 3504 that is the image signal processing device described in any one of the embodiments, and an image video output unit that outputs the video signal or image signal processed by the image signal processing unit 3504 to another device or the like 3605, an audio output unit 3606 for outputting the audio signal reproduced by the recording / reproducing unit 3502 to other devices, a control unit 3507 for controlling each component of the recording / reproducing device 3600, a user Over comprises a like user interface unit 3508 for operating the recording and reproducing apparatus 3600.

  Since the recording / reproducing apparatus 3600 includes the image signal processing unit 3504 that is the image signal processing apparatus described in any one of the seventh and eighth embodiments, the video signal or image input to the input unit 3501 The signal can be output to another device or the like as a video signal or image signal with higher resolution and higher image quality. Therefore, it is possible to suitably realize a high-quality and high-resolution signal conversion apparatus that converts a low-resolution video signal or image signal into a high-quality and high-definition video signal or image signal while increasing the resolution.

  Also, when playing back video content or image content stored in the content storage unit 3503, it can be converted into a higher-resolution and higher-quality video signal or image signal and output to another device or the like.

  Therefore, a recording / playback apparatus that inputs and stores a low-resolution video signal or image signal, converts it into a high-quality and high-definition video signal or image signal while outputting a high resolution during playback / output, and outputs the video signal is suitable. realizable.

  In addition, by performing image processing of the image signal processing unit 3504 after reproduction of video content or image content stored in the content storage unit 3503, data stored in the content storage unit 3503 is a signal output to another device. The resolution is relatively lower than the resolution. Therefore, there is an effect that the content data amount can be relatively reduced and stored.

  Further, the image signal processing unit 3504 may be included in the recording / playback unit 3502, and the above-described image signal processing may be performed during recording. In this case, since it is not necessary to perform the above-described image signal processing during reproduction, the processing load during reproduction can be reduced.

  Here, it has been described that the above-described image signal processing is performed by the image signal processing unit 3504, but may be realized by a control unit 3507 and software. In this case, the image signal processing may be performed by the method described in any one of the seventh embodiment and the eighth embodiment.

  In this embodiment, the recording / playback unit 3502 may perform encoding according to the state of content such as video input from the input unit 3501 during recording, and then record it in the content storage unit 3503.

  In the present embodiment, the recording / reproducing unit 3502 may perform decoding and decoding if content such as video input from the input unit 3501 is encoded during recording.

  Further, the image / video output unit 3605 and the audio output unit 3606 according to the present embodiment may be integrated. In this case, it is possible to use a connector shape that outputs a video signal and an audio signal with a single cable.

  Further, the recording / playback apparatus 3600 may be, for example, an HDD recorder, a DVD recorder, or an apparatus using another storage device. Similarly, the content storage unit 3503 may be, for example, a hard disk drive, a flash memory, or a removable media disk drive.

  The input unit 3501 may be provided with a tuner that receives broadcast waves, may be provided with a LAN connector for connecting to a network, or may be provided with a USB connector. Furthermore, a terminal that digitally inputs a video signal or an audio signal may be provided, or an analog input terminal such as a composite terminal or a component terminal may be provided. A receiving unit that wirelessly transfers data may also be used.

  Further, the image / video output unit 3605 may be provided with a terminal for digitally outputting a video signal, or may be provided with an analog output terminal such as a composite terminal or a component terminal. In addition, a LAN connector for connecting to a network or a USB connector may be used. Further, it may be a transmission unit that transfers data wirelessly. The audio output unit 3606 is the same as the image video output unit 3605.

  Furthermore, the input unit 3501 may include, for example, an imaging optical system and a light receiving element. In this case, the recording / playback apparatus 3600 can be applied to, for example, a digital camera, a video camera, a surveillance camera (surveillance camera system), and the like. At this time, for example, the input unit 3501 captures an image of the object to be photographed on the light receiving element by the imaging optical system, generates image data or video data based on the signal output from the light receiving element, and outputs the image data or video data to the recording / playback unit 3502. That's fine.

  If the recording / playback apparatus 3600 is a digital camera, for example, a plurality of images that are temporally different from each other are recorded, and if the image signal processing of the image signal processing unit 3504 is performed on the plurality of image data, 1 A high-resolution image with high image quality can be obtained. Note that the image processing of the image signal processing unit 3504 may be performed on an image recorded in the content storage unit 3503 when data is output from the digital camera. Further, the image signal processing unit 3504 may be integrated with the recording / playback unit 3502 and the image signal processing unit 3504 to perform image processing of the image signal processing unit 3504 before recording in the content storage unit 3503. In this case, only the enlarged image that the user wants to handle finally needs to be stored in the content storage unit 3503, and management becomes easier when the user handles image data later.

  According to the digital camera described above, high-quality image data having a resolution exceeding the resolution of the light receiving element of the digital camera can be obtained.

  Further, if the recording / playback apparatus 3600 is, for example, a video camera, video captured on the light receiving element by the imaging optical system of the input unit 3501 may be output to the recording / playback unit 3502 as video data. The recording / playback unit 3502 may record video data in the content storage unit 3503, and the image signal processing unit 3504 may generate high-resolution video data from the recorded video data. In this way, high-quality video data having a resolution exceeding the resolution of the light receiving element of the video camera can be obtained. At this time, the image signal processing unit 3504 may generate one piece of still image data using data of a plurality of frames included in the recorded video data. In this way, one piece of high quality image data can be obtained from the video data. Similarly to the case of the digital camera described above, the image processing of the image signal processing unit 3504 may be performed before or after recording the video data in the content storage unit 3503.

  According to the video camera described above, high-quality video data having a resolution exceeding the resolution of the light receiving element of the video camera and high-quality still image data can be obtained using the captured video data.

  In addition, when the recording / playback apparatus 3600 is, for example, a surveillance camera (surveillance camera system), high-quality video data having a resolution exceeding the resolution of the light-receiving element of the surveillance camera, as with the video camera described above, Still image data with high image quality can be obtained using the video data. At this time, for example, even when the input unit 3501 including the imaging optical system and the light receiving element is separated from the recording / reproducing unit 3502 and connected by a network cable or the like, the recording / reproducing unit 3502 has a low resolution. It is possible to increase the resolution by transmitting the video data and then performing image signal processing of the image signal processing unit 3504. Thus, high-resolution video data can be obtained while efficiently using the bandwidth of the transmission network from the input unit 3501 including the imaging optical system and the light receiving element.

  The image display apparatuses according to the thirteenth to sixteenth embodiments and the recording / playback apparatus according to the present embodiment can be an embodiment of the present invention even if the functions and components thereof are integrated. In this case, the video signal or image signal subjected to the above-described image signal processing can be displayed or output to another device, and can be used as any of a display device, a recording / playback device, and an output device, Convenient for users.

  According to the recording / reproducing apparatus in the seventeenth embodiment described above, each image signal of two input image frames included in the input video signal or the input image signal is phase-shifted, and each image signal is detected. Two signals are generated. As a result, four signals are generated from the image signals of the two input image frames. Here, based on the phase difference between the two input image frames, for each of the four signals, a coefficient for canceling and combining the aliasing components of the four signals is calculated for each pixel. For each pixel of the image to be generated, a sum is calculated by multiplying the pixel value of the corresponding pixel included in each of the four signals by each coefficient to generate a pixel value of a new high-resolution image. By performing this for each pixel of the generated image, an image having a higher resolution in the one-dimensional direction than the input image frame is generated.

  This is performed in each of the horizontal direction and the vertical direction, and an image with high resolution in the horizontal direction and an image with high resolution in the vertical direction are generated. The image having the high resolution in the horizontal direction and the image having the high resolution in the vertical direction are subjected to up-rate processing in the vertical direction and the horizontal direction, respectively, and then both are mixed.

  As a result, a high-resolution image with high resolution in both the vertical direction and the horizontal direction can be generated from each image signal of the two input image frames included in the input video signal or the input image signal. That is, a two-dimensional high resolution image can be generated and output.

  Also, the input video signal or the input image signal is recorded in the recording unit, and at the time of reproduction from the recording unit, the vertical and horizontal directions are obtained from the image signals of the two input image frames included in the video signal or the image signal. A two-dimensional high resolution image with high resolution in both directions can be generated and can be output.

  In addition, since the recording / playback apparatus according to the seventeenth embodiment uses two input image frames, it is possible to output a high resolution image with a small amount of image processing. As a result, it is possible to realize a recording / reproducing apparatus that outputs a high-resolution video or image in both the vertical direction and the horizontal direction with less aliasing components.

  The recording / playback apparatus according to Embodiment 18 of the present invention is the same as the recording / playback apparatus according to Embodiment 17, except that the image signal processing unit 3504 shown in FIG. 36 is replaced with the image signal processing apparatus described in Embodiment 9. . Since other configurations are the same as those of the recording / playback apparatus according to the seventeenth embodiment, description thereof is omitted.

  Further, the detailed operation of the image signal processing unit 3504 is as described in the ninth embodiment, and thus the description thereof is omitted.

  According to the recording / reproducing apparatus in the eighteenth embodiment, using two input image frames included in the input video signal or the input image signal, the resolution is higher in the horizontal direction, the vertical direction, and the oblique direction than the input video image or the input image. It is possible to generate a two-dimensional high-resolution image and output it.

  Further, the input video signal or the input image signal is recorded in the recording unit, and at the time of reproduction from the recording unit, the horizontal direction and the vertical direction are obtained from each image signal of the two input image frames included in the video signal or the image signal. A two-dimensional high-resolution image with high resolution in the direction and oblique direction can be generated and output.

  The recording / playback apparatus according to Embodiment 19 of the present invention is the same as the recording / playback apparatus according to Embodiment 17, except that the image signal processing unit 3504 shown in FIG. 36 is replaced with the image signal processing apparatus described in Embodiment 11. . Since other configurations are the same as those of the recording / playback apparatus according to the seventeenth embodiment, description thereof is omitted.

  The detailed operation of the image signal processing unit 3504 is the same as that described in the eleventh embodiment, and a description thereof will be omitted.

  According to the recording / reproducing apparatus in the nineteenth embodiment, four input image frames included in the input video signal or the input image signal are used, and the resolution in the horizontal direction, the vertical direction, and the oblique direction is higher than that of the input video or input image. It is possible to realize a recording / playback apparatus that generates and outputs a converted two-dimensional high-resolution image.

  Further, the input video signal or the input image signal is recorded in the recording unit, and at the time of reproduction from the recording unit, the horizontal direction and the vertical direction are obtained from each image signal of the four input image frames included in the video signal or the image signal. A two-dimensional high-resolution image with high resolution in the direction and oblique direction can be generated and output.

  Further, the resolution improvement effect of the recording / playback apparatus according to the nineteenth embodiment can be improved to a higher frequency component in the oblique direction than the recording / playback apparatus according to the eighteenth embodiment, and a higher-resolution image with higher image quality is output. be able to.

  The recording / playback apparatus according to the twentieth embodiment of the present invention is the same as the recording / playback apparatus according to the seventeenth embodiment, but the image signal processing unit 3504 shown in FIG. 36 is one of the first, third, or fifth embodiments. The image signal processing apparatus described in the embodiment is replaced. Since other configurations are the same as those of the recording / playback apparatus according to the seventeenth embodiment, description thereof is omitted.

  The detailed operation of the image signal processing unit 3504 is the same as that described in the first, third, or fifth embodiment, and thus the description thereof is omitted.

  According to the recording / playback device according to the twentieth embodiment, a high resolution in which two input image frames included in the input video signal or the input image signal are used and the resolution is increased in a one-dimensional direction as compared with the input video or the input image. A recording / reproducing apparatus that generates and outputs an image can be realized.

  In addition, the input video signal or the input image signal is recorded in the recording unit, and at the time of reproduction from the recording unit, from each image signal of the two input image frames included in the video signal or the image signal, in a one-dimensional direction. A high-resolution image with high resolution can be generated and output.

  FIG. 45 shows an image signal processing apparatus according to Embodiment 21 of the present invention. The image processing apparatus according to the present embodiment includes, for example, an input unit (1) to which a frame sequence of a moving image such as a television broadcast signal is input, and a frame input from the input unit (1) in the horizontal and vertical directions. A resolution conversion unit (4) for increasing the combined two-dimensional resolution, a display unit (3) for displaying an image based on the frame whose resolution has been increased by the resolution conversion unit (4), and the resolution conversion A phase conversion unit (4501) for converting phase difference information used in the unit (4). Here, the resolution conversion unit (4) used in the image signal processing apparatus of the present embodiment is the resolution conversion unit (4) shown in FIG. 21 similar to that of the eighth embodiment of the present invention, and thus the description thereof is omitted.

  In the image signal processing device according to the twenty-first embodiment of the present invention, by using the phase conversion unit (4501), the movement of the subject that could not be used for high resolution in the image signal processing device according to the eighth embodiment of the present invention. Information (horizontal phase difference θH (2102) and vertical phase difference θV (2103)) is converted into information (horizontal phase difference θH '(4502) and vertical phase difference θV' (4503)) that can be used for higher resolution . Details of the phase conversion unit (4501) will be described below.

  FIG. 46 shows the operating principle of the phase converter (4501). FIG. 5A shows a state in which the subject moves to the right in the image frame (4601). Here, as an example, it is assumed that the movement of the subject (4602) includes only horizontal movement and does not include vertical movement. Figure (b) is an enlarged view of part (4603) of the image in Figure (a), and the subject outline (4610) is positioned at (4611) as the subject moves (4602). It shows how it changed.

  Here, the pixel (4607) at the position (x0, y0) above the image frame (4601) is the pixel (4606) at the position (x1, y1) one frame before the subject motion (4602). It has moved with it. At this time, the vertical phase difference θV (2103) output from the position estimation unit (2101) shown in FIG. 45 becomes 0, and the coefficient C1 (and the required high resolution in the vertical direction is calculated by the calculation formula shown in FIG. Since C3) becomes unstable, the aliasing component cannot be removed by the aliasing component removal unit (2109) shown in FIG. 45 simply by using the vertical phase difference θV (2103) as it is.

  On the other hand, as shown in FIG. 46 (b), if the luminance value (signal level) of the pixel on the contour line (4610) of the subject is the same, the position of (x0, y0) on the image frame (4601) The pixel (4607) at the position (x2, y2) one frame before has moved downward (4604), or the pixel at the position (x3, y3) one frame before ( 4609) may have been moved in the upper right direction (4605). That is, if the motion information in the downward direction (4604) or the upper right direction (4605) is used, the vertical phase difference θV ′ (4503) whose value is not 0 can be obtained, and the aliasing component removal unit (2109) shown in FIG. Thus, the folded component can be removed. Therefore, if the direction of the same luminance is estimated in the vicinity of the pixel (4606) at the position (x1, y1) corresponding to the original motion (4602) of the subject, the motion of the subject that could not be used for higher resolution Information (horizontal phase difference θH (2102) and vertical phase difference θV (2103)) is converted into information (horizontal phase difference θH '(4502) and vertical phase difference θV' (4503)) that can be used for higher resolution be able to.

  FIGS. 46 (c) (d) (e) collectively show the operation of the phase converter (4501). FIG. 3C shows the original horizontal phase difference θH (2102) and vertical phase difference θV (2103), which are values (x0−x1) and (y0−y1), respectively. FIG. 4D shows the phase conversion operation in the same luminance direction.The pixel (4606) at the position (x1, y1) corresponding to the original motion (4602) of the subject is represented by the subject. The operation converted into the pixel (4608) at the position (x2, y2) on the contour line (4610) is shown. (E) in the figure shows the operation for obtaining the horizontal phase difference θH ′ (4502) and the vertical phase difference θV ′ (4503) after phase conversion based on the operations in (c) and (d) in the figure. (X0−x2) and (y0−y2), that is, θH (2102) + (x1−x2) = θH + ΔθH, and θV (2103) + (y1−y2) = θV + ΔθV, respectively. The resolution conversion unit (4) shown in FIG. 45 may perform the aliasing component removal using the horizontal phase difference θH ′ (4502) and the vertical phase difference θV ′ (4503) after the phase conversion. Note that the pixels (4606), (4608), and (4609) shown in FIG. 46B may be not only actual pixels (real pixels) but also pixels that are interpolated using neighboring real pixels (interpolated pixels). Further, the pixels (4606), (4607), (4608), and (4609) do not have to be on the contour line of the subject, and may have a correspondence relationship that results in the same luminance value.

  FIG. 47 shows a configuration example of the phase conversion unit (4501) that realizes the phase conversion operation shown in FIG. In FIG. 45, first, based on frame # 2 (4604) input from the input unit (1) shown in FIG. 45, the same luminance direction estimation unit (4702) uses a horizontal phase correction value ΔθH (4707). ) And the vertical phase correction value ΔθV (4708), and the horizontal phase difference θH (2102) and vertical phase difference θV (2103) output from the phase estimation unit (2101) shown in FIG. 4703) and 4704 are added and input to the switches (4705) and (4706), respectively. In parallel with this, the horizontal phase difference θH (2102) and the vertical phase difference θV (2103) are input to the phase determination unit (4701), and the resolution conversion unit (4) shown in FIG. If it is determined that it can be used as it is, the switch (4705) (4706) is switched to the upper side, and the horizontal phase difference θH (2102) and vertical phase difference θV (2103) remain unchanged. The phase conversion unit 4501 outputs the horizontal phase difference θH ′ (4602) and the vertical phase difference θV ′ (4603) after the phase conversion. On the other hand, if the phase determination unit (4701) determines that it cannot be used as it is, the switches (4705) and (4706) are switched to the lower side, and the outputs of the adders (4703) and (4704) are phase-converted. The subsequent horizontal phase difference θH ′ (4602) and vertical phase difference θV ′ (4603) are output from the phase conversion unit (4501).

  FIG. 48 shows a method of determining whether the horizontal phase difference θH (2102) or the vertical phase difference θV (2103) can be used as it is in the phase determination unit (4701). In the figure, it is determined whether the coefficient C1 (and coefficient C3) can be obtained using the calculation formula shown in FIG. 9, and either the horizontal phase difference θH (2102) or the vertical phase difference θV (2103) is determined. When one of them becomes 0 and the coefficients C1 and C3 become indefinite, or when one of the horizontal phase difference θH (2102) θH (2102) or the vertical phase difference θV (2103) approaches 0, the coefficients C1 and C3 When it becomes vulnerable to noise or the like due to the increase, it is determined that “the horizontal phase difference θH (2102) or the vertical phase difference θV (2103) cannot be used as it is”.

  Here, the case where the signal input to the input unit (1) shown in FIG. 45 is interlaced scanning will be described.

  First, referring to FIG. 37, general interlace scanning and progressive scanning will be described. FIG. 37A shows the positional relationship of interlaced scanning (2: 1 interlaced) scanning lines, and FIG. 37B shows the positional relationship of progressive scanning scanning lines. In each figure, the horizontal axis represents the position (t) in the time direction (frame direction), and the vertical axis represents the vertical position (v).

  In the progressive scanning shown in FIG. 5B, the scanning lines in the frame (3705) are sequentially scanned. On the other hand, in the interlaced scanning shown in FIG. 5A, scanning lines (3701) to be transmitted or displayed (3701) and scanning lines (3702) to be transmitted or not displayed alternately are alternately displayed. A field (3703) is formed in a repeated manner. In the next field, the positions of the scanning line (3701) and scanning line (3702) are reversed (complementary), and the two fields (3703) (3704) are combined to form one frame (3704). Is done.

  Therefore, in this embodiment, when the signal input to the input unit (1) shown in FIG. 45 is interlaced scanning, the coefficients C0, C1, C2, and C3 shown in FIG. Alternatively, θV) needs to be a value obtained by replacing (θ ± π).

  Therefore, when the phase difference θ becomes ± π and the coefficients C1 and C3 become indefinite, or when the coefficients C1 and C3 increase as the phase difference θ approaches ± π, the noise becomes weak. The horizontal phase difference θH (2102) or the vertical phase difference θV (2103) cannot be used as it is ”.

  An example of the operation of the same luminance direction estimation unit (4702) shown in FIG. 47 will be described with reference to FIG. In FIG. 49A, real pixels (4901) arranged in a lattice form represent pixels that originally exist on the frame. The interpolation pixel (4903) indicates N points (# 1 to #N) (N is a positive integer) arranged on the circumference centering on the pixel (4902) to be processed. This is a pixel interpolated using the pixel (4901). Note that the position of the interpolation pixel (4903) is not limited to the position shown in FIG. 10 (a) .For example, as shown in FIG. 4904) may be arranged in a rectangular shape around the central pixel (4902). From the interpolation pixels (4903) or (4904) arranged in this way, select the pixel with the smallest difference from the luminance value of the center pixel (4902), and select the center pixel (4902) and the selected interpolation. A horizontal phase correction value ΔθH (4707) and a vertical phase correction value ΔθV (4708) are obtained from the direction of the straight line connecting the pixels (4903) or (4904).

  FIG. 50 shows a configuration example of the same luminance direction estimation unit (4702). In the figure, first, a horizontal low-pass filter (hereinafter referred to as LPF) (5001) and a vertical LPF filter (5002) are applied to the input signal of frame # 2 (4504) to obtain the value of the center pixel. . These are filters to reduce the influence of aliasing components that cause the same luminance direction to be erroneously estimated, and the frequency of about 1/2 to 1/4 of the Nyquist frequency band of the input signal is The cut-off frequency may be used. Subsequently, using the pixel interpolators # 1 (5003-1) to #N (5003-N), the interpolation pixel (4903) shown in FIG. 49 (a) or the interpolation pixel shown in FIG. 49 (b) A signal at the position (4904) is generated. This pixel interpolator # 1 (5003-1) to #N (5003-N) performs pixel interpolation from the value of the actual pixel (4901) located near the center pixel (4902) using the sinc function described above. Since a general interpolation LPF to be performed can be used as it is, detailed illustration and description are omitted. Each output of the pixel interpolators # 1 (5003-1) to #N (5003-N) is output from the signal before pixel interpolation (i.e., real pixel =) using the subtracters (5004-1) to (5004-N). Output signal of each of the vertical LPFs (5002)), and using the absolute value converters (5005-1) to (5005-N), the luminance value difference is calculated for each interpolation pixel # 1 to #N. Find the absolute value. The direction selector (5006) selects an interpolation pixel so that this value (absolute value of luminance value difference) is minimized, and the horizontal direction and vertical direction of the position of the selected interpolation pixel and the position of the center actual pixel are selected. The direction difference is output as a horizontal phase correction value ΔθH (4707) and a vertical phase correction value ΔθV (4708). Note that if the cutoff frequencies of the horizontal LPF (5001) and vertical LPF (5002) are set low (that is, the passband is narrowed), the aliasing component becomes small, but at the same time, a fine texture that represents the original same luminance direction. The signal component of (pattern) is also attenuated. Conversely, when the cutoff frequency is set high (that is, when the passband is widened), a fine texture (pattern) signal component remains, but the influence of the aliasing component becomes large. Therefore, since the two are in a trade-off relationship, the characteristics of the horizontal LPF (5001) and the vertical LPF (5002) are the actual image of the input frame (4504), the horizontal phase correction value ΔθH (4707), and the vertical It is preferable to design so as to be optimal while confirming the result of the direction phase correction value ΔθV (4708).

  In FIGS. 49A and 49B, for the sake of simplicity, the center pixel (4902) is illustrated at the same position as the actual pixel. However, in practice, the position estimation unit (2101) illustrated in FIG. The position of the pixel on frame # 2 corresponding to the pixel to be processed on # 1 is obtained, and this is set as the position of the center pixel (4902). Therefore, if the value of horizontal phase difference θH (2102) or vertical phase difference θV (2103) is not an integer multiple of the actual pixel interval (including 0), the center pixel (4902) is also in the same position as the actual pixel. Don't be. In this case, the tap coefficient of the horizontal LPF (5001) shown in FIG. 50 is shifted by the horizontal phase difference θH (2102), and the tap coefficient of the vertical LPF (5002) is shifted by the vertical phase difference θV (2103). . Specifically, when the cutoff frequency of the horizontal LPF (5001) is fc (H) and the Nyquist frequency is fn (H), the tap coefficient Ck (H) of the horizontal LPF (5001) (where k is an integer) Is a generally known sinc function, Ck (H) = 2sin (π * fc (H) * k / fn (H) -θH) / (π * fc (H) * k / fn (H) -θH). Similarly, tap coefficient Ck (V) of vertical LPF (5002) is Ck (V) = 2sin (π * fc (V) * k / fn (V) -θV) / (π * fc (V) * k / fn (V) −θV).

  Also, the interval between the center pixel (4902) and the interpolation pixel (4903) in FIG. 49 (a) (= the radius of the circle), and the interval between the center pixel (4902) and the interpolation pixel (4904) in FIG. 49 (b). (= 1/2 of the length of each side of the rectangle) corresponds to the magnitude of each of the horizontal phase correction value ΔθH (4707) and the vertical phase correction value ΔθV (4708). The interval (the radius of the circle or the length of each side of the rectangle) may be a fixed value or a variable value. That is, when the aliasing distortion is removed using the horizontal phase difference θH ′ (4502) and the vertical phase difference θV ′ (4503) after the phase conversion shown in FIG. 46 (e), the calculation formula shown in FIG. The horizontal phase correction value ΔθH (4707) and the vertical phase correction value ΔθV (4708) are approximately equal to the actual pixel (4901) interval so that the coefficient C1 (and coefficient C3) can be obtained using Alternatively, ΔθH (4707) and ΔθV may be set so that the same luminance direction indicated by the horizontal phase correction value ΔθH (4707) and the vertical phase correction value ΔθV (4708) does not change. Each value may be increased or decreased as appropriate while the ratio of (4708) is kept constant.

  The ratio of ΔθH (4707) and ΔθV (4708) was kept constant so that the same luminance direction indicated by the horizontal phase correction value ΔθH (4707) and the vertical phase correction value ΔθV (4708) did not change. Even if each value is increased or decreased, the coefficient C1 (and coefficient C3) may not be obtained using the calculation formula shown in FIG. For example, in FIG. 49A, when the same luminance direction is horizontal (# 1), the vertical phase correction value ΔθV (4708) is always 0, and the vertical phase difference θV ′ (4603) after phase conversion is obtained. Even if the vertical phase difference θV (2103) is used as it is, the folding component removal unit (2109) shown in FIG. 45 cannot remove the folding component in the vertical direction. In this case, similar to the image signal processing apparatus according to the fifth embodiment of the present invention, the general interpolation low-pass filter (1101) shown in FIG. 11 is bypassed in the aliasing component removal units (2108) and (2109). In addition to the coefficients C0 and C1 described above, the coefficient determiner (1103) newly generates C4, and the multiplier (1102) multiplies the output of the interpolation low-pass filter (1101) by the coefficient C4. The signal may be output in addition to the signal whose resolution has been increased by the adder (1104). The configurations other than the interpolation low-pass filter (1101), multiplier (1102), coefficient determiner (1103), adder (1104), and auxiliary pixel interpolation unit (1105) are the same as those in the third embodiment shown in FIG. Since the configuration is the same, the description thereof is omitted. The operations and configurations of the interpolation low-pass filter (1101), multiplier (1102), coefficient determiner (1103), adder (1104), and auxiliary pixel interpolation unit (1105) are shown in FIGS. Since the operation and configuration are the same as those of the fifth embodiment, description thereof is omitted.

  If the minimum value of the difference between the signal values of the center pixel (4902) and the interpolation pixels (4903) (4904) shown in FIGS. 49 (a) and 49 (b) is larger than a predetermined threshold value, the same luminance It is determined that the direction cannot be estimated, and the phase correction values ΔθH (4707) and ΔθV (4708) are set to 0 by the direction selector (5006) in the same luminance direction estimation unit (4702) shown in FIG. For example, the aliasing component removal units (2108) and (2109) shown in FIG. 45 may be prevented from malfunctioning.

  According to the image signal processing device according to the twenty-first embodiment described above, in addition to the effects of the image signal processing device according to the eighth embodiment, the subject that could not be used for higher resolution in the image signal processing device according to the eighth embodiment. Even in the case of movements, there is an effect that it is possible to realize high resolution.

  The configuration of FIG. 45 describing the image signal processing apparatus according to the present embodiment is based on the configuration of FIG. 21 illustrating the image signal processing apparatus according to the eighth embodiment of the present invention, but is not limited thereto. However, for the image signal processing apparatus according to another embodiment of the present invention, it is clear that the same effect can be obtained even if the phase converter (4501) shown in FIG. 45 is provided. Description is omitted.

  The twenty-second embodiment relates to an image signal processing method that realizes processing equivalent to the image signal processing in the image signal processing apparatus according to the twenty-first embodiment by a control unit that cooperates with software. The image processing apparatus that performs the image signal processing method of the present embodiment is the image processing apparatus shown in FIG.

  FIG. 51 shows an example of a flowchart of the operation of the present embodiment. The flowchart of FIG. 51 is obtained by adding step (5103) to the flowchart shown in FIG. 14 according to the second embodiment of the present invention. Steps (1401) to (1420) are the same as those of the second embodiment of the present invention. Since it is the same operation | movement, description is abbreviate | omitted.

  In step (5103), first, in step (5101), based on the phase difference θ (that is, horizontal phase difference θH or vertical phase difference θV) obtained in step (1405), the calculation formula shown in FIG. It is determined whether or not the coefficient C1 determined based on this is appropriate. If appropriate, the process proceeds to step (1406) and step (1409), and if not appropriate, the process proceeds to step (5102). Here, when determining whether the coefficient C1 is appropriate, if the phase difference θ becomes 0 and the coefficient C1 becomes indefinite, or if the coefficient C1 increases as the phase difference θ approaches 0, noise or the like When it becomes vulnerable, it may be determined that “the coefficient C1 is not appropriate”.

  As described in the twenty-first embodiment, when the signal input to the input unit (1) shown in FIG. 18 is interlaced scanning, the coefficients C0, C1, C2, and C3 shown in FIG. Since θ is a value obtained by replacing (θ ± π), noise occurs when the phase difference θ becomes ± π and the coefficient C1 becomes indefinite, or when the phase difference θ approaches ± π, the coefficient C1 increases. Or the like, it may be determined that “the coefficient C1 is not appropriate”.

  In step (5102), the same luminance direction is estimated in the same manner as the operation described with reference to FIGS. 46 to 50, and the original phase difference θ (that is, horizontal phase difference θH or vertical phase difference θV) is changed to the horizontal direction. The phase correction value ΔθH (4707) or the vertical phase correction value ΔθV (4708) is added to convert it to a new phase difference θ, and the new phase difference θ is used to perform steps (1406) and (1409). move on.

  Based on the above flowchart, if the process starts from step (1401) shown in FIG. 51 and ends in step (1417), the signal buffered in frame buffer # 3 shown in FIG. It can be output to the display unit (3) in pixel units.

  By performing the processing as described above, it is possible to output a high resolution signal to the frame buffer # 3 using the pixel data of the frame buffer # 1 and the frame buffer # 2. When applied to a moving image, the process from step (1401) to step (1417) shown in FIG. 51 may be repeated for each frame.

  The flowchart of FIG. 51 for explaining the operation according to the present embodiment is obtained by adding step (5103) to the flowchart shown in FIG. 14 according to the second embodiment of the present invention. It is clear that adding the step (5103) to the flowchart shown in FIG.

  The ratio of ΔθH (4707) and ΔθV (4708) was kept constant so that the same luminance direction indicated by the horizontal phase correction value ΔθH (4707) and the vertical phase correction value ΔθV (4708) did not change. Even if each value is increased or decreased, the coefficient C1 (and coefficient C3) may not be obtained using the calculation formula shown in FIG. For example, in FIG. 49A, when the same luminance direction is horizontal (# 1), the vertical phase correction value ΔθV (4708) is always 0, and the vertical phase difference θV ′ (4603) after phase conversion is obtained. Even if the vertical phase difference θV (2103) is used as it is, the vertical folding component cannot be removed in step (1420). In this case, similarly to the image signal processing method according to the sixth embodiment of the present invention, steps (1605) and (1606) shown in FIG. 16 are prepared, and the phase difference θ after conversion becomes 0 or near 0. The processing result of step (1606) may be configured to be output to frame buffer # 3. Since Step (1605) and Step (1606) are the same as the operation of the sixth embodiment, description thereof is omitted.

  According to the image signal processing method according to the twenty-second embodiment described above, in addition to the effects of the image signal processing method according to the second embodiment, the subject that cannot be used for increasing the resolution in the image signal processing method according to the second embodiment. Even in the case of movements, there is an effect that it is possible to realize high resolution. That is, it is possible to realize a high resolution process corresponding to a variety of subject movements.

  The image display device according to the twenty-third embodiment of the present invention is the same as the image display device according to the thirteenth embodiment except that the image signal processing unit 3504 shown in FIG. 35 is replaced with the image signal processing device described in the twenty-first or twenty-second embodiment. It is a thing. Since other configurations are the same as those of the image display apparatus according to the thirteenth embodiment, description thereof is omitted.

  The detailed operation of the image signal processing unit 3504 is as described in the twenty-first or twenty-second embodiment, and thus the description thereof is omitted.

  According to the image display device according to the twenty-third embodiment, in addition to the effects of the image display device according to the thirteenth embodiment, even in the case of the movement of the subject that cannot be used for high resolution in the image display device according to the thirteenth embodiment, There is an effect that high resolution can be realized. In addition, it is possible to realize an image display device that displays an image generated by the resolution enhancement processing on the display unit. That is, it is possible to generate a high-resolution image corresponding to a variety of subject movements and display the generated high-resolution image.

  The recording / playback apparatus according to Embodiment 24 of the present invention is the same as the recording / playback apparatus according to Embodiment 17, except that the image signal processing unit 3504 shown in FIG. 36 is replaced with the image signal processing apparatus described in Embodiment 21 or 22. It is a thing. Since other configurations are the same as those of the recording / playback apparatus according to the seventeenth embodiment, description thereof is omitted.

  The detailed operation of the image signal processing unit 3504 is as described in the twenty-first or twenty-second embodiment, and thus the description thereof is omitted.

  According to the recording / playback device according to the twenty-fourth embodiment, in addition to the effects of the recording / playback device according to the seventeenth embodiment, even in the case of a subject motion that could not be used for high resolution in the recording / playback device according to the seventeenth embodiment, There is an effect that high resolution can be realized. In addition, an image generated by the resolution enhancement process can be output. That is, it is possible to generate a high-resolution image corresponding to a variety of subject movements and output the generated high-resolution image.

  FIG. 52 shows an image signal processing apparatus according to Embodiment 25 of the present invention. The image processing apparatus according to the present embodiment includes, for example, an input unit (1) to which an image frame such as a television broadcast signal is input, and a frame input from the input unit (1) that combines the horizontal and vertical directions. A resolution conversion unit (9) for increasing the resolution of a dimension, a display unit (3) for displaying an image based on a frame whose resolution has been increased by the resolution conversion unit (9), and the resolution conversion unit (9 The same luminance direction estimator (4702) for generating phase difference information to be used in (1). Here, the image signal processing apparatus according to the present embodiment removes the input unit (1) for inputting the signal of frame # 2 from the image signal processing apparatus shown in FIG. The estimation unit (2101) is replaced with the same luminance direction estimation unit (4702). Other configurations and operations are the same as those of the image signal processing apparatus shown in FIG.

  The image signal processing device according to the twenty-fifth embodiment of the present invention includes a resolution conversion unit (44) that realizes high resolution processing using only the signal of frame # 1 input from the input unit (1). . The details will be described below.

  In the image signal processing apparatus according to Example 21 described above, the image signal processing apparatus according to Example 8 of the present invention could not be used for high resolution by using the phase converter (4501) shown in FIG. Information on the movement of the subject (horizontal phase difference θH (2102) and vertical phase difference θV (2103)) can be used for higher resolution (horizontal phase difference θH '(4502) and vertical phase difference θV' (4503)) Converted to.

  On the other hand, in the twenty-fifth embodiment of the present invention, assuming that the signal of frame # 2 is the same as the signal of frame # 1, that is, the image is completely stationary, the phase converter (4501) shown in FIG. As shown in FIG. 47, the image signal to be input to the frame is frame # 1 (5201) and the information on the motion of the subject (horizontal phase difference θH (2102) and vertical phase difference θV (2103)) is forcibly set to 0. The configuration of the phase conversion unit (4501) is replaced with only the same luminance direction estimation unit (4702). Hereinafter, this operation will be described in detail.

  FIG. 53 shows the operation principle of the same luminance direction estimation unit (4702) in the image signal processing apparatus according to the present embodiment. FIG. 5A shows a state in which the subject is reflected in the image frame (5301). Note that the image signal processing apparatus according to the present embodiment handles only the signal of one image frame (5301), unlike the image frame (4601) shown in FIG. FIG. 5B is an enlarged view of a part (5302) of the image in FIG. 4A, and shows the outline of the subject (5303).

  If the luminance value (signal level) of the pixel above the contour line (5303) of the subject is the same as shown in FIG. (B), the pixel (5304) at the position (x1, y1) is It can also be considered that the pixel (5305) at the position (x2, y2) has moved in the lower left direction (5306). That is, one image frame (5301) is considered as two virtual image frames, and for each pixel (5304) to be processed on the image frame (5301), another image frame (5301) The horizontal phase difference θH '(4502) and vertical phase difference θV' (4503) are generated by regarding the difference in position (sampling phase difference) from the pixel (5305) in the vicinity of the object to be processed above as motion information. Then, as in the image signal processing apparatus according to the twenty-first embodiment described above, higher resolution can be realized. Note that the pixels (5304) and (5305) shown in FIG. 5B may be not only actual pixels (real pixels) but also pixels (interpolated pixels) interpolated using neighboring real pixels. Further, the pixels (5304) and (5305) do not need to be on the contour line of the subject, and may have a correspondence relationship such that they have the same luminance value.

  53 (c) (d) (e) collectively show the operation of the same luminance direction estimation unit (4702). The figure (c) shows the original horizontal phase difference θH (2102) and vertical phase difference θV (2103), and is used to consider one image frame (5301) as two stationary image frames. The original horizontal phase difference θH (2102) = (x1−x1) = 0 and the original vertical phase difference θV (2103) = (y1−y1) = 0. FIG. 6D shows the phase conversion operation in the same luminance direction.The pixel (5304) at the position (x1, y1) is moved to (x2, y2) above the contour line (5303) of the subject. The operation converted into the pixel (5305) at the position y2) is shown. (E) in the figure shows the operation for obtaining the horizontal phase difference θH ′ (4502) and the vertical phase difference θV ′ (4503) after phase conversion based on the operations in (c) and (d) in the figure. The horizontal phase difference after phase conversion θH ′ (4502) = (x1−x2) = ΔθH, and the vertical phase difference after phase conversion θV ′ (4503) = (y1−y2) = ΔθV.

  Here, the same luminance direction estimation unit (4702) shown in FIG. 52 has the same configuration and operation as the same luminance direction estimation unit (4702) shown in FIG. 50, and ΔθH and the same luminance direction estimation unit (4702) shown in FIG. ΔθV is the same as the horizontal phase correction value ΔθH (4707) and the vertical phase correction value ΔθV (4708) output from the same luminance direction estimation unit (4702) shown in FIG.

  52, the horizontal phase difference θH ′ (4502) = ΔθH (4707) and the vertical phase difference θV ′ (4503) = ΔθV generated by the same luminance direction estimation unit (4702). The aliasing component may be removed using (4708).

  The ratio of ΔθH (4707) and ΔθV (4708) was kept constant so that the same luminance direction indicated by the horizontal phase correction value ΔθH (4707) and the vertical phase correction value ΔθV (4708) did not change. Even if each value is increased or decreased, the coefficient C1 (and coefficient C3) may not be obtained using the calculation formula shown in FIG. For example, when the same luminance direction is horizontal, the vertical phase correction value ΔθV (4708) is always 0, and the folding component removal unit (2109) shown in FIG. 52 removes the folding component in the vertical direction. I can't. In this case, similar to the image signal processing apparatus according to the fifth embodiment of the present invention, the general interpolation low-pass filter (1101) shown in FIG. 11 is bypassed in the aliasing component removal units (2108) and (2109). In addition to the coefficients C0 and C1 described above, the coefficient determiner (1103) newly generates C4, and the multiplier (1102) multiplies the output of the interpolation low-pass filter (1101) by the coefficient C4. The signal may be output in addition to the signal whose resolution has been increased by the adder (1104). The configurations other than the interpolation low-pass filter (1101), multiplier (1102), coefficient determiner (1103), adder (1104), and auxiliary pixel interpolation unit (1105) are the same as those in the third embodiment shown in FIG. Since the configuration is the same, the description thereof is omitted. The operations and configurations of the interpolation low-pass filter (1101), multiplier (1102), coefficient determiner (1103), adder (1104), and auxiliary pixel interpolation unit (1105) are shown in FIGS. Since the operation and configuration are the same as those of the fifth embodiment, description thereof is omitted.

  According to the image signal processing apparatus according to the 25th embodiment described above, it is possible to generate an image with fewer aliasing components and higher resolution than the input image by using one input image frame fewer than in the conventional example. Therefore, when only one image frame is input, the entire image frame or part (one area) of the input image is stationary, and the subject moves violently. If it is impossible to estimate the pixels to be processed, it is possible to realize high resolution even if the corresponding pixels cannot be estimated between the previous and subsequent frames because the image frames are continuous frames, but each one has a completely different content. There is an effect of becoming.

  Further, since the image signal processing apparatus according to the twenty-fifth embodiment uses one input image frame that is smaller than that in the conventional example, the amount of necessary image processing can be reduced as compared with the conventional example.

  The configuration of FIG. 52 describing the image signal processing device according to the present embodiment is based on the configuration of FIG. 45 illustrating the image signal processing device according to the embodiment 21 of the present invention, but is not limited thereto. However, the same luminance direction estimation unit (4702) shown in FIG. 52 is provided for the image signal processing apparatus according to another embodiment of the present invention, and the same applies even when only one frame image is input. Since it is clear that the above effect can be obtained, the description thereof is omitted.

  The twenty-sixth embodiment relates to an image signal processing method in which processing equivalent to the image signal processing in the image signal processing apparatus according to the twenty-fifth embodiment is realized by a control unit that cooperates with software. The image processing apparatus that performs the image signal processing method of the present embodiment is obtained by removing the input unit (1) for inputting frame # 2 and the buffer # 2 (22) from the image processing apparatus shown in FIG. Since the image processing apparatus is the same as that of the second embodiment of the present invention, description thereof is omitted.

  FIG. 55 shows an example of a flowchart of the operation of the present embodiment. In the flowchart of FIG. 55, steps (1403) (1405) (1406) (1418) are deleted from the flowchart shown in FIG. 14 according to the second embodiment of the present invention, and steps (5501) (5502) are newly added. Is. Hereinafter, changes from the flowchart shown in FIG. 14 will be described.

  First, in the image signal processing method according to the present embodiment, high resolution processing is realized using only the signal of frame # 1 input from the input unit (1) of the image processing apparatus shown in FIG. Therefore, the processing steps relating to frame # 2 and frame buffer # 2 in the flowchart shown in FIG. 14 are not necessary, and steps (1403), (1405), (1406), and (1418) are deleted.

  Subsequently, in step (5501) provided instead of step (1405), the same luminance direction is estimated for the processing target pixel set in step (1404) by the operation described with reference to FIG. Then, a phase difference θ (horizontal phase difference θH ′ (4502) or vertical phase difference θV ′ (4503)) is generated, and the process proceeds to step (5502) and step (1409). In step (5502), the phase difference θ generated in step (5501) is regarded as motion information, and motion compensation is performed on pixels near the target pixel in frame buffer # 1. At this time, only the pixel data used in the π / 2 phase shift processing in step (1408), that is, pixel data in a range where a finite number of taps acts, need to be motion-compensated as neighboring pixels. The motion compensation operation is the same as the operation described with reference to FIGS.

  The other steps (1401) and (1402), steps (1407) to (1417), and steps (1419) and (1420) are the same operations as those in the second embodiment of the present invention, and thus description thereof is omitted.

  By performing the processing as described above, it is possible to output a high resolution signal to the frame buffer # 3 using the pixel data of the frame buffer # 1. When applied to a moving image, the process from step (1401) to step (1417) may be repeated for each frame.

  According to the image signal processing method according to the twenty-sixth embodiment described above, it is possible to generate an image with fewer aliasing components and higher resolution than the input image using one input image frame that is smaller than that in the conventional example.

  Further, the image signal processing method according to the twenty-sixth embodiment has an effect that the amount of necessary image processing can be reduced as compared with the conventional example because one input image frame that is smaller than that in the conventional example is used.

  FIG. 54 shows an image signal processing apparatus according to Embodiment 27 of the present invention. The image processing apparatus according to the present embodiment includes, for example, an input unit (1) to which an image frame such as a television broadcast signal is input, and a frame input from the input unit (1) that combines the horizontal and vertical directions. A resolution conversion unit (41) for increasing the resolution of a dimension and a display unit (3) for displaying an image based on the frame whose resolution has been increased by the resolution conversion unit (41) are provided.

  In the resolution conversion unit (41) shown in the figure, based on the input frame image signal (5201), the same luminance direction estimation unit (4702) described above uses the horizontal phase difference θH ′ (4502) = ΔθH ( 4707) and vertical phase difference θV ′ (4503) = ΔθV (4708), and the phase difference information is input to the resolution conversion unit (9-1) and the multipliers (5401) and (5402) are used. The polarity of each phase difference information is inverted (that is, multiplied by (-1)) and input to the resolution conversion unit (9-2). After each resolution conversion unit (9-1) (9-2) performs high resolution processing, an adder (5403) and a multiplier (5404) are used to convert each resolution conversion unit (9-1 ) (9-2) is averaged and output. The operations and configurations of the same luminance direction estimation unit (4702) and the resolution conversion units (9-1) and (9-2) are the same luminance direction estimation unit (4702) and resolution conversion unit according to Example 25 described above. Since it is the same as (9), the description is omitted.

  In the same luminance direction estimation unit (4702), FIG. 53 (b) described above shows a state in which the direction of the neighboring pixel (5305) having the same luminance with respect to the processing target pixel (5304) is estimated. On the other hand, as shown in FIGS. 49 (a) and 49 (b), when attention is paid to the vicinity of the central pixel (4902) to be processed, the direction of the same luminance in a minute region can be regarded as a straight line. This straight line may be extended so as to be point-symmetric about the pixel (4902) to be processed.

  Therefore, the polarity of the horizontal phase difference θH ′ (4502) and vertical phase difference θV ′ (4503) generated by the same luminance direction estimation unit (4702) is inverted by the multipliers (5401) (5402), and the processing target The horizontal phase difference θH ′ (4502) and the vertical phase difference θV can be obtained by converting each phase difference value so that it is point-symmetrical with respect to the pixel (4902) and then inputting it to the resolution converter (9-2). The output result similar to that of the resolution conversion unit (9-1) using the value of '(4503) as it is can be obtained.

  Even if the horizontal phase difference θH '(4502) and vertical phase difference θV' (4503) generated by the same luminance direction estimator (4702) contain errors and noise, the effect is also affected by resolution conversion. Since the parts (9-1) and the resolution converter (9-2) appear to have opposite polarities, it is possible to reduce errors and noises by averaging the outputs of both.

  Furthermore, the number of resolution converters (9) is increased, and the magnification in the multipliers (5401) and (5402) is not limited to (-1), and the horizontal phase difference θH ′ (4502) and the vertical phase difference θV can be set at various magnifications. After converting the value of '(4503) and performing the high resolution processing, the output of each resolution conversion unit (9) may be averaged and output. For example, the number of resolution converters (9) is changed from 2 to 6 shown in FIG. 54, and the horizontal phase difference θH ′ (4502) and the vertical phase difference θV ′ (4503) input to each resolution converter (9). 54 magnifications are added to the two types, 1 × and (−1) times shown in FIG. 54, and 6 types are added by adding 0.5 times, (−0.5) times, 1.5 times, and (−1.5) times. ) Are summed and multiplied by 1/6, and averaged to produce an output signal. Thus, by increasing the number of resolution conversion units (9), it is possible to reduce errors and noises generated in the same luminance direction estimation unit (4702).

  According to the image signal processing device according to the twenty-seventh embodiment described above, in addition to the effects of the image signal processing device according to the twenty-fifth embodiment, the error and noise generated in the same luminance direction estimation unit (4702) can be reduced. Has the effect of becoming possible.

  The image display device according to the twenty-eighth embodiment of the present invention is the same as the image display device according to the thirteenth embodiment, except that the image signal processing unit 3504 shown in FIG. 35 is replaced with the image signal processing device described in the twenty-fifth to twenty-seventh embodiments. It is a thing. Since other configurations are the same as those of the image display apparatus according to the thirteenth embodiment, description thereof is omitted.

  The detailed operation of the image signal processing unit 3504 is the same as that described in the twenty-fifth to twenty-seventh embodiment, and thus the description thereof is omitted.

  The image display apparatus according to the twenty-eighth embodiment has an effect that the amount of necessary image processing can be further reduced because one input image frame that is smaller than that in the conventional example is used.

  The recording / playback apparatus according to Embodiment 29 of the present invention is the same as the recording / playback apparatus according to Embodiment 17, except that the image signal processing unit 3504 shown in FIG. 36 is replaced with the image signal processing apparatus described in Embodiments 25 to 27. It is a thing. Since other configurations are the same as those of the recording / playback apparatus according to the seventeenth embodiment, description thereof is omitted.

  The detailed operation of the image signal processing unit 3504 is the same as that described in the twenty-fifth to twenty-seventh embodiment, and thus the description thereof is omitted.

  According to the recording / reproducing apparatus in the twenty-ninth embodiment, since one input image frame that is smaller than that in the conventional example is used, the amount of necessary image processing can be further reduced.

  FIG. 56 shows an image signal processing apparatus according to Embodiment 30 of the present invention. The image processing apparatus according to the present embodiment includes, for example, an input unit (1) to which a frame sequence of a moving image such as a television broadcast signal is input, and a frame input from the input unit (1) in the horizontal and vertical directions. A resolution conversion unit (43) for increasing the combined two-dimensional resolution and a display unit (3) for displaying an image based on the frame whose resolution has been increased by the resolution conversion unit (43) are provided. .

  The resolution conversion unit (43) included in the image signal processing device according to the thirtieth embodiment of the present invention is the same as the resolution conversion unit (44) similar to the image signal processing device according to the twenty-fifth embodiment of the present invention. 8 includes a resolution conversion unit (4) similar to that of the image signal processing apparatus according to 8, and a mixing unit (5601) for mixing the output signals of both. Since the resolution conversion unit (44) and the resolution conversion unit (4) are the same as the resolution conversion units shown in FIGS. 52 and 21, respectively, description thereof will be omitted. The details of the operation and configuration of the resolution conversion unit (43) will be described below.

  First, the resolution conversion unit (44) similar to the image signal processing apparatus according to the twenty-fifth embodiment of the present invention realizes high resolution using only the signal of frame # 1 input from the input unit (1). Is possible. On the other hand, as described above, the horizontal phase difference θH ′ (4502) = ΔθH (4707) and the vertical phase difference θV ′ generated by the same luminance direction estimation unit (4702) in the resolution conversion unit (44). Depending on the value of (4503) = ΔθV (4708), the coefficient C1 (and coefficient C3) cannot be obtained using the calculation formula shown in FIG. 9, and the effect of increasing the resolution may not be obtained. For example, when the same luminance direction is the horizontal direction or the vertical direction, the effect of increasing the resolution cannot be obtained.

  In addition, the resolution conversion unit (4) similar to the image signal processing apparatus according to the eighth embodiment of the present invention uses both the frame # 1 and frame # 2 signals input from the input unit (1) to achieve high resolution. Realize. At this time, as described above, depending on the values of the horizontal phase difference θH (2102) and the vertical phase difference θV (2103) generated by the position estimation unit (2101) in the resolution conversion unit (4), FIG. In some cases, the coefficient C1 (and the coefficient C3) cannot be obtained using the calculation formula shown in FIG. For example, when the input image is a progressive scanning signal, the effect of increasing the resolution cannot be obtained in a region where the subject is stationary or a region where the movement of the subject is exactly an integer pixel unit. When the input image is an interlaced scanning signal, the effect of increasing the resolution cannot be obtained in a region where the signal value of the image does not change between fields.

  Therefore, in the resolution conversion unit (43) included in the image signal processing apparatus according to the thirtieth embodiment of the present invention, the output signal (SR1 (one frame type)) of the resolution conversion unit (44) and the output of the resolution conversion unit (4) By mixing the signal (SR2 (2-frame type)) with the mixer (5601), the effect of increasing the resolution is enhanced.

  FIG. 57 shows a first configuration example of the mixer (5601). In the figure, an adder (5701) and a multiplier (5702) are used to generate an average value of each signal of SR1 (1-frame type) and SR2 (2-frame type) input to the mixer (5601). Output. In the configuration shown in the figure, each resolution improvement effect of SR1 (1 frame type) and SR2 (2 frame type) is also halved, but the mixer (5601) is the simplest configuration, It can be realized at low cost.

  FIG. 58 shows a second configuration example of the mixer (5601). In the figure, for each signal of SR1 (1-frame type) and SR2 (2-frame type) input to the mixer (5601), a multiplier K (SR3) and a multiplier (5804) are used to obtain a coefficient K (SR1 ) And the coefficient K (SR2), respectively, and the adder (5805) adds both to produce an output. The coefficient K (SR1) and the coefficient K (SR2) are generated by coefficient determiners (5801) and (5802), respectively. The operation of the coefficient determiners (5801) (5802) will be described below.

  As described in the operation of the seventh embodiment of the present invention, the aliasing component removal units (2108) and (2109) shown in FIG. 21 have the phase difference θH (2102) and the phase difference θV (2103) shown in FIG. At the same time, the coefficient C0 to C3 shown in FIG. 9 is generated by the coefficient determiner (109) shown in FIG. At this time, the coefficients C1 and C3 become indefinite when the phase differences θH (2102) and θV (2103) are 0, and the coefficients C1 and C3 are increased as the phase differences θH (2102) and θV (2103) approach 0. In order to prevent it from becoming vulnerable to noise and the like by increasing the coefficient, the coefficient C4 (0 ≦ C4 ≦ 1) shown in FIG. 13 is introduced, and auxiliary pixel interpolation is performed as in the configuration shown in FIG. preferable. In other words, the effect of improving the resolution is obtained when the value of the coefficient C4 is 0.0, but the effect of improving the resolution becomes smaller as the value of the coefficient C4 approaches 1.0.

  Using this property, in the seventh embodiment of the present invention, when the horizontal phase difference θH (2102) is near 0 (that is, the coefficient C4 (horizontal) is near 1.0), the SR (vertical) of the vertical resolution conversion result is strong. Reflected, when the vertical phase difference θV (2103) is near 0 (that is, the coefficient C4 (vertical) is near 1.0), the horizontal resolution conversion result SR (horizontal) is reflected strongly, respectively A coefficient K (horizontal) and a coefficient K (vertical) are determined using the value of the coefficient C4. In order to realize this operation, the coefficient determiner (2301) shown in FIG. 23 performs an operation of K (horizontal) = C4 (horizontal) + (1-C4 (vertical)) / 2 to determine K (horizontal). In the coefficient determiner (2303), K (vertical) = C4 (vertical) + (1-C4 (horizontal)) / 2 was calculated to determine K (vertical).

  Similarly, in this embodiment, coefficients C4 (horizontal) and C4 (vertical) are generated based on the phase difference θH ′ (4502) and the phase difference θV ′ (4503) used in the resolution converter (44). The average value of these values is C4 (SR1). Also, coefficients C4 (horizontal) and C4 (vertical) are generated based on the phase difference θH (2102) and phase difference θV (2103) used in the resolution converter (4), and the average value of these is calculated as C4 (SR2 ). That is, the coefficients C4 (SR1) and C4 (SR2) can be used as an index indicating the degree of the effect of the high resolution in the resolution converter (44) and the resolution converter (4).

  Therefore, in the mixer (5601) shown in FIG. 58, the coefficient determiner (5801) determines K (SR1) by calculating K (SR1) = C4 (SR1) + (1-C4 (SR2) / 2. In the coefficient determiner (5802), K (SR2) = C4 (SR2) + (1-C4 (SR1)) / 2 is calculated to determine K (SR2), and then each coefficient K (SR1) And K (SR2), the output (SR1) of the resolution converter (44) and the output (SR2) of the resolution converter (4) are weighted and mixed and output. It is possible to increase the proportion of the signal having the larger effect of (44) and the resolution conversion unit (4), and to increase the effect of higher resolution.

  According to the image signal processing device according to the embodiment 30 described above, in addition to the effects of the image signal processing device according to the embodiments 25 and 8, the image signal processing device according to the embodiments 25 and 8 is high. Even in the case of the same luminance direction and subject movement of the subject that could not be used for resolution enhancement, it is possible to achieve higher resolution, and there is an effect that the resolution improvement effect can be further increased.

  The configuration of FIG. 56 describing the image signal processing apparatus according to the present embodiment is the same as the resolution conversion unit (44) similar to the image signal processing apparatus according to the embodiment 25 of the present invention and the embodiment 8 of the present invention. Although it is based on the same resolution conversion unit (4) as the image signal processing apparatus according to this embodiment, the present invention is not limited to this, and the one-frame input type image signal processing apparatus according to another embodiment of the present invention and 2 Since it is clear that the same effect can be obtained even if the mixing unit (5601) shown in FIG. 56 is provided for the frame input type image signal processing apparatus, description thereof will be omitted.

  An image display apparatus according to Embodiment 31 of the present invention is the same as the image display apparatus according to Embodiment 13, except that the image signal processing unit 3504 shown in FIG. 35 is replaced with the image signal processing apparatus described in Embodiment 30. . Since other configurations are the same as those of the image display apparatus according to the thirteenth embodiment, description thereof is omitted.

  The detailed operation of the image signal processing unit 3504 is as described in the thirtieth embodiment, and thus the description thereof is omitted.

  According to the image display device according to the example 31, in addition to the effects of the image display device according to the example 28 and the example 13, the image display device according to the example 28 and the example 13 cannot be used for high resolution. Even in the case of the same luminance direction and subject movement of the subject, it is possible to achieve high resolution, and there is an effect that the resolution improvement effect can be further increased.

  The recording / playback apparatus according to Embodiment 32 of the present invention is the same as the recording / playback apparatus according to Embodiment 17, except that the image signal processing unit 3504 shown in FIG. 36 is replaced with the image signal processing apparatus described in Embodiment 30. . Since other configurations are the same as those of the recording / playback apparatus according to the seventeenth embodiment, description thereof is omitted.

  The detailed operation of the image signal processing unit 3504 is as described in the thirtieth embodiment, and thus the description thereof is omitted.

  According to the recording / playback apparatus according to the thirty-second embodiment, in addition to the effects of the recording / playback apparatuses according to the twenty-ninth and seventeenth embodiments, the recording / playback apparatus according to the twenty-ninth and seventeenth embodiments cannot be used for high resolution. Even in the case of the same luminance direction and subject movement of the subject, it is possible to achieve high resolution, and there is an effect that the resolution improvement effect can be further increased.

  Each embodiment of the present invention can be similarly applied to, for example, a DVD player, a magnetic disk player, or a semiconductor memory player in addition to the devices described in the above embodiments. For example, the present invention can also be applied to a portable image display terminal (for example, a mobile phone) for receiving 1-segment broadcasting.

  As the image frame, an image frame of a signal other than the television broadcast signal may be used. Further, for example, a streaming image transmitted via the Internet or an image frame of an image reproduced from a DVD player or an HDD player may be used.

  Further, in each of the above-described embodiments, the description has been given by taking as an example high resolution in frame units. However, the target for higher resolution is not necessarily the entire frame. For example, a part of the frame of the input image or input video may be set as the resolution target. That is, if the image processing according to the embodiment of the present invention described above is performed for a plurality of frames of a part of the frame of the input video, a high-quality enlarged image of the input image or a part of the input video can be obtained. . This can be applied to, for example, an enlarged display of a part of an image.

  In addition, each embodiment of the present invention is not limited to the luminance signal (Y), but also from color signals such as red (R), green (G), and blue (B), and RGB signals by general color space conversion processing. The present invention can also be applied to color difference signals such as converted Cb, Cr, Pb, Pr, U, and V. In this case, “luminance” in the above description may be read as “color” or “color difference”.

  It should be noted that any combination of the above-described embodiments can be an embodiment of the present invention.

According to each of the embodiments of the present invention described above, it is possible to perform processing for suitably converting a low-resolution image into an enlarged image, and it is possible to suitably obtain a high-resolution high-resolution image. That is, it is possible to suitably increase the resolution of the image signal.

In addition, according to the above-described embodiments of the present invention, the number of frames of an image necessary for obtaining a high-quality high-resolution image can be reduced.

It is explanatory drawing of Example 1 which concerns on this invention. It is a figure explaining an example of operation | movement of the general high resolution image signal processing. It is a figure explaining operation | movement of a prior art. It is a figure explaining operation | movement of Example 1 which concerns on this invention. It is explanatory drawing of Example 1 which concerns on this invention. It is explanatory drawing of Example 1 which concerns on this invention. It is explanatory drawing of Example 1 which concerns on this invention. It is explanatory drawing of Example 1 which concerns on this invention. It is explanatory drawing of Example 1 which concerns on this invention. It is explanatory drawing of Example 3 which concerns on this invention. It is explanatory drawing of Example 5 which concerns on this invention. It is explanatory drawing of Example 5 which concerns on this invention. It is explanatory drawing of Example 5 which concerns on this invention. It is explanatory drawing of Example 2 which concerns on this invention. It is explanatory drawing of Example 4 which concerns on this invention. It is explanatory drawing of Example 6 which concerns on this invention. It is a figure explaining the difference of operation of one embodiment of the present invention and the prior art. It is explanatory drawing of Example 2 which concerns on this invention. It is explanatory drawing of Example 12 which concerns on this invention. It is explanatory drawing of Example 7 which concerns on this invention. It is explanatory drawing of Example 8 which concerns on this invention. It is explanatory drawing of Example 7 which concerns on this invention. It is explanatory drawing of Example 7 which concerns on this invention. It is explanatory drawing of Example 7 which concerns on this invention. It is explanatory drawing of Example 7 which concerns on this invention. It is explanatory drawing of Example 7 which concerns on this invention. It is explanatory drawing of Example 9 which concerns on this invention. It is explanatory drawing of Example 9 which concerns on this invention. It is explanatory drawing of Example 9 which concerns on this invention. It is explanatory drawing of Example 9 which concerns on this invention. It is explanatory drawing of Example 9 which concerns on this invention. It is explanatory drawing of Example 9 which concerns on this invention. It is explanatory drawing of Example 10 which concerns on this invention. It is a figure explaining the difference of operation of one embodiment of the present invention and the prior art. It is explanatory drawing of Example 13 which concerns on this invention. It is explanatory drawing of Example 17 which concerns on this invention. It is a figure explaining operation | movement of a prior art. It is explanatory drawing of the 11th Example which concerns on this invention. It is explanatory drawing of the 11th Example which concerns on this invention. It is explanatory drawing of the 11th Example which concerns on this invention. It is explanatory drawing of the 11th Example which concerns on this invention. It is explanatory drawing of the 11th Example which concerns on this invention. It is explanatory drawing of the 12th Example which concerns on this invention. It is a figure explaining the difference of operation of one embodiment of the present invention and the prior art. It is explanatory drawing of the 21st Example which concerns on this invention. It is explanatory drawing of the 21st Example which concerns on this invention. It is explanatory drawing of the 21st Example which concerns on this invention. It is explanatory drawing of the 21st Example which concerns on this invention. It is explanatory drawing of the 21st Example which concerns on this invention. It is explanatory drawing of the 21st Example which concerns on this invention. It is explanatory drawing of the 22nd Example which concerns on this invention. It is explanatory drawing of the 25th Example which concerns on this invention. It is explanatory drawing of the 25th Example which concerns on this invention. It is explanatory drawing of the 27th Example which concerns on this invention. It is explanatory drawing of the 26th Example which concerns on this invention. It is explanatory drawing of the 30th Example which concerns on this invention. It is explanatory drawing of the 30th Example which concerns on this invention. It is explanatory drawing of the 30th Example which concerns on this invention.

Explanation of symbols

1 ... input unit; 2,4,9,41,43,44, ... resolution conversion unit; 3 ... display unit; 5,6,7 ... high resolution task; 10 ... control unit; 11 ... storage unit; 21, 22, 23, 31, 32, 33, 34, 35 ... buffer; 101, 1801, 2101, 3806, 3812 ... position estimation unit; 102, 211, 1802 ... phase difference θ; 103, 104, 1803 ... up-rate device; 105, 107, 1002, 1804,3802 ... delay device; 106,108,1805 ... π / 2 phase shifter; 109,1007,1103,2301,2302,3001,3002,3003, 3004,4101,5801,5802 ... coefficient determiner; 110,111,112,113,1003, 1006,1102,2202,2303,2304,2604,2902,3005, 3006,3007,3008,4102,5401,5402,5404,5702,5803,5804 ... multiplier; 114,1001,1008,1104,2201,2305 , 2603, 2606,2703,2901,3
009,4103,4703,4704,5403,5701,5805 ... adders; 115,1806,2110,3810 ... motion compensation / uprate units; 116,1009,1807,2111,3811 ... phase shift units; 117,1010, 1808, 2108, 2109, 2112, 2705, 2706, 3809 ... aliasing component removal unit; 201, 202, 203, 20
6,1701, 1702,1703,1704,1705,2010,2011,2012,2013,2014,2015,3401,3402,3403,3404,34
05,4401,4402,4403,4404,4504,4601,5201,5301 ... Frame; 204 ... Movement; 205 ... Horizontal line; 207 ... Motion compensation; 208,212,4606,4607,4608,4609,5304,5305 ... Pixels; 209,210 ... Sampling phase; 301,302,303,401,402 ... Original component; 304,305,306,405,406 ... Folded component; 403,404 ... Original component after π / 2 phase shift; 407,408 ... Folded component after π / 2 phase shift; 1004,2704,5004 1101 ... Interpolation low-pass filter; 1105 ... Auxiliary pixel interpolation unit; 2001 ... Horizontal resolution conversion unit; 2002 ... Vertical up-rater; 2003, 2007, 2710, 2711 ... Pixel interpolator; 2004 ... Vertical interpolation unit; ... vertical resolution conversion unit; 2006 ... horizontal up-rate device; 2008 ... horizontal interpolation unit; 2009, 2707, 5601 ... mixer; 2102, 3807, 4502 ... horizontal phase difference θH; 2103, 3808, 4503 ... vertical phase difference θV; 2104, 2105, 3801 ... Horizontal / vertical up-rater; 2106, 3803 ... Horizontal phase shift unit; 2107, 3804 ... Vertical phase shift unit; 2501 ... Horizontal solution Frequency region where effect by degree conversion is large; 2502 ... Frequency region where effect by vertical resolution conversion is large; 2601, 2602, 2605 ... Two-dimensional filter; 2701 ... Diagonal (lower right) phase shift unit; 2702 ... Diagonal (upper right) phase shift 3101 ... diagonal (upper right) frequency region where the effect of resolution conversion is large; 3102 ... frequency region where the effect of diagonal (lower right) resolution conversion is large; 3500 ... image display device; 3501 ... input unit; 3502 ... recording / playback unit; 3503 ... Content storage unit; 3504 ... Image signal processing unit; 3505 ... Display unit; 3506 ... Audio output unit; 3507 ... Control unit; 3508 ... User interface unit; 3600 ... Recording / playback device; 3605 ... Image video output unit; Audio output unit: 3701 ... Actual scan line; 3702 ... Interpolation scan line; 3703 ... Field; 3704, 3705 ... Frame; 4201, 4705, 4706 ... Switcher; 5005 ... Absolute value converter; 3805 ... Horizontal / vertical phase shift unit 4501 ... Phase conversion unit; 4602,4604,4605,5306 ... Subject Motion: 4603, 5302 ... Part of the image; 4610,4611,5303 ... Outline; 4701 ... Phase determination unit; 4702 ... Same luminance direction estimation unit; 4707,4708 ... Phase correction value; 4901 ... Real pixel; 4902 ... Center 4903,4904 ... interpolation pixels; 5001 ... horizontal low-pass filter (LPF); 5002 ... vertical low-pass filter (LPF); 5003 ... pixel interpolator; 5006 ... direction selector

Claims (6)

An input unit for inputting a plurality of image frames;
A resolution conversion unit for obtaining an output image frame by increasing the number of pixels constituting the image frame by synthesizing the plurality of input image frames;
The resolution converter
A position estimation unit that estimates a sampling phase difference using image data on the input image frame serving as a reference and corresponding image data on another input image frame;
A phase converter that converts the sampling phase difference and outputs the converted sampling phase difference;
A motion compensation / uprate unit that compensates for motion of the image data of each input image frame using the information of the converted sampling phase difference and increases the number of pixels;
A phase shift unit that phase-shifts the image data of each image frame with the increased number of pixels by a predetermined amount;
A folding component removal unit that removes the folding component and outputs the result by multiplying the image data before and after the phase shift by a coefficient determined using the information of the sampling phase difference, An image signal processing device. The phase converter is
The same luminance direction estimation unit that estimates the same luminance direction as the processing target pixel in the input image frame and outputs a phase correction value based on the estimation result,
The image signal processing apparatus according to claim 1, wherein the phase correction value is added to the sampling phase difference to obtain the converted sampling phase difference. An input step in which a plurality of image frames are input;
Comprising a resolution conversion step for obtaining an output image frame by increasing the number of pixels constituting the image frame by combining the plurality of input image frames;
In the resolution conversion step,
Perform position estimation processing to estimate the sampling phase difference using the image data on the reference input image frame and the corresponding image data on other input image frames,
Convert the sampling phase difference, perform phase conversion processing to output the converted sampling phase difference,
Performing motion compensation / uprate processing to increase the number of pixels while performing motion compensation on the image data of each input image frame using the information of the sampling phase difference after the conversion,
Perform a phase shift process that phase-shifts the image data of each image frame with the increased number of pixels by a predetermined amount,
A folding component removal process for removing and outputting a folding component is performed by multiplying each image data before and after the phase shift by a coefficient determined by using the information of the sampling phase difference, and performing addition processing. Image signal processing method. The phase conversion step includes
The same luminance direction estimation step of estimating the same luminance direction as the processing target pixel in the input image frame and outputting a phase correction value based on the estimation result;
The image signal processing method according to claim 3, further comprising a step of adding the phase correction value to the sampling phase difference to obtain the converted sampling phase difference. An input unit for inputting a plurality of image frames;
A resolution converter for obtaining an output image frame by increasing the number of pixels constituting the image frame by combining the plurality of input image frames;
A display unit for displaying the image generated by the resolution conversion unit,
The resolution converter
A position estimation unit that estimates a sampling phase difference using image data on the input image frame serving as a reference and corresponding image data on another input image frame;
A phase converter that converts the sampling phase difference and outputs the converted sampling phase difference;
A motion compensation / uprate unit that compensates for motion of the image data of each input image frame using the information of the converted sampling phase difference and increases the number of pixels;
A phase shift unit that phase-shifts the image data of each image frame with the increased number of pixels by a predetermined amount;
A folding component removing unit that removes the folding component and outputs the result by multiplying each image data before and after the phase shift by a coefficient determined using the information of the sampling phase difference, Image display device. The phase converter is
The same luminance direction estimation unit that estimates the same luminance direction as the processing target pixel in the input image frame and outputs a phase correction value based on the estimation result,
6. The image display device according to claim 5, wherein the phase correction value is added to the sampling phase difference to obtain the converted sampling phase difference.

Images