JP2009141444A - Image signal processor, image display apparatus, image signal processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To achieve a high resolution of an input image signal suitably with a small number of frames. <P>SOLUTION: One image frame is generated by entering n (n is an integer of 2 or more) image frames, estimating a sampling phase difference by using image data on the input image frame becoming a reference and each corresponding image data on the other input image frame, accumulating information of the sampling phase difference based on the motion track of image data on the input image frame becoming the reference, holding image data on the input image frame at an accumulation start point which corresponds to image data on the input image frame becoming the reference, and compounding image data on the input image frame becoming the reference and image data on the input image frame at the accumulation start point by using the accumulated information of sampling phase difference. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a technique for increasing the resolution of an image signal, and in particular, by combining a plurality of image frames, the number of pixels constituting the image frame is increased and unnecessary aliasing components are removed to increase the resolution. Regarding technology.

  Recent television receivers have become larger in screen size and do not display image signals input from broadcasting, communication, storage media, etc., as they are, but display them by increasing the number of pixels in the horizontal and vertical directions by digital signal processing. It is generally done. At this time, the resolution cannot be increased only by increasing the number of pixels by an interpolation low-pass filter using a generally known sinc function, a spline function, or the like.

  Therefore, as described in Patent Document 1, Patent Document 2, and Non-Patent Document 1, a plurality of input image frames (hereinafter abbreviated as “frames”) are combined into a single frame. A technique for increasing the number of pixels while increasing the resolution (hereinafter referred to as a conventional technique) has been proposed.

JP-A-8-336046 JP-A-9-69755 Shin Aoki “Super-resolution processing using multiple digital image data”, Ricoh Technical Report pp.19-25, No.24, NOVEMBER, 1998

  In these conventional techniques, high resolution is achieved by three processes: (1) position estimation, (2) wideband interpolation, and (3) weighted sum. Here, (1) position estimation is to estimate the difference in sampling phase (sampling position) of each image data using each image data of a plurality of input image frames. (2) Wideband interpolation increases the image data density by interpolating and increasing the number of pixels (sampling points) using a wide-band low-pass filter that transmits all high-frequency components of the original signal, including aliasing components. It is to become. (3) The weighted sum is a weighted sum corresponding to the sampling phase of each densified data, canceling out aliasing components generated during pixel sampling and simultaneously removing the high-frequency components of the original signal. Is to restore.

  FIG. 2 shows an outline of these high resolution techniques. As shown in (a) in the figure, frame # 1 (201), frame # 2 (202), and frame # 3 (203) on different time axes are input, and these are combined to form an output frame (206). Assume that you get. For simplicity, first consider the case where the subject has moved (204) in the horizontal direction, and consider increasing the resolution by one-dimensional signal processing on the horizontal line (205). At this time, as shown in (b) and (d) in the figure, the position of the signal waveform is shifted in accordance with the amount of movement (204) of the subject in frame # 2 (202) and frame # 1 (201). Occurs. The amount of displacement is obtained by the above (1) position estimation, and as shown in FIG. 5C, the frame # 2 (202) is motion-compensated (207) so that the displacement is eliminated, and the pixels ( The phase difference θ (211) between the sampling phases (209) and (210) of 208) is obtained. Based on this phase difference θ (211), by performing the above (2) wideband interpolation and (3) weighted sum, as shown in FIG. High resolution is realized by generating a new pixel (212) at the position of = π). (3) The weighted sum will be described later.

  Actually, the movement of the subject may be accompanied by movements such as rotation and enlargement / reduction as well as parallel movement, but if the time interval between frames is very small or the movement of the subject is slow, These movements can also be considered by approximating local translation.

  As shown in FIG. 3, when performing the weighted sum of (3) above when the resolution is doubled in the one-dimensional direction by the conventional techniques described in Patent Document 1, Patent Document 2, and Non-Patent Document 1. It is necessary to use signals of at least three frame images. Here, FIG. 3 is a diagram showing a frequency spectrum of each component in a one-dimensional frequency region. In the figure, the distance from the frequency axis represents the signal intensity, and the rotation angle around the frequency axis represents the phase. The weighted sum of (3) above will be described in detail below.

  When the pixel interpolation is performed by the broadband low-pass filter that transmits twice the Nyquist frequency band (frequency band 0 to sampling frequency fs) in the broadband interpolation of (2) above, the same component as the original signal (hereinafter referred to as the original component) And the sum of the aliasing components according to the sampling phase is obtained. At this time, when the above-described (2) wideband interpolation processing is performed on the signals of the three frame images, as shown in FIG. 3 (a), the original components (301), (302), (303) of each frame are displayed. It is well known that the phases are all in agreement, and the phase of the aliasing components (304), (305) and (306) rotates in accordance with the difference in sampling phase of each frame. In order to facilitate understanding of the respective phase relationships, the phase relationship of the original components of each frame is shown in FIG. 5B, and the phase relationship of the folded components of each frame is shown in FIG.

  Here, with respect to the signals of the three frame images, by appropriately selecting the coefficients to be multiplied and performing the above (3) weighted sum, the folded components (304), (305), and (306) of each frame are mutually connected. It can be canceled out and only the original components can be extracted. At this time, the vector sum of the folded components (304) (305) (306) of each frame is set to 0, that is, both the Re axis (real axis) component and the Im axis (imaginary axis) component are set to 0. For this purpose, at least three folding components are required. Therefore, it is necessary to use signals of at least three frame images in order to realize double resolution enhancement, that is, to remove one aliasing component.

  Similarly, as described in Patent Document 1, Patent Document 2, and Non-Patent Document 1, when the resolution is increased with respect to a horizontal / vertical two-dimensional input signal, the aliasing comes from the vertical and horizontal directions. When the band of the original signal doubled both vertically and horizontally, the three folded components overlap, and 2M + 1 = 7 digital data (= 7 frame image signals) were required to cancel them.

  Therefore, the prior art is not economical because the scale of the frame memory and the signal processing circuit is increased. In addition, since it is necessary to accurately estimate the positions of many frame images that are separated in time, the configuration becomes complicated. That is, it is difficult for the conventional technology to increase the resolution of a moving image frame such as a television broadcast signal.

  In addition, interlace scanning is mainly used in current television broadcast signals. However, Patent Document 1, Patent Document 2, and Non-Patent Document 1 disclose that interlace scanning signals themselves have higher resolution, interlace scanning, and so on. There is no disclosure or suggestion about progressive scan conversion (IP conversion).

  In addition, in the current digital television broadcasting using terrestrial and satellite (BS, CS), programs are broadcast with HD (High Definition) image signals in addition to conventional SD (Standard Definition) image signals. . However, not all programs have been replaced with image signals shot with an HD camera.Image signals shot with an SD camera are converted into signals with the same number of pixels as HD using an SD-to-HD converter. It is well known that it is broadcast by switching between programs or scenes.

  With a conventional receiver, if the received signal is an image signal taken with an HD camera, a high-resolution image is played back, and if it is an SD-HD converted (up-con) image signal, a low-resolution image is played back. Therefore, there is a problem that the resolution is frequently switched for each program or for each scene and is unsightly.

  Further, in the above conventional technique, since the resolution enhancement processing is performed using the difference in the sampling phase (sampling position), a signal that does not cause a difference in the sampling phase, that is, a region where the subject is stationary, a subject There is a problem that the effect of increasing the resolution cannot be obtained in a region where the movement of the pixel is an integral multiple of the pixel interval, that is, a region where the sampling phase difference between consecutive frames is close to 0 or 2π.

The present invention has been made in view of the above-described problems of the prior art, and an object thereof is to provide a technique for suitably increasing the resolution of an image signal.

  In order to achieve the above object, an embodiment of the present invention may be configured as described in the claims, for example.

  According to the present invention, it is possible to increase the resolution of an image signal more suitably.

  Embodiments of the present invention will be described below with reference to the drawings.

Moreover, in each drawing, the component to which the same code | symbol was attached | subjected shall have the same function.

  Further, the expression “phase” in each description and each drawing of this specification includes the meaning of “position” on a two-dimensional image when used in a two-dimensional image space. The said position means the position of decimal pixel precision.

  In addition, the expression “up rate” in each description and each drawing of this specification also includes the meaning of “up rate processing”. Further, the expression “upcon” in each description and each drawing of this specification indicates “upconversion processing”. Both of them mean conversion processing for increasing the number of pixels of an image (pixel number increase processing) or conversion processing for enlarging an image (image enlargement conversion processing).

  In addition, the expression “down rate” in each description and each drawing of this specification also includes the meaning of “down rate processing”. In addition, the expression “downconversion” in each description and each drawing of the present specification indicates “downconversion processing”. Both of them mean conversion processing for reducing the number of pixels in the image (pixel number reduction processing) or conversion processing for reducing the image (image reduction conversion processing).

  Further, the expression “motion compensation” in each description and each drawing of the present specification includes a meaning of performing alignment by calculating a phase difference or a sampling phase difference, that is, a spatial position difference.

In the description of each embodiment below, the method described in Reference Document 1 or Reference Document 2 may be used for the above-described (1) position estimation. As for the above-mentioned (2) wideband interpolation, a general low-pass filter having a pass band twice as high as the Nyquist frequency as described in Non-Patent Document 1 may be used.
[Reference 1] Shigeru Ando “Velocity vector distribution measurement system using spatio-temporal differential calculation of images”, Transactions of the Society of Instrument and Control Engineers, pp. 1330-1336, Vol. 22, No. 12, 1986
[Reference 2] Hiroyuki Kobayashi et al. “Phase-Only Correlation of Images Based on DCT Transform”, IEICE Technical Report ITS2005-92, IE2005-299 (2006-02), pp.73-78
In the following embodiments, the notation “SR signal” is an abbreviation of “Super Resolution signal (super-resolution signal)”.

  Embodiments of the present invention will be described below with reference to the drawings.

  FIG. 1 shows an image signal processing apparatus according to Embodiment 1 of the present invention, and features thereof will be described. The image signal processing apparatus according to this embodiment is applied to an image display apparatus such as a television receiver. In the following description of the present embodiment, an image display device will be described as an example of the image signal processing device.

  In FIG. 1, an image signal processing apparatus according to the present embodiment includes an input unit (1) to which a frame sequence of a moving image such as a television broadcast signal is input, and a frame input from the input unit (1). A resolution conversion unit (2) for increasing the resolution and a display unit (3) for displaying an image based on the frame whose resolution is increased by the resolution conversion unit (2) are further provided. As the display unit (3), for example, a plasma display panel, a liquid crystal display panel, or an electron / electrolytic emission display panel is used. Details of the resolution conversion unit (2) will be described below.

  In FIG. 1, first, the position estimation unit (101) performs a corresponding operation on the frame # 2 based on the sampling phase (sampling position) of the pixel to be processed on the frame # 1 input to the input unit (1). The pixel position is estimated, and the sampling phase difference θ (102) is obtained for each pixel.

  Next, frame # 2 is motion-compensated using the information of phase difference θ (102) by the up-raters (103) and (104) of the motion compensation / up-rate unit (115) and aligned with frame # 1. At the same time, the number of pixels in frame # 1 and frame # 2 is doubled to increase the density. The phase shift unit (116) shifts the phase of the densified data by a certain amount. Here, π / 2 phase shifters (106) and (108) can be used as means for shifting the phase of data by a certain amount. Further, in order to compensate for the delay caused by the π / 2 phase shifters (106) and (108), the signals of the frame # 1 and the frame # 2 which have been densified by the delay devices (105) and (107) are delayed.

  In the aliasing component removal unit (117), the phase difference θ (102) is calculated by the coefficient determiner (109) for each output signal of the delay units (105) (107) and the Hilbert transformers (106) (108). The generated coefficients C0, C2, C1, and C3 are respectively multiplied by multipliers (110), (112), (111), and (113), and these signals are added by an adder (114) and output. obtain. This output is supplied to the display unit 3. The position estimation unit (101) can be realized using the above-described conventional technique as it is. Details of the up-raters (103) and (104), the π / 2 phase shifters (106) and (108), and the aliasing component removal unit (117) will be described later.

  FIG. 4 shows the operation of the first embodiment of the present invention. This figure shows the outputs of the delay units (105) and (107) and π / 2 phase shifters (106) and (108) shown in FIG. 1 in a one-dimensional frequency domain. In FIG. 8A, the signals of frame # 1 and frame # 2 after the up-rate output from the delay units (105) and (107) are respectively the original components (401) and (402) and the original sampling frequency ( This is a signal obtained by adding folded components (405) and (406) folded from fs). At this time, the aliasing component (406) is rotated in phase by the above-described phase difference θ (102).

  On the other hand, the frame # 1 and frame # 2 signals output from the π / 2 phase shifters (106) and (108) are the original components (403) and (404) after the π / 2 phase shift, respectively. , A signal obtained by adding the folded components (407) and (408) after the π / 2 phase shift. (B) and (c) show the original component and the aliasing component extracted for easy understanding of the phase relationship between the components shown in (a).

  Here, when the vector sum of the four components shown in FIG. 4B is taken, the Re-axis component is set to 1, the Im-axis component is set to 0, and the four components shown in FIG. When the vector sum is taken, the coefficients to be multiplied by each component are determined so that both the Re-axis and Im-axis components are set to 0, and the weighted sum is taken. Only the components can be extracted. In other words, it is possible to realize an image signal processing apparatus that uses only two frame images to increase the resolution twice in the one-dimensional direction. Details of this coefficient determination method will be described later.

  FIG. 5 shows the operation of the up-raters (103) (104) used in the first embodiment of the present invention. In this figure, the horizontal axis represents frequency, the vertical axis represents gain (value of the ratio of output signal amplitude to input signal amplitude), and shows the “frequency-gain” characteristics of the up-raters (103) and (104). . Here, in the up-rater (103) (104), the new sampling frequency is set to a frequency (2fs) that is twice the sampling frequency (fs) of the original signal, and the position is exactly in the middle of the original pixel interval. Insert a new pixel sanding point (= zero point) into the filter to double the number of pixels and increase the density. Call. At this time, as shown in the figure, due to the symmetry of the digital signal, the characteristic repeats every frequency that is an integral multiple of 2fs.

  FIG. 6 shows a specific example of the up-raters (103) (104) used in the first embodiment of the present invention. This figure shows the tap coefficients of the filter obtained by inverse Fourier transform of the frequency characteristics shown in FIG. At this time, each tap coefficient Ck (where k is an integer) is a generally known sinc function, and is shifted by (−θ) to compensate for the sampling phase difference θ (102) for each pixel, Ck = 2sin (πk + θ) / (πk + θ) may be set. In the up-rate device (103), the phase difference θ (102) is set to 0 and Ck = 2sin (πk) / (πk). In addition, by expressing the phase difference θ (102) by the phase difference in integer pixel units (2π) + the phase difference in decimal pixel units, the phase difference compensation in integer pixel units is realized by a simple pixel shift, and decimal For compensation of the phase difference in units of pixels, the filters of the up-raters (103) and (104) may be used.

  FIG. 7 shows an operation example of the π / 2 phase shifters (106) and (108) used in the first embodiment of the present invention. A generally known Hilbert transformer can be used as the π / 2 phase shifters (106) and (108).

  In FIG. 5A, the horizontal axis represents frequency, and the vertical axis represents gain (value of the ratio of output signal amplitude to input signal amplitude), indicating the “frequency-gain” characteristics of the Hilbert transformer. Here, in the Hilbert transformer, the frequency (2fs) that is twice the sampling frequency (fs) of the original signal is used as a new sampling frequency, and all frequency components except -0 between -fs and + fs are gained. A pass band of 1.0.

  In FIG. 2B, the horizontal axis represents frequency, and the vertical axis represents phase difference (difference in output signal phase with respect to input signal phase), indicating the “frequency-phase difference” characteristic of the Hilbert transformer. Here, the phase of the frequency component between 0 and fs is delayed by π / 2, and the phase of the frequency component between 0 and −fs is advanced by π / 2. At this time, as shown in the figure, due to the symmetry of the digital signal, the characteristic repeats every frequency that is an integral multiple of 2fs.

  FIG. 8 shows an example in which the π / 2 phase shifters (106) and (108) used in the first embodiment of the present invention are configured by Hilbert transformers. This figure shows the filter tap coefficients obtained by inverse Fourier transform of the frequency characteristics shown in FIG. At this time, each tap coefficient Ck may be Ck = 0 when k = 2m (where m is an integer), and Ck = −2 / (πk) when k = 2m + 1.

  The π / 2 phase shifters (106) and (108) used in Embodiment 1 of the present invention can also use differentiators. In this case, if the general expression cos (ωt + α) representing a sine wave is differentiated by t and multiplied by 1 / ω, d (cos (ωt + α)) / dt * (1 / ω) =-sin (ωt + α) = cos (ωt + α + π / 2), and the function of π / 2 phase shift can be realized. In other words, after taking the difference between the value of the target pixel and the value of the adjacent pixel, a π / 2 phase shift function is realized by applying a filter with a frequency / amplitude characteristic of 1 / ω. May be.

  FIG. 9 shows an operation and a specific example of the coefficient determiner (109) used in the first embodiment of the present invention. As shown in FIG. 4A, when the vector sum of the four components shown in FIG. 4B is taken, the Re-axis component is set to 1, the Im-axis component is set to 0, and FIG. If the coefficients to be multiplied by each component are determined so that both the Re-axis and Im-axis components are 0 when the vector sum of the four components shown in (c) is taken, two frame images Can be used to realize an image signal processing apparatus that achieves a resolution twice as high as that in the one-dimensional direction.

  Here, as shown in FIG. 1, the coefficient for the output of the delay unit (105) (the sum of the original component and the folded component of the frame # 1 after the up-rate) is C0, and the output of the π / 2 phase shifter (106) The coefficient for (the sum of the π / 2 phase shift results of the original component and the folded component of frame # 1 after the up-rate) is C1, and the output of the delay unit (107) (the original component of frame # 2 after the up- The coefficient for the sum of the aliasing components) is C2, and the factor for the output of the Hilbert transformer (106) (the sum of the π / 2 phase shift results of the original and aliasing components of frame # 2 after the update) is C3. .

  At this time, if the conditions of FIG. 9 (a) are satisfied, the simultaneous equations shown in FIG. 9 (b) can be obtained from the phase relationships of the components shown in FIGS. 4 (b) and 4 (c). If this is solved, the result shown in FIG. 9 (c) can be derived.

  The coefficient determiner (109) according to the present embodiment outputs coefficients C0, C1, C2, and C3 that satisfy any one of FIGS. 9 (a), 9 (b), and 9 (c).

  As an example, FIG. 9 (d) shows values of the coefficients C0, C1, C2, and C3 when the phase difference θ (102) is changed from 0 to 2π every π / 8. This corresponds to a case where the position of the signal of the original frame # 2 is estimated with an accuracy of 1/16 pixel and motion compensation is performed on the frame # 1. When the value of phase difference θ (102) is less than 0 or 2π or more, use the periodicity of sin function or cos function and add the value of integer multiple of 2π to the value of phase difference θ (102) Alternatively, subtraction may be performed so that the phase difference θ (102) falls within the range of 0 to 2π.

  The up-raters (103) (104) and π / 2 phase shifters (106) (107) require an infinite number of taps to obtain ideal characteristics, but the number of taps is finite. There is no practical problem even if it is cut off and simplified. At this time, a general window function (such as a Hanning window function or a Hamming window function) may be used. If the coefficient of each tap of the simplified Hilbert transformer is the value of the left and right points centered on C0, that is, C (-k) = -Ck (k is an integer), the phase can be shifted by a certain amount. it can.

  Next, a difference in operation between the image signal processing apparatus according to the first embodiment and the above-described conventional technique will be described with reference to FIG. In FIG. 9A, an input image is prepared so that the subject moves in the right direction between frame # 1 (1701) and frame # 5 (1705). At this time, as shown in FIG. 4B, when the sampling phase in each frame is viewed, the corresponding pixel position is 1/4 pixel (= π) between frame # 1 (1701) and frame # 2 (1702). / 2) The position of the corresponding pixel is shifted by 1 pixel (= 2π) between frame # 1 (1701) and frame # 3 (1703), and frame # 1 (1701) and frame # 4 (1704 ), The corresponding pixel position is shifted by 5/4 pixels (= 5π / 2), and the corresponding pixel position is 2 pixels (= 4π) between frame # 1 (1701) and frame # 5 (1705). The subject is intentionally moved so that it is displaced. At this time, the phase of each folded component included in the signal on each frame is expressed as shown in FIG. 7C with reference to the phase of the folded component included in the signal on frame # 1 (1701). Can do. When the resolution of the input image (a) is increased by a factor of 2, in the above-described conventional technique, the aliasing component of any of the three frames from frame # 1 (1701) to frame # 5 (1705) is used. Since the vector sum cannot be reduced to 0, high resolution cannot be realized. On the other hand, if this embodiment is used, for example, the vector sum of the aliasing component can be set to 0 using two adjacent frames (eg, frame # 1 (1701) and frame # 2 (1702)). realizable. That is, the operation status of the present embodiment can be confirmed by using the input image of FIG.

  In the above description of the first embodiment, the description has been given by taking the high resolution in the horizontal direction as an example. However, the embodiments of the present invention are not limited to this, and the vertical and oblique height increases. It can be applied to resolution.

  According to the image signal processing apparatus according to the first embodiment described above, phase shift is performed on each image signal of two input image frames, which is smaller than in the conventional example, and two signals are generated from each image signal. . Thereby, four signals can be generated from the image signals of the two input image frames. Here, based on the phase difference between the two input image frames, for each of the four signals, a coefficient for canceling and combining the aliasing components of the four signals is calculated for each pixel. For each pixel of the image to be generated, a sum is calculated by multiplying the pixel value of the corresponding pixel included in each of the four signals by each coefficient to generate a pixel value of a new high-resolution image. By performing this for each pixel of the generated image, a new high-resolution image can be generated.

  Thereby, the image signal processing apparatus according to the first embodiment can generate an image with less aliasing components and higher resolution than the input image by using two input image frames that are smaller than those in the conventional example.

  In addition, since the image signal processing apparatus according to the first embodiment uses two input image frames that are fewer than the conventional example, the amount of necessary image processing can be reduced as compared with the conventional example. As a result, an image signal processing apparatus that generates an image with less aliasing components and higher resolution than the input image can be realized at a lower cost than the conventional example.

  Next, Embodiment 2 of the present invention will be described with reference to FIGS.

  The second embodiment relates to an image signal processing method in which processing equivalent to the image signal processing in the image signal processing apparatus according to the first embodiment is realized by a control unit that cooperates with software.

  First, an image processing apparatus for realizing the image signal processing method according to the present embodiment will be described with reference to FIG. The image signal processing apparatus shown in FIG. 18 stores, for example, an input unit (1) to which an image signal such as a television broadcast signal is input, and software for processing the signal input from the input unit (1). A control unit (10) that performs image signal processing on a signal input from the input unit (1) in cooperation with software stored in the unit (11), the storage unit (11), and a control unit (10) Frame buffer # 1 (21) used for data buffer in the image signal processing, frame buffer # 2 (22), and the signal after image signal processing output from the control unit (10) to the output unit (3), And buffer # 3 (23) for frame buffering.

  Here, the number of input units (1) included in the image signal processing apparatus shown in FIG. 18 is the same as two, which is the number of frames used for image processing, but only one input unit (1) is provided. One frame may be input continuously.

  In addition, the frame buffer # 1 (21), the frame buffer # 2 (22), and the storage unit (11) for storing software used for the data buffer may each be configured using individual memory chips. Alternatively, one or a plurality of memory chips may be used and each data address may be divided and used.

  In this embodiment, for the image signal input from the input unit (1), the control unit (10) performs image signal processing in cooperation with software stored in the storage unit (11), and the display unit (3) Output to. Details of the image signal processing will be described with reference to FIG.

  FIG. 14 shows an example of a flowchart of the image signal processing method according to the present embodiment. The flowchart of FIG. 14 starts from step (1401), and in step (1418), the image data of each frame is doubled. That is, in step (1402), the image data of frame # 1 is updated and written to frame buffer # 1, and in step (1403), the image data of frame # 2 is updated and written to frame buffer # 2. Here, the up-rate can be realized by clearing the value of each frame buffer to 0 and then writing the data every other pixel.

  Next, in step (1404), the first pixel (for example, the upper left pixel) of frame buffer # 1 is set as a processing target, and processing is performed until all pixel data processing for frame buffer # 1 is completed. Loop.

  In step (1405), the position of the corresponding pixel in the frame buffer # 2 is estimated with reference to the target pixel in the frame buffer # 1, and the phase difference θ is output. At this time, the above-described conventional technique can be used as it is as a method of estimating the position of the corresponding pixel.

  In step (1406), based on the phase difference θ obtained in step (1405), motion compensation is performed on pixels near the corresponding pixel in frame buffer # 2. At this time, only the pixel data used in the π / 2 phase shift processing in step (1408), that is, pixel data in a range where a finite number of taps acts, need to be motion-compensated as neighboring pixels. The motion compensation operation is the same as the operation described with reference to FIGS.

  Subsequently, in step (1419), the phase is shifted by a certain amount with respect to the frame buffer # 1 and the motion compensated frame buffer # 2. That is, in steps (1407) and (1408), the pixel data in each frame buffer is phase-shifted by π / 2.

  Subsequently, in step (1420), the coefficients C0, C1, C2, and C3 set so as to satisfy the conditions of FIGS. 9A, 9B, and 9C based on the phase difference θ are used. ) Are multiplied and added to each other, thereby removing the aliasing component from the pixel data of the frame buffers # 1 and # 2, and outputting the result to the frame buffer # 3. That is, in step (1409), coefficients C0, C1, C2, and C3 are determined based on the phase difference θ, and each coefficient and frame buffer # 1 are determined in steps (1410), (1411), (1412), and (1413). , The pixel data of # 2 and the data after the π / 2 phase shift are respectively multiplied, and then all are added in step (1414) and output to frame buffer # 3. The operation for removing the aliasing component is the same as that described with reference to FIG.

  Subsequently, in step (1415), it is determined whether or not the processing of all the pixels in the frame buffer # 1 has been completed.If the processing has not been completed, the next pixel (for example, the right adjacent pixel) is determined in step (1416). The process is set as a process target and the process returns to step (1405) and the subsequent steps.

  After the image signal processing of the flowchart shown in FIG. 14, the signal buffered in the frame buffer # 3 shown in FIG. 18 can be output to the display unit (3) in frame units or pixel units.

  By performing the processing as described above, it is possible to output a high resolution signal to the frame buffer # 3 using the pixel data of the frame buffer # 1 and the frame buffer # 2. When applied to a moving image, the process from step (1401) to step (1417) may be repeated for each frame.

  As for the image signal processing method according to the second embodiment, as in the description of FIG. 17, the difference in the operation of the above prior art can be confirmed. Is omitted.

  According to the image signal processing method according to the second embodiment described above, phase shift is performed on each image signal of two input image frames, which is smaller than in the conventional example, and two signals are generated from each image signal. . Thereby, four signals can be generated from the image signals of the two input image frames. Here, based on the phase difference between the two input image frames, for each of the four signals, a coefficient for canceling and combining the aliasing components of the four signals is calculated for each pixel. For each pixel of the image to be generated, a sum is calculated by multiplying the pixel value of the corresponding pixel included in each of the four signals by each coefficient to generate a pixel value of a new high-resolution image. By performing this for each pixel of the generated image, a new high-resolution image can be generated.

  As a result, the image signal processing method according to the second embodiment can generate an image with less aliasing components and higher resolution than the input image using two input image frames that are fewer than those in the conventional example.

  In addition, since the image signal processing method according to the second embodiment uses two smaller input image frames than the conventional example, the amount of necessary image processing can be reduced compared to the conventional example.

  FIG. 10 shows a third embodiment of the present invention. The configuration shown in the figure is a simplified version of the configuration shown in FIG. 1 by utilizing the relationship between the coefficients C0, C1, C2, and C3 shown in FIG. 9C. That is, since C0 = C2 = 1/2 and C1 = -C3 =-(1 + cosθ) / (2sinθ), frame # 1 after the up-rate and frame # 2 after the motion compensation / up-rate From these signals, an adder (1001) and a subtracter (1004) generate sum and difference signals. The sum signal passes through the fs cutoff filter (1002), is then multiplied by C0 (= 0.5) by the multiplier (1003), and is input to the adder (1008). Here, the fs cutoff filter (1002) is a filter that cuts off the component of the sampling frequency (fs) before the up-rate as a zero point, and can be realized by using, for example, a tap coefficient indicated by (1011) in FIG. This fs cutoff filter (1002) cannot remove the aliasing component because the gain of the frequency fs is zero due to the `` frequency-gain '' characteristic of the Hilbert transformer (1005) as shown in FIG. The purpose is to prevent an unnecessary component having a frequency fs from remaining. Therefore, if a means capable of π / 2 phase shift including the frequency fs component is used instead of the Hilbert transformer (1005), the fs cutoff filter (1002) becomes unnecessary.

  On the other hand, for the difference signal, the coefficient determined by the coefficient determiner (1007) based on the phase difference (102) after shifting the phase by a certain amount (= π / 2) by the Hilbert transformer (1005) C1 is multiplied by a multiplier (1006) and added by an adder (1008) to obtain an output. Here, the phase shift unit (1009) including the delay unit (1002) and the Hilbert transformer (1005) can be realized with a circuit scale that is half that of the phase shift unit (116) shown in FIG. The coefficient determiner (1007) only needs to output the coefficient C1 shown in FIG. 9 (c). The adder (1001), the subtracter (1004), the multipliers (1003) (1006), the adder ( 1008) and the aliasing component removal unit (1010) including the coefficient determiner (1007) can reduce the number of multipliers, and thus can be realized with a smaller circuit scale than the aliasing component removal unit (117) shown in FIG. .

  As for the image signal processing method according to the third embodiment, as in the description of FIG. 17, the difference in the operation of the above prior art can be confirmed. Is omitted.

  The image signal processing apparatus and the image signal processing method according to the third embodiment can also be applied to increase the resolution in the vertical direction and the oblique direction.

  In addition to the effects of the image signal processing apparatus according to the first embodiment, the image signal processing apparatus according to the third embodiment described above can be realized with a smaller circuit scale than the image signal processing apparatus according to the first embodiment, and can be realized at a lower cost. Can be realized.

  An image signal processing method according to the fourth embodiment of the present invention will be described with reference to FIG.

  The fourth embodiment relates to an image signal processing method in which processing equivalent to image signal processing in the image signal processing apparatus according to the third embodiment is realized by a control unit that cooperates with software. The image processing apparatus that performs the image signal processing method of the present embodiment is the image processing apparatus shown in FIG.

  FIG. 15 shows an example of a flowchart of the operation of the present embodiment. The flowchart of FIG. 15 starts from step (1501), and in step (1518), the image data of each frame is updated. That is, in step (1502), the image data of frame # 1 is updated and written to frame buffer # 1, and in step (1503), the image data of frame # 2 is updated and written to frame buffer # 2. Here, the up-rate can be realized by clearing the value of each frame buffer to 0 and then writing the data every other pixel.

  Next, in step (1504), the first pixel (for example, the upper left pixel) of frame buffer # 1 is set as the processing target, and processing is performed until all pixel data in frame buffer # 1 is processed. Loop.

  In step (1505), the position of the corresponding pixel in the frame buffer # 2 is estimated with reference to the target pixel in the frame buffer # 1, and the phase difference θ is output. At this time, the above-described conventional technique can be used as it is as a method of estimating the position of the corresponding pixel.

  In step (1506), based on the phase difference θ obtained in step (1505), pixels near the corresponding pixel in the frame buffer # 2 are motion compensated. At this time, as the “neighboring pixels”, only the pixel data used in the Hilbert transform processing in step (1510), that is, the pixel data in the range where the finite number of taps acts is compensated for motion. The motion compensation operation is the same as the operation described with reference to FIGS.

  Subsequently, in step (1520), the aliasing component is removed from the pixel data of the frame buffers # 1 and # 2 based on the phase difference θ and output to the frame buffer # 3. First, in step (1507), the value of pixel data in frame buffer # 1 and the value of pixel data in frame buffer # 2 subjected to motion compensation are added. In step (1509), the frequency fs component is cut off. The operation of this fs cutoff filter (1509) is the same as the operation of (1002) shown in FIG.

  In step (1508), the pixel data value of the frame buffer # 2 subjected to motion compensation is subtracted from the pixel data value of the frame buffer # 1. Here, with respect to the subtraction result, the phase is shifted by a certain amount in step (1519). That is, the Hilbert transform is performed in step (1510) using the neighboring data similarly subtracted. The phase shift operation is the same as the operation described with reference to FIGS.

  Subsequently, in step (1511), the data after the addition is multiplied by a coefficient C0 (= 0.5), and in step (1512), a coefficient C1 is determined based on the phase difference θ, and then in step (1513). After multiplying the coefficient C1 by the Hilbert transformed data, the two data are added in step (1514) and output to the frame buffer # 3. The operation of removing the aliasing component is the same as that described with reference to FIG.

  Subsequently, in step (1515), it is determined whether or not the processing of all the pixels in the frame buffer # 1 has been completed.If the processing has not been completed, the next pixel (for example, the right adjacent pixel) is determined in step (1516). The process is set as a process target, and the process returns to step (1505) and subsequent steps. If completed, the process ends at step (1517).

  After the image signal processing of the flowchart shown in FIG. 15, the signal buffered in the frame buffer # 3 shown in FIG. 18 can be output to the display unit (3) in frame units or pixel units.

  By performing the processing as described above, it is possible to output a high resolution signal to the frame buffer # 3 using the pixel data of the frame buffer # 1 and the frame buffer # 2. When applied to a moving image, the process from step (1501) to step (1517) may be repeated for each frame.

  The image signal processing method according to the fourth embodiment can also be confirmed by using FIG. 17 to confirm the difference in operation of the above prior art, but the result is the same as that of the first embodiment, and the description thereof is omitted. To do.

  The image signal processing method according to the fourth embodiment can also be applied to increase the resolution in the vertical direction and the oblique direction.

  The image signal processing method according to the fourth embodiment described above has the same effect of increasing the resolution of the image signal as the image signal processing method according to the second embodiment. Furthermore, the image signal processing method according to the fourth embodiment is less than the image signal processing method according to the second embodiment by sharing the contents of some processing steps as compared with the image signal processing method according to the second embodiment. There is an effect that it is possible to realize the same signal processing with a processing amount (the number of operations).

  FIG. 11 shows a fifth embodiment of the present invention. In the configuration shown in FIG. 9, the coefficients C1 and C3 become indefinite when the phase difference θ is 0 as shown in FIG. 9D, and the coefficients C1 and C3 increase as the phase difference θ approaches 0. Therefore, based on the configuration shown in FIG. 10, when the phase difference θ becomes close to 0, the output from the auxiliary pixel interpolation unit (1105) is switched. It is composed. In other words, a general interpolation low-pass filter (1101) is prepared as a bypass path, and in addition to the above-described coefficients C0 and C1 in the coefficient determiner (1103), C4 is newly generated, and the multiplier (1102) The output of the interpolation low-pass filter (1101) is multiplied by the coefficient C4, and the result is output in addition to the signal whose resolution has been increased by the adder (1104).

  The configuration other than the interpolation low-pass filter (1101), multiplier (1102), coefficient determiner (1103), adder (1104), and auxiliary pixel interpolation unit (1105) is the same as that of the third embodiment shown in FIG. Since it is the same, description is abbreviate | omitted.

  FIG. 12 shows a specific example of the interpolation low-pass filter (1101) used in the fifth embodiment of the present invention. This figure shows the tap coefficient of the filter obtained by inverse Fourier transform of the frequency characteristic with the cutoff frequency being 1/2 of the original sampling frequency fs. At this time, each tap coefficient Ck (where k is an integer) becomes a general sinc function, and may be Ck = sin (πk / 2) / (πk / 2).

  FIG. 13 shows a specific example of the coefficient determiner (1103) used in the fifth embodiment of the present invention. This figure is based on the coefficients C0 and C1 shown in Fig. 9 (d), and normally the new coefficient C4 is 0, but when the phase difference θ is close to 0, the value of the coefficient C1 is forced. The operation of setting the coefficient C4 to 1.0 and setting the coefficient C4 to 1.0 is shown. This operation automatically switches the output of the adder (1104) to the output of the interpolation low-pass filter (1101) when the phase difference θ (102) becomes 0 or close to 0 in the configuration shown in FIG. Will be able to. Note that, as the phase difference θ approaches 0, the coefficient shown in FIG. 12 may be gradually and gradually approximated to the coefficient shown in FIG. Also, when the position estimation unit (101) in FIG. 1 determines that the pixel corresponding to the pixel to be processed on the frame # 1 is not on the frame # 2, the phase difference θ (102) is close to zero. As in the case of the above, each coefficient may be controlled to automatically switch the output of the adder (1104) to the output of the interpolation low-pass filter (1101).

  Note that the image signal processing apparatus according to the fifth embodiment can also confirm the difference in operation of the above-described conventional technique with reference to FIG. 17, but the result is the same as that of the first embodiment, and thus the description thereof is omitted. To do.

  The image signal processing apparatus according to the fifth embodiment can also be applied to increase the resolution in the vertical direction and the oblique direction.

  In addition to the effects of the image signal processing device according to the third embodiment, the image signal processing device according to the fifth embodiment described above has a phase difference θ (102) of 0 or 0 as compared with the image signal processing device according to the third embodiment. The processing result will not be undefined even when it is close to 0 (i.e., still or almost stationary) or when it is determined that the pixel corresponding to the pixel to be processed on frame # 1 is not on frame # 2. This has the effect that a stable output image can be obtained.

  An image signal processing method according to the sixth embodiment of the present invention will be described with reference to FIG.

  Example 6 relates to an image signal processing method in which processing equivalent to image signal processing in the image signal processing apparatus according to Example 5 is realized by a control unit that cooperates with software. The image processing apparatus that performs the image signal processing method of the present embodiment is the image processing apparatus shown in FIG.

  FIG. 16 shows an example of a flowchart of the operation of the present embodiment. As shown in FIG. 9 (d), the processing steps shown in FIG. 9 are performed when the coefficients C1 and C3 become indefinite when the phase difference θ is 0, and as the phase difference θ approaches 0, the coefficient C1 In order to prevent the noise from being weakened by increasing C3, the step (1606) is performed when the phase difference θ becomes 0 or near 0 based on the steps of FIG. 15 described in the fourth embodiment. This processing result is output to the frame buffer # 3. That is, coefficients C0, C1, and C4 are determined based on the phase difference θ in step (1601), and the target pixel data in frame buffer # 1 and its neighboring pixel data are used in step (1602). After performing general interpolation low-pass filter processing, the coefficient C4 is multiplied in step (1603), and the result is added to the outputs of steps (1511) and (1513) in step (1604) to obtain frame buffer # 3. Output to.

  Steps other than these are the same as the processing steps of FIG. 15 described in the fourth embodiment, and a description thereof will be omitted. The coefficient determination operation in step (1601) is the same as the operation shown in FIG. The operation of the interpolation low-pass filter in step (1602) is the same as that shown in FIG.

  Note that the image signal processing method according to the sixth embodiment can also be confirmed by using FIG. 17 to confirm the difference in operation of the above-described conventional technique, but the result is the same as that of the first embodiment, and thus the description thereof is omitted. To do.

  Note that the image signal processing method according to the sixth embodiment can also be applied to increase the resolution in the vertical and oblique directions.

  In the image signal processing method according to the sixth embodiment described above, in addition to the effect of the image signal processing method according to the fourth embodiment, the phase difference θ (102) is 0 or 0 as compared with the image processing method according to the fourth embodiment. Even when it is determined that the pixel corresponding to the pixel to be processed on frame # 1 is not on frame # 2 when it is close (i.e., still or almost still), the processing result is not uncertain, This has the effect that a stable output image can be obtained.

  FIG. 20 shows an image signal processing apparatus according to Embodiment 7 of the present invention. The image processing apparatus according to the present embodiment includes, for example, an input unit (1) to which a frame sequence of a moving image such as a television broadcast signal is input, and a frame input from the input unit (1) in the horizontal and vertical directions. A resolution conversion unit (4) for increasing the combined two-dimensional resolution and a display unit (3) for displaying an image based on the frame whose resolution has been increased by the resolution conversion unit (4) are provided. .

  In this resolution conversion unit (4), by performing resolution conversion processing in each of the horizontal direction and the vertical direction, a component having a large resolution improvement effect is selectively or mixedly output from each result, thereby two-dimensionally. Realize higher resolution. Details of the resolution conversion unit (4) will be described below.

  In FIG. 20, based on frame # 1 (2010) and frame # 2 (2013) input to the input unit (1), a horizontal resolution conversion unit (2001) and a vertical resolution conversion unit (2005) are used. A frame (2011) with an increased number of pixels in the horizontal direction and a frame (2014) with an increased number of pixels in the vertical direction are generated.

  Here, each resolution conversion unit (2001) (2005) uses the configuration of the resolution conversion unit (2) of the image signal processing apparatus according to the first embodiment of the present invention shown in FIG. Each signal processing is performed. At this time, in the horizontal resolution converter (2001), the up-raters (103) (104), delay units (105) (107), and π / 2 phase shifters (106) (108) shown in FIG. Each is configured to perform horizontal up-rate, delay, and π / 2 phase shift.

  Similarly, in the vertical resolution converter (2005), the up-raters (103) (104), delay units (105) (107), and π / 2 phase shifters (106) (108) shown in FIG. Each is configured to perform vertical uprate, delay, and π / 2 phase shift. These can be implemented using the operations shown in FIGS. 5 to 8 and the prior art.

  Each resolution conversion unit (2001) (2005) includes, in place of the configuration of the resolution conversion unit of the image signal processing apparatus according to the first embodiment of the present invention, the image signal processing apparatus according to the third embodiment of the present invention. The present invention can also be realized by using the resolution conversion unit and the resolution conversion unit of the image signal processing apparatus according to the fifth embodiment of the present invention. The following description will be made assuming that the configuration of the resolution conversion unit of the image signal processing apparatus according to the first embodiment of the present invention is used.

  In the present embodiment, assuming that the subject has moved two-dimensionally in the horizontal and vertical directions, the operations shown in FIGS. 1 and 2 are extended two-dimensionally. That is, the position estimation unit ((101) in FIG. 1) and the motion compensation / uprate unit ((115) in FIG. 1) in the horizontal resolution conversion unit (2001) use the subject on the frame # 1 as a reference. As described above, the subject on the frame # 2 is compensated for two-dimensional motion, and the horizontal phase difference θH among the sampling phase differences of the pixels of each frame is determined for the coefficient of the aliasing component removal unit ((117) in FIG. 1). Use.

  Similarly, in the position estimation unit ((101) in FIG. 1) and the motion compensation / uprate unit ((115) in FIG. 1) in the vertical resolution conversion unit (2005), the subject on frame # 1 ( 2016) as a reference, the subject (2017) on frame # 2 is two-dimensionally motion compensated, and among the sampling phase differences of the pixels of each frame, the vertical phase difference θV is the aliasing component removal unit (in FIG. Used to determine the coefficient of 117)). To determine the coefficient of the aliasing component removal unit ((117) in FIG. 1), the operation shown in FIG. 9 may be used as it is.

  Assuming that the subject has moved in an oblique direction, the frame (2011) in which the number of pixels in the horizontal direction has been increased by the horizontal resolution converter (2001) will contain diagonal distortion, but the original input In a component with a low vertical frequency of the signal (such as a vertical line), this distortion is small enough to be ignored. Similarly, the frame (2014) in which the number of pixels in the vertical direction has been increased by the vertical resolution converter (2005) will include diagonal distortion, but the original input signal has a low horizontal frequency component (horizontal line). Etc.), this distortion is small enough to be ignored.

  Using this characteristic, the frame (2011) in which the number of pixels in the horizontal direction is increased according to the signal processing described above is converted into a frame by the vertical interpolation unit (2004) composed of the vertical uprater (2002) and the pixel interpolator (2003). (2012) is generated and used as the SR (horizontal) signal. Here, the pixel interpolator (2003) may use a general vertical low-pass filter that outputs an average value of upper and lower pixel data of a pixel to be interpolated. Similarly, a frame (2014) in which the number of pixels in the vertical direction is increased is generated by a horizontal interpolation unit (2008) including a horizontal up-rate unit (2006) and a pixel interpolator (2007), and a frame (2015) is generated. (Vertical) signal. Here, the pixel interpolator (2007) may use a general horizontal low-pass filter that outputs an average value of the left and right pixel data of the pixel to be interpolated.

  In this way, if the pixel interpolator (2003) (2007) is used to remove the high-frequency component in the direction orthogonal to the direction of the processing target and extract only the low-frequency component, when moving in the oblique direction described above, The influence of the generated distortion can be made small enough to be ignored. The SR (horizontal) signal and the SR (vertical) signal generated by the above processing are mixed by the mixer (2009) to be an output signal and displayed on the display unit (3).

  Here, the detailed configuration and operation of the mixer (2009) will be described. The mixer (2009) may use any of the following three configuration examples.

  FIG. 22 shows a first configuration example of the mixer (2009). In the figure, an adder (2201) and a multiplier (2202) are used to generate and output an average value of SR (horizontal) and SR (vertical) signals input to the mixer (2009). In the configuration shown in the figure, the horizontal and vertical resolution improvement effects are also halved, but the mixer (2009) is the simplest configuration and can be realized at low cost.

  FIG. 23 shows a second configuration example of the mixer (2009). In the figure, for the SR (horizontal) and SR (vertical) signals input to the mixer (2009), a multiplier (2303) and a multiplier (2304) are used to calculate a coefficient K (horizontal) and a coefficient K. Multiply each by (vertical) and add both by the adder (2305) to obtain the output. The coefficient K (horizontal) and the coefficient K (vertical) are generated by coefficient determiners (2301) and (2302), respectively. The operation of the coefficient determiners (2301) and (2302) will be described below.

  The aliasing component removal units (2108) and (2109) shown in FIG. 21 are connected to the coefficient determiner (109) shown in FIG. 1 based on the phase difference θH (2102) and the phase difference θV (2103) shown in FIG. The coefficients C0 to C3 shown in FIG. 9 are generated to perform the aliasing component removal calculation. At this time, the coefficients C1 and C3 become indefinite when the phase differences θH (2102) and θV (2103) are 0, and the coefficients C1 and C3 are increased as the phase differences θH (2102) and θV (2103) approach 0. In order to prevent it from becoming vulnerable to noise and the like by increasing the coefficient, the coefficient C4 (0 ≦ C4 ≦ 1) shown in FIG. 13 is introduced, and auxiliary pixel interpolation is performed as in the configuration shown in FIG. preferable. In other words, the effect of improving the resolution is obtained when the value of the coefficient C4 is 0.0, but the effect of improving the resolution becomes smaller as the value of the coefficient C4 approaches 1.0.

  Using this property, when the horizontal phase difference θH (2102) is near 0 (that is, the coefficient C4 (horizontal) is near 1.0), the SR (vertical) of the vertical resolution conversion result is strongly reflected, and the vertical phase difference θV ( 2103) is close to 0 (i.e., the coefficient C4 (vertical) is close to 1.0), so that the horizontal resolution conversion result SR (horizontal) is strongly reflected, using the values of the coefficient C4 in the horizontal and vertical directions respectively. A coefficient K (horizontal) and a coefficient K (vertical) are determined. In order to realize this operation, for example, the coefficient determiner (2301) shown in FIG. 23 performs an operation of K (horizontal) = C4 (horizontal) + (1-C4 (vertical)) / 2 to determine K (horizontal). The coefficient determiner (2303) determines K (vertical) by performing an operation of K (vertical) = C4 (vertical) + (1-C4 (horizontal)) / 2.

  FIG. 24 summarizes an example of the outputs (coefficient K (horizontal) and coefficient K (vertical)) of the coefficient determiners (2301) and (2302) when the coefficient C4 (horizontal) and the coefficient C4 (vertical) are changed. Show. As shown in the figure, when the coefficient C4 (horizontal) increases, the coefficient K (horizontal) decreases and the coefficient K (vertical) increases, and when the coefficient C4 (vertical) increases, the coefficient K (horizontal) increases. It operates so that the coefficient K (vertical) becomes small.

  When the values of the coefficient C4 (horizontal) and the coefficient C4 (vertical) are equal, the coefficient K (horizontal) and the coefficient K (vertical) are each 0.5. In this way, for the coefficient C4 that changes independently in horizontal and vertical, the coefficient K is determined to be just 1.0 by adding the coefficient K (horizontal) and the coefficient K (vertical), and SR (horizontal) and Mix SR (vertical).

  The third operation and configuration example of the mixer (2009) will be described using FIG. 25 and FIG. 26, respectively. FIG. 25 shows a two-dimensional frequency region in which the horizontal frequency is represented by μ and the vertical frequency is represented by ν. Assuming that the horizontal sampling frequency of the original input image is μs and the vertical sampling frequency is νs, the output of the resolution conversion unit (4) shown in FIGS. 20 and 21 is a vertical frequency in the range of −μs to μs. The frequency ν is a signal in the range of −νs to νs.

  High-frequency components are reproduced by horizontal and vertical resolution conversions, but the high-frequency components originally have a low signal level, so the effect of horizontal resolution conversion is large (μ, ν) = (± μs / 2, 0) component in the frequency region (2501) in the vicinity (particularly (μ, ν) = (+ μs / 2, 0)), a region of a frequency where μ> 0, and (μ, ν) = (− μs / 2, 0), and the component of the frequency region where μ <0), and the effect of vertical resolution conversion is large in the frequency region in the vicinity of (μ, ν) = (0, ± νs / 2) ( 2502) components (particularly including (μ, ν) = (0, + νs / 2), including a frequency region where ν> 0, and (μ, ν) = (0, -νs / 2), component in a frequency region where ν <0).

  Therefore, when these frequency components (2501) and (2502) are extracted and mixed by a two-dimensional filter, components having a large resolution improvement effect can be selectively output.

  FIG. 26 shows a configuration example of a mixer (2009) that extracts components having a large effect by horizontal and vertical resolution conversion. In the figure, a two-dimensional filter (2601) is used to extract a component in the frequency region (2501) having a large resolution improvement effect of SR (horizontal) input to the mixer (2009). Similarly, using the two-dimensional filter (2602), a component in the frequency region (2502) having a large resolution improvement effect of SR (vertical) input to the mixer (2009) is extracted.

  As a component other than the frequency domain (2501) (2502), an adder (2603) and a multiplier (2604) are used to create an average signal of SR (horizontal) and SR (vertical), and a two-dimensional filter (2605) The components other than the passbands of the two-dimensional filters (2601) and (2602) (that is, the remaining components) are extracted. The output signals of the two-dimensional filters (2601), (2602), and (2605) are added by the adder (2606) to obtain the output of the mixer (2009).

Note that the numbers in circles in the two-dimensional filters (2601), (2602), and (2605) shown in the figure indicate examples of the tap coefficients of the respective filters. (The coefficients of each filter are expressed as integers for the sake of simplicity. The original coefficient values are indicated by a circled number and “× 1/16” on the right). For example, in the two-dimensional filter (2601), the original coefficient value is obtained by multiplying each circled number by 1/16. Same for coefficient.)
The two-dimensional filter (2601) is the product of a horizontal bandpass filter and a vertical lowpass filter with ± μs / 2 as the center frequency of the passband, and the two-dimensional filter (2602) has ± νs / 2 as the center frequency of the passband. The product of the vertical bandpass filter and the horizontal lowpass filter may be used, and the two-dimensional filter (2605) may have a characteristic obtained by subtracting the passbands of the two-dimensional filter (2601) and the two-dimensional filter (2602) from the entire band.

  Next, the difference between the processing in the image signal processing apparatus according to the seventh embodiment and the operation of the conventional technique will be described with reference to FIG. The figure (a) shows frame # 1 (3401), frame # 2 (3402), frame # 3 (3403), frame # 4 (3404), frame # 5 (3405) input to the resolution converter (4). (B) shows each frame output from the resolution converter (4). In each frame, the subject moves clockwise by 1/4 pixel, and the subject is intentionally moved so as to make one round in four frames. This movement is continued in the same manner after frame # 6.

  In the prior art described in Patent Document 1, Patent Document 2, and Non-Patent Document 1, as described above, when the resolution is increased with respect to a horizontal / vertical two-dimensional input signal, the aliasing comes from two directions. So, if the bandwidth of the original signal is doubled both vertically and horizontally, the three aliasing components overlap, and 2M + 1 = 7 digital data (= 7 frame image signals) is required to cancel them. It was. Therefore, when a signal that circulates in 4 frames as shown in FIG. 34 (a) is input, independent data cannot be obtained no matter which 7 frames are selected, and the solution by the high resolution processing becomes indefinite. It is not required.

  On the other hand, if this embodiment is used, for example, two adjacent frames (for example, frame # 1 (3401) and frame # 2 (3402), (or frame # 2 (3402) and frame # 3 (3403))) are used. Thus, as shown in FIG. 5B, the horizontal (or vertical) aliasing component can be removed to achieve high resolution. That is, the operation status of the present embodiment can be confirmed by using the input image of FIG. If a generally well-known circular zone plate (CZP: Circular Zone Plate) is used as the pattern of this test pattern, the effect of resolution conversion can be directly viewed on the display unit (3). That is, if the circular zone plate is moved left and right for each frame, an image with improved horizontal resolution is displayed, and if it is moved up and down (or diagonally), an image with improved vertical (or diagonally) resolution is displayed. The effect of improving the resolution according to the moving direction of the test pattern can be confirmed.

  According to the image signal processing apparatus according to the seventh embodiment described above, phase shift is performed on each image signal of two input image frames, and two signals are generated from each image signal. As a result, four signals are generated from the image signals of the two input image frames. Here, based on the phase difference between the two input image frames, for each of the four signals, a coefficient for canceling and combining the aliasing components of the four signals is calculated for each pixel. For each pixel of the image to be generated, a sum is calculated by multiplying the pixel value of the corresponding pixel included in each of the four signals by each coefficient to generate a pixel value of a new high-resolution image. By performing this for each pixel of the generated image, an image having a higher resolution in the one-dimensional direction than the input image frame is generated.

  This is performed in each of the horizontal direction and the vertical direction, and an image with high resolution in the horizontal direction and an image with high resolution in the vertical direction are generated. The image having the high resolution in the horizontal direction and the image having the high resolution in the vertical direction are subjected to up-rate processing in the vertical direction and the horizontal direction, respectively, and then both are mixed.

  As a result, it is possible to generate a high-resolution image in which the resolution is increased in both the vertical direction and the horizontal direction from the image signals of two input image frames that are smaller than in the conventional example. That is, a two-dimensional high resolution image can be generated.

  Further, since the image signal processing apparatus according to the seventh embodiment uses two input image frames that are fewer than those in the conventional example, the amount of necessary image processing can be reduced as compared with the conventional example. Accordingly, it is possible to realize an image signal processing apparatus that generates a high-resolution image with less aliasing components and higher resolution than the input image in both the vertical and horizontal directions at a lower cost than the conventional example. .

  In addition, according to the prior art described in Patent Document 1, Patent Document 2, and Non-Patent Document 1, one-dimensional high resolution such as a horizontal direction and a vertical direction is performed in a plurality of directions using three frames. These results may be input to the mixer (2009) shown in FIG. 20 and mixed, and output as a two-dimensional resolution conversion result. In this case, as shown in FIG. 20, the scale of the signal processing circuit such as the frame memory or the motion estimation unit is larger than the configuration in which the two-dimensional resolution conversion is performed using only two frames. 2. As described in Non-Patent Document 1, it is possible to reduce the scale of a signal processing circuit such as a frame memory or a motion estimation unit, compared to using a signal of at least 7 frames.

  Further, the present invention is not limited to the conventional techniques described in Patent Document 1, Patent Document 2, and Non-Patent Document 1, and other one-dimensional high resolution such as a horizontal direction and a vertical direction is applied by applying a conventional high resolution technology. It is also possible to perform conversion into a plurality of directions, input each result into a mixer (2009) shown in FIG. 20, and mix and output the result as a two-dimensional resolution conversion result.

  In FIG. 20, the case where the resolution of the frame # 1 is converted using the set of the input signals of the frame # 1 and the frame # 2 has been described as an example. # 3, Frame # 1 and Frame # 4, etc. are used to convert the resolution of frame # 1 and mix the results to obtain the final resolution conversion result of frame # 1. .

  As a mixing method at this time, an average value of the results may be taken, or mixing may be performed according to the value of the coefficient C4 (frame) for each frame as shown in FIGS. In this case, as the coefficient C4 (frame), the MAX value (the smaller value) of the coefficient C4 (horizontal) and the coefficient C4 (vertical) for each frame may be used. Also, all the coefficients C4 (horizontal) and C4 (vertical) are compared for each pixel, and the resolution conversion result obtained from the group with the smallest coefficient C4 (i.e., the group with the greatest resolution improvement effect) is obtained for each pixel. The final resolution conversion result of frame # 1 may be selected.

  Thus, for example, with frame # 1 as a reference, if frame # 2 is a frame earlier than frame # 1 and frame # 3 is a future frame than frame # 1, the subject is When changing from `` motion '' to `` still '' (end of movement), resolution conversion processing is performed with frame # 1 and frame # 2, and when the subject is converted from `` still '' to `` motion '' at the time of frame # 1 ( (Beginning of motion) Since each processing result is mixed for each pixel so that resolution conversion processing is performed by frame # 1 and frame # 3, the motion of the subject can be used effectively, and the resolution improvement effect is the largest can do.

  FIG. 21 shows an image signal processing apparatus according to Embodiment 8 of the present invention. The image processing apparatus according to the present embodiment is a modification of the configuration of the seventh embodiment described above, and the processing order of the resolution conversion units (2001) (2005) and the interpolation units (2004) (2008) shown in FIG. On the other hand, the resolution conversion is performed after the interpolation process. As a result, the up-rater ((103) (104) in FIG. 1) in the resolution conversion unit (2001) (2005) and the up-rater in the interpolation unit (2004) (2008) (FIG. 20 (2002) (2006)) and the horizontal resolution conversion unit (2001) and the vertical resolution conversion unit (2005), each position estimation unit ((101) in FIG. 1) Since they can be shared, similar signal processing can be realized with a smaller circuit scale and operation amount.

  In FIG. 21, first, the position estimation unit (2101) performs a corresponding operation on the frame # 2 with reference to the sampling phase (sampling position) of the pixel to be processed on the frame # 1 input to the input unit (1). The pixel position is estimated, and the sampling phase differences θH (2102) and θV (2103) in the horizontal and vertical directions are obtained.

  Next, the motion compensation / uprate unit (2110) uses the uprater (2104) (2105) to perform motion compensation on the frame # 2 using the information on the phase differences θH (2102) and θV (2103) to In addition to matching the position with 1, the number of pixels in frame # 1 and frame # 2 is doubled both horizontally and vertically (4 times in total) to increase the density. The up-raters (2104) and (2105) are obtained by extending the operation / configuration shown in FIGS. 5 and 6 to two dimensions in the horizontal and vertical directions. The phase shift unit (2111) shifts the phase of the densified data by a certain amount.

  At this time, the horizontal phase shifter (2106) performs a horizontal phase shift, and the vertical phase shifter (2107) performs a vertical phase shift. The delay units (105) and (107) shown in FIG. ) And π / 2 phase shifter (108) and the operations and configurations shown in FIGS.

  For each phase-shifted signal, horizontal and vertical folding components are removed by the horizontal folding component removal unit (2108) and vertical folding component removal unit (2109) in the folding component removal unit (2112), respectively. . Next, the output of the horizontal aliasing component removal unit (2108) is pixel-interpolated using the pixel interpolator (2003) to obtain an SR (horizontal) signal, and the output of the vertical aliasing component removal unit (2109) is the pixel. Pixel interpolation is performed using an interpolator (2007) to form an SR (vertical) signal, which is mixed by an mixer (2009) and output.

  The folded component removing units (2108) and (2109) can use the configuration of the folded component removing unit (117) shown in FIG. 1 as it is. As the phase difference θ (102), the folded component removing unit (2108) uses the horizontal phase difference θH (2102), and the folded component removing unit (2109) uses the horizontal phase difference θH (2103) to perform the operation shown in FIG. By performing the above, folding components in the respective directions can be removed.

  In the above description, the phase shift unit (2111) includes the delay units (105) and (107) and the π / 2 phase shifter (108) shown in FIG. 1 and the operations and configurations shown in FIGS. The aliasing component removal units (2108) and (2109) are performed in the same manner as described above, and the configuration of the aliasing component removal unit (117) shown in FIG. 1 is used as it is, but instead, the phase shift unit (2111) The phase shift unit (1009) of FIG. 10 is used for the vertical direction and the horizontal direction, respectively, and the aliasing component removal units (2108) and (2109) may also use the aliasing component removal means (1010) of FIG. Good. Further, at this time, each of the aliasing component removal units (2108) and (2109) may be provided with the auxiliary pixel interpolation unit (1105) of FIG. 11 as in FIG.

  Note that the mixer (2009) is the same as that of the seventh embodiment, and a description thereof will be omitted.

  Also, the operation for the input frame shown in FIG. 34 is the same as that of the seventh embodiment, and thus the description thereof is omitted.

  The image signal processing apparatus according to the eighth embodiment described above has the same effects as the image signal processing apparatus according to the seventh embodiment, but a part of the processing units is shared as compared with the image signal processing apparatus according to the seventh embodiment. By doing so, it is possible to realize the same signal processing with a circuit scale and a calculation amount smaller than those of the image signal processing apparatus according to the seventh embodiment.

  In addition, according to the prior art described in Patent Document 1, Patent Document 2, and Non-Patent Document 1, one-dimensional high resolution such as a horizontal direction and a vertical direction is performed in a plurality of directions using three frames. These results may be input to the mixer (2009) shown in FIG. 21 and mixed, and output as a two-dimensional resolution conversion result. In this case, as shown in FIG. 21, the scale of the signal processing circuit such as the frame memory and the motion estimation unit is larger than the configuration in which the two-dimensional resolution conversion is performed using only two frames. 2. As described in Non-Patent Document 1, it is possible to reduce the scale of a signal processing circuit such as a frame memory or a motion estimation unit, compared to using a signal of at least 7 frames.

  In addition, the present invention is not limited to the conventional techniques described in Patent Document 1, Patent Document 2, and Non-Patent Document 1, and other conventional high resolution techniques are applied to apply a one-dimensional height such as a horizontal direction or a vertical direction. Resolution may be performed in a plurality of directions, and each result may be input to the mixer (2009) shown in FIG. 21 and mixed, and output as a two-dimensional resolution conversion result.

  In FIG. 21, the case where the resolution of the frame # 1 is converted using the set of the input signals of the frame # 1 and the frame # 2 has been described as an example. # 3, Frame # 1 and Frame # 4, etc. are used to convert the resolution of frame # 1 and mix the results to obtain the final resolution conversion result of frame # 1. .

  As a mixing method at this time, an average value of the results may be taken, or mixing may be performed according to the value of the coefficient C4 (frame) for each frame as shown in FIGS. In this case, as the coefficient C4 (frame), the MAX value (the smaller value) of the coefficient C4 (horizontal) and the coefficient C4 (vertical) for each frame may be used. Also, all the coefficients C4 (horizontal) and C4 (vertical) are compared for each pixel, and the resolution conversion result obtained from the group with the smallest coefficient C4 (i.e., the group with the greatest resolution improvement effect) is obtained for each pixel. The final resolution conversion result of frame # 1 may be selected.

  Thus, for example, with frame # 1 as a reference, if frame # 2 is a frame earlier than frame # 1 and frame # 3 is a future frame than frame # 1, the subject is When changing from `` motion '' to `` still '' (end of movement), resolution conversion processing is performed with frame # 1 and frame # 2, and when the subject is converted from `` still '' to `` motion '' at the time of frame # 1 ( (Beginning of motion) Since each processing result is mixed for each pixel so that resolution conversion processing is performed by frame # 1 and frame # 3, the motion of the subject can be used effectively, and the resolution improvement effect is the largest can do.

  FIG. 27 shows an image signal processing apparatus according to Embodiment 9 of the present invention. The image processing apparatus according to the present embodiment has a configuration in which a high-resolution conversion unit for diagonal components in the lower right and upper right directions is added to the configuration example shown in FIG. That is, an oblique (lower right) phase shift unit (2701) and an oblique (upper right) phase shift unit (2702) are added to the phase shift unit (2708), and the aliasing component removal unit (2705) is added to the aliasing component removal unit (2709). ) (2706) and after passing through the pixel interpolators (2710) and (2711), the SR (horizontal), SR (vertical), SR (upper right) and SR (lower right) signals are mixed. Mix at (2707) and use as output. Here, the pixel interpolators (2710) and (2711) may use a general two-dimensional low-pass filter that outputs an average value of upper, lower, left, and right pixel data of a pixel to be interpolated.

  Phase difference information in the oblique direction is required as the phase difference θ, and the phase difference (θH + θV) obtained by adding the horizontal phase difference θH (2102) and vertical phase difference θV (2103) with the adder (2703) is removed. The phase difference (−θH + θV) generated by the subtracter (2704) may be input to the aliasing component removal unit (2706). The configurations and operations of the aliasing component removal units (2106), (2109), (2705), and (2706) are all the same.

  FIGS. 28A to 28D show a horizontal phase shift unit (2106), a vertical phase shift unit (2107), an oblique (lower right) phase shift unit (2701), and an oblique (upper right) phase shift in the two-dimensional frequency domain. Each operation of the unit (2702) is shown. FIGS. 28A to 28D are two-dimensional frequency regions in which the horizontal frequency is represented by μ and the vertical frequency is represented by ν, as in FIG. These phase shift units (2106) (2107) (2701) (2702) have the same configuration as the phase shift unit (116) shown in FIG. 1, and the π / 2 phase shifters (106) (108) therein ) “Frequency-phase difference” characteristics are changed according to each direction.

  That is, in the figure (a), in the horizontal phase shift unit (2106), when the horizontal frequency sampling frequency of the input signal is μs, the frequency in the range of −μs to 0 is the same as the operation shown in FIG. The phase of the component is shifted by π / 2, and the phase of the frequency component in the range of 0 to μs is shifted by -π / 2. Similarly, in the vertical phase shift unit (2107), when the vertical frequency sampling frequency of the input signal is νs, the phase of the frequency component in the range of −νs to 0 is shifted by π / 2, and the range of 0 to νs. The phase of the frequency component of is shifted by -π / 2.

  Similarly, in the oblique (lower right) phase shift unit (2701) and the oblique (upper right) phase shift unit (2702), as shown in FIG. Shift by / 2 or π / 2. These “frequency-phase difference” characteristics are arranged in the horizontal, vertical, diagonal (lower right) and diagonal (upper right) directions according to the two-dimensional sampling points. Can be easily realized.

  FIG. 29 shows a first configuration example of the mixer (2707). In the same figure, each of SR (horizontal), SR (vertical), SR (lower right), SR (upper right) input to the mixer (2707) using an adder (2901) and a multiplier (2902) Generate and output the average value of the signal. The configuration shown in the figure is an example of the simplest configuration of the mixer (2707), but the horizontal, vertical, lower right, and upper right resolution improvement effects are also each reduced to 1/4.

  FIG. 30 shows a second configuration example of the mixer (2707). In the figure, for each signal of SR (horizontal), SR (vertical), SR (bottom right), SR (top right) input to the mixer (2707), a multiplier (3005), a multiplier (3006) Using the multiplier (3007) and multiplier (3008), multiply by the coefficient K (horizontal), coefficient K (vertical), coefficient K (lower right) and coefficient K (upper right), respectively, and adder (3009) These signals are added to produce an output. The coefficient K (horizontal), the coefficient K (vertical), the coefficient K (lower right), and the coefficient K (upper right) are generated by coefficient determiners (3001) (3002) (3003) (3004), respectively. The operation of the coefficient determiners (3001) (3002) (3003) (3004) will be described below.

  The aliasing component removal units (2108), (2109), (2705), and (2706) shown in FIG. 27 have the phase difference θH (2102), phase difference θV (2103), phase difference (θH + θV), position shown in FIG. Based on the phase difference (−θH + θV), the coefficient determiner (109) shown in FIG. 1 generates coefficients C0 to C3 shown in FIG. At this time, when the phase differences θH (2102), θV (2103), (θH + θV), (−θH + θV) are 0, the coefficients C1 and C3 become indefinite, the phase difference θH (2102), In order to prevent the coefficients C1 and C3 from becoming large as θV (2103), (θH + θV), (−θH + θV) approaches 0, the coefficient C4 (0 ≦ C4 ≦ 1) is introduced, and auxiliary pixel interpolation is preferably performed as in the configuration shown in FIG. In other words, the effect of improving the resolution is obtained when the value of the coefficient C4 is 0.0, but the effect of improving the resolution becomes smaller as the value of the coefficient C4 approaches 1.0. Using this property, when the horizontal phase difference θH (2102) is close to 0 (that is, when the coefficient C4 (horizontal) is close to 1.0), the horizontal resolution conversion result SR (horizontal) becomes weak and the horizontal phase difference θH (2102 ) Is not near 0 (i.e., when the coefficient C4 (horizontal) is near 0.0), the coefficient determiner (3001) sets the coefficient K (horizontal) so that the SR (horizontal) of the horizontal resolution conversion result becomes strong. decide.

  As an example of this, the coefficient K (horizontal) = (1 + C4 (horizontal) * 3-C4 (vertical) −C4 (lower right) −C4 (upper right)) / 4 may be set. Similarly, coefficients K (vertical), K (lower right), and K (upper right) are determined by coefficient determiners (3002), (3003), and (3004), respectively. At this time, for coefficient C4 (horizontal), coefficient C4 (vertical), coefficient C4 (lower right), and coefficient C4 (upper right) that change independently, coefficient K (horizontal) + coefficient K (vertical) + coefficient K The coefficient K is determined so that (lower right) + coefficient K (upper right) = 1.0, and SR (horizontal), SR (vertical), SR (lower right), SR (upper right) are mixed.

  31 and 32 show a third operation and a configuration example of the mixer (2707), respectively. FIG. 31 shows a two-dimensional frequency region in which the horizontal frequency is represented by μ and the vertical frequency is represented by ν, as in FIG. In FIG. 31, when the horizontal sampling frequency of the original input image is μs and the vertical sampling frequency is νs, the output of the resolution converter (4) shown in FIG. 27 is the horizontal frequency μ in the range of −μs to μs, The vertical frequency ν is a signal in the range of −νs to νs.

  The effect of the diagonal (upper right) resolution conversion is large, as shown in FIG. 31, in the vicinity of (μ, ν) = (+ μs / 2, + νs / 2) and (μ, ν) = (− μs / 2, -νs / 2) in the vicinity of the frequency region (3101) (particularly (μ, ν) = (+ μs / 2, + νs / 2), with a frequency such that μ> 0 and ν> 0. Region, and (μ, ν) = (− μs / 2, −νs / 2), and a component of a frequency region where μ <0 and ν <0).

  The effect of the diagonal (lower right) resolution conversion is large because (μ, ν) = (+ μs / 2, −νs / 2) as shown in FIG. 31 and (μ, ν) = (− μs). / 2, + νs / 2) in the frequency region (3102) (particularly (μ, ν) = (+ μs / 2, -νs / 2)), and frequencies where μ> 0 and ν <0 And (μ, ν) = (− μs / 2, + νs / 2), and components in a frequency region where μ <0, ν> 0).

  Therefore, when these frequency components (3101) and (3102) are extracted by a two-dimensional filter and mixed together with the frequency components (2501) and (2502) shown in FIG. 25, components having a large resolution improvement effect are selectively output. can do.

  FIG. 32 shows an example of the configuration of a mixer (2707) that extracts components that have a large effect of horizontal, vertical, diagonal (lower right), and diagonal (upper right) resolution conversion. In the figure, a two-dimensional filter (3201) is used to extract a component in the frequency region (3102) having a large resolution improvement effect of SR (lower right) input to the mixer (2707). Similarly, using the two-dimensional filter (3202), the component of the frequency region (3101) having a large resolution improvement effect of SR (upper right) input to the mixer (2707) is extracted. In addition, the two-dimensional filters (2601) and (2602) shown in FIG. 26 extract components in the frequency regions (2501) and (2502) that have a large resolution improvement effect for SR (horizontal) and SR (vertical), respectively. As components other than frequency domain (2501) (2502) (3101) (3102), SR (horizontal), SR (vertical), SR (bottom right), SR using adder (3203) and multiplier (3204) An average signal (upper right) is created, and components other than the passbands of the two-dimensional filters (2601, 2602, 3201, and 3202) are extracted using the two-dimensional filter (3205). The output signals of the two-dimensional filters (2601), (2602), (3201), (3202), and (3205) are added by the adder (3206) to obtain the output of the mixer (2707).

  The numbers in circles in the two-dimensional filters (2601), (2602), (3201), (3202), and (3205) shown in the figure indicate examples of tap coefficients of the respective filters.

  According to the image signal processing apparatus according to the ninth embodiment described above, it is possible to generate a high-resolution image in which the resolution in the oblique direction is increased in addition to the horizontal direction and the vertical direction.

  In addition, according to the prior art described in Patent Document 1, Patent Document 2, and Non-Patent Document 1, one-dimensional (horizontal, vertical, diagonal (lower right), diagonal (upper right)) high resolution using three frames May be performed in a plurality of directions, and each result thereof may be input to the mixer (2707) shown in FIG. 27 and mixed, and output as a two-dimensional resolution conversion result. In this case, as shown in FIG. 27, the scale of the signal processing circuit such as the frame memory and the motion estimation unit is larger than the configuration in which the two-dimensional resolution conversion is performed using only two frames. 2. As described in Non-Patent Document 1, it is possible to reduce the scale of a signal processing circuit such as a frame memory or a motion estimation unit, compared to using a signal of at least 7 frames.

  In addition, the present invention is not limited to the conventional techniques described in Patent Document 1, Patent Document 2, and Non-Patent Document 1, and other one-dimensional (horizontal / vertical / diagonal (lower right)) by applying other conventional high resolution technologies. Even if the resolution (diagonal (upper right)) is increased in a plurality of directions, the respective results are input to the mixer (2707) shown in FIG. 27, mixed, and output as a two-dimensional resolution conversion result. Good.

  Also, in FIG. 27, the case where the resolution of frame # 1 is converted using a set of input signals of frame # 1 and frame # 2 has been described as an example, but other than this, for example, frame # 1 and frame # 1 # 3, Frame # 1 and Frame # 4, etc. are used to convert the resolution of frame # 1 and mix the results to obtain the final resolution conversion result of frame # 1. .

  As a mixing method at this time, an average value of the results may be taken, or mixing may be performed according to the value of the coefficient C4 (frame) for each frame as shown in FIGS. In this case, as the coefficient C4 (frame), the MAX value (the smaller value) of the coefficient C4 (horizontal) and the coefficient C4 (vertical) for each frame may be used. Also, all the coefficients C4 (horizontal) and C4 (vertical) are compared for each pixel, and the resolution conversion result obtained from the group with the smallest coefficient C4 (i.e., the group with the greatest resolution improvement effect) is obtained for each pixel. The final resolution conversion result of frame # 1 may be selected.

  Thus, for example, with frame # 1 as a reference, if frame # 2 is a frame earlier than frame # 1 and frame # 3 is a future frame than frame # 1, the subject is When changing from `` motion '' to `` still '' (end of movement), resolution conversion processing is performed with frame # 1 and frame # 2, and when the subject is converted from `` still '' to `` motion '' at the time of frame # 1 ( (Beginning of motion) Since each processing result is mixed for each pixel so that resolution conversion processing is performed by frame # 1 and frame # 3, the motion of the subject can be used effectively, and the resolution improvement effect is the largest can do.

  An image signal processing method according to Embodiment 10 of the present invention will be described with reference to FIG.

  The tenth embodiment relates to an image signal processing method for realizing a process equivalent to the image signal processing in the image signal processing apparatus according to the ninth embodiment by a control unit that cooperates with software. The image processing apparatus that performs the image signal processing method of the present embodiment is the image processing apparatus shown in FIG.

  FIG. 33 shows an example of a flowchart of the operation of the present embodiment. The flowchart of FIG. 33 starts from step (3301), and in steps (5-1), (5-2), (5-3), and (5-4), horizontal, vertical, diagonal (lower right), diagonal ( Increase the resolution in the upper right). Here, in each step (5-1) (5-2) (5-3) (5-4), the processing step (5) shown in FIG. 14 to FIG. 16 or FIG. 42 to FIG. Any one of the processing steps (5) shown may be executed in the horizontal, vertical, diagonal (lower right), and diagonal (upper right) directions. In other words, the `` frequency-phase '' characteristics such as π / 2 phase shift (1407) (1408) and Hilbert transform (1510) are changed according to the respective directions as shown in FIG. 28, and the phase difference θ is changed to θH , ΘV, (θH + θV), and (−θH + θV). The processing results of steps (5-1), (5-2), (5-3), and (5-4) are written into the respective frame buffers # 3 as described with reference to FIGS. In the following steps (3302-1), (3302-2), (3302-3) and (3302-4), vertical, horizontal, and diagonal pixel interpolation is performed, respectively, to the same number of horizontal and vertical pixels as the output frame. Thus, all the pixels of the two-dimensional frame buffer # 3 are generated. In the subsequent step (3303), the data of each frame buffer # 3 is mixed for each pixel according to the method described with reference to FIGS. 29, 30, and 32, and is output to the output frame buffer # 4. When the operations of the eighth to ninth embodiments are realized by a software program, steps (5-3) and (5-4) for performing oblique processing, and steps for performing pixel interpolation on these results (3302-3) and (3302-4) are not required. In addition, as a mixing method in step (3303), data is mixed according to the method described with reference to FIGS.

  According to the image signal processing method according to the tenth embodiment described above, it is possible to generate a high-resolution image in which the resolution in the oblique direction is increased in addition to the horizontal direction and the vertical direction.

  FIG. 54 shows an image processing apparatus according to Embodiment 11 of the present invention. The image processing apparatus according to the present embodiment uses, for example, an input unit (1) to which a frame sequence of a moving image such as a television broadcast signal is input, and four frames input from the input unit (1). A resolution converter (8) for increasing the resolution twice in the horizontal direction and twice in the vertical direction, and further displaying an image based on the frame that has been increased in resolution by the resolution converter (8). And a display unit (3). This resolution conversion unit (8) performs a phase shift in the horizontal direction, vertical direction, and horizontal / vertical direction for each of the input image signals of the four frames, thereby folding back in the two-dimensional frequency domain. Removes components and achieves two-dimensional high resolution. Details of the resolution conversion unit (8) will be described below.

In FIG. 54, first, a two-dimensional sampling position of a pixel to be processed on frame # 1 input from the input unit (1) by the position estimation units (5406-2), (5406-3), and (5406-4). Using the (sampling position) as a reference, the two-dimensional position of each corresponding image on frame # 2, frame # 3, and frame # 4 is estimated, and the horizontal phase difference θH2 (5407-2), θH3 ( 5407-3), θH4 (5407-4), and vertical phase differences θV2 (5408-2), θV3 (5408-3), and θV4 (5408-4) are obtained. Next, the phase difference θH2 (5407-2) is obtained by the horizontal / vertical uprater (5401-1) (5401-2) (5401-3) (5401-4) of the motion compensation / uprate unit (5410). , ΘH3 (5407-3), θH4 (5407-4), θV2 (5408-2), θV3 (5408-3), θV4 (5408-4) information, frame # 2, frame # 3, The frame # 4 is compensated for motion and aligned with the frame # 1, and the number of pixels in each frame is doubled horizontally and doubled, respectively, for a total density of 4 times. The phase shift unit (5411) converts the phase of the densified data into horizontal phase shifters (5403-1) (5403-2) (5403-3) (5403-4) and vertical phase shifters (5404- 1) Horizontal direction using (5404-2) (5404-3) (5404-4), horizontal / vertical phase shifter (5405-1) (5405-2) (5405-3) (5405-4) Shift by a fixed amount in the vertical, horizontal and vertical directions. Here, a π / 2 phase shifter such as the above-mentioned Hilbert transformer can be used as means for shifting the phase of data by a certain amount. In the aliasing component removal unit (5409), a total of 16 signals from the above-described phase shift unit (5411) and a total of 6 phase difference signals from the phase estimation unit (5412) are used for horizontal and vertical respectively. The output is obtained by removing the aliasing component in the direction of. This output is supplied to the display unit 3. Note that the position estimation units (5406-2), (5406-3), and (5406-4) can be realized using the above-described conventional technique as it is. The horizontal / vertical up-rater (5401-1) (5401-2) (5401-3) (5401-4) extends the operation and configuration shown in FIGS. 5 and 6 to two dimensions in the horizontal and vertical directions. Is. Details of the phase shift unit (5411) and the aliasing component removal unit (5409) will be described later.

  FIG. 55 shows a configuration example of the horizontal / vertical phase shifters (5405-1), (5405-2), (5405-3), and (5405-4). Since the horizontal and vertical phases of the image signal are independent of each other, the horizontal / vertical phase shifter (5405) has the vertical phase shifter (5404) and horizontal phase shifter (5403) as shown in the figure. Can be realized by combining in series. In addition, it is clear that the same operation is achieved even if the connection order is reversed and the horizontal phase shifter (5403) is arranged in front of the vertical phase shifter (5404).

FIG. 56 shows detailed operations of the phase shift unit (5411) and the aliasing component removal unit (5409) described above. FIG. 4A shows a two-dimensional frequency region in which the horizontal frequency is μ and the vertical frequency is ν. When the horizontal sampling frequency of the original input image is μs and the vertical sampling frequency is νs, the signal near the origin (i.e., (μ, ν) = (0, 0)) in FIG. It is well known that a folding component is generated at the position of (μ, ν) = (μs, 0), (μ, ν) = (0, νs), (μ, ν) = (μs, νs). Note that these origin symmetrical positions (i.e., (μ, ν) = (-μs, 0), (μ, ν) = (0, -νs), (μ, ν) = (-μs, -νs) ) Also produces aliasing components, but these are due to the symmetry of the frequency (μ, ν) = (μs, 0), (μ, ν) = (0, νs), (μ, ν) = (μs, It is equivalent to the folded component at the position of νs). In order to increase the resolution twice in the horizontal direction and twice in the vertical direction by the resolution conversion unit (8) shown in FIG. 54, the motion compensation / up-rate unit (5410) performs horizontal and vertical After double the up rate (0 insertion) in each direction and quadrupling the number of pixels, (μ, ν) = (μs, 0), (μ, ν) shown in FIG. 56 (a) ) = (0, νs), (μ, ν) = (μs, νs) may be removed. The operation will be described below.

  In FIG. 56 (b), (μ, ν) = (0, 0), (μ, ν) = (μs, 0), (μ, ν) = (0, νs), (μ, ν) = ( The state of horizontal phase rotation and vertical phase rotation of each component at the position of μs, νs) is shown. As shown in FIG. 4, the phase rotation of the original component does not occur between a plurality of frames having different sampling phases, and only the aliasing component rotates in phase according to the sampling phase difference. Thus, with the phase of the original component as the reference (Re axis), it is considered that the phase shift unit (5411) shown in FIG. 54 generates components of the phase orthogonal axis (Im axis) in the horizontal, vertical, horizontal and vertical directions. 56 (b), the original component (μ, ν) = (0, 0) (that is, # 1) horizontal Re axis (= no horizontal phase rotation) and vertical Re axis Only set the value of the component (no phase rotation in the vertical direction) (the total value of the signals after each phase shift) to `` 1 '' and the values of the other components (i.e., # 2 to # 16) to `` 0 ''. For example, the folded component can be canceled and only the original component can be extracted.

  FIG. 56 (c) shows a matrix arithmetic expression for realizing the phase relationship shown in FIG. 56 (b). In the figure, M is a matrix having 16 × 16 elements, and is an operation indicating horizontal, vertical, horizontal / vertical phase rotation. Details of the matrix M will be described later. Also, the left side of the figure shows the “value” in FIG. 56 (b), and C1ReRe to C4ImIm on the right side are output to the output signals of the phase shift unit (5411) by the aliasing component removal unit (5409) shown in FIG. Indicates the coefficient to be multiplied. That is, for frame # 1 shown in FIG. 54, the output signal of the delay unit (5402-1) is multiplied by the coefficient C1ReRe, the output signal of the horizontal phase shifter (5403-1) is multiplied by the coefficient C1ImRe, and the vertical phase shifter The output signal of (5404-1) is multiplied by a coefficient C1ReIm, and the output signal of the horizontal / vertical phase shifter (5405-1) is multiplied by a coefficient C1ImIm. Similarly, for frame # 2, the output signal of the delay unit (5402-2) is multiplied by the coefficient C2ReRe, the output signal of the horizontal phase shifter (5403-2) is multiplied by the coefficient C2ImRe, and the vertical phase shifter (5404 The output signal of -2) is multiplied by the coefficient C2ReIm, and the output signal of the horizontal / vertical phase shifter (5405-2) is multiplied by the coefficient C2ImIm. For frame # 3, the output signal of the delay unit (5402-3) is multiplied by the coefficient C1ReRe, the output signal of the horizontal phase shifter (5403-3) is multiplied by the coefficient C3ImRe, and the vertical phase shifter (5404-3) The output signal is multiplied by a coefficient C3ReIm, and the output signal of the horizontal / vertical phase shifter (5405-3) is multiplied by a coefficient C3ImIm. For frame # 4, the output signal of the delay unit (5402-4) is multiplied by the coefficient C4ReRe, the output signal of the horizontal phase shifter (5403-4) is multiplied by the coefficient C4ImRe, and the vertical phase shifter (5404-4) The output signal is multiplied by a coefficient C4ReIm, and the output signal of the horizontal / vertical phase shifter (5405-4) is multiplied by a coefficient C4ImIm. If the coefficients C1ReRe to C4ImIm are determined so that the relationship shown in FIG. 56 (c) always holds when all the 16 signals multiplied by the above coefficients are added by the aliasing component removal unit (5409) described later. It is possible to cancel the folded component and extract only the original component.

  FIG. 56 (d) shows details of the matrix M. The matrix M is a matrix having 16 × 16 elements as described above, and is represented by mij (where row number i and column number j are integers satisfying 1 ≦ i ≦ 4 and 1 ≦ j ≦ 4). It consists of a partial matrix with 4x4 elements. This partial matrix mij is classified as shown in (e), (f), (g), and (h) in FIG.

  FIG. 56 (e) shows each element of the partial matrix m1j (that is, m11, m12, m13, m14) when the row number i = 1. This partial matrix m1j is an element acting on the component of the frequency (μ, ν) = (0, 0), and no horizontal / vertical phase rotation occurs regardless of the sampling phase difference between frames. That is, a matrix in which all the elements on the diagonal to the lower right are 1 and the remaining elements are all 0).

  FIG. 56 (f) shows each element of the partial matrix m2j (that is, m21, m22, m23, m24) when the row number i = 2. This partial matrix m2j is an element acting on the component of (μ, ν) = (μs, 0), and according to the horizontal phase difference θHj of sampling (where j is an integer satisfying 1 ≦ j ≦ 4), This is a rotation matrix that rotates the phase in the horizontal direction. That is, a rotation matrix that rotates the phase by θHj around the horizontal frequency axis with # 5 and # 6 and # 7 and # 8 shown in FIG. 56 (b) having a common vertical phase axis as a pair. The horizontal phase difference θH1 when j = 1 is not shown in FIG. 54, but this is the phase difference between the frame # 1 (reference) and the frame # 1 (the processing target = the same as the reference) (= 0) and treated as θH1 = 0. Hereinafter, the vertical phase difference θV1 is similarly treated as θV1 = 0.

  FIG. 56 (g) shows each element of the partial matrix m3j (that is, m31, m32, m33, m34) when the row number i = 3. This partial matrix m3j is an element acting on the component of (μ, ν) = (0, νs), and according to the sampling vertical phase difference θVj (where j is an integer satisfying 1 ≦ j ≦ 4), This is a rotation matrix that rotates the phase in the vertical direction. That is, a rotation matrix that rotates the phase by θVj around the vertical frequency axis with # 9 and # 11 and # 10 and # 12 shown in FIG. 56 (b) having a common horizontal phase axis as a pair.

  FIG. 56 (h) shows each element of the partial matrix m4j (that is, m41, m42, m43, m44) when the row number i = 4. This partial matrix m4j is an element acting on the component of (μ, ν) = (μs, νs), and the sampling horizontal phase difference θHj and vertical phase difference θVj (where j is an integer satisfying 1 ≦ j ≦ 4) ), The rotation matrix rotates the phase in both the horizontal and vertical directions. That is, it is the product of m2j and m3j.

  From another viewpoint, m1j, m2j, and m3j are also rotation matrices that rotate the phase in both the horizontal and vertical directions as in m4j. For m1j, θHj = θVj = 0, for m2j, θVj = 0 In the case of m3j, even if it is assumed that θHj = 0, the partial matrix is the same as described above.

In this way, the matrix M is determined based on each sampling phase difference (θHj, θVj), and a total of 16 coefficients (C1ReRe to C4ImIm) are determined so that the equation shown in FIG. 56 (c) always holds. To do. At this time, an inverse matrix M ⁻¹ for the matrix M is obtained in advance, and the coefficients (C1ReRe to C4ImIm) may be determined by the calculation shown in FIG. 56 (i). As a method for obtaining the inverse matrix M ⁻¹ , a method using a cofactor matrix, a method using a Gauss-Jordan sweep-out method, a method of calculating by dividing into a triangular matrix, etc. are well known. .

  FIG. 57 shows a detailed configuration example of the aliasing component removal unit (5409) shown in FIG. In the figure, the coefficient determination unit (5701) is based on the horizontal phase differences (θH2, θH3, θH4) and vertical phase differences (θV2, θV3, θV4) output from the position estimation unit (5412) shown in FIG. Each coefficient (C1ReRe to C4ImIm) is generated by the inverse matrix operation shown in FIG. 56 (i). These coefficients are multiplied by the signal of each frame output from the phase shift unit (5411) by the multiplier (5702), fully added by the adder (5703), and output from the aliasing component removal unit (5409). (That is, the output signal of the resolution converter (8)). The horizontal phase difference (θH2, θH3, θH4) and vertical phase difference (θV2, θV3, θV4) are generally different for each pixel on the input frame, so the above inverse matrix calculation is performed for each pixel. There is a need to do. At this time, the horizontal phase difference (θH2, θH3, θH4) and the vertical phase difference (θV2, θV3, θV4) are representative phase differences (for example, an integral multiple of π / 8 as shown in FIG. 9D). ), The coefficients (C1ReRe to C4ImIm) may be generated in advance and tabulated using a ROM (Read Only Memory) or the like. Since this is well known as a general table reference method, illustration is omitted.

FIG. 58 shows another configuration example of the aliasing component removal unit (5409) shown in FIG. In the above description, when determining a total of 16 coefficients (C1ReRe to C4ImIm) so that the equation shown in FIG. 56 (c) always holds, an inverse matrix M ⁻¹ for the matrix M is obtained in advance. The coefficients (C1ReRe to C4ImIm) are to be determined by the calculation shown in FIG. 56 (i). ^In some cases, ⁻¹ does not exist and the coefficients (C1ReRe to C4ImIm) cannot be determined. Whether the inverse matrix M ⁻¹ exists is determined by calculating the inverse matrix M ⁻¹ in the coefficient determination unit (5701), using a cofactor matrix, using a Gauss-Jordan sweep-out method, and a triangular matrix If the inverse matrix M ⁻¹ does not exist, it can be easily determined in the calculation process such as the calculation method, and the frame # 1 and the frame # are processed by the resolution conversion unit (4) shown in FIG. The output signal may be switched so that the output signal is obtained using 2. That is, using the horizontal folding component removal unit (2108), the vertical folding component removal unit (2109), the pixel interpolator (2003) (2007), and the mixer (2009) shown in FIG. Resolution conversion result based on frame # 1 and frame # 2 output from 5411) and horizontal phase difference θH2 (5407-2) and vertical phase difference θV2 (5408-2) output from the position estimation unit And the output of the adder (5703) described above using the switch (5801). Instead of binary switching using the switch (5801), the output of the adder (5703) and the output of the mixer (2009) are continuously mixed (that is, weighted addition). For example, in the vicinity of a pixel where the inverse matrix M ⁻¹ does not exist, the mixing ratio of the output of the mixer (2009) may be increased.

  By the aliasing component removal processing described above, in the two-dimensional frequency region shown in FIG. 56 (a), the resolution improvement effect can be achieved from the center to (μ, ν) = (μs, 0) in the horizontal direction. In the vertical direction, the effect of improving the resolution can be achieved from the center to (μ, ν) = (0, νs). In the oblique direction, the effect of improving the resolution can be achieved from the center to (μ, ν) = (μs, νs).

  Here, also in the image signal processing apparatus and the image signal processing method according to the seventh embodiment, the resolution is increased in the oblique direction in addition to the horizontal direction and the vertical direction. As shown in FIG. 31, it does not reach (μ, ν) = (μs, νs).

  Therefore, the image signal processing device shown in FIG. 54 has an effect that the resolution can be improved to a high frequency component in an oblique direction as compared with the image signal processing device according to the seventh embodiment.

  Next, the difference in operation between the image signal processing apparatus according to the eleventh embodiment of the present invention and the prior art will be described with reference to FIG. FIG. 5A shows a frame # 1 (5301), a frame # 2 (5302), a frame # 3 (5303), a frame # 4 (5304), input to the resolution converter (8) shown in FIG. Frame # 5 (5305) is shown, and FIG. 5B shows each frame output from the resolution converter (8). In each frame, the subject moves clockwise by 1/4 pixel, and the subject is intentionally moved so as to make one round in four frames. This movement is continued in the same manner from frame # 5.

In the conventional techniques described in Patent Document 1, Patent Document 2, and Non-Patent Document 1, as described above, when the resolution is increased with respect to a horizontal / vertical two-dimensional input signal, folding is performed from two vertical and horizontal directions. Therefore, if the bandwidth of the original signal doubles both vertically and horizontally, the three aliasing components overlap, and 2M + 1 = 7 digital data (= 7 frame image signals) is required to cancel them. I was trying. Therefore, when a signal that circulates in 4 frames as shown in FIG. 34 (a) is input, independent data cannot be obtained no matter which 7 frames are selected, and the solution by the high resolution processing becomes indefinite. It is not required.

  On the other hand, using the twentieth to twenty-first embodiments, for example, four adjacent frames (frame # 1 (5301) and frame # 2 (5302), frame # 3 (5303), frame # 4 (5304)) are used. Thus, as shown in FIG. 5B, high resolution can be realized by removing the aliasing components in the horizontal direction, vertical direction, and horizontal / vertical direction. That is, the operation status of the present embodiment can be confirmed by using the input image of FIG. If a generally well-known circular zone plate (CZP: Circular Zone Plate) is used as the pattern of this test pattern, the effect of resolution conversion can be directly viewed on the display unit (3). That is, if the circular zone plate is moved so as to make one round in four frames as shown in FIG. 34 (a), an image with improved horizontal resolution and vertical resolution is always displayed. The effect can be confirmed.

  As described above, the image signal processing apparatus according to the eleventh embodiment has a plurality of types of phase shifts (horizontal direction, vertical direction, horizontal / vertical direction) with respect to each image signal of four input image frames. Is performed to generate four signals from each of the image signals. As a result, 16 signals are generated from the image signals of the four input image frames. Here, based on the phase difference between the four input image frames, for each of the 16 signals, a coefficient for canceling the aliasing component of the 16 signals and combining them is calculated for each pixel. For each pixel of the generated image, the sum of the corresponding pixel value of each of the 16 signals multiplied by each coefficient is calculated to generate a new pixel value of the high-resolution image.

  Thus, the image signal processing apparatus according to the eleventh embodiment can generate a high-resolution image in which the diagonal components in the lower right direction and the upper right direction are increased in addition to the horizontal direction and the vertical direction.

  Further, the resolution improvement effect by the image signal processing device according to the eleventh embodiment is that the resolution can be improved to a higher frequency component in the oblique direction than the image signal processing device according to the ninth embodiment, and a high-resolution image with higher image quality. Can be generated.

  An image signal processing method according to Embodiment 12 of the present invention will be described with reference to FIGS. 59 and 19.

  The twelfth embodiment relates to an image signal processing method in which processing equivalent to the image signal processing in the image signal processing apparatus according to the eleventh embodiment is realized by a control unit that cooperates with software.

  Here, an image processing apparatus for realizing the image signal processing method according to the present embodiment will be described with reference to FIG. The image signal processing apparatus shown in FIG. 19 stores, for example, an input unit (1) to which an image signal such as a television broadcast signal is input, and software for processing the signal input from the input unit (1). A control unit (10) that performs image signal processing on a signal input from the input unit (1) in cooperation with software stored in the unit (11), the storage unit (11), and a control unit (10) Output from the frame buffer # 1 (31), frame buffer # 2 (32), frame buffer # 3 (33), frame buffer # 4 (34), and the control unit (10) used as a data buffer in the image signal processing A buffer # 5 (35) for frame-buffering the image signal processed signal output to the unit (3).

  Here, the number of input units (1) included in the image signal processing apparatus shown in FIG. 19 is the same as four, which is the number of frames used for image processing. One frame may be input continuously.

  In addition, the frame buffer # 1 (31), the frame buffer # 2 (32), the frame buffer # 3 (33), the frame buffer # 4 (34), and the storage unit (11) for storing software are used for the data buffer. Each may be configured using individual memory chips, or one or a plurality of memory chips may be used to divide and use each data address.

  In this embodiment, for the image signal input from the input unit (1), the control unit (10) performs image signal processing in cooperation with software stored in the storage unit (11), and the display unit (3) Output to. Details of the image signal processing will be described with reference to FIG.

  The flowchart in FIG. 59 starts from step (5901), and in steps (5902-1), (5902-2), (5902-3), and (5902-4), the image data of each frame is doubled both horizontally and vertically. Update to That is, the image data of frame # 1 is updated at step (5902-1) and written to frame buffer # 1, and the image data of frame # 2 is updated at step (5902-2) to frame buffer # 1. 2 is written, the image data of frame # 3 is updated at step (5902-3) and written to frame buffer # 3, and the image data of frame # 4 is updated at step (5902-4) to frame Write to buffer # 4. Here, the up-rate can be realized by writing the data every other horizontal pixel and every other vertical pixel after once clearing the value of each frame buffer to 0.

Next, in step (5903), the first pixel (for example, the upper left pixel) of frame buffer # 1 is set as the processing target, and processing is performed until all pixel data processing for frame buffer # 1 is completed. Loop.

  In step (5904-2), the position of the corresponding pixel in the frame buffer # 2 is estimated based on the target pixel in the frame buffer # 1, and the horizontal phase difference θH2 and the vertical phase difference θV2 are output. Similarly, in step (5904-3), the position of the corresponding pixel in the frame buffer # 3 is estimated with reference to the target pixel in the frame buffer # 1, and the horizontal phase difference θH3 and the vertical phase difference θV3 are output. . In step (5904-4), the position of the corresponding pixel in the frame buffer # 4 is estimated based on the target pixel in the frame buffer # 1, and the horizontal phase difference θH4 and the vertical phase difference θV4 are output. At this time, the above-described conventional technique can be used as it is as a method of estimating the position of the corresponding pixel.

  In step (5905-2), based on the horizontal phase difference θH2 and vertical phase difference θV2 obtained in step (5904-2), pixels near the corresponding pixel in the frame buffer # 2 are motion compensated. In this motion compensation operation, the operation described with reference to FIGS. 5 and 6 may be performed in the same manner for each of the horizontal direction and the vertical direction. Similarly, in step (5905-3), based on the horizontal phase difference θH3 and vertical phase difference θV3 obtained in step (5904-3), a pixel near the corresponding pixel in frame buffer # 3 is moved. To compensate. In step (5905-4), based on the horizontal phase difference θH4 and vertical phase difference θV4 obtained in step (5904-4), motion compensation is performed on pixels near the corresponding pixel in frame buffer # 4. To do.

  Subsequently, in step (5913), steps (5906-1), (5906-2), (5906-) are applied to frame buffer # 1, frame buffer # 2, frame buffer # 3, and frame buffer # 4 subjected to motion compensation. 3) The horizontal phase is shifted by a certain amount by (5906-4), and the vertical phase is shifted by a certain amount by steps (5907-1), (5907-2), (5907-3) and (5907-4). Further, for the result of steps (5907-1) (5907-2) (5907-3) (5907-4), steps (5908-1) (5908-2) (5908-3) (5908-4) ) Shifts the horizontal phase by a certain amount, thereby shifting both the horizontal and vertical phases by a certain amount. That is, the pixel data in each frame buffer is phase shifted by π / 2 in the horizontal direction and the vertical direction.

  Subsequently, in step (5909), all the 16 coefficients (C1ReRe to C1ReRe through the method shown in FIG. 56) based on the horizontal phase differences (θH2, θH3, θH4) and the vertical phase differences (θV2, θV3, θV4). C4ImIm) is multiplied, and each output of step (5913) is multiplied by the above coefficients and added (weighted addition), so that frame buffer # 1, frame buffer # 2, frame buffer # 3, and frame buffer # 4 The aliasing component is removed from the pixel data and output to frame buffer # 5. The operation of removing the aliasing component is the same as the operation described with reference to FIG.

Subsequently, in step (5910), it is determined whether or not the processing of all the pixels in the frame buffer # 1 has been completed.If not, the next pixel (for example, the right adjacent pixel) is The process is set as a process target, and the process returns to steps (5904-2), (5904-3), and (5904-4). If completed, the process ends in step (5912).

  By performing the processing as described above, using the pixel data of frame buffer # 1, frame buffer # 2, frame buffer # 3, and frame buffer # 4, a high-resolution signal is output to frame buffer # 5. Can do. When applied to a moving image, the processing from step (5901) to step (5912) may be repeated for each frame.

54, 57, 58, and 59, the number of input frames has been described as four. However, the present invention is not limited to this, and n frames (however, n is 4 or more). (Integer integer) may be input, and four frames suitable for the resolution conversion process described above may be selected and used. For example, when outputting the inverse matrix operation shown in FIG. 56 (i), 4 frames used for resolution conversion processing from n frames so that the number of pixels in which the inverse matrix M ⁻¹ does not exist is minimized. This frame may be selected and switched for each pixel or for each region composed of a plurality of pixels.

  Therefore, the image signal processing method according to the twelfth embodiment has an effect that the resolution can be improved to a higher frequency component in the oblique direction than the image signal processing method according to the tenth embodiment. The details of the effect are the same as the effect of the image signal processing apparatus shown in FIG.

  As described above, the image signal processing method according to the twelfth embodiment has a plurality of types of phase shifts (horizontal direction, vertical direction, horizontal / vertical direction) with different directions for each image signal of four input image frames. Is performed to generate four signals from each of the image signals. As a result, 16 signals are generated from the image signals of the four input image frames. Here, based on the phase difference between the four input image frames, for each of the 16 signals, a coefficient for canceling the aliasing component of the 16 signals and combining them is calculated for each pixel. For each pixel of the generated image, the sum of the corresponding pixel value of each of the 16 signals multiplied by each coefficient is calculated to generate a new pixel value of the high-resolution image.

  Accordingly, the image signal processing method according to the twelfth embodiment can generate a high-resolution image in which the diagonal components in the lower right direction and the upper right direction are increased in addition to the horizontal direction and the vertical direction.

  Further, the resolution improvement effect by the image signal processing method according to the twelfth embodiment can improve the resolution to a high frequency component in an oblique direction as compared with the image signal processing method according to the tenth embodiment, and generates a high-resolution image with higher image quality. be able to.

  In the image signal processing apparatus or the image signal processing method according to the first to twelfth embodiments described above, the case where the number of pixels is doubled while improving the resolution of the image has been described as an example. For example, the number of pixels can be increased to a power of 2 (= 2 times, 4 times, 8 times, etc.) by operating the image signal processing apparatus or the image signal processing method multiple times or in multiple stages. is there. In other words, by performing signal processing using two input image frames, the number of pixels is doubled to form an intermediate image frame, and then, two additional intermediate image frames are used to create a new input image frame. As a result, an output image frame in which the number of pixels is further doubled can be obtained. In this case, an output image frame having four times the number of pixels can be obtained as compared with the input image frame. Similarly, if the signal processing is repeated a total of three times, the number of pixels of the output image frame is eight times that of the input image frame. At this time, the number of input image frames necessary to obtain one output image frame is also a power of 2 (= 2, 4, 8,...).

  The final output image is output with the number of pixels other than the power of 2 (= 2 times, 4 times, 8 times, etc.) by performing a general resolution conversion process after the image processing. It is also possible.

  FIG. 35 shows an image display apparatus according to Embodiment 13 of the present invention. The image display apparatus according to the present embodiment is an image display apparatus configured to perform the image signal processing described in any one of the seventh embodiment and the eighth embodiment.

  In the figure, an image display device 3500 includes, for example, an input unit 3501 for inputting a broadcast signal, video content, image content, and the like via a broadcast wave including a television signal, a network, and the like, and an input unit 3501 A recording / playback unit 3502 for recording or playing back content, a content storage unit 3503 for recording content by the recording / playback unit 3502, and a video signal or an image signal played back by the recording / playback unit 3502 are either of Embodiment 7 or Embodiment 8. An image signal processing unit 3504 that is an image signal processing device described in one embodiment, a display unit 3505 that displays a video signal or an image signal processed by the image signal processing unit 3504, and a recording / playback unit 3502 for playback An audio output unit 3506 that outputs the audio signal, a control unit 3507 that controls each component of the image display device 3500, and a user interface that allows the user to operate the image display device 3500. It provided with a such as the face portion 3508.

  The detailed operation of the image signal processing unit 3504 is as described in the seventh embodiment or the eighth embodiment, and thus the description thereof is omitted.

  Since the image display device 3500 includes the image signal processing unit 3504 that is the image signal processing device described in any one of the seventh embodiment and the eighth embodiment, the video signal or the image input to the input unit 3501 The signal can be displayed on the display portion 3505 as a video signal or image signal with higher resolution and higher image quality. Therefore, even when a signal having a resolution lower than that of the display device of the display unit 3505 is input from the input unit 3501, it is possible to perform high-quality and high-definition display while increasing the resolution of the reproduction signal.

  In addition, when playing back video content or image content stored in the content storage unit 3503, it can be converted into a video signal or image signal with higher resolution and higher image quality and displayed on the display unit 3505.

  In addition, by performing image processing of the image signal processing unit 3504 after reproduction of video content or image content stored in the content storage unit 3503, data stored in the content storage unit 3503 is displayed at a resolution displayed on the display unit 3505. Relatively lower resolution than Therefore, there is an effect that the content data amount can be relatively reduced and stored.

  Further, the image signal processing unit 3504 may be included in the recording / playback unit 3502, and the above-described image signal processing may be performed during recording. In this case, since it is not necessary to perform the above-described image signal processing during reproduction, the processing load during reproduction can be reduced.

  Here, it has been described that the above-described image signal processing is performed by the image signal processing unit 3504, but may be realized by a control unit 3507 and software. In this case, the image signal processing may be performed by the method described in any one of the seventh embodiment and the eighth embodiment.

  In this embodiment, the recording / playback unit 3502 may perform encoding according to the state of content such as video input from the input unit 3501 during recording, and then record it in the content storage unit 3503.

  In the present embodiment, the recording / reproducing unit 3502 may perform decoding and decoding if content such as video input from the input unit 3501 is encoded during recording.

  In the image display apparatus according to the present embodiment, the content storage unit 3503 is not necessarily required. In this case, the recording / playback unit 3503 may perform playback of content such as video input from the input unit 3501 without recording.

  Also in this case, the video signal or image signal input to the input unit 3501 can be displayed on the display unit 3505 as a video signal or image signal with higher resolution and higher image quality.

  The image display apparatus 3500 may be, for example, a plasma television, a liquid crystal television, a cathode ray tube, a projector, or an apparatus using other devices. Similarly, the display unit 3505 may be, for example, a plasma panel module, an LCD module, or a projector device. The content storage unit 3503 may be, for example, a hard disk drive, a flash memory, or a removable media disk drive. The audio output unit 3506 may be a speaker, for example. The input unit 3501 may be provided with a tuner that receives broadcast waves, may be provided with a LAN connector for connecting to a network, or may be provided with a USB connector. Furthermore, a terminal that digitally inputs a video signal or an audio signal may be provided, or an analog input terminal such as a composite terminal or a component terminal may be provided. Further, it may be a receiver that transfers data wirelessly.

  According to the image display apparatus according to the thirteenth embodiment described above, each image signal of the two input image frames included in the input video signal or the input image signal is phase-shifted, and each image signal is detected. Two signals are generated. As a result, four signals are generated from the image signals of the two input image frames. Here, based on the phase difference between the two input image frames, for each of the four signals, a coefficient for canceling and combining the aliasing components of the four signals is calculated for each pixel. For each pixel of the image to be generated, a sum is calculated by multiplying the pixel value of the corresponding pixel included in each of the four signals by each coefficient to generate a pixel value of a new high-resolution image. By performing this for each pixel of the generated image, an image having a higher resolution in the one-dimensional direction than the input image frame is generated.

  This is performed in each of the horizontal direction and the vertical direction, and an image with high resolution in the horizontal direction and an image with high resolution in the vertical direction are generated. The image having the high resolution in the horizontal direction and the image having the high resolution in the vertical direction are subjected to up-rate processing in the vertical direction and the horizontal direction, respectively, and then both are mixed.

  As a result, a high-resolution image with high resolution in both the vertical direction and the horizontal direction can be generated from each image signal of two input image frames included in the input video signal or the input image signal. That is, a two-dimensional high resolution image can be generated and displayed on the display unit.

  In addition, since the image display apparatus according to the thirteenth embodiment uses two input image frames, high-resolution display can be realized with a small image processing amount. Accordingly, it is possible to realize an image display apparatus that displays a high-resolution video or image on the display unit in both the vertical direction and the horizontal direction with few aliasing components.

  An image display device according to Embodiment 14 of the present invention is the same as the image display device according to Embodiment 13, except that the image signal processing unit 3504 shown in FIG. 35 is replaced with the image signal processing device described in Embodiment 9. . Since other configurations are the same as those of the image display apparatus according to the thirteenth embodiment, description thereof is omitted.

  Further, the detailed operation of the image signal processing unit 3504 is as described in the ninth embodiment, and thus the description thereof is omitted.

  With the image display device according to the fourteenth embodiment, the input video signal or two input image frames included in the input image signal are used, and the resolution is higher in the horizontal, vertical, and diagonal directions than the input video or input image. It is possible to generate a high resolution image. Further, it is possible to realize an image display device that displays this on the display unit.

  An image display device according to Embodiment 15 of the present invention is obtained by replacing the image signal processing unit 3504 shown in FIG. 35 with the image signal processing device described in Embodiment 11 in the image display device according to Embodiment 13. . Since other configurations are the same as those of the image display apparatus according to the thirteenth embodiment, description thereof is omitted.

  The detailed operation of the image signal processing unit 3504 is the same as that described in the eleventh embodiment, and a description thereof will be omitted.

  With the image display device according to the fifteenth embodiment, the input video signal or the four input image frames included in the input image signal are used, and the resolution is higher in the horizontal, vertical, and diagonal directions than the input video or input image. An image display device that generates a high-resolution image and displays it on the display unit can be realized.

  Further, the resolution improvement effect by the image display device according to the fifteenth embodiment is possible to improve the resolution to a high frequency component in an oblique direction as compared with the image display device according to the fourteenth embodiment, and display a higher resolution image with higher image quality. Can do.

  The image display device according to the sixteenth embodiment of the present invention is the same as the image display device according to the thirteenth embodiment, except that the image signal processing unit 3504 shown in FIG. 35 is one of the first, third, or fifth embodiments. The image signal processing apparatus described in the embodiment is replaced. Since other configurations are the same as those of the image display apparatus according to the thirteenth embodiment, description thereof is omitted.

  The detailed operation of the image signal processing unit 3504 is as described in the first embodiment, the third embodiment, or the fifth embodiment, and thus the description thereof is omitted.

  With the image display device according to the sixteenth embodiment, a high resolution in which the input video signal or two input image frames included in the input image signal are used and the resolution is increased in a one-dimensional direction as compared to the input video or the input image. An image display device that generates an image and displays the image on a display unit can be realized.

  FIG. 36 shows a recording / playback apparatus according to Embodiment 17 of the present invention. The recording / reproducing apparatus according to the present embodiment is a recording / reproducing apparatus configured to perform the image signal processing described in any one of the seventh embodiment and the eighth embodiment.

  In the figure, for example, a recording / playback apparatus 3600 includes, for example, an input unit 3501 for inputting a broadcast signal, video content, image content, and the like via a broadcast wave including a television signal, a network, and the like, and an input from the input unit 3501 A recording / playback unit 3502 for recording or playing back the recorded content, a content storage unit 3503 for recording the content by the recording / playback unit 3502, and a video signal or an image signal played back by the recording / playback unit 3502 according to the seventh or eighth embodiment. An image signal processing unit 3504 that is the image signal processing device described in any one of the embodiments, and an image video output unit that outputs the video signal or image signal processed by the image signal processing unit 3504 to another device or the like 3605, an audio output unit 3606 for outputting the audio signal reproduced by the recording / reproducing unit 3502 to other devices, a control unit 3507 for controlling each component of the recording / reproducing device 3600, a user Over comprises a like user interface unit 3508 for operating the recording and reproducing apparatus 3600.

  Since the recording / reproducing apparatus 3600 includes the image signal processing unit 3504 that is the image signal processing apparatus described in any one of the seventh and eighth embodiments, the video signal or image input to the input unit 3501 The signal can be output to another device or the like as a video signal or image signal with higher resolution and higher image quality. Therefore, it is possible to suitably realize a high-quality and high-resolution signal conversion apparatus that converts a low-resolution video signal or image signal into a high-quality and high-definition video signal or image signal while increasing the resolution.

  Also, when playing back video content or image content stored in the content storage unit 3503, it can be converted into a higher-resolution and higher-quality video signal or image signal and output to another device or the like.

  Therefore, a recording / playback apparatus that inputs and stores a low-resolution video signal or image signal, converts it into a high-quality and high-definition video signal or image signal while outputting a high resolution during playback / output, and outputs the video signal is suitable. realizable.

  In addition, by performing image processing of the image signal processing unit 3504 after reproduction of video content or image content stored in the content storage unit 3503, data stored in the content storage unit 3503 is a signal output to another device. The resolution is relatively lower than the resolution. Therefore, there is an effect that the content data amount can be relatively reduced and stored.

  Further, the image signal processing unit 3504 may be included in the recording / playback unit 3502, and the above-described image signal processing may be performed during recording. In this case, since it is not necessary to perform the above-described image signal processing during reproduction, the processing load during reproduction can be reduced.

  Here, it has been described that the above-described image signal processing is performed by the image signal processing unit 3504, but may be realized by a control unit 3507 and software. In this case, the image signal processing may be performed by the method described in any one of the seventh embodiment and the eighth embodiment.

  In this embodiment, the recording / playback unit 3502 may perform encoding according to the state of content such as video input from the input unit 3501 during recording, and then record it in the content storage unit 3503.

  In the present embodiment, the recording / reproducing unit 3502 may perform decoding and decoding if content such as video input from the input unit 3501 is encoded during recording.

  Further, the image / video output unit 3605 and the audio output unit 3606 according to the present embodiment may be integrated. In this case, it is possible to use a connector shape that outputs a video signal and an audio signal with a single cable.

  Further, the recording / playback apparatus 3600 may be, for example, an HDD recorder, a DVD recorder, or an apparatus using another storage device. Similarly, the content storage unit 3503 may be, for example, a hard disk drive, a flash memory, or a removable media disk drive.

  The input unit 3501 may be provided with a tuner that receives broadcast waves, may be provided with a LAN connector for connecting to a network, or may be provided with a USB connector. Furthermore, a terminal that digitally inputs a video signal or an audio signal may be provided, or an analog input terminal such as a composite terminal or a component terminal may be provided. A receiving unit that wirelessly transfers data may also be used.

  Further, the image / video output unit 3605 may be provided with a terminal for digitally outputting a video signal, or may be provided with an analog output terminal such as a composite terminal or a component terminal. In addition, a LAN connector for connecting to a network or a USB connector may be used. Further, it may be a transmission unit that transfers data wirelessly. The audio output unit 3606 is the same as the image video output unit 3605.

  Furthermore, the input unit 3501 may include, for example, an imaging optical system and a light receiving element. In this case, the recording / playback apparatus 3600 can be applied to, for example, a digital camera, a video camera, a surveillance camera (surveillance camera system), and the like. At this time, for example, the input unit 3501 captures an image of the object to be photographed on the light receiving element by the imaging optical system, generates image data or video data based on the signal output from the light receiving element, and outputs the image data or video data to the recording / playback unit 3502. That's fine.

  If the recording / playback apparatus 3600 is a digital camera, for example, a plurality of images that are temporally different from each other are recorded, and if the image signal processing of the image signal processing unit 3504 is performed on the plurality of image data, 1 A high-resolution image with high image quality can be obtained. Note that the image processing of the image signal processing unit 3504 may be performed on an image recorded in the content storage unit 3503 when data is output from the digital camera. Further, the image signal processing unit 3504 may be integrated with the recording / playback unit 3502 and the image signal processing unit 3504 to perform image processing of the image signal processing unit 3504 before recording in the content storage unit 3503. In this case, only the enlarged image that the user wants to handle finally needs to be stored in the content storage unit 3503, and management becomes easier when the user handles image data later.

  According to the digital camera described above, high-quality image data having a resolution exceeding the resolution of the light receiving element of the digital camera can be obtained.

  Further, if the recording / playback apparatus 3600 is, for example, a video camera, video captured on the light receiving element by the imaging optical system of the input unit 3501 may be output to the recording / playback unit 3502 as video data. The recording / playback unit 3502 may record video data in the content storage unit 3503, and the image signal processing unit 3504 may generate high-resolution video data from the recorded video data. In this way, high-quality video data having a resolution exceeding the resolution of the light receiving element of the video camera can be obtained. At this time, the image signal processing unit 3504 may generate one piece of still image data using data of a plurality of frames included in the recorded video data. In this way, one piece of high quality image data can be obtained from the video data. Similarly to the case of the digital camera described above, the image processing of the image signal processing unit 3504 may be performed before or after recording the video data in the content storage unit 3503.

  According to the video camera described above, high-quality video data having a resolution exceeding the resolution of the light receiving element of the video camera and high-quality still image data can be obtained using the captured video data.

  In addition, when the recording / playback apparatus 3600 is, for example, a surveillance camera (surveillance camera system), high-quality video data having a resolution exceeding the resolution of the light-receiving element of the surveillance camera, as with the video camera described above, Still image data with high image quality can be obtained using the video data. At this time, for example, even when the input unit 3501 including the imaging optical system and the light receiving element is separated from the recording / reproducing unit 3502 and connected by a network cable or the like, the recording / reproducing unit 3502 has a low resolution. It is possible to increase the resolution by transmitting the video data and then performing image signal processing of the image signal processing unit 3504. Thus, high-resolution video data can be obtained while efficiently using the bandwidth of the transmission network from the input unit 3501 including the imaging optical system and the light receiving element.

  The image display apparatuses according to the thirteenth to sixteenth embodiments and the recording / playback apparatus according to the present embodiment can be an embodiment of the present invention even if the functions and components thereof are integrated. In this case, the video signal or image signal subjected to the above-described image signal processing can be displayed or output to another device, and can be used as any of a display device, a recording / playback device, and an output device, Convenient for users.

  According to the recording / reproducing apparatus in the seventeenth embodiment described above, each image signal of two input image frames included in the input video signal or the input image signal is phase-shifted, and each image signal is detected. Two signals are generated. As a result, four signals are generated from the image signals of the two input image frames. Here, based on the phase difference between the two input image frames, for each of the four signals, a coefficient for canceling and combining the aliasing components of the four signals is calculated for each pixel. For each pixel of the image to be generated, a sum is calculated by multiplying the pixel value of the corresponding pixel included in each of the four signals by each coefficient to generate a pixel value of a new high-resolution image. By performing this for each pixel of the generated image, an image having a higher resolution in the one-dimensional direction than the input image frame is generated.

  This is performed in each of the horizontal direction and the vertical direction, and an image with high resolution in the horizontal direction and an image with high resolution in the vertical direction are generated. The image having the high resolution in the horizontal direction and the image having the high resolution in the vertical direction are subjected to up-rate processing in the vertical direction and the horizontal direction, respectively, and then both are mixed.

  As a result, a high-resolution image with high resolution in both the vertical direction and the horizontal direction can be generated from each image signal of the two input image frames included in the input video signal or the input image signal. That is, a two-dimensional high resolution image can be generated and output.

  Also, the input video signal or the input image signal is recorded in the recording unit, and at the time of reproduction from the recording unit, the vertical and horizontal directions are obtained from the image signals of the two input image frames included in the video signal or the image signal. A two-dimensional high resolution image with high resolution in both directions can be generated and can be output.

  In addition, since the recording / playback apparatus according to the seventeenth embodiment uses two input image frames, it is possible to output a high resolution image with a small amount of image processing. As a result, it is possible to realize a recording / reproducing apparatus that outputs a high-resolution video or image in both the vertical direction and the horizontal direction with less aliasing components.

  The recording / playback apparatus according to Embodiment 18 of the present invention is the same as the recording / playback apparatus according to Embodiment 17, except that the image signal processing unit 3504 shown in FIG. 36 is replaced with the image signal processing apparatus described in Embodiment 9. . Since other configurations are the same as those of the recording / playback apparatus according to the seventeenth embodiment, description thereof is omitted.

  Further, the detailed operation of the image signal processing unit 3504 is as described in the ninth embodiment, and thus the description thereof is omitted.

  According to the recording / reproducing apparatus in the eighteenth embodiment, using two input image frames included in the input video signal or the input image signal, the resolution is higher in the horizontal direction, the vertical direction, and the oblique direction than the input video image or the input image. It is possible to generate a two-dimensional high-resolution image and output it.

  Further, the input video signal or the input image signal is recorded in the recording unit, and at the time of reproduction from the recording unit, the horizontal direction and the vertical direction are obtained from each image signal of the two input image frames included in the video signal or the image signal. A two-dimensional high-resolution image with high resolution in the direction and oblique direction can be generated and output.

  The recording / playback apparatus according to Embodiment 19 of the present invention is the same as the recording / playback apparatus according to Embodiment 17, except that the image signal processing unit 3504 shown in FIG. 36 is replaced with the image signal processing apparatus described in Embodiment 11. . Since other configurations are the same as those of the recording / playback apparatus according to the seventeenth embodiment, description thereof is omitted.

  The detailed operation of the image signal processing unit 3504 is the same as that described in the eleventh embodiment, and a description thereof will be omitted.

  According to the recording / reproducing apparatus in the nineteenth embodiment, four input image frames included in the input video signal or the input image signal are used, and the resolution in the horizontal direction, the vertical direction, and the oblique direction is higher than that of the input video or input image. It is possible to realize a recording / playback apparatus that generates and outputs a converted two-dimensional high-resolution image.

  Further, the input video signal or the input image signal is recorded in the recording unit, and at the time of reproduction from the recording unit, the horizontal direction and the vertical direction are obtained from each image signal of the four input image frames included in the video signal or the image signal. A two-dimensional high-resolution image with high resolution in the direction and oblique direction can be generated and output.

  Further, the resolution improvement effect of the recording / playback apparatus according to the nineteenth embodiment can be improved to a higher frequency component in the oblique direction than the recording / playback apparatus according to the eighteenth embodiment, and a higher-resolution image with higher image quality is output. be able to.

  The recording / playback apparatus according to the twentieth embodiment of the present invention is the same as the recording / playback apparatus according to the seventeenth embodiment, but the image signal processing unit 3504 shown in FIG. 36 is one of the first, third, or fifth embodiments. The image signal processing apparatus described in the embodiment is replaced. Since other configurations are the same as those of the recording / playback apparatus according to the seventeenth embodiment, description thereof is omitted.

  The detailed operation of the image signal processing unit 3504 is the same as that described in the first, third, or fifth embodiment, and thus the description thereof is omitted.

  According to the recording / playback device according to the twentieth embodiment, a high resolution in which two input image frames included in the input video signal or the input image signal are used and the resolution is increased in a one-dimensional direction as compared with the input video or the input image. A recording / reproducing apparatus that generates and outputs an image can be realized.

  In addition, the input video signal or the input image signal is recorded in the recording unit, and at the time of reproduction from the recording unit, from each image signal of the two input image frames included in the video signal or the image signal, in a one-dimensional direction. A high-resolution image with high resolution can be generated and output.

  A twenty-first embodiment in which the present invention is applied to an interlace progressive scanning line (hereinafter referred to as I-P conversion) will be described with reference to FIGS. Prior to this, the conventional general I-P conversion operation will be described with reference to FIGS.

  FIG. 37A shows the positional relationship of scanning lines for interlaced scanning (2: 1 interlaced), and FIG. 37B shows the positional relationship of scanning lines for progressive scanning. In each figure, the horizontal axis represents the position (t) in the time direction (frame direction), and the vertical axis represents the vertical position (v). In the interlaced scanning shown in FIG. 6A, the scanning line (3701) that is transmitted or displayed (actual scanning line) and the scanning line (3702) that is skipped and not transmitted or displayed are alternately repeated in the field (3703). ) Is formed. In the next field, the positions of the scanning line (3701) and scanning line (3702) are reversed (complementary), and the two fields (3703) (3704) are combined to form one frame (3704). Is done. The field (3703) is converted to the frame (3705) by interpolating the scanning line (3702) that is skipped and not transmitted or displayed by interpolating from the neighboring actual scanning line (3701) to the frame (3705). Convert to the progressive scan of b).

  As a conventional representative method for realizing this I-P conversion, there is a motion adaptive IP conversion shown in FIG. 38A and a motion compensation IP conversion shown in FIG. (a) Motion adaptive IP conversion uses inter-field interpolation to generate signals on interpolated scanning lines using signals on actual scanning lines of past (or future) fields when the subject in the image is stationary. When the subject is moving, it is common to perform intra-field interpolation (3802) that generates signals on the interpolated scanning lines using the signals on the upper and lower scanning lines in the same field. is there. At this time, when the subject is completely stationary, an ideal progressive scan image can be obtained by inter-field interpolation (3801), but if the subject moves even slightly, even if inter-field interpolation (3801) is used, the inter-frame interpolation (3801) It is well known that even if interpolation (3802) is used, a progressive scan image with reduced vertical resolution is obtained. On the other hand, (b) motion compensated IP conversion calculates the motion vector (3803) by estimating the motion of the subject for each pixel, and interpolated scanning lines based on this motion vector (3803) including the motion in the horizontal direction. Generate. For example, FIG. 5B shows a case where the subject once moves downward, then remains stationary for two field periods, and then moves upward. At this time, if the motion vector just points from the actual scan line to the interpolated scan line, an ideal progressive scan image is obtained, but in other cases (when the interpolated scan line is not pointed to from the actual scan line), it is vertical. It is well known that a progressive scan image with reduced resolution is obtained.

  FIG. 39 shows the operation of the embodiment 21 according to the present invention for improving the drawbacks of the conventional I-P conversion. Each field (# 1, # 2 ...) of interlaced scanning shown in Fig. 11 (a) is regarded as a frame with the number of scanning lines being 1/2, and the position of the scanning line (= vertical sampling phase) is determined for each field. Using the change, a frame is generated by vertical resolution conversion (3901) (3902). For example, frame # 2 is generated by vertical resolution conversion (3901) using field # 1 and field # 2, and frame # 3 is generated by vertical resolution conversion (3902) using field # 2 and field # 3. Thereafter, the subsequent fields are similarly processed to continuously generate frames. At this time, as each vertical resolution conversion (3901) (3902), the resolution conversion unit (2) shown in FIG. 1 or the like can be used as it is, but it is necessary to change the operation for each field as shown in FIG. is there.

  The operation according to Example 21 of the present invention will be described in detail with reference to FIG. FIG. 4A shows the position of the original interlaced scanning line. When the subject is stationary, the motion vector (4001) passing through the actual scanning line indicates the interpolation scanning line of the next field at the same vertical position as shown in FIG. Here, each field (# 1, # 2, etc.) is regarded as a frame with half the number of scanning lines, and the position of the scanning line (= sampling phase in the vertical direction) does not change for each field. When a direction offset (spatial position difference) is added and moved, the motion vector (4002) at rest moves up and down for each field as shown in FIG. That is, when the phase difference θ (102) is obtained by the phase estimator (101) shown in FIG. 1 by using the signal at the scanning line position shown in FIG. 5B as an input, the vertical resolution conversion (3901) (3902) is performed. Will output incorrect results.

  FIG. 41 shows a configuration according to Embodiment 21 of the present invention aimed at preventing the above malfunction. The configuration shown in the figure is obtained by adding an offset correction unit (4103) to the configuration of the embodiment that performs resolution conversion in the one-dimensional direction (vertical direction here) shown in FIG. ) And the previous (past) field # (k-1) from the input unit (1), vertical resolution conversion is performed, and an image is output to the display unit (3). The offset correction unit (4103) adds a phase difference offset (θoffset) (4102) to the phase difference θ obtained by the position estimation unit (101) by the adder (4101) to obtain a new phase difference θ (102). This is used for signal processing of the motion compensation / up-rate unit (115) and the aliasing component removal unit (117). Here, when k = 2n (where n is an integer, i.e., k is an even number), the phase difference offset (θoffset) (4102) is π, and the value of the phase difference θ (102) is half the input scanning line interval. Corrected downward (only the difference in the spatial position of the nearest scanning line between the two fields), and when k = 2n + 1 (n is an integer, that is, k is an odd number), the phase difference offset (θoffset) ( 4102) is -π, and the value of phase difference θ (102) is corrected upward by 1/2 of the input scanning line interval (by the difference in the spatial position of the nearest scanning line between the two fields). To do. Thereby, the vertical motion of the stationary motion vector (4002) shown in FIG. 40 (b) can be corrected. In order to correspond to a two-dimensional image including the horizontal direction, the position estimation unit (101) outputs the horizontal phase difference θH and the vertical phase difference θV, and the motion compensation / uprate unit (115) is two-dimensional. You just have to. That is, after adding the above-described phase difference offset (θoffset) (4102) to the vertical phase difference θV to obtain a new vertical phase difference θV, the tap coefficient Ck (= 2sin) shown in FIG. Apply a vertical filter with (πk + θV) / (πk + θV)). On the other hand, general motion compensation is performed in the horizontal direction. In other words, the tap coefficient shown in FIG. 12 is shifted in the sampling phase by the horizontal phase difference θH, and Ck = sin (πk / 2 + θH) / (πk / 2 + θH) is defined as the tap coefficient without up-grading in the horizontal direction. You just need to apply a horizontal filter. Each process of the phase shift unit (116) and the aliasing component removal unit (117) performs the vertical phase shift (Hilbert transform) and aliasing component removal (coefficient determination and weighted addition) without changing the operation as described above. Do. With these processes, it is possible to perform resolution conversion on the input of interlaced scanning and convert it to progressive scanning.

  At this time, the coefficients C0, C1, C2, and C3 in the aliasing component removal unit (117) are values obtained by replacing the phase difference θ shown in FIG. 9 with (θ ± π). That is, C0 = C2 = 1/2, C1 =-(1 + cos (θ ± π)) / (2sin (θ ± π)) = (1-cosθ) / sinθ, C3 = (1 + cos (θ ± π)) / (2sin (θ ± π)) =-(1-cosθ) / sinθ. At this time, it is possible to prevent the coefficients C1 and C3 from becoming indefinite when the phase difference θ is ± π, and the coefficients C1 and C3 to increase as the phase difference θ approaches ± π, thereby making it vulnerable to noise and the like. Therefore, the fifth embodiment of the present invention shown in FIG. 11 may be configured to switch to the output from the auxiliary pixel interpolation unit (1105) when the phase difference θ is in the vicinity of ± π. That is, in the specific example of the coefficient determiner (1103) used in the fifth embodiment of the present invention shown in FIG. 13, when θ in the figure is read as (θ ± π) and the phase difference θ is in the vicinity of ± π. In addition, the value of the coefficient C1 may be forced to 0 and the value of the coefficient C4 may be set to 1.0. With this operation, in the configuration shown in FIG. 11, when the phase difference θ (102) becomes ± π or in the vicinity of ± π, the output of the adder (1104) is automatically converted to the output of the interpolation low-pass filter (1101). It becomes possible to switch. Note that the phase difference θ may approach ± π and gradually gradually approach the coefficient shown in FIG. 13 from the coefficient shown in FIG. Also, when the position estimation unit (101) in FIG. 1 determines that the pixel corresponding to the pixel to be processed on the frame # 1 is not on the frame # 2, the phase difference θ (102) is ± π. The coefficients may be controlled in the same manner as when the vicinity is reached, and the output of the adder (1104) may be automatically switched to the output of the interpolation low-pass filter (1101).

  20, 21, and 27, the phase difference offset (θoffset) (4102) shown in FIG. 41 with respect to the value of the vertical phase difference θV output from the position estimation unit. ), Two-dimensional resolution conversion can be realized for interlaced scanning input signals.

  Further, it is also possible to add an offset to the vertical position of the pixel to the output progressive frame image to obtain an interlace scanning mode with a high scanning line density. For example, when the 480i format (interlaced scanning mode with 480 scanning lines) is input and converted to 960i (interlaced scanning mode with 960 scanning lines), the 480i format is temporarily converted to 480p by the above technique of the present invention. After converting to a format (progressive scanning form with 480 scanning lines), using a general interpolation filter, every other frame (for example, frames # 2, # 4, # 6, etc.) is set to 1 in the vertical direction. It is only necessary to shift by / 2 pixels (= 1/2 scanning lines).

  Note that frame # 1, frame # 2, frame # 3, and the like described above may be temporally discontinuous frames or may be in reverse order in time. Also, field # 1, field # 2, field # 3, etc. are temporally discontinuous if the value of the phase difference offset (θoffset) is determined in consideration of the positional relationship of the scanning lines as shown in FIG. Field, or in reverse order of time. For example, when performing resolution conversion processing with odd fields or even fields as inputs, the value of the phase difference offset (θoffset) may be set to zero.

  According to the image signal processing apparatus according to the twenty-first embodiment described above, each field of interlace scanning (# 1, # 2,...) Is regarded as a frame with the number of scanning lines being ½, and every other frame covers the entire field. Move with offset in the vertical direction. Of the continuous images generated in this way, position estimation is performed for two images, a phase difference is calculated, and the phase difference is corrected by a phase difference offset corresponding to the offset. Here, phase shift is performed on the signals of the two images, and two signals are generated from each image signal. As a result, four signals are generated from the two image signals. Here, based on the corrected phase difference, for each signal of the four signals, a coefficient for canceling and combining the aliasing components of the four signals is calculated for each pixel. For each pixel of the image to be generated, a sum is calculated by multiplying the pixel value of the corresponding pixel included in each of the four signals by each coefficient to generate a pixel value of a new high-resolution image. By performing this for each pixel of the generated image, a new high-resolution image is generated, and the high-resolution image is output as a frame image for progressive scanning.

  As a result, the image signal processing apparatus according to the twenty-first embodiment can generate a progressive scan image with little reduction in vertical resolution using two interlaced scan fields.

  Further, since the image signal processing apparatus according to the twenty-first embodiment uses two input image frames, the amount of necessary image processing is small. Accordingly, it is possible to realize an image signal processing apparatus that generates a progressive scan image with little reduction in vertical resolution at a low cost.

  An image signal processing method according to Embodiment 22 of the present invention will be described with reference to FIG.

  The twenty-second embodiment relates to an image signal processing method that realizes processing equivalent to the image signal processing in the image signal processing apparatus according to the twenty-first embodiment by a control unit that cooperates with software. The image processing apparatus that performs the image signal processing method of the present embodiment is the image processing apparatus shown in FIG.

  FIG. 42 shows an example of a flowchart of the operation of the image signal processing apparatus according to the present embodiment. The flowchart of FIG. 42 adds a step (4201) for correcting the offset shown in FIG. 40 (b) to the flowchart shown in FIG. 14, and also inputs each input of steps (1402) and (1403) to frame #. 1. Frame # 2 is changed to field # 1 and field # 2.

  Other steps are the same as those in the flowchart shown in FIG.

  Here, in the step (4201) of correcting the offset shown in FIG. 42, the phase difference offset θoffset is determined based on the vertical positional relationship of the scanning lines shown in FIG. 40B, and added to the phase difference θ. This is a step for performing processing. That is, when vertical resolution conversion is performed by inputting the field #k (where k is an integer) and the previous (past) field # (k-1), k = 2n (n is an integer, that is, k is an even number. ), The phase difference offset (θoffset) is π, and the value of the phase difference θ is corrected downward by 1/2 of the input scanning line interval, k = 2n + 1 (n is an integer, that is, k is an odd number) In this case, the phase difference offset (θoffset) is set to −π, and the value of the phase difference θ is corrected upward by 1/2 of the input scanning line interval. As a result, the vertical motion of the stationary motion vector 4002 shown in FIG. 40B can be corrected, and resolution conversion can be performed on the input of interlaced scanning without changing the operation of other steps. Will be able to do.

  Note that frame # 1, frame # 2, frame # 3, and the like described above may be temporally discontinuous frames or may be in reverse order in time. Also, field # 1, field # 2, field # 3, etc. are temporally discontinuous if the value of the phase difference offset (θoffset) is determined in consideration of the positional relationship of the scanning lines as shown in FIG. Field, or in reverse order of time. For example, when performing resolution conversion processing with odd fields or even fields as inputs, the value of the phase difference offset (θoffset) may be set to zero.

  According to the image signal processing method according to the twenty-second embodiment described above, each field of interlace scanning (# 1, # 2,...) Is regarded as a frame with the number of scanning lines being ½, and every other frame is applied to the entire field. Move with offset in the vertical direction. Of the continuous images generated in this way, position estimation is performed for two images, a phase difference is calculated, and the phase difference is corrected by a phase difference offset corresponding to the offset. Here, phase shift is performed on the signals of the two images, and two signals are generated from each image signal. As a result, four signals are generated from the two image signals. Here, based on the corrected phase difference, for each signal of the four signals, a coefficient for canceling and combining the aliasing components of the four signals is calculated for each pixel. For each pixel of the image to be generated, a sum is calculated by multiplying the pixel value of the corresponding pixel included in each of the four signals by each coefficient to generate a pixel value of a new high-resolution image. By performing this for each pixel of the generated image, a new high-resolution image is generated, and the high-resolution image is output as a frame image for progressive scanning.

  As a result, the image signal processing method according to the twenty-second embodiment can generate a progressive scan image with little decrease in vertical resolution by using two fields of interlace scanning.

  Further, the image signal processing method according to the twenty-second embodiment has an effect of reducing the amount of necessary image processing since two input image frames are used.

  The image signal processing device according to the twenty-third embodiment of the present invention is the same as the image signal processing device according to the twenty-first embodiment shown in FIG. 41 except that the phase shift unit (116) is replaced with the phase shift unit (1009) in FIG. The removal unit (117) is replaced with the folded component removal unit (1010) of FIG.

  Other configurations are the same as those of the image signal processing apparatus shown in FIG.

  The image signal processing apparatus according to the twenty-third embodiment described above can be realized with a smaller circuit scale than the image signal processing apparatus according to the twenty-first embodiment in addition to the effects of the image signal processing apparatus according to the twenty-first embodiment. It can be realized at cost.

  An image signal processing method according to Embodiment 24 of the present invention will be described with reference to FIG.

  The twenty-fourth embodiment relates to an image signal processing method in which processing equivalent to the image signal processing in the image signal processing apparatus according to the twenty-third embodiment is realized by a control unit that cooperates with software. The image processing apparatus that performs the image signal processing method of the present embodiment is the image processing apparatus shown in FIG.

  FIG. 43 shows an example of a flowchart of the operation of the image signal processing apparatus according to the present embodiment. 43 adds the step (4301) for correcting the offset shown in FIG. 40 (b) to the flowchart shown in FIG. 15, and also inputs each input in steps (1502) and (1503) to frame #. 1. Frame # 2 is changed to field # 1 and field # 2.

  Other steps are the same as those in the flowchart shown in FIG.

  The details of the operation in step (4301) for correcting the offset are the same as those in step (4201) for correcting the offset shown in FIG.

  The image signal processing method according to the twenty-fourth embodiment described above has the same effect of increasing the resolution of the image signal as the image signal processing method according to the twenty-second embodiment. Furthermore, the image signal processing method according to the twenty-fourth embodiment is less than the image signal processing method according to the twenty-second embodiment by sharing the contents of some processing steps as compared with the image signal processing method according to the twenty-second embodiment. There is an effect that it is possible to realize the same signal processing with a processing amount (the number of operations).

  The image signal processing device according to Embodiment 25 of the present invention is the same as the image signal processing device according to Embodiment 21 shown in FIG. 41 except that the phase shift unit (116) is replaced with the phase shift unit (1009) of FIG. The removal unit (117) is replaced with the aliasing component removal unit (1010) of FIG. 11, and the auxiliary pixel interpolation unit (1055) shown in FIG. 11 is further provided.

  Other configurations are the same as those of the image signal processing apparatus shown in FIG.

  In addition to the effects of the image signal processing device according to the twenty-first embodiment, the image signal processing device according to the twenty-fifth embodiment described above has a phase difference θ after the addition of the phase difference offset, compared to the image signal processing device according to the twenty-first embodiment. A stable output image without indeterminate processing results even when the value becomes 0 or near 0 or when it is determined that the pixel corresponding to the pixel to be processed on field # 1 is not on field # 2. Can be obtained.

  An image signal processing method according to Embodiment 26 of the present invention will be described with reference to FIG.

  The twenty-sixth embodiment relates to an image signal processing method in which processing equivalent to the image signal processing in the image signal processing apparatus according to the twenty-fifth embodiment is realized by a control unit that cooperates with software. The image processing apparatus that performs the image signal processing method of the present embodiment is the image processing apparatus shown in FIG.

  FIG. 44 shows an example of a flowchart of the operation of the image signal processing apparatus according to the present embodiment. The flowchart of FIG. 44 adds a step (4401) for correcting the offset shown in FIG. 40 (b) to the flowchart shown in FIG. 16, and also inputs the inputs of steps (1402) and (1403) to the frame #. 1. Frame # 2 is changed to field # 1 and field # 2.

  The other steps are the same as those in the flowchart shown in FIG.

  The details of the operation in the step (4401) for correcting the offset are the same as those in the step (4201) for correcting the offset shown in FIG.

  In addition to the effect of the image signal processing method according to the twenty-second embodiment, the image signal processing method according to the twenty-sixth embodiment described above is more effective than the image signal processing method according to the twenty-second embodiment. A stable output image without indeterminate processing results even when the value becomes 0 or near 0 or when it is determined that the pixel corresponding to the pixel to be processed on field # 1 is not on field # 2. Can be obtained.

  The image display device according to the twenty-seventh embodiment of the present invention is the same as the image display device according to the thirteenth embodiment except that the image signal processing unit 3504 shown in FIG. In place of the image signal processing apparatus described in the above. Since other configurations are the same as those of the image display apparatus according to the thirteenth embodiment, description thereof is omitted.

  The detailed operation of the image signal processing unit 3504 is the same as that described in the twenty-first, twenty-third, or twenty-fifth embodiment, and thus the description thereof is omitted.

  According to the image display apparatus of the twenty-seventh embodiment, each interlaced scanning field (# 1, # 2,...) Is regarded as a frame having a scanning line number of 1/2, and every other frame is offset vertically. Add and move. Of the continuous images generated in this way, position estimation is performed for two images, a phase difference is calculated, and the phase difference is corrected by a phase difference offset corresponding to the offset. Here, phase shift is performed on the signals of the two images, and two signals are generated from each image signal. As a result, four signals are generated from the two image signals. Here, based on the corrected phase difference, for each signal of the four signals, a coefficient for canceling and combining the aliasing components of the four signals is calculated for each pixel. For each pixel of the image to be generated, a sum is calculated by multiplying the pixel value of the corresponding pixel included in each of the four signals by each coefficient to generate a pixel value of a new high-resolution image. By performing this for each pixel of the generated image, a new high-resolution image is generated, and the high-resolution image is displayed on the display unit as a frame image for progressive scanning.

  Thus, the image display apparatus according to the twenty-seventh embodiment can generate a progressive scan image with little decrease in vertical resolution using two fields of interlaced scan and display the same on the display unit.

  Further, since the image display apparatus according to the twenty-seventh embodiment uses two input image frames, the amount of necessary image processing is small. As a result, an image display device that generates and displays a progressive scan image with little reduction in vertical resolution can be realized at low cost.

  The recording / playback apparatus according to Embodiment 28 of the present invention is the same as the recording / playback apparatus according to Embodiment 17, except that the image signal processing unit 3504 shown in FIG. The image signal processing apparatus described in one embodiment is replaced. Since other configurations are the same as those of the recording / playback apparatus according to the seventeenth embodiment, description thereof is omitted.

  The detailed operation of the image signal processing unit 3504 is as described in the twenty-first, twenty-third, or twenty-fifth embodiment, and thus the description thereof is omitted.

  According to the recording / reproducing apparatus in the twenty-eighth embodiment of the present invention, each field of interlace scanning (# 1, # 2,...) Is regarded as a frame having the number of scanning lines of 1/2, and every other frame is perpendicular to the entire field. Move with an offset in the direction. Of the continuous images generated in this way, position estimation is performed for two images, a phase difference is calculated, and the phase difference is corrected by a phase difference offset corresponding to the offset. Here, phase shift is performed on the signals of the two images, and two signals are generated from each image signal. As a result, four signals are generated from the two image signals. Here, based on the corrected phase difference, for each signal of the four signals, a coefficient for canceling and combining the aliasing components of the four signals is calculated for each pixel. For each pixel of the image to be generated, a sum is calculated by multiplying the pixel value of the corresponding pixel included in each of the four signals by each coefficient to generate a pixel value of a new high-resolution image. By performing this for each pixel of the generated image, it is possible to generate a new high-resolution image, generate the high-resolution image as a progressive scan frame image, and output this.

  Thus, the recording / playback apparatus according to the twenty-eighth embodiment can generate and output a progressive scan image with little reduction in vertical resolution using two interlaced scan fields.

  In addition, the interlaced scanning input video signal is recorded in the recording unit, and at the time of reproduction from the recording unit, a progressive scanning image with little reduction in vertical resolution is generated by using two fields of the interlaced scanning to be reproduced. Can be output.

  In addition, since the recording / playback apparatus according to the twenty-eighth embodiment uses two input image frames, the amount of necessary image processing is small. As a result, it is possible to realize a recording / playback apparatus that generates and outputs a progressive scan image with little reduction in vertical resolution at low cost.

  As described above, in the conventional techniques described in Patent Document 1, Patent Document 2, and Non-Patent Document 1, since the resolution enhancement processing is performed using the difference in sampling phase (sampling position), sampling is performed. There is a problem that the effect of increasing the resolution cannot be obtained in a signal in which a phase difference does not occur, that is, in a region where the subject is stationary or a region where the motion of the subject is an integer pixel unit. In order to solve this problem, the operation and configuration of the image signal processing apparatus according to the embodiment 29 of the present invention will be described with reference to FIGS.

  First, the operation according to Embodiment 29 of the present invention will be described with reference to FIG. In this figure, (a) the subject on the input frame moves over time as frame #a (4501), frame #b (4502), frame #c (4503), frame #d (4504), etc. It shall be. At this time, if the direction in which the camera is projected is moving between frame #a (4501) and frame #b (4502), (b) the entire screen is unified as the position estimation result (motion vector). The result (4505) is obtained. Also, between the frame #b (4502) and the frame #c (4503), assuming that the camera is stationary and only the person is moving, (b) only the person part is included as the position estimation result (motion vector). The result (4506) is obtained that the motion and the background part are stationary. Also, if both the person and the background are stationary between frame #c (4503) and frame #d (4504), (b) the position estimation results (motion vectors) are all stationary results (zero vectors). can get.

  Here, in the prior art described in Patent Literature 1, Patent Literature 2, and Non-Patent Literature 1, the region where the subject is moving, that is, (b) the region where motion is detected as the position estimation result (motion vector) ( (4505) and (4506) person part) can achieve the effect of higher resolution, but the area where the subject is not moving, that is, (b) no motion was detected as the position estimation result (motion vector) The area (the background portion of (4506) and the entire screen of (4507)) cannot achieve the effect of increasing the resolution.

  Therefore, as shown in FIG. 45 (c) high resolution signal processing method, the area where the subject is moving, that is, (b) the area where motion is detected as the position estimation result (motion vector) ((4505), and In the (4506) person portion), the same high resolution processing as before is performed, and the area where the subject is not moving, i.e., the area where no motion is detected as the position estimation result (motion vector) ((4506 In the background portion of () and the entire screen of (4507), the high resolution processing is performed by synthesizing each pixel with another frame in which motion is detected. That is, in the portion of the person (4505) and (4506), the frame #b and the frame #a are combined to increase the resolution. On the other hand, in the background portion of (4506), frame #c and frame #a are synthesized for each pixel to increase the resolution (4509). In addition, in the entire screen of (4507), the frame #d, and the combined result (frame # a + # b) of frame #a and frame #b are combined to increase the resolution (4510). In this way, by selectively changing the frame to be combined for each pixel according to the presence or absence of the movement of the subject, it becomes possible to increase the resolution of the entire frame regardless of the movement of the subject. It should be noted that the area where the movement of the subject is an integer pixel unit may be processed similarly to the area where the subject is stationary.

  The configuration of an image signal processing apparatus according to Embodiment 29 of the present invention will be described with reference to FIG. The configuration shown in the figure assumes that the frame # 1 is the latest frame and the frame # 2 is one frame older than the frame # 1, and the configuration of the image signal processing apparatus according to the first embodiment of the present invention ( The pixel / phase difference replacement unit (4607) is added to FIG. 1), and the frame # after the up-rate is determined according to the value of the phase difference θ (102) output from the position estimation unit (101). After replacing the pixel value of 2 and the value of the phase difference θ (102) for each pixel with the frame (4608) and the phase difference θ (4609), respectively, the motion compensation / uprate unit (115) described above and the phase The shift unit (116) and the aliasing component removal unit (117) are used to perform the same operation as the image signal processing apparatus according to the first embodiment of the present invention. Note that the motion compensation / uprate unit (115) has a configuration of an uprater (118) and a motion compensator (119) instead of the uprater (104) shown in FIG. As described with reference to FIG. 5, the operation of the up-rater (104) is changed to “the number of pixels by inserting a new pixel sampling point (= zero point) at a position intermediate the original pixel interval. Is divided into the operation of `` double the density to increase the density '' and the operation of `` filtering with all the frequencies between -fs and + fs being a passband with a gain of 2.0 '', and the former is the up-rater (118), The latter is merely replaced with the configuration of the motion compensator (119), and the motion compensation / up-rate units (115) shown in FIGS. 1 and 46 have substantially the same configuration.

  In the pixel / phase difference replacement unit (4607), the frame buffer (4603) and the adder (4606) and the frame buffer (4605) configured in a loop are used to output from the motion compensator (119). Record the frame and the cumulative value of the phase difference θ (102). In addition, when it is determined by the processability determination unit (4601) for each pixel whether or not normal resolution enhancement processing is possible according to the criteria described later, when it is determined that normal resolution enhancement processing is possible The switcher (4602) and switcher (4604) are both switched to the (a) side, the frame (4608) value is used as the output value of the up-rater (118), and the phase difference θ (4609) is the phase difference. As the same value as θ (102), the same operation as the configuration shown in FIG. 1 is performed.

  On the other hand, if it is determined that normal high resolution processing is not possible, both the switcher (4602) and switcher (4604) are switched to (b) side, and the value of frame (4608) is set to the value of frame buffer (4603). By using the adder (4606) to set the phase difference θ (4609) to the sum of the phase difference θ (102) and the output value of the frame buffer (4605) using the adder (4606). High resolution processing is performed using pixel values and phase differences of past frames. The frame buffer (4603) is configured to input and write the output value of the motion compensator (119), and the frame buffer (4605) is configured to input and write the output value of the adder (4606). Constitute. At this time, the data write (input) address of each frame buffer (4603) (4605) is set to point to the data at the position corresponding to the target pixel of frame # 1, and each frame buffer (4603) (4605) The data read (output) address is the data at the same position as the pixel on frame # 2 corresponding to the target pixel of frame # 1, based on the phase difference θ (102) output by the position estimation unit (101). To point to.

  If the pixel / phase difference replacement unit (4607) is configured as described above and it is determined that normal resolution enhancement processing is possible, the operation of the image signal processing apparatus shown in FIG. The operation is the same as described, and the normal resolution enhancement process shown in the first embodiment of the present invention is performed. At the same time, the past history (cumulative) is reset for the corresponding pixel position value of each frame buffer (4603) (4605), the latest motion compensation result is stored in the frame buffer (4603), and the frame buffer (4605) The latest value of the phase difference θ (102) is written.

  On the other hand, when it is determined that the normal resolution enhancement processing is impossible, the past pixel data of one frame or more is read from the frame buffer (4603), and the frame buffer (4605) reads from the frame buffer (4603). The phase difference θ between the output pixel data and the target pixel of frame # 1, that is, the phase difference θ (102) is accumulated based on the trajectory that the target pixel of frame # 1 has passed over each past frame. Since the value is output, even if the subject is stationary or the region has only motion in units of integer pixels, if the past frame has motion in units of decimal pixels, the high level shown in FIG. Resolution processing can be realized. Since the value is not yet written in each frame buffer (4603) (4605) immediately after the start of processing, both the switch (4602) and the switch (4604) are used at least for the period of the first one frame or more ( It is necessary to switch to the a) side and fix it so that erroneous data is not read from each frame buffer (4603) (4605). Also, depending on the value of the phase difference θ (102), the same position as the pixel on the frame # 2 corresponding to the target pixel of the frame # 1 may not exist. In this case also, the switch (4602) and the switch Both (4604) may be switched to the (a) side and fixed, and normal high resolution processing may be performed. Also, before reading the data in each frame buffer (4603) (4605), it is necessary to control so that other data is not written to the same address, but two equivalent frame buffers are prepared, The above control can be realized by using a general double buffer configuration in which two processes of data reading and data writing are made parallel by switching appropriately. Since these can be easily realized by the prior art, illustration is omitted. Also, in the configuration of the image signal processing apparatus shown in FIG. 46, the other parts except the pixel / phase difference replacement unit (4607) are the same as those shown in FIG.

  FIG. 47 shows an example of criteria for determining for each pixel whether or not normal resolution enhancement processing is possible in the processability determination unit (4601), that is, the switch (4602) and the switch (4604) are controlled. An example of how to do this is shown. The phase difference θ and the coefficients C0, C1, C2, and C3 shown in the figure are the same as the phase difference θ and the coefficients C0, C1, C2, and C3 shown in FIG. In the image signal processing device according to the fifth embodiment of the present invention and the image signal processing method according to the sixth embodiment, the coefficients C1 and C3 are indefinite when the phase difference θ is 0, or the phase difference θ is 0. In order to prevent the coefficients C1 and C3 from increasing and becoming vulnerable to noise and the like as they approach, switching to auxiliary pixel interpolation was performed when the phase difference θ was close to 0. Based on the same criterion, the switch (4602) and the switch (4604) are normally switched to the (a) side, and switched to the (b) side when the phase difference θ is close to 0. Note that the relationship between the operations of θ, the switch (4602), and the switch (4604) shown in FIG. 47 is not limited to this. For example, not only when θ = 0, but also when 0 ≦ θ ≦ π / 8 or 15π / 8 ≦ θ <= 2π (equivalent to 0), the switch (4602) and switch (4604) It may be switched to the b) side, and the “near 0” range may be set in consideration of the influence of noise and the like.

  Further, when the present invention is applied to an interlace / progressive scanning line (hereinafter referred to as IP conversion), it will be described in the image signal processing apparatus according to the twenty-first embodiment of the present invention and the image signal processing method according to the twenty-second embodiment. As described above, the coefficients C0, C1, C2, C3, and the states of the switch (4602) and the switch (4604) are values obtained by replacing the phase difference θ shown in FIG. 47 with (θ ± π). . That is, C0 = C2 = 1/2, C1 =-(1 + cos (θ ± π)) / (2sin (θ ± π)) = (1-cosθ) / sinθ, C3 = (1 + cos (θ ± π)) / (2sin (θ ± π)) =-(1-cosθ) / sinθ. At this time, it is possible to prevent the coefficients C1 and C3 from becoming indefinite when the phase difference θ is ± π, and the coefficients C1 and C3 to increase as the phase difference θ approaches ± π, thereby making it vulnerable to noise and the like. Therefore, in the image signal processing device according to the twenty-first embodiment of the present invention and the image signal processing method according to the twenty-second embodiment, switching to auxiliary pixel interpolation is performed when the phase difference θ is close to ± π. In the embodiment, the switching device (4602) and the switching device (4604) are normally switched to the (a) side according to the same criterion, and the (b) side when the phase difference θ is close to ± π. Switch to.

  Further, the pixel / phase difference replacement unit (4607) described above includes the configuration of FIG. 10 showing the third embodiment according to the present invention, the configuration of FIG. 11 showing the fifth embodiment according to the present invention, and the implementation according to the present invention. The configuration of FIG. 20 showing Example 7, the configuration of FIG. 21 showing Example 8 according to the present invention, the configuration of FIG. 27 showing Example 9 according to the present invention, and Example 21 according to the present invention were shown. The same applies to the configuration of FIG.

  In the image signal processing apparatus according to the 29th embodiment described above, when a high-resolution image is generated using a plurality of images and their phase differences, the region where the subject is stationary or the motion of the subject is an integer. Even in the region where the pixel unit is used, if there is a motion of a decimal pixel unit in the past, the effect of increasing the resolution can be obtained.

  The thirtieth embodiment relates to an image signal processing method in which processing equivalent to the image signal processing in the image signal processing apparatus according to the twenty-ninth embodiment is realized by a control unit that cooperates with software.

  First, an image processing apparatus for realizing the image signal processing method according to the present embodiment will be described with reference to FIG. The image signal processing apparatus shown in FIG. 48 stores, for example, an input unit (1) to which an image signal such as a television broadcast signal is input, and software for processing the signal input from the input unit (1). A control unit (10) that performs image signal processing on a signal input from the input unit (1) in cooperation with software stored in the unit (11), the storage unit (11), and a control unit (10) Output from frame buffer # 1 (21), frame buffer # 2 (22), frame buffer # 4 (36), frame buffer # 5 (37), and control unit (10) used for data buffer in the image signal processing A buffer # 3 (23) for frame-buffering the image signal processed signal output to the unit (3).

  Here, the number of input units (1) included in the image signal processing apparatus shown in FIG. 48 is the same as two, which is the number of frames used for image processing, but only one input unit (1) is provided. One frame may be input continuously.

  In addition, the frame buffer # 1 (21), the frame buffer # 2 (22), the frame buffer # 4 (36), the frame buffer # 5 (37), and the storage unit (11) for storing software are used for the data buffer. Each may be configured using individual memory chips, or one or a plurality of memory chips may be used to divide and use each data address.

  In this embodiment, for the image signal input from the input unit (1), the control unit (10) performs image signal processing in cooperation with software stored in the storage unit (11), and the display unit (3) Output to. Details of the image signal processing will be described with reference to FIG.

  FIG. 49 shows an example of a flowchart of the image signal processing method according to the present embodiment. The flowchart of FIG. 49 is obtained by replacing step (1406) in the flowchart of FIG. 14 showing the image signal processing method according to the second embodiment of the present invention with step (4907) in FIG. In step (1418), the image data of each frame is updated twice. That is, in step (1402), the image data of frame # 1 is updated and written to frame buffer # 1, and in step (1403), the image data of frame # 2 is updated and written to frame buffer # 2. Here, the up-rate can be realized by clearing the value of each frame buffer to 0 and then writing the data every other pixel.

  Next, in step (1404), the first pixel (for example, the upper left pixel) of frame buffer # 1 is set as a processing target, and processing is performed until all pixel data processing for frame buffer # 1 is completed. Loop.

  In step (1405), the position of the corresponding pixel in the frame buffer # 2 is estimated with reference to the target pixel in the frame buffer # 1, and the phase difference θ is output. At this time, the above-described conventional technique can be used as it is as a method of estimating the position of the corresponding pixel.

  In step (4907), based on the phase difference θ obtained in step (1405), first, in step (4901), a normal resolution enhancement process is performed according to the example of the determination criterion described with reference to FIG. It is determined whether or not it is possible. If possible, the process proceeds to step (4902), and if not possible, the process proceeds to step (4903).

  In step (4902), frame buffer # 2 is selected as a processing target, and the flow proceeds to step (4905).

  On the other hand, in step (4903), frame buffer # 4 is selected as a processing target, and the flow proceeds to step (4904). In step (4904), based on the phase difference θ output in step (1405), data at the same position (corresponding pixel position) as the pixel on frame buffer # 2 corresponding to the target pixel in frame buffer # 1 Is added to the phase difference θ output in step (1405) to obtain a new phase difference θ.

  In step (4905), the phase difference θ is written in frame buffer # 5, and the flow advances to step (4906). At this time, the data write address of the frame buffer # 5 is set so as to indicate the data at the position corresponding to the target pixel of the frame # 1.

  In step (4906), based on the phase difference θ, the pixel data value is compensated by motion-compensating the pixels in the vicinity of the corresponding pixel in the frame buffer selected in step (4902) or step (4903). Are written in the frame buffer # 4, and the process proceeds to step (1413) and step (1408). At this time, only the pixel data used in the π / 2 phase shift processing in step (1408), that is, pixel data in a range where a finite number of taps acts, need to be motion-compensated as neighboring pixels. The motion compensation operation is the same as the operation described with reference to FIGS.

  Subsequently, in step (1419), the phase is shifted by a certain amount using each neighboring pixel with respect to each of the frame buffer # 1 and the output pixel of step (4906). That is, in steps (1407) and (1408), the pixel data in each frame buffer is phase-shifted by π / 2.

  Subsequently, in step (1420), the coefficients C0, C1, C2, and C3 set so as to satisfy the conditions of FIGS. 9A, 9B, and 9C based on the phase difference θ are used. ) Is multiplied and added to remove the aliasing component from the pixel data in frame buffers # 1 and # 2, and output to frame buffer # 3. That is, in step (1409), coefficients C0, C1, C2, and C3 are determined based on the phase difference θ, and each coefficient and frame buffer # 1 are determined in steps (1410), (1411), (1412), and (1413). , The pixel data of # 2 and the data after the π / 2 phase shift are respectively multiplied, and then all are added in step (1414) and output to frame buffer # 3. The operation for removing the aliasing component is the same as that described with reference to FIG.

  Subsequently, in step (1415), it is determined whether or not the processing of all the pixels in the frame buffer # 1 has been completed.If the processing has not been completed, the next pixel (for example, the right adjacent pixel) is determined in step (1416). The process is set as a process target and the process returns to step (1405) and the subsequent steps.

  After the image signal processing of the flowchart shown in FIG. 49, the signal buffered in the frame buffer # 3 shown in FIG. 48 can be output to the display unit (3) in frame units or pixel units.

  By performing the processing as described above, it is possible to output a high resolution signal to the frame buffer # 3 using the pixel data of the frame buffer # 1 and the frame buffer # 2. When applied to a moving image, the process from step (1401) to step (1417) may be repeated for each frame.

  Further, the above-mentioned step (4907) includes step (1506) of the flowchart of FIG. 15 showing the fourth embodiment according to the present invention, step (1506) of the flowchart of FIG. 16 showing the sixth embodiment of the present invention, Step (1406) of the flowchart of FIG. 42 showing Example 22 according to the present invention, Step (1506) of the flowchart of FIG. 43 showing Example 24 of the present invention, and Example 26 of the present invention were shown. 44 can be similarly applied by replacing step (1506) in the flowchart of FIG.

  According to the image signal processing method according to the thirtieth embodiment described above, when a high-resolution image is generated using a plurality of images and their phase differences, the region where the subject is stationary or the motion of the subject Even in a region where the number of pixels is in the unit of integer pixels, if there is a motion in units of decimal pixels in the past, the effect of increasing the resolution can be obtained.

  As described above, in the conventional techniques described in Patent Document 1, Patent Document 2, and Non-Patent Document 1, since the resolution enhancement processing is performed using the difference in sampling phase (sampling position), sampling is performed. There is a problem that the effect of increasing the resolution cannot be obtained in a signal in which a phase difference does not occur, that is, in a region where the subject is stationary or a region where the motion of the subject is an integer pixel unit. In order to solve this problem, the configuration of an image signal processing apparatus according to Embodiment 31 of the present invention will be described with reference to FIG.

  In FIG. 50, an image signal processing apparatus according to Embodiment 31 of the present invention has n input units (where n is an integer of 3 or more) input units (for example, a moving image frame sequence such as a television broadcast signal). 1), a resolution conversion unit (2) for increasing the resolution of the frame input from the input unit (1), and further displaying an image based on the frame whose resolution has been increased by the resolution conversion unit (2) And a display unit (3). The operations of the motion compensation / uprate unit (115), the phase shift unit (116), and the aliasing component removal unit (117) constituting the resolution conversion unit (2) are shown in FIG. 1 according to the first embodiment of the present invention. Therefore, the description is omitted. The configuration and operation of the (n−1) position estimation units (101-2) (101-3)... (101-n) and the pixel / phase difference replacement unit (5003) will be described below.

  In the figure, the pixel to be processed on the frame # 1 input to the input unit (1) by the (n-1) position estimation units (101-2) (101-3) (101-n) With reference to the sampling phase (sampling position), the position of the corresponding pixel on each of (n-1) frames # 2, frame # 3 ... frame #n is estimated, and the sampling phase difference for each pixel θ2 (102-2), θ3 (102-3)... θn (102-n) are obtained. Subsequently, with respect to each sampling phase difference θ2 (102-2), θ3 (102-3)... Θn (102-n) in the processability determination unit (5001) in the pixel / phase difference replacement unit (5003). For example, based on the determination criterion shown in FIG. 51, it is determined whether or not the normal high resolution processing is possible. Thereafter, in the selection unit (5002), for example, according to the selection criterion shown in FIG. 52, (n−1) frames # 2, frame # 3,..., Frame #n and (n−1) sampling phase differences θ2 From (102-2), θ3 (102-3) ... θn (102-n), select a set of one frame (5004) and phase difference θ (5005) corresponding to this frame, and motion compensation Input to the up-rate unit (115), and the resolution conversion unit (2) performs high resolution processing.

  FIG. 51 shows an example of a criterion for determining, for each pixel, whether or not the normal resolution enhancement process is possible in the processability determination unit (5001). The phase difference θ and the coefficients C0, C1, C2, and C3 shown in the figure are the same as the phase difference θ and the coefficients C0, C1, C2, and C3 shown in FIG. In the image signal processing device according to the fifth embodiment of the present invention and the image signal processing method according to the sixth embodiment, the coefficients C1 and C3 are indefinite when the phase difference θ is 0, or the phase difference θ is 0. In order to prevent the coefficients C1 and C3 from increasing and becoming vulnerable to noise and the like as they approach, switching to auxiliary pixel interpolation was performed when the phase difference θ was close to 0. Based on the same determination criteria, it is determined whether or not normal resolution enhancement processing is possible. Note that the determination result as to whether θ and normal resolution enhancement processing shown in FIG. 51 are possible is not limited to this. For example, not only when θ = 0 but also when 0 ≦ θ ≦ π / 8 or 15π / 8 ≦ θ <= 2π (equivalent to 0), it is determined that normal high resolution processing is impossible. The range of “near 0” may be set in consideration of the influence of noise and the like.

  Further, when the present invention is applied to an interlace / progressive scanning line (hereinafter referred to as IP conversion), it will be described in the image signal processing apparatus according to the twenty-first embodiment of the present invention and the image signal processing method according to the twenty-second embodiment. As described above, the coefficients C0, C1, C2, C3, and the states of the switch (4602) and the switch (4604) are values obtained by replacing the phase difference θ shown in FIG. 51 with (θ ± π). . That is, C0 = C2 = 1/2, C1 =-(1 + cos (θ ± π)) / (2sin (θ ± π)) = (1-cosθ) / sinθ, C3 = (1 + cos (θ ± π)) / (2sin (θ ± π)) =-(1-cosθ) / sinθ. At this time, it is possible to prevent the coefficients C1 and C3 from becoming indefinite when the phase difference θ is ± π, and the coefficients C1 and C3 to increase as the phase difference θ approaches ± π, thereby making it vulnerable to noise and the like. Therefore, in the image signal processing device according to the twenty-first embodiment of the present invention and the image signal processing method according to the twenty-second embodiment, switching to auxiliary pixel interpolation is performed when the phase difference θ is close to ± π. In the embodiment, the same determination criterion may be used to determine that normal high resolution processing is impossible when the phase difference θ is in the vicinity of ± π.

  FIG. 52 shows an operation when the number of frames n = 7 as an example of the operation of the selection unit (5002). First, based on the determination result of the processability determination unit (5001) described above, (n-1) frames # 2, frame # 3,..., Frame #n and (n-1) input to the selection unit (5002) Of the sampling phase differences θ2 (102-2), θ3 (102-3) ... θn (102-n), only the set of frames and phase differences θ determined to be capable of normal high resolution processing. select. Here, when there is one selected set, the set is output as a frame (5004) and a phase difference θ (5005). On the other hand, when the selected group is two or more, one of them is further selected and output as the frame (5004) and the phase difference θ (5005). At this time, for example, a group of frames having the smallest frame difference (time difference) from the reference frame # 1 may be selected. Furthermore, if there are multiple frames that have the same frame difference (i.e., past frames and future frames as seen from frame # 1), the set of either one (for example, the past) is selected. Just choose. In addition, when there is no pair of the frame determined to be capable of normal high resolution processing and the phase difference θ, the frame having the smallest frame difference (time difference) from the reference frame # 1 (in the example of FIG. Select the group # 2).

  The pixel / phase difference replacement unit (5001) described above increases the number of input frames from 2 to n by increasing the number of input units (1) from 2 to n (where n is an integer of 3 or more). 10, the configuration of FIG. 10 showing the third embodiment of the present invention, the configuration of FIG. 11 showing the fifth embodiment of the present invention, the configuration of FIG. 20 showing the seventh embodiment of the present invention, and the present invention. The configuration of FIG. 21 showing the eighth embodiment, the configuration of FIG. 27 showing the ninth embodiment according to the present invention, and the configuration of FIG. 41 showing the twenty-first embodiment according to the present invention can be similarly applied.

  According to the image signal processing device according to the embodiment 31 described above, when a high resolution image is generated using a plurality of images and their phase differences, the area where the subject is stationary between adjacent frames Even in an area where the movement of the subject is in units of integer pixels, if there is a movement in units of decimal pixels between n frames (where n is an integer of 3 or more) to be processed including the past or future, high resolution can be achieved. An effect is obtained.

  Each embodiment of the present invention can be similarly applied to, for example, a DVD player, a magnetic disk player, or a semiconductor memory player in addition to the devices described in the above embodiments. For example, the present invention can also be applied to a portable image display terminal (for example, a mobile phone) for receiving 1-segment broadcasting.

  As the image frame, an image frame of a signal other than the television broadcast signal may be used. Further, for example, a streaming image transmitted via the Internet or an image frame of an image reproduced from a DVD player or an HDD player may be used.

  Further, in each of the above-described embodiments, the description has been given by taking as an example high resolution in frame units. However, the target for higher resolution is not necessarily the entire frame. For example, a part of the frame of the input image or input video may be set as the resolution target. That is, if the image processing according to the embodiment of the present invention described above is performed for a plurality of frames of a part of the frame of the input video, a high-quality enlarged image of the input image or a part of the input video can be obtained. . This can be applied to, for example, an enlarged display of a part of an image.

  It should be noted that any combination of the above-described embodiments can be an embodiment of the present invention.

According to each of the embodiments of the present invention described above, it is possible to perform processing for suitably converting a low-resolution image into an enlarged image, and it is possible to suitably obtain a high-resolution high-resolution image. That is, it is possible to suitably increase the resolution of the image signal.

In addition, according to the above-described embodiments of the present invention, the number of frames of an image necessary for obtaining a high-quality high-resolution image can be reduced.

It is explanatory drawing of Example 1 which concerns on this invention. It is a figure explaining an example of operation | movement of the general high resolution image signal processing. It is a figure explaining operation | movement of a prior art. It is a figure explaining operation | movement of Example 1 which concerns on this invention. It is explanatory drawing of Example 1 which concerns on this invention. It is explanatory drawing of Example 1 which concerns on this invention. It is explanatory drawing of Example 1 which concerns on this invention. It is explanatory drawing of Example 1 which concerns on this invention. It is explanatory drawing of Example 1 which concerns on this invention. It is explanatory drawing of Example 3 which concerns on this invention. It is explanatory drawing of Example 5 which concerns on this invention. It is explanatory drawing of Example 5 which concerns on this invention. It is explanatory drawing of Example 5 which concerns on this invention. It is explanatory drawing of Example 2 which concerns on this invention. It is explanatory drawing of Example 4 which concerns on this invention. It is explanatory drawing of Example 6 which concerns on this invention. It is a figure explaining the difference of operation of one embodiment of the present invention and the prior art. It is explanatory drawing of Example 2 which concerns on this invention. It is explanatory drawing of Example 12 which concerns on this invention. It is explanatory drawing of Example 7 which concerns on this invention. It is explanatory drawing of Example 8 which concerns on this invention. It is explanatory drawing of Example 7 which concerns on this invention. It is explanatory drawing of Example 7 which concerns on this invention. It is explanatory drawing of Example 7 which concerns on this invention. It is explanatory drawing of Example 7 which concerns on this invention. It is explanatory drawing of Example 7 which concerns on this invention. It is explanatory drawing of Example 9 which concerns on this invention. It is explanatory drawing of Example 9 which concerns on this invention. It is explanatory drawing of Example 9 which concerns on this invention. It is explanatory drawing of Example 9 which concerns on this invention. It is explanatory drawing of Example 9 which concerns on this invention. It is explanatory drawing of Example 9 which concerns on this invention. It is explanatory drawing of Example 10 which concerns on this invention. It is a figure explaining the difference of operation of one embodiment of the present invention and the prior art. It is explanatory drawing of Example 13 which concerns on this invention. It is explanatory drawing of Example 17 which concerns on this invention. It is a figure explaining operation | movement of a prior art. It is a figure explaining operation | movement of a prior art. It is explanatory drawing of Example 21 which concerns on this invention. It is explanatory drawing of Example 21 which concerns on this invention. It is explanatory drawing of Example 21 which concerns on this invention. It is explanatory drawing of Example 22 which concerns on this invention. It is explanatory drawing of Example 24 which concerns on this invention. It is explanatory drawing of Example 26 which concerns on this invention. It is explanatory drawing of the 32nd Example which concerns on this invention. It is explanatory drawing of the 32nd Example which concerns on this invention. It is explanatory drawing of the 32nd Example which concerns on this invention. It is explanatory drawing of the 33rd Example which concerns on this invention. It is explanatory drawing of the 33rd Example which concerns on this invention. It is explanatory drawing of the 34th Example which concerns on this invention. It is explanatory drawing of the 34th Example which concerns on this invention. It is explanatory drawing of the 34th Example which concerns on this invention. It is a figure explaining the difference of operation of one embodiment of the present invention and the prior art. It is explanatory drawing of the 11th Example which concerns on this invention. It is explanatory drawing of the 11th Example which concerns on this invention. It is explanatory drawing of the 11th Example which concerns on this invention. It is explanatory drawing of the 11th Example which concerns on this invention. It is explanatory drawing of the 11th Example which concerns on this invention. It is explanatory drawing of the 12th Example which concerns on this invention.

Explanation of symbols

1 ... input unit; 2,4 ... resolution conversion unit; 3 ... display unit; 5,6,7 ... high resolution task; 10 ... control unit; 11 ... storage unit; 21,22,23,31,32,33 , 34, 35, 36, 37, 460 3, 4605 ... buffer; 101, 1801, 2101, 5406, 5412 ... position estimation unit; 102, 211, 1802 ... phase difference θ; 103, 104, 1803 ... up-rate device; 105, 107, 1002, 1804 , 5402 ... delay device; 106, 108, 1805 ... π / 2 phase shifter; 109, 1007, 1103, 2301, 2302, 3001, 3002, 3003, 3004, 5701 ... coefficient determiner; 110, 111, 112, 113, 1003, 1006, 1102, 2202 , 2303, 2304, 2604, 2902, 3005, 3006, 3007, 3008, 5702 ... multiplier; 114, 1001, 1008, 1104, 2201, 2305, 2603, 2606, 2703, 2901, 3009, 4101, 5703,4606 ... 115, 1806, 2110, 5410 ... motion compensation / up-rate unit; 116, 1009, 1807, 2111, 5411 ... phase shift unit; 117, 1010, 1808, 2108, 2109, 2112, 2705, 2706, 5409 ... Folding component removal unit; 201, 202, 203, 20
6,1701, 1702,1703,1704,1705,2010,2011,2012,2013,2014,2015,3401,3402,3403,3404,3405,5301,5302,5303,5304 ... Frame; 204 ... Movement; 205 ... 207 ... motion compensation; 208,212 ... pixel; 209,210 ... sampling phase; 301,302,303,401,402 ... original component; 304,305,306,405,406 ... folding component; 403,404 ... original component after π / 2 phase shift; 1004, 2704 ... subtractor; 1005 ... Hilbert transformer; 1101 ... interpolation low-pass filter; 1105 ... auxiliary pixel interpolator; 2001 ... horizontal resolution converter; 2002 ... vertical up-rater; 2003, 2007, 2710, 2711 ... pixel 2004 ... Vertical interpolation unit; 2005 ... Vertical resolution conversion unit; 2006 ... Horizontal up-rate unit; 2008 ... Horizontal interpolation unit; 2009,2707 ... Mixer; 2102,5407 ... Horizontal phase difference θH; 2103,5408 ... Vertical Phase difference θV; 2104, 2105, 5401 ... Horizontal / vertical up-rater; 2106, 5403 ... Horizontal phase shift unit; 2107, 5404 ... Vertical phase shift unit; 2501 ... Frequency region where the effect of flat resolution conversion is large; 2502 ... Frequency region where the effect of vertical resolution conversion is large; 2601, 2602, 2605 ... Two-dimensional filter; 2701 ... Diagonal (lower right) phase shift unit; 2702 ... Diagonal (upper right) phase Shift unit: 3101 ... Frequency region where the effect of diagonal (upper right) resolution conversion is large; 3102 ... Frequency region where the effect of diagonal (lower right) resolution conversion is large; 3500 ... Image display device; 3501 ... Input unit; 3502 ... Recording / playback unit 3503 ... Content storage unit; 3504 ... Image signal processing unit; 3505 ... Display unit; 3506 ... Audio output unit; 3507 ... Control unit; 3508 ... User interface unit; 3600 ... Recording / playback apparatus; 3605 ... Image video output unit; ... Audio output unit; 3701 ... actual scanning line; 3702 ... interpolation scanning line; 3703 ... field; 3704,3705 ... frame; 3801 ... inter-field interpolation; 3802 ... intra-field interpolation; 3803,4001,4002 ... motion vector; 3902… Vertical resolution conversion; 4 102 ... Phase difference offset; 4103 ... Offset correction unit; 4602,4604 ... Switcher; 5405 ... Horizontal / vertical phase shift unit; 4601,5001 ... Processable determination unit; 4607,5003 ... Pixel / phase difference replacement unit; 5002 ... Select part

Claims (10)

An input unit for inputting a plurality of image frames;
With respect to the plurality of image frames input by the input unit, image data on a reference input image frame among the plurality of image frames and an input image temporally continuous with the reference input image frame The image data composition processing by the first image data composition method for compositing the image data on the frame, the image data on the reference input image frame, and the reference input image frame are not temporally unsatisfactory. A resolution conversion unit that performs switching between image data combining processing by the second image data combining method for combining image data on continuous input image frames; and
An image signal processing apparatus comprising:   The resolution conversion unit is configured to determine the first data based on a difference between a position of the image data on the reference input image frame and a position of the image data on the input image frame temporally continuous to the reference input image frame. 2. The image signal processing apparatus according to claim 1, wherein the first image data synthesizing method and the second image data synthesizing method are switched.   When the position of the image data on the input image frame that is temporally continuous with the reference input image frame does not change from the position of the image data on the reference input image frame, the resolution conversion unit 2. The image signal processing apparatus according to claim 1, wherein the image data is synthesized by the second image data synthesis method. An input unit for inputting a plurality of image frames;
A position estimation unit that estimates a sampling phase difference using image data on a reference input image frame among the plurality of image frames input by the input unit and corresponding image data on another input image frame; ,
A sampling phase difference accumulating unit for accumulating information of the sampling phase difference based on a trajectory of movement of image data on the reference input image frame;
Using the information of the accumulated sampling phase difference, a resolution conversion unit that synthesizes the image data on the reference input image frame and the image data on the input image frame at the accumulation start time to generate an output image frame;
An image signal processing apparatus comprising: An input unit for inputting a plurality of image frames;
Sampling using image data on a reference input image frame among the plurality of image frames and corresponding image data on other input image frames other than the reference input image frame among the plurality of image frames A position estimation unit for estimating a phase difference;
A selection unit that selects one input image frame from the other input image frames using the information of the sampling phase difference;
A resolution converter that combines the selected input image frame and the selected input image frame to generate an output image frame;
An image display apparatus comprising: a display unit that displays an output image frame generated by the resolution conversion unit.   The selection unit determines whether a sampling phase difference between the reference input image frame and an input image frame temporally continuous with the reference input image frame is close to 0 or 2π, and the determination 6. The image display device according to claim 5, wherein one input image frame is selected from the other input image frames based on a result. An input step for inputting a plurality of image frames;
For the plurality of image frames, image data on an input image frame serving as a reference among the plurality of image frames, and image data on an input image frame temporally continuous to the input image frame serving as the reference. Image data combining processing by the first image data combining method to be combined, image data on the reference input image frame, and input image frame temporally discontinuous to the reference input image frame A resolution conversion step of switching image data combining processing by a second image data combining method for combining image data;
An image signal processing method comprising:   The resolution conversion step is based on the difference between the position of the image data on the reference input image frame and the position of the image data on the input image frame temporally continuous to the reference input image frame. 8. The image signal processing method according to claim 7, wherein the first image data synthesizing method and the second image data synthesizing method are switched.   In the resolution conversion step, when the position of the image data on the input image frame that is temporally continuous with the reference input image frame does not change from the position of the image data on the reference input image frame, 8. The image signal processing method according to claim 7, wherein the image data is synthesized by the second image data synthesis method. An input step in which a plurality of image frames are input;
Position estimation step for estimating a sampling phase difference using image data on the input image frame serving as a reference and each corresponding image data on another input image frame among the plurality of image frames input in the input step When,
A sampling phase difference accumulation step for accumulating the information of the sampling phase difference based on a trajectory of movement of image data on the reference input image frame;
Using the accumulated sampling phase difference information, resolution conversion for generating an output image frame by combining image data on the reference input image frame and image data on the input image frame at the accumulation start time Steps,
An image signal processing method comprising:

Images