KR100505516B1 - Video signal conversion device and method - Google Patents

Abstract

An image signal conversion apparatus for converting a first image signal into a second image signal, wherein the first and second image signals include images of a plurality of different frames. A motion detector is operable to detect motion of the first image signal between the first frame and the second frame, and the processing circuitry is based on the assumption of the pixel at the position corresponding to the detected motion. Generate a second video signal. The device can be used to produce an output signal having the same resolution as the first image signal with reduced or eliminated aliasing distortion, or higher resolution in the vertical and / or horizontal directions.

Description

Video signal conversion device and method

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention generally relates to image processing, and more particularly, to an apparatus and method for converting image signals that reduce aliasing distortion due to undersampling of an image. The present invention also relates to a technique for generating a high resolution image by increasing the number of pixels of a low resolution image.

With modern digital imaging techniques, analog video signals are sampled and stored as digital data for sequential playback on cathode ray tubes (CRTs) or other displays. The design goal is to display these images with the highest possible resolution and little distortion.

Images displayed on CRTs, such as images from television signals, video tape recorders (VTRs), or digital versatile disc (DVD) players, are generated by repeated scanning of electron beams in the horizontal direction. As shown in Fig. 25, each horizontal scan is performed from left to right, and after each scan, the beams are repositioned in the vertical direction so that the beams go back to the leftmost and start the next horizontal scan. The fluorescent screen of the CRT is irradiated with three electron beams to illuminate each blue, green and red phosphors distributed in small units on the screen. Phosphorescence creates light spots that correspond to the intensity of the electron beams, and a collection of all the points produces an image. Thus, the displayed image can be regarded as a collection of these points, ie pixels.

Since the image displayed on the CRT derived from the analog signal consists of a set of luminescent pixels, the image can be thought of as a digital signal obtained by sampling the original image at pixel locations. Thus, if the original analog image is sampled at sufficient sampling intervals in the horizontal and vertical directions to generate the same number of points as the number of pixels on the CRT, then the set of image data can be stored digitally. When played back sequentially, images of approximately the same resolution as in the strict analog recording / playback approach are obtained.

Sampling theorem indicates that an analog signal can be completely reconstructed from a set of evenly spaced discrete samples in time, assuming that the signal is sampled at a rate of at least twice the highest frequency component of the signal. When sampling the original image, if the sampling theory is not satisfied, aliasing distortion occurs in the displayed image. To correct the aliasing distortion, a prefilter is used to compensate for undersampling in the horizontal (scanning) direction, but such a prefilter is typically not provided in the vertical direction. As such, aliasing distortion in the vertical direction is a common problem.

26 illustrates aliasing distortion of an image displayed in the vertical direction. Each of the frames N and N + 1 shows four pixels P ₁ to P ₄ in a predetermined column. Signals S _N and S _{N + 1} represent the amount of change in the image level of the original image in the vertical direction for a given column, where the amplitude level is shown horizontally in the figure. Thus, for example, in frame N, the brightness of the image is higher at pixel P ₂ than at pixel P ₁ . Now, if the highest spatial frequency component of the original image in the vertical direction has a period less than twice the horizontal space between pixels, aliasing distortion occurs because the sampling theory is not satisfied. This is the case of signals S _N and S _{N + 1} in FIG. 26. For example, the signal S _N ′, which is an approximation to the sampling signal of the frame N, is significantly different from the principle signal S _N. With aliasing distortion, even when a filter is used that eliminates aliasing in the vertical direction, high frequency components of the original signal are lost during reproduction. This aliasing distortion causes attenuation in signal processing such as Y / C separation, noise cancellation, and qualitative improvement.

Note that although undersampling of the image always results in a reduced resolution as discussed, the observer's effect on image quality depends on how the scene changes from frame to frame. If the frame including the aliasing distortion is significantly changed from frame to frame as in the case of FIG. 26, an image or blurring that is not naturally seen from the user's point of view occurs. If the picture remains stationary, the aliasing noise is not great. In any case, the resolution is always attenuated by undersampling and the signal of a standard television broadcast is intended only for a limited number of horizontal sweeps per frame, eliminating aliasing and recapturing the original picture with improved quality ( recapture) A practical method is needed.

It is therefore an object of the present invention to provide a method for converting an undersampled video signal into an output video signal with reduced or eliminated aliasing distortion.

Still another object of the present invention is to generate a high resolution image from a low resolution image.

Another object of the present invention is to improve the quality of displayable images generated from low quality input image signals.

Various other objects and advantages of the present invention will be readily apparent to those skilled in the art, in particular new characteristics being indicated in the appended claims.

In an embodiment of the present invention, an image signal conversion apparatus for converting a first image signal into a second image signal is provided, and the first and second image signals each include images of a plurality of different frames. The apparatus includes a motion detector operable to detect motion of the first image signal between the first frame and the second frame, and processing for generating a second image signal based on pixel assumptions at positions corresponding to the detected motion. It includes a circuit.

The device is used to produce an output signal having the same resolution as the first video signal with reduced or eliminated aliasing distortion, or higher resolution in the horizontal and / or vertical directions. The motion detector preferably detects motion of the first image signal by an amount finer than the pixel size of the first image signal. The processing circuit stores the image of the first image signal and has a storage capacity having a storage capacity larger than the amount of data in one image of the first image signal, and writes the first image signal to the resolution generation memory and writes a new image signal from the memory. And a controller operable to control reading of. The controller writes the first video signal to the memory in accordance with the detected motion of the first video signal.

In another embodiment, the image conversion device comprises a region divider defining at least first and second image regions of the image, the motion of which is detected in each image region. In this case, the first and second image regions of the second image signal are generated based on the assumption of the pixels at the positions corresponding to the motion detected in the first and second image regions of the first image signal, respectively. The combiner combines the first and second image regions of the second signal to generate a composite image.

In one application, the first image signal is a high definition or high definition (HD) image signal as a second image signal having twice the resolution of a standard definition signal or a standard definition signal in both the horizontal and vertical directions. SD video signal is converted into. In some frames, pixels of high definition (HD) signals may also be generated by adaptive processing techniques.

1, a simplified block diagram of a television receiver 10 according to the present invention is shown. The receiver 10 converts the input television signal into a digital signal, corrects the digital signal for aliasing distortion while enhancing the resolution, and also converts the distortion corrected digital signal back to an analog signal for display.

The receiver 10 includes a tuner 1 for demodulating an input television broadcast signal of a selected band received via an antenna or transmission line. The demodulated signal is applied to a low pass filter (LPF) 2 which acts as an aliasing distortion prefilter to filter the high frequency components of the television signal. The output signal of the filtered LPF 2 is applied to an analog-to-digital (A / D) converter 3, where it is sampled and digitized. The digitized signal is then provided to a distortion corrector 4 which reduces or essentially eliminates aliasing distortion in the vertical direction of the digitized television signal. An image processing circuit such as a Y / C separator 5 is used between the distortion corrector 4 and the D / A converter 6 (or alternatively between the tuner 1 and the LPF 2). The Y / C separator 5 separates the luminance from the color difference in the output signal from the distortion corrector 4. Note that in the illustrated configuration, a composite signal (luminance + color difference) is supplied to the A / D converter 3, and sampling is performed according to the frequency of the color difference subcarrier. Alternatively, if Y / C separation of the television signal is performed before the signal is applied to the A / D converter 3, sampling is performed at a frequency on the order of 13.5 MHz. In any case, the D / A converter 6 converts the distortion corrected output signal of the distortion corrector 4 into an analog signal for display on a cathode ray tube (CRT) 7.

It is now described to reproduce the original image from the image having aliasing distortion in the vertical direction, that is, perpendicular to the horizontal scanning line. Referring to Fig. 2A, the principle of the aliasing distortion removal method executed in the distortion corrector 4 is described. The change amount of the image signal in the vertical direction in the predetermined column C _i of each image frame is represented by the signals S _N to S _{N + 4} for the frames N to N + 4, respectively. In each frame, pixels P ₁ to P _K of columns C _i are placed at respective positions of the conventional horizontal scan lines SL ₁ -SL _K. The signal level of each pixel is shown horizontally in the figure--for example, in frame N, signal levels V _1- (V _K ) correspond to pixels P _1- (P _K ), respectively. .

According to one embodiment of the present invention, pixels are created at locations between conventional horizontal scan lines to remove aliasing distortion and to improve the resolution of the image that is selectively displayed. Pixels between these scan lines are generated from frame to frame according to the motion of the image. To illustrate this concept, first a reference is made to frame N, where the amount of signal change of the original image S _{N is} shown along column C _i for that frame. Due to the high spatial frequency in the signal S _N , the vertical sampling at the position corresponding to the horizontal scanning line is insufficient to reproduce all the image information therein, whereby without aliasing distortion of the present invention, aliasing distortion Will be created. That is, the vertical image signal based on simply sampling one frame at the same time in the conventional horizontal scanning line resembles the distorted signal S _N '(shown in dashed lines), which is the original image signal S _N. Significantly different).

Frame-to-frame motion of the video signal is detected to generate pixels between scan lines that recapture the original signal. In frame N, signal values V ₁ -V _K for each pixel are stored in memory. Subsequently, the motion of the image is detected between the frames N and N + 1. (The method of detecting frame to frame motion is discussed later.) In the example of FIG. 2A, the image is moved down by the distance D ₁ between frame N and (N + 1). In frame N + 1, the image is resampled on the horizontal scan line to generate signal values V ₁ '-(V _K ') for pixels P _1- (P _K ) of frame N + 1. These values are stored in memory. In addition, interscan pixels P _1a , P _2a ,..., P _Ka are generated and stored in memory, where pixel P _1a is frame N relative to pixel P ₁ . With the signal value of V1 determined from the sample, the pixel P _2a has the signal value of V ₂ determined from the frame N sample for the pixel P ₂ . The position of pixel P _1a in frame N + 1 is the distance D ₁ from scan line SL ₁ , which is determined based on the motion from frame N to (N + 1). Similarly, pixel P _2a lies at a distance D ₁ below scan line SL ₂ .

The pixel generation process is repeated for frame N + 2, where the image signal S _{N + 2} is the signal value (V ₁ '')-for each pixel P _1- (P _K ) of the frame. Sampled at the horizontal scan line to determine (V _K ''), and motion from frame N + 1 to frame N + 2 is detected. Assuming the image is moved down by the distance D ₂ , (V ₁ '), (V ₂ '), ..., (corresponding to a predetermined value stored in memory for frame N + 1). Pixels P _1b , P _2b ,..., And P _Kb having a V _K 'value are generated, each located at a distance D ₂ below each horizontal scan line SL ₁ . Similarly, frame N + 2 is pixel P _1a with values of (V ₁ ), (V ₂ ), ..., (V _K ), respectively, as previously stored in memory for frame N + 1. ), (P _2a ), ..., (P _Ka ), each located at a distance D ₁ + D ₂ below (SL ₁ ).

The pixel generation process proceeds for sequential frames until a predetermined number of additional pixels are generated. In the example of FIG. 2A, the process continues to frame N + 4 with a pixel five times as large as frame N in the vertical direction. 2B shows a pixel between scan line SL ₁ and SL ₂ of frame N + 4. Video frame (N + 2) and (N + 3) the distance between the (D ₃₎ as also assumed to move down by a distance (D ₄₎ between the frame (N + 3) and (N + 4) do. Thus, the frame (N + 4) are located (SL ₁₎ signal (S _{N + 4)} level value (V ₁ '''') a pixel (P ₁₎ as well each (V ₁ of the '' corresponding to the in (P _1d ), (P _1c ), (P _1b ), and (P _1a ) of the values '), (V ₁ ''), (V ₁ '), and (V ₁ ), where (P _1d ) is the distance D ₄ below (SL ₁ ) and (P _1c ) is the distance D ₃ + D ₄ below (SL ₁ ). Of course, more or fewer additional pixels may be generated as a function of the desired resolution. In any case, in the given example, every frame following frame N + 4 will have five times the pixels of the horizontal scanning line in the vertical direction.

Thus, pixel generation is accomplished by assuming a pixel at a position corresponding to motion. However, it is desirable to represent the image as a substantially uniformly spaced sample. In the above example, if the inter-frame motion is not uniform such that the distances D ₁ -D ₄ are not equal, the resulting pixels are left unevenly spaced. However, by rounding the interframe motion properly, evenly spaced pixels can be derived and stored in memory. Assuming that the frames N + 4 are represented by evenly spaced pixels in the vertical direction (5 times the number of scan lines in this example), then each frame will be represented by the same number of evenly spaced pixels. Can be.

In the example of FIG. 2A, it is assumed that the intra-frame motion of the image is strictly in the vertical direction. Horizontal motion must also be considered to remove aliasing distortion in the vertical direction and increase the number of pixels when the original image is moved both in the horizontal and vertical directions. To this end, motion vectors are calculated from frame to frame as will be discussed in more detail below. In addition, images of other parts within the frame move in different directions. Certain embodiments of the present invention also take this diverging motion into account by separating and analyzing different blocks of each frame.

3 is a diagram illustrating a general concept of improving resolution and eliminating aliasing by generating pixels based on image data from a previous frame and frame to frame image movement. The upper part of the figure shows frames N to (N + 3) each having image data 101-104 along a common horizontal scanning line. Assuming the image is moved incrementally down from frame N to (N + 3), additional pixels are generated between the scan lines based on the data from the previous frame and the image motion as described above. Thus, frame N + 1 is promoted to pixel data 101 of frame N; The pixel data 101, 102 is added to the frame N + 2, and the pixel data 101, 102, and 103 are added to the frame N + 3 so as to be perpendicular to the original frame. As a result, a high resolution image having four times the pixels is generated.

Referring now to FIG. 4, a block diagram illustrating the configuration of the distortion corrector 4 is shown. The frame memory 11 receives input image data applied to the distortion corrector. It is assumed that the input image data corresponds to the undersampled image signal so that aliasing distortion is generally given when no resolution enhanced pixel is added and the frame is simply reproduced. The frame memory 11 is composed of a current frame memory 11A for storing image data of a current frame and a preceding frame memory 11B for storing image data of a frame immediately preceding the current frame. Motion detector 12 detects a motion vector representing the motion of the current frame relative to the preceding frame. This interframe motion is detected in a subtle amount than the pixel size (or pixel to pixel space) in the distorted image in the vertical direction. The given discussion assumes that the images of the entire frame are moved together so that the motion from the preceding frame to the current frame is the same in all parts of each frame. In this case, only one motion vector is detected by the detector 12 for each frame. The motion vector is provided to the controller 14.

One way to determine the motion vector is to perform multiple comparisons between blocks of equal size pixels in adjacent frames, for example blocks of 8x8 or 16x16 pixels. Starting with the main block of the current frame, the image features of a plurality of blocks (reference blocks) in the preceding frame are compared to the main block. When the reference block in the preceding frame is found to have the image feature closest to the main block, the reference block can be determined to be moved to the position of the main block, and a motion vector can be determined.

The main block is compared to the reference block by calculating the absolute difference between the pixel values of the pixels in the reference block for the corresponding pixels of the reference block. Thus, the signal value of the pixel P ₁₁ (1 row 1 column) in the main block is subtracted from the signal value of the pixel P ₁₁ in the reference block. The difference value is called an error. The errors of each reference block for the main block are summed and the reference block with the minimum total error is determined so that the motion vector is approximately in the direction corresponding to the positional relationship between the reference block and the main block. In particular, as shown in FIG. 5, linear interpolation between reference blocks is performed to more accurately determine the motion vector. The intersection of a pair of linear interpolated lines with opposite slopes defines the relative position where the minimum error occurs between the main block and the assumed reference block. The intersection defines the motion vector for the main block.

Continuing with reference to FIG. 4, a scene change detector 13 is used to detect scene changes in television signals by analyzing the difference between images from frame to frame. In general, when the difference between the image of the current frame and the preceding frame exceeds a predetermined threshold, a scene change is detected. Methods for detecting scene changes are already known in the art. For example, the scene change detector 13 is configured by analyzing the error between the main block and the reference block in a manner similar to determining the motion vector and finding the minimum error value for each main block forming the current frame. If the sum of the minimum error values for each block exceeds a predetermined threshold, a scene change is detected and the scene change signal is provided to the controller 14.

The controller 14 controls to write the data from the frame memory 11 to the resolution generating memory 15 in accordance with the motion vector supplied from the motion detector 12 and the signal from the scene change detector 13. The resolution generating memory 15 may store more image data than data in one low resolution frame in the frame memory 11 so as to store newly generated high resolution frames that are sequentially shifted. The controller 14 controls the recording of the image data from the memory 11 to the memory 15 by determining an appropriate storage address in the memory 15 according to the motion vector. The high resolution frame is generated in the memory 15 in a manner similar to that described above in connection with FIG. 2A. Briefly, the data in the current frame memory 11A (or from the preceding frame memory 11B) is stored at the same pixel position in the frame along the scan line, so as to generate pixels between scan lines (pixels with enhanced resolution). 15) is delivered in high resolution "frames". Subsequently, the image data of the next frame is recorded in the memory 15 at the position interpreted according to the motion vector. The controller 14 supplies an address pointer relative to the memory 15 to perform data transfer to the analyzed position, and leaves at least some image data of previous frames stored in the memory 15 intact in the same storage location. Once the frame is generated, it is sent from the memory 15 to the vertical low pass filter 16 under the control of the controller 14.

Following the screen change, as is apparent from FIG. 2A, it takes the time required to receive and process several low resolution frames to produce a true high resolution frame. Thus, when scene changes occur, interpolation between low resolution image data is performed to initially generate additional pixels in the memory 15.

6 illustrates storing image data in the resolution generating memory 15. The memory 15 may store image data of (P _H ') pixels in the horizontal (scanning) direction and (P _V ') pixels in the vertical direction. For example, in the following description, P _H '> P _H and P _V '> 4P _V , where P _H and P _V are the number of pixels in the horizontal and vertical directions for the low resolution frames stored in the frame memory 11, respectively. Assume that In this case, the resolution generating memory 15 may store more than four times as many pixels in the vertical direction and a larger number of pixels in the horizontal direction than the low resolution image. The term " distorted image " or " distorted data " used hereinafter is used to refer to an image or data stored in the frame memory 11 in which the data corresponds to an image undersampled in the vertical direction. However, as described above, distorted reproduction images are produced only when data is read directly from the frame memory 11 and reproduced frame by frame. In this embodiment, the "distorted" image data is used to generate an undistorted reproduction image.

In the resolution generating memory 15, the data storage location can be defined with an absolute address and a relative address. 6 shows data storage according to an absolute address, where the absolute address includes all storage locations of the memory 15. The first row and column of the absolute address storage area are represented by zero rows and zero columns. As such, the pixels in the (i + 1) th row and the (j + 1) th column are represented by absolute addresses (i, j).

As shown in FIG. 7, in the absolute address storage array of the memory 15, an access region 15a corresponding to a P _H × 4P _V pixel is defined. The position of the upper left point of the access area 15a is defined by a relative address pointer R _P as indicated by an arrow in FIG. 7. The address pointer R _P may actually be a code word supplied by the controller 14 to control a circuit (not shown) in the memory 15. The address pointer operates to control the position at which incoming image data is to be recorded. In the case of a larger absolute address area, the access area 15a also begins with rows 0 and 0.

8 illustrates a method in which video data transmitted from the frame memory 11 is recorded in the access area 15a. In the horizontal direction of the access area, image data of the same number of pixels as in the frame of the frame memory 11 is stored, while in the vertical direction, data of four times as many pixels are stored. Thus, as shown by the hatched regions in FIG. 8, writing the distorted image from the frame memory 11 to the access region 15a in the horizontal direction is sequentially performed from an address defined by the address pointer R _P. It is executed but every four lines in the vertical direction. In the grating of FIG. 8, pixel data between scan lines, for example, rows 1-3, 5-7, and the like, are generated and stored according to a motion vector to generate a high resolution frame in the vertical direction. In order to generate a higher resolution than the distorted image, the motion vectors are detected in units more delicate than the pixel size of the frame in the frame memory 11.

Data storage in resolution generation memory 15 is now described in more detail with reference to the flowchart of FIG. 9. When the controller 14 receives a signal from the scene change detector 13 indicating a scene change from the preceding frame to the current frame, it resets (clears) all data in the memory 15 (step). S1). It is noted here that the scene change signal is also received when the first frame of data is received by the frame memory 11. (The operation by the controller 14 is the same as to whether data of the first frame is received or a scene change occurs.) In step S2, the controller 14 determines that the distorted data of the current frame memory 11A is not present. As shown in FIG. 8, recording is made in the access area 15a every fourth line. Since the storage position is reset in advance, a gap is given to the access region 15a between the storage positions corresponding to the scanning lines at this point. These gaps are filled through interpolation performed by the controller 14 in step S3. That is, the controller 14 calculates interpolation values for the pixels between scan lines, and causes these values to be stored as data in the access area 15a. Then, the image data including the interpolation value is read out from the memory 15 in step S4 and passed to the vertical low pass filter 16.

In step S5, the data in the current frame memory 11A is shifted to the previous frame memory 11B, the data of the next frame is received and stored in the current frame memory 11A, which again causes a scene change? Determine. Otherwise, motion is detected by the motion detector 12 (step S6), and the motion vector is calculated. Then, the controller 14 moves the relative address pointer according to the motion (step S7), and the data is written into the high resolution frame in the memory 15 according to the motion vector. This operation can be executed in a plurality of ways. One method is to transfer the previous data stored in the access area 15a to a temporary buffer memory in the controller 14 and rewrite the data back into the access area 15a but shift according to the motion vector. Following the shift, data of the current frame memory 11A (not yet written to the memory 15) is written to every same line to the same access area 15a previously defined, i.e., to the same scan line position. As such, some of the previous data is overwritten. In this approach, as previously described in Figures 2A and 3, where data from each preceding frame is sequentially shifted in a direction corresponding to motion, the previous high resolution frame data is shifted to each frame in accordance with motion. (However, one difference is that interpolated data is not assumed in the method of FIG. 2A or FIG. 3-a gap of high resolution frames is assumed up to the fifth frame of FIG. 2A and to the fourth frame of FIG. 3.)

While the above approach of shifting the storage location of high resolution data in the direction relative to the motion vector is sufficient to produce a suitable high resolution frame, a good approach is to shift the new low resolution data of the current frame memory 11A to the previous data in the access area. The previous data in the access area 15a is kept in the same storage location while recording at the location. In other words, the data of frame N + 1 is written to a storage location with an absolute address different from the data of frame N whenever the motion vector for frame N + 1 is not zero. At the same time, since the absolute address of the old data in the memory 15 remains the same, the net effect is the shift of the old data over the new data. This approach is implemented by simply corresponding the motion vector but shifting the relative address pointer in the opposite direction. Thus, the relative addresses of the low resolution (distorted) data remain the same (e.g., rows 0, 4, etc. as shown in Figure 8), but the relative addresses of the preceding high resolution frame data change in the direction associated with the motion vector. do. Basically, the physical access area 15a of the memory 15 is shifted to each new relative address pointer.

With each new motion vector, the relative address pointer is moved by the same number of pixels as the number of pixels in the horizontal direction of the motion vector. However, in the vertical direction, the relative address pointer is moved by the number of high resolution pixels that are four times the number of low resolution pixels in the vertical component of the motion vector (since there are four times the pixels in the low resolution frame in the vertical direction of the high resolution frame). Of course, the relative address pointer is rounded to the nearest pixel corresponding to the motion vector (or the motion vector is rounded). So, for example, if the vertical component of the motion vector is between 1/8 and 3/8 of the original low resolution pixel size, the relative address pointer is shifted by one vertical unit. (Another method for simple rounding of motion vectors is discussed later.)

Continuing with reference to FIG. 9, recording new low resolution (distorted) image data in the memory 15 according to the relative address pointer (step S8) generates a new high resolution frame, and then in step S9 the vertical LPF It is read at 16. If in step S10 it is determined that the low resolution data is no longer applied to the frame memory 11, the routine is terminated. Otherwise, the routine returns to step S5 where the process is repeated for each sequential input image data frame.

Thus, by repeating steps S5 to S9 several times without changing the scene, interpolation data that fills the gap between scan line samples is replaced by image samples sequentially. In several frames after the scene change, the image frame becomes a true high resolution image, and the highest frequency in the vertical direction contained in the original image is less than half of the frequency corresponding to one quarter of the horizontal scanning period of the distorted image. When reproduced, the reproduced image does not have aliasing distortion in the vertical direction.

Referring to FIG. 4, the vertical LPF 16 operates to limit high frequency components in the image data from the resolution generating memory 15 by filtering in the vertical direction with a low pass filter. The purpose of such a filter process is to prevent aliasing distortion even when sequentially reproduced images are low resolution images. For example, if only a low resolution display is available, a low resolution image is displayed but nonetheless aliasing distortion is removed. As a result, when the images are reproduced sequentially, the observer does not recognize the unnatural or blurred motion image. The output signal of the vertical LPF 16 is applied to the frame memory 17 having the access area 15a of the memory 15 of the same storage capacity. The frame memory temporarily stores image data, and then, for example, produces a low resolution image which is read every fourth line and has the same number of lines as the original distorted image, but with no aliasing distortion.

Thus, the method described above makes it possible to improve the image quality and to avoid the drawbacks associated with certain signal processing operations such as Y / C separation, noise reduction, and the like.

A person skilled in the art can produce a high resolution image from an undersampled image according to a motion vector from frame to frame, and then filter it with a vertical LPF, and also read only the low resolution image. It will be readily understood to provide a. On the other hand, if the input low resolution image data is simply applied to the vertical LPF, such aliasing distortion removal is not realized.

If the CRT 6 can display a high resolution image by using more horizontal scan lines (e.g., four times the scan line) than the undersampled frame, the image data can be output directly for display. Thus, in this case the vertical LPF 16 is not used, and the data is provided directly to the frame memory 17 so that the data is read therefrom so as to be selectively displayed rather than every four lines.

When the highest frequency in the vertical direction included in the original image is higher than 1/2 of the frequency corresponding to one quarter of the horizontal scanning period, the sampling theory does not satisfy the above approach of generating four times the pixels, so the vertical direction Aliasing distortion in is not completely removed. Nevertheless, the method continues to be useful in that aliasing distortion is substantially reduced.

In the above embodiment, the number of pixels in the vertical direction is increased by a factor of four, but in other methods, more or less interline pixels may be generated for the purpose of displaying a high resolution image or a low resolution image having a reduced aliasing distortion.

In the above-described distortion corrector 4a, it is assumed that all images of a predetermined frame are moved together, i.e., the same distance from frame to frame in the same direction. In other words, the motion vectors for all parts of each frame are assumed to be the same. The assumption is generally valid for certain applications, such as pan and tilt operations. However, in most video applications different portions of each frame are moved in different ratios and / or in different directions. Embodiments of the present invention, which will be described later, can reduce aliasing distortion while considering such independent movement.

Referring now to FIG. 10, shown is a block diagram of another embodiment of a distortion corrector in accordance with the present invention. The distortion corrector 4b has the function of identifying frame-to-frame motion for each frame of a plurality of parts and generating a high resolution frame based on the motion. High resolution frames are used to generate low resolution or high resolution images with no aliasing distortion in any case.

The distortion corrector 4b is basically the same frame memory 11, motion detector 12, scene change detector 13, vertical low pass filter 16, and frame memory as described above for the distortion corrector 4a. (17). The motion detector 12 is operated as described above to analyze the difference between the plural reference blocks of the previous frame and the main block to obtain the motion vector of the current frame main block, e.g. 8x8 or 16x16 pixels, from the preceding frame. Decide In the distortion corrector 4a, the motion vector of one main block is sufficient to determine the motion of the entire frame image. On the other hand, with the distortion corrector 4b, a plurality of motion vectors corresponding to different main blocks are determined for each frame. The motion vectors of the various main blocks are provided to a region divider 21 which divides the frame region into multiple regions for independent processing operations. The region is based on the difference between the motion vector and the adjacent pixels and is defined through other operations as necessary, such as smoothing. For example, if the screen includes a fixed or slow moving object such as a background and a fast moving object such as an airplane, the area divider 21 defines the first area of the frame corresponding to the background and the second area corresponding to the plane. . In FIG. 10, the region divider 21 is applied to define two regions per frame, but it is understood that it may alternatively be configured to divide each frame into three or four regions in addition to only two regions.

Once the area divider 21 defines the image area for the current frame, it is passed through the switch 22A and only data from the second area to be temporarily stored in the memory 23A. By controlling the switches 22A and 22B to be transmitted through the switch 22B to be stored in this memory 23B, the reading of the image data from the current frame memory 11A is controlled. The region divider 21 also provides the controller 14A with a motion vector for one zone in the first region and a controller with motion vector for one zone in the second region. The scene change detector 13 supplies a scene change signal to the controllers 14A and 14B. The controller 14A controls writing of data from the memory 23A to the resolution generating memory 15A by dynamically changing the relative address pointer based on the motion vector in the first area for each frame. As a result, interscan pixels are basically created at positions according to the motion vectors in the manner described above. Thereby, a high resolution frame or frame portion having a resolution four times that of the frame memory 11A in the vertical direction is generated in the memory 15A. Similarly, the image data of the second frame region is transferred from the memory 23B to the memory 15B under the control of the controller 14B according to the second motion vector to make a high resolution frame or frame portion in the memory 15B. That is, except that only a part of the entire image is enhanced and stored in each of the memories 15A and 15B, each address pointer is dynamically moved in the manner described above for the controller 14 and the memory 15 of FIG. By determining each of the controllers 14A and 14B operate in association with the memory 15A and 15B, respectively.

The data of each high resolution frame generated in memories 15A and 15B is transferred to combiner 24 under the control of controllers 14A and 14B, respectively. The combiner 24 combines the data of the two frame portions to produce the image data of the composite frame and provides it to the vertical LPF 16. Thus, the output image data of the combiner 24 is similar to the memory 15 in the distortion corrector 4a. The vertical LPF 16 and the frame memory 17 execute the processing as described above to produce a low resolution image from which the aliasing distortion is removed. Conversely, if a high resolution image is desired, the vertical LPF is designed or bypassed with a different frequency response than discussed above.

Distortion correctors 4a and 4b have been described as being operated to generate interscan pixels in the vertical direction based on a motion vector (ie, a motion vector corresponding to both vertical and horizontal motion of the image), but in the horizontal direction Image data for additional pixels of may also be generated. That is, high resolution pixels can also be created in memory 15 (or memory 15A, 15B) with positions between low resolution pixel locations in the horizontal direction in a similar manner as described for the vertical direction. As a result, aliasing distortion can be further eliminated in the horizontal direction, so that a horizontally higher resolution can also be obtained.

There is a special case where the access area is increased than the storage area of the resolution generating memory 15 (or 15A, 15B) depending on the position of the relative address pointer. In this case, the data of the increased area can be obtained by enlarging the storage area of the resolution generating memory to the adjacent area. X≥P _H 'and Y≥P _V' of when the home position and, indicated by the absolute address (X, Y) to be included in the access area, the position is (mod (X, P from the storage area of the memory 15 _H '), mod (Y, P _V ') is maintained at the absolute address (where mod (a, b) represents the remainder when "a" is divided by "b"). Another way of maintaining the increased data is that the controller 14 transfers all the data to the buffer memory and then writes it back to the new access area with a new address pointer, thereby accessing the access area containing the most recent data held therein. Reallocate.

In the above-described embodiment, a high resolution frame is generated through the assumption of interscan pixels at a position corresponding to a motion vector. The high resolution frame is then used to generate a low resolution or high resolution image with reduced (or eliminated) aliasing distortion. Another method of performing such image reproduction is as follows: Assume that the original image without aliasing distortion is represented by P and the distorted (undersampled) image based on it is represented by P '. The relationship between the two is represented as follows:

[Equation 1]

P '= f (P)

Here, f () represents a function of subsampling an image in parentheses.

Similarly, the inverse of f () is represented by g ():

[Equation 2]

P = g (P ')

Therefore, if it becomes possible to reduce the aliasing distortion, since it is theoretically possible to reproduce the original image P from the distorted image P ', it is possible to reduce the aliasing distortion without having to detect the motion of the distorted image. Become.

The function g () is a learning process in which the original image is considered to contain a particular teaser-data and the continuous frame of the distorted image P 'is considered to contain a learning-data. Can be obtained through In this case, the gap between the scan lines of the distorted image may be filled based on the function g (), whereby the original image P may be reproduced.

Except for controlling aliasing distortion, the present invention relates to making an enlarged image with enhanced resolution or converting a low or standard resolution image (hereinafter referred to as SD image) into a high resolution image (hereinafter referred to as HD image). Including other applications.

Referring now to FIG. 11, a block diagram of an embodiment of a television receiver 100 for converting a television signal including an SD image into an HD image for display is shown. Receiver 100 differs from receiver 10 in FIG. 1 by using resolution converter 34 instead of distortion corrector 4 and using high resolution CRT 36 instead of CRT 7. The other component is a receiver 10 with a Y / C converter (not shown) that is typically used between the resolution converter 34 and the D / A converter 6 or between the LPF 2 and the A / D converter 3. )

12 is a block diagram of an embodiment of a resolution converter 34. The resolution converter 34 operates to segment regions of low resolution frames according to different motions in each region, generates high resolution images for each region, and combines high resolution images for each region to produce a composite image. It is similar in many respects to the distortion corrector 4b of FIG. The resolution converter 34 is configured to enhance the resolution in both the horizontal and vertical directions by generating pixels at positions between adjacent horizontal pixels and between adjacent vertical pixels of the SD image based on the motion vector for each frame. The transducer 34 may generate some high resolution pixels by the classification and adaptive processing techniques discussed in detail later.

The resolution converter 34 allows the divider 21 'as well as the frame memory 11, the motion detector 12, and the scene change detector 13 as described above to move each frame into M frame areas other than just two areas. Except for dividing, it includes an area divider 21 'such as the area divider 21 of FIG. Each of the M high resolution object generators is operated to generate a high resolution object for each frame region defined by the area divider 21 '. Each object generator 41i (i = 1, 2, ..., M) includes a switch 22i, a buffer memory 23i, a controller 14i, a resolution generating memory 15i, a write flag memory 42i, And a pixel generator 43i.

Each write flag memory 42i stores the same number of write flags as the number of pixels stored in the associated resolution generating memory 15i. The write flag is a bit flag indicating whether or not the pixel data of the SD video is stored at the address of the corresponding memory 15i (for example, the flag is 1 if the pixel data is stored at the address, and 0 otherwise). . The write flag is controlled to be set / reset by the controller 14i.

Referring to FIG. 13, each resolution generating memory 15i stores pixel data of P _H 'pixels in the horizontal direction and P _V ' pixels in the vertical direction when P _H '≥2P _H and P _V ' ≥2P _V. It can be designed for storage in, where P _H and P _V are each a number of pixels in the horizontal and vertical directions of SD frame stored in the frame memory 11. In this example, the memory 15i may store pixel data of a pixel twice as large as the SD image in the vertical direction and a pixel twice as large in the horizontal direction. The memory access area 15ai is composed of a storage location for storing pixel data of 2P _H x 2P _V pixels.

The procedure of writing the frame of the SD data into the resolution generating memory 15i is now described with reference to the flowchart of FIG. In step (S11), if the scene change detector 13 detects the reception of change in the scene or the first frame of data, a scene change signal, each controller (14 ₁₎ is provided in (14 _M). Then, each controller 14i clears the associated memory 15i in step S12 (deleting previous frame pixel data), and resets all write flags in the write flag memory 41i to initial values (step S13). . The distorted SD image data of the associated frame area in the current frame memory 11A is recorded in the memory 15i by closing the switch 22i at an appropriate time in the frame data reading operation from the frame memory 11A (step S14). . Thus, only image data in the associated area as defined by the area divider 21 'is transferred through the switch 22i and the memory 23i. In particular, the SD data is written into the access area of the memory 15i at pixel positions alternated in the horizontal and vertical directions. So, for example, the SD frame is arranged in M pixel rows (rows 1 to M) to N pixel columns (columns 1 to N) and the access area of the memory 15i is 2M rows (rows 1 to 2M). Assume a storage array corresponding to 2N columns (columns 1 to 2N). At this time, when data of pixels 1 and 1 corresponding to one row and one column of the SD frame is transferred to the object generator 41i, it is first stored in the storage location of the access area corresponding to the HD pixels of two rows and two columns of the access regions. do. Preferably, the storage location is also two rows, two columns in the storage array to facilitate pixel generation. Similarly, SD data of pixels 2 and 2 of the SD frames are first stored in four rows and four columns of the access area. The write flag is set at the same time for each of the HD storage locations where the SD data is thus stored. The image data is read out from the memory 15i by the pixel generator 43i at this time, but it is not deleted from the memory 15i (when there is no screen change) because it is used to form part of the next high resolution frame.

If no scene change signal is received in step S11, then in step S16 each controller 14i receives a motion vector from the area divider 21 'corresponding to the motion in the associated image area of the current frame. The controller 14i calculates a new relative address pointer R _P corresponding to the motion vector and supplies it to the resolution conversion memory 15i. Subsequently, the routine proceeds to step S14 in which the distorted image data of the current frame is immediately written to the memory 15i according to the determined relative address pointer, whereby the data stored prior to the preceding frame of the memory 15i is replaced with the new one. Shifted with respect to data (since the data of the previous frame is stored and kept in the same position according to the previous address pointer while the data of the current frame is stored according to the new address pointer). As described above, some of the previously stored data are typically overwritten at this stage. The write flag is set for the storage location where the new data is stored in step S15 (if it has not already been set previously).

Therefore, when the recording process proceeds from frame to frame in each resolution generating memory 15i, an HD image having a resolution twice that of the corresponding SD image in the horizontal and vertical directions is gradually completed.

To further illustrate the generation of the HD image, reference is made to FIG. 15 which illustrates the interframe movement of the small solid triangular object 47 and the associated storage of pixel data representing it. The left side of the figure shows an array of objects 47 and nested SD pixels P _SD , and the right side shows access areas 15a _N to 15a _{(N +} for frames N to (N + 3), respectively). _{3) shows} the storage location MC in the). In a low resolution image, the SD pixel P _{SD in} the boundary region of the object 47 represents the object by, for example, having some brightness and / or having a different color than the neighboring pixel. In any case, object 47 goes to pixels (a), (b), (c), and (d) in frame (N), and pixels (f), (g) in frame (N + 1). Is assumed to be represented by pixels (h), (i), (j), (k) in frame (N + 2) and pixels (l), (m) in frame (N + 3), where Pixels change from frame to frame due to interframe motion.

According to the above-described method, SD pixel data is recorded in the access areas 15a _N to 15a _{(N + 3) in the} area hatched to the alternate storage positions. In frame N, storage locations a 'through e' are filled with pixel data a through e corresponding to SD pixels. Between frames N and (N + 1), the object is placed at an SD pixel interval (adjacent) such that the relative address pointer R _P of frame N + 1 is moved down by one HD pixel storage location as shown. Up half of the interval between the midpoints of the SD pixels), and the pixel data for pixels f and g are recorded. On the other hand, the absolute storage position for storing pixel data a '-e' remains the same in frame N + 1. Thus, pixels b and g are at the same position in the SD image, while corresponding " pixels " (b ') and (g') are in the access area 15a _{(N + 1)} representing the HD image. It can be seen that they are shifted with respect to each other. The process continues with the address pointer R _P being moved from its previous position in the opposite direction to the object motion until a complete HD image is formed in frame N + 3. Note that if the object remains stationary or the motion is slightly from frame to frame, more frames are taken to generate a full HD image based solely on generating HD pixel data according to the motion vector.

16 is a flowchart for explaining a routine for reading image data from the high resolution object generator 41i. The reading routine follows the routine of Fig. 14 every time new image data is written to the memory 15i, i.e., every time recording of data for a particular frame is completed. In step S21, the controller 14i reads out image data from the storage position of the access area 15i and supplies the data to the pixel generator 23. In step S22, the pixel generator 23 determines whether image data has been stored for a particular high resolution pixel corresponding to a particular address based on the write flag in the write flag memory 42i. For example, assuming that the frame N corresponds to the first frame after the scene change in consideration of FIG. 15, the storage positions c 'and (d') include stored image data, but (c ') and The storage location 48 between (d ') is emptied. As such, the write flag for (c ') and (d') is set to 1, while for the storage location 48 is zero. Thus, before displaying image data for a current frame such as frame N, pixel data needs to be generated for high definition (HD) pixels corresponding to empty storage locations, such as storage location 48.

Therefore, in step S23, if the storage location under consideration is empty as indicated by the associated recording flag, image data is generated by the pixel generator 43i. Image data can be generated in a variety of ways, such as by interpolating between adjacent pixels or by more complex prediction techniques, such as the classification and adaptive processing methods described later. In any case, once image data is generated for a pixel under consideration (or read directly from memory 15i if already stored), the pixel data for that pixel is output from object generator 41i in step S24. And supplied to the combiner 24. If it is determined that all the image data of the high resolution frame under consideration has been read in step S25, the reading process for that frame is completed. Otherwise, the routine returns to step S21 of reading data from the next storage location and generating pixel data once again if necessary.

FIG. 17 illustrates a simplified flowchart of a processing step 23 of generating image data for a pixel corresponding to an empty storage location by interpolation or prediction techniques. In step S31, it is determined based on the write flag whether the pixel adjacent to the pixel under consideration includes pixel data stored in the memory 15i. Subsequently, in step S32, data is read from the storage location in the access area 15ai corresponding to these neighboring pixels. Pixel data for the pixel under consideration is generated in step S33 based on the neighboring pixel data.

Generation of pixel data by adaptive processing

If the high resolution frame in the resolution generating memory 15i has an empty storage location as described above, a simple interpolation technique that generates pixel data for the empty storage location is insufficient to recapture the high frequency components of the original image. The present applicant has previously proposed an image converter apparatus for converting an SD image into an HD image including high frequency components not included in the SD image. U.S. Patent No. "Digital Data Conversion Equipment and Method Thereof" granted on May 14, 1996. See 5,517,588. Thus, the same or similar adaptive processing may be used by the pixel generator 43i to generate pixel data for empty storage locations. That is, the adaptive process can be used to generate "fill-in" pixels in addition to those generated through assuming pixels at positions corresponding to the detected image motion. An adaptive process for performing such pixel generation is now described.

The adaptive process determines the predictive value of the HD image pixel based on the linear connection between the SD image and the predetermined estimation coefficient. For example, the predicted value E [y] for the pixel value y of the HD pixels forming the HD image may be determined by using a linear combination model. This model is a linear representation of the pixel values (hereinafter referred to as training data) of the SD pixel (x ₁ ), (x ₂ ), ..., and predetermined prediction coefficients (w ₁ ), (w ₂ ), ... Defined as a combination. In this case, the predicted value E [y] is expressed as follows:

[Equation 3]

E [y] = W ₁ X ₁ + W ₂ X ₂ + ...

To generalize the above, a matrix Y 'consisting of a set of prediction values E [y] for y = y ₁ to y _n is assumed. That is, the matrix Y 'is defined as the product of the matrix W, which is a set of prediction coefficients w, and the matrix X, which is a set of training data. Thus, the observation (Equation 4) is obtained as follows:

[Equation 4]

       XW = Y '

here,

One method of finding the predicted value E [y] close to the HD pixel value y includes applying the least squares method to the equation (4) above. In this case, matrix Y 'is considered to be the sum of matrix Y and matrix E, where matrix Y consists of a set of actual HD pixel values (called leader data) and matrix E ) Consists of the remaining set of "e" of predicted values E [y]. Thus, the following remainder is derived from Equation 4 above:

[Equation 5]

       XW = Y + E

here,

Square root error is defined as the sum of the squares of the remainder. In other words,

[Equation 6]

Thus, the prediction coefficient w _i for obtaining the prediction value E [y] close to the HD pixel value can be obtained by minimizing the square root error of Equation 6.

When the square root error is differentiated by the prediction coefficients w _i , if the result is 0, the value for w _i that satisfies the following equation 7 is the optimal value for obtaining the prediction value E [y] close to the HD pixel value:

[Equation 7]

When Equation 6 is differentiated by the prediction coefficient w _i , the following Equation 8 is obtained:

[Equation 8]

From Equations 7 and 8, the following Equation 9 is derived:

[Equation 9]

Given the relationship between the training data (x), the prediction coefficient (w), the leader data (y), and the rest (e), the set of prediction equations (10) can be obtained as follows:

[Equation 10]

The number of equations in the set of equation (10) corresponds to the number of prediction coefficients (w). The optimal prediction coefficient w can be obtained by solving Equation 10 that can be solved by the prior art, for example, by using a Gauss-Jordan deletion method. Note that the matrix composed of prediction coefficients w must be a static matrix so that Equation 10 can be solved.

Thus, with the above-described adaptive processing, it is possible to derive an optimal predicted value E [y] that is closer to the HD pixel value (i.e., closer to the HD pixel value that would be present when the HD signal is in principle received by the television receiver than the SD signal). The optimum prediction coefficient w is obtained for the purpose. Adaptive processing differs from interpolation in that high-frequency components of the original image contained in the HD image, although not in the SD image, can be restored. As long as only the above equation 1 is concerned, the adaptive process is similar to the interpolation process using an interpolation filter. However, in the adaptive processing, the prediction coefficients corresponding to the tap coefficients of the interpolation filter can be obtained by learning using leader data. As a result, the high frequency components included in the HD image may be resaved to easily obtain a high resolution image.

Referring now to FIG. 18, a block diagram of an image converter apparatus 200 for converting an SD image into an HD image is shown. The device 200 is used as part of the pixel generator 43i to generate HD pixel data when needed, i.e. whenever the HD pixel storage location in the resolution generation memory is empty. The input SD image signal is applied to both the adaptive processor 204 and the classifying circuit 201, which classifies the class tap generator 202 and the class determining circuit 203. In the classification circuit 201, HD pixels (hereinafter referred to as displayed pixels) whose prediction values are found in the adaptive processing are classified into predetermined grades based on the characteristics of the set of SD pixels in a predetermined positional relationship close to the displayed pixels. The pixel value for the set of SD pixels associated with the displayed HD pixel is referred to as the grade tab of the displayed pixel.

A rating tap based on the input SD image is extracted in rating tap generator 202 and provided to rating circuit 203 for detecting a pattern of a set of SD pixels that form a rating tap for each displayed pixel. Since the pattern is based on the characteristics of each pixel in the set, it is a function of pixel values. For example, one pattern corresponds to the uniformity of the pixel, the second pattern corresponds to the pixel value increasing in the upper right direction, and the third pattern corresponds to the pixel value increasing in the lower left direction. The value assigned prior to the detected pattern is then supplied to the adaptation processor 204 to indicate the grade of the indicated pixel.

19 illustrates the positional relationship of rank taps for associated HD pixels. It is assumed that the HD image is composed of pixels shown and indicated by x, and the corresponding SD image is composed of pixels indicated by O. Thus, the illustrated SD image includes 1/4 the number of pixels of the HD image. The spacing between the midpoints of the HD columns and the HD rows is half that for the SD rows and columns. In Fig. 19, SD pixels at positions (i + 1) (SD column i + 1) from the left and (j + 1) (SD row (j + 1) positions from the top are denoted by X _ij . HD pixels at (i '+ 1) and HD row (j' + 1) positions are represented by Y _{i'j '} , so, for example, of SD pixel X _ij and HD pixel Y _{(2i) (2j)} . The positions coincide with each other (note that "grade tap" and "prediction tap" are actually image data values for a particular pixel. However, for simplicity, the "tap" is described as being the pixel itself. )

To explain how the class tap is defined, it is assumed that the displayed pixel is an HD pixel Y ₄₄ having the same position as the SD pixel X ₂₂ . The rank tab for this marked pixel is the closest nine SD pixels within a 3 x 3 SD pixel rectangle centered about the marked HD pixel Y ₄₄ . Thus, in FIG. 19, the grade tap is an SD pixel within a region defined by the boundary T _c , that is, SD pixels X ₁₁ , X ₂₁ , X ₃₁ , X ₁₂ , and X ₂₂ . , (X ₃₂ ), (X ₁₃ ), (X ₂₃ ), and (X ₃₃ ). The rank tap generator 202 extracts pixel values for these SD pixels as rank taps for the indicated pixels. If the displayed pixel is adjacent to an HD pixel that matches the SD pixel, then a rating tap such as for the matching HD pixel may be defined. So, for example, if the displayed pixels are (Y ₅₄ ), (Y ₅₅ ), (Y ₄₅ ), etc., then the rating taps for these HD pixels are the same as for the "matching" HD pixels Y ₄₄ . It is also possible to form different grade taps for mismatched pixels, such as (Y ₅₄ ), (Y ₅₅ ), or (Y ₄₅ ).

The classification circuit 203 detects a pattern of the grade taps of the displayed pixels (the grade taps are provided by the grade tap generator 202). In other words, a pattern such as the uniformity of nine SD pixels, etc. in a positional relationship close to the indicated pixel is detected to identify which class should be assigned to the indicated pixel. The pattern value corresponding to the detected pattern is output in the class of the displayed pixel, and is supplied from the adaptive processor 204 to the address terminal AD of the coefficient read only memory (ROM) 207.

Typically, 8 bits or the like are allocated to the pixels forming the image. Assuming 8 bits are allocated to the SD pixel, for example, if nine SD pixels are used for the rank tap as in the example of FIG. 19, the number of possible pixel values per rank tap is (2 ⁸ ) ⁹ As high as As a result, the function of processing the pattern detection processing operation at high speed is prohibited.

Therefore, before sorting is performed, it is desirable to reduce the number of bits allocated to the SD pixels of the class tap. For example, to perform such bit reduction, adaptive dynamic range coding (ADRC) may be performed. In a first step in ADRC processing, a pixel having a maximum pixel value (hereinafter referred to as a maximum pixel) of the nine SD pixels forming a processing block and a pixel having a minimum pixel value in a processing block (hereinafter referred to as a minimum pixel) ) Is detected. The difference DR between the pixel value MAX of the maximum pixel and the pixel value MIN of the minimum pixel is then calculated. The value DR is represented by the local dynamic reference value of the processing block, and each pixel value forming the processing block is quantized back to a number K bits smaller than the number of bits originally assigned for each pixel. In other words, the pixel value MIN of the minimum pixel is subtracted from each pixel value forming the processing block, and each subtraction result is divided by DR / 2K. As a result, each pixel value constituting the processing block can be represented by K bits. So, for example, when K = 1, the maximum number of nine SD pixel patterns is (2 ¹ ) ⁹ . Thus, the maximum number of patterns is significantly reduced compared to when no ADRC processing is performed.

Continuing with reference to FIG. 18, the adaptive process is executed in an adaptive processor 204 that includes a predictive tap generator 205, a predictive value calculator 206, and a predictive coefficient ROM 207. The prediction tap generator 205 extracts data of a plurality of SD pixels having a predetermined positional relationship with respect to the displayed pixel. This extracted pixel is provided to the prediction value calculator 206 as prediction taps x1, x2, ..., which determine the prediction value for the HD pixel based on the prediction coefficient and the prediction tap.

The predictive tap corresponds to a pixel having a high position correlation with the displayed HD pixel. For example, if the displayed pixel as shown in FIG. 19 is pixel Y ₄₄ and the grade tap within boundary T _c is formed as described above, prediction tap generator 205 is defined by boundary T _p . 5 x fall within the surrounding region 5 SD pixels, that is, determines the prediction tap to the block of the SD pixels (X ₀₀₎ to (X _44). If the displayed pixel is a pixel adjacent to Y ₄₄ , such as pixel Y ₅₄ , Y ₄₅ , or Y ₅₅ , then the same as for pixel Y ₄₄ , that is, pixels X ₀₀ to X ₄₄ , a prediction tap is formed. However, it is possible to define different prediction taps when the mismatched pixels such as (Y ₄₅ ), (Y ₅₄ ), or (Y ₅₅ ) are the displayed pixels.

Prediction coefficient ROM 207 stores the prediction coefficients found by the learning previously performed with the grade. When the ROM 207 receives a rating supplied from the classification circuit 203, the ROM 207 reads a prediction coefficient stored at an address corresponding to the supplied rating, and supplies the prediction coefficient to the prediction value calculator 206. .

Thus, both the prediction coefficients corresponding to the class of the displayed pixels and the prediction taps corresponding to the displayed pixels are supplied to the calculator 206. In the calculator 206, the operation according to Equation 3 is the prediction received from the SD pixel data x1, x2, ... and ROM 207 forming the predictive tap from the predictive tap generator 205. This is done using the coefficients w1, w2, ... As a result, the predicted value E [y] of the displayed pixel y is determined, which is output as the pixel value for the HD pixel. The process is repeated by representing each HD pixel with the indicated pixel, and when all HD pixels are so represented and predictions derived for them, the complete SD image is converted to an HD image.

Referring now to FIG. 20, shown is a block diagram of a learning apparatus 210 that executes a learning process to calculate prediction coefficients to be stored in ROM 207 of FIG. 18. The HD image data to be leader data in the learning process is supplied to both the thinning circuit 211 and the leader data sampling circuit 146. In the thinning circuit 211, the number of pixels of the HD image is reduced by thinning so that the HD image is converted into the SD image. The number of HD image pixels is halved in both the horizontal and vertical directions, thereby forming the SD image. The SD image is supplied to the classification circuit 212 and the prediction tap generator 145. Note that rather than forming the SD image from the HD image, the SD image is directly applied to the classification circuit 212 from the SD camera corresponding to the HD image from the HD camera.

In the classification circuit 212 or the prediction tap generator 145, the same processing as that performed in the classification circuit 201 or the prediction tap generator 205 in FIG. 18 is executed, whereby the class and the prediction tap of the displayed pixels are output. do. The grade output by the classification circuit 212 is applied to the address terminal AD of both the prediction tap memory 147 and the leader data memory 148. The predictive tap output from the predictive tap generator 145 is applied to the predictive tap memory 147 where the tap is stored at an address corresponding to a rating supplied from the classification circuit 212.

In the leader data sampling circuit 146, an HD pixel that becomes the pixel displayed by the classification circuit 212 and the prediction tap generator 145 is extracted from the HD image supplied thereto. The extracted taps are stored in leader data memory 148 as leader data with the grade calculated at a common address location. The process is repeated for all HD pixels of the HD image input to the device 210 for learning purposes. Since the prediction tap memory 147 and the leader data memory 148 are each configured to store a plurality of kinds of information at the same address location, the plurality of learning data x and leader data y classified into the same class are basically used. Can be stored at the same address location.

The calculator 149 reads HD pixel data which is prediction taps or leader data which are learning data stored at the same address position from the prediction tap memory 147 or the leader data memory 148, respectively. Based on this data, the calculator 149 calculates the prediction coefficients by, for example, a least square method that minimizes the error between the prediction value and the leader data. In other words, in the calculator 109, the above equation (10), which is the above-described prediction equation, is formed into a grade, and the prediction coefficient is obtained when solving these equations. The prediction coefficients are stored at address locations corresponding to the ranks in the coefficient ROM 207 of FIG. Matching each of the "as independent pixels _{(Y 45), (Y 54} ), or (Y ₅₅₎ in the equation (10) is the prediction tap are each also as" matching "pixels (Y ₄₄₎ 19 be the same Note that it is "unlocked" for HD pixels.

In the example of Fig. 19, nine grade taps and 25 prediction taps are formed for each displayed HD pixel, and classification and adaptation processing (hereinafter referred to as classification adaptive processing) is executed accordingly. However, for the indicated HD pixels near the edges of the image area, the assumptions of the 9 rank taps and 25 prediction taps are no longer valid. Thus, it is desirable to form different arrangements of rank and prediction taps for these HD pixels and to calculate the prediction coefficients in a similar manner as described above to include this special case.

Thus, the classification-adaptation process described above is used to compensate for pixel generation in resolution generation memory 15i when HD pixels are not otherwise generated according to the associated motion vector. For example, following scene changes, generating an HD video from an SD video based on motion only takes several motion frames. So, in the initial frame following the scene change, a classification adaptation procedure can sometimes be used.

Data recording according to the motion vector

As described above in connection with the embodiment of FIG. 4, each motion vector is rounded to a distance corresponding to the distance between midpoints of adjacent high resolution pixels in order to generate pixels at positions corresponding substantially to the image motion. So, for example, in an embodiment where HD pixel data is generated from SD pixel data, if the motion vector of a given frame corresponds to motion between 0.5 and 1.5 HD pixel spaces in the vertical (or horizontal) direction, then the motion vector is It is rounded up by one HD pixel interval in the vertical (or horizontal) direction and the address pointer is moved accordingly to write new SD data to memory. Alternatively, the address pointer may be moved only if the motion vector is approximately equal to or within a predetermined range of integer multiples of the HD pixel interval (i.e., integer multiple of one half of the SD pixel interval). In this case, there are many examples in which there is no change in the address pointer so that the empty address is given more frequently for the HD pixel data in the resolution generating memory 15i. Thus, the empty address can be filled by interpolation or through classification adaptive processing.

As another method of rounding the motion vector to produce high resolution pixel data accordingly, another approach is to use a higher resolution storage array (enlarged in the resolution generating memory 15 or 15 ⁱ ) to reduce the rounding error of the motion vector. Access area). When a higher resolution frame is generated, pixel data may be read by skipping some data to produce a high resolution image, but not alternating rows and alternating columns, for example, but not as high resolution as can be realized. .

To illustrate the later approach, the resolution converter 34, designed to convert SD images to HD images, has a pixel storage location of four times the SD image in both the horizontal and vertical directions, i.e., 16 times the pixel intensity compared to the SD image. It can be corrected by using a resolution conversion memory 15 _i ′ having. This approach is illustrated in FIG. 21 where the memory 15 _i 'is designed with P _H '> = 4P _H columns and P _V '> = 4P _V rows, where P _H and P _V are the horizontal and vertical directions of the SD frame, respectively. The number of pixels in. The access area 15a _i is designed with 4P _H x 4P _V pixels. The relative address pointer R _P basically controls the exact position of the access area 15a _i in the larger memory area of the memory 15 _{i in the} manner described above.

The SD pixel is written every fourth position in the access area 15a _i 'in the horizontal and vertical directions after moving the relative address pointer according to the motion vector. The motion vector is rounded up to a unit corresponding to one quarter of the SD pixel interval before determining the relative address pointer. However, even in this case, if the x and y components of the motion vector of the SD image deviate significantly from an integer multiple of 1/4 of the SD pixel interval, no additional SD pixels are written to the memory. For example, if the motion vector is determined to be less than 1/8 of the SD pixel interval, a new high resolution frame is formed equal to the previous high resolution frame and no new SD pixel data is recorded.

22-24 further illustrate the extended access area approach. 22 illustrates an SD pixel array portion of the pixel P _SD . The object OB _N-4 is assumed to be formed of pixels in the frame N-4. The position of the object is moved from frame to frame until it reaches the position of object OB _N in frame N. The pixels forming the image are recorded in the access area extended to the area of 4P _H x 4P _V as described above. The storage state is shown in the access area 15a _i ′ of FIG. 23, where pixels of different frames are shown with different hatching processes. Motion vectors corresponding to frame to frame motion are also created in FIG. 23. 24 is an enlarged view of a memory storage location or cell MC of FIG. 23. After the motion of four frames from frame N-4 to frame N, many storage locations remain empty. They can be filled by interpolation and also by classification adaptive processing or the like. Alternatively, the bin position can be filled by simply inserting a pixel value, such as the closest storage location, into the bin position.

Dotted lines shown in Figs. 23 and 24 represent alternating rows and columns of pixel data. To read the HD pixel data from the access area 15a _i ′, it is sufficient to read the data in every other row and every other column, ie from the storage location along the dotted line. This method provides higher resolution than is possible by using an access area with only a 2P _H x 2P _V array of storage locations.

Referring to FIG. 27, another embodiment 34 ′ of the resolution converter 34 is shown. This embodiment is similar to FIG. 12 except that the complete image of each frame is designed for the case where the same amount of motion is moved from frame to frame together. That is, the resolution converter is functionally similar to the distortion corrector 4a of FIG. Thus, a single high resolution image generator 41 without switch 22 or buffer memory 23 is used in place of the high resolution object generators 411-41M, and the area divider 21 and combiner 24 are removed. All other features are as described above with respect to the embodiment of FIG. 12.

In each of the above-described embodiments of the present invention using the resolution generating memory 15 or 15 _i , when the previous high resolution frame data has already been generated in the resolution generating memory, from the next frame, that is, from the current frame memory 11A. The data storage of is described by moving the address pointer according to the motion vector and overwriting some of the previous data. As another method for that approach, the motion vector of the previous frame is stored and compared with the motion vector of the current frame to determine which motion vector has the x and y components closest to the distance corresponding to one high resolution pixel interval, The data is stored accordingly. In other words, data associated with a motion vector with a smaller rounding error is selected to be stored. In the memory of FIG. 23, this means selecting data associated with the motion vector closest to the nearest multiple of one quarter of the SD pixel interval.

While the invention has been shown and described in connection with particularly preferred embodiments, those skilled in the art will readily appreciate that various changes and modifications may be made to the described embodiments without departing from the spirit and scope of the invention. For example, although the described embodiment is described to reduce aliasing distortion and selectively generate high resolution images, the present invention can also be used to expand the images. Also, while the above process is described as processing data based on frame to frame, the process can be used to process a portion of a frame, such as a field unit. The image can also be displayed on a display other than a CRT, such as a liquid crystal display. In addition, the received video signal may be a digital video signal. In this case, the A / D converter (for example in FIG. 1 or FIG. 11) can be removed. Moreover, the present invention can be used to convert interlaced scanned images into sequential scanned images. Accordingly, such changes and modifications and other variations and modifications are intended to be included within the scope of the invention as defined by the appended claims.

1 is a block diagram of a television receiver according to the present invention;

2A-2B and 3 illustrate pixel data generation in accordance with the present invention.

4 is a block diagram of a distortion corrector shown in FIG.

5 is a graph illustrating a method of determining a motion vector for an image.

6 to 8 illustrate data storage in a resolution generating memory.

9 is a flowchart illustrating a routine for generating high resolution frames.

10 is a block diagram of a distortion corrector.

11 is a block diagram of another embodiment of a receiver in accordance with the present invention.

12 is a block diagram of a resolution converter in the receiver shown in FIG.

FIG. 13 is a diagram showing a data storage location of a resolution generating memory of the converter shown in FIG. 12; FIG.

Fig. 14 is a flowchart showing a routine for recording image data in the memory of the resolution converter.

FIG. 15 is a diagram illustrating image data storage in a resolution converter according to a motion vector of an image. FIG.

Fig. 16 is a flowchart for explaining a routine of reading image data from a resolution converting memory in a resolution converter.

Fig. 17 is a flowchart showing a processing procedure for generating high definition pixel data.

Fig. 18 is a block diagram of a circuit for generating high definition image data from standard definition image data using prediction coefficients.

FIG. 19 illustrates rank taps and prediction taps. FIG.

20 is a block diagram of a learning device for determining prediction coefficients.

FIG. 21 illustrates data storage in a resolution generating memory having an extended access area. FIG.

22 is a diagram illustrating movement of an image in accordance with a frame.

23 and 24 are diagrams for explaining pixel data storage of the image shown in FIG. 22;

25 illustrates horizontal scanning lines of a CRT display.

FIG. 26 is a diagram illustrating undersampling of an image signal in a vertical direction. FIG.

27 is a block diagram of another resolution converter in accordance with the present invention.

* Description of the symbols for the main parts of the drawings *

1: tuner 2: low pass filter (LPF)

3: analog to digital converter 4: distortion corrector

5: Y / C separator 6: digital to analog converter

7: cathode ray tube (CRT) 11: frame memory

101, 102, 103: pixel data 12: detector

13: scene change detector 14: controller

Claims (56)

An image signal conversion apparatus for converting a first image signal into a second image signal, wherein the first and second image signals include images of a plurality of different frames. A motion detector operable to detect motion of said first image signal between a first frame and a second frame; And a processing circuit for generating the second image signal based on the assumption of the pixel at the position corresponding to the detected motion. The method of claim 1, And the second video signal has a higher resolution than the first video signal. The method of claim 1, And the second image signal has pixel data of a greater number of pixels than the first image signal. The method of claim 3, wherein And the processing circuit generates the second image signal having pixel data of more pixels than the first image signal in a direction perpendicular to the scanning lines associated with the first image signal. The method of claim 1, And the first and second frames are adjacent frames. The method of claim 1, And each of the pictures of the plurality of different frames comprises a picture of a complete frame. The method of claim 1, And each of the images of the plurality of different frames comprises a portion of each frame. The method of claim 1, And a low pass filter for filtering high frequency components of the second image signal. The method of claim 1, The motion detector detects motion of the first image signal by an amount finer than the pixel size of the first image signal, and the processing circuitry A resolution creating memory for storing an image of the first image signal and having a storage capacity greater than the amount of data in one image of the first image signal; A controller operable to control the writing of the first image signal data to the resolution producing memory and to control the data reading of a new image signal from the memory having a higher quality than the first image signal, the controller being the first And the controller for recording the first image signal data into the memory in accordance with the detected motion of the image signal. The method of claim 9, And the controller controls reading of the new video signal as the second video signal. The method of claim 9, A low pass filter for filtering the new video signal; Converting the filtered new video signal into the second video signal formed of the same number of pixels as the number of pixels in one image of the first video signal, and having less aliasing distortion than the first video signal. And an output unit for outputting the second image signal. The method of claim 9, A scene change detector for detecting a scene change of the first image signal and generating a scene change signal accordingly, wherein the controller changes the scene change to reset values of storage locations of the memory to predetermined values; Video signal conversion device responsive to the signal. The method of claim 9, The data of the first frame of the first image signal is caused to be stored in a first storage location of the resolution generating memory, The data of the second frame of the first image signal may be respectively shifted with respect to the first storage positions by an amount corresponding to the motion of the first image signal between the first frame and the second frame. An image signal conversion apparatus caused to be stored in second storage locations, thereby generating high quality frame data including both the first frame data and the second frame data. The method of claim 13, The first frame data is stored in the resolution generating memory in a first access area, The second frame data is caused to be stored in a second access area having the same size as the first access area without all of the first frame data being deleted, and the second access area corresponds to the detected motion. Shifted for 1 access area, And the controller is operative to provide a relative address pointer to the memory that determines locations of the first and second access regions. The method of claim 9, The processing circuit includes a detector for detecting empty positions in the memory in which data of the first image signal is not stored; And a pixel generator for generating pixel data at the empty positions. The method of claim 15, And the detector comprises a flag memory for storing flags indicating which storage locations of the resolution generating memory contain stored image data of the first image signal. The method of claim 15, And the pixel generator is operative to determine image values for pixels at locations corresponding to the bin positions by interpolation between pixel data already stored at neighboring storage positions of the bin positions. The method of claim 15, The pixel generator is configured to detect features of the first image signal and determine a class corresponding to the detected features; Prediction data memory for storing prediction data for each class; And circuitry for generating the pixel data for an empty storage location in accordance with the predictive data read from the predictive data memory corresponding to the determined class. The method of claim 18, And the prediction data for each class is generated by learning using a learning image signal having a higher resolution than at least the first signal. An image signal conversion apparatus for converting a first image signal into a second image signal, wherein the first and second image signals each include images of a plurality of different frames. A motion detector operable to detect motion of said first image signal between at least a first frame and a second frame in at least first and second regions of each frame; An area divider for defining at least first and second image regions of the image of the first image signal based at least on the detected motion; Processing circuitry for generating first and second image regions of the second image signal based on assumptions of pixels at positions corresponding to the detected motion in the respective first and second image regions; And a combiner for combining the first and second image regions of the second image signal to form a composite image signal. The method of claim 20, And the second video signal has a higher resolution than the first video signal. The method of claim 20, And the second image signal has pixel data of a greater number of pixels than the first image signal. The method of claim 20, And the second image signal is generated as pixel data of more pixels than the first image signal in a direction perpendicular to the scan lines associated with the first image signal. The method of claim 20, And a filter for filtering the second image signal. The method of claim 20, The motion detector detects motion of the first image signal by an amount finer than the pixel size of the first image signal, and the processing circuitry performs individual processing operations on the first and second image regions of the first signal. First and second circuit portions for execution, Each processing section stores data in each image region of the first image signal and has a storage capacity having a storage capacity larger than the amount of data in one image of the first image signal; A controller operable to control recording of first image signal data into the resolution generating memory and reading data of a new image signal from the memory having a higher quality than the first image signal, the controller being configured in an associated image region. And the controller for recording the first image signal data into the memory in accordance with the detected motion of the first image signal of the controller. The method of claim 25, A scene change detector for detecting a scene change of the first image signal and accordingly generating a scene change signal, wherein each controller is configured to reset values in storage locations of an associated resolution generating memory to a predetermined value; A video signal conversion device responsive to a scene change signal. The method of claim 25, A low pass filter for filtering the second video signal; An output unit for converting the filtered second signal into an output video signal formed of the same number of pixels as the number of pixels in one image of the first video signal, and outputting the second video signal, the output And the image signal further comprises the output unit having less aliasing distortion than the first image signal. The method of claim 27, The output unit includes a frame memory for storing the filtered second signal, and data is skipped from the frame memory by periodically skipping lines of data to produce the output signal having a lower quality than the second image signal. Video signal conversion device to be read. The method of claim 25, Wherein each circuit portion of the processing circuit is A detector for detecting empty positions in an associated resolution generating memory in which data of the first image signal is not stored; And a pixel generator for generating pixel data at the empty positions. The method of claim 29, And the detector comprises a flag memory for storing flags indicating which storage locations of the resolution generating memory contain stored image data of the first image signal. The method of claim 29, And the pixel generator is operative to determine image values for pixels at locations corresponding to the bin positions by interpolation between pixel data already stored at neighboring storage positions of the bin positions. The method of claim 29, The pixel generator comprises a determiner for detecting features of the first image signal and for determining a rank corresponding to the detected features; Prediction data memory for storing prediction data for each class; And circuitry for generating the pixel data for an empty storage location in accordance with the predictive data read from the predictive data memory corresponding to the determined class. The method of claim 32, And the prediction data for each class is generated by learning using a learning video signal having a higher resolution than at least the first signal. The method of claim 1, The first image signal includes a standard definition (SD) signal, and the second image signal includes a high definition (HD) signal having a higher resolution than the SD signal in both a horizontal direction and a vertical direction. Video signal conversion device. The method of claim 9, The first image signal represents a standard definition (SD) image, the second image signal represents a high definition (HD) image having a higher resolution than the SD image in a horizontal direction and a vertical direction, An access area is defined in the resolution generating memory for each generated high resolution image, with storage locations corresponding to double pixels in the vertical direction and double pixels in the horizontal direction with respect to the SD image. Video signal conversion device. The method of claim 9, The first image signal represents a standard definition (SD) image, the second image signal represents a high definition (HD) image having a higher resolution than the SD image in a horizontal direction and a vertical direction, Defined in the resolution generating memory for each high resolution image generated, with access positions whose pixels are four times the SD image in the vertical direction and storage positions corresponding to the pixels four times the SD image in the horizontal direction. Become, And skipping alternate storage locations of pixel data causing the image data to be read from the resolution generating memory, thereby forming an HD image having twice the pixels of the SD image in each of the horizontal and vertical directions. A method for converting a first video signal into a second video signal, wherein each of the first and second video signals comprises a plurality of different frames of images. Detecting motion of the first video signal between a first frame and a second frame; Generating pixel data by assuming a pixel at a position corresponding to the detected motion; Generating the second image signal based on the generated pixel data. The method of claim 37, And the second video signal has a higher resolution than the first video signal. The method of claim 37, And the second image signal is generated with pixel data of a greater number of pixels than the first image signal. The method of claim 39, And the second image signal is generated as pixel data of more pixels than the first image signal in a direction perpendicular to the scan lines associated with the first image signal. The method of claim 37, And the first and second frames are adjacent frames. The method of claim 37, And low pass filtering the second image signal to reduce aliasing distortion that may be present therein. The method of claim 37, Detecting the motion comprises detecting the motion of the first image signal by an amount smaller than the pixel size of the first image signal, Storing the image of the first image signal in a resolution generating memory having a storage capacity larger than the amount of data in one image of the first image signal; Controlling writing first video signal data to the resolution generating memory according to the detected motion of the first video signal; Controlling reading of data of a new video signal from the memory, wherein the new video signal is of higher quality than the first video signal. The method of claim 43, Controlling said reading comprises controlling reading data of said new video signal as said second video signal. The method of claim 43, Low pass filtering the new video signal to produce a filtered signal; Converting the filtered image signal into the second image signal formed of the same number of pixels as the number of pixels in one image of the first image signal; Outputting the second video signal with less aliasing distortion than the first video signal. The method of claim 43, Detecting a scene change of the first video signal; If a scene change is detected, resetting data in storage locations of the resolution generating memory to predetermined values. The method of claim 43, Generating a relative address pointer according to the detected motion, and not deleting all the image data in the memory of the first frame, but images of the second frame in the memory at positions according to the relative address pointer Recording data, thereby generating data of a higher quality frame including both the first frame data and the second frame data, wherein the second frame data stores the first frame data. Stored in shifted locations relative to storage locations. The method of claim 43, Detecting empty positions in the memory in which data of the first image signal is not stored; Generating pixel data for pixels corresponding to the empty positions. 49. The method of claim 48 wherein And the detecting step includes storing flags indicating which storage locations of the resolution generating memory contain stored image data of the first image signal. 49. The method of claim 48 wherein Generating the pixel data comprises interpolating between image values of neighboring pixels in which the pixel data has already been stored. 49. The method of claim 48 wherein Generating the pixel data, Detecting features of the first image signal and determining a class corresponding to the detected features; Storing predictive data in predictive data memory for each class; Generating the pixel data for an empty storage location in accordance with the predictive data read from the predictive data memory corresponding to the determined rank. 49. The method of claim 48 wherein Generating the predictive data for each class by learning using a learning video signal having at least a higher resolution than the first signal. The method of claim 37, Wherein the first image signal comprises a standard clear (SD) signal and the second image signal comprises a high definition (HD) signal having a higher resolution than the SD signal in both horizontal and vertical directions. The method of claim 43, The first image signal represents a standard definition (SD) image, the second image signal represents a high definition (HD) image having a higher resolution than the SD image in horizontal and vertical directions, Storing SD image data in an access area of the resolution generating memory for each of the generated high resolution images, each access area being twice the pixels of the SD image in the vertical direction and the SD image in the horizontal direction; And storing the storage locations corresponding to twice the pixels. The method of claim 43, The first image signal represents a standard definition (SD) image, the second image signal represents a high definition (HD) image having a higher resolution than the SD image in both horizontal and vertical directions, Storing SD image data in an access region of the resolution generating memory for each generated high resolution image, wherein each access region is four times the pixels of the SD image in the vertical direction and four of the SD image in the horizontal direction; Said storage step having storage positions corresponding to double pixels; Reading pixel data from the resolution generating memory by skipping alternate storage locations, thereby forming an HD image having twice the pixels of the SD image in each of the horizontal and vertical directions. How to include more. The method of claim 37, Detecting the motion comprises detecting motion of the first image signal by an amount smaller than a pixel size of the first image signal in a plurality of image regions of the image, Defining at least first and second image regions of the image of the first image signal based at least on the detected motion; Generating first and second image regions of the second image signal based on assumptions of pixels at positions corresponding to the detected motion in the respective first and second image regions; Combining the first and second image regions of the second image signal to form a composite image signal.