JP2009188470A - Apparatus and method for attaining high resolution - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To solve a matter that the effect of high resolution by super-resolution processing may not be attained because a motion vector cannot be extracted in a moving video including no motion between continuous images, e.g., a moving video containing a quiescent state or a moving video obtained from movie by pull-down translation. <P>SOLUTION: A technology presented ensures high resolution of the video signal of a moving video where images having no motion may continue. More specifically, it determines whether there is a motion in adjoining images (fields) of the moving video or not, and performs super-resolution processing for an image involving a motion. Consequently, high resolution can be attained even for the video signal of the moving video where images having no motion may continue. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The technical field relates to higher resolution video.

  Regarding the above technical field, Patent Document 1 discloses that “high-resolution data of an original signal having a Nyquist frequency or higher is restored from a plurality of sets of digital data having different sampling positions to obtain high-resolution data with less blur”. , “Three sets of digital data obtained by sampling the same object at different positions are input 1. The first data is input to the reference data buffer 2, and the position difference estimation unit 3 A position difference (Δ1, 2) with respect to the data is obtained, and the weight calculation unit 4 obtains a weight according to the position difference, and the wideband LPF and the resolution increasing unit 6 perform the LPF process on the input data, Sampling is performed at an interval narrower than the sampling interval, and the result is stored in the intermediate buffer 7. The product-sum unit 8 adds the weighted data to the intermediate buffer 7 and adds them together. "The technical idea is disclosed. That.

  In addition, Patent Document 2 states that “when interlaced video data having a field frequency of 60 Hz obtained by converting original progressive video data having a frame frequency of 24 Hz is converted to output progressive video data having a frame frequency of 60 Hz. For the purpose of making the motion of the video look smooth. "The original progressive video data with a frame frequency of 24 Hz is converted into interlace video data with a field frequency of 60 Hz by 2-3 pulldown, and the interlace video data is converted into It converts to progressive video data with a frame frequency of 60 Hz, and further interpolates this progressive video data to obtain output progressive video data, directly from the interlaced video data after 2-3 pulldown. Can be converted into progressive video data interpolation process. "Technical idea is disclosed.

JP-A-8-336046 JP 2004-15700 A

  Patent Document 1 discloses a technique for synthesizing one high-resolution image from a plurality of low-resolution images. As an application of this technique, a technique for obtaining a higher resolution image by using a plurality of field images (field images) in moving image data has been studied. In particular, it is effective to reduce the number of original field images required for increasing the resolution in order to reduce the resolution increasing process. Therefore, position estimation means for estimating the sampling phase difference using the field image on the reference frame and the corresponding field image on the other frame, and the image of each input image frame using the information of the sampling phase difference Motion compensation and up-rate means to compensate for data and increase the number of pixels by n times to increase the density, phase shift means to phase-shift the image data of each densified image frame by a fixed amount, and sampling position Coefficient determination means for determining a coefficient using phase difference information, and folding that outputs by removing the (n-1) aliasing components by multiplying each image data before and after the phase shift by multiplying the coefficient. A technique for generating one high-resolution image having n times the resolution from n low-resolution still images has been studied by providing component removal means. If this process is used, two low-resolution still images are sufficient to generate one high-resolution image having double the resolution. A process for obtaining a higher-resolution image using a plurality of field images (field images) or frame images (frame images) in the moving image data is particularly referred to as super-resolution processing.

  However, a high-resolution image cannot be synthesized if there is no movement between a plurality of original low-resolution images. For this reason, for example, there is a problem in increasing the resolution of a pull-down image obtained by converting a movie image recorded on a film into a television signal.

  The movie records 24 frames per second on the film. On the other hand, in the television, for example, in the NTSC system adopted in Japan, the signal includes 30 frames and 60 fields per second. This is a method called an interlace method. When scanning one screen, the odd-numbered scanning lines and the even-numbered scanning lines are divided into two times, and one screen is completed by two scans. Data consisting only of odd-numbered scanning lines or only even-numbered scanning lines obtained in one scan is called one field, and one screen consisting of all scanning lines obtained by combining two fields is called one frame. Therefore, scanning is performed 60 times per second. Therefore, in order to convert a video recorded on a film as a movie into a television signal for broadcasting or recording, conversion between 24 Hz and 60 Hz is necessary. A method called 2-3 pull-down is used here. As shown in Fig. 3, 2-3 pull-down generates 2 fields, 3 fields, 2 fields, and 3 fields from each frame on the film, thereby producing an NTSC signal of 60 fields per second. Similarly, to generate 50 fields of PAL and SECAM television signals for 1 second using movie film as a source, a method called 2-2 pulldown is used to generate 2 fields of television signals from each frame on the film. Is used.

  Patent Document 2 is examining smoothing the motion of the moving image in the time axis direction by calculating between the frames for the signal subjected to the video conversion by 2-3 pulldown, but the motion of the moving image is smooth. The resolution (number of pixels) does not change.

  For this reason, for example, if super-resolution processing is applied to a signal after 2-3 pull-down as it is, fields derived from the same frame of the video source on the film are continuous in the signal after pull-down as shown in FIG. If you try to perform super-resolution processing using a simple sequence of multiple fields, there may be no time difference between at least two consecutive fields in the multiple fields. The effect of processing may not be obtained. This is not limited to 2-3 pull-down, but is a common problem when super-resolution processing is applied to a general pull-down method in which fields having no time difference are continuous.

  Therefore, for example, the present application provides a technique for increasing the resolution of a video signal of a moving image in which images that do not move with each other may continue. Specifically, it is determined whether or not there is a motion in an adjacent image of the moving image, and super-resolution processing is performed on the motioned image.

  According to the above means, for example, it is possible to increase the resolution of a moving image video signal in which images that do not move with each other may continue.

  An example (example) of an embodiment suitable for the present invention will be described. However, the implementation of the present invention is not limited to this embodiment.

  FIG. 1 shows an example of the configuration of the high resolution device of the first embodiment. The high resolution device is, for example, a television, a display, a set top box (set top box), a recorder, a player, or the like.

  The video is input from a tuner 105 that receives and demodulates a broadcast wave or a media reading unit 106 that reads data from media such as an HDD, a BD, or a memory. However, the present invention is not limited to this and may be input from the Internet, for example. Further, the tuner 105 and the media reading unit 106 may be separate from the apparatus.

  The inter-adjacent field motion detection unit 101 compares adjacent fields in the input video signal and detects whether there is motion. When there is motion between adjacent fields, the motion vector extraction unit 102 extracts the motion vector between the fields, and performs super-resolution processing on each field using the motion vector extracted by the super-resolution processing unit 103. The video signal with high resolution is output. In addition, although the high resolution by a field is demonstrated, the high resolution by a frame may be sufficient (it is the same also in another Example).

  When the inter-adjacent field motion detection unit determines that there is no motion between adjacent fields in the input signal, a past field having motion between the field from the field storage / selection unit 104 that temporarily stores the input signal is displayed. Select and extract a motion vector from the field. As a result, a motion vector can be extracted even when there is no motion between adjacent fields, and each field can be super-resolved and a high resolution field can be output.

  Here, the past field selected by the field storage / selection unit 104 is not limited to the previous field. For example, for an input video signal that stops when there is a moving image and continues for a while, the field immediately before stopping is stored in the field storage / selection unit 104, and each field that is still By continuing the motion vector extraction between the image and the field immediately before the image is stationary, it is possible to continue outputting the high resolution field even while the image is stationary.

  Next, the operation will be described with reference to FIG.

  The input video signal (S201) is stored in the field storage / selection unit (S202). Further, it is determined whether or not there is movement between the current field in the input signal and the previous field (S203). If it is determined that there is no motion, select a field with a temporal difference stored in the past (S204) and perform motion vector extraction (S205), and perform super-resolution processing using the extracted motion vector. The output is made after the execution (S206) (S207).

  If it is determined that there is motion between the current field and the previous field in the input video signal (S203), the motion vector is extracted as it is between consecutive fields (S205) and extracted. Super-resolution processing is performed using the motion vector (S206), and then output (S207).

  As described above, even when there is no motion between adjacent fields in the input signal, it is possible to obtain a high resolution output by performing super-resolution processing.

  The configuration and operation of the inter-adjacent field motion detection unit 101 will be described with reference to FIG. First, a difference signal between two consecutive fields is generated by calculating a difference between a signal that has passed through the one-field delay unit 301 and a signal that has not passed through the subtraction unit 302. However, in the case of the composite signal, since the color signal is inverted, the color signal is removed by the low-pass filter 303 in order to extract only the luminance signal. The difference signal between the fields generated in this way is compared with a threshold value by the comparison unit 304 to detect whether there is a significant difference between corresponding pixels between two consecutive fields. It is determined whether the signal obtained by integrating the output of the comparison unit 304 for one field by the one-field integration unit 305 exceeds a predetermined threshold value. If it is below the threshold, it can be determined that there is no motion between adjacent fields.

  The first embodiment is an example that can be generally applied to a video in which there is no motion between adjacent fields. In the second and subsequent embodiments, a particularly preferable embodiment will be described for a 2-3 pull-down video.

  FIG. 4 shows a configuration example of the high resolution device of the second embodiment.

  The 2-3 pull-down video detection unit 401 detects whether or not the input TV video signal is a video signal derived from a movie film converted into a TV signal by 2-3 pull-down. When the input signal is not a signal that has been pulled down 2-3, the motion vector extraction unit 102 simply extracts a motion vector between each two fields, and each motion vector extracted by the super-resolution processing unit 103 Super-resolution processing is performed on the field to output a high-resolution video signal.

  When the 2-3 pull-down video detection unit 401 determines that the input signal is a signal after 2-3 pull-down, the input signal is temporarily stored in the field storage / selection unit 104 and is derived from a different frame on the movie film. Select two fields and extract motion vectors. As a result, motion vectors can be extracted from each field, and each field can be subjected to super-resolution processing to output a high-resolution field.

  FIG. 5 shows an operation example of the apparatus.

  First, it is determined whether or not the input video signal is a video signal converted by a pull-down method (S502). As a result, if it is determined to be a pull-down video signal, the field is stored (S503), and two fields with temporal differences originating from different frames on the original film are selected (S504) to extract motion vectors. (S505), super-resolution processing is performed using the extracted motion vector (S506), and output (S507).

  If it is determined that the input video signal is not a pull-down video signal, motion vectors are extracted between consecutive fields as they are (S505), and super-resolution processing is performed using the extracted motion vectors (S506). Output (S507).

  With this operation, even when the input is a pull-down video signal, it is possible to obtain a high resolution output by performing super-resolution processing.

  FIG. 6 shows a configuration example of the 2-3 pull-down video detection unit 401. First, the inter-field motion detection unit 600 detects whether there is motion between two consecutive fields. More specifically, the difference signal between two consecutive fields is generated by calculating the difference between the signal that has passed through the one-field delay unit 301 and the signal that has not passed through the subtraction unit 302. However, in the case of the composite signal, since the color signal is inverted, the color signal is removed by the low-pass filter 303 in order to extract only the luminance signal. The difference signal between the fields generated in this way is compared with a threshold value by the comparison unit 304 to detect whether there is a significant difference between corresponding pixels between two consecutive fields. A signal obtained by holding the output of the inter-field motion detection unit 600 for one field by the one-field hold unit 610 is input to the five-field period detection unit 620 to determine whether there is a five-field period. If there are 5 field periods, it can be determined that the input signal was a 2-3 pull-down signal. The reason why the difference signal for one field is held here is to compare the entire one screen. The meaning of detecting the period of 5 fields is that if there is a 2-3 pull-down signal, the movement between adjacent fields repeats “No”, “Yes”, “No”, “No”, “Yes” as shown in FIG. This utilizes the fact that the difference between adjacent fields is repeated with a period of 5 fields.

  The first embodiment can be applied to general images in which there is no motion between adjacent fields, and can also support 2-3 pull-down images. However, the load is heavy because it is necessary to constantly monitor and compare two adjacent fields. On the other hand, in Example 2, because it is detected specifically for 2-3 pull-down video, it is expected that there will be few examples where the time that is not the target time is switched in a short time, and once it is detected, the monitoring frequency is changed. It is possible to reduce the load on the detection unit by lowering or predicting two or three fields that will not move.

  Note that the 2-3 pull-down video detection unit 401 may detect that the video is a 2-3 pull-down video based on a separate signal attached to the video.

  FIG. 7 is a schematic diagram showing an example of field selection in the field storage / selection unit 104. FIG. 7 shows the correspondence when a movie image recorded on a film with 24 frames per second is pulled down 2-3 and converted to a 60-field television signal per second. Two frames A1 and A2 are generated from the frame A on the film, and three fields B1, B2 and B3 are generated from the frame B. When trying to perform super-resolution processing between two adjacent fields for a 60Hz TV signal after 2-3 pulldown, it comes from different frames on the film such as (A2, B1) (B3, C1). Since there is motion between fields, it is possible to extract the motion vector and perform super-resolution processing to obtain the effect of high resolution, but (A1, A2) (B1, B2) (B2, B3) etc. Motion vectors cannot be extracted between fields originating from the same frame, and the effect of increasing the resolution by super-resolution processing cannot be obtained. In contrast, as shown in the lower part of the figure, motion vectors can be extracted by using two fields that are always derived from different frames on the film, such as (A2, B1) (A2, B2). It is possible to obtain the effect of high resolution by super-resolution processing.

  FIG. 8 shows an example of the configuration of the high resolution device of the third embodiment. Here, the overall flow of the third embodiment will be described with reference to FIG.

  As in the first embodiment, the 2-3 pull-down video detection unit 101 detects whether or not the input TV video signal is a video signal derived from a movie film converted into a TV signal by 2-3 pull-down. The motion vector detection unit 102 simply extracts a motion vector between two consecutive fields. If the 2-3 pull-down video detection unit 101 determines that the input signal is not a 2-3 pull-down signal, simply use the motion vector between two consecutive fields extracted by the motion vector extraction unit 102 to perform super-resolution. The processing unit 103 performs super-resolution processing and outputs a high-resolution video signal.

  If the 2-3 pull-down video detection unit 101 determines that the input signal is a signal after 2-3 pull-down, the motion vector storage / selection unit 801 extracts the motion vector between two consecutive fields extracted by the motion vector extraction unit 102. Are temporarily stored, and motion vectors extracted from two fields derived from different frames on the movie film are selected and used for the super-resolution processing in the super-resolution processing unit 103. As a result, super-resolution processing using an effective motion vector for each field becomes possible, and each field can be super-resolution processed to output a high-resolution field.

  Next, the operation will be described with reference to FIG.

  First, motion vector extraction between consecutive fields is performed on the input video signal (S901). The next operation differs depending on whether or not the input video signal is a video signal converted by the pull-down method (S903). If it is determined that the video signal is a pull-down video signal, the extracted motion vector is stored (S904), and the motion vector extracted from two fields with temporal differences derived from different frames on the original film is selected. (S905) Super-resolution processing is performed (S906) and then output (S907).

  When it is determined that the input video signal is not a pull-down video signal, super-resolution processing is performed using the motion vector extracted from the continuous field as it is (S906) and then output (S907).

  As described above, even when the input is a pull-down video signal, it is possible to obtain a high-resolution output by performing super-resolution processing.

  FIG. 10 shows a configuration example of the high resolution device of the fourth embodiment. Here, the overall flow of the fourth embodiment will be described with reference to FIG.

  As in the first embodiment, the 2-3 pull-down video detection unit 101 detects whether or not the input TV video signal is a video signal derived from a movie film converted into a TV signal by 2-3 pull-down. The motion vector detection unit 102 simply extracts a motion vector between two consecutive fields. The super-resolution processing unit 103 performs super-resolution processing regardless of whether or not a valid motion vector is detected by the motion vector detection unit 102. When the 2-3 pull-down video detection unit 101 determines that the input signal is not a 2-3 pull-down signal, the super-resolution processing unit 103 simply performs super-resolution processing using the motion vector between two consecutive fields. Output the result.

  When the 2-3 pull-down video detection unit 101 determines that the input signal is a signal after 2-3 pull-down, the output of the super-resolution processing unit 103 is temporarily stored in the field storage / selection unit 1001 and is stored on the movie film. Select the motion vector extracted from two fields derived from different frames and select and output the super-resolution field. As a result, it is possible to select and output only the super-resolution processing result using an effective motion vector for each field.

  Next, the operation will be described with reference to FIG.

  First, motion vectors are extracted between consecutive fields from the input video signal (S1101) (S1102), and super-resolution processing is performed using the extracted motion vectors (S1103). The next operation differs depending on whether or not the input video signal is a video signal converted by the pull-down method (S1104). If it is determined to be a pull-down video signal, the super-resolved field is stored (S1105), and motion vectors extracted from two fields with temporal differences originating from different frames on the original film are stored. The field subjected to super-resolution processing is selected (S1106) and output (S1107).

  If it is determined that the input video signal is not a pull-down video signal, it is output as it is (S1107).

  As described above, even when the input is a pull-down video signal, it is possible to obtain a high-resolution output by performing super-resolution processing.

  Next, the difference between the second and third embodiments and the fourth embodiment and the discrimination method will be described with reference to FIG. 7 again.

In the present embodiment, the high-resolution field immediately before output after super-resolution processing is stored, selected, and output. This means that, for example, in FIG. 7, a high-resolution field obtained by performing super-resolution processing using A2 and B1 is output at the timing of outputting field B1, and further, for example, at the timing of outputting B3, it is also superposed using A2 and B1. This means that the same high resolution field that has undergone resolution processing is output. On the other hand, in the second embodiment, at the timing of outputting B1, a high-resolution field obtained by performing super-resolution processing using the motion vector extracted from A2 and B1 is extracted from A2 and B3 at the timing of outputting B3. The resolution-enhanced field obtained by performing the super-resolution processing using the motion vector thus output is output. B1 and B3 should be the same signal because they are derived from the same frame on the film, but they are not completely the same signal due to mechanical noise during film running. Strictly different signals with different noises. Therefore, in the second embodiment, the high resolution field output at the timing of outputting B1 and the high resolution field output at the timing of outputting B3 are not completely coincident signals. In contrast, in this embodiment, since the signal stored in the field selection / storage unit is repeatedly output, the high resolution field output at the timing of outputting B1 and the output of the signal at the timing of outputting B3. The resolution field is completely the same signal. By observing the output of the super-resolution display device using this property, it is possible to discriminate between Example 2 and this Example based on whether or not the same high resolution field is always output within 5 fields. It is.
Similarly, in Example 3, the high resolution field in which A2 and B1 were subjected to super-resolution processing at the timing of outputting B1, and the high resolution field in which super-resolution processing was applied to A2 and B3 at the timing of outputting B3. The resolution field is output. Unlike the second embodiment, the motion vectors used for the super-resolution processing are the same in the third embodiment and the third embodiment. However, in the third embodiment, the outputs at both timings are not always the same signal. . Therefore, if the output of the super-resolution display device is observed using this property, it is possible to distinguish between the third embodiment and the present embodiment depending on whether or not the same high resolution field is always output within five fields. Is possible.

  The 2-3 pulldown has been described above, but other pulldown methods are also applicable. For example, a 2-2 pull-down video detector that detects whether an input TV video signal is a movie film-derived video converted to a PAL or SECAM television signal by 2-2 pull-down 2-3 The pull-down video detection unit 401 may be mounted instead of or together. The operation when the super-resolution display device of the present embodiment processes 2-2 pull-down video is the same as in FIG. 5, but in the 2-3 pull-down detection unit, whether there is a motion between fields has a 5-field period. On the other hand, the 2-2 pull-down detector detects whether or not it has 4 fields or 2 fields.

1 is a block diagram illustrating a configuration example of a super-resolution display device according to a first embodiment. FIG. 3 is a flowchart illustrating an operation example of the super-resolution display device according to the first embodiment. FIG. 3 is a block diagram illustrating a configuration example of a motion detection unit between adjacent fields of the super-resolution display device according to the first embodiment. FIG. 4 is a block diagram illustrating a configuration example of a super-resolution display device according to a second embodiment. FIG. 7 is a flowchart showing an operation example of the super-resolution display device of the second embodiment. FIG. 6 is a block diagram illustrating a configuration example of a 2-3 pull-down video detection unit of the super-resolution display device according to the second embodiment. Principle diagram of Example 2 FIG. 6 is a block diagram illustrating a configuration example of a super-resolution display device according to a third embodiment. FIG. 7 is a flowchart showing an operation example of the super-resolution display device according to the third embodiment. FIG. 4 is a block diagram illustrating a configuration example of a super-resolution display device according to a fourth embodiment. FIG. 9 is a flowchart showing an operation example of the super-resolution display device according to the fourth embodiment.

Explanation of symbols

101 Motion detector between adjacent fields
102 Motion vector detector
103 Super-resolution processor
104 Field storage / selection section
105 tuner
106 Media reader
301 1 field delay
302 Subtraction part
303 Low pass filter
304 Comparison part
305 1 field integration unit
401 2-3 pull-down video detector
600 Inter-field motion detector
610 1 field hold section
620 5-field period detector
801 Motion vector memory / selection unit
1001 Field storage / selection section

Claims (8)

A high-resolution device for increasing the resolution of a moving image including an image of a first resolution into a moving image including an image of a second resolution higher than the first resolution,
A motion vector extraction unit for extracting a motion vector from a plurality of images of a video;
A high-resolution processing unit that increases the resolution of a moving image using the extracted motion vector;
A motion detection unit between adjacent images that detects whether or not images without motion are consecutive in the video;
When the motion detection unit between adjacent images detects that images without motion are continuous, the second image having the higher resolution is output for the continuous images due to the difference from the image before the motion disappears. And a high resolution device having a control unit. The high resolution device according to claim 1,
A storage unit for storing the image of the first resolution;
When the motion detection unit between adjacent images detects that images without motion are continuous, the control unit reads out the image of the first resolution before the motion disappears from the images stored in the storage unit. And a resolution increasing device for inputting the vector to the vector extracting unit. The high resolution device according to claim 1,
A storage unit for storing the motion vector extracted by the motion vector extraction unit;
When the motion detection unit between adjacent images detects that images without motion are continuous, the control unit moves the image after disappearing from the image before motion disappears among the images stored in the storage unit. A resolution increasing apparatus that reads out a vector and inputs the vector to the motion detection unit between adjacent images. The high resolution device according to claim 1,
A storage unit that stores an image of the second resolution that has been increased in resolution by the resolution enhancement processing unit;
When the motion detection unit between adjacent images detects that images without motion are continuous, the control unit reads out the second resolution image stored in the storage unit and inputs the image to the vector extraction unit. High resolution device. The high resolution device according to any one of claims 1 to 4,
The adjacent image motion detection unit is a high-resolution device that is a pull-down video detection unit that detects whether or not the video has been subjected to pull-down processing for conversion from film to a television signal. The high resolution device according to claim 1,
The pull-down video detection unit is a high-resolution device that detects whether or not a corresponding pixel difference between two consecutive images exceeds a condition, and determines whether or not there is a five-image cycle for the detected image. A method for increasing the resolution of a video,
It is determined whether there is a difference larger than a threshold value between the first image and the second image arranged on the time axis included in the video,
If it is determined that there is a difference, a resolution enhancement process is performed based on the second image,
A resolution enhancement method for performing a resolution enhancement process based on the first image or an image before the first image on a time axis when it is determined that there is no difference. A method for increasing the resolution of a video,
It is determined whether there is a difference larger than a threshold value between the first image and the second image arranged on the time axis included in the video,
If it is determined that there is a difference, a resolution enhancement process is performed based on the second image,
A resolution-enhancing method of outputting an image that has been subjected to resolution-enhancement processing based on the first image or an image preceding the time axis with respect to the first image when it is determined that there is no difference.

Images