JP2011045147A - Rendering method and rendering device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To perform interlace/progressive conversion processing and the write of image data to a free frame memory in a period during which one of frame memories is free. <P>SOLUTION: A timing control part outputs a write request to a write part, and the write part returns an acknowledgment to the write request to the timing control part when the read of the image data from a display frame memory by a read part is ended when the image data not read yet by the read part remains in a write frame memory at present, and returns the acknowledgment to the write request to the timing control part without delay after the write request is input when the image data not read yet by the read part does not remain in the write frame memory at present. From the time when the timing control part receives input of the acknowledgment from the write part, an image processing part performs image processing, and the write part writes the image data obtained through the image processing to the write frame memory. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a rendering method and a rendering apparatus.

Conventionally, when viewing television on a personal computer, video signals of NTSC (National Television Standards Committee), PAL (Phase Alternation by Line), or SECAM (SEquential Couleur A Memoire) are non-interlaced (progressive) of the personal computer. In order to display on a monitor, combing removal processing was applied to a frame created by interleaving an even field and an odd field.
JP 2001-339694 A JP 2002-185933 A

  When the above processing is performed, the number of frames displayed per second is about 30 or about 25, so that there is a problem that the smoothness of the motion is inferior to the image before the processing. In addition, depending on the combing removal method, there is a problem that the entire video is blurred or the detail of the video is crushed.

  As a method for improving such a problem, for example, there is an interlace / progressive conversion method in which each field of the NTSC signal is converted into one frame so that about 60 frames can be displayed per second. In this interlace / progressive conversion method, the motion of a video is determined in units of pixels for each field of an interless signal, and an interpolation method is adaptively determined depending on whether the pixel is a moving part or a stationary part. Switch.

  Patent Document 1 describes an invention aimed at reducing the failure of motion detection that occurs when interlaced / progressive conversion is performed on a high-speed scroll screen. Since the threshold value for detection is variable, an extra buffer memory for one frame is required.

  Patent Document 2 describes an invention aimed at improving the accuracy of motion detection in interlaced / progressive conversion with motion correction, but it is 1 field before, 2 fields before, 3 fields before and 6 fields. It is necessary to use the previous field data, and this requires an extra buffer memory for four fields.

  Further, in the interlace / progressive conversion method, since the motion cannot be correctly determined, the moving image portion is determined to be a still image portion, and accordingly, interpolation processing for still images may be applied to the portion. . When still image interpolation processing is applied to the moving image portion, dust-like noise is included in the video. Such noise is visually noticeable.

  The present invention provides an interlace / progressive conversion process and an image to an empty frame memory in a period in which one of two frame memories which are alternately a write frame memory and a display frame memory is empty. An object of the present invention is to provide a rendering method and a rendering apparatus that allow data to be written.

  According to the first aspect of the present invention, two frame memories that are a write frame memory and a display frame memory are alternately and complementarily used, and the timing control unit outputs a write request to the write unit. And the writing unit reads the image data from the display frame memory by the reading unit when the image data not yet read by the reading unit remains in the current writing frame memory. When the processing is completed, an acknowledgment to the write request is returned to the timing control unit, and if there is no image data that has not been read by the reading unit in the current write frame memory, the write request is input. A step of returning an affirmative response to the write request to the timing control unit without delay, and the timing control unit Since the positive response is input from the writing unit, the image processing unit performs image processing, and the writing unit writes the image data obtained by the image processing to a writing frame memory; and the timing The control unit compares the time when the image processing and writing are completed with a reference time, and updates the interval parameter based on the comparison result, and the timing control unit adds the interval parameter to the reference time. The step of setting the time as the next reference time, and at the time when the timing control unit added the interval parameter from the time when the write request was last output to the writing unit, the writing unit again, There is provided a rendering method comprising: outputting a write request; and repeating the above operation.

  According to the second aspect of the present invention, the two frame memories which become the write frame memory and the display frame memory alternately and complementarily, and the timing controller outputs a write request to the writer. And the writing unit finishes reading the image data from the display frame memory by the reading unit when the image data not yet read by the reading unit remains in the current writing frame memory. When the timing control unit returns an affirmative response to the write request, and no image data that has not yet been read by the reading unit remains in the current write frame memory, the write request is input. Means for returning an affirmative response to the write request to the timing control unit without delay, and the timing control unit Is input from the writing unit, the image processing unit performs image processing, the writing unit writes the image data obtained by the image processing into a writing frame memory, and the timing control unit Comparing the time when the image processing and writing are completed with a reference time, and updating the interval parameter according to the comparison result, and the timing control unit adding the time when the interval parameter is added to the reference time next time And the timing control unit again sends a write request to the writing unit at the time when the interval parameter is added from the time when the writing request was previously output to the writing unit. There is provided a rendering apparatus comprising: means for outputting; and means for repeating the above-described operation.

  According to the third aspect of the present invention, the two frame memories which become the write frame memory and the display frame memory alternately and complementarily, and the timing controller outputs a write request to the writer. And the writing unit finishes reading the image data from the display frame memory by the reading unit when the image data not yet read by the reading unit remains in the current writing frame memory. When the timing control unit returns an affirmative response to the write request, and no image data that has not yet been read by the reading unit remains in the current write frame memory, the write request is input. Means for returning an affirmative response to the write request to the timing control unit without delay, and the timing control unit Is input from the writing unit, the image processing unit performs image processing, the writing unit writes the image data obtained by the image processing into a writing frame memory, and the timing control unit Comparing the time when the image processing and writing are completed with a reference time, and updating the interval parameter according to the comparison result, and the timing control unit adding the time when the interval parameter is added to the reference time next time And the timing control unit again sends a write request to the writing unit at the time when the interval parameter is added from the time when the writing request was previously output to the writing unit. A program for causing a computer to function as a rendering device comprising: means for outputting; and means for repeating the above-described operation. It is.

  According to the present invention, using two frame memories that are alternately and alternately a write frame memory and a display frame memory, the timing control unit outputs a write request to the writing unit; When image data that has not yet been read out by the reading unit remains in the current writing frame memory, the writing unit reads the image data from the display frame memory when the reading unit finishes. When an acknowledgment to the write request is returned to the timing control unit, and there is no image data that has not yet been read out by the reading unit in the current write frame memory, there is no delay after inputting the write request. Returning an affirmative response to the write request to the timing control unit; and Is input from the writing unit, the image processing unit performs image processing, the writing unit writes the image data obtained by the image processing to a writing frame memory, and the timing control unit Comparing the time at which the image processing and writing are completed with a reference time, and updating the interval parameter according to the result of the comparison, and the timing control unit adding the time at which the interval parameter is added to the reference time next time And the timing control unit makes a write request to the writing unit again at the time when the interval parameter is added from the time when the timing control unit previously output the writing request to the writing unit. And a step of repeating the above operation, so that the write frame memory and the display frame memory are alternately and complementarily complemented. It is possible to write image data to the frame memory which interlace / progressive conversion process and the vacant period in which any of the two frame memories to be re is free is performed.

It is a block diagram which shows the structure of the interlace / progressive conversion apparatus by Embodiment 1 of this invention. It is a 1st figure for demonstrating the effect of Embodiment 1 of this invention. It is a 2nd figure for demonstrating the effect of Embodiment 1 of this invention. 3 is a flowchart for explaining an interlace / progressive conversion method performed by the interlace / progressive conversion device shown in FIG. 1. It is a block diagram which shows the structure of the interlace / progressive conversion apparatus by Embodiment 2 of this invention. It is a timing diagram for demonstrating operation | movement of the interlace / progressive conversion apparatus by Embodiment 3 of this invention. It is a block diagram which shows the structure of the interlace / progressive conversion apparatus by Embodiment 3 of this invention. It is a block diagram which shows the structural example of the noise removal filter by Embodiment 4 of this invention. It is a block diagram which shows the structure of the interlace / progressive conversion apparatus containing the renderer by Embodiment 5 of this invention. FIG. 10 is a first timing diagram for explaining the operation of the interlace / progressive conversion device shown in FIG. 9. FIG. 10 is a second timing chart for explaining the operation of the interlace / progressive conversion device shown in FIG. 9. It is a flowchart for demonstrating operation | movement of the interlace / progressive conversion apparatus which has a film image conversion function by Embodiment 6 of this invention. It is a 1st timing diagram for demonstrating operation | movement of the interlace / progressive conversion apparatus which has a film image conversion function by Embodiment 6 of this invention. It is a 2nd timing diagram for demonstrating operation | movement of the interlace / progressive conversion apparatus which has a film image conversion function by Embodiment 6 of this invention. It is a 3rd timing diagram for demonstrating operation | movement of the interlace / progressive conversion apparatus which has a film image conversion function by Embodiment 6 of this invention. It is a 4th timing diagram for demonstrating operation | movement of the interlace / progressive conversion apparatus which has a film image conversion function by Embodiment 6 of this invention.

  The best mode for carrying out the present invention will be described below in detail with reference to the drawings.

[Embodiment 1]
FIG. 1 is a block diagram showing a configuration of an interlace / progressive conversion apparatus according to Embodiment 1 of the present invention.

  The first field memory 101-1 stores image data of a field one field ahead of the field to be processed. The second field memory 101-2 stores image data of the field to be processed. The third field memory 101-3 stores the image data of the field immediately before the field to be processed.

  The image data writing unit 103 writes the image data in the first to third field memories 101-1, 101-2, and 101-3. The first to third field memories 101-1, 101-2, and 101-3 have a ring buffer configuration, and image data of three fields are stored in these field memories in a circular manner. May be.

  The moving image interpolation pixel generation unit 105 generates moving image interpolation pixels based on the image data stored in the second field memory 101-2. For example, the number of the field currently being processed is n, the vertical position of the pixel currently being processed as a pixel to be interpolated in that field, i, and the pixel to be interpolated in that field When the horizontal position of the pixel currently being processed is j, a moving image interpolation pixel A (n, i, j) is generated according to equation (1).

ここで、ｘ（ｎ，ｉ−１，ｊ）は、処理の対象となっているフィールドの処理の対象となっている画素の１つ上の画素の値であり、ｘ（ｎ，ｉ＋１，ｊ）は、処理の対象となっているフィールドの処理の対象となっている画素の１つ下の画素の値である。上記では、これらの２つの画素の値の平均値を処理の対象となっている画素の値としているが、これらの２つの画素のうちの１つの画素の値を処理の対象となっている画素の値としてもよい。また、処理の対象となっているフィールドの画素のうちの処理の対象となっている画素の所定の近傍の画素の値の加重平均を処理の対象となっている画素の値としてもよい。

Here, x (n, i−1, j) is the value of the pixel one pixel above the pixel to be processed in the field to be processed, and x (n, i + 1, j). ) Is the value of the pixel one pixel below the pixel to be processed in the field to be processed. In the above, the average value of the values of these two pixels is the value of the pixel to be processed, but the value of one pixel of these two pixels is the pixel to be processed. It is good also as the value of. Also, a weighted average of the values of pixels in a predetermined vicinity of the pixel to be processed among the pixels in the field to be processed may be used as the value of the pixel to be processed.

  The still image interpolation pixel generation unit 107 generates still image pixels based on the pixel data stored in the first field memory 101-1 and the pixel data stored in the third field memory 101-3. Generate. For example, the number of the field currently being processed is n, the vertical position of the pixel currently being processed as a pixel to be interpolated in that field, i, and the pixel to be interpolated in that field If the horizontal position of the pixel currently being processed is j, a moving image interpolation pixel B (n, i, j) is generated according to equation (2).

ここで、ｘ（ｎ−１，ｉ，ｊ）は、処理の対象となっているフィールドの１つ前のフィールドの画素のうち、処理の対象となっている画素と同一の位置（ｉ，ｊ）にある画素の値であり、ｘ（ｎ＋１，ｉ，ｊ）は、処理の対象となっているフィールドの１つ後のフィールドの画素のうち、処理の対象となっている画素と同一の位置（ｉ，ｊ）にある画素の値である。上記では、これらの２つの画素の値の平均値を処理の対象となっている画素の値としているが、これらの２つの画素のうちの１つの画素の値を処理の対象となっている画素の値としてもよい。また、処理の対象となっているフィールドの近傍のフィールドの画素のうちの処理の対象となっている画素と同一の位置にある画素の値の加重平均を処理の対象となっている画素の値としてもよい。

Here, x (n−1, i, j) is the same position (i, j) as the pixel to be processed among the pixels in the field immediately before the field to be processed. ), And x (n + 1, i, j) is the same position as the pixel to be processed among the pixels in the next field after the field to be processed. This is the value of the pixel at (i, j). In the above, the average value of the values of these two pixels is the value of the pixel to be processed, but the value of one pixel of these two pixels is the pixel to be processed. It is good also as the value of. Also, the value of the pixel subject to processing is the weighted average of the values of the pixels in the same position as the pixel subject to processing among the pixels in the field in the vicinity of the field subject to processing. It is good.

  The difference calculation unit 109 obtains the difference C (n, i, j) for the pixel to be processed in the field to be processed according to the equation (3).

ここで、ｘ（ｎ−１，ｉ，ｊ）は、処理の対象となっているフィールドの１つ前のフィールドの画素のうち、処理の対象となっている画素と同一の位置（ｉ，ｊ）にある画素の値であり、ｘ（ｎ＋１，ｉ，ｊ）は、処理の対象となっているフィールドの１つ後のフィールドの画素のうち、処理の対象となっている画素と同一の位置（ｉ，ｊ）にある画素の値である。差分Ｃ（ｎ，ｉ，ｊ）は、動き量の基本となる。

Here, x (n−1, i, j) is the same position (i, j) as the pixel to be processed among the pixels in the field immediately before the field to be processed. ), And x (n + 1, i, j) is the same position as the pixel to be processed among the pixels in the next field after the field to be processed. This is the value of the pixel at (i, j). The difference C (n, i, j) is the basis of the amount of motion.

  The difference correction unit 111 obtains a corrected difference D (n, i, j) according to the equation (4).

ここで、Δｃの値は、ノイズのレベル等により決定され、補正値保持部１１３に格納されている。

Here, the value of Δc is determined by the noise level or the like and stored in the correction value holding unit 113.

  The difference accumulating unit 115 updates the accumulated motion amount m (i, j) stored in the accumulated motion amount memory 117 according to the equation (5).

ここで、ＴＨの値は、視覚特性等により決定され、しきい値保持部１１９に格納されている。ＴＨの値は、例えば、１６である。

Here, the value of TH is determined by visual characteristics or the like and stored in the threshold value holding unit 119. The value of TH is 16, for example.

  If a certain amount of motion is detected by calculating the cumulative motion amount m (i, j) according to the above equation, the cumulative motion amount m (i, j) A value obtained by adding 1 to the accumulated motion amount in the previous field and subtracting 1 or when the motion amount is large is a threshold value TH, and decreases by 1 every time one field has passed since then. By doing so, the motion detection is tailed. Thereby, the possibility that the moving image part is erroneously determined to be a still image part after movement can be reduced. When the moving image part is erroneously determined to be a still image part, an interpolation value completely different from the interpolation value calculated when correctly processed as the moving image part is calculated, which causes a visual problem. On the other hand, if the still image portion is erroneously determined to be a moving image portion, only the resolution is lowered, and this is not a visual problem. In particular, even if the still image portion around the portion where the motion is actually detected is erroneously determined to be a moving image portion, the reduction in the resolution of the portion is not visually recognized due to the masking effect due to the portion that actually moves. . Therefore, the advantage of introducing the formula (5) and reducing the possibility that the moving image portion is erroneously determined as the still image portion is that the possibility that the still image portion is erroneously determined as the moving image portion is increased. Therefore, the introduction of the equation (5) is useful for improving the image quality as a whole. Also, many still image portions such as telop portions where characters are fixed are not mistakenly determined to be moving image portions even if the formula (5) is introduced.

  The motion amount calculation unit 121 calculates the motion amount M (i, j) according to the equation (6).

合成補間画素生成部１２３は、式（７）に従って合成補間画素ｘ’（ｎ，ｉ，ｊ）を計算する。

The synthetic interpolation pixel generation unit 123 calculates a synthetic interpolation pixel x ′ (n, i, j) according to the equation (7).

選択部１２５は、ライン毎に実在画素ｘ（ｎ，ｉ，ｊ）と補間画素ｘ’（ｎ，ｉ，ｊ）を切り換えて、プログレッシブ走査の画像データを出力する。

The selection unit 125 switches the actual pixel x (n, i, j) and the interpolated pixel x ′ (n, i, j) for each line, and outputs progressive scan image data.

  The interless / progressive conversion device shown in FIG. 1 can be realized by hardware, but the computer reads and executes a program for causing the computer to function as the interless / progressive conversion device shown in FIG. Can also be realized.

  Next, the effect of calculating the motion amount M (i, j) according to the equation (6) will be described.

  As shown in FIG. 2, it is assumed that a black vertical line moves to the right on a white flat background. Further, in order to facilitate the consideration, it is assumed that the motion accumulation amount m (i, j) is calculated according to the equation (8).

そうすると、式（６）を用いずに、仮に、式（９）に従って、動き量Ｍ（ｉ，ｊ）を計算することとすると、図２に示すように、動きが検出されずに、静止画補間画素により補間が行われることとなり、コーミングが発生する。

Then, if the motion amount M (i, j) is calculated according to the equation (9) without using the equation (6), as shown in FIG. Interpolation is performed by the interpolation pixel, and combing occurs.

すなわち、図２において、四角の画素は元々ある画素であり、丸の画素は補完するべき画素である。例えば、画素１３４と画素３３４との差分の絶対値が画素２３４の動き量となるが、この値はゼロであるため、画素２３４は、静止画画素であると間違って判断される。そうすると、画素１３４と画素３３４の平均値が画素２３４の補間値となり、画素２３４は白となってしまう。画素２１４、２５４についても同様である。そうすると、画素２２４、２４４、２６４が正しく黒であるのに対し、画素２１４、２３４、２５４は間違って白となってしまい、コーミングが発生する。

That is, in FIG. 2, square pixels are originally pixels, and round pixels are pixels to be complemented. For example, the absolute value of the difference between the pixel 134 and the pixel 334 is the amount of motion of the pixel 234. Since this value is zero, the pixel 234 is erroneously determined to be a still image pixel. Then, the average value of the pixel 134 and the pixel 334 becomes the interpolation value of the pixel 234, and the pixel 234 becomes white. The same applies to the pixels 214 and 254. Then, while the pixels 224, 244, and 264 are correctly black, the pixels 214, 234, and 254 are erroneously white and combing occurs.

  On the other hand, if the motion amount M (i, j) is calculated according to the equation (6) according to the present invention, the motion is correctly detected and interpolation is performed by the moving image interpolation pixels as shown in FIG. No combing occurs.

  That is, in FIG. 3, as in the case of FIG. 2, the square pixels are originally pixels, and the round pixels are pixels to be complemented. For example, the smaller of the absolute value of the difference between the pixel 134 and the pixel 334, the sum of the absolute value of the difference between the pixel 044 and the pixel 244, and the threshold value TH is the amount of motion of the pixel 234. Since it is the threshold value TH, the pixel 234 is correctly determined as a moving image pixel. Then, the average value of the pixel 224 and the pixel 244 becomes the interpolation value of the pixel 234, and the pixel 234 becomes black. The same applies to the pixels 214 and 254. Then, since the pixels 224, 244, and 264 are correctly black and the pixels 214, 234, and 254 are also correctly black, no combing occurs.

  Thus, by calculating the motion amount M (i, j) according to the equation (6), it is possible to avoid erroneously determining that a moving image pixel is a still image pixel.

  Next, an interlace / progressive conversion method performed by the interlace / progressive conversion apparatus shown in FIG. 1 will be described.

  Referring to FIG. 4, first, the moving image interpolation pixel generation unit 105 generates a moving image interpolation pixel according to Expression (1) (step S201). Next, the still image interpolation pixel generation unit 107 generates a still image interpolation pixel according to Expression (2) (step S203). Next, the difference calculation unit 109 calculates the difference according to the equation (3) (step S205). Next, the difference correction unit 111 corrects the difference according to the equation (4) (step S207). Next, the difference accumulating unit 115 updates the accumulated motion amount according to Expression (5) (step S209). Next, the motion amount calculation unit 121 calculates a motion amount according to Equation (6) (step S211). Next, the synthetic | combination interpolation pixel production | generation part 123 produces | generates a synthetic | combination interpolation pixel according to Formula (7) (step S213).

[Embodiment 2]
When interlace / progressive conversion is performed in the configuration of the first embodiment, the timing of the moving image may be delayed with respect to the sound accompanying the moving image. In the second embodiment, the timing of the moving image is not delayed with respect to the sound. For this purpose, processing is performed without using image data of future fields.

  FIG. 5 is a block diagram showing a configuration of an interlace / progressive conversion apparatus according to Embodiment 2 of the present invention.

  The first field memory 131-1 stores image data of a field to be processed. The second field memory 131-2 stores image data of the field immediately before the field to be processed. The third field memory 131-3 stores image data of the field two fields before the field to be processed. The fourth field memory 131-4 stores image data of a field three fields before the field to be processed.

  The image data writing unit 133 writes the image data to the first to fourth field memories 131-1, 131-2, 131-3, and 131-4. Note that the first to fourth field memories 131-1, 131-2, 131-3, and 131-3 have a ring buffer configuration, and image data of four fields is cyclically transferred to these field memories. You may make it store.

  The moving image interpolation pixel generation unit 105 generates a moving image interpolation pixel based on the image data stored in the first field memory 131-1. For example, the number of the field currently being processed is n, the vertical position of the pixel currently being processed as a pixel to be interpolated in that field, i, and the pixel to be interpolated in that field If the horizontal position of the pixel currently being processed is j, moving image interpolation pixel A (n, i, j) is generated according to equation (10).

ここで、ｘ（ｎ，ｉ−１，ｊ）は、処理の対象となっているフィールドの処理の対象となっている画素の１つ上の画素の値であり、ｘ（ｎ，ｉ＋１，ｊ）は、処理の対象となっているフィールドの処理の対象となっている画素の１つ下の画素の値である。上記では、これらの２つの画素の値の平均値を処理の対象となっている画素の値としているが、これらの２つの画素のうちの１つの画素の値を処理の対象となっている画素の値としてもよい。また、処理の対象となっているフィールドの画素のうちの処理の対象となっている画素の所定の近傍の画素の値の加重平均を処理の対象となっている画素の値としてもよい。

Here, x (n, i−1, j) is the value of the pixel one pixel above the pixel to be processed in the field to be processed, and x (n, i + 1, j). ) Is the value of the pixel one pixel below the pixel to be processed in the field to be processed. In the above, the average value of the values of these two pixels is the value of the pixel to be processed, but the value of one pixel of these two pixels is the pixel to be processed. It is good also as the value of. Also, a weighted average of the values of pixels in a predetermined vicinity of the pixel to be processed among the pixels in the field to be processed may be used as the value of the pixel to be processed.

  The still image interpolation pixel generation unit 137 generates a still image pixel based on the pixel data stored in the second field memory 131-2. For example, the number of the field currently being processed is n, the vertical position of the pixel currently being processed as a pixel to be interpolated in that field, i, and the pixel to be interpolated in that field When the horizontal position of the pixel currently being processed is j, a moving image interpolation pixel B (n, i, j) is generated according to equation (11).

ここで、ｘ（ｎ−１，ｉ，ｊ）は、処理の対象となっているフィールドの１つ前のフィールドの画素のうち、処理の対象となっている画素と同一の位置（ｉ，ｊ）にある画素の値である。但し、処理の対象となっているフィールドの過去の近傍のフィールドの画素のうちの処理の対象となっている画素と同一の位置にある画素の値の加重平均を処理の対象となっている画素の値としてもよい。

Here, x (n−1, i, j) is the same position (i, j) as the pixel to be processed among the pixels in the field immediately before the field to be processed. ). However, the pixel subject to processing is a weighted average of the values of the pixels at the same position as the pixel subject to processing among the pixels in the past neighboring field of the field subject to processing. It is good also as the value of.

  The difference calculation unit 139 obtains a difference C (n, i, j) for the pixel to be processed in the field to be processed according to the equation (12).

ここで、ｘ（ｎ−３，ｉ，ｊ）は、処理の対象となっているフィールドの３つ前のフィールドの画素のうち、処理の対象となっている画素と同一の位置（ｉ，ｊ）にある画素の値であり、ｘ（ｎ−１，ｉ，ｊ）は、処理の対象となっているフィールドの１つ前のフィールドの画素のうち、処理の対象となっている画素と同一の位置（ｉ，ｊ）にある画素の値である。差分Ｃ（ｎ，ｉ，ｊ）は、動き量の基本となる。

Here, x (n−3, i, j) is the same position (i, j) as the pixel to be processed among the pixels in the field three fields before the field to be processed. ), And x (n-1, i, j) is the same as the pixel to be processed among the pixels in the field immediately before the field to be processed. Is the value of the pixel at position (i, j). The difference C (n, i, j) is the basis of the amount of motion.

  The difference correction unit 141 obtains a corrected difference D (n, i, j) according to the equation (13).

ここで、Δｃの値は、ノイズのレベル等により決定され、補正値保持部１１３に格納されている。

Here, the value of Δc is determined by the noise level or the like and stored in the correction value holding unit 113.

  The difference accumulating unit 145 updates the accumulated motion amount m (i, j) stored in the accumulated motion amount memory 147 according to the equation (14).

ここで、ＴＨの値は、視覚特性等により決定され、しきい値保持部１４９に格納されている。ＴＨの値は、例えば、１６である。

Here, the value of TH is determined by visual characteristics or the like and stored in the threshold value holding unit 149. The value of TH is 16, for example.

  If a certain amount of motion is detected by calculating the cumulative motion amount m (i, j) according to the above equation, the cumulative motion amount m (i, j) A value obtained by adding 1 to the accumulated motion amount in the previous field and subtracting 1 or when the motion amount is large is a threshold value TH and decreases by 1 every time one field elapses. By doing so, the motion detection is tailed. Thereby, the possibility that the moving image part is erroneously determined to be a still image part after movement can be reduced.

  The motion amount calculation unit 151 calculates the motion amount M (i, j) according to the equation (15).

合成補間画素生成部１５３は、式（１６）に従って合成補間画素ｘ’（ｎ，ｉ，ｊ）を計算する。

The synthetic interpolation pixel generation unit 153 calculates a synthetic interpolation pixel x ′ (n, i, j) according to the equation (16).

選択部１５５は、ライン毎に実在画素ｘ（ｎ，ｉ，ｊ）と補間画素ｘ’（ｎ，ｉ，ｊ）を切り換えて、プログレッシブ走査の画像データを出力する。

The selection unit 155 switches the actual pixel x (n, i, j) and the interpolation pixel x ′ (n, i, j) for each line, and outputs progressive scan image data.

  Although the interlace / progressive conversion device shown in FIG. 5 can be realized by hardware, the computer reads and executes a program for causing the computer to function as the interless / progressive conversion device shown in FIG. Can also be realized.

[Embodiment 3]
When the configuration of the first embodiment is followed, as shown in FIG. 6, in the fields before and after the boundary of the scene change (field B2 and field A3 in the example of FIG. 6), all the interpolated pixels are interpolated (within the field). Interpolation) should be done. However, depending on the content of the image data, still image interpolation may be performed for some of the interpolation pixels. Therefore, in the third embodiment, a scene change is detected for each field, and moving image interpolation (intra-field interpolation) is forcibly performed for all interpolation pixels in fields before and after the boundary of the scene change.

  FIG. 7 is a block diagram showing a configuration of an interlace / progressive conversion apparatus according to Embodiment 3 of the present invention.

  As is clear from comparison between FIG. 7 and FIG. 1, the interlace / progressive conversion apparatus according to the third embodiment is different from the interlace / progressive conversion apparatus according to the first embodiment in that a scene change detection unit 161 is added. Only the difference. Therefore, only the scene change detection unit 161 and its peripheral part will be described.

  Referring to FIG. 6, when processing is performed on the field B2, it is detected that the field A2 which is the field immediately before the field B2 and the field A3 which is the field immediately after the field B2 belong to different scenes. That's fine. Similarly, when processing is performed on the field A3, it is only necessary to detect that the field B2 which is the field immediately before the field A3 and the field B3 which is the field immediately after the field A3 belong to different scenes. Therefore, it is only necessary to detect that the field immediately before and the field immediately after the field being processed belongs to different scenes. In order to detect whether they belong to different scenes, for example, it is determined whether the cross-correlation value between both fields is below a threshold value, or the cross-correlation value between both fields is What is necessary is just to judge whether it changed suddenly.

  For this purpose, the scene change detection unit 161 inputs the image data of the field immediately after the current field from the first field memory 101-1, and the current field from the third field memory 101-3. A scene change is detected by inputting the image data of the previous field and calculating the cross-correlation value between the two image data. When a scene change is detected, the scene change detection unit 161 sets all cumulative motion amounts stored in the cumulative motion amount memory 117 to the threshold value TH. By doing so, all the interpolation pixels in the field become moving image interpolation pixels.

  Although the interlace / progressive conversion device shown in FIG. 7 can be realized by hardware, the computer reads and executes a program for causing the computer to function as the interless / progressive conversion device shown in FIG. Can also be realized.

[Embodiment 4]
In the fourth embodiment, a noise removal filter is arranged before the image data writing unit 133. The noise in the flat portion is removed by this noise removal filter.

  FIG. 8 shows a configuration example of the noise removal filter. Referring to FIG. 8, the noise removal filter includes delay circuits 171-1 to 171-6, multipliers 172-1 to 172-7, an adder 173, a comparator 174, and a selector 175.

  The delay circuits 171-1 to 171-6 delay pixels in units of one pixel. Multipliers 172-1 to 172-7 and adder 173 constitute a product-sum circuit. By appropriately selecting the coefficients of the multipliers 172-1 to 172-7 (eg, 1/8, 1/8, 1/8, 2/8, 1/8, 1/8, 1/8), the delay The circuits 171-1 to 171-6, the multipliers 172-1 to 172-7, and the adder 173 are low-pass filters. The comparator 174 has a value obtained by subtracting a predetermined value from the value of the output data of the delay circuit 171-3 and a value obtained by adding the predetermined value to the value of the output data of the delay circuit 171-3. Whether or not it is between. The selector 175 has a value obtained by subtracting a predetermined value from the value of the output data of the delay circuit 171-3 and a value obtained by adding the predetermined value to the value of the output data of the delay circuit 171-3. If it is between, the output data of the adder 173 is selected, and if not, the output data of the delay circuit 171-3 is selected. By doing so, it is possible to remove noise on the flat portion.

  The noise removal filter shown in FIG. 8 can be realized by hardware, but can also be realized by the computer reading and executing a program for causing the computer to function as the noise removal filter shown in FIG. it can.

[Embodiment 5]
In the first to fourth embodiments, the renderer is not considered. However, as shown in FIG. 9, a renderer 191 exists in the final stage of the interlace / progressive conversion apparatus. The renderer 191 has a function of outputting image data to the monitor at the monitor clock frequency, horizontal synchronization frequency, and frame frequency.

  Referring to FIG. 9, the renderer 191 includes a writing unit 192, an A frame memory 193, a B frame memory 194, and a reading unit 195. The A frame memory 193 and the B frame memory 194 alternately and complementarily become a write frame memory and a display frame memory.

  The writing unit 192 writes the image data to the A frame memory 193 and the B frame memory 194 that are currently the writing frame memory. The reading unit 195 reads image data from the A frame memory 193 and the B frame memory 194 that are currently the display frame memory, and supplies the image data to the monitor.

  The field memory group 101 (or 131) shown in FIG. 9 is a group of field memories 101-1 to 101-3, for example, in the example of FIG. The image data reading / interpolating unit 182A (or 182B or 182C) includes, for example, a block group surrounded by a frame 182A in FIG. 1 in the example of FIG. The writing time memory 181 stores the time when the image data writing unit 103 (or 133) writes the image data into the field memory group 101 (or 131). In the timing control unit 183, the image data reading / interpolating unit 182A (or 182B or 182C) reads and interpolates the image data, and supplies the progressive image data obtained as a result to the writing unit 192. Reference numeral 192 controls the timing for writing image data to the A frame memory 193 and the B frame memory 194 which are currently the write frame memory.

  Next, in particular, timing control performed by the timing control unit 183 will be described with reference to FIG.

  Referring to FIG. 10, the time when the image data writing unit 103 (or 133) first writes the image data into the field memory 101 (or 131) is defined as T-In (1). The time T-In (1) is written in the write time memory 181. Thereafter, each time the image data writing unit 103 (or 133) writes the image data into the field memory 101 (or 131), the writing time T-In (2), T-In (3), T-In ( 4),... Are written in the write time memory 181.

  At time T-In (1), the timing control unit 183 outputs a write request to the writing unit 192. In FIG. 10, the left end of the VRAM acquisition strip is the time when the write request is output. At this time, since the image data has already been written in the B frame memory 194 which is a writing frame memory (the image data in the B frame memory 194 has not yet been read), the A frame memory by the reading unit 195 Waiting until time TR (1) when the process of reading image data from 183 is completed and the A-frame memory 193 becomes free occurs. At time TR (1), the writing unit 192 receives the timing control unit. A positive response (ACK) is returned to 183. In FIG. 10, the right end of the VRAM acquisition strip is the time when an affirmative response is returned. Accordingly, the width of the VRAM acquisition strip represents a waiting period from when the timing control unit 183 outputs a write request to the writing unit 192 until the writing unit 192 returns an acknowledgment to the timing control unit 183.

  Then, the image data reading / interpolating unit 182A (or 182B or 183C) starts the interlace / progressive conversion process from time TR (1). In parallel with the interlace / progressive conversion process, the writing unit 192 writes the converted image data in the A frame memory 193. In FIG. 10, a portion represented by a strip of processing is a period during which interlace / progressive conversion processing and writing are performed.

  When the image data reading / interpolating unit 182A (or 182B or 183C) finishes the interlace / progressive conversion process, the timing control unit 183, as will be described next, uses diff (2) and T-Out ( Perform the calculation of 2). In FIG. 10, the strip portion of the calculation is a period for performing this calculation and the like. First, the interlace / progressive conversion process and the write end time are compared with a time T-Out (1) delayed by a predetermined offset time ΔT from the time T-In (1). In this example, since the former is slower than the latter, it is determined that “delay” has occurred. In this example, the “delay” is equal to or greater than a predetermined value. In this case, the timing control unit 183 sets Diff (2) = Diff (1) + α1. Here, Diff (1) is an initial value of Diff (i). diff (i) is a time difference from T-Out (i) to T-Out (i + 1). Therefore, T−Out (i + 1) = T−Out (i) + diff (i). In addition, the time when the timing control unit 183 outputs a write request to the writing unit 192 for the i-th time is always a time preceding the time T-Out (i) by the offset time ΔT. Accordingly, the timing control unit 183 outputs the write request to the writing unit 192 at the i-th time, and the timing control unit 183 outputs the write request to the writing unit 192 at the i-th time. The time is delayed by diff (i) from the time to be performed.

  Since the timing at which the timing control unit 183 outputs the write request to the writing unit 192 for the first time is T-In (1), the timing control unit 183 receives the writing unit 192 for the second time. On the other hand, the time when the write request is output is T-In (1) + diff (1). At the time when the timing control unit 183 outputs a write request to the writing unit 192 for the second time, the image data has already been written in the A frame memory 193 that is the writing frame memory (A frame memory Since the image data written in 183 has not yet been read out), the time T-R when the reading unit 195 completes the reading of the image data from the B frame memory 194 and the B frame memory 194 is freed. Waiting until (2) occurs, and the writing unit 192 returns an affirmative response (ACK) to the timing control unit 183 at time TR (2). Then, from time TR (2), the image data reading / interpolating unit 182A (or 182B or 183C) starts the process of interlace / progressive conversion. In parallel with the interlace / progressive conversion process, the writing unit 192 writes the converted image data in the B frame memory 193.

  When the image data reading / interpolating unit 182A (or 182B or 183C) finishes the interlace / progressive conversion process, the timing control unit 183 compares the end time with the time T-Out (2). . However, since T−Out (i + 1) = T−Out (i) + diff (i) as described above, time T−Out (2) = T−out (1) + diff (1). In this example, since the end time of the interlace / progressive conversion process is later than the time T-Out (2), it is determined that a “delay” has occurred. In this example, the “delay” is less than a predetermined value. In this case, the timing control unit 183 sets Diff (3) = Diff (2).

  The timing at which the timing control unit 183 outputs a write request to the writing unit 192 for the third time is T-In (1) + diff (1) + diff (2). At the time when the timing control unit 183 outputs a write request to the writing unit 192 for the third time, image data has already been written in the B frame memory 194 which is a writing frame memory. Waits until time TR (3) when the A frame memory 194 is freed after the reading process is completed, and at time TR (3), the writing unit 192 acknowledges the timing control unit 183 with an acknowledgment ( ACK). Then, from time TR (3), the image data reading / interpolating unit 182A (or 182B or 183C) starts processing for interlace / progressive conversion. In parallel with the interlace / progressive conversion process, the writing unit 192 writes the converted image data in the A frame memory 193.

  When the image data reading / interpolating unit 182A (or 182B or 183C) finishes the interlace / progressive conversion process, the timing control unit 183 compares the end time with the time T-Out (3). . However, since T−Out (i + 1) = T−Out (i) + diff (i) as described above, time T−Out (3) = T−out (2) + diff (2). In this example, since the end time of the interlace / progressive conversion process is earlier than the time T-Out (3), it is determined that “advance” has occurred. In this example, the “advance” is equal to or greater than a predetermined value. In this case, the timing control unit 183 sets Diff (4) = Diff (3) −α2.

  The timing at which the timing control unit 183 outputs a write request to the writing unit 192 for the fourth time is T-In (1) + diff (1) + diff (2) + diff (3). At the time when the timing control unit 183 outputs a write request to the writing unit 192 for the fourth time, the B frame memory 194 that is the writing frame memory is free, so no waiting occurs, and the time Immediately after, the writing unit 192 returns an affirmative response (ACK) to the timing control unit 183. The image data reading / interpolating unit 182A (or 182B or 183C) starts the interlace / progressive conversion process immediately after the acknowledgment (ACK) is returned. In parallel with the interlace / progressive conversion process, the writing unit 192 writes the converted image data in the A frame memory 193.

  When the image data reading / interpolating unit 182A (or 182B or 183C) finishes the interlace / progressive conversion process, the timing control unit 183 compares the end time with the time T-Out (4). . However, since T−Out (i + 1) = T−Out (i) + diff (i) as described above, time T−Out (4) = T−out (3) + diff (3). In this example, since the end time of the interlace / progressive conversion processing is earlier than the time T-Out (4), it is determined that “advance” has occurred. In this example, the “advance” is equal to or greater than a predetermined value. In this case, the timing control unit 183 sets Diff (5) = Diff (4) −α2.

  Hereinafter, the same thing is repeated. In this way, the diff value is updated depending on whether it is advanced or delayed, and the next write request output time is the time obtained by adding diff to the current write request output time, and the next interlace / The time when the progressive conversion process is completed is compared with the time obtained by adding diff to the current time T-out, and updated again depending on whether the value of diff is advanced or delayed. As a result, interlace / progressive conversion processing and writing of image data to the free frame memory are performed during a period when either the A frame memory 193 or the B frame memory is free. In the convergence state, the value of diff vibrates as shown in FIG.

  In addition, when the operating frequency of the renderer is slow, the cycle of the write time T-In may be shorter than the cycle of the read time T-Out. In such a case, when the time from the last time T-In written in the write time memory 181 to the current time T-Out is equal to or longer than a predetermined time, the timing control unit 183 -Discard the image data written in In. This is shown in FIG.

  Although the renderer 191 and the timing control unit 183 shown in FIG. 9 can be realized by hardware, the computer reads and executes a program for causing the computer to function as the renderer 191 and the timing control unit 183 shown in FIG. This can also be realized.

[Embodiment 6]
In the sixth embodiment, about 60 fps interlaced image data created from 24 fps film image data is displayed as the original 24 fps film image data.

  Referring to FIG. 12, first, it is determined whether the image data of two fields before and the image data of the current field are the same (step S221). For this determination, for example, the absolute value of the difference for each pixel is obtained, and it is determined whether or not the absolute value is below a threshold value for all or some of the pixels in the field.

  If it is determined that they are the same (no change) (“no change” in step S221), it is determined whether or not the variable “pattern count” is zero (step S222). "Pattern count" changes as 5, 4, 3, 2, 1, 5, 4, 3, 2, 1, 5 ... during the period when image data created from a film image is displayed The period during which an image not created from a film image is displayed is zero.

  If the “pattern count” is zero (“= 0” in step S222), the “pattern count” is initialized to 5, and the variable “24 fps possibility” is incremented by 1 (step S224). ). By initializing “pattern count” to 5 in step S224, the phase of “pattern count” is initialized. “Possibility of 24 fps” indicates the possibility that the current field is obtained from a film image, and the larger this value, the higher the possibility.

  If the “pattern count” is not zero (“≠ 0” in step S222), step S224 is skipped. This occurs when the image is a still image or when it is determined that there is a change but there is no change by mistake.

  If it is determined that the image data of two fields before and the image data of the current field are not the same (changed) (“changed” in step S221), it is determined whether or not “pattern count” is zero. Judgment is made (step S223).

  If the “pattern count” is not zero (“≠ 0” in step S223), the “24 fps possibility” is increased by 1 (step S226). However, the upper limit value of “24 fps possibility” is 11 as an example. Therefore, when the value is increased by 1 to 12, the value is 11. Of the fields obtained from the film image, when a non-overlapping field is being processed, step S226 is executed.

  When the “pattern count” is zero (“= 0” in step S223), the “24 fps possibility” is decreased by 1, and when the “24 fps possibility” is 10 or more, “ The “pattern count” is initialized to 5 (step S225). However, when the “24 fps possibility” is decreased by 1, the lower limit value of the “24 fps possibility” is set to zero as an example. Therefore, when it is decreased by 1 and becomes -1, it is set to zero. If an overlapping field among the fields obtained from the film image is processed, but it is erroneously determined as “changed” in step S221, step S225 is executed. Also, when processing a field that is not obtained from a film image, step S225 is executed.

  If “≠ 0” in step S224, the process proceeds to step S227. The process also proceeds from step S224, S225, or S226 to step S227.

  In step S227, it is determined whether or not the “pattern count” is zero. “Pattern count” being zero means that the current field is not obtained from a film image. In such a case (“= 0” in step S227), the interlace / Progressive conversion processing is performed, and the resulting image is displayed (step S231). For the interlace / progressive conversion processing, Embodiments 1 to 4 are used.

  If the “pattern count” is not zero (“≠ 0” in step S227), the “pattern count” is decreased by 1 (step S228). As a result, when the fields obtained from the film image are continuous, the “pattern count” is decreased by 1 for each field.

  Next, it is determined whether or not the “24 fps possibility” is 5 or more (step S229).

  If the “24 fps possibility” is less than 5 (“<5” in step S229), the current field may not have been obtained from a film image even if the “pattern count” is not zero. Is high, interlace / progressive conversion processing is performed, and an image obtained thereby is displayed (step S231).

  If “24 fps possibility” is 5 or more (“> = 5” in step S229), it is determined whether “pattern count” is 2 or 0 (step S230).

  When the “pattern count” is 2 or 0, the field after 2 fields or 4 fields from the field in which the image data of 2 fields before and the image data of the current field are determined to be the same is currently handled. Means that Such a field is the latter half of the two fields obtained from the film image, except for duplicate fields. Therefore, in such a case (“= 2 or = 0” in step S230), a frame is created by combining the previous field and the current field, and this frame is displayed (step S232). In order to create a frame by combining the previous field and the current field, the interlace / progressive conversion processing apparatus performs processing after treating all areas of the current frame as still image areas. Just do it.

  If the “pattern count” is neither 2 nor 0, the image data of the last frame is displayed again as it is (step S233).

  If “= 0” in step S227 and “<5” in step S229, interlace / progressive conversion processing is performed and the resulting frame is displayed (step S231).

  The operation when there is no determination error in step S221 is shown in FIGS. FIG. 15 shows the operation when the first type determination error in step S221 occurs once after the processing shown in FIGS. 13 and 14 is performed. Further, FIG. 16 shows an operation when the second type determination error occurs three times in step S221 after the processing shown in FIGS. 13 and 14 is performed. As shown in FIGS. 15 and 16, it can be seen that the correct process is finally selected even if a determination error occurs in step S221.

  The display method shown in FIG. 12 can be realized by hardware, but can also be realized by the computer reading and executing a program for causing the computer to perform the display method shown in FIG.

101-1 to 101-3 Field memory 103 Image data writing unit 105 Moving image interpolation pixel generation unit 107 Still image interpolation pixel generation unit 109 Difference calculation unit 111 Difference correction unit 113 Correction value holding unit 115 Difference accumulation unit 117 Accumulated motion amount Memory 119 Threshold value holding unit 121 Motion amount calculation unit 123 Synthetic interpolation pixel generation unit 125 Selection unit

Claims (7)

Using two frame memories that are alternately and complementarily a write frame memory and a display frame memory,
A timing control unit outputting a write request to the writing unit;
In the case where image data that has not yet been read by the reading unit remains in the current writing frame memory, the writing unit has finished reading image data from the display frame memory by the reading unit, When the timing control unit returns an affirmative response to the write request and there is no image data that has not yet been read out by the reading unit in the current write frame memory, a delay occurs after the write request is input. And returning a positive response to the write request to the timing control unit,
When the timing control unit inputs the acknowledgment from the writing unit, the image processing unit performs image processing, and the writing unit stores the image data obtained by the image processing in a writing frame memory. Writing step;
The timing control unit compares a time when the image processing and writing are completed with a reference time, and updates an interval parameter according to a result of the comparison;
The timing control unit sets a time obtained by adding the interval parameter to the reference time as a next reference time;
The timing control unit outputs the write request to the writing unit again at the time when the interval parameter is added from the time when the write request was previously output to the writing unit, and the above operation Repeating steps,
A rendering method comprising:   The rendering method according to claim 1, wherein the image processing is an interless / progressive conversion processing. The rendering method according to claim 2, wherein
The interlace / progressive conversion process is:
In a method for converting interlaced image data into progressive image data,
Calculating a first amount of motion for a pixel to be interpolated for the field of interest;
Calculating a second amount of motion for pixels in the vertical direction from pixels to be interpolated in the target field among the pixels of the field adjacent to the target field;
Calculating a first cumulative motion amount based on the first motion amount;
Calculating a second cumulative motion amount based on the second motion amount;
Calculating a third motion amount based on the first cumulative motion amount and the second cumulative motion amount;
Calculating a value of a pixel to be interpolated based on the third amount of motion;
A rendering method comprising: The rendering method according to claim 1, wherein
Comparing the time when the image processing and writing are completed with a reference time, and updating the interval parameter according to the result of the comparison,
If the reference time is a predetermined value or more earlier than the time when the writing is completed, increasing the interval parameter;
If the reference time is a predetermined value or more later than the time when the writing is completed, decreasing the interval parameter;
Otherwise, a step that does not change the spacing parameter;
A rendering method comprising: The rendering method according to claim 1, wherein
A step of discarding the image data written in the buffer memory if the reference time is later than the time when the image data to be processed next is written in the buffer memory by a predetermined time or more; A rendering method, further comprising: Two frame memories alternately and complementarily as a write frame memory and a display frame memory;
Means for outputting a write request to the writing unit by the timing control unit;
In the case where image data that has not yet been read by the reading unit remains in the current writing frame memory, the writing unit has finished reading image data from the display frame memory by the reading unit, When the timing control unit returns an affirmative response to the write request and there is no image data that has not yet been read out by the reading unit in the current write frame memory, a delay occurs after the write request is input. And means for returning an affirmative response to the write request to the timing control unit,
When the timing control unit inputs the acknowledgment from the writing unit, the image processing unit performs image processing, and the writing unit stores the image data obtained by the image processing in a writing frame memory. Means for writing;
Means for comparing the time when the image processing and writing are completed with a reference time, and updating an interval parameter according to a result of the comparison;
Means for setting the timing control unit as a next reference time a time obtained by adding the interval parameter to the reference time;
The timing control unit is configured to output the write request to the writing unit again at the time when the interval parameter is added from the time when the write request was previously output to the writing unit. Means to repeat,
A rendering device comprising: Two frame memories alternately and complementarily as a write frame memory and a display frame memory;
Means for outputting a write request to the writing unit by the timing control unit;
In the case where image data that has not yet been read by the reading unit remains in the current writing frame memory, the writing unit has finished reading image data from the display frame memory by the reading unit, When the timing control unit returns an affirmative response to the write request and there is no image data that has not yet been read out by the reading unit in the current write frame memory, a delay occurs after the write request is input. And means for returning an affirmative response to the write request to the timing control unit,
When the timing control unit inputs the acknowledgment from the writing unit, the image processing unit performs image processing, and the writing unit stores the image data obtained by the image processing in a writing frame memory. Means for writing;
Means for comparing the time when the image processing and writing are completed with a reference time, and updating an interval parameter according to a result of the comparison;
Means for setting the timing control unit as a next reference time a time obtained by adding the interval parameter to the reference time;
The timing control unit is configured to output the write request to the writing unit again at the time when the interval parameter is added from the time when the write request was previously output to the writing unit. Means to repeat,
A program for causing a computer to function as a rendering device.

Images