JP2007166530A - Frame frequency conversion method, frame frequency converter and frame frequency conversion program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To execute a frame frequency conversion method without losing the smoothness of moving images and without increasing an operation amount. <P>SOLUTION: When it is judged that either deframing or frame insertion is required in a certain input frame, the deframing processing or frame insertion processing is executed over a plurality of frames then, and the time to suspend the deframing processing or the frame insertion processing is provided thereafter. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a frame frequency conversion method, a frame frequency conversion device, and a frame frequency conversion program for generating an output moving image having a second frame frequency based on an input moving image having a first frame frequency.

  In recent years, a personal computer has a built-in or connected TV tuner, and the personal computer can display, record, and play back TV broadcasts received by the TV tuner. ing.

  In Japan and several other countries, the NTSC system is adopted for television broadcasting. In the NTSC system, the field frequency is about 59.94 Hz (= 60 × (1000/1001) Hz) instead of 60 Hz in order to adjust the frequency with the color subcarrier frequency. Therefore, the frame frequency is not about 30 Hz but about 29.97 Hz (= 30 × (1000/1001) Hz). The frame frequency of the moving image obtained by performing IP (Interlace / Progressive) conversion processing on the signal output from the NTSC tuner is 59.94 Hz. Here, the IP conversion process is a process of converting an interlaced moving image into a progressive moving image.

  On the other hand, a typical refresh rate (corresponding to a frame frequency) of a television monitor of a personal computer is just 60 Hz. Many personal computers can also set other refresh rates.

  Then, since the frame frequency of the television image to be displayed is different from the refresh rate of the television monitor, the personal computer needs to perform processing for solving this difference.

Conventionally, as a process for this, some frames are thinned out or displayed twice (for example, see Patent Document 1).
JP 2004-80327 A

  However, when the frame frequency is adjusted by thinning out some frames or displaying twice, the smoothness of the moving image is lost before and after the thinning and at the twice-displayed portion, which is unnatural. .

  In order to solve this problem, if frame frequency conversion is performed over all frames, the amount of calculation increases. For example, if this is performed on a multitasking operating system, the time allocated to other tasks, The computer resources are reduced, and the response time to the user in other tasks becomes longer.

  Therefore, the present invention generates an output moving image of the second frame frequency based on the input moving image of the first frame frequency without losing the smoothness of the moving image, and has a frame frequency with a small amount of calculation for that purpose. It is an object to provide a conversion method, a frame frequency conversion device, and a frame frequency conversion program.

  According to the present invention, in a frame frequency conversion method for generating an output moving image having a second frame frequency based on an input moving image having a first frame frequency, a frame dropping process is performed for each frame of the input moving image. A start determination step for determining whether or not it is necessary to start a frame insertion process; and when the start determination step determines that the frame dropping process or the frame insertion process needs to be started, the input moving image A process execution step for performing the frame dropping process or the frame inserting process over a plurality of frames of an image, and the start determination being performed continuously when the frame dropping process or the frame inserting process is completed. In the step, until it is determined that the start of the frame dropping process or the frame inserting process is necessary again, the frame dropping process or the frame inserting process is performed. Frame frequency conversion method characterized by pause is provided.

  In the frame frequency conversion method, the start determination step needs to start the frame dropping process or the frame insertion process based on a time difference between the frame synchronization signal of the input moving image and the frame signal of the output moving image. It may be determined whether or not.

  In the frame frequency conversion method, in the start determination step, the data of the input moving image is written therein according to the first frame frequency, and the data of the output moving image is stored there according to the second frame frequency. When the amount of moving image data temporarily stored in the display memory read from the above becomes a predetermined value or more, the time for waiting for the input moving image data to be written to the display memory becomes a predetermined value or more. In this case, it may be determined that the frame dropping process needs to be started.

  In the frame frequency conversion method, the frame dropping process or the frame inserting process over a plurality of frames of the input moving image may include a weight variable, the frame dropping process or the frame inserting in a certain frame of the input moving image. Initializing to a value corresponding to the number of frames to be processed; changing the weight variable for each frame of the input video; and the weight variable changed for each frame of the input video. And generating a frame signal of an output moving image by weighting and adding a plurality of frame signals of the input moving image.

  In the frame frequency conversion method described above, when it is determined in the start determination step that the start of the frame drop process or the frame insert process is necessary, the start frame of the frame drop process or the frame insert process is a moving image. A still image / moving image determining step for determining whether the frame is a still image frame or a still image frame, and the start frame of the frame dropping process or the frame inserting process is determined to be a moving image frame in the still image / moving image determining step. The frame dropping process or the frame inserting process over a plurality of frames of the input moving image is performed only when the frame dropping process or the frame inserting process is started in the still image / video determination step. If it is determined that there is another frame drop process or another frame insertion process, Good.

  In the frame frequency conversion method, the other frame dropping process or the other frame inserting process is a process of discarding a certain frame of the input moving image or a process of overlapping a certain frame of the input moving image, respectively. You may do it.

  In the frame frequency conversion method, the start determination step may include the frame over the plurality of frames of the input moving image according to the interval of the input frames determined to require the start of the frame drop process or the frame insertion process. You may make it further provide the parameter adjustment step which adjusts the parameter in a frame drop process or the said frame insertion process.

  In the frame frequency conversion method, in the parameter adjustment step, the number of frames to be subjected to the frame dropping process or the frame inserting process over a plurality of frames of the input moving image may be adjusted.

  In the frame frequency conversion method, in the parameter adjustment step, the number of frames to be subjected to the frame dropping process or the frame inserting process over a plurality of frames of the input moving image is determined by the frame dropping process or the frame inserting process. You may make it adjust so that it may become below the space | interval of the input frame judged that the start is required.

  In the frame frequency conversion method, in the parameter adjustment step, the number of frames to be subjected to the frame dropping process or the frame inserting process over a plurality of frames of the input moving image is determined by the frame dropping process or the frame inserting process. An adjustment may be made so that the interval is less than the interval between input frames that are determined to need to be started, thereby providing a period for pausing the frame dropping process or the frame inserting process.

  According to the present invention, in a frame frequency conversion method for generating an output moving image having a second frame frequency based on an input moving image having a first frame frequency, a frame dropping process is performed for each frame of the input moving image. A start determination step for determining whether or not it is necessary to start a frame insertion process; and when the start determination step determines that the frame dropping process or the frame insertion process needs to be started, the input moving image A process execution step for performing the frame dropping process or the frame inserting process over a plurality of frames of an image, and the start determination being performed continuously when the frame dropping process or the frame inserting process is completed. In the step, until it is determined that the start of the frame dropping process or the frame inserting process is necessary again, the frame dropping process or the frame inserting process is performed. To pause.

  Accordingly, the smoothness of the moving image can be prevented from being lost by the frame dropping process or the frame inserting process over a plurality of frames, and conversion can be performed by providing a period for pausing the frame dropping process or the frame inserting process. Can reduce the amount of computation.

  The best mode for carrying out the present invention will be described below in detail with reference to the drawings.

[Embodiment 1]
FIG. 1 is a block diagram showing a configuration of a system including a frame frequency conversion apparatus according to Embodiment 1 of the present invention.

  Referring to FIG. 1, this system includes an NTSC decoder 101, an image data writing unit 103, a frame buffer group 105, an output processing unit 107, a display memory 125, a monitor 127, and a parameter specifying unit 129.

  The frame buffer group 105 includes a first frame buffer 105-1, a second frame buffer 105-2, and a third frame buffer 105-3.

  The output processing unit 107 includes an image data reading unit 109, an IP conversion processing unit 111, a frame drop / frame insertion determination unit 115, a composition processing unit 117, a temporary storage memory 119, a format conversion unit 121, and a display processing unit 123.

  The NTSC decoder 101 decodes the input NTSC signal and outputs a baseband moving image signal.

  The image data writing unit 103 writes each frame of the moving image signal input from the NTSC decoder 101 to each frame buffer 105-1 to 105-3 of the frame buffer group 105. For example, the image data writing unit 103 treats the first frame buffer 105-1, the second frame buffer 105-2, and the third frame buffer 105-3 like a ring buffer and cyclically handles them. Each frame of the moving image signal may be written into the buffer. Further, the number of frame buffers is not limited to three, and may be larger.

  The output processing unit 107 performs output processing for each field of the NTSC signal, for example, using each field vertical synchronization signal of the NTSC signal output from the NTSC decoder 101 as a trigger. The output processing unit 107 can be realized by hardware, but can also be realized by a computer reading and executing a program for causing the computer to function as the output processing unit 107.

  The output processing unit 107 writes the moving image signal obtained as a result of the output processing in the display memory 125.

  The moving image signal written in the display memory 125 is read using the vertical synchronizing signal (for example, frame refresh signal) of the monitor 127 as a vertical synchronizing signal, and the moving image based on this is displayed on the monitor 127.

  The respective units of the output processing unit 107 will be described. The image data reading unit 109 reads out each frame of the input moving image signal from each of the frame buffers 105-1 to 105-3 of the frame buffer group 105. For example, the image data reading unit 109 treats the first frame buffer 105-1, the second frame buffer 105-2, and the third frame buffer 105-3 like a ring buffer, and cyclically selects these. Each frame of the input moving image signal may be read from the buffer.

  The IP conversion processing unit 111 performs IP conversion processing. The parameter specifying unit 129 specifies a parameter used in the IP conversion process by the IP conversion processing unit 111. For example, when the IP conversion processing unit 111 has a configuration as shown in FIG. 19, a threshold value to be stored in the threshold value holding unit 519 is designated.

  The frame drop / frame insertion determination unit 115 determines whether frame drop or frame insertion is necessary in relation to the current field of the input moving image (this corresponds to the frame after IP conversion). To do.

  The composition processing unit 117 synthesizes the output moving image based on the progressive input moving image input from the IP conversion processing unit 111. Specifically, the synthesis processing unit 117 synthesizes each output frame based on one or more input frames. Thereby, a frame drop process or a frame insertion process over a plurality of frames is realized.

  The temporary storage memory 119 is used by the synthesis processing unit 117 for performing the synthesis process, and temporarily stores one frame included in the progressive input moving image.

  The format conversion unit 121 converts the format of the progressive output moving image input from the synthesis processing unit 117. For example, the format conversion unit 121 converts the array of YUV data from the output of the IP conversion processing unit 111 to one suitable for a graphic board including the display memory 125.

  The display processing unit 123 writes the format-converted output moving image input from the format conversion unit 121 in the display memory 125.

  Next, output processing performed by the output processing unit 107 will be described. In the output processing unit 107, the frame frequency of the monitor 127 (hereinafter referred to as “output frame frequency”) is based on the field frequency of the input NTSC signal (input frame frequency after IP conversion, hereinafter referred to as “input frame frequency”). If the output frame frequency is higher than the input frame frequency, the frame drop process is performed over a plurality of frames to absorb the difference between the input frame frequency and the output frame frequency. By performing the frame insertion process over a period of time, the difference between the input frame frequency and the output frame frequency is absorbed.

  First, output processing performed by the output processing unit 107 when the output frame frequency is lower than the input frame frequency will be described with reference to FIG.

  First, the IP conversion processing unit 111 performs IP conversion processing (step S201). Details of the IP conversion process will be described later.

  Next, the frame drop / frame insertion determination unit 115, for example, a field synchronization signal input from the NTSC decoder 101 (which has an input frame frequency) and a refresh signal input from the monitor 127 (which is an output frame frequency). Based on the above, it is determined whether or not the frame drop start process needs to be opened (step S203).

  If the frame drop / frame insertion determination unit 115 determines that it is necessary to perform frame drop start processing (YES in step S203), the composition processing unit 117 temporarily stores the current input frame obtained in step S201. The data is stored in the storage memory 119 (step S205), and a variable counter is initialized to 30 (step S207). Here, the initial value 30 is an example, and another value may be used as the initial value. This initial value corresponds to the number of frames for which the frame dropping process is continued. For example, the initial value may be changed according to the load status of the computer. If the load is high, the amount of frame drop processing can be reduced by reducing the initial value. Conversely, if the load is low, the initial value can be increased to reduce the number of frames for which frame drop processing is performed. The image quality can be increased and the image quality can be improved. If the frame drop / frame insertion determination unit 115 determines that it is necessary to start the frame drop process (YES in step S203), the format conversion process (step S217) and the display process (step S219) are performed in the input frame. ) Is omitted, the output frame can be reduced by one.

  When the frame drop / frame insertion determination unit 115 determines that it is not necessary to perform the frame drop start process (NO in step S203), the composition processing unit 117 determines whether or not the counter value exceeds zero. (Step S209).

  If the value of the counter exceeds zero (YES in step S209), this means that the current frame belongs to a plurality of frames to be subjected to the frame dropping process. The process proceeds to S211.

  On the other hand, when the value of the counter does not exceed zero (NO in step S209), this means that the current frame does not belong to a plurality of frames to be subjected to the frame dropping process (that is, the previous frame dropping process has already ended). In order to omit the frame dropping process, the process proceeds to step S221.

  In step S211, the synthesis processing unit 117 synthesizes an output frame based on the input frame and the temporary storage frame. The input frame is an input frame generated by the IP conversion process (step S201) in the current output process, and in short, is the current input frame. The temporary storage frame is an input frame generated in the IP conversion process (step S201) in the output process prior to the current output process and temporarily stored in the temporary storage memory 119 in step S205 or S213. Simply put, it is the previous input frame.

  In the synthesis, for example, each pixel y (m, i, j) in the i-th row and j-th column of the output frame m is calculated according to the following equation.

y (m, i, j) = w × x (n−1, i, j) + (1−w) × x (n, i, j)
Here, w represents a weight, x (n−1, i, j) represents a pixel in the i-th row and j-th column of the storage frame, and x (n, i, j) represents the input frame. Represents a pixel in the i-th row and j-th column. Moreover, w is calculated according to the following formula.

w = counter / 30
However, the denominator 30 is the initialization in step S207. If this initial value is another value, the denominator also changes to that value.

  After step S211, the composition processing unit 117 stores the current input frame obtained in step S201 in the temporary storage memory 119 (step S213), and decrements the counter by 1 (step S215).

  On the other hand, in step S221, the composition processing unit 117 sets the input frame itself as an output frame.

  Following step S215 or step S221, the format conversion processing unit 121 performs format conversion processing (step S217), and the display processing unit 123 performs display processing (step S219).

  FIG. 3 shows an operation example.

  Referring to FIG. 3, it is determined that frame drop processing is necessary in the 40th input frame. Steps S201, S205, and S207 are executed, and an output frame corresponding to only the 40th input frame is not generated.

  In the 41st to 70th input frames, the frame dropping process is continuously performed. Step S201, Step S211, Step S213, Step S215, Step S217, and Step S219 are executed for each input frame. As shown in FIG. 3, the counter decreases by 1 for each input frame. Further, the weight decreases linearly from 1 to 0.03 as shown in FIG. The output frame column in FIG. 3 indicates the position of the output frame by the frame number of the input frame. Also, the difference column in FIG. 3 shows the difference between each output frame and the previous output frame.

  If frames are dropped by a method that simply drops one frame as in the conventional example, the interval between the output frames is 2 in the frame that is determined to require frame dropping, and the output moving image skips. It becomes discontinuous.

  On the other hand, in the present embodiment, the interval between output frames does not become 2 in a frame that is determined to require frame dropping. Further, in the period when frame dropping is performed, the difference between each output frame and the previous output frame is 1.03. This value is not significantly different from the normal difference 1. Therefore, in the present embodiment, it is possible to avoid losing the smoothness of the moving image even if the frame dropping process is performed. Further, the period for performing the frame dropping process is only the period corresponding to the number of frames corresponding to the value set in the counter in step S207, and after that period, the frame dropping process is required again in step S203. Since the frame dropping process can be paused until a frame determined to be present arrives, the amount of calculation can be reduced. Therefore, for example, even if a large amount of computation is required for the frame dropping process, it is not necessary during the period when the frame dropping process is paused. Can be transferred to other tasks.

  Next, output processing performed by the output processing unit 107 when the output frame frequency is higher than the input frame frequency will be described with reference to FIG.

  First, the IP conversion processing unit 111 performs IP conversion processing (step S251). Details of the IP conversion process will be described later.

  Next, the frame drop / frame insertion determination unit 115, for example, a field synchronization signal input from the NTSC decoder 101 (which has an input frame frequency) and a refresh signal input from the monitor 127 (which is an output frame frequency). Based on the above, it is determined whether or not the frame insertion start process needs to be performed (step S253).

  If the frame drop / frame insertion determination unit 115 determines that it is necessary to perform frame insertion start processing (YES in step S253), the composition processing unit 117 temporarily stores the current input frame obtained in step S251. The data is stored in the storage memory 119 (step S255), and a variable counter is initialized to 30 (step S257). Here, the initial value 30 is an example, and another value may be used as the initial value. From step S257, the process proceeds to step S261.

  When the frame drop / frame insertion determination unit 115 determines that it is not necessary to perform the frame insertion start process (NO in step S253), the composition processing unit 117 determines whether the counter value exceeds zero. (Step S259).

  When the value of the counter exceeds zero (YES in step S259), this means that the current frame belongs to a plurality of frames to be subjected to frame insertion processing. The process proceeds to S261.

  On the other hand, when the value of the counter does not exceed zero (NO in step S259), this means that the current frame does not belong to a plurality of frames to be subjected to frame insertion processing (that is, the previous frame insertion processing has already been performed). In order to omit the frame insertion process, the process proceeds to step S281.

  In step S261, the composition processing unit 117 synthesizes an output frame based on the input frame and the temporarily saved frame. The input frame is an input frame generated by the IP conversion process (step S251) in the current output process, and is simply the current input frame. The temporary storage frame is an input frame generated in the IP conversion process (step S251) in the output process before the current output process and temporarily stored in the temporary storage memory 119 in step S255 or S263. Simply put, it is the previous input frame.

  In the synthesis, for example, each pixel y (m, i, j) in the i-th row and j-th column of the output frame m is calculated according to the following equation.

y (m, i, j) = (1−w) × x (n−1, i, j) + w × x (n, i, j)
Here, w represents a weight, x (n−1, i, j) represents a pixel in the i-th row and j-th column of the storage frame, and x (n, i, j) represents the input frame. Represents a pixel in the i-th row and j-th column. Compared to the case of dropping frames, the weights are reversed. Moreover, w is calculated according to the following formula.

w = counter / 30
After step S261, the composition processing unit 117 stores the current input frame obtained in step S251 in the temporary storage memory 119 (step S263), and decrements the counter by 1 (step S265).

  On the other hand, in step S281, the composition processing unit 117 sets the input frame itself as an output frame.

  Following step S265 or step S281, the format conversion processing unit 121 performs format conversion processing (step S267), and the display processing unit 123 performs display processing (step S279).

  Next, the composition processing unit 117 determines whether or not the counter value is zero (step S271).

  If it is determined that the value of the counter is zero (YES in step S271), the counter is set to a value less than zero (eg, “−1”) (step S272). In this way, YES is not determined in step S271 until the next frame dropping process is completed. Next, the composition processing unit 117 uses the input frame itself as an output frame (step S273), the format conversion processing unit 121 performs format conversion processing (step S275), and the display processing unit 123 performs display processing (step S275). S279).

  If the frame drop / frame insertion determination unit 115 determines that it is necessary to start the frame insertion process (YES in step S253), the output frame synthesized in step S261 in the input frame is subjected to the format conversion process ( Since it is output by the step S217) and the display process (step S219), it is not reduced by one frame unlike the frame drop process. If it is determined that the counter value is zero, in addition to the output synthesis process (step S261), the format conversion process (step S267), and the display process (step S269), an output frame generation process (step S273), Since the format conversion process (step S275) and the display process (step S279) are executed, the output frame can be increased by one.

  FIG. 5 shows an operation example.

  Referring to FIG. 5, it is determined that frame insertion processing is necessary in the 40th input frame. Step S251, Step S255, Step S257, Step S261, Step S263, Step S265, Step S267, Step S269, and Step S271 are executed, and one output frame is generated.

  In the 41st to 69th input frames, the frame insertion process is continuously performed. Step S251, Step S255, Step S261, Step S263, Step S265, Step S267, Step S269 and Step S271 are executed for each input frame. As shown in FIG. 5, the counter decreases by 1 for each input frame. Further, the weight decreases linearly from 1 to 0.03 as shown in FIG. The output frame column in FIG. 5 indicates the position of the output frame by the frame number of the input frame. Also, the difference column in FIG. 5 shows the difference between each output frame and the previous output frame.

  If frames are dropped by a method of simply repeating one frame twice, the output frame interval becomes zero in a frame in which it is determined that frame insertion is necessary, and the output moving image becomes stagnant and discontinuous. turn into.

  On the other hand, in this embodiment, the output frame interval does not become zero in a frame that is determined to require frame insertion. Also, during the period in which frame insertion is performed, the difference between each output frame and the previous output frame is 0.97. This value is not significantly different from the normal difference 1. Therefore, in this embodiment, it is possible to avoid losing the smoothness of the moving image even if the frame insertion process is performed. Further, the period for performing the frame insertion process is only the period corresponding to the number of frames corresponding to the value set in the counter in step S257, and after that period, the frame insertion process is required again in step S253. Since the frame insertion process can be paused until a frame determined to be present arrives, the amount of calculation can be reduced. Therefore, for example, even if a large amount of computation is required for the frame insertion process, this is not necessary during the period when the frame insertion process is paused. Can be transferred to other tasks.

  The operation of the output processing unit 107 has been described separately for the case where the output frame frequency is lower than the input frame frequency and the case where the output frame frequency is higher. In practice, a case-by-case process is realized to potentially deal with any case. In practice, however, the refresh rate may vary depending on which monitor is used as the monitor 127. Once the monitor 127 and its operation mode are determined, the refresh rate of the monitor varies. However, even if it fluctuates, it is a long-term fluctuation, so in practice, the process described with reference to FIG. 2 or the like or the process described with reference to FIG.

[Embodiment 2]
In the first embodiment, after the IP conversion processing is performed by the IP conversion processing unit 111, the composition processing unit 117 in the subsequent stage performs a frame drop process or a frame insertion process as necessary.

  In contrast, in the second embodiment, two IP conversion processing units (that is, the first IP conversion processing unit 111B and the second IP conversion processing unit 111C) are provided, and the synthesis processing unit 117 is omitted.

  One IP conversion processing unit (that is, the first IP conversion processing unit 111B) performs normal IP conversion processing (that is, IP conversion processing that does not include frame drop processing and frame insertion processing), and the other as necessary. The IP conversion processing unit (that is, the second IP conversion processing unit 111C) performs an IP conversion process including a frame drop process or a frame insertion process. Accordingly, the first IP conversion processing unit 111B of the second embodiment is the same as the IP conversion processing unit 111 of the first embodiment.

  FIG. 6 is a block diagram showing a configuration of a system including a frame frequency conversion device according to Embodiment 2 of the present invention.

  Referring to FIG. 6, this system includes an NTSC decoder 101, an image data writing unit 103B, a frame buffer group 105B, an output processing unit 107B, a display memory 125, a monitor 127, and a parameter specifying unit 129.

  The frame buffer group 105B includes a first frame buffer 105-1, a second frame buffer 105-2, a third frame buffer 105-3, and a fourth frame buffer 105-4.

  The output processing unit 107B includes an image data reading unit 109B, a first switch 131, a first IP conversion processing unit 111B, a second IP conversion processing unit 111C, a frame drop / frame insertion determination unit 115, and a second switch 133. A format conversion unit 121, a display processing unit 123, and a control unit 135.

  Since the NTSC decoder 101, the display memory 125, the monitor 127, and the parameter designation unit 129 of the second embodiment are the same as those of the first embodiment, redundant description is omitted.

  The image data writing unit 103B writes each frame of the moving image signal input from the NTSC decoder 101 to each frame buffer 105-1 to 105-4 of the frame buffer group 105. For example, the image data writing unit 103B uses the first frame buffer 105-1, the second frame buffer 105-2, the third frame buffer 105-3, and the fourth frame buffer 105-4 as a ring buffer. Thus, each frame of the moving image signal may be written in these buffers in a circular manner. The number of frame buffers is not limited to four, and may be larger.

  The output processing unit 107B performs output processing for each field of the NTSC signal, for example, using each field vertical synchronization signal of the NTSC signal output from the NTSC decoder 101 as a trigger. The output processing unit 107B can be realized by hardware, but can also be realized by a computer reading and executing a program for causing the computer to function as the output processing unit 107B.

  The output processing unit 107B writes the moving image signal obtained as a result of the output processing in the display memory 125.

  Explaining each part of the output processing unit 107B, the image data reading unit 109B reads each frame of the input moving image signal from each frame buffer 105-1 to 105-4 of the frame buffer group 105. For example, the image data reading unit 109 makes the first frame buffer 105-1, the second frame buffer 105-2, the third frame buffer 105-3, and the fourth frame buffer 105-4 like a ring buffer. The frames of the input moving image signal may be read out from these buffers in a circular manner.

  The frame drop / frame insertion determination unit 115 determines whether frame drop or frame insertion is necessary in relation to the current field of the input moving image (this corresponds to the frame after IP conversion). To do.

  The control unit 135 determines which of the first IP conversion processing unit 111B and the second IP conversion processing unit 111C should perform the IP conversion processing based on the determination result by the frame drop / frame insertion determination unit 115. Judging.

  The first switch 131 and the second switch 133 switch the path of the image signal from the image data reading unit 109B to the format conversion unit 121 based on the determination result by the control unit 135.

  The format conversion unit 121 converts the format of the progressive output moving image input from the second switch 133 in the same manner as in the first embodiment.

  The display processing unit 123 writes the format-converted output moving image input from the format conversion unit 121 in the display memory 125.

  Next, output processing performed by the output processing unit 107B will be described. When the output frame frequency is lower than the input frame frequency, the output processing unit 107B absorbs a shift between the input frame frequency and the output frame frequency by performing a frame dropping process, and the output frame frequency is the input frame frequency. If it is higher, the frame insertion process is performed to absorb the difference between the input frame frequency and the output frame frequency.

  First, output processing performed by the output processing unit 107B when the output frame frequency is lower than the input frame frequency will be described with reference to FIG.

  First, the frame drop / frame insertion determination unit 115, for example, uses a field synchronization signal (which has an input frame frequency) input from the NTSC decoder 101 and a refresh signal (which has an output frame frequency input from the monitor 127). Based on the above, it is determined whether or not it is necessary to open the frame drop start process (step S301).

  If the frame drop / frame insertion determination unit 115 determines that it is necessary to perform frame drop start processing (YES in step S301), the control unit 135 initializes a counter that is a variable to 30 (step S305). ). Here, the initial value 30 is an example, and another value may be used as the initial value. If the frame drop / frame insertion determination unit 115 determines that it is necessary to start the frame drop process (YES in step S301), the format conversion process (step S313) and the display process (step S315) are performed in the input frame. ) Is omitted, the output frame can be reduced by one.

  When the frame drop / frame insertion determination unit 115 determines that it is not necessary to perform the frame drop start process (NO in step S301), the control unit 135 determines whether the counter value exceeds zero. (Step S307).

  If the value of the counter exceeds zero (YES in step S307), this means that the current frame belongs to a plurality of frames to be subjected to the frame drop process, so the image data reading unit 109B to the format conversion unit The first switch 131 and the second switch 133 are switched so that the second IP conversion processing unit 111C enters between 121 and 121, and the second IP conversion processing unit 111C performs IP conversion including frame drop processing. Processing is performed (step S309). Next, the control unit 135 decreases the counter by 1 (step S311).

In the IP conversion processing including the frame dropping process performed by the second IP conversion processing unit 111C, the weight w is set to w = counter / 30.
And frame synthesis is performed using this.

  On the other hand, when the value of the counter does not exceed zero (NO in step S307), this means that the current frame does not belong to a plurality of frames to be subjected to the frame dropping process (that is, the previous frame dropping process has already been performed). In order to omit the frame dropping process, the process proceeds to step S317.

  In step S317, the control unit 135 switches the first switch 131 and the second switch 133 so that the first IP conversion processing unit 111B is inserted between the image data reading unit 109B and the format conversion unit 121. The first IP conversion processing unit 111B performs normal IP conversion processing.

  Subsequent to step S317 or step S309, an input moving image obtained by normal IP conversion processing (step S317) performed by the first IP conversion processing unit 111B or frame dropping performed by the second IP conversion processing unit 111C The format conversion processing unit 121 performs format conversion processing on the input moving image obtained by the IP conversion processing including processing (step S309) (step S313), and the display processing unit 123 performs display processing (step S313). Step S315).

  The operation example of the frame dropping process according to the second embodiment and the effects described with reference thereto are the same as those illustrated in FIG. 3 according to the first embodiment and the effects described with reference thereto.

  Next, output processing performed by the output processing unit 107B when the output frame frequency is higher than the input frame frequency will be described with reference to FIG.

  First, the frame drop / frame insertion determination unit 115, for example, uses a field synchronization signal (which has an input frame frequency) input from the NTSC decoder 101 and a refresh signal (which has an output frame frequency input from the monitor 127). Based on the above, it is determined whether or not the frame insertion start process needs to be performed (step S351).

  If the frame drop / frame insertion determination unit 115 determines that it is necessary to perform the frame insertion start process (YES in step S351), the control unit 135 initializes a variable counter to 30 (step S355). ). Here, the initial value 30 is an example, and another value may be used as the initial value. From step S355, the process proceeds to step S359.

  When the frame drop / frame insertion determination unit 115 determines that it is not necessary to perform the frame insertion start process (NO in step S351), the control unit 135 determines whether the counter value exceeds zero. (Step S357).

  If the value of the counter exceeds zero (YES in step S357), this means that the current frame belongs to a plurality of frames to be subjected to frame insertion processing, and therefore IP conversion processing including frame insertion processing is performed. In step S359, the process proceeds to step S359.

  On the other hand, if the value of the counter does not exceed zero (NO in step S357), this means that the current frame does not belong to a plurality of frames to be subjected to frame insertion processing (that is, the previous frame insertion processing has already ended). In order to perform normal IP conversion processing and omit frame insertion processing, the process proceeds to step S379.

  In step S359, the control unit 135 switches the first switch 131 and the second switch 133 so that the second IP conversion processing unit 111C is inserted between the image data reading unit 109B and the format conversion unit 121. The second IP conversion processing unit 111C performs IP conversion processing including frame insertion processing.

In the IP conversion process including the frame insertion process performed by the second IP conversion processing unit 111C, the weight w is set to w = counter / 30.
And frame synthesis is performed using this.

  Following step S359, the control unit 135 decrements the counter by 1 (step S363).

  In step S379, the control unit 135 switches the first switch 131 and the second switch 133 so that the first IP conversion processing unit 111B is inserted between the image data reading unit 109B and the format conversion unit 121. The first IP conversion processing unit 111B performs normal IP conversion processing.

  Subsequent to step S379 or step S359, an input moving image obtained by a normal IP conversion process (step S379) performed by the first IP conversion processing unit 111B or a frame insertion performed by the second IP conversion processing unit 111C The format conversion processing unit 121 performs format conversion processing on the input moving image obtained by the IP conversion processing including processing (step S359) (step S365), and the display processing unit 123 performs display processing (step S365). Step S367).

  Next, the control unit 135 determines whether or not the counter value is zero (step S369).

  If it is determined that the value of the counter is zero (YES in step S369), the counter is set to a value less than zero (eg, “−1”) (step S370). By doing so, YES is not determined in step S369 until the end of the next frame dropping process. Next, the control unit 135 switches the first switch 131 and the second switch 133 so that the first IP conversion processing unit 111B is inserted between the image data reading unit 109B and the format conversion unit 121. The first IP conversion processing unit 111B performs normal IP conversion processing (step S371). Next, the format conversion processing unit 121 performs format conversion processing on the input moving image obtained by the normal IP conversion processing (step S371) performed by the first IP conversion processing unit 111B (step S373). The display processing unit 123 performs display processing (step S375).

  If the frame drop / frame insertion determination unit 115 determines that it is necessary to start the frame insertion process (YES in step S351), the second IP conversion process (that is, the frame insertion process is performed in the input frame). Including the IP conversion process) (step S359), the output frame generated by the format conversion process (step S365) and the display process (step S367) is output, so that one frame is not reduced unlike the frame drop process. If it is determined that the counter value is zero, the second IP conversion process (that is, the IP conversion process including the frame insertion process) (step S359), the format conversion process (step S365), and the display process ( In addition to step S367), the first IP conversion process (that is, normal IP conversion process) (step S371), the format conversion process (step S2373), and the display process (step S375) are executed. Can be increased.

  The operation example of the frame insertion process according to the second embodiment and the effects described with reference thereto are the same as those illustrated in FIG. 5 according to the first embodiment and the effects described with reference thereto.

[Embodiment 3]
As described above, an object of the present invention is to not lose the smoothness of a moving image, and to prevent a calculation amount from being greatly increased even if processing is performed so as not to lose the smoothness of a moving image. . On the other hand, the input moving image is generally a moving image, but also includes a frame of a still image. When an input frame that is determined to require frame drop processing or frame insert processing is a still image frame, frame drop is realized by simply discarding the frame, or by overlapping the frame. Even if the insertion process is realized, the smoothness of the still image is not lost. In addition, when the input frame (target frame) determined to require the frame drop process or the frame insertion process and all the input frames in the vicinity thereof are still image frames, the frame is simply discarded by discarding the target frame. Even if the frame insertion processing is realized by realizing dropping or overlapping the frames, the degree to which the smoothness of the still image is not lost is further increased.

  Therefore, in the third embodiment, when it is determined that the frame drop process or the frame insertion process is necessary, it is determined whether or not the current input frame is a moving image frame. When it is determined that the current input frame is a moving image frame, the frame dropping process or the frame insertion process as described in the first or second embodiment is performed, and the current input frame is a still image frame. If it is determined, a frame dropping process or a frame inserting process having a smaller calculation amount than the frame dropping process or the frame inserting process described in the first or second embodiment is performed.

  The frame dropping process or the frame inserting process having a smaller calculation amount than the frame dropping process or the frame inserting process as described in the first or second embodiment is, for example, a frame dropping process or a target by simply discarding the target frame. This is a frame insertion process by simply overlapping frames.

  Further, instead of determining whether or not the current input frame is a moving image frame, it may be determined whether or not the current input frame and all input frames in the vicinity thereof are moving image frames.

  The third embodiment adds the above function by modifying the first embodiment. Therefore, the configuration and operation of the system according to the third embodiment are basically the same as those of the first embodiment. Therefore, the description will be focused on the points of the third embodiment different from the first embodiment.

  FIG. 9 shows the configuration of the output processing unit 107C according to the third embodiment. Next, only differences from the output processing unit 107 according to the first embodiment illustrated in FIG. 1 will be described.

  The IP conversion processing unit 111 includes a moving pixel / still pixel determination unit 141. Further, a moving image frame / still image frame determination unit 143, a first switch 145, a still image frame frame drop / frame insertion processing unit 147, a second switch 149, and a control unit 151 are added.

  The moving image / still pixel determination unit 141 determines whether each pixel of a frame to be synthesized by the current IP conversion processing unit 111 is a pixel included in the moving image portion or a pixel included in the still image portion. Here, the moving image portion and the still image portion may spatially be in units of pixels, may be in units of blocks of multiple pixels, or may be objects represented by images. May be the unit. In addition, the moving image portion and the still image portion may be determined for each frame as to whether the moving image portion or the still image portion is a moving image portion or a still image portion. It may be determined. Further, whether each pixel is a pixel included in the moving image portion or a pixel included in the still image portion may be represented by a binary value, and the degree thereof is represented by a multi-value or a continuous value. Also good. When the IP conversion processing unit 111 has a configuration as shown in FIG. 19, a difference calculation unit 509, a difference correction unit 511, a correction value holding unit 513, a difference accumulation unit 515, an accumulated motion amount memory 517, and a threshold value holding The unit 519 and the motion amount calculation unit 521 may be used as the moving pixel / still pixel determination unit 141.

  The moving image frame / still image frame determination unit 143 is information regarding whether each pixel of the current frame input from the moving pixel / still pixel determination unit 141 is a pixel included in the moving image portion or a pixel included in the still image portion. Based on the above, it is determined whether the current frame is a moving image frame or a still image frame. For example, whether each pixel is a pixel included in the moving image portion or a pixel included in the still image portion is represented by a multivalue that increases as the possibility that the pixel is a pixel included in the moving image increases. If the ratio of all the pixels whose value is equal to or less than the first threshold is equal to or greater than the second predetermined value, it is determined that the frame is a still image frame. It may be. Further, when the value obtained by adding the multivalues for all the pixels in the frame is equal to or smaller than the third predetermined value, the frame may be determined to be a still image frame. Also, these threshold values may be changed based on the current CPU load.

  When the frame drop / frame insertion determination unit 115 determines that frame drop processing or frame insertion processing is necessary, the control unit 151 further determines whether the frame currently output by the IP conversion processing unit 111 is a moving image frame. Whether the frame is an image frame is determined by a signal input from the moving image frame / still image frame determination unit 143.

  Then, when it is determined that the frame drop process or the frame insertion process is currently necessary, and the control unit 151 determines that the frame currently output by the IP conversion processing unit 111 is a moving image frame, the control unit 151 then selects a plurality of frames. The synthesis processing unit 117 determines that the frame dropping process or the frame insertion process corresponding to the moving image frame should be performed over a plurality of frames. For this purpose, the control unit 151 switches the first switch 145 and the second switch 149 so that the composition processing unit 117 is inserted between the IP conversion processing unit 111 and the format conversion unit 121. Then, the composition processing unit 117 performs frame dropping processing or frame inserting processing corresponding to the moving image frame as described in the first embodiment over the plurality of frames.

  On the other hand, the control unit 151 determines that a frame drop process or a frame insertion process is currently necessary, and determines that the frame currently output by the IP conversion processing unit 111 is a still image frame. The frame drop / frame insertion processing unit 147 determines that the frame drop or frame insertion processing corresponding to the still image frame should be performed. Therefore, the control unit 151 includes the first switch 145 and the second switch 149 so that the frame drop / frame insertion processing unit 147 for the still image frame is inserted between the IP conversion processing unit 111 and the format conversion unit 121. Switch. The still image frame frame dropping / frame insertion processing unit 147 performs frame dropping processing or frame insertion processing corresponding to the still image frame. However, the display processing unit 123 performs processing for displaying the signal input from the format conversion unit 121 multiple times (multiple writing to the display memory), thereby realizing frame insertion corresponding to a still image frame. May be.

  In addition, when it is determined that the frame dropping process or the frame inserting process is not currently required, the control unit 151 also includes the still image frame frame between the IP conversion processing unit 111 and the format conversion unit 121. The first switch 145 and the second switch 149 are switched so that the drop / frame insertion processing unit 147 does not enter. However, the composition processing unit 117 or the still image frame frame dropping / frame insertion processing unit 147 may be inserted, and these portions may perform bypass processing.

  Next, output processing performed by the output processing unit 107C when the output frame frequency is lower than the input frame frequency will be described.

  This process is represented by the flowchart shown in FIG. Only differences from the corresponding flowchart of the first embodiment shown in FIG. 2 will be described. Step S223 is inserted between step S203 and step S205. In step S223, it is determined whether or not the current frame is a moving image frame. If the current frame is a moving image frame (YES in step S223), the process proceeds to step S205 as in the first embodiment. If the current frame is a still image frame (NO in step S223), steps S205 and S207 are skipped.

  Therefore, if it is determined that the frame drop process is necessary (YES in step S203) and it is determined that the current frame is a still image frame (NO in step S223), the format conversion process (step S217) is performed. ) And display processing (step S219) are not performed, so one frame is dropped. Further, since step S207 is also skipped, the counter is not initialized to 30. Accordingly, in the output process for the next input frame, after it is determined that the frame drop start process is not necessary (NO in step S203), the counter is zero (NO in step S209), so the input frame is used as it is. (Step S221). Therefore, a simple one frame drop is realized, and the output frame synthesis process (step S211) can be omitted.

  Next, output processing performed by the output processing unit 107C when the output frame frequency is higher than the input frame frequency will be described.

  This process is represented by the flowchart shown in FIG. Only differences from the corresponding flowchart of the first embodiment shown in FIG. 4 will be described. Step S285 is inserted between step S253 and step S255. In step S285, it is determined whether or not the current frame is a moving image frame. If the current frame is a moving image frame (YES in step S285), the process proceeds to step S255 as in the first embodiment.

  When the current frame is a still image frame (NO in step S285), the input frame itself is used as an output frame (step S287), and the format conversion processing unit 121 performs format conversion processing (step S289), and display processing is performed. The unit 123 performs display processing of the signal obtained in step S289 twice (steps S291 and S293). That is, the signal for one frame obtained in step S289 is written twice in the display memory 125.

  Therefore, when it is determined that the frame insertion process is necessary (YES in step S253) and the current frame is determined to be a still image frame (NO in step S285), the format conversion process (step S289) is performed. ) Is displayed twice (steps S291 and S293), so that one frame is displayed twice. Further, since step S275 is skipped, the counter is not initialized to 30. Accordingly, in the output process for the next input frame, after it is determined that the frame insertion start process is not necessary (NO in step S253), the counter is zero (NO in step S259), so the input frame is used as it is. (Step S281). Therefore, simple one-frame insertion is realized, and the output frame synthesis process (step S261) can be omitted.

[Embodiment 4]
In the fourth embodiment, the function described in the introduction part of the third embodiment is added by modifying the second embodiment. Therefore, the configuration and operation of the system according to the fourth embodiment are basically the same as those of the second embodiment. Therefore, the difference between the fourth embodiment and the second embodiment will be mainly described.

  FIG. 12 shows a configuration of the output processing unit 107D according to the fourth embodiment. Next, only differences from the output processing unit 107B according to the second embodiment illustrated in FIG. 6 will be described.

  The first IP conversion processing unit 111B includes a moving pixel / still pixel determination unit 141. Note that the moving pixel / still pixel determination unit 141 may be included in the second IP conversion processing unit 111C, or is shared by the first IP conversion processing unit 111B and the second IP conversion processing unit 111C. It may be a thing. In addition, a moving image frame / still image frame determination unit 143 is added. The control unit 135 is changed to a control unit 135B.

  The moving picture element / still picture pixel determination unit 141 and the moving picture frame / still picture frame determination unit 143 are the same as those in the third embodiment, and thus redundant description is omitted.

  When the frame drop / frame insertion determination unit 115 determines that a frame drop process or a frame insertion process is necessary, the control unit 135B further displays a frame that is currently output by the first IP conversion processing unit 111B as a moving image frame. Whether it is a still image frame or not is determined by a signal input from the moving image frame / still image frame determination unit 143.

  Then, the control unit 135B determines that the frame dropping process or the frame insertion process is currently necessary, and determines that the frame currently output by the first IP conversion processing unit 111B is a moving image frame. Then, the second IP conversion processing unit 111C determines that an IP conversion process including a frame dropping process or a frame insertion process corresponding to the moving image frame should be performed over a plurality of frames. Therefore, the control unit 135B switches the first switch 131 and the second switch 133 so that the second IP conversion processing unit 111C is inserted between the image data reading unit 109B and the format conversion unit 121. Then, the second IP conversion processing unit 111C performs an IP conversion process including a frame dropping process or a frame insertion process corresponding to a moving image frame as described in the second embodiment over the plurality of frames.

  On the other hand, the control unit 135B determines that a frame drop process or a frame insertion process is currently necessary, and determines that the frame currently output by the first IP conversion processing unit 111B is a still image frame. Determines that the first IP conversion processing unit 111B should perform IP conversion processing including frame drop processing or frame insertion processing corresponding to a still image frame. For this purpose, the control unit 135B switches the first switch 131 and the second switch 133 so that the first IP conversion processing unit 111B is inserted between the image data reading unit 109B and the format conversion unit 121. Then, the first IP conversion processing unit 111B performs an IP conversion process including a frame drop process or a frame insertion process corresponding to the still image frame. However, the display processing unit 123 performs processing for displaying the signal input from the format conversion unit 121 multiple times (multiple writing to the display memory), thereby realizing frame insertion corresponding to a still image frame. May be.

  In addition, when it is determined that the frame drop process or the frame insertion process is not currently required, the control unit 135B determines that the first IP conversion processing unit 111B should perform a normal IP conversion process. For this purpose, the control unit 135B switches the first switch 131 and the second switch 133 so that the first IP conversion processing unit 111B is inserted between the image data reading unit 109B and the format conversion unit 121. Then, the first IP conversion processing unit 111B performs normal IP conversion processing.

  Next, output processing performed by the output processing unit 107D when the output frame frequency is lower than the input frame frequency will be described.

  This process is represented by the flowchart shown in FIG. Only differences from the corresponding flowchart of the first embodiment shown in FIG. 7 will be described. Step S319 is inserted between step S301 and step S305. In step S319, it is determined whether or not the current frame is a moving image frame. If the current frame is a moving image frame (YES in step S319), the process proceeds to step S305 as in the second embodiment. If the current frame is a still image frame (NO in step S319), step S305 is skipped.

  Therefore, when it is determined that the frame drop process is necessary (YES in step S301) and the current frame is determined to be a still image frame (NO in step S319), the format conversion process (step S313) is performed. ) And display processing (step S315) are not performed, so one frame is dropped. Since step S305 is also skipped, the counter is not initialized to 30. Therefore, in the output process for the next input frame, after it is determined that the frame drop start process is not necessary (NO in step S301), the counter is zero (NO in step S307), so that the normal IP conversion process ( First IP conversion processing) is performed (step S317). Therefore, a simple frame drop is realized, and the second IP conversion process (step S309) can be omitted.

  Next, output processing performed by the output processing unit 107D when the output frame frequency is higher than the input frame frequency will be described.

  This process is represented by the flowchart shown in FIG. Only differences from the corresponding flowchart of the second embodiment shown in FIG. 8 will be described. Step S377 is inserted between step S351 and step S355. In step S377, it is determined whether or not the current frame is a moving image frame. If the current frame is a moving image frame (YES in step S377), the process proceeds to step S355 as in the second embodiment.

  If the current frame is a still image frame (NO in step S377), the first IP conversion processing unit 111B performs normal IP conversion processing (step S379), and the format conversion processing unit 121 performs format conversion. Processing is performed (step S381), and the display processing unit 123 performs display processing of the signal obtained in step S289 twice (steps S383 and S385). That is, the signal for one frame obtained in step S383 is written twice in the display memory 125.

  Therefore, when it is determined that the frame insertion process is necessary (YES in step S351) and the current frame is determined to be a still image frame (NO in step S377), a normal IP conversion process ( Since the frame generated in step S379) is displayed twice (steps S383 and S385), one frame is displayed twice. In addition, since step S355 is skipped, the counter is not initialized to 30. Accordingly, in the output process for the next input frame, since it is determined that the frame insertion start process is not necessary (NO in step S355), the counter is zero (NO in step S357), so the second IP conversion process Instead of (Step S359), the first IP conversion process (Step S375) is performed. Therefore, simple one-frame insertion is realized. By determining that the first IP conversion process (step S375) has a smaller calculation amount than the second IP conversion process (step S359), it is determined that the frame insertion process is necessary (YES in step S351). If it is determined that the current frame is a still image frame (NO in step S377), it is determined that frame insertion processing is necessary (YES in step S351), and the current frame is a moving image frame. The amount of calculation can be reduced compared to the case where it is determined that (YES in step S377).

[Embodiment 5]
In the first and second embodiments, the initial value of the counter is set to 30 so that the frame dropping process or the frame inserting process is performed over about 30 frames. It is also possible to set the initial value of the counter to another value. However, if the initial value of the counter is too large, the effect of reducing the amount of calculation is diminished. Conversely, if the initial value of the counter is too small, the effect of smoothing the movement is diminished. The value of 30 is a value determined in consideration of both.

  When the output frame frequency is slightly lower than the input frame frequency, setting the initial value of the counter to 30 can reduce the ratio of the time for continuing the frame dropping process to the necessary time. it can.

  However, if the output frame frequency is much lower than the input frame frequency, a frame drop request is requested in a certain frame, and the next frame drop request is issued before the frame drop process for a series of frames started from that time is completed. It can happen. In such a case, the frame dropping process over the current series of frames is interrupted, and the next frame dropping process over a series of frames starting from the interrupted frame is entered. If it does so, an output frame will become discontinuous between them, and the effect of not losing the smoothness of a motion will fade.

  In the fifth embodiment, such a problem is solved. For this purpose, the initial value of the counter is dynamically changed according to the interval of frame drop requests.

  Since the basic configuration of the system of the fifth embodiment is the same as that of the first embodiment, description thereof is omitted.

  In the fifth embodiment, the output process shown in FIG. 2 is changed to the output process shown in FIG.

  The “composite counter” shown in FIG. 15 corresponds to the “counter” of the first embodiment, and is used for weighting in output frame synthesis (step S409). The “composite counter” is dynamically initialized in the frame drop start process (step S405).

  Roughly speaking, the “N counter” reflects the frame interval at which a frame drop request occurs.

  Referring to FIG. 15, first, the IP conversion processing unit 111 performs IP conversion processing (step S401). Details of the IP conversion process will be described later.

  Next, the frame drop / frame insertion determination unit 115 receives a field synchronization signal (which has an input frame frequency) input from the NTSC decoder 101 and a refresh signal (which has an output frame frequency) input from the monitor 127. )), It is determined whether or not it is necessary to open the frame drop start process (step S403).

  If the frame drop / frame insertion determination unit 115 determines that it is necessary to perform frame drop start processing (YES in step S403), the composition processing unit 117 performs frame drop start processing (step S405). This will be described later.

  The frame drop start process (step S405) is a replacement of step S205 and step S207 of the first embodiment.

  Accordingly, if the frame drop / frame insertion determination unit 115 determines that it is necessary to start the frame drop process (YES in step S403), the format conversion process (step S417) and the display process (step S417) are performed in the input frame. Since step S419) is omitted, the output frame can be reduced by one.

  When the frame drop / frame insertion determination unit 115 determines that it is not necessary to perform the frame drop start process (NO in step S403), the combination processing unit 117 determines whether the value of the “composite counter” exceeds zero. Is determined (step S407).

  If the value of the “composite counter” exceeds zero (YES in step S407), this means that the current frame belongs to a plurality of frames to be subjected to the frame dropping process. Then, the process proceeds to step S409.

  On the other hand, if the value of the “composite counter” does not exceed zero (NO in step S407), this means that the current frame does not belong to a plurality of frames to be subjected to the frame dropping process (that is, the previous frame dropping). This means that the process has already been completed), and thus the process proceeds to step S421 in order to omit the frame drop process.

  In step S409, the composition processing unit 117 synthesizes an output frame based on the input frame and the temporarily saved frame. The input frame is an input frame generated in the IP conversion process (step S401) in the current output process, and in short, is the current input frame. The temporary storage frame is generated in the IP conversion process (step S401) in the output process prior to the current output process, and is stored in the temporary storage memory 119 in step S405 (more specifically, step S405-13) or S411. This is an input frame that is temporarily stored. In short, it is the previous input frame.

  In the synthesis, for example, each pixel y (m, i, j) in the i-th row and j-th column of the output frame m is calculated according to the following equation.

y (m, i, j) = w × x (n−1, i, j) + (1−w) × x (n, i, j)
Here, w represents a weight, x (n−1, i, j) represents a pixel in the i-th row and j-th column of the storage frame, and x (n, i, j) represents the input frame. Represents a pixel in the i-th row and j-th column. Moreover, w is calculated according to the following formula. The “composite counter initial value” in the following formula will be described later.

w = Composition counter / synthesis counter initial value Next to step S409, the composition processing unit 117 stores the current input frame obtained in step S401 in the temporary storage memory 119 (step S411), and sets “composition counter” to 1. The “N counter” is increased by 1 (step S415).

  On the other hand, in step S421, the composition processing unit 117 sets the input frame itself as an output frame.

  Following step S415 or step S421, the format conversion processing unit 121 performs format conversion processing (step S417), and the display processing unit 123 performs display processing (step S419).

  Next, details of step S405 will be described with reference to FIG.

  The “composite counter initial value” shown in FIG. 16 is a value used as the initial value of the “composite counter” as indicated by the character. This also changes dynamically.

  First, it is determined whether the “N counter” is 30 or more (step S405-1). Here, 30 is a number given as an example corresponding to the initial value of “counter” being 30 in the first embodiment, and “N counter” may be compared with other values. However, the value compared with the “N counter” in step S405-1 defines the maximum value of the number of frames in which a series of frame drop processing is continued.

  If the “N counter” is 30 or more (YES in step S405-1), the numerical value 30 is substituted for the “composite counter initial value” (step S405-3). The numerical value 30 here is adjusted to 30 in step S405-1, and when 30 in step S405-1 is changed to another value, the numerical value 30 also changes to that value.

  If the “N counter” is not 30 or more (NO in step S405-1), it is determined whether the “N counter” is larger than the “synthesis counter initial value” (step S405-5).

  If “N counter” is larger than “composite counter initial value” (YES in step S405-5), “composite counter initial value” is increased by 1 (step S405-7).

  If “N counter” is not greater than “composite counter initial value” (NO in step S405-5), it is determined whether “N counter” is smaller than “composite counter initial value” (step S405). -9).

  When “N counter” is smaller than “composite counter initial value” (YES in step S405-9), a value obtained by subtracting 1 from the value of “N counter” is substituted into “composite counter initial value”. (Step S405-11).

  If “N counter” is not smaller than “composite counter initial value” (NO in step S405-9), “N counter” is 30 or less, and “N counter” and “composite counter initial value” are Means equal.

  After step S405-1, step S405-1, and step S405-1, the process proceeds to step S405-13. Moreover, also when it is NO at step S405-9, it progresses to step S405-13.

  In step S405-13, the composition processing unit 117 stores the current input frame obtained in step S401 in the temporary storage memory 119.

  Next, “synthesis counter initial value” is substituted into “synthesis counter” (step S405-15).

  Finally, the “N counter” is initialized with zero (step S405-17).

  FIG. 17 shows an operation example. In this operation example, it is determined that the frame dropping process is necessary every 10 frames, and according to this, the “composite counter initial value” finally converges to 9, and the “composite counter” is 9 over 10 frames. 1 to 0 is counted one by one. Frame dropping is continuously performed at a rate of 1 frame per 10 frames.

  Referring to FIG. 17, it is determined that frame drop processing is necessary in the 40th input frame. Since “N counter”> 30 (YES in step S405-1), the “composite counter” is initialized to 30.

  In the 50th input frame, it is determined that frame drop processing is necessary. “Composition counter initial value” is 30, and “N counter” is 9. Therefore, not “N counter”> 30 (NO in step S405-1), but not “N counter”> “composite counter initial value” (NO in step S405-5), “N counter” <“composite counter initial” Value "(YES in step S405-9). Accordingly, the “synthesis counter initial value” is initialized to “N counter” −1 = 9-1 = 8 (step S405-11).

  In the 60th input frame, it is determined that frame drop processing is necessary. The “synthesis counter initial value” is 8, and the “N counter” is 9. Therefore, not “N counter”> 30 (NO in step S405-1), but “N counter”> “composition counter initial value” (YES in step S405-5). Accordingly, the “synthesis counter initial value” is increased by 1 to 9 (step S405-7).

  In the 70th input frame, it is determined that frame drop processing is necessary. The “composite counter initial value” is 9, and the “N counter” is 9. Therefore, not “N counter”> 30 (NO in step S405-1), but not “N counter”> “composite counter initial value” (NO in step S405-5), “N counter” <“composite counter initial” It is not “value” (NO in step S405-9). Therefore, the “synthesis counter initial value” remains at 9. From this point on, it has entered a steady state.

  The operation in the 80th input frame is the same as the operation in the 70th input frame.

  FIG. 18 shows changes in “N counter” (indicated by diamonds), “composite counter initial value” (indicated by squares), and “composite counter” (indicated by Δ) among the values shown in FIG. The “N counter” is reset every time there is a frame drop request, and thereafter increments by 1 for each frame. The “synthesis counter initial value” fluctuates at first, but finally converges to 9. The “composite counter” is initialized every time there is a frame drop request, and is decremented by 1 for each frame. In the convergence state, every time there is a frame drop request, the “composite counter” is initialized to the converged “composite counter initial value” and is decremented by 1 for each frame.

  In the above-described operation example, by adjusting the initial value of the “composite counter”, the next frame dropping process is started before the frame dropping process for the number of frames corresponding to the initial value of the “compositing counter” is completed. You can avoid doing that.

  However, when the initial value of the “composite counter” is adjusted in this way, it is possible to avoid losing the smoothness of the screen at the boundary between two series of frame drop processing, but the amount of calculation cannot be reduced.

  In order to solve this and reduce the amount of calculation, for example, when it is determined NO in step S405-9 for a predetermined number of times or more, the variable “composite counter initial value” is stabilized, In step S405-15, instead of substituting “composition counter initial value” for “composition counter”, the ratio of frames for which frame dropping processing is performed to “composition counter initial value” in step S405-15. A value obtained by multiplying by a coefficient (a number greater than zero and less than 1) indicating may be substituted into the “synthesis counter”. Further, this coefficient may be changed according to the load condition of the computer. In order to determine stabilization, for example, a variable indicating stability is introduced, this variable is reset in step S405-3, step S405-7, and step S405-11, and NO in step S405-9. If this happens, increase this variable. In step S405-15, when this variable is greater than or equal to a predetermined value, the “composite counter initial value” is obtained by multiplying by a coefficient (a number greater than zero and less than 1) indicating the proportion of frames to be subjected to frame drop processing. The obtained value is assigned to the “composite counter”.

  When the “composite counter initial value” is almost equal to the “N counter” every time the frame dropping start process is entered, the stable state is entered. Therefore, when such a stable state is entered, in step S405-15, instead of substituting the “composite counter initial value” for the “composite counter”, a frame for frame dropping processing is added to the “composite counter initial value”. A value obtained by multiplying by a coefficient R (a number greater than zero and less than 1, for example, 0.3) indicating a ratio of the entire frame to the “composite counter” may be substituted. Further, this coefficient may be changed according to the load condition of the computer. In order to determine the stable state, for example, a variable “stability counter” indicating the stability is introduced.

  For example, the frame drop start process as shown in FIG. 16 is changed to that shown in FIG.

  In step S405-21 following step S405-3, the “stability counter” is reset to zero.

In step S405-23 after step S405-7 or step S405-11,
“N counter” −A <“composite counter” <“N counter” + A
It is determined whether or not is established. If not established (NO in step S405-23), the “stability counter” is reset to zero (step S405-25). If true (YES in step S405-23), the "stability counter" is incremented by 1 (step S405-27).

  In this way, only when the stable state is entered, step S405-27 is executed continuously, and the “stability counter” continues to increase. Note that A used in step S405-23 may be zero, but may take a value of about 1 or 2 in consideration of fluctuations.

  Before entering step S405-15, step S405-29 is provided, where it is determined whether or not the “stability counter” exceeds M.

  If not exceeded (NO in step S405-29), the stable state has not been entered, and as in the case of FIG. 16, the "composite counter initial value" is substituted into the "composite counter" (step S405-15). .

  If it exceeds (YES in step S405-29), since it is in a stable state, unlike the case of FIG. 16, as described above, the “composite counter initial value” is set to the entire frame to be subjected to frame dropping processing. A value obtained by multiplying by a coefficient R (a number exceeding zero and less than 1, for example, 0.3) indicating a ratio to the frame is substituted into the “synthesis counter” (step S405-31).

  Next, in order to avoid the “stability counter” from continuing to increase, the “stability counter” is initialized to M (step S405-33).

  In addition, when the frame drop start process of FIG. 19 is performed instead of the frame drop start process of FIG. 16, the weight used in step S409 of FIG. 15 is calculated according to the following equation.

w = composite counter / (R × composite counter initial value)
[Normal IP conversion processing]
Next, the configuration of the IP conversion processing unit 111 or the first IP conversion processing unit 111B and the normal IP conversion processing (IP conversion processing not including frame drop processing and frame insertion processing) performed by the IP conversion processing unit 111B will be described.

  FIG. 20 is a block diagram illustrating a configuration of the IP conversion processing unit 111 or the first IP conversion processing unit 111B and its peripheral part.

  The first field memory 501-1 stores the image data of the field one field ahead of the field to be processed. The second field memory 501-2 stores image data of a field to be processed. The third field memory 501-3 stores image data of the field immediately before the field to be processed.

  The first field memory 501-1 to the third field memory 501-3 are included in the frame buffer group 105 shown in FIG. 1 or the frame buffer group 105B shown in FIG. One field memory corresponds to one frame buffer. In FIG. 20, the image data reading unit 109 shown in FIG. 1 or the image data reading unit 109B and the first switch 131 shown in FIG. 6 are omitted.

  The image data writing unit 103 writes image data in the first to third field memories 501-1, 501-2, and 501-3. The first to third field memories 501-1, 501-2, and 501-3 have a ring buffer configuration, and the image data of three fields are stored in these field memories in a circular manner. May be.

  The moving image interpolation pixel generation unit 505 generates a moving image interpolation pixel based on the image data stored in the second field memory 501-2.

  The moving image interpolation pixel generation unit 505 basically has, for example, the field number currently being processed as n, and the vertical direction of the pixel currently being processed as a pixel to be interpolated in that field. When the position is i and the horizontal position of the pixel currently being processed as a pixel to be interpolated in that field is j, the moving image interpolation pixel A (n, i, j) is represented by the equation (1). Generate. Here, i is a position when the actual scanning line and the scanning line to be interpolated are counted together.

Here, x (n, i−1, j) is the value of the pixel one pixel above the pixel to be processed in the field to be processed, and x (n, i + 1, j). ) Is the value of the pixel one pixel below the pixel to be processed in the field to be processed. In the above, the average value of the values of these two pixels is the value of the pixel to be processed, but the value of one pixel of these two pixels is the pixel to be processed. It is good also as the value of. Also, a weighted average of the values of pixels in a predetermined vicinity of the pixel to be processed among the pixels in the field to be processed may be used as the value of the pixel to be processed.

  The still image interpolation pixel generation unit 507 generates a still image pixel based on the pixel data stored in the first field memory 501-1 and the pixel data stored in the third field memory 501-3. Generate. For example, the number of the field currently being processed is n, the vertical position of the pixel currently being processed as a pixel to be interpolated in that field, i, and the pixel to be interpolated in that field If the horizontal position of the pixel currently being processed is j, a moving image interpolation pixel B (n, i, j) is generated according to equation (2).

Here, x (n−1, i, j) is the same position (i, j) as the pixel to be processed among the pixels in the field immediately before the field to be processed. ), And x (n + 1, i, j) is the same position as the pixel to be processed among the pixels in the next field after the field to be processed. This is the value of the pixel at (i, j). In the above, the average value of the values of these two pixels is the value of the pixel to be processed, but the value of one pixel of these two pixels is the pixel to be processed. It is good also as the value of. Also, the value of the pixel subject to processing is the weighted average of the values of the pixels in the same position as the pixel subject to processing among the pixels in the field in the vicinity of the field subject to processing. It is good.

  The difference calculation unit 509 obtains a difference C (n, i, j) for the pixel to be processed in the field to be processed according to the equation (3).

Here, x (n−1, i, j) is the same position (i, j) as the pixel to be processed among the pixels in the field immediately before the field to be processed. ), And x (n + 1, i, j) is the same position as the pixel to be processed among the pixels in the next field after the field to be processed. This is the value of the pixel at (i, j). The difference C (n, i, j) is the basis of the amount of motion.

  The difference correction unit 511 obtains a corrected difference D (n, i, j) by correcting C according to Expression (4).

Here, the value of Δc is determined by the noise level or the like and stored in the correction value holding unit 513.

  The difference accumulating unit 515 updates the accumulated motion amount m (i, j) stored in the accumulated motion amount memory 517 according to the equation (5).

Here, the value of TH is determined by visual characteristics or the like and stored in the threshold value holding unit 519. The value of TH is 16, for example.

  If a certain amount of motion is detected by calculating the cumulative motion amount m (i, j) according to the above equation, the cumulative motion amount m (i, j) A value obtained by adding 1 to the accumulated motion amount in the previous field and subtracting 1 or when the motion amount is large is a threshold value TH, and decreases by 1 every time one field has passed since then. By doing so, the motion detection is tailed. Thereby, the possibility that the moving image part is erroneously determined to be a still image part after movement can be reduced. When the moving image part is erroneously determined to be a still image part, an interpolation value completely different from the interpolation value calculated when correctly processed as the moving image part is calculated, which causes a visual problem. On the other hand, if the still image portion is erroneously determined to be a moving image portion, only the resolution is lowered, and this is not a visual problem. In particular, even if the still image portion around the portion where the motion is actually detected is erroneously determined to be a moving image portion, the reduction in the resolution of the portion is not visually recognized due to the masking effect due to the portion that actually moves. . Therefore, the advantage of introducing the formula (5) and reducing the possibility that the moving image portion is erroneously determined as the still image portion is that the possibility that the still image portion is erroneously determined as the moving image portion is increased. Therefore, the introduction of the equation (5) is useful for improving the image quality as a whole. In addition, since many still image parts such as a telop part in which characters are fixed, no motion is detected from the part, even if the expression (5) is introduced, it is erroneously a moving image part. It will not be judged.

  The motion amount calculation unit 521 calculates the motion amount M (i, j) according to Equation (6).

  The synthetic interpolation pixel generation unit 523 calculates a synthetic interpolation pixel x ′ (n, i, j) according to the equation (7).

  The selection unit 525 switches the actual pixel x (n, i, j) and the interpolation pixel x ′ (n, i, j) for each line, and outputs progressive scan image data.

  The IP conversion processing unit 101 whose configuration is shown in FIG. 20 can be realized by hardware, but the computer records a program for causing the computer to function as the IP conversion processing unit 101 whose configuration is shown in FIG. It can also be realized by reading from the medium and executing it.

  Next, the effect of calculating the motion amount M (i, j) according to the equation (6) will be described.

  As shown in FIG. 21, it is assumed that a black vertical line moves to the right on a white flat background. Further, in order to facilitate the consideration, it is assumed that the motion accumulation amount m (i, j) is calculated according to the equation (8).

  Then, if the motion amount M (i, j) is calculated according to the equation (9) without using the equation (6), as shown in FIG. Interpolation is performed by the interpolation pixel, and combing occurs.

  That is, in FIG. 21, square pixels are originally pixels, and round pixels are pixels to be complemented. For example, the absolute value of the difference between the pixel 134 and the pixel 334 is the amount of motion of the pixel 234. Since this value is zero, the pixel 234 is erroneously determined to be a still image pixel. Then, the average value of the pixel 134 and the pixel 334 becomes the interpolation value of the pixel 234, and the pixel 234 becomes white. The same applies to the pixels 214 and 254. Then, while the pixels 224, 244, and 264 are correctly black, the pixels 214, 234, and 254 are erroneously white and combing occurs.

  On the other hand, if the motion amount M (i, j) is calculated according to the equation (6) according to the present invention, the motion is correctly detected and interpolation is performed by the moving image interpolation pixel as shown in FIG. No combing occurs.

  That is, in FIG. 22, as in the case of FIG. 21, the square pixel is originally a pixel, and the round pixel is a pixel to be complemented. For example, the larger of the sum of the absolute value of the difference between the pixel 044 and the pixel 244 and the threshold value TH in the absolute value of the difference between the pixel 134 and the pixel 334 is the amount of movement of the pixel 234. Since the value is the threshold value TH, the pixel 234 is correctly determined as a moving image pixel. Then, the average value of the pixel 224 and the pixel 244 becomes the interpolation value of the pixel 234, and the pixel 234 becomes black. The same applies to the pixels 214 and 254. Then, since the pixels 224, 244, and 264 are correctly black and the pixels 214, 234, and 254 are also correctly black, no combing occurs.

  Thus, by calculating the motion amount M (i, j) according to the equation (6), it is possible to avoid erroneously determining that a moving image pixel is a still image pixel.

  In Expression (6), when the input field corresponding to the number n is an even field, m (i, j) is a motion amount detected from the even field, and m (i + 1, j) is , The amount of motion detected from the odd field. Similarly, in Equation (6), when the input field corresponding to the number n is an odd field, m (i, j) is the amount of motion detected from the odd field, and m (i + 1, j) Is the amount of motion detected from the even field. That is, the motion amount M (i, j) is used by utilizing the fact that the motion that could not be detected from one of the even field and the odd field can be detected from the other field. I am going to do that.

  Next, an interlace / progressive conversion method performed by the IP conversion processing unit 101 whose configuration is shown in FIG. 20 will be described.

  Referring to FIG. 23, first, the moving image interpolation pixel generation unit 505 generates a moving image interpolation pixel according to Expression (1) (step S581). Next, the still image interpolation pixel generation unit 507 generates a still image interpolation pixel according to Expression (2) (step S583). Next, the difference calculation unit 309 calculates the difference according to the equation (3) (step S585). Next, the difference correction unit 511 corrects the difference according to the equation (4) (step S587). Next, the difference accumulation unit 515 updates the accumulated motion amount according to the equation (5) (Step S589). Next, the motion amount calculation unit 521 calculates the motion amount according to Equation (6) (step S591). Next, the synthetic | combination interpolation pixel production | generation part 523 produces | generates a synthetic | combination interpolation pixel according to Formula (7) (step S595).

  The IP conversion processing unit 111 or the first IP conversion processing unit 111B may have another configuration.

[IP conversion processing including frame drop processing or frame insertion processing]
Next, the configuration of the second IP conversion processing unit 111C and the IP conversion process including the frame dropping process and the frame insertion process performed by the second IP conversion processing unit 111C will be described.

  Referring to FIG. 17, in the output frame column, the temporal position of each output frame is shown based on the number of the input frame. In an IP conversion process including a frame drop process and a frame insertion process, data in a plurality of input fields are used to synthesize output frames at these positions.

  This principle will be described with reference to FIGS. 24 and 25. FIG. 24 and 25, the input field f (n-1), f (n), f (n + 1), f (n + 2) and the output frame F (m ) At a certain horizontal position. In addition, what is indicated by a double circle in the output frame is a pixel (target pixel) y (m, i, j) that is currently generated. Here, i is a line number in the frame, and j is a pixel number in the line. The input field f (n) is a field immediately before the output frame F (m), and the input field f (n + 1) is an input field immediately after the input field f (n).

  FIG. 24A shows the target pixel y (m) when the input field f (n) has a pixel x (n, i, j) spatially located at the same position as the target pixel y (m, i, j). , I, j) shows an example of a method for generating a moving image interpolation pixel ym (m, i, j). In the input field f (n), the pixel spatially closest to the target pixel y (m, i, j) is the pixel x (n, i, j). In the input field f (n + 1), the target pixel y ( The pixels spatially closest to m, i, j) are x (n + 1, i-1, j) and x (n + 1, i + 1, j).

  When the time of the output frame F (m) is tout, the time of the input field f (n) is tin1, and the time of the input field f (n + 1) is tin2, the moving image interpolation pixel ym (m, i, j) is generated. For this purpose, the weights w1 and w2 in the weighted addition are calculated by the following equation (12).

  Then, the moving image interpolation pixel ym (m, i, j) is calculated by the following equation (13).

  However, the correspondence with the second embodiment will be described. In the frame dropping process, the input field f (n) is the previous input field in the second embodiment, and the input field f (n + 1) is the second embodiment. Is the current input field, w1 is the weight w in the second embodiment, and w2 is (1-w) in the third embodiment. Here, the weight w is a counter / 30.

  In the frame insertion process, the input field f (n) is the previous input field in the second embodiment, the input field f (n + 1) is the current input field in the second embodiment, and w1 is (1−w) in the second embodiment, and w2 is the weight w in the third embodiment.

  That is, in the second embodiment, the position of the output frame is approximately expressed using a counter, and the weights w1 and w2 are approximately expressed using the approximate value of the position of the output frame.

  The above correspondence also applies to the following explanation.

  However, the moving image interpolation pixel ym (m, i, j) may be calculated by the following formula (14) or formula (15).

  Further, when the weight w1> w2, the expression (14) may be used, and if not, the expression (15) may be used.

  Also, input pixels other than the input pixels appearing in Expression (13), Expression (14), and Expression (15) (for example, pixels in the vicinity of the position (i, j) of the fields f (n) and f (n + 1)) are also included. The moving image interpolation pixel ym (m, i, j) may be calculated by using this.

  FIG. 24B shows the target pixel y when the input field f (n + 1) includes a pixel x (n, i ′, j) that is spatially located at the same position as the target pixel y (m, i ′, j). An example of a method for generating a moving image interpolation pixel ym (m, i ′, j) for (m, i ′, j) is shown. In the input field f (n + 1), the pixel spatially closest to the target pixel y (m, i ′, j) is the pixel x (n + 1, i ′, j). In the input field f (n), the target pixel The pixels spatially closest to y (m, i ′, j) are x (n, i′−1, j) and x (n, i ′ + 1, j).

  When the time of the output frame F (m) is tout, the time of the input field f (n) is tin1, and the time of the input field f (n + 1) is tin2, the moving image interpolation pixel ym (m, i ′, j) is generated. The weights w1 and w2 in the weighted addition for calculating are calculated by the equation (12) shown above.

  Then, the moving image interpolation pixel ym (m, i ', j) is calculated by the following equation (16).

  However, the moving image interpolation pixel ym (m, i ', j) may be calculated by the following equation (17) or equation (18).

  Further, when weight w1> w2, the expression (17) may be used, and if not, the expression (18) may be used.

  In addition, input pixels other than the input pixels appearing in the equations (16), (17), and (18) (for example, pixels in the vicinity of the position (i, j) of the field f (n) or f (n + 1)) are also included. The moving image interpolation pixel ym (m, i ′, j) may be calculated by using this.

  FIG. 25A shows the target pixel y (m) when the input field f (n) has a pixel x (n, i, j) that is spatially located at the same position as the target pixel y (m, i, j). , I, j) shows an example of a method for generating a still image interpolation pixel ys (m, i, j). Basically, the still image interpolation pixel ys (m, i, j) is a still image interpolation pixel ys (m, i, j) among pixels of one or more input frames near the output frame F (m). ) Based on pixels located in the same spatial position. In the input field f (n), the pixel spatially located at the same position as the target pixel y (m, i, j) is the pixel x (n, i, j), and in the input field f (n + 2), the space In particular, the pixel at the same position as the target pixel y (m, i, j) is x (n + 2, i, j).

  When the time of the output frame F (m) is tout, the time of the input field f (n) is tin1, and the time of the input field f (n + 2) is tin2, the still image interpolation pixel ys (m, i, j) is generated. The weights w1 and w2 in the weighted addition for calculating are calculated by the equation (12) shown above.

  Then, the still image interpolation pixel ys (m, i, j) is calculated by the following equation (19).

  However, the still image interpolation pixel ys (m, i, j) may be calculated by the following equation (20) or equation (21).

  Further, when weight w1> w2, the expression (20) may be used, and if not, the expression (21) may be used.

  In addition, input pixels other than the input pixels appearing in Expression (19), Expression (20), and Expression (21) (for example, x (n−2, i, j), x (n + 4, i, j)) are also used. The still image interpolation pixel ys (m, i, j) may be calculated.

  FIG. 25B illustrates the target pixel y when the input field f (n + 1) includes a pixel x (n + 1, i ′, j) spatially identical to the target pixel y (m, i ′, j). An example of a method for generating a still image interpolation pixel ys (m, i ′, j) for (m, i ′, j) is shown. Basically, the still image interpolation pixel ys (m, i ′, j) is a still image interpolation pixel ys (m, i ′) among pixels of one or more input frames near the output frame F (m). , J) is generated on the basis of pixels located at the same spatial position. In the input field f (n−1), the pixel spatially located at the same position as the target pixel y (m, i ′, j) is the pixel x (n−1, i ′, j), and the input field f In (n + 1), the pixel spatially located at the same position as the target pixel y (m, i ′, j) is x (n + 1, i ′, j).

  When the time of the output frame F (m) is tout, the time of the input field f (n−1) is tin1, and the time of the input field f (n + 1) is tin2, the still image interpolation pixel ys (m, i ′, j ) To calculate the weights w1 and w2 in the weighted addition for generating () by the equation (12) shown above.

  Then, the still image interpolation pixel ys (m, i ', j) is calculated by the following equation (22).

  However, the still image interpolation pixel ys (m, i ', j) may be calculated by the following formula (23) or formula (24).

  Further, when the weight w1> w2, the expression (23) may be used, and if not, the expression (24) may be used.

  In addition, input pixels other than the input pixels appearing in Expression (22), Expression (23), and Expression (24) (for example, x (n−3, i, j), x (n + 3, i, j)) are also used. The still image interpolation pixel ys (m, i ′, j) may be calculated.

  In this way, the moving image interpolation pixel ym (m, i, j) and the still image interpolation pixel ys (m, i, j) for the target pixel are calculated. Here, i represents i and i 'as a representative.

  Then, the motion amount M (i, j) for the target pixel is obtained, and the moving image interpolation pixel ym (m, i, j) and the still image interpolation pixel ys (m, i, j) are calculated with weights based on the motion amount. The target pixel y (m, i, j) is generated by weighted addition.

  As a method for calculating the motion amount M (i, j) and a weighted addition method for generating the target pixel y (m, i, j), the same methods as those in a normal IP conversion process can be used. Hereinafter, these will be described.

  When the input field f (n) includes a pixel x (n, i, j) that is spatially identical to the target pixel y (m, i, j), the difference C (m, i, j) is determined according to equation (25) below.

  If the input field f (n + 1) includes a pixel x (n + 1, i ′, j) spatially identical to the target pixel y (m, i ′, j), the difference C (m, i ′, j) is obtained according to the equation (26) shown below.

The difference C (m, i, j) or C (m, i ', j) is the basis of the amount of motion.

  Next, the corrected difference D (m, i, j) is obtained by correcting C (m, i, j) according to the equation (27) below. Here, i represents i and i ', and so on.

Here, the value of Δc is a value determined by the noise level or the like.

  Next, the accumulated motion amount m (i, j) stored in the memory is updated according to the equation (28) shown below.

Here, the value of TH is a value determined by visual characteristics or the like. The value of TH is 16, for example.

  The effect of calculating the accumulated motion amount m (i, j) according to the equation (28) shown above is as described in connection with the normal IP conversion process.

  Next, the motion amount M (i, j) is calculated according to the following equation (29).

  In Equation (29), one of m (i, j) and m (i + 1, j) is the cumulative motion amount detected from the input even field, and the other is the cumulative motion detected from the input odd field. Amount.

  Next, the target pixel y (m, i, j) is calculated according to the equation (30) shown below.

  Next, the configuration of the second IP conversion processing unit 111C and its peripheral part will be described with reference to FIG.

  Referring to FIG. 26, the image data writing unit 103 receives the interlaced moving image output from the NTSC decoder 101, and stores each field in the first to fourth field memories 601-1 to 601-4. Write. Further, whether each field is an even field or an odd field is detected from, for example, a synchronization signal of an input moving image, or is input from the NTSC decoder 101, and information related thereto is also written in the field memory. Even when the number of field memories is increased and the output frame time fluctuates, the input fields f (n−1), f (n), f (n + 1), f (n + 2) are surely stored in any field memory. ) May be stored.

  The first field memory 601-1 to the fourth field memory 501-4 are included in the frame buffer group 105B shown in FIG. 6, and two field memories are one frame buffer. Corresponding to In FIG. 26, the image data reading unit 109B and the first switch 131 shown in FIG. 6 are omitted.

  The even / odd field reference unit 637 has fields f (n−1), f (n), f (n + 1), and f (n + 2) from the first to fourth field memories 601-1 to 601-4 as even fields. Get information about whether the field is an odd field.

  The weight calculation unit 639 relates to whether the weight w and the fields f (n−1), f (n), f (n + 1), and f (n + 2) input from the control unit 135 are even fields or odd fields, respectively. Based on the information, a moving image interpolation pixel ym (m, i, j) and a still image interpolation pixel ys (m, i, j) are generated for each pixel y (m, i, j) of the output frame F (m). To calculate the weight for

  That is, for example, whether the moving image interpolation pixel ym (m, i, j) is calculated based on the equation (13) or the equation (16) is the input field f (n) is an even field. Or an odd field, and information on whether the line number i of the target pixel y (m, i, j) is an even number or an odd number, and further, weighted addition in each case Is calculated based on the weight w. As described above, the expression (14) or the expression (15) is used fixedly or adaptively instead of the expression (13), or the expression (17) or the expression (18) is used instead of the expression (16). May be calculated in a fixed or adaptive manner.

  Further, for example, whether the still image interpolation pixel ys (m, i, j) is calculated based on the equation (19) or the equation (22) is an even field. It is determined on the basis of information on whether there is an odd field or an odd field and information on whether the line number i of the target pixel y (m, i, j) is even or odd, and the weights in each case The weight in addition is calculated based on the weight w. As described above, the expression (20) or the expression (21) is used fixedly or adaptively instead of the expression (19), or the expression (23) or the expression (24) is used instead of the expression (22). May be calculated in a fixed or adaptive manner.

  The moving image interpolation pixel generation unit 605 inputs the weight necessary for generating the moving image interpolation pixel ym (m, i, j) for each target pixel y (m, i, j) from the weight calculation unit 329, and Necessary pixels are read from the second to third field memories 601-2 to 601-3 based on the weights, and a still image interpolation pixel ym (m, i, j) is determined based on the input weights and the read pixels. Generate.

  The still image interpolation pixel generation unit 607 inputs the weight necessary for generating the still image interpolation pixel ys (m, i, j) for each target pixel y (m, i, j) from the weight calculation unit 329. The necessary pixels are read from the first to fourth field memories 601-1 to 601-4 based on the weights, and the still image interpolation pixel ys (m, i, j is based on the input weights and the read pixels. ) Is generated.

  The difference calculation unit 609 relates to information on whether the field f (n) is an even field or an odd field and whether the line number i of the target pixel y (m, i, j) is an even number or an odd number. Based on the information, it is determined which pixel in which field should be calculated, and based on the result, necessary pixels are determined as the first field memory 601-1 and the third field memory 601-. 3 or the second field memory 601-2 and the fourth field memory 601-4, and based on the read pixel, the difference C (m, i, j) based on the equation (25) or the equation (26) Is calculated.

  The difference correction unit 611 calculates a corrected difference D (m, i, j) by correcting C according to the equation (27).

  Here, the value of Δc is determined by the noise level or the like and stored in the correction value holding unit 613.

  The difference accumulating unit 615 updates the accumulated motion amount m (i, j) stored in the accumulated motion amount memory 617 according to the equation (28).

  Here, the value of TH is determined by visual characteristics or the like and stored in the threshold value holding unit 619. The value of TH is 16, for example.

  The motion amount calculator 621 calculates the motion amount M (i, j) according to the equation (29).

  The synthetic interpolation pixel generation unit 623 calculates a synthetic interpolation pixel y (m, i, j) according to the equation (30). The combined interpolation pixel y (m, i, j) is written into the display memory 105-1.

  In addition, although the 2nd IP conversion process part 111C which shows the structure in FIG. 26 can also be implement | achieved by a hardware, in order to make a computer function as the 2nd IP conversion process part 111C which shows the structure in FIG. This program can also be realized by a computer reading the program from a recording medium and executing it.

  In addition, the first IP conversion processing unit 111B and the second IP conversion processing unit 111C include many common components, and the motion amount is calculated using the first IP conversion processing and the second IP conversion processing. It is necessary to carry out continuously without distinction. Accordingly, the difference calculation unit 509, the difference correction unit 511, the correction value holding unit 513, the difference accumulation unit 515, the accumulated motion amount memory 517, the threshold value holding unit 519, and the motion amount calculation unit 521 are respectively converted into the difference calculation unit 609, There is also a form shared with the difference correction unit 611, the correction value holding unit 613, the difference accumulation unit 615, the accumulated motion amount memory 617, the threshold value holding unit 619 and the motion amount calculation unit 621. Further, even when both the first IP conversion process and the second IP conversion process are omitted in the operations of FIGS. 7 and 13, the motion amount calculation portion operates.

[Frame drop determination processing (1)]
In the first embodiment, the image signal output from the IP conversion processing unit 111 that does not have the frame drop / frame insertion processing function is subjected to frame drop processing or frame insertion processing as necessary by the synthesis processing unit 117 in the subsequent stage. Thereafter, the data is written into the display memory 125 via the format conversion unit 121 and the display processing unit 123. The image signal output from the IP conversion processing unit 111 is written in the display memory 125 using the field frequency of the input NTSC signal as the frame frequency.

  On the other hand, an image signal is read from the display memory 125 with the frame frequency of the monitor (equal to the refresh rate) as the frame frequency, and an image based on the image signal is displayed on the monitor 127.

  Accordingly, the frame frequency for writing to the display memory 125 is the field frequency of the input NTSC signal, and the frame frequency for reading from the display memory 125 is the frame frequency of the monitor.

  When an image signal is to be written into the display memory 125, if the amount of the image signal stored in the display memory is greater than or equal to a predetermined value, the writing is waited and the amount of the image signal becomes less than a predetermined value. It is possible to write the image signal that was being written. Therefore, the time (waiting time) for which writing is waited can be detected from the time difference between the write request signal indicating that the image signal is to be written and the write permission signal for permitting writing.

  When this waiting time exceeds a predetermined value, it is determined that frame dropping is necessary.

  In the second embodiment, the image signal output from the first IP conversion processing unit 111B or the second IP conversion processing unit 111C passes through the second switch 133, the format conversion unit 121, and the display processing unit 123. Are written in the display memory 125. The image signal output from the first IP conversion processing unit 111B or the second IP conversion processing unit 111C is written in the display memory 125 using the field frequency of the input NTSC signal as the frame frequency.

  Therefore, also in the second embodiment, when the above-described waiting time becomes equal to or greater than a predetermined value, it is determined that frame dropping is necessary.

[Frame drop determination process (2) and frame insertion determination process]
Each field synchronization signal of the input NTSC signal is detected as an input frame synchronization signal, and each frame synchronization signal of the monitor 127 is detected as an output frame synchronization signal. When a monitor 127 having a frame frequency higher than the field frequency of the input NTSC signal is connected, the corresponding output frame synchronization signal is gradually delayed with respect to each input frame synchronization signal. Therefore, for each input frame synchronization signal, the delay of the corresponding output frame synchronization signal is a predetermined value (for example, 15 milliseconds if the input frame period is 16.7 milliseconds (1/60 seconds)) or more. At that time, it is determined that frame insertion is necessary.

  When a monitor 127 having a frame frequency lower than the field frequency of the input NTSC signal is connected, the output frame synchronization signal corresponding to each input frame synchronization signal gradually advances. Therefore, for each input frame synchronization signal, the advance of the corresponding output frame synchronization signal is a predetermined value (for example, 2 milliseconds if the input frame period is 16.7 milliseconds (1/60 seconds)) or more. At that time, it is determined that frame dropping is necessary.

  The second IP conversion processing unit 111C may have another configuration.

It is a block diagram which shows the structure of the system containing the frame frequency converter by Embodiment 1 of this invention. 2 is a flowchart for explaining an output process performed by an output processing unit shown in FIG. 1 when an output frame frequency is lower than an input frame frequency. It is a table | surface for demonstrating the operation example when the frame drop process is performed by the frame frequency converter by Embodiment 1 of this invention. 2 is a flowchart for explaining an output process performed by an output processing unit shown in FIG. 1 when an output frame frequency is higher than an input frame frequency. It is a table | surface for demonstrating the operation example when the frame insertion process is performed by the frame frequency converter by Embodiment 1 of this invention. It is a block diagram which shows the structure of the system containing the frame frequency converter by Embodiment 2 of this invention. 7 is a flowchart for explaining an output process performed by the output processing unit shown in FIG. 6 when the output frame frequency is lower than the input frame frequency. 7 is a flowchart for explaining an output process performed by an output processing unit shown in FIG. 6 when an output frame frequency is higher than an input frame frequency. It is a block diagram which shows the structure of the output process part by Embodiment 3 of this invention. 10 is a flowchart for explaining an output process performed by the output processing unit shown in FIG. 9 when the output frame frequency is lower than the input frame frequency. 10 is a flowchart for explaining an output process performed by the output processing unit shown in FIG. 9 when the output frame frequency is higher than the input frame frequency. It is a block diagram which shows the structure of the output process part by Embodiment 4 of this invention. 13 is a flowchart for explaining output processing performed by the output processing unit shown in FIG. 12 when the output frame frequency is lower than the input frame frequency. 13 is a flowchart for explaining output processing performed by the output processing unit shown in FIG. 12 when the output frame frequency is higher than the input frame frequency. 10 is a flowchart for explaining output processing performed by the output processing unit shown in FIG. 1 when the output frame frequency is lower than the input frame frequency in the fifth embodiment of the present invention. 16 is a flowchart illustrating details of an example of a frame drop start process illustrated in FIG. 15. It is a table | surface for demonstrating the operation example when the frame drop process is performed by the frame frequency converter by Embodiment 5 of this invention. It is a graph for demonstrating the operation example when the frame drop process is performed by the frame frequency converter by Embodiment 5 of this invention. 16 is a flowchart showing details of another example of the frame drop start process shown in FIG. 15. It is a block diagram which shows the structural example of the IP conversion process part shown in FIG. 1, or the 1st IP conversion process part shown in FIG. FIG. 21 is a first diagram for explaining an effect obtained by the IP conversion processing unit having the configuration shown in FIG. 20. FIG. 21 is a second diagram for explaining an effect obtained by the IP conversion processing unit having the configuration shown in FIG. 20. FIG. 21 is a flowchart for describing IP conversion processing performed by an IP conversion processing unit having the configuration shown in FIG. 20. FIG. FIG. 7 is a first diagram for explaining the principle of IP conversion processing performed by a second IP conversion processing unit shown in FIG. 6. FIG. 7 is a second diagram for explaining the principle of IP conversion processing performed by the second IP conversion processing unit shown in FIG. 6. It is a block diagram which shows the structural example of the 2nd IP conversion process part shown in FIG.

Explanation of symbols

101 NTSC decoder 103 Image data writing unit 105 Frame buffer group 107 Output processing unit 109 Image data reading unit 111 IP conversion processing unit 111B First IP conversion processing unit 111C Second IP conversion processing unit 115 Frame drop / frame insertion determination Unit 117 synthesis processing unit 119 temporary storage memory 121 format conversion unit 123 display processing unit 125 display memory 127 monitor 135 control unit

Claims (21)

In a frame frequency conversion method for generating an output moving image having a second frame frequency based on an input moving image having a first frame frequency,
A start determination step for determining whether it is necessary to start a frame dropping process or a frame inserting process for each frame of the input moving image;
A process execution step of performing the frame dropping process or the frame inserting process over a plurality of frames of the input moving image when it is determined in the start determining step that the frame dropping process or the frame inserting process needs to be started. When,
With
If the frame dropping process or the frame inserting process is completed, until it is determined that the start of the frame dropping process or the frame inserting process is necessary again in the start determination step that is continuously performed. A frame frequency conversion method characterized by pausing the frame dropping process or the frame inserting process. The frame frequency conversion method according to claim 1,
In the start determination step, it is determined whether or not it is necessary to start the frame dropping process or the frame insertion process based on a time difference between the frame synchronization signal of the input moving image and the frame signal of the output moving image. And a frame frequency conversion method. The frame frequency conversion method according to claim 1,
In the start determination step, the data of the input moving image is written therein according to the first frame frequency, and the data of the output moving image is temporarily read out from the data according to the second frame frequency. When the amount of accumulated moving image data exceeds a predetermined value, the time for waiting for writing of the input moving image data to the display memory exceeds a predetermined value. A frame frequency conversion method characterized by determining that a start is necessary. The frame frequency conversion method according to claim 1,
The frame dropping process or the frame inserting process over a plurality of frames of the input moving image includes:
Initializing a weight variable in a certain frame of the input moving image to a value corresponding to the number of frames to be subjected to the frame dropping process or the frame inserting process;
Changing the weight variable for each frame of the input video;
Generating a frame signal of an output moving image by weighting and adding a plurality of frame signals of the input moving image using the weight variable changed for each frame of the input moving image;
A frame frequency conversion method comprising: The frame frequency conversion method according to claim 1,
When it is determined in the start determining step that the frame drop process or the frame insert process needs to be started, the start frame of the frame drop process or the frame insert process is a moving image frame or a still image frame. A still image / moving image determination step for determining whether or not there is,
The frame dropping process or the frame inserting process over a plurality of frames of the input moving image only when the start frame of the frame dropping process or the frame inserting process is determined to be a moving image frame in the still image / moving image determining step. Do
When it is determined in the still image / moving image determination step that the start frame of the frame dropping process or the frame inserting process is a still image frame, another frame dropping process or another frame inserting process is performed. Frame frequency conversion method. The frame frequency conversion method according to claim 5, wherein
The other frame dropping process or the other frame inserting process is a process of discarding a certain frame of the input moving image or a process of overlapping a certain frame of the input moving image, respectively. Method. The frame frequency conversion method according to claim 1,
The frame dropping process or the frame inserting process over a plurality of frames of the input moving image according to the interval of the input frames determined to require the frame dropping process or the frame inserting process to be started in the start determining step. A frame frequency conversion method further comprising a parameter adjustment step of adjusting a parameter in The frame frequency conversion method according to claim 7,
In the parameter adjustment step, a frame frequency conversion method is characterized in that the number of frames to be subjected to the frame dropping process or the frame inserting process over a plurality of frames of the input moving image is adjusted. The frame frequency conversion method according to claim 8,
In the parameter adjustment step, it is determined that the number of frames subject to the frame dropping process or the frame inserting process over a plurality of frames of the input moving image needs to start the frame dropping process or the frame inserting process. A frame frequency conversion method, wherein adjustment is performed so that the input frame interval is equal to or less than the input frame interval. The frame frequency conversion method according to claim 8,
In the parameter adjustment step, it is determined that the number of frames subject to the frame dropping process or the frame inserting process over a plurality of frames of the input moving image needs to start the frame dropping process or the frame inserting process. A frame frequency conversion method characterized in that adjustment is made to be less than an input frame interval, thereby providing a period during which the frame drop processing or frame insertion processing is paused. In a frame frequency conversion device that generates an output moving image having a second frame frequency based on an input moving image having a first frame frequency,
Start determination means for determining whether it is necessary to start frame drop processing or frame insertion processing for each frame of the input moving image;
Processing execution means for performing the frame dropping process or the frame inserting process over a plurality of frames of the input moving image when the start determining means determines that the frame dropping process or the frame inserting process needs to be started. When,
With
When the frame dropping process or the frame insertion process is completed, the process execution unit is required to start the frame dropping process or the frame insertion process again with the start determination unit operating continuously. The frame frequency conversion apparatus, wherein the frame dropping process or the frame inserting process is paused until it is determined that The frame frequency conversion device according to claim 11, wherein
The start determining means determines whether or not it is necessary to start the frame dropping process or the frame inserting process based on a time difference between the frame synchronization signal of the input moving image and the frame signal of the output moving image. A frame frequency converter characterized by the above. The frame frequency conversion device according to claim 11, wherein
The start determination means temporarily stores the input moving image data in accordance with the first frame frequency, and temporarily stores it in a display memory from which the output moving image data is read out in accordance with the second frame frequency. When the amount of accumulated moving image data exceeds a predetermined value, the time for waiting for writing of the input moving image data to the display memory exceeds a predetermined value. A frame frequency conversion apparatus characterized by determining that a start is necessary. The frame frequency conversion device according to claim 11, wherein
The process execution means includes
Means for initializing a weight variable in a certain frame of the input moving image to a value corresponding to the number of frames to be subjected to the frame dropping process or the frame inserting process;
Means for changing the weight variable for each frame of the input moving image;
Means for generating a frame signal of an output moving image by weighting and adding a plurality of frame signals of the input moving image using the weight variable changed for each frame of the input moving image;
A frame frequency conversion device comprising: The frame frequency conversion device according to claim 11, wherein
When the start determination unit determines that the frame drop process or the frame insertion process needs to be started, the start frame of the frame drop process or the frame insertion process is a moving image frame or a still image frame. A still image / moving image judging means for judging whether or not there is,
Only when the start frame of the frame dropping process or the frame inserting process is determined to be a moving image frame by the still image / moving image determining means, the process executing means is configured to drop the frame dropping over a plurality of frames of the input moving image. Process or insert the frame,
The image processing apparatus further includes means for performing another frame dropping process or another frame inserting process when the still image / video determining unit determines that the start frame of the frame dropping process or the frame inserting process is a still image frame. A frame frequency converter characterized by the above. The frame frequency conversion device according to claim 15,
The other frame dropping process or the other frame inserting process is a process of discarding a certain frame of the input moving image or a process of overlapping a certain frame of the input moving image, respectively. apparatus. The frame frequency conversion device according to claim 11, wherein
The frame dropping process or the frame inserting process over a plurality of frames of the input moving image according to the interval of the input frames determined by the start determining means to start the frame dropping process or the frame inserting process. A frame frequency conversion apparatus, further comprising parameter adjustment means for adjusting the parameters in (1). The frame frequency conversion device according to claim 17,
The frame frequency conversion device, wherein the parameter adjusting means adjusts the number of frames to be subjected to the frame dropping process or the frame inserting process over a plurality of frames of the input moving image. The frame frequency conversion device according to claim 18,
The parameter adjusting means determines that the number of frames subject to the frame dropping process or the frame inserting process over a plurality of frames of the input moving image needs to start the frame dropping process or the frame inserting process. A frame frequency conversion apparatus that performs adjustment so as to be equal to or less than an input frame interval. The frame frequency conversion device according to claim 18,
The parameter adjusting means determines that the number of frames subject to the frame dropping process or the frame inserting process over a plurality of frames of the input moving image needs to start the frame dropping process or the frame inserting process. The frame frequency conversion apparatus is characterized in that an adjustment is made so that the interval is less than an input frame interval, thereby providing a period during which the frame dropping process or the frame inserting process is paused.   A frame frequency conversion program for causing a computer to perform the frame frequency conversion method according to any one of claims 1 to 10.