TW202409960A - Method for handling irregular input for video processor - Google Patents

Abstract

An operational method for a video processor includes the following steps: receiving a series of input frames; calculating a buffer stage value according to the series of input frames, wherein the buffer stage value corresponds to a status of the input frames stored in a frame buffer of the video processor; and selecting a frame set from the input frames stored in the frame buffer for generating an interpolated frame as an output frame to be output by the video processor according to the buffer stage value.

Description

Method for processing irregular input of a video processor

The present invention refers to a method for motion estimation and motion compensation (MEMC), and in particular, to a method that can be used to handle irregular inputs in motion estimation and motion compensation.

Motion Estimation and Motion Compensation (MEMC) is a technology used for frame interpolation, which allows a series of image frames to be displayed at a higher frame rate. For example, if a 30 Hz original video (such as a movie video) needs to be displayed at a frequency of 60 Hz, an interpolation frame needs to be added between every two adjacent input frames of the original video to achieve twice the frame rate.

Assuming that the original video from the video provider has a frame rate of 24 Hz and the image to be displayed on the display panel is 60 Hz, frame conversion technology is used to convert every two input frames into five output frames. In order to avoid judder on the output picture, interpolation can be performed using the cadence of film mode 32, in which two input frames can be combined to generate at least one interpolated frame. Motion estimation and motion compensation technology allow the video processor to determine the appropriate input frame and perform interpolation using the correct phase coefficient to output a smooth picture.

Traditionally, when the video processor obtains the information of the input frame and the output frame and determines the video mode based on it, a correspondence table recording the input frame and related phase information can be used to perform interpolation, and the video processor can perform interpolation by looking up the table. Generate interpolated frames. However, if the input frame does not follow the input rules but the interpolation operation is still based on a lookup table, the video processor may take the wrong source input frame and/or use the wrong phase coefficient to perform interpolation, resulting in the output The screen appears to be shaking. For example, if the video processor in a mobile phone receives input frames according to the instruction mode of the Mobile Industry Processor Interface (MIPI) specification, the input timing of the input frames is easily affected by the Central Processing Unit (Central Processing Unit). , CPU) or graphics processing unit (Graphics Processing Unit, GPU) state interference, so when the graphics processing unit is too busy, it will cause a delay in the input image. Delays or irregularities in input timing are often difficult to predict, and therefore conventional lookup table methods may not be able to properly handle such irregular input problems.

For example, in an application that converts 24 Hz input video to 60 Hz output video, every 2 input frames are used to generate 5 output frames. Therefore, the input update signal used to indicate the receipt of the input frame usually has a repeating sequence. "10010", as shown in Table 1. By querying a reference table, it can be obtained that the phase step is equal to 2/5, so that the phase coefficients form a recursive sequence of 0, 2/5, 4/5, 1/5 and 3/5, and the interpolation frame uses the current input The frame and the previous input frame are generated with corresponding phase coefficients. index 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 input frame A B C D E F Enter update signal 1 0 0 1 0 0 1 0 1 0 1 0 0 1 0 phase step 2/5 2/5 2/5 2/5 2/5 2/5 2/5 2/5 2/5 2/5 2/5 2/5 2/5 2/5 2/5 current frame A A A B B B C C D D E E E F F previous frame Z Z Z A A A B B C C D D D E E Phase coefficient 0 2/5 4/5 1/5 3/5 0 2/5 4/5 1/5 3/5 0 2/5 4/5 1/5 3/5 Table 1

However, as shown in Table 1, input frame C is expected to arrive during the output frame at index 6, but is delayed until the next output frame at index 7. According to the reference table, since the input frame C has not yet arrived, the video processor will fetch the input frames B and A from the specific frame buffer to generate the interpolated frame for index 6. Therefore, the same frame group is selected from index 5 to index 6 to perform interpolation, but the phase coefficient changes from 3/5 to 0, causing the output picture to roll back abnormally and easily causing image jitter in the output picture.

The same problem can also occur in applications with regular or pulldown, where the input frame is received as a repeated frame at 60 Hz and the output frame is output at 60 Hz, as shown in Table 2. index 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Input frame A1 A2 A3 B1 B2 B3 C1 C2 D1 D2 E1 E2 E3 F1 F2 Input update signal 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 Frame Difference 1 0 0 1 0 0 1 0 1 0 1 0 0 1 0 Phase Step 2/5 2/5 2/5 2/5 2/5 2/5 2/5 2/5 2/5 2/5 2/5 2/5 2/5 2/5 2/5 Current frame A A A B B B C C D D E E E F F Previous frame Z Z Z A A A B B C C D D D E E Phase coefficient 0 2/5 4/5 1/5 3/5 0 2/5 4/5 1/5 3/5 0 2/5 4/5 1/5 3/5 Table 2

In this example, input frames A2 and A3 are generated by copying input frame A1, input frames B2 and B3 are generated by copying input frame B1, input frame C2 is generated by copying input frame C1, and And so on. The frame difference value indicates whether the currently received input frame is different from the previous input frame or is a copy of the previous input frame. Likewise, due to the delay of input frame C1, an abnormal rollback of the output picture occurs during the output frame of index 6.

In view of this, there is a real need to improve learning technology.

Therefore, the main purpose of the present invention is to propose an image processing method for a video processor to handle the problem of irregular input by judging abnormal situations and correspondingly selecting frame groups and calculating phase steps, thereby solving the above-mentioned problems. problem.

An embodiment of the present invention discloses an operating method for a video processor, which includes the following steps: receiving a series of input frames; calculating a buffer stage value according to the series of input frames, wherein the buffer stage value The value corresponds to the state of the input frame stored in a frame buffer in the video processor; and based on the buffer level value, a frame group is selected from the input frame stored in the frame buffer to generate an interpolation frame , as an output frame output by the video processor.

Another embodiment of the present invention discloses an operating method for a video processor, which includes the following steps: receiving a series of input frames; and based on an input frame rate of one of the series of input frames and a series of outputs output by the video processor. The relative relationship between the output frame rate of one frame is used to determine an expected input spacing value; to determine whether an actual input spacing value is different from the expected input spacing value; when it is determined that the actual input spacing value is different from the expected input spacing value, calculate an Phase step and obtain the phase step, the phase step is different from a normal phase step determined according to the relative relationship between the input frame rate and the output frame rate; and using the calculated phase step, according to a The frame group is selected to generate an interpolated frame, which is output as an output frame in the series of output frames.

Figure 1 is a schematic diagram of a display system 10 according to Embodiment 1 of the present invention. As shown in FIG. 1 , the display system 10 can be, for example, a television or a screen, which can receive and display original video having a series of image frames provided by a video providing unit 12 , which can be, for example, a digital video disc (digital video disc). Digital Video Disc (DVD) player or video streaming service provider, which can communicate with the display system 10 through a wired or wireless network. The display system 10 includes a video processor 106, a display driving device 108 and a display panel 110. Generally, the frame rate of the original video is different from the frame rate to be displayed on the display panel 110 . The video processor 106 may include a video control integrated circuit (Video Controller IC) and a frame rate converter IC (Frame Rate Converter IC, FRC IC) for converting original video with a usually low frame rate to generate A series of output frames with a higher frame rate are displayed through the display panel 110 . Alternatively, the video processor 106 can be a video control integrated circuit with a frame rate conversion function. That is, the frame rate converter 104 shown in Figure 1 can be regarded as an independent frame rate conversion integrated circuit or located in the video control integrated circuit. Frame rate conversion circuit inside the body circuit. In this example, when the frame rate converter 104 receives an input video of 24 Hz with 2 input frames A and B, it can convert the 2 input frames A and B into 5 image frames A, A of 60 Hz. , A, B, B, and the video processor 106 can perform motion estimation and motion compensation (MEMC) according to these image frames to generate appropriate interpolation frames.

In another embodiment, the video processor 106 may include a graphics processing unit (GPU) and an independent frame rate conversion integrated circuit, or the video processor 106 may be a graphics processing unit including a built-in frame rate conversion circuit.

The video processor 106 may also include a frame buffer 105 . The frame buffer 105 can be used to store the received input frames, so that the video processor 106 can obtain the selected input frames from the frame buffer 105 to perform interpolation. In this example, frame buffer 105 is integrated with video processor 106 (eg, a video control integrated circuit). In another embodiment, video processor 106 may access input frames from one or more independent frame buffers, which are external memory devices separate from video processor 106 .

The display driver 108 can convert the image data into a data voltage signal, and drive the display panel 110 to display the image frame through the data voltage signal. The display driver 108 can include a timing controller, a source driver, a gate driver, and/or any other device that can be used to drive the display panel 110. The display panel 110 can be of any type, such as a liquid crystal display panel (LCD), a light-emitting diode (LED) display, a plasma display panel (PDP), but is not limited thereto.

As described above, conventional video processors can obtain a frame group and generate interpolated frames using phase coefficients determined based on a reference table, where the selected frames and their associated phase coefficients are determined based on the input frame rate and output The frame rate is determined in advance by referring to the information recorded in the table. Therefore, if the input frame fails to follow the expected input timing, the output image frame will not be generated normally. In order to solve or alleviate this problem, the present invention proposes a method that can dynamically determine the frame group and phase step/coefficient for interpolation within each output frame period, by appropriately selecting the current frame for interpolation and previous frames and recalculates the phase steps and phase coefficients to make the output video smoother and minimize the impact of irregular input timing.

FIG. 2 is a flowchart of an image processing process 20 according to an embodiment of the present invention. The image processing process 20 can be implemented in a video processor, such as the video processor 106 in FIG. 1. As shown in FIG. 2, the image processing process 20 includes the following steps:

Step 200: Start.

Step 202: Receive a series of input frames.

Step 204: Calculate a buffer stage value based on the series of input frames, wherein the buffer stage value corresponds to the status of the input frames stored in a frame buffer in the video processor.

Step 206: According to the buffer level value, select a frame group from the input frames stored in the frame buffer to generate an interpolation frame as an output frame output by the video processor.

Step 208: End.

According to the image processing process 20, the video processor may receive a series of input frames (step 202). In one embodiment, the series of input frames may be received by the video processor from a front-end device, such as the video providing unit 12 in FIG. 1 .

During each output frame with or without receiving an input frame, the video processor may calculate a buffer level value based on the reception of the input frame (step 204). The buffer level value corresponds to the status of the input frame stored in a frame buffer of the video processor. More specifically, the buffer level value may represent the number of input frames in the frame buffer that have not been used to generate interpolated frames through interpolation operations. The video processor can use the buffer level value to monitor the frame buffer's input frame storage status. The video processor then selects a frame group from the frame buffer according to the buffer level value to generate an interpolated frame as an output frame (step 206).

Table 3 lists the buffer level values during multiple consecutive output frames under the input timing of a series of input frames. Table 3 also lists the frame group used to generate interpolation frames during each output frame (including a current frame and a previous frame) and their corresponding phase coefficients, where each output frame period is represented by an index value. The interpolation frame can be used as an output frame and is output by the video processor to the back-end display driver device and display panel. A new phase start indication and a buffer level value are also listed in Table 3. Table 3 shows the conversion of 24 Hz input video to 60 Hz output video with a phase step of 2/5 and a recursive sequence of phase coefficients of 0, 2/5, 4/5, 1/5 and 3. /5. index 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 input frame A B C D E F Enter update signal 1 0 0 1 0 1 0 1 0 0 1 0 0 1 0 New phase start indication 1 1 1 1 1 1 buffer level value 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 current frame A A A B B C C C D D E E E F F previous frame Z Z Z A A B B B C C D D D E E Phase coefficient 0 2/5 4/5 1/5 3/5 0 2/5 4/5 1/5 3/5 0 2/5 4/5 1/5 3/5 table 3

Specifically, the video processor receives a series of input frames A, B, C, D, E and F.... The input update signal may be used to indicate the reception of an input frame, where a value of 1 indicates that an input frame was received during the output frame period, and a value of 0 indicates that no input frame was received during the output frame period. When the front-end video providing unit sends an input frame to the video processor, the video providing unit may provide an input update signal to the video processor to instruct the transmission of the input frame. The new phase start indication can be used to indicate the status of the phase coefficient, where a value of 1 indicates that the phase coefficient of the previous output frame plus the phase step reaches 1 or exceeds 1, thus re-starting the accumulation of the new phase period from 0. The buffer level value can be preset to a default value, which is 0 in this example. Since the buffer level value represents the number of input frames in the frame buffer that have not yet been used to generate interpolated frames, the buffer level value can be incremented by 1 when a new input frame arrives, and the phase coefficient reaches 1 at the beginning of a new phase period And when accumulating again from 0, the buffer level value can be reduced by 1. In this example, the arrival of a new input frame may be indicated by an input update signal, and a new phase period may be indicated by a new phase start indication.

More specifically, if a new input frame is received, the buffer level value can be increased by 1 when the new input frame enters the frame buffer. If the phase coefficient reaches 1 and starts accumulating again from 0, it means that the interpolation frame is generated based on the next input frame group. At this time, since an input frame is added to the frame buffer to generate the interpolation frame, it can be Decrease the buffer level value by 1.

For example, during the output frame period of index 4, input frame B arrives, so the buffer level value is increased by 1; at the same time, a new phase starts and the phase coefficient starts to accumulate again from 0, so the buffer level value is reduced by 1. The combination of the addition and subtraction operations results in the buffer level value remaining unchanged, that is, it remains at 0 during the output frame period of index 4. Similarly, during the output frame period of index 6, input frame C arrives, so the buffer level value is increased by 1; at the same time, a new phase starts and the phase coefficient returns to 0, so the buffer level value is reduced by 1. The combination of the addition and subtraction operations results in the buffer level value remaining unchanged and continuously at 0.

However, within the output frame period at index 8, input frame D arrives earlier than the expected arrival time (i.e., index 9). In this case, the input update signal indicates the arrival of a new input frame, so the buffer level value is increased by 1, and the phase coefficient is increased to 4/5 but a new phase period is not entered. As a result, the buffer level value becomes 1 during the output frame at index 8, and the video processor will select the input frame group used to generate the interpolation frame based on the buffer level value.

More specifically, if the buffer level value is equal to a preset value (e.g., 0), the video processor may select a newly arrived input frame as a current frame and select a previously received input frame (earlier than the newly arrived input frame) as a previous frame to generate an interpolation frame. For example, during an output frame of index 4, the buffer level value is equal to 0, the newly arrived/received input frame B is selected as the current frame and its previous frame A is selected as the previous frame to perform interpolation to generate an interpolation frame. During the output frame of index 6, the buffer level value is equal to 0, the newly arrived/received input frame C is selected as the current frame and its previous frame B is selected as the previous frame to perform interpolation to generate the interpolation frame.

If the buffer level value is greater than the default value (for example, increased to 1), the video processor can select the previously received input frame as the current frame and select another previously received input frame (which is earlier than the input frame selected as the current frame). ) as the previous frame to generate the interpolated frame. For example, during the output frame at index 8, the buffer level value is equal to 1, the previously received input frame C is selected as the current frame and its previous frame B is selected as the previous frame, used to perform interpolation to produce the interpolation Fill frame.

Note that, during the next output frame period (i.e., index 9), the phase coefficients are re-accumulated from 0 and a new phase period begins, so the buffer level value is reduced to 0. In this case, the video processor can select the newly arrived input frame D as the current frame and select its previous frame C as the previous frame to generate the interpolated frame.

In the conventional alternative of using a reference table to perform interpolation, the video processor must take out the newly arrived input frame and its previous input frame as a frame group to perform interpolation. In this interpolation method, one input frame is larger than expected. If the time arrives earlier, an abnormal rollback of the screen may occur. In contrast, according to embodiments of the present invention, the video processor can select the input frame group for interpolation based on the buffer level value. Therefore, the selected input frame group will be correct, especially for an input When frames arrive early, interpolated frames can be generated normally and displayed smoothly.

In one embodiment, the buffer level value can also be used to handle the input frame delay situation. With a buffer level value of 0 (meaning there are no input frames in the frame buffer that have not yet been used to perform interpolation), when the phase coefficient is to exceed 1 but there are no new input frames arriving, the video processor cannot obtain the new Arriving input frames to perform interpolation. Such a situation is called "buffer underflow", which means that an input frame is expected to be taken during an output frame to generate an interpolation frame but the input frame has not been received during the output frame. When a buffer underflow occurs, the video processor can still generate interpolated frames using the same input frame group as during the previous output frame (i.e., the frame group remains unchanged), and set the phase coefficients to force the maximum value.

Table 4 shows the case where a buffer underflow occurs due to the delay of input frame D. During the output frame period of index 9, input frame D is expected to be received, but no input frame is actually received, and the phase coefficient is about to exceed 1 and enter the next phase cycle. At the same time, the buffer level value is equal to the minimum value 0, which means that no new input frame is stored in the frame buffer that can be used as a new frame group for interpolation, so the system determines that a buffer underflow occurs. In this case, the frame group used for interpolation remains unchanged, where input frame C is still selected as the current frame and input frame B is still selected as the previous frame. The phase coefficient reaches its maximum value, such as 127/128, which is the maximum possible value less than 1. The use of the maximum phase coefficient makes the interpolated frame generated similar to the input frame C before the input frame D arrives. In this way, abnormal picture regression can be avoided and the impact of input frame delay can be minimized. index 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Input frame A B C D E Input update signal 1 0 0 1 0 1 0 0 0 0 1 0 0 1 0 New phase start indication 1 1 1 1 1 Buffer level value 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Current frame A A A B B C C C C C D D D E E Previous frame Z Z Z A A B B B B B C C C D D Phase coefficient 0 2/5 4/5 1/5 3/5 0 2/5 4/5 127/128 127/128 0 2/5 4/5 1/5 3/5 Table 4

It is worth noting that the current frame and previous frames used to perform interpolation are taken from the frame buffer. Assume that the video processor can access a total of N frame buffers (or a frame buffer that can accommodate N input frames). If the buffer level value is equal to N-1, the selected frame group contains input that is earlier than the new arrival. The (N-1)th input frame before the frame (as the current frame) and the Nth input frame earlier than the newly arrived input frame (as the previous frame). The latter will be overwritten by the newly arrived input frame, causing the video processor to obtain the wrong input frame to perform interpolation.

The above situation is called "buffer overflow", which means that a previously received input frame stored in the frame buffer and expected to be used to generate the interpolation frame is overwritten by the newly arrived/received input frame. To solve the problem of buffer overflow, when the buffer level value reaches N-1 (where N is the number of frame buffers of the video processor, indicating that the frame buffer of the video processor has the ability to store N input frames), the buffer level value can be forced to decrease by 1, and then the input frame group used to perform interpolation is selected accordingly. That is, the video processor can select the input frames stored in the frame buffer but not overwritten by the currently arriving input frame as a frame group to perform interpolation, and the phase coefficients are forced to be re-accumulated from 0.

Table 5 shows the case where a buffer overflow occurs due to the early arrival of input frames F and G. In this example, the number of frame buffers is 3, which means that the frame buffer of the video processor can store up to 3 input frames, and the frame buffer may include 3 buffer units B0, B1, and B2, as shown in Table 5. Therefore, a buffer overflow occurs when the buffer level value reaches 2. index … 6 7 8 9 10 11 12 13 14 15 16 17 Input frame … C D E F G H Input update signal … 1 0 0 1 0 1 1 1 0 0 1 0 New phase start indication … 1 1 1 0 0->1 1 Buffer level value … 0 0 0 0 0 0 1 2->1 1 1 1 1 Current frame … C C C D D E E F F F G G Previous frame … B B B C C D D E E E F F Phase coefficient … 0 2/5 4/5 1/5 3/5 0 2/5 0 2/5 4/5 1/5 3/5 Buffer unit B0 … A A A D D D D G G G G G Buffer unit B1 … B B B B B E E E E E H H Buffer unit B2 … C C C C C C F F F F F F table 5

In detail, the received input frames are sequentially and iteratively written into the buffer units B0, B1 and B2. In this example, input frame D is written to buffer unit B0 to overwrite input frame A, input frame E is written to buffer unit B1 to overwrite input frame B, input frame F is written to buffer unit B2 to overwrite input frame C, Input frame G is written to buffer unit B0 to overwrite input frame D. During the output frames at indexes 12 and 13, input frames F and G arrive faster than expected, and the buffer level value is about to increase to 2 when input frame G arrives. At this time, it is determined that a buffer overflow occurs and the buffer Input frame D in unit B0 is overwritten by input frame G. In this case, the buffer level value is reduced from 2 to 1, and the video processor selects a frame group based on the reduced buffer level value of 1, which selects input frame F as the current frame and input frame E as the previous frame to perform interpolation. Supplement. At the same time, according to the new phase start indication, the phase coefficient is set to force it to enter a new phase period and re-accumulate from 0. Then, the arrival of subsequent input frames H returns to normal timing. Therefore, the video processor can select the frame group in the normal sequence according to the buffer level value, and the phase coefficients continue to accumulate at the normal phase step 2/5.

In the same example, if the frame group for interpolation is selected without considering the buffer overflow, during the output frame of index 13, the video processor expects to fetch the input frame E stored in the buffer unit B1 as the current frame and expects to fetch the input frame D stored in the buffer unit B0 as the previous frame to perform interpolation (where the phase coefficient is 4/5). However, the input frame D stored in the buffer unit B0 has been overwritten by the input frame G that arrived earlier, causing an overflow, resulting in it fetching an erroneous input frame to perform interpolation. In contrast, the buffer overflow judgment and related operation method proposed in the present invention can ensure that suitable input frames are selected as frame groups to perform interpolation, making the output picture smoother.

According to the buffer level value, the video processor can select a suitable input frame to perform interpolation, and under a limited number of frame buffers, the video processor can take appropriate countermeasures when a buffer overflow or underflow occurs. The judgment of buffer overflow or underflow can be performed dynamically during each output frame, so the buffer level value should be updated at any time to adaptively select the frame group for interpolation. Even in the case of severe input irregularities and input frames continue to arrive quickly or no input frames arrive for a long time, frame interpolation can be handled well when a buffer overflow or underflow occurs. The above method enables the video processor to select the best frame group to perform interpolation, which can ensure that the output picture will not be abnormally regressed. In addition, in order to make the output picture smoother, the present invention proposes a new method for adjusting or modifying the phase step when irregular input occurs.

FIG. 3 is a flowchart of an image processing process 30 according to an embodiment of the present invention. The image processing process 30 can be implemented in a video processor, such as the video processor 106 in FIG. 1. As shown in FIG. 3, the image processing process 30 includes the following steps:

Step 300: Start.

Step 302: Receive a series of input frames.

Step 304: Determine an expected input spacing value based on the relative relationship between the input frame rate of one of the series of input frames and the output frame rate of one of the series of output frames output by the video processor.

Step 306: Determine whether an actual input spacing value is different from the expected input spacing value.

Step 308: When it is determined that the actual input spacing value is different from the expected input spacing value, calculate a phase step and obtain the phase step. The phase step is different from a normal phase step determined based on the relative relationship between the input frame rate and the output frame rate. Phase steps.

Step 310: Use the calculated phase step to generate an interpolation frame according to a selected frame group, and output it as an output frame in the series of output frames.

Step 312: End.

According to the image processing flow 30, the video processor may receive a series of input frames (step 302). In one embodiment, the series of input frames may be received by the video processor from a front-end device, which may be, for example, the video providing unit 12 in FIG. 1.

For each received input frame, the video processor may generate an expected input spacing value (hereinafter referred to as the "expected value") and an actual input spacing value (hereinafter referred to as the "actual value"). Specifically, the expected value is determined based on the relative relationship between the input frame rate of the input frame received by the video processor and the output frame rate of the output frame output by the video processor (step 304). For example, in an application that converts a 30 Hz input video to a 60 Hz output video, the expected value of each input frame is 2; in an application that converts a 24 Hz input video to a 60 Hz output video, a The expected values for a series of input frames are 3 and 2 alternating.

In addition, the video processor can determine the actual value based on the input timing of the input frame, and continuously monitor the actual value to determine whether each actual value is the same or different from the corresponding expected value (step 306). If the input frame arrives at the normal timing of the input sequence, the actual value will continue to be the same as the corresponding expected value; if an input frame arrives earlier or later than its expected arrival time, the irregular input timing will cause the actual value to be different from the expected value.

When the actual value is the same as the expected value, the normal phase step can be used. For example, in an application that converts a 24 Hz input video to a 60 Hz output video, the video processor can use a phase step of 2/5; and in an application that converts a 30 Hz input video to a 60 Hz output video , the video processor can use a phase step of 1/2. On the other hand, when the video processor determines that the actual value is different from the expected value, the phase step can be recalculated according to the irregular input timing (step 308). The calculated phase step should be different from the phase step based on the input frame. The normal phase step is determined by the relative relationship between the frame rate and the output frame rate. The video processor then uses the calculated phase step to generate an interpolated frame according to the selected frame group (which can be determined according to the buffer level value, as explained in the above embodiment), and outputs it as an output frame (step 310 ). In other words, under normal input timing, the phase step has a normal value; and when irregular input timing occurs, the phase step needs to be recalculated and has another value different from its normal value. The value of the phase step can be dynamically calculated based on the actual input timing of the input frame, as detailed below.

Table 6 lists the expected and actual values corresponding to a series of input frames received during multiple consecutive output frames. Table 6 shows the input timing of the input frame and includes various parameters, such as the input update signal, the current frame and the previous frame (i.e., frame group) used to generate the interpolated frame, and the phase coefficient used for interpolation. These parameters are the same as the contents of the previous table and will not be described in detail here. index 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 input frame A B C D E F Enter update signal 1 0 0 1 0 1 0 0 1 0 1 0 0 1 0 frame count value 0 1 2 0 1 0 1 2 0 1 0 1 2 0 1 actual value 2 3 2 3 2 3 expected value 2 3 2 3 2 3 current frame A A A B B C C C D D E E E F F previous frame Z Z Z A A B B B C C D D D E E Phase coefficient 0 2/5 4/5 1/5 3/5 0 2/5 4/5 1/5 3/5 0 2/5 4/5 1/5 3/5 Table 6

Table 6 also lists a frame count value that can be used to count the number of output frame periods since the previous input frame was received. The frame count value is reset to 0 when a new input frame arrives, and then starts counting, using the newly arrived input frame as the previous input frame. The frame count is a value that represents the number of output frame periods between the newly arrived current input frame and the previous input frame.

Therefore, based on the input frames expected to be received and the input frames actually received, the counting results of the frame count values can be used to calculate the expected value and the actual value respectively. More specifically, the expected value may represent the amount of output frame period that is expected to elapse between receipt of the previous input frame and receipt of the current input frame. In an embodiment where the input frame rate is 24 Hz and the output frame rate is 60 Hz, the expected values are 3 and 2 alternating. For example, the expected value corresponding to input frame A is 2, the expected value corresponding to input frame B is 3, the expected value corresponding to input frame C is 2, and so on.

The actual value may represent the number of output frame periods that have actually elapsed between the reception of the previous input frame and the reception of the current input frame. Therefore, the actual value is updated when a new input frame arrives. In the embodiment shown in Table 6, the input frames arrive following normal timing, so the actual values are all the same as the expected values. In this case, the video processor may perform interpolation using a phase coefficient calculated based on a normal phase step (i.e., 2/5), which is determined based on the relative relationship between the input frame rate of 24 Hz and the output frame rate of 60 Hz.

Table 7 shows the situation in which the actual value is different from the expected value due to the irregular input timing of the input frame in the embodiment of the present invention. In Table 7, the input frame F arrives early during the output frame period of index 13, causing the actual value of the corresponding input frame F to be different from the expected value. More specifically, according to the input timing corresponding to the 24 Hz to 60 Hz conversion application, the expected value of the input frame F is 3, but the early arrival of the input frame F makes the actual value of the input frame F 2. When the video processor detects that the actual value is different from the expected value, it can recalculate the phase step and obtain a phase step equal to 9/20, and then generate the subsequent phase coefficient according to the recalculated phase step 9/20. index … 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Input frame … C D E F G H Input update signal … 1 0 0 1 0 1 0 1 0 0 1 0 1 0 0 Frame count value … 0 1 2 0 1 0 1 0 1 2 0 1 0 1 2 Actual value … 2 3 2 2 3 2 Expected value … 2 3 2 3 3 2 New phase start indication … 1 1 1 1 1 1 Remaining input frames … 2 1 2 1+1 1 2 Remaining output frames … 5 4 3 2 1 5 4 3 2+2 3 2 1 5 4 3 Phase Step … 2/5 2/5 2/5 2/5 2/5 2/5 2/5 2/5 2/5 9/ 20 9/ 20 9/ 20 9/ 20 2/5 2/5 Current frame … C C C D D E E E F F G G H H H Previous frame … B B B C C D D D E E F F G G G Phase coefficient … 0 2/5 4/5 1/5 3/5 0 2/5 4/5 1/5 13/20 2/ 20 11/20 0 2/5 4/5 Table 7

After a few output frames using the recalculated phase step 9/20, the video processor knows that the phase coefficient has returned to 0 and the input frames are back to following normal timing. Therefore, the video processor restarts interpolation with the appropriate frame group using the normal phase step 2/5.

Table 7 lists two other parameters: the remaining input frames and the remaining output frames. The remaining input frames represent the number of remaining input frames used to generate interpolation frames within a cycle determined by the relative relationship between the input frame rate and the output frame rate; while the remaining output frames represent the number of remaining output frames to be generated and output within the same cycle. In other words, the remaining input frames refer to the input frames to be used for interpolation within the remaining output frame period of this cycle, and the remaining output frames refer to the output frames to be output within the remaining output frame period of this cycle. In an embodiment where the input frame rate is 24 Hz and the output frame rate is 60 Hz, each cycle includes 5 output frame periods. Therefore, in the cycle where the input frame arrives normally, the remaining input frame number starts to count down from 2 with the new phase start indication (the new phase start indication represents that the phase enters a new phase cycle and the frame group used for interpolation is updated), and the remaining output frame number starts to count down from 5 in each output frame period where an output frame is output.

When input frames arrive irregularly, the video processor can calculate the phase step based on the number of remaining input frames and the number of remaining output frames. For example, during the output frame period of index 13, the video processor learns that the actual value is different from the expected value, and then begins to execute the process of recalculating the phase step. The video processor can first update the values of the remaining input frames and the remaining output frames based on the current status of the input frames, output frames and phase coefficients.

More specifically, the video processor may add 1 to the remaining input frame number to generate an updated remaining input frame number, and add the current actual value (i.e., the actual input spacing value) to the remaining output frame number to generate an updated remaining output frame number. In the embodiment of Table 7, the above operation is performed during the output frame period of index 14, wherein the remaining input frame number is added by 1, and the remaining output frame number is added by 2, which is the actual value corresponding to the input frame F that arrives early.

Therefore, the phase step (Phase_Step) can be obtained by subtracting the current phase coefficient (Present_Phase) from the number of remaining input frames to be updated (Rest_Input_Frame_Cnt) and then divided by the number of remaining output frames to be updated (Rest_Output_Frame_Cnt). The formula is as follows: .

It is worth noting that the normal phase step is basically determined based on the relative relationship between the input frame rate and the output frame rate. Through the phase step, motion estimation and motion compensation operations enable the image information of one or more input frames to be evenly distributed on multiple output frames according to their frame rates, thereby generating an output picture with a higher frame rate and smoothness. In the embodiment of the present invention, the recalculation of the phase step also conforms to a similar concept, that is, the image information of irregularly arriving input frames should also be evenly distributed on multiple output frames, so that the impact of irregular arrival on the output video is minimized, and the calculation based on the remaining input frame number and the remaining output frame number helps to achieve this goal.

For example, in the embodiment of Table 7, the input frame F arrives early during the output frame of index 13, so the phase coefficient used to generate the interpolated frame should be accelerated to cope with the early arrival of the input frame F, and thus a larger phase step should be used. In this example, the phase step obtained is equal to 9/20, which is slightly larger than the normal phase step of 2/5. In detail, the remaining input frame number is used to monitor the status of the input, and the remaining input frame number plus 1 indicates that the next input frame G is taken into consideration to recalculate the phase coefficient. Next, the remaining input frame number is updated to deduct the current phase coefficient (i.e., 1/5), which means that the phase value that has passed is excluded from the calculation, leaving only the remaining phase value. The remaining output frames are used to monitor the status of the output. Accordingly, as the denominator of the formula used to calculate the phase step, the remaining output frames need to be added to the actual value (i.e. 2) to generate the updated remaining output frames, which is equal to 4, which means that the next phase cycle corresponding to the input frame G is taken into account to disperse the impact of irregular input. In this example, the phase step can be calculated and obtained as follows: .

In this case, this phase step is available for the next 4 output frame periods, which is indicated by updating the value 4 for the number of remaining output frames. In other words, the calculated phase step continues to be valid until the number of remaining output frames approaches 0 and returns to its maximum value of 5. For example, in the embodiment of Table 7, phase step 9/20 is used for the four output frame periods of indexes 15 to 18, and the phase coefficient will return to 0. Then, since subsequent input frames are received with normal timing, the video processor restarts interpolation using normal phase step 2/5.

In this example, the irregular input frame F arrives during the output frame at index 13. The video processor then updates the remaining input frame number and the remaining output frame number during the next output frame at index 14 to calculate and obtain the phase step. level. This calculated phase step then begins for the next output frame period at index 15. Based on the relevant rules of this phase step calculation, the impact of irregular input can be evenly distributed to multiple output image frames, making the output video smoother and minimizing the impact of irregular input. In addition, the phase step and its related phase coefficients can be dynamically determined according to the timing of the input frame, so a suitable phase step can be obtained under any irregular conditions. In other words, the method of calculating phase steps can be applied to various irregular situations, such as input frames arriving early and input frames being delayed. The parameters used to calculate the phase step are such that the phase coefficient converges to 0 at the end of the phase adjustment period. In this way, subsequent phase coefficients can still continue to roll normally according to the normal phase steps.

In addition, the method of dynamically calculating phase steps can also be applied to more complex situations. For example, if the phase step is calculated or adjusted based on the first irregular input frame, and the second irregular input frame arrives during the above-mentioned phase adjustment period, the phase step can be recalculated to In response to the second irregular input frame, the phase coefficient will follow the newly calculated phase step until the input frame returns to the original pattern and the phase coefficient returns to normal.

As mentioned above, the traditional phase step is determined by looking up the reference table according to the input frame rate and the output frame rate. Even when the problem of irregular input occurs and input frames arrive with irregular timing, the reference table can only be applied to a few typical cases. In contrast, the image processing method of the present invention can dynamically monitor the input status (such as monitoring the actual value and the expected value) and correspondingly calculate the phase step according to the parameters (such as the number of remaining input frames and the number of remaining output frames). Therefore, the video processor can calculate and obtain appropriate phase steps under different situations, which are not limited to situations that can be predicted by the reference table. In this case, the present invention can handle more complex input irregularities better than the traditional method using a reference table.

It is worth noting that the purpose of the present invention is to provide an image processing method and a related video processor that can be used to process irregular input. Those skilled in the art can make modifications or changes accordingly, but are not limited to this. For example, in the above embodiment, the output frame rate is 60 Hz, and the input frame is received at 24 Hz without cadence or pulldown. In another embodiment, the image processing method of the present invention can also be applied to input frames with regularity or pulldown, which can be received at a frame rate of 60 Hz, such as the input sequence A1, A2, A3, B1, B2... For example, in one embodiment, the frame difference value can be used to infer the frame count value and then determine the actual value. Therefore, the video processor can determine the irregular input and calculate the phase step in the same way.

Similarly, input frames with regularity can also be applied to the embodiment of selecting the frame group for interpolation based on the buffer level value. In an exemplary embodiment, repeated input frames may be written to the same buffer unit. For example, input frame A2 is used to overwrite its previous input frame A1, and input frame A3 is used to overwrite its previous input frame A2, and the next input frame B1 is written into another buffer unit. According to the above rules for writing input frames to the frame buffer, the buffer level value can be calculated in the same way and used to select the frame group for interpolation. Buffer overflow and buffer underflow can also be determined based on the buffer level value in the same way.

In one embodiment, the method of selecting the frame group to perform interpolation and the method of calculating the phase step can be performed together to implement a complete image processing flow. Figure 4 is a flow chart of an image processing process 40 according to an embodiment of the present invention. The image processing flow 40 may be a combination of frame group selection and phase step calculation implementations, and may also be implemented in a video processor, such as the video processor 106 in Figure 1 . As shown in Figure 4, the image processing process 40 includes the following steps:

Step 402: Receive an input frame.

Step 404: Calculate the phase step.

Step 406: Calculate the phase coefficient according to the phase step.

Step 408: Calculate the buffer level value based on the input frame stored in the frame buffer.

Step 410: Detect buffer overflow and buffer underflow based on the buffer level value.

Step 412: Based on the buffer level value and the detection results of buffer overflow and buffer underflow, select the frame group used to perform interpolation and adjust the phase coefficient.

Step 414: Detect irregular input frames.

Step 416: If the input frame is irregular, dynamically calculate the phase step.

Step 418: If the input frame is input normally, use the normal phase step.

Step 420: Perform motion estimation and motion compensation to generate an interpolated frame.

Step 422: Output the interpolated frame as an output frame.

It should be noted that the image processing method of the present invention can be performed dynamically during each output frame period, so the image processing flow 40 represents the operation of the video processor during an output frame period. First, the video processor can receive an input frame and write the input frame into the frame buffer (step 402). The input frame can be received during a portion of the output frame period (without pull-down) or during each output frame period (with pull-down). Then, the video processor calculates a normal phase step based on the relative relationship between the input frame rate and the output frame rate (step 404). Therefore, the phase coefficient can be calculated by accumulating the phase step, where the current phase coefficient is equal to the previous phase coefficient plus the phase step (step 406).

Next, the video processor calculates the buffer level value based on the status of the input frame stored in the frame buffer (step 408). More specifically, the buffer level value can be incremented by one when an input frame arrives/received, and decremented by one at the beginning of a new phase period. Based on the dynamically changing buffer level value, the video processor can detect whether buffer overflow and buffer underflow occur (step 410). Buffer overflow and underflow represent that the input frame and the interpolation frame (i.e., the output frame) are in an unbalanced state. Buffer overflow occurs when input frames arrive faster than interpolation frames are produced such that a previous input frame intended for interpolation is overwritten by a newly arriving input frame. Buffer underflow occurs when the delay in the input frame is so long that the input frame expected for the current interpolation has not yet been received and written to the frame buffer.

Next, the video processor selects a frame group to perform interpolation according to the buffer level value. If the above detection result indicates that buffer overflow or underflow occurs, the phase coefficient can be adjusted (step 412). When a buffer overflow occurs, the frame group used to perform interpolation can be changed to use the stored input frames that have not been overwritten by the currently arriving input frame, and the phase coefficients are forced to start a new phase period. When buffer underflow occurs, the video processor performs interpolation using the same set of input frames as during the previous output frame, and the phase coefficients are forced to their maximum values.

The video processor then detects and determines whether the reception of the input frame is normal or irregular (step 414). If the input frame does not arrive within the expected output frame period, it represents an irregular input, which can be known by comparing the actual value with the expected value. If it is determined that the input of the input frame is normal (ie, arrives during the expected output frame), the video processor uses the normal phase step to calculate the phase coefficient during the next output frame (step 418). If it is determined that the input of the input frame is irregular (ie, arrives during an unexpected output frame), the video processor can dynamically adjust and calculate the phase step for calculating the phase coefficient during the next output frame (step 416). The phase step can be calculated with reference to the number of remaining input frames and the number of remaining output frames.

Next, based on the frame group obtained by the above selection scheme and using the phase step and phase coefficient calculated based on the above procedure, the video processor can perform motion estimation and motion compensation to generate an interpolation frame (step 420), and the interpolation frame It can be used as an output frame to be output to the display panel (step 422).

To sum up, the present invention proposes an image processing method that can be used in a video processor to deal with the problem of irregular input. In one embodiment, the video processor may monitor a buffer level value that indicates the status of the input frame stored in the frame buffer. Based on the buffer level value, the video processor can select appropriate input frames as frame groups to perform interpolation, determine whether buffer overflow or buffer underflow occurs, and then take appropriate countermeasures. In one embodiment, the video processor can detect the input timing of the input frame to determine whether the input frame follows normal timing. If irregular input occurs, the video processor can calculate the phase step accordingly, so that the impact of the irregular input is evenly distributed to multiple subsequent output frames, thereby making the output picture smooth and minimizing the impact of the irregular input.

In addition, in one embodiment, the video processor can first determine and handle the buffer overflow or buffer underflow situation according to the status of the frame buffer to determine a suitable frame group, wherein when the buffer overflows The phase coefficient should also be well controlled when underflow or underflow occurs. The video processor can then determine whether irregular input occurs. If the irregular input does not cause buffer overflow or underflow, it means that the irregular input can be processed by the existing frame buffer, and the phase step calculation scheme of the present invention can be used to generate appropriate phase steps and phase coefficients. . In this case, by dynamically determining the frame group used to perform interpolation and calculating the phase step while taking into account the status of the frame buffer, the video processor of the present invention can handle various irregular situations. Therefore, for the motion estimation and motion compensation operations performed during each output frame, the video processor can select the best frame group from the frame buffer and calculate and obtain the appropriate phase step. In this way, the video processor can produce a smooth output picture under any irregular input situation, and the impact of irregular input can be minimized. The above are only preferred embodiments of the present invention, and all equivalent changes and modifications made in accordance with the patentable scope of the present invention shall fall within the scope of the present invention.

10:Display system 12: Video providing unit 104: Frame rate converter 105: Frame buffer 106:Video processor 108:Display driver 110:Display panel 20, 30, 40: Image processing process 200～208, 300～312, 402～422: steps

FIG. 1 is a schematic diagram of a display system of an embodiment of the present invention. FIG. 2 is a flow chart of an image processing process of an embodiment of the present invention. FIG. 3 is a flow chart of another image processing process of an embodiment of the present invention. FIG. 4 is a flow chart of another image processing process of an embodiment of the present invention.

20:Image processing process

200~208: steps

Claims (19)

An operating method for a video processor, including: Receive a series of input frames; Calculate a buffer stage value based on the series of input frames, where the buffer stage value corresponds to the state of the input frames stored in a frame buffer in the video processor; and According to the buffer level value, a frame group is selected from the input frames stored in the frame buffer to generate an interpolation frame as an output frame output by the video processor. The method of claim 1, wherein the buffer level value represents the number of input frames stored in the frame buffer but not yet used to generate the interpolation frame. The method of claim 1, wherein the buffer level value is increased by 1 when a new input frame in the series of input frames arrives, and the buffer level value is decreased by 1 at the beginning of a new phase period. The method of claim 1, wherein the step of selecting the frame group from the input frames stored in the frame buffer to generate the interpolation frame includes: When the buffer level value is equal to a preset value, a newly arrived input frame in the series of input frames is selected as a current frame, and a previously received input frame in the series of input frames is selected as a previous frame to generate The interpolated frame. The method as described in claim 4, wherein the default value is 0. The method as described in claim 4, wherein the step of selecting the frame group from the input frame stored in the frame buffer for generating the interpolation frame further includes: When the buffer level value is greater than the preset value, selecting a first previously received input frame in the series of input frames as the current frame, and selecting a second previously received input frame before the first input frame as the previous frame, for generating the interpolation frame. The method as described in claim 1 further comprises: Determining whether a buffer underflow occurs according to the buffer level value, wherein the buffer underflow represents that an input frame in the series of input frames that is expected to be taken during an output frame period to generate the interpolation frame has not been received during the output frame period. A method as described in claim 7, wherein when it is determined that the buffer underflow occurs, a phase coefficient reaches a maximum value and the frame group used to generate the interpolated frame remains unchanged. The method described in request 1 also includes: Determine whether a buffer overflow has occurred according to the buffer level value, where the buffer overflow represents a previous frame in the series of input frames that is stored in the frame buffer and is expected to be accessed to generate the interpolation frame. The receive input frame is overwritten by a new receive input frame in the series of input frames. The method described in request item 9, wherein the step of determining whether the buffer overflow occurs based on the buffer level value includes: When the buffer level value is equal to the number of one frame buffer of the video processor minus 1, it is determined that the buffer overflow has occurred. The method of claim 9, wherein when it is determined that the buffer overflow occurs, the buffer level value is forcibly decremented by 1, and then the frame group is selected based on the buffer level value. An operating method for a video processor, comprising: receiving a series of input frames; judging an expected input spacing value based on a relative relationship between an input frame rate of the series of input frames and an output frame rate of a series of output frames output by the video processor; judging whether an actual input spacing value is different from the expected input spacing value; when judging that the actual input spacing value is different from the expected input spacing value, calculating a phase step and obtaining the phase step, the phase step being different from a normal phase step determined based on the relative relationship between the input frame rate and the output frame rate; and Using the calculated phase step, an interpolated frame is generated according to a selected frame group and output as an output frame in the series of output frames. The method of claim 12, wherein the expected input spacing value represents an expected elapsed output frame period from the reception of a previous input frame in the series of input frames to the reception of a current input frame in the series of input frames. quantity, and the actual input spacing value represents the number of actual elapsed output frame periods from the reception of the previous input frame to the reception of the current input frame. The method as described in claim 12, wherein the step of calculating the phase step comprises: Calculating the phase step according to a remaining input frame number and a remaining output frame number, wherein the remaining input frame number represents the number of remaining input frames used to generate the interpolation frame in a cycle determined according to the relative relationship between the input frame rate and the output frame rate, and the remaining output frame number represents the number of remaining output frames in the cycle. The method of claim 14, wherein when it is determined that the actual input distance value is different from the expected input distance value, the remaining input frame number is increased by 1 to generate an updated remaining input frame number, and the current actual input distance value The value is added to the number of remaining output frames to produce an updated number of remaining output frames. A method as described in claim 15, wherein the phase step is calculated by subtracting a current phase coefficient from the updated residual input frame number and then dividing the result by the updated residual output frame number. The method of claim 12, wherein the calculated phase step continues to be valid until the remaining output frame number approaches 0 and returns to a maximum value. The method of claim 12, wherein the actual input spacing value that is different from the expected input spacing value indicates that an input frame in the series of input frames has irregular input timing. The method described in request 12 also includes: When it is determined that the actual input distance value is the same as the expected input distance value, the normal phase step is adopted.

Images