KR20130105374A - Image processing apparatus and control method thereof - Google Patents

Abstract

PURPOSE: Method of optimizing an image of an image processing device is provided to make video play easy and to reduce the occupancy of central processing unit (CPU) and system resource. CONSTITUTION: A video reception unit receives (S11) an input image. An image processing unit processes the input image and outputs an output image. A display unit displays the output image. A storage unit stores data and information of the input image. A control unit compares (S12) an image processing speed of the input image and a frame rate of the input image, configures (S13) a frame rate of the output image to be smaller than the frame rate of the input image when the frame rate of the input image is greater than the image processing speed of the input image and controls (S14) the number of processed frames of the input image according to a configured result to decrease. [Reference numerals] (AA) Start; (BB) End; (S11) Receive an input image; (S12) Is the frame rate of the input image larger than the image processing speed of the input image ?; (S13) Set the frame rate of an output image smaller than that of the input image; (S14) Reduce the number of processed frames of the input image according to the setting

Description

Image processing apparatus and control method thereof {IMAGE PROCESSING APPARATUS AND CONTROL METHOD THEREOF}

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a reproduction technology of an image processing apparatus, and more particularly to a method for optimizing an image of an image processing apparatus.

An image processing apparatus based on an existing embedded facility or a given platform provides the user with two different choices: video hardware decoding / display and software decoding / display. When the user selects a hardware decoding / display option, the existing image processing apparatus performs a call of hardware decoding / display using a platform related interface, but an implementation thereof is provided by each hardware supplier. When the user selects the software decoding / display option, the existing image processing apparatus decodes a video using a software decoder and calls the software display interface to display.

All the modules in the existing embedded or platform based image processing system are not related to the hardware system, apply the multi-threaded software playback control, the CPU occupancy and occupancy of system resources are relatively high, and the decoding speed is relatively slow. However, the software solution cannot play video in SD, 720p, and HD resolutions smoothly, such as MPEG-4 and H.264. However, when using a hardware decoding scheme, the underlying interfaces and implementations of different hardware are somewhat different, so the universality and portability of the hardware decoding scheme is also relatively low. On the other hand, since the hardware decoder has limited support for video formats (eg, MPEG4, H.264) and profile, level and resolution, there may be different representations on different hardware facilities. By using a common software decoding scheme, the problems with the hardware decoding scheme can be overcome, but the performance is very limited.

The optimization of the existing video player is mainly focused on the optimization of the decoder itself and the memory reduction task, and each module such as IDCT, inverse quantization, entropy decoding, motion compensation, loop filter, and resolution reduction decoding and color conversion in the video decoder. However, the focus is on algorithm optimization or specialized hardware optimization for modules such as image scale, but there is a problem that the generality of the optimization is low and the optimization level is limited.

An object of the present invention is to solve the problems of the conventional image processing apparatus described above, to provide an image processing apparatus and a control method for optimizing a playback image to improve the smoothness of video playback and reduce the occupancy of CPU and system resources. have.

The image processing apparatus for achieving the object of the present invention, the image receiving unit for receiving an input image; An image processor which processes the input image and outputs an output image; A display unit which displays the output image; A storage unit for storing data and information of the input image; The image processing speed of the input image is compared with the frame rate of the input image. When the frame rate of the input image is greater than the image processing speed of the input image, the frame rate of the output image is greater than the frame rate of the input image. It may be achieved by the image processing apparatus, characterized in that it comprises a control unit for setting to be small, so as to control the number of frames to be processed image of the input image according to the set result.

The control unit may calculate a real time reduction rate of a frame of the input image according to the image processing speed of the input image and the frame rate of the output image, and perform downsampling of the frame of the input image according to the calculated real time reduction rate. Can be done.

The control unit may select a predetermined frame of the input image to perform a reduction of the number of frames processed by the frame priority according to a group of picture (GOP) structure by the downsampling, and select the predetermined Frames can be discarded without image processing.

Here, the control unit calculates a reduction rate of the frame of the output image according to the frame rate of the output image and the frame rate of the input image,

If the predetermined frame is discarded, the number of frames for which the predetermined frame is discarded equally according to the reduction rate of the frame of the output image, and the number of frames for which the image is processed may be reduced is the reduction rate of the frame of the output image. When the number of the frames of the input image is smaller than the number of frames, the number of frames for which the image processing is performed may be discarded.

The controller may determine the image processing speed and the value when the number of frames for which the reduction of the number of frames processed by the image is smaller than the number of predetermined frames of the input image by the reduction rate of the frames of the output image. Calculating a real-time reduction rate of the frame of the input image according to the frame rate of the output image, selecting a predetermined frame of the input image for performing a reduction of the frame to output the image according to the calculated real-time reduction rate, and selecting The predetermined frame can be image processed without image output.

Herein, the control unit calculates a ratio of the image processing speed of one frame of the input image in each GOP to the average image processing speed of the frame of the input image image-processed, and the calculated ratio and the image are output. The value calculated by multiplying the reduction rate of the frame of the output image by the reduction of the frame to be the real-time reduction rate in the GOP, and the reduction of the frame to output the image evenly using the real-time reduction rate in the GOP. Can be.

The image processing unit may further include a converting unit converting the YCbCr format from the RGB format, and the control unit converts the input image from the YCbCr format to the RGB format by including a predetermined conversion operation. The computational amount of time can be reduced.

Herein, the control unit may be configured to perform a conversion operation including shifting, table lookup, logic operation, and a small amount of addition, such that vertex, elimination multiplication, multipoint simultaneous operation, table lookup, and removal are performed. Law and limiting operations can be performed.

The apparatus may further include a user input unit capable of inputting a video signal, and wherein the controller uses a point calculation based on the size of the screen area of the display unit when there is an input regarding smooth reproduction priority from the user input unit. The resolution of the output image can be reduced.

The image processor may further include a decoder configured to decode an input image, and the controller may be configured to convert the YCbCr format into the RGB format and perform image processing of the input image by arranging the converter and the decoder in series. Can be done.

The controller may reduce the resolution at a ratio of 1/2 to 1/16.

Here, the storage unit further includes an output image storage unit for storing the output image,

The control unit compares the resolution of the output image processed by the image processing unit with the size of the screen area of the display unit, and displays the output image and the input image or adjusts the output image based on the comparison result. The image can be displayed.

When the resolution of the output image is less than 1/2 of the size of the screen area of the display unit, the controller enlarges the image using the bilinear interpolation while maintaining the aspect ratio of the input image with respect to the output image. When at least one of the horizontal and vertical sides of the output image corresponds to the screen area of the display unit, the other side may be positioned at the center of the side corresponding to the screen area of the display unit.

Herein, when the resolution of the output image is smaller than a size of the screen area of the display unit, the controller may arrange one of the output image or the input image in the center of the display unit screen area. I can display it.

When the resolution of the output image is larger than a size of a display area of the display unit, the controller reduces the size of the output image using a recent Lean method while maintaining the aspect ratio of the input image. When at least one of the horizontal side or the vertical side corresponds to the screen area of the display unit, the other side may be positioned at the center of the side corresponding to the screen area of the display unit.

Herein, the control unit performs vertices on coordinate operations in order to optimize the latest Lean method and the bilinear interpolation method, constructs a table for coordinates necessary for transformation during the calculation of the frame, and the coordinates of the frame are without calculation. It is calculated through table search and additive operation, and the RGB color of the pixel can be processed simultaneously by the calculation of the above-described method.

On the other hand, the control method of the image processing apparatus for achieving the object of the present invention, the step of receiving an input image; Comparing the image processing speed of the input image with the frame rate of the input image; Setting the frame rate of the output image to be smaller than the frame rate of the input image if the frame rate of the input image is greater than the image processing speed of the input image; The method may be achieved by a method of controlling an image processing apparatus, the method including reducing the number of frames to be image-processed in the input image.

The reducing of the number of frames may include calculating a real time reduction rate of a frame of the input image according to an image processing speed of the input image and a frame rate of the output image, and calculating the real time reduction rate of the input image according to the calculated real time reduction rate. And performing downsampling of the frames of the frames.

The performing of the downsampling may include selecting a predetermined frame of the input image to reduce the number of frames processed by the frame priority according to a GOP (Group of Picture) structure by the downsampling. The predetermined frame may be discarded without image processing.

The discarding without the image processing may include calculating a reduction rate of the frame of the output image according to the frame rate of the output image and the frame rate of the input image, and uniformly according to the reduction rate of the frame of the output image. If the number of frames for which the reduction of the number of frames to be image-processed is discarded is smaller than the number of frames of the input image by the reduction rate of the frame of the output image, Discarding all frames for which reduction may be performed may include.

The discarding without the image processing may include: when the number of frames in which the reduction of the number of frames processed by the image is performed is smaller than the number of predetermined frames of the input image by the reduction rate of the frames of the output image. A predetermined frame of the input image for calculating a real-time reduction rate of the frame of the input image according to the image processing speed and the frame rate of the output image, and for reducing the frame to output the image according to the calculated real-time reduction rate The selected predetermined frame may further include image processing without image output.

The discarding of all the frames may include calculating a ratio of the image processing speed of one frame of the input image in each GOP to the average image processing speed of the frame of the input image. And a value calculated by multiplying the reduction rate of the frame of the output image by the reduction of the frame for outputting the image as a real time reduction rate in the GOP, and reducing the frame for outputting the image using the real time reduction rate in the GOP. It may further comprise the step of performing evenly.

The reducing of the number of frames may further include reducing a calculation amount during conversion by converting the input image from a YCbCr format to an RGB format by including a predetermined conversion operation. .

Here, the predetermined transform operation may be transformed, including shifting, table checking, logic operation, and a small amount of additive methods, such as vertex, elimination multiplication, multipoint simultaneous operation, table checking, elimination, and limiting. You can perform the operation.

The reducing of the calculation amount may include reducing the resolution of the output image by using a point calculation based on the size of the screen region on which the image is output when there is an input regarding a smooth reproduction priority by the user. can do.

The reducing of the calculation amount may include converting the YCbCr format from the YCbCr format to the RGB format and a decoding unit configured to process the input image in series to convert from the YCbCr format to the RGB format. And performing image processing in parallel.

In the reducing of the resolution, the resolution may be reduced by a ratio of 1/2 to 1/16.

The reducing of the number of the frames may include comparing the resolution of the output image and the size of the screen area on which the output image is output, and displaying or outputting the output image and the input image based on a result of the comparison. The method may further include displaying an image by adjusting the image.

In the displaying of the image, when the resolution of the output image is smaller than 1/2 of the size of the screen area to which the output image is output, the bilinear interpolation method is maintained while maintaining the aspect ratio of the input image with respect to the output image. The method may further include positioning the other side at the center of the side corresponding to the screen area of the display unit when the at least one of the horizontal side or the vertical side of the output image corresponds to the screen area of the display unit.

The displaying of the image may include outputting one of the output image or the input image when the output image is smaller than a resolution of the output image and the size of the screen area to which the output image is output. And displaying the image by being disposed at the center of the screen area.

In the displaying of the image, when the resolution of the output image is larger than a size of a screen area in which the output image is output, the latest image is maintained while maintaining the aspect ratio of the input image with respect to the output image. And at least one of a horizontal side or a vertical side of the output image corresponds to a screen area on which the output image is output, and positioning the other side at the center of a side corresponding to the screen area of the display unit. .

Here, to optimize the recent Lean method and the bilinear interpolation method, it performs a vertex of the coordinate operation, and constructs a table for the coordinates necessary for the transformation during the calculation of the frame, the coordinates of the frame without the operation It is calculated through the calculation, and the RGB method of the pixel can be processed simultaneously by the calculation of the recent method.

According to the present invention, it is realized by using software that is not related to hardware, which is a multimedia playback method with universality and can support a wide range of video formats.

In addition, by performing optimized combinations for video decoding, color conversion, image scale, decoding speed control, video frame display, system memory use, and frame buffer, it is possible to provide universality with the final goal of smoothness of video playback. In addition, it can also effectively reduce the occupancy of the CPU and system resources, and improve the smoothness of playback.

1 is a block diagram of an image processing apparatus software based on a predetermined platform according to an embodiment of the present invention.
FIG. 2 is a logic diagram of the image processing apparatus software shown in FIG.
3 is a schematic diagram of the length and the numerical value of each section of the limiting table according to the RGB rapid switching method of the image processing apparatus according to an embodiment of the present invention,
4 is a diagram illustrating a method of simultaneously processing four brightness pixels and two chrominance components through first order conversion calculation in a method of converting from a YCbCr 4: 2: 0 sub-sampling format to RGB in an image processing apparatus according to an embodiment of the present invention; It is a schematic diagram,
5 is a schematic diagram showing a half resolution reduction conversion from YCbCr 4: 2: 0 sub-sampling format to RGB of an image processing apparatus according to an embodiment of the present invention.
FIG. 6 illustrates coordinates of a target image of an image processing apparatus according to an exemplary embodiment of the present invention, and positions S of pixels corresponding to the original image and four adjacent pixels A, B, C, It is a schematic diagram of D,
7 is a block diagram illustrating a configuration of an image processing apparatus according to an embodiment of the present invention.
8 is a block diagram illustrating a configuration of an image processing apparatus according to another embodiment of the present invention.
9 is a control flowchart of an image processing apparatus according to another embodiment of the present invention.

The present invention may be embodied in many different forms and is not limited to the embodiments described herein. Parts which are not directly related to the present invention are omitted for convenience of description, and the same or similar components are denoted by the same reference numerals throughout the specification.

BRIEF DESCRIPTION OF THE DRAWINGS To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be described in detail with reference to the accompanying drawings.

The software structure of an image processing apparatus based on an embedded facility or a predetermined platform is similar. FIG. 1 illustrates a structure of image processing apparatus software based on a predetermined platform according to an embodiment of the present invention. Here, the uppermost layer is a predetermined application layer, and realizes a player call through local functions of JAVA and JNI. The image processing apparatus is composed of a media analysis module and a player core. The media analysis module analyzes a file transfer protocol and a media file format, and the player core is a logic core of the entire image processing apparatus. It is composed of decoding module, video display module and audio playback module, and each module operates one thread. The logic control module operates the main thread to demultiplex the multimedia data and control the logic of the entire player. The other four modules also run one thread each for video decoding, display, audio decoding, and playback. Here, the audio / video decoding module calls the open source image processing apparatus software frame FFMPEG, respectively, to perform software decoding. Through the call of a predetermined adapter module, the display module of the video proceeds to display the video using the surface interface of the predetermined platform, and the audio playback module uses the audio track interface of the predetermined system. Proceed with audio playback. Here, the predetermined surface and the audio track are both software interfaces for a predetermined platform.

The logic configuration of the multithreaded player core is shown in FIG. The demultiplexing thread reads the input multimedia data and demultiplexes the buffer by inputting a buffer, and outputs a data packet of a raw stream (for example, compressed data of one frame of MPEG-4 and H.264). Store in audio and video packet buffers. The audio and video decoding thread reads data packets from its input buffer, calls the FFMPEG software decoder to perform decoding, and outputs the decoded YCbCr data and PCM data. RGB conversion is performed on the YCbCr data output by the video decoder, and the RGB data after the conversion is adjusted to the same size as the surface frame buffer through the scale, and written directly to the surface frame buffer. In this way, when displaying a video frame, system memory that needs to be allocated and worked can be omitted. The video display thread controls the display timing of the RGB data in each frame buffer by visiting the shared data and logic control information, and also performs the same logic control as the audio playback thread to complete the synchronized playback of the audio / video frame.

Based on the analysis of the existing technology, the present invention makes improvements to the existing technology in several aspects as follows.

1) Among the technical solutions for optimizing existing video decoders, optimization is generally performed independently for each module. Therefore, each module is not connected and optimized. Therefore, the hitability and universality for optimizing the entire video player is low, and the optimization level is limited.

2) Almost all of the existing video decoding optimization methods apply a predetermined frame dropping algorithm based on uniform resampling or image content in time.

3) Among existing optimization algorithms for color space conversion, the conversion speed is improved by applying algorithm-level optimization and hardware-specific optimization.

Among the software of an image processing apparatus based on an embedded facility or a platform according to the present invention, an optimization method for a video decoding and display module includes several aspects as follows. That is, a dynamic predetermined frame dropping method based on the decoding speed, a fast switching method from YCbCr to RGB color space, and an image rapid scaling method of arbitrary proportion. Hereinafter, the method will be described in detail.

1) Predetermined Frame Dropping Method Based on Decoding Rate

The action of the frame dropping method is to perform down sampling (e.g., when sampling down a video of 30 frames / second to a video of 20 frames / second, the sampling rate is 3: 2) on the video frame on the time axis. By lowering the frame rate of the video, the average speed of video decoding is accelerated under the assumption that the user's impression is not lowered. The algorithm is a predetermined frame dropping method based on the decoding speed, and includes two stages of loss decoding and loss display. The former discards input data directly without processing the video frame selected for discarding, while the latter performs decoding on the video frame selected for discarding first, but does not store and display the decoded video frame. The method specifically includes three parts as follows.

i) Loss decoding and loss display selection mechanism

In the moving picture sequence, if any one frame data is discarded due to the existence of inter-frame prediction and the decoding is not performed, decoding of subsequent frames may lack reference frames, resulting in decoded matching and error propagation in the prediction link. . The basis for selecting the loss decoding or the loss display is as follows. That is, when loss decoding according to the target frame rate, a picture that does not serve as a reference frame of other frames is preferentially selected, and if such a picture does not exist or is discarded, the frame rate can still be lowered to the target frame rate. If not, use the method of loss display.

ii) Ross decoding method

The normal B frame, the lowest B frame of the classification B frame, the lowest P frame of the classification P frame, and the last 1 frame P frame in the closed loop GOP are not referred to as reference pictures of other frames. Therefore, according to the method, such a frame can be selected as a discardable frame. Depending on the result of the calculation of the target frame rate and the raw frame rate, the decoder can select from among the frames that can be discarded and select all or part of the frames that can be discarded. That is, the decoder does not process the input data of such a frame, does not update the buffer and index of the reference picture, directly discards the input data packet, and waits for input data of the next frame. According to the method, it is possible to save time required for full frame decoding, color space conversion, scale and display for a certain amount of frames, and to significantly improve the average decoding and display speed of a video.

iii) Los display method

If there is no B frame, classification P frame, closed loop GOP in the GOP structure, or ii) the raw frame rate cannot be lowered to the target frame rate through the loss decoding method of the loss display method, it is further improved according to the present method. Choose the method of loss display. The method does not alter the decoding process. That is, all video frames are decoded, but the selected discardable frame does not output the decoded image, thereby saving color space conversion, scale, and display time of such frames. A detailed method of selecting a discardable frame is as follows.

Calculate the frame dropping rate f (remove the frame dropping rate of the previous step from the target frame dropping rate) required in this step, and decode time tc of the current frame in N frame intervals and average decoding timeta of all frames before the current frame. The real time frame dropping rate is set to fr = f × tc ÷ ta. Herein, the N frame is one real time frame dropping unit, and a scheme of uniform frame dropping is applied among one real time frame dropping unit, the frame dropping rate is fr, and the one frame is performed within one real time frame dropping unit. Select a frame that can be discarded according to the frame dropping rate. The purpose of using the real time frame dropping rate is to improve the frame dropping rate within a time period where decoding is relatively slow and to reduce the frame dropping rate within a time period where decoding is relatively fast. In addition, by selecting a strategy of uniform frame dropping within a unit of one N-frame, the average speed of video playback is improved and the smoothness of video playback and user's viewing are not affected.

On the other hand, this is independent of how the player discards the perceived video frame. According to this method, the video frame selected for discard is not output from the decoder, so that the perceived video frame in the audio / video synchronization logic is reduced as much as possible.

2) Rapid Switching Method from YCbCr to RGB Color Space

During the conversion of the RGB color space to the YCbCr image output from the decoder, floating point arithmetic, memory operations, and axis point calculation consume a large amount of system resources and outputs. Therefore, the present invention has optimized the conversion method. And, mainly, the following steps.

i) Algorithm level optimization

Algorithm-level optimization significantly reduces CPU-consuming operations by including vertexing, elimination multiplication, multipoint concurrency, table lookup, elimination and limiting operations for YCbCr to RGB conversion operations.

ii) RGB resolution reduction

,

,

....

의 비율 규칙에 따른 해상도 저감 RGB전환을 포함한다. 상기 방법을 통하여, RGB전환 연산에 참여하는 픽셀 포인트 수를 대폭 감소시켜 전환 속도를 향상시킨다.When the YCbCr picture resolution is much larger than the size of the target display buffer, and the user selects a smooth playback priority play option, the present invention is directed to a ratio rule (i.e., the same ratio reduction in the horizontal and vertical directions) for the image after decoding. By performing the resolution reduction RGB conversion, the RGB image size after the conversion can approach the size of the target display buffer. here,

,

,

....

Resolution RGB conversion according to the ratio rule of the. Through the above method, the number of pixel points participating in the RGB conversion operation is greatly reduced to improve the conversion speed.

iii) parallel conversion

In software decoders that support multithreaded parallel decoding, the YCbCr to RGB conversion module does work before the intermediate decoding thread is completed. In this way, the switching module and the decoding module in the respective decoding thread are serialized, and the switching operation for each frame picture is also parallel. The method is very effective for the optimization of multi-core platforms and improves the speed of decoding and conversion to some extent even for a single core platform and does not affect the performance of the entire player.

3) Arbitrary proportional image rapid scaling method

The RGB image size after switching may be different from the size of the target display buffer. The present invention can scale the output image to the size of the target display buffer using any proportional image rapid scaling method. Specifically, the following content is included.

i) Non-memory operation to write directly to the target display buffer

In order to prevent extra memory allocation and memory operations, the scale module obtains the point of the target display buffer address through the adapter layer before scaling the moving picture of each frame, and the scale module stores the memory for the target picture. Without assignment, the target picture is written directly to the address indicated by the point, and the moving picture display module then determines when to display the RGB picture already written at the address.

ii) Select scaling method and display mode according to scaling

Different scales or display modes are applied to images of different resolutions.

If the image resolution is smaller than 1/2 of the screen size (i.e., both horizontal and horizontal sizes are smaller than 1/2 of the size corresponding to the target display buffer), the image is enlarged and enlarged by applying the bilinear interpolation method. The aspect ratio of the source image is maintained, and one side that does not reach the full screen size is placed at the center position of the side corresponding to the display area, and the area where the size of both sides is insufficient is displayed as the black side (all RGB components have initial values of 0). do. This is because the decoding speed of a small resolution image is sufficiently fast, so that the bilinear interpolation method with relatively high output and relatively good image enlargement quality can be applied.

Otherwise, if the picture resolution is less than or equal to the screen size (that is, all the seedling sizes are less than or equal to the size corresponding to the target display buffer), the picture is directly centered in the display area without scaling. The remaining display area is displayed in black side. This is because the image resolution is similar to the display size, it is possible to save time and skip the calculation of the image magnification when displaying directly according to the raw size.

Otherwise, the image is reduced to the screen size using the most recent method, the aspect ratio of the source image is maintained during the reduction process, and one side that does not reach the full screen size is placed at the center position of the side corresponding to the display area. In other words, the areas with insufficient size on both sides are displayed as black sides. This is because the quality of the image reduced by the recent Lean method is similar to the quality of the image by the bilinear interpolation method, but the amount of calculation is much smaller than that of the bilinear interpolation method.

iii) Algorithm Level Optimization of Bilinear Interpolation and Recent Lean Methods

Since both the bilinear interpolation method and the recent Lean method require floating point operations for pixel coordinates, and the calculation of coordinates occupies a maximum amount of output, the present invention first optimizes the vertex method for the two methods. Next, a table is constructed during the calculation of the first frame with respect to the coordinates to be used during the coordinate calculation, and after that, the coordinates of all the frames can be obtained through table inspection and simple additive operation without calculating.

Since the Lean method does not need to calculate pixel values, it is possible to process three color components of one pixel at the same time through a first order operation. For example, for the RGB565 format, two-byte pixel value operations can be performed first, thereby further reducing output.

The technical solution according to the present invention can achieve the following advantageous effects. In other words, after performing some optimization on video decoding and display of embedded software or multimedia software player based on Android (predetermined) platform, normal video format (MPEG-1 / 2/4, H.264, etc.) It can support smooth playback of a video of SD (720 x 480p, 720 x 576i) resolution, 720p (1280 x 720p) resolution, and partial HD (1920 x 1080) resolution. Here, the three major optimization stages each average 30% (1/3 frame dropping rate) and 75% (6GB parallel decoding on a double-core CPU without lowering the resolution when compared to the corresponding unoptimized algorithm). And conversion) and 90% (the reduction of 1024 × 562 resolution in HD) can be reduced.

Hereinafter, the present invention will be described in detail with reference to one specific embodiment.

1) Basic playback process of universal multimedia software player

The basic playback process of the multimedia software player is as follows.

Step 1: The user launches the player's application, selects an input file, and activates the player.

Step 2: Analyze the file transfer protocol in the media analysis module, call the function of the corresponding protocol to read the multimedia data, analyze the file format and coding / decoding information, initialize the corresponding resources of the FFMPEG, the player core Initialize

Step 3: The player core operates corresponding resources according to the media information output from the media analysis module, opens and positions the audio / video software decoder and the audio / video player, the main logic control thread, the video decoding thread, and the display. Activate thread, audio decoding thread and playback thread.

Step 4: The main logic control thread obtains the size and address of the frame buffer area, places the size in the video decoding thread, and places the size and address in the video frame scale module.

Step 5: The video decoding thread places the FFMPEG software decoder, operates the multithreaded decoding mode to set the number of parallel threads TC, and FFMPEG waits for parallel decoding of the input data according to the multi-decoding thread according to the arrangement.

Step 6: The main logic control thread starts reading the file data and stores it in the input buffer area in FIG. If the data amount of the buffer area is larger than the set threshold TH1, the main thread starts demultiplexing the audio / video data using FFMPEG.

Step 7: The audio / video data output after demultiplexing is stored in the audio packet buffer area and the video packet buffer area in FIG. 2, respectively. When the total amount of all audio / video data stored in the packet buffer exceeds the set threshold TH2 and the quantity of audio packets and the quantity of the video packet exceeds the set threshold TH3, the audio / video decoding thread is executed in the buffer area. Read and decode the audio / video packets, respectively. Here, the color space conversion from YCbCr to RGB is performed in parallel during the multi-threading decoding process.

Step 8: After the audio / video starts decoding, the speed change of demultiplexing, audio / video decoding, and playback is not uniform, so the main logic control thread operates the buffer mechanism to prevent overflow of memory usage, and demultiplexing output. Make the data input and consumption rate relatively stable. When starting to play normally, the buffer state is playing. If there is no audio or video data in the audio / video packet buffer area, the audio / video decoding thread and playback thread are notified to stop the operation temporarily. At this time, only an input exists in the audio / video packet buffer area and no output exists. This state is the buffering state. In the case of buffering state, if the total amount of audio and video data in the audio / video packet buffer area exceeds the set threshold TH4 and both the quantity of audio packets and the quantity of video packets exceed the set threshold TH3. Notify the audio / video decoding thread and the playback thread to continue, changing the buffer state to playing. In the playing state, if the total amount of data in the audio / video packet buffer area exceeds the threshold TH2, the demultiplexing is temporarily stopped, where only output is present and no input is present in the audio / video packet buffer area. This prevents overflow of memory usage and modifies the state to consuming. In the case of the consumer state, if the total amount of audio and video data in the audio / video packet buffer area is smaller than the set threshold TH4, the demultiplexing is restored, and the state is changed to the normal playing state. At this time, an input and an output exist simultaneously in the audio / video packet buffer area.

Step 9: After the audio / video decoding thread successfully outputs audio / video data after decoding one frame, the audio playback thread and the video display thread respectively read the frame data, so that the current system time is the first frame video playback. The time between the start time is obtained and the system clock STC is compared with the display time stamp PTS of the audio or video of one frame currently decoded and output. If STC <PTS is also PTS-STC <TH5, or STC> PTS is also STC-PTS <TH6, play the current audio frame into the Audio Track or play the RGB image of the current video frame. Output to the scale module. Otherwise, if STC <PTS, the audio playback thread or video display thread waits 5 milliseconds before continuing to compare. Otherwise, the current audio / video frame is determined to be perceptual and still proceeds with playback or display for that frame but requires updating the system clock STC = STC- (STC-PTS). This is to ensure smooth playback of the current audio / video, but allows slowing down the system clock.

Step 10: The image scale module acquires the RGB image of the current video frame, scales the image to the size of the surface frame buffer at a high speed, and stores and displays it directly in the target display buffer area.

Steps 1 to 10 are basic steps for normally opening and playing the media file. The limit values and the constants TC, TH1, TH2, TH3, TH4, TH5, and TH6 are preset batch values (for example, TC = 6, TH1 = TH2 = 5M bytes, TH3 = 30, TH4 = 3M bytes, TH5 = TH6 = 10 milliseconds).

2) Specific Embodiment of Dynamic Egonomic Frame Dropping Based on Decoding Rate

The action of the dynamic self-adaptive frame dropping method based on the decoding speed according to the present invention is to perform down sampling on the video frame on the time axis, and lower the frame rate of the video on the premise that the user's viewing is not reduced. We are going to optimize for speed. The main steps are as follows.

Step 1: After the decoder acquires the information of the coder / decoder format, the decoder sets the target frame rate ft according to the format of the coder / decoder, the bit rate, the raw frame rate f (frame / second), the difference, and the level information. For example, in one embodiment, when a target frame rate is set to 20 frames / second for a H.264 video, 1280 × 720 resolution, HP differential, 4.0 level, 8M bit rate, 30 frame / second frame rate video, The sampling rate in time is 3: 2. The frame dropping rate fd = f-ft is calculated to indicate the number of frames to be discarded every second.

Step 2: The structure information of the picture set (GOP) is obtained from the coder / decoder format information, and the determination is performed, and the following steps are performed.

a) If there are general B frames, lowest B frames of classification B frames, lowest P frames of classification P frames, and last one frame P in closed loop GOP, these frames are defined as discardable frames and marked as D frames. . If the frequency of the D frame is fD> fd, uniform loss decoding is performed for the D frame. That is, every fd D frames are uniformly discarded every second, no decoding is performed on the discarded D frames, no decoding data is output, the buffer or index of the reference frame is not updated, and the decoder does not output the input data. Emits. In addition, the method no longer performs subsequent steps.

b) If there is a D frame in the GOP and the frequency of the D frame is fD ≤ fd, discard all the D frames, do not decode the discarded D frames, do not output the decoded data, and buffer the reference frame or It does not update the index, but emits decoder input data. If fD = fd, the method no longer performs the subsequent step, otherwise performs the subsequent loss display step.

c) If there is no D frame in the GOP, denote fD = 0.

d) The target frequency for calculating the loss display is fR = fd-fD, which indicates the number of frames for which the loss display should be performed every second.

e) Set the length N of the real-time frame dropping unit and the length of the GOP equally.

f) Calculate the decoding time tc of the current frame and the average decoding time ta of all the frames before the current frame at intervals of N-1 frames.

g) Calculate the real time frame dropping rate. fr = fR × tc × ta.

h) Within one real time frame dropping unit, a uniform loss display is performed using the frame dropping rate fr. In other words, the frames selected to be discarded do not output a picture after decoding, thereby saving the time required for color space conversion, scale and display of such frames.

By using a real-time frame dropping strategy, it is possible to improve the frame dropping rate within a time period where decoding is relatively slow, to reduce the frame dropping rate within a time period when decoding is relatively fast, and to uniformly within a unit of one N frame. By selecting the strategy of frame dropping, the average speed of video playback may be improved, and the smoothness of video playback and user's viewing may not be affected.

It should be pointed out here that the target frame rate ft in this method is only a reference value in one calculation and not a frame rate actually reproduced. It may be a variable frame rate for actual reproduction, the average frame rate of which is close to the target frame rate. Since the demultiplexing thread has already extracted the time stamp of each audio / video packet from the analysis of the file format, abandoning the decoding or display of some video frames does not affect the display and synchronization of other video frames. On the other hand, this is independent of how the player discards the perceived video frame. According to this method, the video frame selected for discard is not output from the decoder, so that the perceived video frame in the audio / video synchronization logic is reduced as much as possible.

3) Specific Embodiment of Rapid Switching from YCbCr to RGB Color Space

If the target platform does not support the display of YCbCr video format and only supports RGB format picture output, the player must proceed to the RGB color space for YCbCr picture output from the decoder. The known method of color conversion from YCbCr to RGB is as follows.

R = 1.164 (Y-16) + 1.596 (Cr-128)

G = 1.164 (Y-16)-0.813 (Cr-128)-0.391 (Cb-128)

B = 1.164 (Y-16) + 2.018 (Cb-128)

Because floating point operations, memory operations, and axis point calculations consume a large amount of system resources and throughput, the method for optimizing for the transition from the YCbCr 4: 2: 0 subsampling format to the RGB565 format according to the present invention is mainly performed in the following steps. It includes.

Step 1: The process of performing algorithm level optimization on the conversion operation includes the following steps.

Step 1-1: The vertex method is applied to the floating-point arithmetic operation, 10-bit enlargement is performed on the transform coefficient using the proportional element K = 1024, and then rounded off through rounding. Through this, the result of applying the vertex method to the conversion formula is as follows.

Here, R ', G' and B 'are magnified 1024 times with respect to R, G and B, respectively. In other words,

Step 12: Set up and use the conversion table, and compute by elimination multiplication.

A) Three conversion tables of 32 bits are defined. That is, Tables T3T1, T4T2, and T5, where the indices of each table are all Y, Cb, or Cr pixel values of 0 to 255, and the implications of each table are as follows (where “<<” is the left shifting operation). Is displayed).

aa) Table T3T1 indicates the portion where Cr is the index during conversion. here,

이고, The first 16 bits are:

ego,

이며, The last 16 bits are:

Is,

를 얻을 수 있다.As a result,

Can be obtained.

bb) Table T4T2 indicates a portion where Cb is the index during the switching. here,

이고, 이에 후속 단계의 폭제한 테이블 중 B 컴포넌트 시작 위치의 오프셋 4608을 포함한다.The first 16 bits are:

This includes offset 4608 of the B component start position in the delimiting table of the subsequent step.

이고, 이에 후속 단계의 폭제한 테이블 중 G 컴포넌트 시작 위치의 오프셋 2304를 포함한다.The last 16 bits are:

This includes offset 2304 of the G component start position in the delimiting table of the subsequent step.

를 얻을 수 있다.As a result,

Can be obtained.

cc) Table T5 indicates the portion of the conversion whose Y is the index.

.

.

B) Using the conversion table to calculate the RGB value, the calculation method is as follows.

Where “>>” denotes a right shift operation, (short) denotes taking 16 bits below the computed value, and the three components of RGB of one pixel are one variable of type 16-bit unsigned short It can be stored, and the calculation method is as follows.

Where "|" represents a logic or operation.

Step 13: Set up and use the limiting table, remove the conditional limit and operate.

The RGB value calculated in step 12 may exceed the range [0, 255], so the width limit must be applied to the RGB value before calculating rgb. In other words,

However, this primary calculation introduces two comparisons, and the present invention optimizes this by applying the method of the explosive table. The definition of the limit table is as follows.

static unsigned short crop_shift [(256 + 2 * 1024) * 3] = {0,}

The array crop_shift [] assigns values as follows when the player core initializes.

for (int i = 0; i <1024; i ++)

{

crop_shift [i] = 0;

    crop_shift [2304 + i] = 0;

    crop_shift [4608 + i] = 0;

    crop_shift [i + 1024 + 256] = (0x1F << 11);

    crop_shift [2304 + (i +1024 + 256)] = (0x3F << 5);

    crop_shift [4608 + (i + 1024 + 256)] = 0x1F;

}

for (int i = 0; i <256; i ++)

{

    crop_shift [i + 1024] = ((i >> 3) << 11);

    crop_shift [2304 + (i + 1024)] = ((i >> 2) << 5);

    crop_shift [4608 + (i + 1024)] = (i >> 3);

}

Since the shift format when the target format is RGB565 and 16-bit RGB data is displayed as one pixel must also be taken into consideration, the data length and value of each section in the array are as shown in FIG.

3 is divided into three 2304 data length intervals, where each interval is further divided into 1024 + 256 + 1024 sub-intervals of three data lengths, for a total of nine sub-intervals. Each sub-section in three sections displays the following content from left to right, respectively.

1) If the R / G / B component is smaller than 0 out of 1024 zero values, that is, RGB565, the limit is advanced to zero.

2) 256 values of 0-255 pass-shifting represent 5/6/5 bits of the R / G / B component of RGB565 in RGB565.

3) Maximum pass shifting of 1024 R / G / B components indicates 5/6/5 bits of the R / G / B component in RGB565.

Step 14: Multipoint Concurrent Operation

As shown in Fig. 4, in an image having an output format of YCbCr 4: 2: 0 sub-sampling format, one Cb or Cr pixel corresponds to four Y pixel points. Therefore, RGB conversion can be performed for four pixel points at the same time.

Following the optimization of step 1, an example code of an algorithm for converting an image of a narrow YCbCr 4: 2: 0 sub-sampling format into an RGB565 image is as follows.

rgb0 = (unsigned short *) dest;

rgb1 = rgb0 + width;

unsigned short * crop = crop_shift + 1024;

for (h = height; h> 0; h)

{

for (w = width; w> 0; w = 4)

{

... ...

... ...

r_add = (short) (T3T1 [Cr]);

g_add = (short) T4T2 [Cb] + (short) (T3T1 [Cr] >> 16));

b_add = (short) (T4T2 [Cb] >> 16);

y_add = T5 [y0 ++];

* rgb0 ++ = (unsigned short) (crop [y_add + r_add] | crop [y_add + g_add] | crop [y_add + b_add]);

y_add = T5 [y0];

* rgb0 ++ = (unsigned short) (crop [y_add + r_add] | crop [y_add + g_add] | crop [y_add + b_add]);

y_add = T5 [y1 ++];

* rgb1 ++ = (unsigned short) (crop [y_add + r_add] | crop [y_add + g_add] | crop [y_add + b_add]);

y_add = T5 [y1];

* rgb1 ++ = (unsigned short) (crop [y_add + r_add] | crop [y_add + g_add] | crop [y_add + b_add]);

}

... ...

... ...

}

In the above code, rgb0 is a point pointing to an RGB565 image address, rgb1 is the next row pixel of rgb0, width and height are the width and height of the image, respectively, y0 is the brightness pixel address of the YCbCr image, and y1 is the value of y0. Next row pixel, Cb and Cr are chroma pixel values.

Step 2: If the user selects the option of "Priority on smooth playback" before playback, use the dot calculation to realize RGB conversion of resolution reduction, reduce the number of pixel points participating in the calculation, and further advance the speed of RGB conversion. Accelerate 5 is an exemplary diagram of 1/2 resolution reduced RGB switching. In Fig. 5, only a quarter of the pixels (pixel points surrounded by dashed lines) participate in the calculation. In the same way, in the 1/3 resolution reduction RGB conversion, only 1/9 of the pixels participate in the calculation. You can continue to infer in this way.

The steps of the resolution reduction RGB conversion method are as follows.

Step 21: calculate the resolution reduction ratio 1 / R (R is a positive integer) according to the width and height of the raw image and the surface_width and height surface_height of the target display buffer placed in the RGB conversion module in the player core. , make width / R and height / R closest to surface_width and surface_height. In other words,

Step 22: In calculating the gap point, as shown in Fig. 5, in the calculation of Step 1, only one pixel point is selected at intervals of (R1) pixels in the horizontal and vertical directions to participate in the calculation. Therefore, only width × height / R2 pixel points in the whole image participate in the RGB conversion. The width of the RGB image after the conversion is width / R, and the height is height / R. The width and height are input to the next module for further processing.

Step 3: A parallel RGB conversion is realized by serially converting the YCbCr to RGB module with the decoding module in the FFMPE's intermediate decoding thread.

4) Specific embodiments of an image rapid scale of any proportion

The converted RGB image size is width × height or (width / R) × (height / R), which may be different from the size surface_width × surface_height of the target display buffer. The present invention scales the output image to the size of the target display buffer by using any proportional image rapid scaling method. Specific implementation steps are as follows.

Step 1: Write Non-Memory Operations Directly to the Target Display Buffer

In order to prevent extra memory allocation and memory operations, the scale module obtains the point of the target display buffer address through the adapter layer before scaling the moving picture of each frame, and the scale module stores the memory for the target picture. The target picture is directly written in the address indicated by the point without allocation, and the moving picture display module then determines when to display the RGB picture already written in the address.

Step 2: choose scale method and display mode according to scaling

Different scales or display modes are applied to images of different resolutions.

Step 21: If the image resolution is smaller than 1/2 of the screen size (i.e., all of the transverse sizes are smaller than 1/2 of the size corresponding to the target display buffer), the image is enlarged by applying the bilinear interpolation method and enlarged. In the process of maintaining the aspect ratio of the source image, one side that does not reach the full screen size is placed at the center position of the side corresponding to the display area, and the black side (the RGB components are all initial values 0). Display). This is because the decoding speed of a small resolution image is sufficiently fast, so that the bilinear interpolation method with relatively high output and relatively good image enlargement quality can be applied.

Step 22: Otherwise, if the picture resolution is less than or equal to the screen size (i.e., all the seedling sizes are less than or equal to the size corresponding to the target display buffer), display the picture directly without scaling the picture. Copies to the center of the area, and displays the remaining display area with black edges. This is because the image resolution is similar to the display size, it is possible to save time and skip the calculation of the image magnification when displaying directly according to the raw size.

Step 23: Otherwise, the image is reduced to the screen size using the latest Lean method, maintains the aspect ratio of the source image in the process of reducing, and the center position of the side corresponding to the display area in which one side does not reach the full screen size. Placed in, the areas with insufficient size on both sides are displayed as black sides. This is because the quality of the image reduced by the recent Lean method is similar to the quality of the image by the bilinear interpolation method, but the amount of calculation is much smaller than that of the bilinear interpolation method.

Step 3: Algorithm-level Optimization of Bilinear Interpolation and Recent Lean Methods

A) Calculation method according to bilinear interpolation and recent Lean method

If the width of the image that needs to be scaled is W and the height is H, then W = width or width / R (using a resolution reduction RGB conversion), H = height or height / R (using a resolution reduction RGB conversion). The target picture is the target display buffer, and the sizes are surface_width and surface_height. The scaling r is calculated as follows.

이고, 상기 좌표값은 부동 소수점이고, 이가 원 화상 중에서의 근접 픽셀은 A, B, C, D이다.Where r is a floating point, min is the minimum, and r is a unified scaling element in the horizontal and vertical directions. As shown in Fig. 6, any one point S whose coordinate is (x, y) in the target image is converted to a position corresponding to the original image through the ratio r is

And the coordinate value is a floating point, and the neighboring pixels in the original image are A, B, C, and D.

aa) The method for calculating the S point pixel value according to the recent method is as follows.

That is, one floating-point coordinate is obtained through reverse switching of the image coordinate after the scale, and the integer coordinate is obtained after the singular processing. The pixel value corresponding to this integer coordinate in the original image is the pixel value of the target pixel immediately after the scale. According to the above method, the calculation is simple and the calculation speed is high, but it can cause a serration effect on the image after the scale.

bb) A method for calculating the S point pixel value according to the bilinear interpolation method is as follows.

In other words, a single floating point coordinate is obtained by inverting the coordinate of the image after the scale, and the pixel value obtained by performing a weighting operation according to a specific weight on four pixels around the position corresponding to the original image of the floating point is immediately scaled. Pixel value of the target pixel. The present invention applies four quarters by weight. That's a 4 taps average filter. These filters have the smoothing of low pass filtering and can remove serrations to some extent, but can also cause blurring of the hard contours.

Comparing the two methods, the recent Lean method has a small output, but when the image is enlarged by a relatively large proportion, the image quality is relatively low, and serrations are relatively large. However, when reducing a relatively large proportional image, the image quality of the two methods is similar. Therefore, the present invention applies the scale and display selection method in step 2 to trade off the image quality and the yield.

B) Optimization of calculation method according to bilinear interpolation and recent Lean method

When the image is scaled using two methods, the most time-consuming operation is to produce floating point pixel coordinates. The present invention optimizes the two methods by applying the following steps.

aa) Calculate the raising flag useX.

일 경우, useX = 1이고, 반대로, useX = 0이다.if

, UseX = 1, and vice versa.

useX indicates whether to use the x-axis scaling to achieve unified scaling, a value of 0 indicates using the y-axis scaling, and a value of 1 indicates the use of the x-axis scaling.

bb) The upper left coordinate of the screen display area in the target display buffer and the size of the display area are calculated as follows.

if (useX == 1)

{

    startX = 0;

    width = surface_width;

    height = H * surface_width / W;

    startY = MAX ((surface_height height) / 2 1, 0);

}

else

{

    startY = 0;

    width = W * surface_height / H;

    height = surface_height;

    startX = MAX ((surface_width width) / 2 1, 0);

}

cc) Four global arrays are defined and the integer coordinate conversion table is expressed as follows.

static int * T_yWyH;

static int * T_xWxH;

static int * T_corDst;

static int * T_corSrc;

When the first frame video image is to be scaled, a space is assigned to a coordinate conversion table and assigned a value as follows.

T_yWyH = (int *) malloc ((height + 1) * sizeof (int));

T_xWxH = (int *) malloc ((width +1) * sizeof (int));

T_corDst = (int *) malloc ((height + 1) * sizeof (int));

T_corSrc = (int *) malloc ((height + 1) * sizeof (int));

for (y = 0; y <height; y ++)

{

T_yWyH [y] = (useX? (Y * W / surface_width): (y * H / surface_height));

T_corDst [y] = (y + startY) * surface_width;

T_corSrc [y] = T_yWyH [y] * W;

}

for (x = 0; x <width; x ++)

{

T_xWxH [x] = (useX? (X * W / surface_width): (x * H / surface_height));

}

dd) During the scaling process of each frame, the pixel value of the final image is stored toward the address (16-bit unsigned short type point dest) of the target display buffer.

The scale of the recent Lean method is advanced using the following method.

int destY, srcY;

for (y = 0; y <height; y ++)

{

destY = T_corDst [y];

srcY = T_corSrc [y];

for (x = 0; x <width; x ++)

{

        dest [destY + (x + startX)] = src [srcY + T_xWxH [x]];

    }

}

The scale of the bilinear interpolation is performed using the following method.

unsigned short destR, destG, destB;

unsigned short srcR00, srcG00, srcB00;

unsigned short srcR01, srcG01, srcB01;

unsigned short srcR10, srcG10, srcB10;

unsigned short srcR11, srcG11, srcB11;

int destY, srcY0, srcY1, srcX0, srcX1;

for (y = 0; y <height; y ++)

{

destY = T_corDst [y];

    srcY0 = T_corSrc [y];

    srcY1 = T_corSrc [y + 1];

    for (x = 0; x <width; x ++)

    {

        srcX0 = T_xWxH [x];

        srcX1 = T_xWxH [x + 1];

        srcR00 = ((src [srcY0 + srcX0] & 0xf800) >> 11) << 3;

        srcG00 = ((src [srcY0 + srcX0] & 0x07e0) >> 5) << 2;

        srcB00 = (src [srcY0 + srcX0] & 0x001f) << 3;

        srcR01 = ((src [srcY1 + srcX0] & 0xf800) >> 11) << 3;

        srcG01 = ((src [srcY1 + srcX0] & 0x07e0) >> 5) << 2;

        srcB01 = (src [srcY1 + srcX0] & 0x001f) << 3;

        srcR10 = ((src [srcY0 + srcX1] & 0xf800) >> 11) << 3;

        srcG10 = ((src [srcY0 + srcX1] & 0x07e0) >> 5) << 2;

        srcB10 = (src [srcY0 + srcX1] & 0x001f) << 3;

        srcR11 = ((src [srcY1 + srcX1] & 0xf800) >> 11) << 3;

        srcG11 = ((src [srcY1 + srcX1] & 0x07e0) >> 5) << 2;

        srcB11 = (src [srcY1 + srcX1] & 0x001f) << 3;

        destR = (srcR00 + srcR01 + srcR10 + srcR11) >> 2;

        destG = (srcG00 + srcG01 + srcG10 + srcG11) >> 2;

        destB = (srcB00 + srcB01 + srcB10 + srcB11) >> 2;

        dest [destY + (x + startX)] = ((destR >> 3) << 11) | ((destG >> 2) << 5) | (destB >> 3);

}

}

As described above, the optimization of the two scale methods through the vertex method and the coordinate conversion table includes only the integer addition method, the table search, and the simple logic operation, so the output is drastically lowered. On the other hand, as can be seen from the above, the yield of the bilinear interpolation method is several tens of times of the recent Lean method, and the recent Lean method can simultaneously process 16-bit RGB values of one pixel. Therefore, the present invention uses only bilinear interpolation to zoom in on a small resolution image having a sufficiently fast decoding speed, and uses the recent Lean method to zoom out on a large picture with relatively slow decoding.

After processing the last one frame image, the space of four coordinate conversion tables is released to rearrange resources.

As can be seen from the above embodiment, all the modules in the multimedia software player after optimization according to the technical solution of the present invention are realized by using software which is not related to hardware, which is a multimedia playback method with universality and various It can support a wide range of video formats.

In addition, the present invention performs an optimized combination of video decoding, color conversion, image scale, decoding speed control, video frame display, system memory use, and frame buffer, thereby achieving smoothness of video playback as a final goal. Not only that, but also it can effectively reduce the occupancy of the CPU and system resources and improve the smoothness of the reproduction.

In the image processing apparatus according to the embodiment of the present invention, the term in the above-described embodiment may be defined as follows.

Frame dropping may mean dropping a frame and may mean reducing it.

The frame rate may be referred to as frame rate below.

Loss decoding may mean reducing the number of frames processed or discarding the frames.

The loss display may mean image processing to reduce the number of frames or discard the frames without image output.

The target frame may be a frame of the output image.

The target frame rate may be expressed as a frame rate of the output image.

The raw frame rate may be a frame rate of the input image.

The target frame dropping rate may mean a ratio of reducing the number of frames of the output image or discarding the frames.

The real time frame dropping rate may mean a real time reduction rate of a frame.

The decoder may mean a decoding unit for decoding the input image.

The above terms may be applied to a description of an image processing apparatus according to an embodiment of the present invention to be described below.

7 is a block diagram illustrating a configuration of an image processing apparatus according to an exemplary embodiment. An image processing apparatus according to an embodiment of the present invention may include an image receiver, an image processor, a display, a storage, and a controller.

The image receiver receives the image signal / image data by wire or wirelessly and transmits the image signal / image data to the image processor. The image receiver may be provided in various ways according to the standard of the received image signal and the implementation form of the display apparatus 100. For example, the image receiver may receive a radio frequency (RF) signal, composite video, component video, super video, SCART, high definition multimedia interface (HDMI), and DisplayPort. ), A UDI (Unified Display Interface), or a wireless signal (wireless HD) standards, etc. can be received. The video receiver may include a tuner for tuning the broadcast signal for each channel if the video signal is a broadcast signal. The receiver receives a broadcast signal as a video signal from a broadcast signal transmitter (not shown) such as a TV broadcast signal, receives a video signal from a video device such as a DVD player or a BD player, receives a video signal from a PC, or A video signal may be received from a mobile device such as a phone or a smart pad, a video signal may be received through a network such as the Internet, or video content stored in a storage medium such as a USB storage medium may be input as the video signal. In another embodiment, the image signal may not be received through the receiver, but may be stored and provided in a storage unit (not shown).

The type of the image processing process performed by the image processing unit is not limited. For example, the decoding process corresponding to the image format of the image data, the deinterlacing process for converting the interlaced image data into the progressive process de-interlacing, scaling to adjust image data to a predetermined resolution, noise reduction for improving image quality, detail enhancement, and frame refresh rate conversion. can do.

The image processing unit is implemented as an image processing board (not shown) by mounting on a printed circuit board a system-on-chip (SOC) that integrates these various functions, or individual components capable of independently performing each of these processes. It can be built into the device. The image processor processes a predetermined image signal to display an image. The image processor may be configured to display an image including an item of at least one GUI. The image processed by the image processor may be output to a monitor or a display device such as a TV and displayed.

Types of image processing processes performed by the image processing unit include, for example, de-multiplexing for distributing a predetermined signal into signals for each characteristic, decoding, and interlace schemes corresponding to image formats of the image signals. De-interlacing to convert the video signal of the video signal progressively, noise reduction for image quality improvement, detail enhancement, frame refresh rate conversion, etc. It may include a decoder (not shown) for decoding the source image corresponding to the image format of the encoded source image and a frame buffer (not shown) for storing the decoded source image in units of frames.

The display unit displays an image based on the image signal output from the image processor. The implementation of the display unit is not limited, such as liquid crystal, plasma, light emitting diode, organic light emitting diode, surfaceconduction electronemitter, carbon nanotube It can be implemented by various display methods such as nanotubes and nanocrystals. The display unit may further include additional components according to the implementation manner. For example, when the display unit is a liquid crystal type, the display unit liquid crystal display panel (not shown), a backlight unit (not shown) for supplying light thereto, and a panel driving substrate (not shown) for driving the panel (not shown) It may include.

The storage unit may allow data to remain even when power is cut off from the image processing apparatus, and may be provided as a writable nonvolatile memory (Writable ROM) to reflect changes of the user. That is, the storage unit may be provided as one of a flash memory, an EPROM, and an EEPROM.

The controller may perform overall control in the image processing apparatus. In particular, the image processing speed of the input image is compared with the frame rate of the input image. When the frame rate of the input image is greater than the image processing speed of the input image, the frame rate of the output image is set to be smaller than the frame rate of the input image. The number of frames processed by the image of the input image may be reduced according to the set result.

8 is a block diagram illustrating a configuration of an image processing apparatus according to another exemplary embodiment. The image processing apparatus according to an exemplary embodiment of the present invention may further include a conversion unit, an output image storage unit of a decoding unit and a storage unit, and a user input unit in the image processing apparatus of FIG. 7.

The conversion unit may be provided to convert from the YCbCr format to the RGB format.

The decoding unit may implement multiplexing / demultiplexing with a configuration mainly serving as an image processing unit.

The user input unit transmits various preset control commands or information to the controller according to a user's operation and input. The user input unit may be implemented as a menu key or an input panel installed outside the display device, or a remote controller separated from the display device. Alternatively, the user input unit may be integrally implemented with the display unit. When the display unit is a touch screen, the user may transmit a preset command to the controller by touching an input menu (not shown) displayed on the display unit. Can be.

Here, the control unit calculates a real time reduction rate of the frame of the input image according to the image processing speed of the input image and the frame rate of the output image, and the image processing unit to perform downsampling of the frame of the input image according to the calculated real time reduction rate. Can be controlled.

Frames of the normal B frame, the lowest B frame of the classification B frame, the lowest P frame of the classification P frame, and the last 1 frame P in the closed loop GOP can be selected as discardable frames. The control unit selects a predetermined frame of the input image for performing the reduction of the number of frames processed according to the frame priority according to the GOP (Group of Picture) structure by downsampling, and discards the selected predetermined frame without image processing. Can be.

The control unit calculates a reduction rate of the frame of the output image according to the frame rate of the output image and the frame rate of the input image, and discards the predetermined frame evenly according to the reduction rate of the frame of the output image. When the number of frames on which the reduction of the number of frames to be imaged can be performed is smaller than the number of predetermined frames of the input image by the reduction rate of the frames of the output image, all of the reduction of the number of frames to be imaged can be performed. You can control to drop frames.

The controller may be configured according to the image processing speed and the frame rate of the output image when the number of frames in which the reduction of the number of frames to be processed is performed is smaller than the number of predetermined frames of the input image by the reduction rate of the frames of the output image. Calculate the real time reduction rate of the frame of the input image, select a predetermined frame of the input image to reduce the frame to output the image according to the calculated real time reduction rate, and the selected predetermined frame can be processed without image output. Can be controlled.

The controller calculates a ratio between the image processing speed of one frame of the input images in each GOP and the average image processing speed of the frames of the input image, and by reducing the calculated ratio and the frame for outputting the image. The value calculated by multiplying the reduction rate of the frame of the output image may be a real time reduction rate in the GOP, and the control may be performed to uniformly reduce the frame for outputting the image using the real time reduction rate in the GOP.

When the input image is converted from the YCbCr format to the RGB format, the control unit may include a predetermined conversion operation to convert the input image, thereby reducing the amount of computation during conversion.

The control unit allows the predetermined conversion operation to be transformed, including shifting, table lookup, logic operations, and a small amount of additions, so that vertex, elimination multiplication, multipoint simultaneous operation, table lookup, elimination, and limiting operations can be performed. You can control to perform the operation.

The controller may reduce the resolution of the output image by using a point calculation based on the size of the screen area of the display unit when there is an input related to the smooth reproduction priority from the user input unit.

The controller may arrange the converter and the decoder in series to perform a conversion from the YCbCr format to the RGB format and image processing of the input image in parallel. The controller can reduce the resolution at a ratio of 1/2 to 1/16.

The controller compares the resolution of the output image processed by the image processor and the size of the display screen area, and displays the image by displaying the output image and the input image or adjusting the output image based on the result of the comparison. Can be controlled.

If the resolution of the output image is less than 1/2 of the size of the screen area of the display, the controller enlarges the image using the bilinear interpolation while maintaining the aspect ratio of the input image. If at least one corresponds to the screen area of the display unit, the image processor may be controlled to position the other side at the center of the side corresponding to the screen area of the display unit.

If the resolution of the output image is smaller than the size of the display area, the controller may control the display of the image by arranging one of the output image or the input image in the center of the display area.

If the resolution of the output image is larger than the size of the screen area of the display unit, the controller reduces the size of the output image by using the latest method while maintaining the aspect ratio of the input image, and at least one of the horizontal and vertical sides of the output image. When the control panel corresponds to the screen area of the display unit, the other side may be controlled to be positioned at the center of the side corresponding to the screen area of the display unit.

The control unit vertices the coordinate operation to optimize the Lean method and the bilinear interpolation method, constructs a table for the coordinates necessary for the transformation when the frame is calculated, and coordinates of the frame without the operation The image processor may control the image processor to simultaneously process the RGB colors of the pixels by calculating the latest method.

9 is a control flowchart of an image processing apparatus according to another embodiment of the present invention. The operation of the image processing apparatus according to an embodiment of the present invention will be described with reference to FIG. 9.

The input image is received through the image receiver (S11). The storage unit stores information about an image of the input image, that is, a format, a bit rate, a frame rate of the input image, a resolution, and the like of the decoding unit of the image processor. The controller compares whether the frame rate of the input image is greater than the image processing speed of the input image (S12). If the frame rate of the input image is greater than the image processing speed of the input image, the frame rate of the output image is set to be smaller than the frame rate of the input image (S13). If the image processing speed of the input image is larger than the frame rate of the input image, the image processor performs normal image processing. The number of frames processed in the input image is reduced according to the frame rate of the set output image (S14).

As described above, in order to optimize an image, a method of reducing a frame so that a predetermined frame does not become an image output and a method of reducing an amount of calculation while converting an input image from a YCbCr format to an RGB format (resolution reduction, image processing and conversion parallel processing) , A predetermined calculation method) and a method of optimizing the image by adjusting the output image may be performed in parallel and alone with the operation of the image processing apparatus of FIG. 9.

As a result, the user can watch a smooth image without stopping during viewing, that is, without buffering.

In addition, the present invention performs an optimized combination of video decoding, color conversion, image scale, decoding speed control, video frame display, system memory use, and frame buffer, thereby achieving smoothness of video playback as a final goal. Not only that, but also it can effectively reduce the occupancy of the CPU and system resources and improve the smoothness of the reproduction.

The above description is only a description of a preferred embodiment of the present invention and is not intended to limit the present invention. All modifications, equivalent replacements, improvements, and the like, within the spirit and principles of the present invention, should all fall within the protection scope of the present invention.

While the embodiments of the present invention have been shown and described, those skilled in the art will appreciate that various modifications, additions and substitutions are possible, without departing from the scope or spirit of this invention. Accordingly, the scope of the invention is not to be determined by the embodiments described so far but on the basis of the appended claims and their equivalents.

1: Image processing device
100:
110: storage unit
112: output image storage unit
120: video receiver
130:
132: converter
134: decoding unit
140:

Claims (32)

An image processing apparatus comprising:
An image receiving unit which receives an input image;
An image processor which processes the input image and outputs an output image;
A display unit which displays the output image;
A storage unit for storing data and information of the input image;
The image processing speed of the input image is compared with the frame rate of the input image. When the frame rate of the input image is greater than the image processing speed of the input image, the frame rate of the output image is greater than the frame rate of the input image. And a controller configured to reduce the number and control the number of frames processed by the input image to be reduced according to the set result. The method of claim 1,
The control unit calculates a real time reduction rate of a frame of the input image according to the image processing speed of the input image and the frame rate of the output image, and performs downsampling of the frame of the input image according to the calculated real time reduction rate. Image processing apparatus, characterized in that. 3. The method of claim 2,
The control unit may select a predetermined frame of the input image to perform a reduction in the number of frames processed by the frame priority according to a GOP (Group of Picture) structure by the downsampling. An image processing apparatus, characterized in that discarded without an image processing process. The method of claim 3,
The controller calculates a reduction rate of a frame of the output image according to the frame rate of the output image and the frame rate of the input image.
If the predetermined frame is discarded, the number of frames for which the predetermined frame is discarded equally according to the reduction rate of the frame of the output image, and the number of frames for which the image is processed may be reduced is the reduction rate of the frame of the output image. If less than the number of the predetermined frame of the input image by the image processing apparatus, characterized in that for discarding all the frames that can be reduced the number of frames to be processed image. The method of claim 3,
The controller may control the image processing speed and the output image when the number of frames in which the reduction of the number of frames to be processed may be performed is smaller than the number of predetermined frames of the input image by the reduction rate of the frames of the output image. Calculating a real-time reduction rate of a frame of the input image according to a frame rate of the selected image, selecting a predetermined frame of the input image for performing a reduction of a frame to output an image according to the calculated real-time reduction rate, and selecting the selected predetermined The frame is an image processing apparatus, characterized in that the image processing without image output. 5. The method of claim 4,
The controller calculates a ratio between the image processing speed of one frame of the input images in each GOP and the average image processing speed of a frame of the input image processed, and outputs the calculated ratio and the image. The value calculated by multiplying the reduction rate of the frame of the output image by the reduction of the frame is used as the real time reduction rate in the GOP, and the reduction of the frame for outputting the image is performed using the real time reduction rate in the GOP. An image processing apparatus. The method of claim 1,
The image processing unit further includes a conversion unit for converting from YCbCr format to RGB format,
And converting the input image from the YCbCr format to the RGB format by including a predetermined conversion operation to reduce the amount of computation during conversion. The method of claim 7, wherein
The control unit may be configured to perform a conversion operation including shifting, table lookup, logic operation, and a small amount of addition, such that vertex, elimination multiplication, multipoint simultaneous operation, table lookup, removal addition, and the like. And an exponential limiting operation. The method of claim 7, wherein
Further comprising a user input unit for inputting a video signal,
And the controller is further configured to reduce the resolution of the output image by using a point calculation based on the size of the screen area of the display unit when there is an input related to a smooth reproduction priority from the user input unit. The method of claim 7, wherein
The image processing unit further includes a decoding unit for decoding the input image,
And the controller is configured to perform the conversion of the YCbCr format to the RGB format and image processing of the input image in parallel by arranging the converter and the decoder in series. 10. The method of claim 9,
And the control unit reduces the resolution at a ratio of 1/2 to 1/16. The method of claim 1,
The storage unit further includes an output image storage unit for storing the output image,
The control unit compares the resolution of the output image processed by the image processing unit with the size of the screen area of the display unit, and displays the output image and the input image or adjusts the output image based on the comparison result. An image processing apparatus characterized by displaying an image. The method of claim 12,
When the resolution of the output image is less than 1/2 of the size of the screen area of the display unit, the controller enlarges the image using the bilinear interpolation while maintaining the aspect ratio of the input image with respect to the output image. And at least one of the horizontal side and the vertical side of the display unit is positioned at the center of the side corresponding to the screen region of the display unit. The method of claim 12,
When the resolution of the output image is smaller than a size of the screen area of the display unit, the controller is configured to display an image by arranging one of the output image or the input image in the center of the display unit screen area. Image processing apparatus, characterized in that. The method of claim 12,
When the resolution of the output image is larger than a size of a display area of the display unit, the controller reduces the size of the output image using a recent Lean method while maintaining the aspect ratio of the input image. And when at least one of the vertical sides corresponds to the screen area of the display unit, the other side is positioned at the center of the side corresponding to the screen area of the display unit. 16. The method according to claim 13 or 15,
The control unit performs vertices on coordinate operations in order to optimize the latest Lean method and the bilinear interpolation method, constructs a table for coordinates necessary for transformation during operation of the frame, and the coordinates of the frame are examined without a table. And an additive operation, and simultaneously processing the RGB colors of the pixels by calculating the latest Lean method. A method of controlling an image processing apparatus,
Receiving an input image;
Comparing the image processing speed of the input image with the frame rate of the input image;
Setting the frame rate of the output image to be smaller than the frame rate of the input image if the frame rate of the input image is greater than the image processing speed of the input image;
And reducing the number of frames to be image-processed of the input image according to a set result. 18. The method of claim 17,
Reducing the number of frames,
Calculating a real time reduction rate of a frame of the input image according to the image processing speed of the input image and the frame rate of the output image, and performing downsampling of the frame of the input image according to the calculated real time reduction rate; Control method of an image processing apparatus. 19. The method of claim 18,
The downsampling may include:
The downsampling selects a predetermined frame of the input image to reduce the number of frames processed according to a frame priority according to a group of picture (GOP) structure, and the predetermined frame is selected without performing image processing. Control method of the image processing apparatus comprising the step of discarding. 20. The method of claim 19,
Discarding without the image processing process,
Calculating a reduction rate of the frame of the output image according to the frame rate of the output image and the frame rate of the input image,
The predetermined frames of the input image are discarded evenly according to the reduction rate of the frames of the output image, and the number of frames for which the reduction of the number of frames processed by the image is performed is determined by the reduction rate of the frames of the output image. If less than the number of, the control method of the image processing apparatus comprising the step of discarding all the frames that can be reduced the number of frames to be processed image. 20. The method of claim 19,
Discarding without the image processing process,
When the number of frames in which the reduction of the number of frames to be imaged can be performed is smaller than the number of predetermined frames of the input image by the reduction rate of the frames of the output image, the image processing speed and the frame rate of the output image Calculating a real time reduction rate of a frame of the input image and selecting a predetermined frame of the input image for reducing a frame for outputting an image according to the calculated real time reduction rate, wherein the selected predetermined frame is an image output And controlling the image without the image. 21. The method of claim 20,
Discarding all the frames,
Calculating a ratio between the image processing speed of one frame of the input image and the average image processing speed of the frame of the input image in the GOP, and reducing the frame to output the calculated ratio and the image. And a value calculated by multiplying a reduction rate of the frame of the output image by a real time reduction rate in the GOP, and uniformly reducing a frame for outputting an image using the real time reduction rate in the GOP. Control method of processing device. 18. The method of claim 17,
Reducing the number of frames,
And converting the input image from the YCbCr format to the RGB format, including a predetermined conversion operation, thereby reducing the amount of computation during conversion. 24. The method of claim 23,
The predetermined transform operation can perform a transform operation including shifting, table lookup, logic operation, and a small amount of additions to perform vertexing, elimination multiplication, multipoint simultaneous operation, table lookup, elimination, and limiting operations. Control method of an image processing apparatus, characterized in that performed. 24. The method of claim 23,
Reducing the amount of calculation,
And reducing the resolution of the output image by using a point calculation based on the size of the screen region on which the image is output when there is an input related to a smooth reproduction priority by a user. 24. The method of claim 23,
Reducing the amount of calculation,
And converting the YCbCr format from the YCbCr format to the RGB format in parallel by arranging a converting unit converting the YCbCr format into the RGB format and performing a video processing of the input image. Control method of an image processing apparatus. 26. The method of claim 25,
Reducing the resolution,
A method of controlling an image processing apparatus, characterized by reducing the resolution at a ratio of 1/2 to 1/16. 18. The method of claim 17,
Reducing the number of frames,
Comparing the resolution of the output image with the size of the screen area on which the output image is output, and displaying the image by displaying the output image and the input image or adjusting the output image based on a result of the comparison. Control method of an image processing apparatus. 29. The method of claim 28,
Displaying the image,
When the resolution of the output image and the output image are smaller than 1/2 of the size of the screen area to be output, the image is enlarged using a double linear interpolation while maintaining the aspect ratio of the input image. If at least one of a horizontal side or a vertical side corresponds to the screen area of the display unit, positioning the other side at the center of the side corresponding to the screen area of the display unit. 29. The method of claim 28,
Displaying the image,
If the resolution of the output image and the output image is smaller than the size of the screen area to be output, less than a predetermined range, one of the output image or the input image is placed in the center of the screen area to which the output image is output image Control method of the image processing apparatus comprising the step of displaying. 29. The method of claim 28,
Displaying the image,
When the resolution of the output image is larger than a size of a screen area in which the output image is output, the image is reduced by using the latest Lean method while maintaining the aspect ratio of the input image with respect to the output image. And when the at least one of the vertical sides corresponds to the screen area on which the output image is output, positioning the other side at the center of the side corresponding to the screen area of the display unit. 32. The method of claim 29 or 31,
In order to optimize the most recent method and the bilinear interpolation method, a vertex of coordinate operations is performed, and a table for coordinates necessary for transforming a frame is calculated, and the coordinates of the frame are used for table search and additive operation without calculation. The control method of the image processing apparatus, which is calculated through, and simultaneously processing the RGB color of the pixel by the operation of the latest method.

Images