KR102274747B1 - CODEC, SYSTEM ON CHIP(SoC) INCLUDING THE SAME, AND DATA PROCESSING SYSTEM INCLUDING THE SoC - Google Patents

Abstract

A codec according to an embodiment of the present invention includes a codec CPU that receives a current frame, determines a type of a received current frame, and sets rate control parameters of the current frame, and a second codec including the current frame. Allocating all target bits for a group of picture (GOP), and allocating a target bit for each of the frames included in the first GOP based on the determined type of the current frame and set rate control parameters A bit rate predictor, and adjusting the rate control parameters of the current frame, and using the target bit for each of the frames allocated by the bit rate predictor and the adjusted rate control parameters for each of the blocks included in the current frame It may include a code rate controller that generates a quantization parameter.

Description

Codec, system on chip including same, and data processing system including system on chip {CODEC, SYSTEM ON CHIP (SoC) INCLUDING THE SAME, AND DATA PROCESSING SYSTEM INCLUDING THE SoC}

An embodiment according to the concept of the present invention relates to a codec, in particular, predicting the bit-rate allocated to the current GOP regardless of the bit-rate of the previous group of picture (GOP), and predicting the bit-rate in the I-frame. The present invention relates to a codec capable of adjusting a quantization parameter for an I-frame and reducing the amount of bits generated in the I-frame, a system on a chip including the codec, and a processing system including the system on chip.

BACKGROUND With the development of Internet protocol (IP) networks, video communication over the IP networks has become very popular. Unlike traditional video transmission through a cable network, various methods for video compression have been sought for efficient transmission of video transmission over an IP network.

On the other hand, since video transmission through an IP network of a mobile device has a relatively high possibility of data packet loss in the transmission process, users are inconvenient such as video cutout and image quality degradation due to frame drop. .

In a mobile device, especially when compressing an image with little motion, a skip mode occurs a lot in a P-frame, so that a lot of bits are generated when compressing an I-frame. In this case, when the compressed I-frame is transmitted, the buffer limit of the receiving terminal is exceeded, and there is a problem in that a frame drop occurs.

A technical problem to be achieved by the present invention is to adjust the bit-rate so that a low bit-rate can be allocated to a frame to which many bits are allocated, a codec capable of encoding the frame, and a system on including the codec A chip and a data processing system including the system-on-a-chip are provided.

A codec according to an embodiment of the present invention includes a codec CPU that receives a current frame, determines a type of a received current frame, and sets rate control parameters of the current frame, and a second codec including the current frame. Allocating all target bits for a group of picture (GOP), and allocating a target bit for each of the frames included in the first GOP based on the determined type of the current frame and set rate control parameters A bit rate predictor, and adjusting the rate control parameters of the current frame, and using the target bit for each of the frames allocated by the bit rate predictor and the adjusted rate control parameters for each of the blocks included in the current frame It may include a code rate controller that generates a quantization parameter.

When the type of the current frame is an I-frame (intra frame), the bit rate predictor allocates all target bits for the first GOP regardless of the second GOP including the previous frame, and A target bit for the current frame may be allocated using the entire target bit.

When the type of the current frame is not an I-frame (intra frame), the bit rate predictor determines the difference between the total target bits allocated for the first GOP and the number of bits used to encode previous frames included in the first GOP. It is possible to allocate the target bit for the current frame using

The bit rate predictor may allocate the target bit for the current frame based on the number of unprocessed frames included in the first GOP.

and an encoder for encoding the current frame and outputting the encoded current frame based on the quantization parameter for each of the blocks, and a codec memory for storing the encoded current frame, wherein the codec CPU is configured to: Based on the encoded current frame stored in the codec memory, it is possible to check whether the codec memory is saturated.

When the codec memory is saturated, the codec CPU may skip encoding of the remaining frames included in the first GOP.

The code rate controller adjusts the quantization parameter for each of the blocks included in the current frame, transmits the quantization parameter adjusted for each block to the encoder, and the encoder uses the adjusted quantization parameter for each block Thus, encoding may be performed on each of the blocks included in the current frame.

A system-on-chip according to an embodiment of the present invention includes a pre-processing circuit that processes image data, generates and outputs a current frame using the processed image data, and receives the current frame and encodes the current frame. a codec, wherein the codec includes a codec CPU that receives a current frame, determines a type of a received current frame, and sets rate control parameters of the current frame, a first group of picture (GOP) including the current frame ), a bit rate predictor for allocating target bits for each of the frames included in the first GOP based on the determined type of the current frame and the set rate control parameters, and the current frame. A code rate controller that adjusts rate control parameters and generates a quantization parameter for each of the blocks included in the current frame by using the target bit for each of the frames allocated by the bit rate predictor and the adjusted rate control parameters. may include

When the type of the current frame is an I-frame (intra frame), the bit rate predictor allocates all target bits for the first GOP regardless of the second GOP including the previous frame, and assigns the entire target bits to the allocated first GOP. A target bit for the current frame may be allocated using the entire target bit for .

When the type of the current frame is not an I-frame (intra frame), the bit rate predictor is the difference between all target bits allocated for the first GOP and the number of bits used to encode previous frames included in the first GOP can be used to allocate the target bit for the current frame.

The bit rate predictor may be included in the first GOP and allocate the target bit for the current frame based on the number of unprocessed frames.

and an encoder for encoding the current frame and outputting the encoded current frame based on the quantization parameter for each of the blocks, and a codec memory for storing the encoded current frame, wherein the codec CPU is configured to: Based on the encoded current frame stored in the codec memory, it is possible to check whether the codec memory is saturated.

When the codec memory is saturated, the codec CPU may skip encoding of the remaining frames included in the first GOP.

The code rate controller adjusts the quantization parameter for each of the blocks included in the current frame, transmits the quantization parameter adjusted for each block to the encoder, and the encoder uses the adjusted quantization parameter for each block Thus, encoding may be performed on each of the blocks included in the current frame.

When a video call application is executed, the data processing system according to an embodiment of the present invention includes a camera device for generating image data, processing image data received from the camera device, and generating a current frame using the processed image data. a pre-processing circuit that outputs, and a codec that receives the current frame and encodes the current frame, wherein the codec receives the current frame, determines a type of the received current frame, and A codec CPU that sets rate control parameters, allocates all target bits for a first group of picture (GOP) including the current frame, and based on the determined type of the current frame and the set rate control parameters, to the first GOP a bit rate predictor for allocating a target bit for each of the included frames, and adjusting the rate control parameters of the current frame, and calculating the target bit and adjusted rate control parameter parameters for each of the frames allocated by the bit rate predictor It may include a code rate controller that generates a quantization parameter for each of the blocks included in the current frame by using it.

When the type of the current frame is an I-frame (intra frame), the bit rate predictor allocates all target bits for the first GOP regardless of the second GOP including the previous frame, and assigns the entire target bits to the allocated first GOP. A target bit for the current frame may be allocated using the entire target bit for .

When the type of the current frame is not an I-frame (intra frame), the bit rate predictor is the difference between all target bits allocated for the first GOP and the number of bits used to encode previous frames included in the first GOP can be used to allocate the target bit for the current frame.

The bit rate predictor may be included in the first GOP and allocate the target bit for the current frame based on the number of unprocessed frames.

and an encoder for encoding the current frame and outputting the encoded current frame based on the quantization parameter for each of the blocks, and a codec memory for storing the encoded current frame, wherein the codec CPU is configured to: Based on the encoded current frame stored in the codec memory, it is possible to check whether the codec memory is saturated.

The code rate controller adjusts the quantization parameter for each of the blocks included in the current frame, transmits the quantization parameter adjusted for each block to the encoder, and the encoder uses the adjusted quantization parameter for each block Thus, encoding may be performed on each of the blocks included in the current frame.

The codec according to an embodiment of the present invention and devices including the same have the effect of encoding the I-frame by limiting the amount of encoded bits of the I-frame.

Accordingly, the codec and the devices including the codec are effective in reducing a frame drop in image data transmission and reducing image quality degradation.

In order to more fully understand the drawings recited in the Detailed Description of the Invention, a detailed description of each drawing is provided.
1 is a block diagram of a data processing system according to an embodiment of the present invention.
FIG. 2 is a block diagram of a video communication system including the data processing system shown in FIG. 1 .
3 is a schematic block diagram of the codec shown in FIGS. 1 and 2 .
FIG. 4 is a detailed block diagram of the codec shown in FIG. 3 .
FIG. 5 is a block diagram for explaining a method in which the bit rate predictor of FIG. 4 allocates a target bit for a group of picture (GOP) when a current frame is an I-frame.
6 is a block diagram for explaining a method for the bit rate predictor of FIG. 4 to allocate a target bit for a GOP when the current frame is not an I-frame.
FIG. 7 is a block diagram illustrating a method in which the code rate controller of FIG. 4 adjusts a quantization parameter.
FIG. 8 is a graph for explaining buffer saturation of a block for adjusting the quantization parameter of FIG. 7 .
9 is a table for explaining a method of adjusting the quantization parameter of FIG. 7 .
10A is a graph illustrating a target bit for a frame encoded by the encoder of FIG. 4 in a normal mode.
FIG. 10(b) is a graph illustrating a target bit for a frame encoded by the encoder of FIG. 4 in a flat mode.
11 is a graph for explaining the operation of the codec when the codec memory of FIG. 4 is saturated.
12 is a flowchart illustrating a normal mode and a flat mode according to an embodiment of the present invention.
13 is a flowchart illustrating an operation of a codec in a flat mode according to an embodiment of the present invention.
14 is a flowchart for explaining the operation of the codec when the codec memory is saturated.

Specific structural or functional descriptions of the embodiments according to the concept of the present invention disclosed in this specification are only exemplified for the purpose of explaining the embodiments according to the concept of the present invention, and the embodiments according to the concept of the present invention are It may be implemented in various forms and is not limited to the embodiments described herein.

Since the embodiments according to the concept of the present invention may have various changes and may have various forms, the embodiments will be illustrated in the drawings and described in detail herein. However, this is not intended to limit the embodiments according to the concept of the present invention to specific disclosed forms, and includes all modifications, equivalents, or substitutes included in the spirit and scope of the present invention.

Terms such as first or second may be used to describe various elements, but the elements should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another, for example without departing from the scope of the inventive concept, a first component may be termed a second component and similarly a second component A component may also be referred to as a first component.

When a component is referred to as being “connected” or “connected” to another component, it may be directly connected or connected to the other component, but it is understood that other components may exist in between. it should be On the other hand, when it is said that a certain element is "directly connected" or "directly connected" to another element, it should be understood that the other element does not exist in the middle. Other expressions describing the relationship between components, such as "between" and "immediately between" or "neighboring to" and "directly adjacent to", etc., should be interpreted similarly.

The terms used herein are used only to describe specific embodiments, and are not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present specification, terms such as "comprise" or "have" are intended to designate that a feature, number, step, operation, component, part, or combination thereof described herein exists, but one or more other features It is to be understood that it does not preclude the possibility of the presence or addition of numbers, steps, operations, components, parts, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present specification. does not

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram of a data processing system according to an embodiment of the present invention.

The data processing system 10 includes a TV, digital TV (DTV), internet protocol TV (IPTV), personal computer (PC), desktop computer, lap-top computer, computer workstation, It may be implemented as a tablet PC, a video game platform (or video game console), a server, or a mobile computing device.

The mobile computing device includes a mobile phone, a smart phone, a personal digital assistant (PDA), an enterprise digital assistant (EDA), a digital still camera, a digital video camera, and a portable digital assistant (PMP). multimedia player), PND (personal navigation device or portable navigation device), mobile internet device (MID), wearable computer, internet of things (IoT) device, internet of everything (IoE) )) device, or may be implemented as an e-book.

The data processing system 10 may refer to various devices capable of processing 2D (dimensional) or 3D graphics data and displaying the processed data.

The data processing system 10 may include a camera 50 , a system on chip (SoC) 100 , a display 200 , an input device 210 , and a second memory 220 . Although the second memory 220 is shown outside the SoC 100 in FIG. 1 , the second memory 220 may be implemented inside the SoC 100 according to an embodiment.

The camera 50 may be implemented as a CMOS image sensor. The camera 50 may photograph a subject, generate first data IM for the subject, and output the generated first data IM to the SoC 100 . The first data IM may be still image data or moving image data.

SoC 100 may control overall operation of data processing system 10 . For example, SoC 100 may be an integrated circuit (IC), motherboard, application processor (AP), or mobile capable of performing operations according to embodiments of the present invention to be described herein. (mobile) may mean an AP.

That is, the SoC 100 processes the first data IM output from the camera 50 , and displays the processed data on the display 200 , stores it in the second memory 220 , or another data processing system. can be sent to The data IM output from the camera 50 may be transmitted to the pre-processing circuit 110 through a MIPI® camera serial interface (CSI).

SoC 100 includes pre-processing circuit 110 , codec 120 , CPU 130 , first memory 140 , display controller 150 , memory controller 160 , bus 170 , modem 180 . ), and a user interface 190 .

The pre-processing circuit 110 , the codec 120 , the CPU 130 , the first memory 140 , the display controller 150 , the memory controller 160 , the modem 180 , and the user interface 190 are bus Through 170, data may be transmitted or received from each other. For example, the bus 170 is a PCI bus (Peripheral Component Interconnect Bus), PCI Express (PCI Express (PCIe)) bus, AMBA (Advanced High Performance Bus), AHB (Advanced High Performance Bus), APB (Advanced Peripheral Bus), It may be implemented as an Advanced eXtensible Interface (AXI) bus, or a combination thereof, but is not limited thereto.

The pre-processing circuit 110 receives the first data IM output from the camera 200 , processes the received first data IM, and outputs the second data FI generated according to the processing result. It can output to the codec 120 . For example, the pre-processing circuit 110 may transmit the second data FI generated according to the processing result to the codec 120 through the bus 170 or directly.

Each of the first data IM and the second data FI may mean a frame (or picture). Hereinafter, for convenience of description, each data (IM and FI) is referred to as a current frame (or a current picture).

According to an embodiment, the pre-processing circuit 110 may be implemented as an image signal processor (ISP). For example, the ISP may convert the first data IM having the first data format into the second data FI having the second data format. For example, the first data IM may be data having a Bayer pattern and the second data FI may be YUV data, but is not limited thereto.

In FIG. 1 , the pre-processing circuit 110 is illustrated as being implemented inside the SoC 100 , but according to embodiments, the pre-processing circuit 110 may be implemented outside the SoC 100 .

The codec 120 may perform an encoding (or encoding) operation on each of a plurality of blocks included in the current frame FI.

The encoding operation is a joint picture expert group (JPEG), motion picture expert groups (MPEG), MPEG-2, MPEG-4, VC-1, H.264, H.265, or High Efficiency Video Coding (HEVC). An image data encoding technique such as the like may be used, but is not limited thereto.

Although the codec 120 in FIG. 1 is implemented as a hardware codec, the codec according to the technical idea of the present invention may be implemented as a hardware codec or a software codec. The software codec may be executed by the CPU 130 .

The CPU 130 may control the operation of the SoC 100 .

A user may provide input to SoC 100 , such that CPU 130 may execute one or more applications (eg, software applications (APP) 135 ).

According to an embodiment, any one of the applications 135 executed by the CPU 130 may mean a video call application. In addition, the applications 135 executed by the CPU 130 may include an operating system (OS), a word processor application, a media player application, a video game application, and/or a graphical user interface (GUI). )) may include an application, but is not limited thereto. Hereinafter, for convenience of description, the video call application is called an application 135 .

The first memory 140 may receive and store data encoded (or encoded) by the codec 120 as the application 135 is executed under the control of the memory controller. Also, the first memory 140 may transmit data stored by the application 135 to the CPU 130 or the modem 180 under the control of the memory controller.

The first memory 140 may write data for the application 135 executed by the CPU 130 , and read data about the application 135 stored in the first memory 140 .

For example, the first memory 140 may be implemented as a volatile memory such as a static random access memory (SRAM) or a nonvolatile memory such as a read only memory (ROM).

The display controller 150 may transmit data output from the codec 120 or the CPU 130 to the display 200 . The display 200 includes a monitor, a TV monitor, a projection device, a thin film transistor-liquid crystal display (TFT-LCD), a light emitting diode (LED) display, an organic LED (OLED) display, an active-matrix (AMOLED) display. It may be implemented as an OLED) display or a flexible display.

For example, the display controller 150 may transmit data to the display 200 through a MIPI display serial interface (DSI).

The input device 210 may receive a user input input from a user and transmit an input signal corresponding to the user input to the user interface 190 .

The input device 210 may be implemented as a touch panel, a touch screen, a voice recognizer, a touch pen, a keyboard, a mouse, a track point, etc., but is not limited thereto. For example, when the input device 210 is a touch screen, the input device 210 may include a touch panel and a touch panel controller. Also, when the input device 210 is a voice recognizer, the input device 210 may include a voice recognition sensor and a voice recognition controller.

The input device 210 may be connected to the display 200 or may be implemented separately from the display 200 .

According to an embodiment, when the user executes the application icon 205 displayed on the display 200 using the input device 210 , the input device 210 may generate an input signal.

The application icon 205 may be created by an application 135 that may be executed on the CPU 130 . A plurality of application icons may be displayed on the display 200 . For example, when the application 135 is a video call application, the application icon 205 may be an icon for executing the video call application.

The input device 210 may transmit an input signal to the user interface 190 .

The user interface 190 may receive an input signal from the input device 210 and transmit data corresponding to the input signal to the CPU 130 .

According to an embodiment, the CPU 130 may receive data corresponding to an input signal and execute the video call application 135 in response to the data.

The memory controller 160 may read data stored in the second memory 220 under the control of the codec 120 or the CPU 130 and transmit the read data to the codec 120 or the CPU 130 . have. Also, the memory controller 160 may write data output from the codec 120 or the CPU 130 to the second memory 220 under the control of the codec 120 or the CPU 130 .

The second memory 220 may be implemented as a volatile memory and/or a nonvolatile memory. The volatile memory may include random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), thyristor RAM (T-RAM), zero capacitor RAM (Z-RAM), or twin (TTRAM). Transistor RAM).

The nonvolatile memory includes electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic RAM (MRAM), spin-transfer torque MRAM (MRAM), ferroelectric RAM (FeRAM), and phase change (PRAM). RAM) or RRAM (resistive RAM).

In addition, the nonvolatile memory may include a multimedia card (MMC), an embedded MMC (eMMC), a universal flash storage (UFS), a solid state drive or solid state disk (SSD), a USB flash drive, or a hard disk drive. (hard disk drive (HDD)) may be implemented.

The modem 180 may output data encoded by the codec 120 or the CPU 130 to the outside using a wireless communication technology. The wireless communication technology may mean Wi-Fi (WI-FI), WiBRO (WIBRO), 3G wireless communication, long term evolution (LTETM), long term evolution-advanced (LTE-A), or broadband LTE-A can

FIG. 2 is a block diagram of a video communication system including the data processing system shown in FIG. 1 . Referring to FIG. 2 , the video communication system 20 may include a first data processing system 10 - 1 and a second data processing system 10 - 2 that can communicate with each other through a channel 300 . . The video communication system 20 may refer to a video telephony system.

The structure and operation of each of the first data processing system 10-1 and the second data processing system 10-2 are substantially the same or similar.

The first data processing system 10 - 1 may include a video camera 50 - 1 , a codec 120 - 1 , a buffer 140 - 1 , and a modem 180 - 1 .

The first data processing system 10-1 encodes (or encodes) data INPUT received from the video camera 50-1, and processes the encoded data EI through the channel 300 as second data. can be transmitted to the system 10-2.

The video camera 50 - 1 may be substantially the same as the camera 50 shown in FIG. 1 , and the codec 120 - 1 may be substantially the same as the codec 120 shown in FIG. 1 , and a buffer Reference numeral 140 - 1 may be substantially the same as the first memory 140 shown in FIG. 1 , and the modem 180 - 1 may be substantially the same as the modem 180 shown in FIG. 1 .

The second data processing system 10 - 2 may receive the encoded data EI transmitted from the first data processing system 10 - 1 through the channel 300 .

The second data processing system 10 - 2 may include a display 200 - 2 , a codec 120 - 2 , a buffer 140 - 2 , and a modem 180 - 2 .

The modem 180 - 2 may transmit the encoded data EI transmitted from the first data processing system 10 - 1 to the buffer 140 - 2 . The modem 180 - 2 may be substantially the same as the modem 180 illustrated in FIG. 1 .

The buffer 140 - 2 may receive the encoded data EI from the modem 180 - 2 and transmit the encoded data EI to the codec 120 - 2 . The buffer 140 - 2 may be substantially the same as the first memory 140 shown in FIG. 1 .

The codec 120 - 2 may receive the coded data EI and decode (or decode) the coded data EI. For example, the codec 120 - 2 may include a function of a decoder.

The display 200 - 2 may display data decoded by the codec 120 - 2 . The display 200 - 2 may be substantially the same as the display 200 illustrated in FIG. 1 .

The first data processing system 10 - 1 and the second data processing system 10 - 2 may perform bidirectional communication through the channel 300 . According to embodiments, the channel 300 is Wi-Fi (WI-FI), WiBro (WIBRO), 3G wireless communication, LTE (long term evolution), LTE-A (long term evolution-advanced), or broadband LTE-A can support

3 is a schematic block diagram of the codec shown in FIGS. 1 and 2 .

1 to 3 , the codec 120 may include a codec CPU 122 , a hardware block 126 , and a codec memory 128 .

The codec CPU 122 may store the current frame FI output from the pre-processing circuit 110 in the codec memory 128 .

The codec CPU 122 may determine the type of the current frame FI and generate the type information TI using the determined type of the current frame FI. The codec CPU 122 may determine rate control parameters of the current frame FI and generate parameter information PI using the determined rate control parameters.

The rate control parameter may refer to parameters capable of adjusting (or controlling) the bit-rate of the current frame FI.

The codec CPU 122 may output the type information TI, the parameter information PI, and the current frame FI to the hardware block 126 .

The firmware 124 executed in the codec CPU 122 determines whether the current frame FI is an i-frame, a p-frame, or a b-frame, according to a characteristic of a group of picture (GOP), and the result of the determination The type information TI may be generated according to

According to embodiments, a group of picture (GOP) may include at least one of an i-frame, a b-frame, and a p-frame. For example, among the frames included in the GOP, a first frame may be an i-frame, and all others may be an i-frame, a b-frame, and/or a p-frame. According to embodiments, the number of frames included in the GOP and/or the order of frames having different types transmitted from the pre-processing circuit 110 may be variously changed.

The firmware 124 executed in the codec CPU 122 may determine the rate control parameters of the current frame FI and generate parameter information PI according to the determination result.

The rate control parameters of the current frame (FI) are, the complexity according to each type of frames, the size of the GOP (eg, total target bits for the GOP), the number of frames per second (or pictures per second) corresponding to the frame-rate, It may include the number of bits per second corresponding to the bit-rate, a normalization constant, and/or an initial parameter. The initial parameter may include a first constant (k2), a second constant (r_seq), an initial frame quantization parameter, and an initial buffer saturation (d0).

The codec CPU 122 may calculate the complexity of the current frame FI for each GOP.

The codec CPU 122 may calculate the complexity xi, xp, or xb of the current frame FI by using Equation 1.

[Equation 1]

xi=160*(br)/115, xp=60*(br)/115, and xb=42*(br)/115

Here, xi is a complexity when the current frame FI is an i-frame, xp is a complexity when the current frame FI is a p-frame, and xb is a complexity when the current frame FI is a b-frame. br may mean the bit-rate of the current frame FI or the number of bits per second.

For example, the codec CPU 122 may determine the bit-rate br, and use the determined bit-rate to determine the complexity of the i-frame (xi), the complexity of the p-frame (xp), and/or the b-frame. We can determine the complexity (xb) of

For example, if the bit-rate is 115, xi is 160, xp is 60, and xb is 42.

The parameter information PI may include the rate control parameters of the current frame FI. For example, the parameter information PI may include the complexity xi, xp, or xb of the current frame FI, the GOP size, the number of bits per second, and/or the number of frames per second.

The codec CPU 122 may transmit an encoding (or encoding) end signal DI for the current frame FI output from the hardware block 126 to the pre-processing circuit 110 .

The codec CPU 122 may check whether the codec memory 128 is saturated and generate a check signal BF corresponding to the check result.

The hardware block 126 may encode the current frame FI using the type information TI and the parameter information PI transmitted from the codec CPU 122 .

The hardware block 126 may store the encoded current frame EI in the codec memory 128 .

When encoding (or encoding) of the current frame FI is finished, the hardware block 126 may transmit an encoding (or encoding) end signal DI to the codec CPU 122 .

The codec memory 128 may receive and store the encoded current preview EI transmitted from the hardware block 126 .

The codec memory 128 may include a plurality of buffers and/or a plurality of virtual buffers. In some embodiments, the codec memory 128 may be implemented as a volatile memory or a nonvolatile memory.

FIG. 4 is a detailed block diagram of the codec shown in FIG. 3 . 3 and 4 , the codec 120 may include a codec CPU 122 , a hardware block 126 , and a codec memory 128 . The hardware block 126 may include a bit rate predictor 126 - 1 , a code rate controller 126 - 2 , and an encoder 126 - 3 .

The bit rate predictor 126 - 1 may allocate a target bit for the current frame FI by using the type information TI and the parameter information PI transmitted from the codec CPU 122 .

The target bit may mean the number of bits (or bit capacity) allocated by the bit rate predictor 126 - 1 to encode (or encode) the current frame FI.

According to an embodiment, the bit rate predictor 126 - 1 allocates all target bits for the GOP including the current frame FI, and uses the type information TI and the parameter information PI of the current frame FI. Thus, it is possible to allocate a target bit to each of the frames included in the GOP.

FIG. 5 is a block diagram for explaining a method in which the bit rate predictor 126 - 1 of FIG. 4 allocates all target bits for a group of picture (GOP) when the current frame is an I-frame.

4 and 5, when the type information (TI) of the current frame (FI = FI1 = I1)) indicates an i-frame, the bit rate predictor 126-1, the second GOP including the previous frame All target bits TB1 for the first GOP (GOP1) may be allocated irrespective of the target bits.

According to an embodiment, when the target bit allocated for the second GOP (GOP0) is different from the target bit used when actually encoding (or encoding) the second GOP (GOP0), the bit rate predictor 126 - 1, the second GOP Regardless of the difference in target bits generated in (GOP0), all target bits TB1 for the first GOP (GOP1) may be allocated.

That is, the bit rate predictor 126 - 1 may allocate the entire target bit TB1 to the entire I-frame I1 and the P-frames (P1 to PN, where N is a natural number equal to or greater than 2). The bit rate predictor 126 - 1 may calculate the total target bits TB1 for the first GOP GOP1 using Equation 2 .

[Equation 2]

TB=K*br/f

Here, TB denotes the total target bits TB1 of the first GOP (GOP1) including the current frame FI, K denotes the size of the first GOP (GOP1), and br denotes the bit-rate of the current frame FI. (bit-rate), and f denotes a frame-rate (frame-rate or picture-rate).

For example, if the bit-rate br is 300, the size K of the first GOP GOP1 is 30, and the number of frames f is 30, the total target bits TB becomes 300. Further, when the bit-rate br is 300, the size K of the first GOP GOP1 is 300, and the number of frames f is 30, the total target bits TB becomes 3000.

The bit rate predictor 126 - 1 may allocate the target bit for the current frame FI or FI1 by using the total target bit TB1 for the predicted first GOP (GOP1). For example, the bit rate predictor 126 - 1 may allocate (or calculate) a target bit for the current frame FI or FI1 using Equation 3 .

[Equation 3]

T=(xL/kL)/(xi/ki+Np*xp/kp+Nb*xb/kb)*R

where xL is the complexity of the L-th frame (L is a natural number), xi is the complexity of the i-frame, xp is the complexity of the p-frame, xb is the complexity of the b-frame, kn is the normalization constant of the n-th frame, and ki is the Normalization constant of i-frame (ki=1), kp is normalization constant of p-frame (kp=1), kb is normalization constant of b-frame (kb=1.4), Np is p-frame not processed in GOP The number, Nb, is the number of unprocessed b-frames in the GOP, and R is the number of bits that are not yet used while encoding the GOP.

6 is a block diagram for explaining a method for the bit rate predictor of FIG. 4 to allocate a target bit for a GOP when the current frame is not an i-frame.

4 and 6, when the type information TI of the current frame (FI=FI2=P2) indicates that it is not an i-frame, the bit rate predictor 126-1 determines that the current frame FI is i - Since all target bits for the first GOP (GOP1) have already been allocated in the frame-in step, all target bits TB2 for the first GOP (GOP1) are not allocated.

According to an embodiment, the bit rate predictor 126 - 2 encodes the entire target bit TB2 allocated for the first GOP (GOP1) and the previous frames I1 and P1 included in the first GOP (GOP1) (or A target bit for the current frame FI, FI2, or P2 may be allocated using a difference RB of the number of bits PB used for encoding). The difference between the total target bits TB2 allocated for the first GOP (GOP1) and the number of bits (PB) used to encode (or encode) the previous frames I1 and P1 included in the first GOP (GOP1) remains It may be expressed as the number of bits (RB).

The bit rate predictor 126 - 1 may allocate a target bit for the current frame FI, FI2, or P2 based on the number D of unprocessed frames included in the first GOP (GOP1).

For example, the number of unprocessed frames (D) is that if the current frame (FI, FI2, or P2) is a p-frame and the total number of p-frames is 29, the number of unprocessed frames (D) is 28. do.

The bit rate predictor 126 - 1 may calculate a target bit for the current frame FI, FI2, or P2 by using Equation (3).

The bit rate predictor 126 - 1 may transmit bit information BI including the target bit for the current frame FI to the code rate controller 126 - 2 . Also, the bit rate predictor 126 - 1 may transmit the parameter information PI and the current frame FI to the code rate controller 126 - 2 .

The bit information BI may include target bits for the current frame FI, and may include all target bits allocated for the first GOP GOP1 including the current frame FI. Also, the bit information BI may include a target bit for each of the frames included in the first GOP GOP1.

The code rate controller 126 - 2 may adjust rate control parameters of the current frame FI based on the received parameter information PI.

The code rate controller 126 - 2 quantizes each of the blocks included in the current frame FI using the target bit and the adjusted parameters for each of the frames allocated by the bit rate predictor 126 - 2 . A parameter (QP) can be created.

FIG. 7 is a block diagram illustrating a method in which the code rate controller of FIG. 4 adjusts a quantization parameter.

4 and 7 , the code rate controller 126 - 2 may generate a quantization parameter (eg, QP, QP1, or QP1') for each of the blocks included in the current frame FI.

For example, it is assumed that the current frame FI includes 4*4 blocks. The number of blocks included in the current frame is not limited thereto. The current frame FI may include 16 blocks BL1 to BL16, and the code rate controller 126-2 generates each quantization parameter QP1 to QP16 for each block BL1 to BL16. can

According to an embodiment, the code rate controller 126 - 2 may generate a quantization parameter QP1 for the first block BL1. The code rate controller 126 - 2 may adjust the quantization parameter QP1 for the first block BL1 and generate the adjusted quantization parameter QP1'.

The code rate controller 126 - 2 may transmit the adjusted quantization parameters (QP=QP1' to QP16') for each of the blocks included in the current frame FI to the encoder 126 - 3 .

The code rate controller 126 - 2 may calculate a quantization parameter (QP) for each of the blocks using Equation (4).

[Equation 4]

QP=(k2/31)*dn/r_seq

Here, QP is a quantization parameter, k2 is a first constant, r_seq is a second constant, and dn is buffer saturation.

FIG. 8 is a graph for explaining buffer saturation of a block for adjusting the quantization parameter of FIG. 7 . 4, 7, and 8, EST represents a function for a target bit allocated for each block included in the current frame, and REAL is a function for the number of bits actually used for each block included in the current frame. indicates

dn may mean buffer saturation of the nth block included in the current frame.

The code rate controller 126-2 is actually used in encoding (or encoding) for the previous block (n-1 th block) from the encoder 126-3 in order to calculate the buffer saturation dn of the n th block. may receive the number of bits (BS).

The code rate controller 126-2 may calculate the buffer saturation dn using Equation 5.

[Equation 5]

dn=d0+BS-(ETB/TBL)*(n-1)

Here, dn is the buffer saturation for the current block (eg, the nth block), d0 is the initial buffer saturation, EB is the target bit allocated for encoding (or encoding) for the previous block, and BS is the encoding for the previous block. The number of bits used in (or encoding), ETB is the total target bits allocated for encoding (or encoding) for the current frame FI, TBL is the total number of blocks included in the current frame FI, n is the current It means the order of the current block included in the frame.

That is, the buffer saturation dn means the difference between the number of bits actually used in the previous block BS and the target bits EB allocated in the previous block.

9 is a table for explaining a method of adjusting the quantization parameter of FIG. 7 .

4, 7, and 9 , k2 and r_seq may be included in rate control parameters for the current frame FI. Accordingly, the codec CPU 122 may determine the k2 value and the r_seq value. According to an embodiment, k2 and r_seq are constant values.

The code rate controller 126 - 2 may adjust the constant k2 and the constant r_seq, and adjust the quantization parameters QP or QP1 to QP16 using the adjusted k2 and the adjusted r_seq.

According to an embodiment, the code rate controller 126 - 2 may maintain constant k2 and r_seq values in a normal mode (NM).

The code rate controller 126-2 may multiply the k2 value by A1(A1>1) in the flat mode (FM_) and multiply the r_seq value by B1(B1<1). That is, the code rate The controller 126-2 may increase the value of k2 by multiplying the value of k2 by a value greater than 1. Also, the code rate controller 126-2 may decrease the value of r_seq by multiplying the value of r_seq by a value smaller than 1. have.

The normal mode (NM) may mean a mode in which the code rate controller 126-2 does not change k2 and r_seq, and the flat mode (FM) is a mode in which the code rate controller 126-2 changes k2 and r_seq. mode can mean.

The code rate controller 126-2 may calculate the adjusted quantization parameters QP or QP1' to QP16' using Equation (4).

The code rate controller 126 - 2 may transmit the adjusted quantization parameters QP or QP1' to QP16' for each of the blocks included in the current frame FI to the encoder 126 - 3 .

The encoder 126 - 3 may encode (or encode) the current frame FI in block units by using quantization parameters (QP=QP1 to QP16). Also, the encoder 126 - 3 may encode (or encode) the current frame FI in units of blocks by using the adjusted quantization parameters (QP=QP1' to QP16').

According to an embodiment, in the normal mode (NM), the encoder 126 - 3 encodes (or encodes) the current frame FI in block units using the unaltered quantization parameters (QP=QP1 to QP16). can In the flat mode FM, the encoder 126 - 3 may encode (or encode) the current frame FI in blocks by using the changed quantization parameters (QP=QP1' to QP16').

When encoding of the current frame FI is completed, the encoder 126 - 3 may transmit an encoding completion signal DI to the codec CPU 122 . The encoding completion signal DI may include information on the number of bits used while the current frame FI is encoded by the encoder 126 - 3 .

When encoding (or encoding) of the current frame FI is completed, the encoder 126 - 3 may store the encoded current frame EI in the codec memory 128 .

10A is a graph showing the number of bits for a frame encoded by the encoder 126-3 of FIG. 4 operating in a normal mode.

The graph of FIG. 10 (a) shows the number of bits (BIT1) used in encoding for the current frame (eg, the nth frame) in the normal mode.

According to an embodiment, when the current frame (eg, the nth frame) is an i-frame, the graph of FIG. 10A may indicate the number of bits BIT1 used in encoding for the current frame.

FIG. 10B is a graph showing the number of bits (BIT2) for a frame encoded by the encoder 126-3 of FIG. 4 operating in a flat mode.

Referring to FIG. 10B , the graph of FIG. 10B shows the number of bits (BIT2<BIT1) used in encoding for the current frame (eg, the nth frame) in the flat mode.

According to an embodiment, when the current frame (eg, the n-th frame) is an i-frame, the graph of FIG. 10B may indicate the number of bits (BIT2) used in encoding for the current frame.

Referring to FIGS. 10A and 10B , the number of bits required to encode the current frame in the normal mode is greater than the number of bits required to encode the current frame in the flat mode. That is, the number of bits required to encode the current frame FI is significantly reduced in the flat mode.

11 is a graph for explaining the operation of the codec when the codec memory of FIG. 4 is saturated. 4 and 11 , the codec CPU 122 checks whether the codec memory 128 is saturated based on the coded current frame EI stored in the codec memory 128, and checks signal BF. can create

When the number of bits of the coded frames stored in the codec memory 128 reaches the memory capacity allocated to the GOP including the current frame FI, the codec CPU 128 may end encoding for the current frame FI. .

According to an embodiment, the codec CPU 122 includes all target bits allocated to the GOP including the current frame FI and the encoded (or encoded) frames (EI1 to EIN) for the GOP stored in the codec memory 128 . ) can be compared to the total number of bits.

When the codec memory 128 is full, the codec CPU 122 may end encoding for the remaining unprocessed frames included in the GOP. For example, the codec CPU 128 may skip encoding for the remaining frames until the saturation of the codec memory 128 falls below a certain amount. The predetermined amount may be determined by a user's setting or a preset standard. Also, the codec CPU 128 may transmit the encoding end signal DI to the preprocessing circuit 110 .

When the codec memory 128 is not full, the codec CPU 122 may continue to encode the remaining unprocessed frames included in the GOP.

12 is a flowchart illustrating a normal mode and a flat mode according to an embodiment of the present invention.

1, 4, and 12 , the codec CPU 122 may determine whether encoding for the current frame FI is performed in the flat mode or the normal mode ( S101 ). The CPU 130 may transmit a result of the determination to the codec CPU 122 .

When it is determined that the encoding of the current frame FI is performed in the normal mode (No in S101), the codec CPU 122 encodes the current frame FI in the normal mode (S103).

The bit rate predictor 126 - 1, when the current frame (FI) is an i-frame, using the number of bits remaining after encoding in the GOP including the previous frame, all target bits for the GOP including the current frame (FI) can be assigned (S105). For example, the number of bits remaining after encoding in the GOP including the previous frame may be additionally allocated to all target bits for the GOP including the current frame FI ( S105 ).

The code rate controller 126-2 may transmit the unadjusted quantization parameter QP for each of the blocks included in the current frame FI to the encoder 126-3 without adjusting the quantization parameter QP. . The encoder 126 - 3 may encode the current frame FI using the unadjusted quantization parameter QP ( S107 ).

When it is determined that the encoding of the current frame FI is performed in the flat mode, the codec CPU 122 may encode the current frame FI in the flat mode (S109).

13 is a flowchart illustrating an operation of a codec in a flat mode according to an embodiment of the present invention. 4 and 13 , the codec CPU 122 may determine rate control parameters of the current frame FI ( S201 ).

The bit rate predictor 126 - 1 is, when the current frame (FI) is an i-frame, regardless of the number of bits remaining after encoding in the previous GOP including the previous frame, the entire GOP including the current frame (FI) A target bit may be allocated (S203).

The bit rate predictor 126 - 1 may allocate a target bit to each of the frames included in the GOP including the current frame FI ( S205 ).

The code rate controller 126 - 2 may adjust the rate control parameters of the current frame FI and adjust the quantization parameters for each block of the current frame FI ( S207 ). The code rate controller 126 - 2 may transmit the adjusted quantization parameter (QP) to the encoder 126 - 3 .

The encoder 126 - 3 may encode the current frame FI in block units by using the adjusted quantization parameter ( S209 ). When encoding (or encoding) of the current frame FI is completed, the encoder 126 - 3 may transmit an encoding end signal DI to the codec CPU 122 .

The codec CPU 122 may determine whether the current frame is the last frame of the GOP (S211). For example, when the next frame of the current frame FI is an i-frame, the current frame FI may be the last frame of the GOP.

When the current frame FI is the last frame of the GOP, the bit rate predictor 126 - 1 may allocate all target bits for the new GOP including the next frame.

When the current frame FI is not the last frame of the GOP, the bit rate predictor 126 - 1 may predict the target bit for the next frame without allocating all target bits for the GOP.

14 is a flowchart for explaining the operation of the codec when the codec memory is saturated. 4 and 14 , the encoder 126 - 3 may encode the current frame FI ( S301 ). The encoder 126 - 3 may complete encoding of the current frame FI and store the encoded current frame EI in the codec memory 128 ( S303 ).

The codec CPU 122 may check whether the codec memory 128 is saturated and generate a check signal BF (S305). The codec CPU 122 compares the number of bits of the encoded frames stored in the codec memory 128 with the total target bits that can be stored in the memory space of the codec memory 128 allocated to the GOP, and determines whether or not saturation is achieved according to the comparison result can be checked (S307).

When the codec memory 128 is not saturated, the encoder 126 - 3 may encode the next frame of the current frame FI ( S307 ). When the codec memory 128 is saturated, the codec CPU 122 may skip encoding for the next frame (S307). The codec CPU 122 may skip encoding of the remaining frames until the saturation of the codec memory 128 falls below a certain amount.

Although the present invention has been described with reference to the embodiment shown in the drawings, this is merely exemplary, and those skilled in the art will understand that various modifications and equivalent other embodiments are possible therefrom. Accordingly, the true technical protection scope of the present invention should be determined by the technical spirit of the appended claims.

10, 10-1: data processing system
50: camera
110: preprocessing circuit
120: codec
122: codec CPU
126: hardware block
126-1: bit rate predictor
126-2: code rate controller
126-3: encoder
128: codec memory
130: CPU
135: application
140: first memory
180: modem
200: display
220: second memory

Claims (10)

a codec CPU that receives a current frame, determines a type of the received current frame, and sets rate control parameters of the current frame;
Allocate all target bits for a first group of picture (GOP) including the current frame, and based on the determined type of the current frame and set rate control parameters, each of the frames included in the first GOP a bit rate predictor for allocating target bits for ; and
Adjusts the rate control parameters of the current frame, and generates a quantization parameter for each of the blocks included in the current frame by using the target bit for each of the frames allocated by the bit rate predictor and the adjusted rate control parameters Including a code rate controller to
The bit rate predictor,
When operating in the normal mode, all target bits are allocated to the first GOP by using the number of bits remaining after encoding in the second GOP including the previous frame,
When operating in the flat mode, allocating all target bits for the first GOP, regardless of the number of bits remaining after encoding in the second GOP including the previous frame,
The code rate controller is
When operating in the normal mode, without adjusting the quantization parameter,
A codec for adjusting the quantization parameter when operating in the flat mode. According to claim 1,
When the type of the current frame is an I-frame (intra frame),
The bit rate predictor allocates all target bits for the first GOP regardless of the second GOP including the previous frame, and uses the all target bits for the allocated first GOP to determine the target bit for the current frame Allocating codec. According to claim 1,
When the type of the current frame is not an I-frame (intra frame),
The bit rate predictor allocates the target bit for the current frame by using a difference between all target bits allocated for the first GOP and the number of bits used to encode previous frames included in the first GOP. 4. The method of claim 3,
The bit rate predictor is a codec for allocating the target bit for the current frame based on the number of unprocessed frames included in the first GOP. According to claim 1,
an encoder that encodes the current frame based on the quantization parameter for each of the blocks and outputs the encoded current frame; and
Further comprising a codec memory for storing the encoded current frame,
The codec CPU checks whether the codec memory is saturated based on the encoded current frame stored in the codec memory. 6. The method of claim 5,
When the codec memory is saturated, the codec CPU skips encoding of the remaining frames included in the first GOP. 6. The method of claim 5,
The code rate controller adjusts the quantization parameter for each of the blocks included in the current frame, and transmits the adjusted quantization parameter for each block to the encoder,
and the encoder performs encoding on each of the blocks included in the current frame by using the adjusted quantization parameter for each block. a pre-processing circuit that processes the image data and generates and outputs a current frame by using the processed image data; and
and a codec for receiving the current frame and encoding the current frame,
The codec is
a codec CPU that receives a current frame, determines a type of the received current frame, and sets rate control parameters of the current frame;
Allocate all target bits for a first group of picture (GOP) including the current frame, and based on the determined current frame type and set rate control parameters, target bits for each of the frames included in the first GOP a bit rate predictor that assigns and
Adjusts the rate control parameters of the current frame, and generates a quantization parameter for each of the blocks included in the current frame by using the target bit for each of the frames allocated by the bit rate predictor and the adjusted rate control parameters Including a code rate controller to
The bit rate predictor,
When operating in the normal mode, all target bits are allocated to the first GOP by using the number of bits remaining after encoding in the second GOP including the previous frame,
When operating in the flat mode, allocating all target bits for the first GOP, regardless of the number of bits remaining after encoding in the second GOP including the previous frame,
The code rate controller is
When operating in the normal mode, without adjusting the quantization parameter,
a system on chip for adjusting the quantization parameter when operating in the flat mode. 9. The method of claim 8,
an encoder that encodes the current frame based on the quantization parameter for each of the blocks and outputs the encoded current frame; and
Further comprising a codec memory for storing the encoded current frame,
The codec CPU checks whether the codec memory is saturated based on the encoded current frame stored in the codec memory. When the video call application is executed, the camera device for generating video data;
a pre-processing circuit for processing the image data received from the camera device, and generating and outputting a current frame using the processed image data; and
and a codec for receiving the current frame and encoding the current frame,
The codec is
a codec CPU that receives a current frame, determines a type of the received current frame, and sets rate control parameters of the current frame;
Allocate all target bits for a first group of picture (GOP) including the current frame, and based on the determined current frame type and set rate control parameters, target bits for each of the frames included in the first GOP a bit rate predictor that assigns and
Adjust the rate control parameters of the current frame, and use the target bit for each of the frames allocated by the bit rate predictor and the adjusted rate control parameter parameters to determine a quantization parameter for each of the blocks included in the current frame A code rate controller for generating
The bit rate predictor,
When operating in the normal mode, all target bits are allocated to the first GOP by using the number of bits remaining after encoding in the second GOP including the previous frame,
When operating in the flat mode, allocating all target bits for the first GOP, regardless of the number of bits remaining after encoding in the second GOP including the previous frame,
The code rate controller is
When operating in the normal mode, without adjusting the quantization parameter,
a data processing system for adjusting the quantization parameter when operating in the flat mode.

Images