CN115174916A - Video data storage method and device, electronic equipment and readable storage medium - Google Patents

Video data storage method and device, electronic equipment and readable storage medium Download PDF

Info

Publication number
CN115174916A
CN115174916A CN202110355145.3A CN202110355145A CN115174916A CN 115174916 A CN115174916 A CN 115174916A CN 202110355145 A CN202110355145 A CN 202110355145A CN 115174916 A CN115174916 A CN 115174916A
Authority
CN
China
Prior art keywords
video
video frame
decoder
frame
block
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202110355145.3A
Other languages
Chinese (zh)
Inventor
赵娟萍
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Guangdong Oppo Mobile Telecommunications Corp Ltd
Original Assignee
Guangdong Oppo Mobile Telecommunications Corp Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Guangdong Oppo Mobile Telecommunications Corp Ltd filed Critical Guangdong Oppo Mobile Telecommunications Corp Ltd
Priority to CN202110355145.3A priority Critical patent/CN115174916A/en
Priority to PCT/CN2022/077080 priority patent/WO2022206212A1/en
Publication of CN115174916A publication Critical patent/CN115174916A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
    • H04N19/136—Incoming video signal characteristics or properties
    • H04N19/137—Motion inside a coding unit, e.g. average field, frame or block difference
    • H04N19/139—Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
    • H04N19/156—Availability of hardware or computational resources, e.g. encoding based on power-saving criteria
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/42—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation
    • H04N19/423—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation characterised by memory arrangements
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/42—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation
    • H04N19/43—Hardware specially adapted for motion estimation or compensation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/42—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation
    • H04N19/43—Hardware specially adapted for motion estimation or compensation
    • H04N19/433—Hardware specially adapted for motion estimation or compensation characterised by techniques for memory access

Abstract

The embodiment of the application provides a video data storage method, a video data storage device, an electronic device and a readable storage medium, wherein the method comprises the following steps: in response to a decoding process of a video frame to be reconstructed, reading a reference video frame of the video frame to be reconstructed from a main storage external to a video decoder in the absence of the reference video frame in the main storage and in the presence of the reference video frame in the main storage; storing the read reference video frame into an internal buffer of the video coder. By adopting the method, the power consumption of the electronic equipment can be reduced.

Description

Video data storage method and device, electronic equipment and readable storage medium
Technical Field
The present application relates to the field of video processing technologies, and in particular, to a video data storage method and apparatus, an electronic device, and a computer-readable storage medium.
Background
With the development of new video standards, such as MPEG-1/2/4/5, h.263/264/265/266, vp8/9, etc., the picture size of video is larger and the frame rate of video is higher, resulting in a conventional video data storage mode, which has a problem of high power consumption of electronic devices.
Disclosure of Invention
The embodiment of the application provides a video data storage method and device, electronic equipment and a computer readable storage medium, which can reduce the power consumption of the electronic equipment.
A video data storage method, comprising:
in response to a decoding process of a video frame to be reconstructed, reading a reference video frame of the video frame to be reconstructed from a main storage external to a video coder in a case where the reference video frame does not exist in the main storage and exists in the main storage; and
storing the reference video frame in main storage into an internal buffer of the video coder.
A method of video data reading, the method comprising:
responding to the decoding processing of the acquired video code stream, reading a reference video block which does not exist in external storage of the video decoder from a main storage outside the video decoder to an internal cache of the video decoder, and then performing compensation or block copying;
alternatively, motion compensation or block copy is performed after the reference video block is read from external storage of the video decoder.
A video data storage apparatus comprising:
a first reading module, configured to, in response to a decoding process of a video frame to be reconstructed, read a reference video frame of the video frame to be reconstructed from a main storage outside a video decoder in a case where the reference video frame does not exist in the main storage and exists in the main storage outside the video decoder; and
a reference video frame storage module for storing the reference video frame read from main storage into an internal buffer of the video decoder.
A video data reading apparatus comprising:
a second reading module, configured to respond to a decoding process of the acquired video stream, read, from a main storage outside the video decoder, a reference video block that is not stored outside the video decoder to an internal cache of the video decoder, and then perform a compensation or block copy operation;
and the third reading module is used for performing motion compensation or block copying after reading the reference video block from the external storage of the video decoder.
An electronic device comprising a memory and a processor, the memory having stored therein a computer program that, when executed by the processor, causes the processor to perform the steps of the methods.
A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the respective method.
In response to the decoding processing of the video frame to be reconstructed, reading the reference video frame from the main storage under the condition that the reference video frame does not exist in the external storage of the video decoder and the reference video frame exists in the main storage outside the video decoder, and storing the reference video frame read from the main storage into the internal cache of the video decoder; that is, video data present in the external storage of the video decoder is not stored in the internal buffer of the video decoder, and video data in the main storage is stored in the internal buffer of the video decoder. Compared with the conventional method that the video data is placed in the decoder internal buffer by the decoder no matter the video data is located in the external storage or the main storage, in the embodiment, only the video data which is not located in the external storage and is located in the main storage is placed in the decoder internal buffer, so that the use of the internal buffer of the video decoder is reduced, the efficiency of the decoder internal buffer is increased, and the power consumption is further reduced specially for the data which is only located in the main storage.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a block diagram illustrating a conventional video data storage method according to an embodiment;
FIG. 2 is a block diagram of a conventional video decoder data processing architecture in one embodiment;
FIG. 3 is a schematic diagram illustrating a conventional video data storage scheme according to another embodiment;
FIG. 4 is a graphical illustration of system power consumption in a conventional manner in one embodiment;
FIG. 5 is a flow diagram of a method for video data storage in one embodiment;
FIG. 6 is a block diagram of an embodiment of a video data store;
FIG. 7 is an architectural diagram illustrating the use of a system cache in one embodiment;
FIG. 8 is a block diagram illustrating an architecture for using system caches in one embodiment;
FIG. 9 is a diagram illustrating a decoding process in one embodiment;
FIG. 10 is a flow diagram of a method for video data reading in one embodiment;
FIG. 11 is a flow diagram illustrating reconstruction of a video frame in one embodiment;
FIG. 12 is a diagram illustrating an example of a conventional manner of storing internal memory in a video decoder;
FIG. 13 is a diagram illustrating the storage of internal storage in a video decoder, according to one embodiment;
FIG. 14 is a block diagram of a video data storage device of an embodiment;
FIG. 15 is a block diagram of a video data reading apparatus according to an embodiment;
fig. 16 is a schematic diagram of the internal structure of the electronic device in one embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
It will be understood that the terms "first," "second," and the like as used herein may be used herein to describe various data, but the data is not limited by these terms. These terms are only used to distinguish one datum from another. For example, a first order condition may be referred to as a second order condition, and similarly, a second order condition may be referred to as a first order condition, without departing from the scope of the present application. Both the first order condition and the second order condition are order conditions, but they are not the same order condition.
In one embodiment, conventional electronic devices basically use DRAM as a storage space for video frames. Fig. 1 is a schematic diagram illustrating an architecture of a conventional video data storage method in an embodiment. Fig. 1 includes a Central Processing Unit (CPU), a Video Decoder (VDEC), a Display Processing Unit (Display Processing Unit), and a dynamic random access memory Controller (DRAM Controller). The central processing unit comprises a master port (master), the video decoder comprises an internal cache (cache) and a master port (master), the display processor comprises a master port (master), and the dynamic random access memory controller comprises a slave port (slave). The arrows in fig. 1 represent video data. When video data exists, the video decoder can acquire the video data from the dynamic random access memory controller to perform video decoding, reconstruct the video data to obtain a video frame, and transmit the reconstructed video frame to the display processor to be displayed.
Fig. 2 is a block diagram of a conventional video decoder data processing architecture in one embodiment. The Video bitstream (Video Bitstreams) is input to a Video decoder for decoding, and Video frames (frame buffers) and some Temporal data (Temporal data) such as Temporal Motion Vectors (TMVs) are generated. And the video bit stream, video frames, and temporary data are stored in the DRAM.
Conventional video decoders can have corresponding buffering strategies for handling various video bitstreams. However, with the new video standards, such as h.265/266, AV, MPEG-5, etc., for the video with larger and larger size and higher frame rate, the conventional technology adopts the architecture as shown in fig. 3, and the bandwidth of DRAM is widened or the DRAM frequency is increased to speed up the data throughput. Fig. 3 is a schematic diagram illustrating a conventional video data storage method according to another embodiment. The figure includes a central processing unit, a video decoder, a display processor, and a Multi-channel DRAM controller (Multi-channel DRAM), wherein the Multi-channel DRAM controller may be used to control a plurality of DRAMs.
FIG. 4 is a graphical illustration of system power consumption in a conventional manner in one embodiment. In fig. 4, the abscissa is the Position in the image (Picture Position) and the ordinate is the system power consumption (system power). The dotted line represents the upper System Power Barrier. As shown in fig. 4, the system DRAM needs to consume a higher system power to meet the operating speed requirement of the video decoder. In the case where the video decoder relies heavily on DRAM or other power consuming storage, the video decoder may not be able to meet the requirements or the system may overheat due to the limited upper limit of power consumption that the system can provide. The conventional approach causes the video decoder to consume more power if it is done at the expected time.
In order to solve the problem of high power consumption of electronic devices, embodiments of the present application provide a video data storage method, by responding to a decoding process of a video frame to be reconstructed, a reference video frame of the video frame to be reconstructed does not exist in an external storage of a video decoder, and in a case where the reference video frame exists in a main storage outside the video decoder, the reference video frame is read from the main storage; storing the reference video frame read from the main storage into an internal buffer of the video decoder; compared with the conventional method that the video data is placed in the decoder internal buffer by the decoder no matter the video data is located in the external storage or the main storage, in the embodiment, only the video data which is not located in the external storage and is located in the main storage is placed in the decoder internal buffer, so that the use of the internal buffer of the video decoder is reduced, the efficiency of the decoder internal buffer is increased, and the power consumption is further reduced specially for the data which is only located in the main storage.
The video data storage method is described below with reference to specific embodiments. In the embodiments of the present application, the main memory is exemplified as DRAM. Fig. 5 is a flow chart of a video data storage method in one embodiment. The video data storage method in this embodiment is described by taking an example of the video data storage method running on an electronic device. As shown in fig. 5, the video data storage method includes steps 502 to 504.
Step 502, in response to the decoding process of the video frame to be reconstructed, reading the reference video frame from the main storage in the case that the reference video frame of the video frame to be reconstructed does not exist in the external storage of the video decoder and the reference video frame exists in the main storage outside the video decoder.
The video frame to be reconstructed refers to a video frame which is not successfully constructed. After the video is compressed, if a video frame needs to be obtained, decoding processing needs to be performed. External storage of the video decoder refers to storage located in the electronic device and other than main storage and the internal buffer of the video decoder. For example, the external storage of the video decoder may be at least one of a system cache, a system buffer. Main memory refers to Main memory (Main memory), which is referred to as Main memory for short. Main storage is an important component of computer hardware that stores instructions and data and is directly accessible at random by a Central Processing Unit (CPU). The main Memory may specifically be a DRAM (Dynamic Random Access Memory).
The reference video frame can be used for reconstructing a target video frame corresponding to the video frame to be reconstructed. The reference video frame may be a video frame displayed before or after the video frame to be reconstructed. The reference video frame is specifically a video frame that has been reconstructed before the video frame to be reconstructed, or the reference video frame may also be a reconstructed portion of the video frame to be reconstructed itself, for example, a block vector (block vector) used for intra block copy in the screen content coding technology identifies the taken picture range. The reference picture range specification allowed in the screen content coding technology has been detailed in various video standards, and is not described herein in detail.
Specifically, in response to the decoding processing of the video frame to be reconstructed, the electronic device obtains a target motion vector or block vector corresponding to the video frame to be reconstructed. Detecting whether a reference video frame of a video frame to be reconstructed exists in an external storage of a video decoder. In the case where there is no reference video frame corresponding to the target motion vector or block vector in the external storage of the video decoder, and there is a reference video frame in the main storage external to the video decoder, the electronic device reads the reference video frame from the main storage and stores the read reference video frame into the internal buffer of the video decoder.
Step 504 stores the reference video frame read from main storage into an internal buffer of the video decoder.
Specifically, the electronic device stores the reference video frames read from the main storage into an internal buffer of the video decoder.
In the video data storage method in this embodiment, in response to the decoding processing of the video frame to be reconstructed, when the reference video frame of the video frame to be reconstructed does not exist in the external storage of the video decoder and the reference video frame exists in the main storage outside the video decoder, the reference video frame is read from the main storage, and the reference video frame read from the main storage is stored in the internal buffer of the video decoder, that is, the video data existing in the external storage of the video decoder does not exist in the internal buffer of the video decoder, and the video data in the main storage can exist in the internal buffer of the video decoder. Compared with the conventional method that the video data is placed in the decoder internal buffer by the decoder no matter the video data is located in the external storage or the main storage, in the embodiment, only the video data which is not located in the external storage and is located in the main storage is placed in the decoder internal buffer, so that the use of the internal buffer of the video decoder is reduced, the efficiency of the decoder internal buffer is increased, and the power consumption is further reduced specially for the data which is only located in the main storage.
In one embodiment, as shown in fig. 6, an architecture diagram of video data storage in one embodiment is shown. The white part indicates that it is stored in the DRAM, i.e. this part of the data is subsequently stored in the internal buffer of the video decoder, and the grey part indicates that it is stored in the external memory of the video decoder. In contrast to the storage scheme of fig. 3, the video bitstream, the video frame data, and the temporary data in fig. 6 are partially stored in the DRAM, and partially stored in the external memory of the video decoder.
In one embodiment, each frame of video is divided into a plurality of blocks. Reading a reference video frame from a main storage external to a video coder in the absence of a reference video frame of a video frame to be reconstructed in the external storage of the video coder and in the presence of a reference video frame in the main storage, comprising:
in the case where there is no reference video block corresponding to the video frame to be reconstructed in the external storage of the video decoder, and there is a reference video block in the main storage external to the video decoder, the reference video block is read from the main storage.
Storing a reference video frame read from main storage into an internal buffer of a video decoder, comprising: the reference video blocks read from main storage are stored into an internal buffer of the video decoder.
The block refers to a partial area of a video frame. For example, the size of the block may be 16 × 16 pixels, 64 × 64 pixels, and the like, but is not limited thereto.
Specifically, in response to the decoding processing of the acquired video, the electronic device divides each video frame in the video into a plurality of blocks and obtains a motion vector corresponding to each block. The electronic device determines a reference video block in a reference video frame to which a video frame to be reconstructed refers based on a motion vector of a current block in a current video frame. The electronic device reads the reference video block from the main storage if the reference video block corresponding to the video frame to be reconstructed does not exist in the external storage of the video decoder and if the reference video block exists in the main storage outside the video decoder. The electronics store the reference video block read from main storage into an internal buffer of the video decoder.
In this embodiment, when the reference video block corresponding to the video frame to be reconstructed does not exist in the external storage of the video decoder and the reference video block exists in the main storage outside the video decoder, the electronic device reads the reference video block from the main storage and stores the reference video block into the internal cache of the video decoder.
In one embodiment, the video data storage method further comprises: in the case where the reference video frame is present in the external storage of the video coder, the reference video frame is read from the external storage of the video coder.
Specifically, in the case that a reference video frame exists in the external storage of the video decoder, the electronic device reads the reference video frame from the external storage of the video decoder, and reconstructs a target video frame corresponding to the video frame to be reconstructed based on the reference video frame read from the external storage of the video decoder.
In this embodiment, when the reference video frame exists in the external storage of the video decoder, the reference video frame is read from the external storage of the video decoder, and data access is performed through the external storage of the video decoder, so that the efficiency of the internal cache of the decoder can be increased, and further power consumption reduction is performed specifically for data that is still only left in main storage.
In one embodiment, in response to the decoding process of the video frame to be reconstructed, in a case where a reference video frame of the video frame to be reconstructed does not exist in the external storage of the video decoder and a reference video frame exists in the main storage external to the video decoder, before reading the reference video frame from the main storage, the video data storage method further includes:
and (a 1) responding to the decoding processing of the acquired video, and acquiring a motion vector corresponding to each video frame in the video.
The type of video is not limited. For example, the downloaded video may be stored locally in the electronic device, or the video may be obtained from a server. The video may be a television show video, a movie video, a television show video, or the like. The length of the video is not limited, and may be, for example, 10 seconds, 1 minute, 10 minutes, or the like, without being limited thereto. The decoding process refers to a process of decoding and restoring the encoded video to reconstruct a video frame. The decoding process may also be referred to as a decoding process. A Motion Vector (MV) represents the relative displacement between a block of a current video frame and a matching block in its reference image. The Block Vector (BV) represents the relative displacement between a Block of the current video frame and a matching Block in the current image frame that it has reconstructed.
Specifically, the electronic device obtains, by the video decoder, a motion vector corresponding to each video frame in the video in response to a decoding process of the obtained video. It is understood that, in response to the decoding processing of the acquired video, the electronic device performs decoding processing on a part of video frames in the video to obtain a motion vector corresponding to each video frame. For example, the video includes 100 video frames, and the electronic device may perform a coding process on the first 50 video frames of the 100 video frames.
And (a 2) determining the number of times of reference of each video frame based on the motion vector corresponding to each video frame.
The reference number of times is the number of times that a video frame is referred to by another video frame when the video frame is reconstructed. For example, video frame a is referenced by video frame B to be reconstructed, video frame C … … to be reconstructed, and video frame Z to be reconstructed, for a total of 26 times.
Specifically, the electronic device determines a reference video frame corresponding to each video frame based on a motion vector corresponding to each video frame in the video; and counting the number of times of reference of the reference video frame. Taking a video frame X, a video frame Y, and a video frame Z as examples, the video frame X is referred to by the video frame Y and the video frame Z, then the video frame X is taken as a reference video frame of the video frame Y and the video frame Z, and then the number of times the video frame X is referred to is 2 times.
And (a 3) storing the video frame with the reference times meeting the first time condition into a main storage outside the video decoder.
Wherein, the first time condition can be adjusted according to the requirement. The number of times characterized by the first order condition is less than the number of times characterized by the second order condition. For example, the first order condition is satisfied as being equal to 1.
Specifically, the electronic device stores the video frames whose number of times of reference satisfies the first number of times condition in main storage outside the video decoder. The subsequent video coder may read the video frame satisfying the first condition from main storage and store in an internal buffer of the video coder.
Step (a 4), storing the video frame with the reference times meeting the second time condition into an external storage of the video decoder; and the times represented by the second time condition are greater than the times represented by the first time condition.
Wherein, the second time condition can be adjusted according to the requirement. The second order condition characterizes a number of times greater than the first order condition. For example, the condition that the number of times to be referred satisfies the first number of times is that the number of times to be referred satisfies 1, and the condition that the number of times to be referred satisfies the second number of times is that the number of times to be referred satisfies more than 1. Alternatively, the condition that the number of times to be referred to satisfies the first number of times is that the number of times to be referred to is 2 or less, the condition that the number of times to be referred to satisfies the second number of times is that the number of times to be referred to is greater than 2, and the like are not limited thereto.
Specifically, the electronic device stores the video frames, the number of times of which is referred to satisfies the second time condition, in an external storage of the video decoder. External storage of a video decoder refers to storage that is located in the electronic device but not located in the video decoder. The external storage of the video decoder may be an on-chip memory (on-chip memory), for example, the external storage of the video decoder may be at least one of a system cache and a system cache region.
In the video data storage method in this embodiment, in response to decoding processing of an acquired video, a motion vector corresponding to each video frame in the video is obtained, and the number of times of reference of each video frame is determined based on the motion vector of each video frame, so that it can be known that the video frame is to be used several times; storing the video frames with the reference times meeting the first time condition into an internal cache of a video decoder, storing the video frames with the reference times meeting the second time condition into an external memory of the video decoder, namely storing the video frames with less reference times into the internal cache of the video decoder, and storing the video frames with more reference times into the external memory of the video decoder; the use of an internal buffer of the video decoder is reduced, and external storage with low power consumption is utilized, so that the power consumption of the electronic equipment is reduced. In addition, because the internal buffer of the video decoder is limited, the video frames which are used for a few times are buffered in the video decoder, the video frames which are buffered externally can be updated more quickly, and the external buffer can be fully utilized to accelerate the decoding processing.
In one embodiment, the external storage of the video decoder includes at least one of a system cache or a system cache region. Both the system cache and the system buffer belong to an on-chip memory. And both the system cache and the system cache region belong to Static Random-Access Memory (SRAM). In this embodiment, by storing the video frame satisfying the second frequency condition in at least one of the system cache and the system cache region, the speed of reading data from the system cache and the system buffer region is fast and the power consumption is low, so that the video data from the system cache and the system buffer region does not need to be stored once in the decoder, the decoding speed of the decoder is maintained, and the power consumption of the electronic device is reduced.
In one embodiment, determining the number of times each video frame is referred to based on the motion vector corresponding to each video frame includes: determining a reference video frame referred to by the current video frame based on the motion vector of the current video frame; the number of times of being referred to of the reference video frame is counted.
Specifically, the electronic device decodes the acquired video, and obtains a motion vector of the video frame by decoding. The motion vectors of the video frames are used to represent the displacement between the current video frame and the reference video frame. The electronic equipment determines a reference video frame which is referred by the current video frame based on the motion vector of the current frame, and counts the referred times of the reference video frame, so that the referred times of each video frame can be obtained. For example, the description will be given by taking the current video frame as a video frame C, and reference video frames referred to by the video frame C as a video frame a and a video frame B. Then the number of times video frame a is referenced is 1 and the number of times video frame B is referenced is also 1. Then, when the current video frame is the video frame D, and the reference video frames referred to by the video frame D are also the video frame a and the video frame B, the number of times that the video frame a is referred to is 2, and the number of times that the video frame B is referred to is also 2, so that the number of times that each video frame in the video is referred to can be obtained.
In the video data storage method in this embodiment, a reference video frame referred to by a current video frame is determined based on a motion vector of the current video frame, the number of times that the reference video frame is referred to is counted, that is, which of the current video frame and what of the motion vector is known, that is, what of the reference video frame referred to by the current video frame is obtained, so that the number of times that each video frame in a video is referred to is obtained, and video data is stored based on the number of times that the reference video frame is referred to is obtained.
In one embodiment, the video data storage method further comprises: and when the number of the video frames between the current video frame and the target reference video frame reaches a preset number, stopping counting the number of times of being referred of the target reference video frame, and obtaining the number of times of being referred of the target reference video frame. The preset number can be set according to actual requirements. The preset number is used to characterize the number of maximum interval frames between the video frame and the reference video frame. Because the interval between the current video frame and the referenced video frame is not too many frames, when the number of the video frames between the current video frame and the target reference video frame reaches the preset number, the video frame behind the current video frame does not refer to the target reference video frame any more, and therefore the referenced times of the target reference video frame can be directly obtained.
In one embodiment, each frame of video is divided into a plurality of blocks; determining a reference video frame to which the current video frame refers based on the motion vector of the current video frame, comprising: a reference video block in a reference video frame to which the current video frame refers is determined based on a motion vector of the current block in the current video frame.
Counting the number of times a reference video frame is referred to, including: and counting the number of times of reference of the reference video block in the reference video frame.
Storing the video frame whose reference number satisfies the first number condition in a main storage outside the video decoder, including: the reference video blocks with the reference times satisfying the first time condition are stored in a main storage outside the video decoder.
Storing the video frame with the reference times meeting the second time condition into an external storage of the video decoder, wherein the method comprises the following steps: and storing the reference video block with the reference times meeting the second time condition into an external storage of the video decoder.
The block refers to a partial area of a video frame. For example, the size of the block may be 16 × 16 pixels, 64 × 64 pixels, and the like, but is not limited thereto.
Specifically, in response to the decoding processing of the acquired video, the electronic device divides each video frame in the video into a plurality of blocks and obtains a motion vector corresponding to each block. The electronic device determines a reference video block in a reference video frame to which the current video frame refers based on a motion vector of the current block in the current video frame. The electronic device counts the number of times the reference video block is referred to. The electronic device stores the reference video blocks with the reference times meeting the first time condition into an internal buffer of the video decoder. The electronic device stores the reference video block with the reference times meeting the second time condition into an external storage of the video decoder.
In this embodiment, the reference video block to be referred to is determined based on the motion vector of the current block, the number of times of reference of the reference video block is counted, the reference video block whose number of times of reference satisfies the first number of times condition is stored in the main memory, and the reference video block whose number of times of reference satisfies the second number of times condition is stored in the external memory of the video decoder.
In this embodiment, as shown in fig. 7, an architecture diagram of a system cache used in one embodiment is shown. The video display system comprises a central processing unit, a video decoder, a display processor, a dynamic random access memory controller and a system cache. A system cache (system cache) may retrieve data from the DRAMC. The system cache size in fig. 7 is 4MB (megabytes). And the system cache belongs to one of SRAM (Static Random-Access Memory). Then, in response to the decoding processing of the acquired video, motion vectors corresponding to the video frames in the video are acquired through a video decoder, and the number of times of reference of each video frame is determined based on the motion vectors corresponding to the video frames; reading the video frames with the reference times meeting the first time condition from the DRAM and storing the video frames in an internal cache of the video decoder; and reading the partial video frames with the reference times meeting the second time condition from the DRAM and storing the partial video frames in a system cache. It is understood that the system cache may not be directly connected to the DRAMC, as just one example in fig. 6.
In this embodiment, as shown in fig. 8, an architecture diagram of a system cache area used in one embodiment is shown. The video display system comprises a central processing unit, a video decoder, a display processor, a dynamic random access memory controller and a system buffer. The system cache size in fig. 7 is 4MB (megabytes). And the system buffer belongs to one of the Static Random-Access memories (SRAMs). Then, in response to the decoding processing of the acquired video, motion vectors corresponding to the video frames in the video are acquired through a video decoder, and the number of times of reference of each video frame is determined based on the motion vectors corresponding to the video frames; reading the video frames with the reference times meeting the first time condition from the DRAM and storing the video frames in an internal buffer of the video decoder; and directly storing the video frames with the reference times meeting the second time condition in a system cache.
In one embodiment, the video data storage method further comprises: in the case where the internal buffer in the video decoder is full and new video frames are stored, the least important video frames in the internal buffer are deleted.
Specifically, under the condition that the internal cache of the video decoder is full and new video frames need to be stored, the electronic device deletes the least important video frames in the internal cache, so that the video decoder can be ensured to store data all the time, and the operation of the video decoder is ensured. The least important may be the oldest video frame or may be judged in other ways.
In one embodiment, as shown in FIG. 9, a decoding process is schematically illustrated in one embodiment. The figure includes I frames, B frames, and P frames. The I frame belongs to intraframe compression, and only the information of the I frame is needed to be utilized during decoding. P frames are forward predictive coded frames, i.e., P frames need to be decoded by referring to information of previous related frames when being decoded. The B frame is a bidirectional predictive coding frame, and when decoding, the B frame needs to refer to both the previous existing frame and the frame to be decoded later. It will be appreciated that some video standards allow both reference objects of a B frame to be previous frames. The Display Order (Display Order) is from left to right as shown in fig. 9. And the Decoding Order (Decoding Order) is the Order 0, 3, 2, 4, 1, 7, 6, 8, 5 in fig. 9. The decoding order in fig. 9 is different from the display order. The arrows indicate references, such as display order 2B frames, which are reconstructed with reference to display order 0I frames and display order 1P frames, and display order 2B frames may in turn serve as reference video frames for display order 1 and display order 3 frames. Thereby, video frames of the video may be reconstructed. The blank part in the figure represents storage in DRAM, i.e. subsequently in the internal buffer of the video decoder. The grey parts represent storage in external storage of the video decoder. Then the display order 0I frame, display order 2B frame, display order 3B frame, display order 4P frame, display order 6B frame, and display order 8B frame all have a portion of the tiles stored in external storage at the video decoder and another portion stored in DRAM or an internal buffer at the video decoder. And the B frame of display order 1, the B frame of display order 5, and the B frame of display order 7 are all stored in the DRAM. In this embodiment, after some analysis of the video, it is known which frames or blocks are not suitable for being stored in the DRAM, or are not suitable for being stored in the internal buffer of the video decoder, and are suitable for being stored in the external storage located outside the video decoder, such as at least one of the system cache and the system buffer. Under the condition of such storage distribution, the decoder can properly utilize the internal cache of the decoder, so that the data in the main storage only outside the video decoder can survive longer in the internal cache of the decoder, the system bandwidth is further reduced, and the purpose of extremely saving power is achieved.
The basic concept of the embodiments of the present application is to make good use of each available on-chip memory (on-chip memory, i.e. external storage and internal storage of the video decoder) to make the DRAM access low, and the video decoder internal cache policy is changed when there is assistance from other on-chip memories. The method comprises the steps of storing data which are repeatedly used for a plurality of times in a low-power-consumption storage space such as a system cache or a system buffer area in a whole reference video frame or a partial block of a reference video frame in a video, and transmitting the data which cannot be put into the cache or the system buffer area to an internal cache of a video decoder for processing.
A video data storage method, comprising:
and (b 1) responding to the decoding processing of the acquired video, and acquiring a motion vector corresponding to each video frame in the video.
And (b 2) determining a reference video block in a reference video frame referred by the current video frame based on the motion vector of the current block in the current video frame.
And (b 3) counting the number of times of reference of the reference video block in the reference video frame.
And (b 4) storing the reference video block with the reference times satisfying the first time condition into a main storage outside the video decoder.
And (b 5) storing the reference video block with the reference times meeting the second time condition into an external storage of the video decoder. And the times represented by the second time condition are greater than the times represented by the first time condition.
And (b 6) in response to the decoding processing of the video frame to be reconstructed, reading the reference video block from the main storage if the reference video block corresponding to the video frame to be reconstructed does not exist in the external storage of the video decoder and if the reference video block exists in the main storage outside the video decoder. The external storage of the video decoder includes at least one of a system cache or a system cache region.
And (b 7) storing the read reference video block into an internal buffer of the video decoder.
And (b 8) reading the reference video frame from the external storage of the video decoder in the case that the reference video frame exists in the external storage of the video decoder.
And (b 9) deleting the video frames which are not needed in the internal buffer under the condition that the internal buffer in the video decoder is full and new video frames are stored. (the least desirable may be the oldest or selected by other methods of assessment).
In the video data storage method in this embodiment, in response to decoding processing of an acquired video, a motion vector corresponding to each video frame in the video is obtained, and the number of times of reference of each video frame is determined based on the motion vector of each video frame, so that it can be known that the video frame is to be used several times; storing the video frames with the reference times meeting the first time condition into an internal cache of a video decoder, storing the video frames with the reference times meeting the second time condition into an external memory of the decoder, namely storing the video frames with less reference times into the internal cache of the video decoder, and storing the video frames with more reference times into the external memory of the decoder; compared with the conventional method that the video data is placed in the decoder internal buffer by the decoder no matter the video data is located in the external storage or the main storage, in the embodiment, only the video data which is not located in the external storage and is located in the main storage is placed in the decoder internal buffer, so that the use of the internal buffer of the video decoder is reduced, the efficiency of the decoder internal buffer is increased, and the power consumption is further reduced specially for the data which is only located in the main storage. In addition, because the internal buffer of the video decoder is limited and the processing speed of decoding processing through the internal buffer is high, video frames which are used for a few times are buffered in the video decoder, the updating speed of the buffered video frames can be higher, and the buffer of the video decoder can be fully utilized to accelerate the decoding processing.
FIG. 10 is a flowchart of a video data reading method according to one embodiment. The video data storage method in this embodiment is described by taking an example of the video data storage method running on an electronic device. As shown in fig. 10, the video data storage method includes steps 1002 to 1004.
In step 1002, in response to the decoding process of the acquired video bitstream, the reference video blocks that are not stored outside the video decoder are read from the main storage outside the video decoder to the internal buffer of the video decoder for performing compensation or block copy.
Wherein primary storage refers to primary storage of the electronic device. The main memory may be, for example, DRAM. Motion Compensation, also called Motion Compensation, is a method to describe the difference between nearby frames (nearby here means nearby in the coding relation, two frames are not necessarily nearby in the playing order), and specifically how each small block of the previous frame moves to a certain position in the video frame to be reconstructed.
Specifically, in response to a decoding process of an acquired video code stream, the electronic device acquires a target motion vector of a video frame to be reconstructed; the electronic equipment reads a reference video block which is not stored outside the video decoder from a main storage outside the video decoder to an internal buffer of the video decoder according to the target motion vector and then performs compensation; the reference video block is a block of a video frame to which the video frame to be reconstructed refers. Or the electronic equipment acquires a block vector of a video frame to be reconstructed; the reference video blocks which are not stored outside the video decoder are read from the main storage according to the block vectors to an internal buffer of the video decoder for block copying. The video frame to be reconstructed refers to a video frame which is not successfully constructed yet. The target motion vector of the video frame to be reconstructed represents the displacement between the current video frame and the reference video block. The video data stored in the internal buffer of the video decoder may or may not be the same as the video data stored in the external memory of the video decoder. For example, the video includes 100 video frames, which are video frame 1, video frame 2, and video frame 3 …, video frame 100. The internal buffer of the video decoder comprises a video frame 1 … video frame 50, and the external storage of the video decoder comprises a video frame 51 and a video frame 52 … video frame 100.
Step 1004, motion compensation or block copy is performed after the reference video block is read from the external storage of the video decoder.
Specifically, the electronic device performs motion compensation after reading the reference video block from the external storage of the video decoder. The electronic device reads the reference video block from the external storage of the video decoder and reconstructs a target video frame based on the read reference video block. For example, the electronic device performs weighted fusion reconstruction on the read reference video blocks to obtain a target video frame. Alternatively, the electronic device reads the reference video block from external storage of the video decoder and performs block copy.
In the video data reading method in this embodiment, in response to the decoding processing of the acquired video stream, the reference video block that is not stored in the external of the video decoder is read from the main storage and then is subjected to motion compensation or block copy after being buffered in the internal of the video decoder, or the reference video block is read from the external of the video decoder and then is subjected to motion compensation or block copy, that is, part of the video data can be read from the outside of the video decoder, so that the external storage with low power consumption is used, and the use of the internal buffer of the video decoder is also reduced, thereby reducing the power consumption of the electronic device.
In one embodiment, in response to the decoding processing of the acquired video bitstream, the reference video blocks that do not exist in external storage of the video decoder are read from main storage external to the video decoder to an internal buffer of the video decoder for compensation; or after reading the reference video block from the external storage of the video decoder, performing motion compensation or block copy, including:
in response to the decoding processing of the acquired video code stream, searching from an external storage of the video decoder, and under the condition that a reference video block is searched in the external storage, performing action compensation or block copying after reading the reference video block from the external storage;
in the case where the reference video block is not found in the external storage, the reference video block is found from the external main storage of the video decoder, and the compensation or block copy is performed after the reference video block is read from the main storage to the internal buffer of the video decoder.
Specifically, the electronic device searches a corresponding reference video block from an external memory of the video decoder according to a target motion vector of a video frame to be reconstructed. And under the condition that the reference video block corresponding to the video frame to be reconstructed is found in the external storage, the reference video block corresponding to the video frame to be reconstructed is read from the external storage and then motion compensation or block copying is carried out. In the case where the reference video block corresponding to the video frame to be reconstructed is not found in the external storage, the electronic device finds the reference video block corresponding to the video frame to be reconstructed from the main storage, and performs compensation or block copy after reading the reference video block from the main storage to an internal buffer of the video decoder. The reference video block is a part of the video area in the reference video frame. The number of reference video blocks corresponding to a video frame to be reconstructed is not limited. For example, 1, 2, 3 … is not limited thereto.
Taking the external storage as a system cache, the main storage as a DRAM, and the internal cache of the video decoder as a VDEC cache as examples, the electronic device searches a corresponding reference video block from the system cache according to a target motion vector of a video frame to be reconstructed. And under the condition that the system cache finds the corresponding reference video block, reading the reference video block corresponding to the video frame to be reconstructed from the system cache. And under the condition that the reference video block corresponding to the video frame to be reconstructed is not found in the system cache, finding the reference video block corresponding to the video frame to be reconstructed from the DRAM, reading the reference video block from the main storage to the VDEC cache, and then performing compensation.
In this embodiment, when all the reference video blocks corresponding to the video frames to be reconstructed are not found in the external storage, the reference video blocks corresponding to the remaining video frames to be reconstructed are found from the main storage, and the reference video blocks are read from the main storage to the internal buffer of the video decoder for performing compensation or block copy.
In this embodiment, when the reference video block is found in the external storage of the video decoder, the reference video block is read from the external storage and then motion compensation or block copy is performed; when the reference video block is not found in the external storage, the reference video block is found again from the external main storage of the video decoder, the reference video block is found from the main storage, and the reference video block is read from the main storage and acts as compensation or block copy after being cached in the video decoder; the video decoder internal buffer only reads the data from the main storage, so that unnecessary buffer operation in the video decoder can be saved compared with a traditional mode that the video decoder internal buffer is consistent with the data stored in the external storage, and the video decoder internal buffer has smaller amount of video data, so that the main storage bandwidth can be further reduced, and the power consumption of the electronic equipment can be reduced.
In one embodiment, the video data reading method further includes: responding to the decoding processing of the acquired video, and acquiring a motion vector corresponding to each video frame in the video; determining the number of times of reference of each video frame based on the motion vector corresponding to each video frame; reading the video frame with the reference times meeting the first time condition from a main storage outside the video decoder and then storing the video frame into an internal cache of the video decoder; storing the video frame with the reference times meeting the second time condition into an external storage of the video decoder; the second order condition characterizes a number of times greater than the first order condition.
In this embodiment, in response to the decoding processing of the acquired video, a motion vector corresponding to each video frame in the video is obtained, and the number of times of reference of each video frame is determined based on the motion vector of each video frame, so that it is known that the video frame is to be used several times; storing the video frames with the reference times meeting the first time condition into an internal cache of a video decoder from a main memory, storing the video frames with the reference times meeting the second time condition into an external memory of the decoder, namely storing the video frames with the low reference times into the internal cache of the video decoder, and storing the video frames with the high reference times into the external memory of the decoder; the use of an internal buffer of the video decoder is reduced, and external storage with low power consumption is utilized, so that the power consumption of the electronic equipment is reduced. In addition, because the internal buffer of the video decoder is limited, the video frames which are used for a few times are buffered in the video decoder, the video frames which are buffered externally can be updated more quickly, and the external buffer can be fully utilized to accelerate the decoding processing.
In one embodiment, the external storage of the video decoder includes at least one of a system cache, a system buffer. Both the system cache and the system buffer belong to an on-chip memory. And both the system cache and the system cache region belong to Static Random-Access Memory (SRAM). In this embodiment, by storing the video frames satisfying the second order condition in at least one of the system cache and the system buffer, the speed of reading data from the system cache and the system buffer is fast and the power consumption is low, so that the decoding speed of the video decoder can be maintained, and the data stored in these caches cannot be placed in the internal cache of the decoder.
In one embodiment, as shown in fig. 11, a flow chart of reconstructing a video frame in one embodiment is shown. The video frames include an a video frame 1102, a B video frame 1104, a C video frame 1106, and a video frame 1108 to be reconstructed, where the a video frame 1102, the a video frame 1104, and the a video frame 1106 are all already reconstructed video frames, and the video frame 1108 to be reconstructed is not already reconstructed. The white parts in the a video frame 1102, the a video frame 1104 and the a video frame 1106 are stored in a DRAM, that is, an internal buffer of a video decoder; blocks of gray shaded portions are stored in external storage of the video coder. And a video frame 1102, B video frame 1104, and C video frame 1106 are all reference video frames of video frame to be reconstructed 1108. As can be seen, different blocks of a video frame may be stored in different stores. Fig. 11 illustrates an example of 16 × 16 blocks and a video frame 1108 to be reconstructed as a B frame.
A small square in the video frame to be reconstructed represents a current block. The reference blocks in fig. 11 are not reused. The electronic device obtains sub-target motion vectors for the 1 st, 2 nd, 3 rd, 4 th, 5 th, 6 th, and 7 th of the video frame to be reconstructed, and may determine a reference block based on the sub-target motion vectors. For example, the 1 st block of the video frame 1108 to be reconstructed may be reconstructed from the 0 block in the a video frame 1102 and the 1 block in the a video frame 1104. The 2 nd block of video frame 1108 to be reconstructed may be reconstructed from 2 blocks in a video frame 1102 and 3 blocks in a video frame 1106. The 3 rd block of video frame 1108 to be reconstructed may be reconstructed from 4 blocks in a video frame 1104 and 5 blocks in a video frame 1104. The 4 th block of video frame 1108 to be reconstructed may be reconstructed from 6 blocks in a video frame 1104 and 7 blocks in a video frame 1106. The 5 th block of video frame 1108 to be reconstructed may be reconstructed from 8 blocks in a video frame 1102 and 9 blocks in a video frame 1104. The 6 th block of video frame 1108 to be reconstructed may be reconstructed from the a block in a video frame 1104 and the B block in a video frame 1106. The 7 th block of video frame 1108 to be reconstructed may be reconstructed from the C block of a video frame 1102 and the D block of a video frame 1104.
In one embodiment, as shown in fig. 12, a schematic diagram of a storage manner stored inside a conventional video decoder in one embodiment is shown. Taking the block in fig. 11 as an example, the storage manner stored inside the video decoder in fig. 12 stores the blocks in order. For example, first-step block 0, second- step blocks 1, 2, 3, 4, 5, 6, and 7, and when a new block is stored, third- step blocks 6, 7, 8, 9, A, B, C and D are stored. That is, the internal buffer of the video decoder in fig. 12 buffers any block read from any area, whether read from DRAM (white portion) or read from external storage of the video decoder (gray portion). In this way, the video data which does not need the internal buffer of the video decoder at all to save the DRAM throughput is also stored in the internal buffer of the video decoder, and the reading and writing energy consumption of the internal buffer of the video decoder is wasted.
In one embodiment, as shown in fig. 13, a schematic diagram of a storage manner stored inside a video decoder in one embodiment is shown. The white portion in the internal buffer of the video decoder refers to the block originally stored in the DRAM, i.e., stored in the internal buffer of the video decoder. In contrast to the approach of fig. 12, the video decoder of fig. 13 stores only the blocks from the DRAM in the internal buffer. Therefore, the blocks stored in the external memory of the video decoder and the blocks stored in the internal cache of the video decoder are not stored repeatedly, the blocks stored in the DRAM are distinguished, the blocks stored in the DRAM are read into the internal cache of the video decoder, and the other part of blocks are stored in the external memory of the video decoder, so that the reading of the DRAM is reduced on the whole in the decoding process, the bandwidth of the DRAM is reduced, and the power consumption of electronic equipment is reduced. The data in the system buffer/cache is electricity-saving, the access speed is fast and controllable, if the data is cached again by using the VDEC internal cache, the data does not help the decoding speed and consumes more energy for accessing the SRAM once. In addition, if the system cache/buffer which saves electricity already stores partial frame buffer data, the residual data which is not stored in the system cache/buffer part enters the VDEC internal cache, and the significance of saving DRAM bandwidth is substantially achieved.
In one embodiment, table 1 is a schematic diagram of the difference between the power consumption of SRAM and DRAM in one embodiment. The energy consumption of SRAM (Static Random-Access Memory) is about 5PJ (Pi Jiaoer), while the energy consumption of DRAM is about 640PJ (Pi Jiaoer), which is 2 orders of magnitude different from each other. The relative power consumption of the SRAM is 10 in terms of relative power consumption 2 And the relative power consumption of DRAM is 10 4 The difference between the two is 2 orders of magnitude. Although it is described in Table 1 that accessing DRAM consumes more than 100 times of power as compared to accessing SRAM even if there is a system buffer/cache (system cache/cache) in the electronic deviceMemory), the amount is limited, and it is impossible to expand the on-chip SRAM beyond the VDEC without limitation. Therefore, the internal cache of the original video decoder is better used, the required DRAM bandwidth can be reduced under the condition of using the system cache/buffer originally, and more extreme power saving and low power consumption are achieved.
TABLE 1
Storing names Energy consumption (energy pJ) Relative Energy consumption (Relative Energy Cost)
Static random access memory 5 10 2
Dynamic random access memory 640 10 4
It should be understood that, although the steps in the flowcharts of fig. 5 and 10 are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least some of the steps in fig. 5 and 10 may include multiple sub-steps or multiple stages that are not necessarily performed at the same time, but may be performed at different times, and the order of performing the sub-steps or stages is not necessarily sequential, but may be performed alternately or alternately with other steps or at least some of the sub-steps or stages of other steps.
Fig. 14 is a block diagram of a video data storage device according to an embodiment. As shown in fig. 14, a video data storage apparatus includes a first reading module 1402 and a reference video frame storing module 1404, wherein:
a first reading module 1402, configured to, in response to a decoding process of a video frame to be reconstructed, read a reference video frame from a main storage if a reference video frame of the video frame to be reconstructed does not exist in the external storage of the video decoder and if the reference video frame exists in the main storage outside the video decoder; and
a reference video frame storage module 1404, configured to store the reference video frame read from the main storage into an internal buffer of the video decoder.
In the video data storage apparatus in this embodiment, in response to the decoding processing of the video frame to be reconstructed, the reference video frame of the video frame to be reconstructed does not exist in the external storage of the video decoder, and when the reference video frame exists in the main storage outside the video decoder, the reference video frame is read from the main storage, and the reference video frame read from the main storage is stored in the internal buffer of the video decoder, that is, the video data existing in the external storage of the video decoder does not store in the internal buffer of the video decoder, and the video data in the main storage stores in the internal buffer of the video decoder.
In one embodiment, each frame of video is divided into a plurality of video blocks. The first reading module 1402 is configured to read a reference video block from main storage outside the video decoder if the reference video block corresponding to the video frame to be reconstructed does not exist in the external storage of the video decoder, and if the reference video block exists in the main storage outside the video decoder. The reference video frame storage module 1404 is used to store the reference video blocks read from main storage into the internal buffer of the video decoder.
In this embodiment, when the reference video block corresponding to the video frame to be reconstructed does not exist in the external storage of the video decoder and the reference video block exists in the main storage, the electronic device reads the reference video block from the main storage and stores the reference video block into the internal buffer of the video decoder.
In one embodiment, the first reading module 1402 is further configured to read the reference video frame from the external storage of the video decoder in the event that the reference video frame exists in the external storage of the video decoder.
In this embodiment, when the reference video frame exists in the external storage of the video decoder, the reference video frame is read from the external storage of the video decoder, and data access is performed through the external storage of the video decoder, so that the efficiency of the internal cache of the decoder can be increased, and further power consumption reduction is performed specifically for data that is still only left in main storage.
In one embodiment, the video data storage apparatus further comprises a motion vector acquisition module, a number determination module, a first storage module, and a second storage module, wherein:
the motion vector acquisition module is used for responding to the decoding processing of the acquired video and acquiring a motion vector corresponding to each video frame in the video;
the number determining module is used for determining the number of times of reference of each video frame based on the motion vector corresponding to each video frame;
a first storage module, configured to store the video frames with the reference times satisfying a first time condition into a main storage of a video decoder;
the second storage module is used for storing the video frames with the reference times meeting the second time condition into the external storage of the video decoder; and the times represented by the second time condition are greater than the times represented by the first time condition.
In this embodiment, in response to the decoding processing of the acquired video, a motion vector corresponding to each video frame in the video is obtained, and the number of times of reference of each video frame is determined based on the motion vector of each video frame, so that it is known that the video frame is to be used several times; storing the video frames with the reference times meeting the first time condition into an internal cache of a video decoder, storing the video frames with the reference times meeting the second time condition into an external memory of the decoder, namely storing the video frames with less reference times into the internal cache of the video decoder, and storing the video frames with more reference times into the external memory of the decoder; the use of an internal buffer of the video decoder is reduced, and external storage with low power consumption is utilized, so that the power consumption of the electronic equipment is reduced. In addition, because the internal buffer of the video decoder is limited, the video frames which are used for a few times are buffered in the video decoder, the video frames which are buffered externally can be updated more quickly, and the external buffer can be fully utilized to accelerate the decoding processing.
In one embodiment, the number determining module is configured to determine a reference video frame to which the current video frame refers based on a motion vector of the current video frame; and counting the number of times of reference of the reference video frame.
The video data storage device in this embodiment determines a reference video frame referred to by a current video frame based on a motion vector of the current video frame, counts the number of times that the reference video frame is referred to, i.e., what the current video frame is and what the motion vector is, i.e., what the reference video frame referred to by the current video frame is, so as to obtain the number of times that each video frame in a video is referred to, and stores video data based on the number of times that the reference video frame is referred to, thereby reducing power consumption of electronic equipment.
In one embodiment, the number determining module is further configured to stop counting the number of times that the target reference video frame is referred to when the number of video frames between the current video frame and the target reference video frame reaches a preset number, and obtain the number of times that the target reference video frame is referred to. Because the interval between the current video frame and the referenced video frame is not too many frames, when the number of the video frames between the current video frame and the target reference video frame reaches the preset number, the video frame behind the current video frame does not refer to the target reference video frame any more, and therefore the referenced times of the target reference video frame can be directly obtained.
In one embodiment, each frame of video is divided into a plurality of blocks. The number determining module is used for determining a reference video block in a reference video frame referred by the current video frame based on the motion vector of the current block in the current video frame; counting the number of times of reference of a reference video block in a reference video frame; storing the reference video blocks with the reference times meeting the first time condition into an internal cache of the video decoder; and storing the reference video block with the reference times meeting the second time condition into an external storage of the video decoder.
In this embodiment, the reference video block to be referred to is determined based on the motion vector of the current block, the number of times of being referred to of the reference video block is counted, the reference video block whose number of times of being referred to satisfies the first number of times condition is stored in the internal storage of the video decoder, and the reference video block whose number of times of being referred to satisfies the second number of times condition is stored in the external storage of the video decoder.
In one embodiment, the external storage of the video decoder includes at least one of a system cache or a system cache region. In this embodiment, by storing the video frame satisfying the second frequency condition in at least one of the system cache and the system buffer, the speed of reading data from the system cache and the system buffer is fast and the power consumption is low, so that the decoding speed of the video decoder can be maintained, and the power consumption of the electronic device can be reduced.
In one embodiment, the main storage external to the video decoder comprises dynamic random access memory.
In one embodiment, the first storage module is further configured to delete the least desirable video frame in the internal buffer if the internal buffer in the video decoder is full and a new video frame is stored. In this embodiment, when the internal buffer of the video decoder is full and a new video frame needs to be stored, the electronic device deletes the video frame that is least needed in the internal buffer, so that the video decoder can always store data, and the operation of the video decoder is guaranteed. For example, the oldest can be used to determine which video frames are least needed. The least desirable video frame may be, for example, the oldest. In other embodiments, other methods of scoring may be used to select the least desirable video frame.
Fig. 15 is a block diagram of a video data reading apparatus according to an embodiment. As shown in fig. 15, a video data reading apparatus includes a second reading module 1502 and a third reading module 1504, wherein:
a second reading module 1502, configured to, in response to a decoding process on the acquired video stream, read a reference video block that is not stored outside the video decoder from a main storage outside the video decoder to an internal buffer of the video decoder, and then perform a compensation or block copy operation;
alternatively, the first and second liquid crystal display panels may be,
a third reading module 1504, configured to perform motion compensation or block copying after reading the reference video block from the external storage of the video decoder.
In the video data reading apparatus in this embodiment, in response to the decoding processing of the acquired video stream, the reference video block that is not stored in the external of the video decoder is read from the main storage and then motion compensation or block copy is performed on the reference video block to the internal buffer of the video decoder, or the reference video block is read from the external storage of the video decoder and then motion compensation or block copy is performed, that is, a part of video data is stored in the external of the video decoder, and the external storage with low power consumption is used, so that the use of the internal buffer of the video decoder is reduced, and the bandwidth of the main storage is also reduced, thereby reducing the power consumption of the electronic device.
In an embodiment, the second reading module 1502 is configured to, in response to a decoding process on the obtained video stream, search from an external storage of the video decoder, and in a case that a reference video block is found in the external storage, perform motion compensation or block copy after reading the reference video block from the external storage;
the third reading module 1504 is used to find the reference video chunk from the external main storage of the video decoder, and then read the reference video chunk from the main storage to the internal buffer of the video decoder for compensation or block copy.
In this embodiment, when the reference video block is found from the external storage of the video decoder, the reference video block is read from the external storage; under the condition that the reference video block is not found in the external storage, the reference video block is found again from the main storage, the reference video block is found from the main storage, and the reference video block is read from the main storage and acts as compensation after being cached in the video decoder; the video decoder internal buffer reads only the data from the main storage, and compared with the traditional mode that the video decoder internal buffer is consistent with the data stored in the external storage, the unnecessary buffer operation in the video decoder can be saved, and the power consumption of the electronic equipment is reduced.
In one embodiment, the video data reading apparatus further includes a storage module, configured to obtain, in response to a decoding process on the obtained video, a motion vector corresponding to each video frame in the video; determining the number of times of reference of each video frame based on the motion vector corresponding to each video frame; storing the video frame with the reference times meeting the first time condition into a main storage outside the video decoder; storing the video frame with the reference times meeting the second time condition into an external storage of the video decoder; and the times represented by the second time condition are greater than the times represented by the first time condition.
In this embodiment, in response to the decoding processing of the acquired video, a motion vector corresponding to each video frame in the video is obtained, and the number of times of reference of each video frame is determined based on the motion vector of each video frame, so that it is known that the video frame is to be used several times; storing the video frames with the reference times meeting the first time condition into a main memory, and storing the video frames with the reference times meeting the second time condition into an external memory of a decoder, namely storing the video frames with less reference times into the main memory, and storing the video frames with more reference times into the external memory of the decoder; thus, the use of an internal buffer of the video decoder is reduced, and external storage with low power consumption is utilized, thereby reducing the power consumption of the electronic device. In addition, because the internal buffer of the video decoder is limited, the video frames which are used for a few times are buffered in the video decoder, the video frames which are buffered externally can be updated more quickly, and the external buffer can be fully utilized to accelerate the decoding processing. In one embodiment, the external storage of the video decoder includes at least one of a system cache, a system buffer. In this embodiment, by storing the video frame satisfying the second frequency condition in at least one of the system cache and the system buffer, the speed of reading data from the system cache and the system buffer is fast and the power consumption is low, so that the decoder does not need to store video data from the system cache and the system buffer again, thereby reducing the power consumption of the electronic device.
The division of the modules in the video data storage device and the video data reading device is merely for illustration, and in other embodiments, the video data storage device and the video data reading device may be divided into different modules as needed to complete all or part of the functions of the video data storage device and the video data reading device.
For specific limitations of the video data storage device and the video data reading device, reference may be made to the above limitations of the method, which are not described herein again. The respective modules in the video data storage apparatus and the video data reading apparatus described above may be wholly or partially implemented by software, hardware, and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
Fig. 16 is a schematic internal structure diagram of an electronic device in one embodiment. As shown in fig. 16, the electronic apparatus includes a processor and a memory connected by a system bus. Wherein, the processor is used for providing calculation and control capability and supporting the operation of the whole electronic equipment. The memory may include a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The computer program is executable by a processor to implement a video data storage method and a video data reading method provided in the following embodiments. The internal memory provides a cached execution environment for the operating system computer programs in the non-volatile storage medium. The electronic device may be any terminal device such as a mobile phone, a tablet computer, a PDA (Personal Digital Assistant), a POS (Point of Sales), a vehicle-mounted computer, and a wearable device.
The implementation of each module in the video data storage device and the video data reading device provided in the embodiments of the present application may be in the form of a computer program. The computer program may be run on a terminal or a server. Program modules constituted by such computer programs may be stored on the memory of the electronic device. Which when executed by a processor, performs the steps of the method described in the embodiments of the present application.
The embodiment of the application also provides a computer readable storage medium. One or more non-transitory computer-readable storage media containing computer-executable instructions that, when executed by one or more processors, cause the processors to perform the steps of the video data storage method.
The embodiment of the application also provides a computer readable storage medium. One or more non-transitory computer-readable storage media containing computer-executable instructions that, when executed by one or more processors, cause the processors to perform the steps of the video data reading method.
A computer program product containing instructions which, when run on a computer, cause the computer to perform a video data storage method.
A computer program product comprising instructions which, when run on a computer, cause the computer to perform a video data reading method.
Any reference to memory, storage, database, or other medium used herein may include non-volatile and/or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM), which acts as external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous Link DRAM (SLDRAM), rambus (Rambus) direct RAM (RDRAM), direct bused dynamic RAM (DRDRAM), and Rambus Dynamic RAM (RDRAM).
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the present application. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.

Claims (15)

1. A video data storage method, comprising:
in response to a decoding process of a video frame to be reconstructed, reading a reference video frame of the video frame to be reconstructed from a main storage external to the video decoder in the absence of the reference video frame in the main storage and in the presence of the reference video frame in the main storage; and
storing the reference video frame read from the main storage into an internal buffer of the video coder.
2. The method of claim 1, wherein each frame of video is divided into a plurality of blocks;
the reading of a reference video frame of a video frame to be reconstructed from a main storage external to the video coder in the absence of the reference video frame in the external storage and in the presence of the reference video frame in the main storage, comprising:
reading a reference video block corresponding to a video frame to be reconstructed from a main storage external to a video coder if the reference video block does not exist in the main storage and if the reference video block exists in the main storage;
the storing the reference video frame read from the main storage into an internal buffer of the video coder, comprising:
storing the reference video block read from the main storage into an internal buffer of the video coder.
3. The method of claim 1, further comprising:
reading the reference video frame from an external storage of a video coder in the event that the reference video frame is present in the external storage of the video coder.
4. The method of claim 1, further comprising:
responding to the decoding processing of the acquired video, and acquiring a motion vector corresponding to each video frame in the video;
determining the number of times of reference of each video frame based on the motion vector corresponding to each video frame;
storing the video frame with the reference times meeting the first time condition into a main storage outside the video decoder;
storing the video frame with the reference times meeting a second time condition into an external storage of the video coder, wherein the times represented by the second time condition are larger than the times represented by the first time condition.
5. The method of any of claims 1 to 4, wherein the external storage of the video coder comprises at least one of a system cache or a system cache region.
6. The method of any of claims 1 to 4, wherein the main storage external to the video coder comprises dynamic random access memory.
7. A method for reading video data, the method comprising:
responding to the decoding processing of the acquired video code stream, reading a reference video block which does not exist in external storage of the video decoder from a main storage outside the video decoder to an internal cache of the video decoder, and then performing compensation or block copying;
or motion compensation or block copy after reading the reference video block from external storage of the video decoder.
8. The method of claim 7, wherein the action of compensating or block copying is performed after reading, in response to the decoding process of the retrieved video bitstream, reference video blocks that are not stored outside the video decoder from a main storage outside the video decoder into an internal buffer of the video decoder; or after reading the reference video block from the external storage of the video decoder, performing motion compensation or block copy, including:
searching from an external storage of the video decoder in response to the decoding processing of the acquired video code stream, and performing action compensation or block copying after reading a reference video block from the external storage under the condition that the reference video block is searched from the external storage;
in the case where no reference video block is found in the external storage, the reference video block is found from the external main storage of the video decoder, and compensation or block copy is performed after the reference video block is read from the main storage to the internal buffer of the video decoder.
9. The method of claim 7, further comprising:
responding to the decoding processing of the acquired video, and acquiring a motion vector corresponding to each video frame in the video;
determining the number of times of reference of each video frame based on the motion vector corresponding to each video frame;
storing the video frame with the reference times meeting the first time condition into a main storage outside the video decoder;
storing the video frame with the reference times meeting a second time condition into an external storage of the video coder, wherein the times represented by the second time condition are larger than the times represented by the first time condition.
10. The method of any of claims 7 to 9, wherein the external storage of the video coder comprises at least one of a system cache or a system cache region.
11. The method of any of claims 7 to 9, wherein the main storage external to the video coder comprises dynamic random access memory.
12. A video data storage apparatus, comprising:
a first reading module, configured to, in response to a decoding process of a video frame to be reconstructed, read a reference video frame of the video frame to be reconstructed from a main storage outside a video decoder in a case where the reference video frame does not exist in the main storage and exists in the main storage outside the video decoder; and
a reference video frame storage module for storing the reference video frames read from the main storage into an internal buffer of the video decoder.
13. A video data reading apparatus, comprising:
a second reading module, configured to respond to a decoding process of the acquired video stream, read, from a main storage outside the video decoder, a reference video block that is not stored outside the video decoder to an internal cache of the video decoder, and then perform a compensation or block copy operation;
alternatively, the first and second electrodes may be,
and the third reading module is used for performing motion compensation or block copying after reading the reference video block from the external storage of the video decoder.
14. An electronic device comprising a memory and a processor, the memory having stored thereon a computer program, wherein the computer program, when executed by the processor, causes the processor to carry out the steps of the method according to any of the claims 1 to 11.
15. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 11.
CN202110355145.3A 2021-04-01 2021-04-01 Video data storage method and device, electronic equipment and readable storage medium Pending CN115174916A (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN202110355145.3A CN115174916A (en) 2021-04-01 2021-04-01 Video data storage method and device, electronic equipment and readable storage medium
PCT/CN2022/077080 WO2022206212A1 (en) 2021-04-01 2022-02-21 Video data storage method and apparatus, and electronic device and readable storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110355145.3A CN115174916A (en) 2021-04-01 2021-04-01 Video data storage method and device, electronic equipment and readable storage medium

Publications (1)

Publication Number Publication Date
CN115174916A true CN115174916A (en) 2022-10-11

Family

ID=83455179

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110355145.3A Pending CN115174916A (en) 2021-04-01 2021-04-01 Video data storage method and device, electronic equipment and readable storage medium

Country Status (2)

Country Link
CN (1) CN115174916A (en)
WO (1) WO2022206212A1 (en)

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103729449B (en) * 2013-12-31 2017-02-15 上海富瀚微电子股份有限公司 Reference data access management method and device
US10516891B2 (en) * 2015-11-20 2019-12-24 Intel Corporation Method and system of reference frame caching for video coding
US20170188033A1 (en) * 2015-12-23 2017-06-29 Mediatek Inc. Method and Apparatus of Bandwidth Estimation and Reduction for Video Coding
US20210127125A1 (en) * 2019-10-23 2021-04-29 Facebook Technologies, Llc Reducing size and power consumption for frame buffers using lossy compression

Also Published As

Publication number Publication date
WO2022206212A1 (en) 2022-10-06

Similar Documents

Publication Publication Date Title
US10200706B2 (en) Pipelined video decoder system
US20180268571A1 (en) Image compression device
US8619862B2 (en) Method and device for generating an image data stream, method and device for reconstructing a current image from an image data stream, image data stream and storage medium carrying an image data stream
US9612962B2 (en) Performing cache bank operations in offset sequences from first bank
CN106470323B (en) The storage method and equipment of video data
CN101714161A (en) Cache device and method for embedded browser
US9832477B2 (en) Data encoding with sign data hiding
CN104125458A (en) Lossless stored data compression method and device
US8963809B1 (en) High performance caching for motion compensated video decoder
US20160373757A1 (en) Analytics Assisted Encoding
US8687083B2 (en) High resolution digital image capturing apparatus and reference pixel memory storage space configuration method
US10757430B2 (en) Method of operating decoder using multiple channels to reduce memory usage and method of operating application processor including the decoder
US7843460B2 (en) Method and apparatus for bandwidth corruption recovery
CN112995679A (en) Motion adaptive coding of video
KR20120066305A (en) Caching apparatus and method for video motion estimation and motion compensation
EP1584069A1 (en) Video frame correlation for motion estimation
US7881367B2 (en) Method of video coding for handheld apparatus
TWI601075B (en) Motion compensation image processing apparatus and image processing method
US8514237B2 (en) Two dimensional memory caching apparatus for high definition video
US9363524B2 (en) Method and apparatus for motion compensation reference data caching
CN115174916A (en) Video data storage method and device, electronic equipment and readable storage medium
US8548275B2 (en) Image processing device and image processing method
CN110545402A (en) underground monitoring video processing method, computer equipment and storage medium
WO2022115999A1 (en) Data processing method and data processing device
CN113535606A (en) Data processing method and device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination