CN113747171A - Self-recovery video decoding method - Google Patents
Self-recovery video decoding method Download PDFInfo
- Publication number
- CN113747171A CN113747171A CN202110903528.XA CN202110903528A CN113747171A CN 113747171 A CN113747171 A CN 113747171A CN 202110903528 A CN202110903528 A CN 202110903528A CN 113747171 A CN113747171 A CN 113747171A
- Authority
- CN
- China
- Prior art keywords
- frame
- image
- data
- decoding
- data packet
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 58
- 238000011084 recovery Methods 0.000 title claims abstract description 17
- 238000001914 filtration Methods 0.000 claims abstract description 18
- 230000007246 mechanism Effects 0.000 claims abstract description 12
- 238000012544 monitoring process Methods 0.000 claims abstract description 8
- 238000012216 screening Methods 0.000 claims abstract description 6
- 230000006870 function Effects 0.000 claims description 31
- 230000008569 process Effects 0.000 claims description 25
- 230000009191 jumping Effects 0.000 claims description 11
- 238000004590 computer program Methods 0.000 claims description 9
- 230000005540 biological transmission Effects 0.000 claims description 6
- 238000012545 processing Methods 0.000 claims description 5
- 238000006243 chemical reaction Methods 0.000 claims description 3
- 230000002159 abnormal effect Effects 0.000 abstract description 2
- 238000004422 calculation algorithm Methods 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/44—Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/65—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using error resilience
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/80—Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Compression Or Coding Systems Of Tv Signals (AREA)
Abstract
The invention relates to a self-recovery video decoding method, which comprises the steps of firstly, transplanting ffmpeg to a Haisi platform, and modifying source codes of the ffmpeg to enable the ffmpeg to be adapted to a Haisi chip. Secondly, starting a dynamic library of the ffmpeg to combine parameter frames and image frames of data transmitted by the network to obtain complete image information, then establishing a filtering mechanism, dynamically screening error frames, analyzing a coding rule, adjusting the sequence of data frames in an error data packet, and transmitting the decodable data packet to a hard decoding module of the Haesi chip. And finally, establishing an image acquisition thread, acquiring decoded image data from the decoding channel, and starting a self-recovery script monitoring decoding program to prevent abnormal exit, thereby completing a decoding task.
Description
Technical Field
The invention belongs to a video decoding technology under a linux system, and particularly relates to a self-recovery video decoding method.
Background
Hi3559AV100 is a professional 8K Ultra HD Mobile Camera SOC, provides digital video recording of 8K30/4K120 broadcast-level image quality, supports multi-channel Sensor input, supports H.265 coding output or video-level RAW data output, integrates high-performance ISP processing, and simultaneously adopts an advanced low-power-consumption process and a low-power-consumption architecture design, and provides excellent image processing capability for users. Hi3559AV100 supports the leading multi-channel 4K Sensor input in the industry, multi-channel ISP image processing, the high dynamic range technical standard of HDR10 and multi-channel panoramic hardware splicing. In support of 8K30/4K120 video recording, Hi3559AV100 provides hardened 6-Dof digital anti-shake, reducing reliance on mechanical holders.
However, the Hi3559AV100 belongs to the hard decoding category, and when the protocol frame does not completely conform to the decoding protocol or the number of error frames is large, the decoding efficiency is low or decoding is impossible, and the hard decoding module cannot extract the parameter information for the image information with parameter frames. In addition, there are cases where the hard decoding module may be abnormally exited due to a memory leak or a device error.
Disclosure of Invention
The technical problem solved by the invention is as follows: the method is designed and realized by aiming at the characteristics of Haisi Hi3559AV100 chips and combining the properties of a hard decoding module and the characteristics of soft decoding under the application scene that a system is linux and based on a self-recovery mechanism.
The technical solution of the invention is as follows:
in a first aspect, a method for decoding a self-healing video, comprises the steps of:
1) the compiling attribute and parameter of the ffmpeg are configured, and the ffmpeg dynamic library is transplanted to the Haisi platform;
2) creating a network receiving task to receive original compressed data transmitted by a host;
3) creating a data packet acquiring thread, and acquiring an image data packet with a parameter-containing frame by using an ffmpeg dynamic library;
4) establishing a filtering mechanism, screening error frames, combining the error frames into a decodable data packet and sending the decodable data packet to a hard decoding module of the chip;
5) acquiring a decoding image in a decoding module;
6) and starting a self-recovery script monitoring decoding program.
Optionally, the method for starting the self-recovery script monitoring decoding program in step 6) specifically includes:
61) setting a self-recovery script S09Decode under the system/etc/rcS/path;
62) acquiring a process number of decode _ video operation;
63) acquiring the running state of the decode _ video process by using a ps command;
64) if the running state of the decode _ video is the running state stillRunning, delaying for 5ms, and returning to step 62); executing the next step if decode _ video is in the exit state;
65) adding an exit decoding channel in an initialization function of decode, releasing a video cache block and releasing system resources;
66) entering a directory where the decode _ video is located by using a cd command, and executing a decode program;
67) return to step 62).
Optionally, the step 1) of dynamically storing ffmpeg into the haisi platform specifically includes:
11) firstly, configuring ffmpeg compiling attribute, and configuring parameters according to a platform type, a cpu type, a codec attribute, a format conversion attribute and a cross compiling attribute;
12) modify acodec.h file, add variable: the parameter frame length SEI _ len and the array SEI _ BUF are determined, and meanwhile, the parameter frame SIZE SEI _ BUF _ SIZE is determined according to the application layer requirements;
13) adding a parameter frame acquiring function in a decode _ nal _ sei _ prefix function in a hevc _ sei.c file: obtaining a parameter SIZE in a function, assigning the SIZE to SEI _ len, judging whether the SIZE is smaller than or equal to SEI _ BUF _ SIZE, copying data in a context parameter array gb to SEI _ BUF if a condition is met, wherein the copy length is the SIZE, and an array subscript i of the copied SEI _ BUF is an index of the gb array divided by 8, namely SEI _ BUF [ i ] ═ gb _ BUF [ index/8 ];
14) then, executing a configure command, and generating decoding libraries libavcodec, libavformat, libavutil and libswscale under the subfolder lib of the configuration folder;
15) and finally, copying the dynamic library to a/usr/lib path of a decoding board card.
Optionally, the receiving the original compressed image transmitted by the host in step 2) specifically includes:
21) firstly, acquiring a receiving ip and a port in a configuration file;
22) then, using the received ip address and the port number as parameters to create and bind the network socket, and after the creation is successful, performing step 23);
23) secondly, clearing the receiving buffer zone, and entering step 24) after waiting for receiving the image data sent by the network;
24) judging whether the length of the image data received this time is greater than zero, if so, carrying out the next step, otherwise, returning to the step 23);
25) judging whether the protocol frame header meets the protocol requirement, if not, discarding the frame; and if the image frame is matched with the image frame, storing the image frame in a data receiving area.
Optionally, the creating and acquiring a data packet thread in step 3) specifically includes:
321) acquiring an image data array first address pointer bufPtr and a length bufLen read from a network;
322) judging whether the current data length bufLen is larger than 0, if so, continuing the next step, and if not, exiting the decoding process of the data and waiting for the next data transmission of the network;
323) transmitting a data array first address pointer bufPtr and a length bufLen to a soft decoding module, segmenting data by frames by using a library function av _ parser _ parse2, recording and continuing the next step if a complete image frame data packet can be successfully obtained from the array, and quitting the decoding process if the complete image frame data packet cannot be obtained from the array, and waiting for the next image data array transmitted by the network;
324) storing the length ret of the image frame divided in the data array at the time, removing the length ret of the image frame divided at the time from the total length bufLen of the data array, and moving a first address pointer bufPtr forwards, wherein the moving times are equal to the length ret of the image frame data packet;
325) and putting the image complete data packet segmented this time into a queue to be decoded.
Optionally, the method for establishing a filtering mechanism and screening an error frame in step 4) specifically includes:
41) judging the image type, analyzing the number of P frames contained in the image data packets of the slice number required by different types, and recording the data structure variable PSlicENumber of the channel;
42) judging whether the size of the current data packet is larger than the maximum number specified by the protocol frame, if so, exiting the filtering process of the data packet, returning to the step 41 after receiving a new image data packet, and otherwise, carrying out the next step;
43) circularly traversing the data in the data packet, sequentially judging whether the current frame accords with the h265 protocol frame header, if so, performing the next step, and if not, jumping to the step 49);
44) judging whether the current frame is a P frame or an I frame, if so, increasing the number temp _ PSlicENumber of the slots in the data packet transmitted by the channel at this time, and entering the next step; if not, jump to step 46);
45) if the temp _ PSlicENumber of the current data packet is equal to 2, judging whether p _ slice is 0 again, if so, setting p _ size to i, and updating and recording a slice flag p _ slice to 1;
46) judging whether the current frame type is SPS, if so, setting the mark top to be 1, otherwise, directly entering 48);
47) continuously judging whether the current frame is a parameter frame, if so, copying the position i in the current data packet array ptr to a data structure Idr _ buf, storing frame header information before the sei frame, recording the data Size Idr _ Size of the currently stored frame header as i, setting the first-appearing sps identifier entryFlag as 1, and resetting top to be 0; step 48) after completion;
48) judging whether the image data of the current channel needs to be inserted into an I frame, and the entryFlag is 1, if the two conditions are satisfied, judging whether the current frame type is a P frame, if the current frame type is the P frame, changing the byte content of the frame type of the data packet from 0x02 to 0x26, and changing the P frame to the I frame; otherwise, directly entering the step 49);
49) continuing to check whether the ptr in the data packet has other data to be traversed for query, if so, returning to the step 43), and if not, continuing to the next step;
410) judging whether the insertIDR mark and the entryFlag of the current channel are both 1, if so, continuing the next step, and if not, performing the step 413);
411) judging whether the current temp _ PSlicENumber is equal to the PSlicENumber which the image type of the channel should have, if so, then the next step, if not, discarding the data packet, and exiting the decoding process of the data packet; returning to step 41) after receiving a new image data packet;
412) copying the data of which the Size of the complete image frame in the current data packet is Size into Idr _ Buf, starting from the Idr _ Size of the Idr _ Buf at the copying position, increasing the Size of the Size by the Idr _ Size, copying the data of the Idr _ Buf into ptr again, wherein the copied data is Size;
413) judging whether the current temp _ PSlicENumber is equal to the PSlicENumber which the channel image type should have, if yes, carrying out the next step, and if not, carrying out the step 415);
414) judging whether p _ slice is 1, if so, assigning p _ size to size, and clearing p _ slice; otherwise, go directly to step 415);
415) recording the size of the image parameter frame of the current channel in seiSize, copying the parameter frame of the current image to a decodeArray of an image data structure to be decoded of the current channel, and taking the parameter number seiCount as a time stamp u64PTS of the image to be decoded;
416) inquiring the decoding state of the current channel, and saving the frame number of the decoded image in the state structure body to u32 DecodeSetreamFrames;
417) acquiring a first address ptr and a data length size of image data to be decoded of a current channel and a time stamp u64PTS, and sending the data to a hard decoding module in a streaming form;
418) circularly inquiring the working state of the decoder until a new decoded image is obtained;
419) calling a Haesi library function to inquire the channel state of the decoder;
420) judging whether the number of bytes needing to be decoded in the channel is 0, if so, entering the next step, and if not, delaying 1ms and jumping to a step 419);
421) whether the number of decoded picture frames of the decoder channel is not equal to the previously saved u32 decoderstreamframes or the insertIDR of the channel is marked as 1, if so, going to the next step, otherwise, going to step 423);
422) clearing the insrtIDR identifier, increasing the number of the parameter frames, judging whether seiCount is greater than the maximum SEI _ NUM specified by the protocol, clearing the parameter frame number identifier seiCount, and entering step 426); otherwise, directly entering the next step;
423) resetting the decoder, proceeding to step 424);
424) judging whether the decoder reset flag resetFlag is 0, if so, entering the next step, if not, continuing to wait for 1ms, and jumping to the step 423);
425) setting a reset state to 1, calling a library function to stop a video stream receiving task of a decoding channel, destroying the decoding channel, restarting the decoding channel, setting an insertIDR (idle IDR) inserted into an I frame to 1, setting the reset state to 0, and entering the next step;
426) and finishing the filtering process.
Optionally, the method for acquiring a decoded image in the decoding module in step 5) specifically includes:
51) creating a decoding image obtaining thread;
52) entering an image acquisition cycle;
53) inquiring whether the reset state in the channel is 1, if so, delaying for 1ms, continuing to inquire the reset state, and if not, carrying out the next step;
54) calling a library function HI _ MPI _ VDEC _ GetFrame to acquire a memory address of image storage; if the failure is failed, jumping to the step 53), if the failure is successful, entering the next step;
55) setting a reset flag resetFlag to 1;
56) traversing parameter frames in the decodeArray structure, comparing whether the stored timestamp u64PTS is consistent with the u64PTS of the current decoding image or not, and if so, sending the image data;
57) creating a network sending task, and sending the data stored in the memory address in the step 54) to the host.
In a second aspect, a processing apparatus comprises:
a memory for storing a computer program;
a processor for calling and running the computer program from the memory to perform the method of the first aspect.
A computer readable storage medium having stored thereon a computer program or instructions which, when executed, implement the method of the first aspect.
A computer program product comprising instructions for causing a computer to perform the method of the first aspect when the computer program product is run on a computer.
Compared with the prior art, the invention has the advantages that:
the invention adopts an ffmpeg decoding library to obtain parameter frame information and data frame information in an original frame, obtain a complete image frame data packet, establish a filtering mechanism to screen error frames, adjust the frame sequence in the data packet, transmit a decodable complete data packet to a hard decoding module to complete a decoding function, and monitor abnormal exit of a decoding program module by using a self-recovery mechanism, thereby ensuring continuous decoding.
Drawings
Fig. 1 is a flowchart illustrating a self-recovery video decoding method according to the present invention.
Detailed Description
In order to solve the video decoding problem in the linux system, a method of combining ffmpeg soft decoding and haisi chip module hard decoding based on a filtering mechanism is adopted in combination with fig. 1. The present invention is further described below.
1) Transplanting ffmpeg to Haisi platform
First, the ffmpeg compile attribute is configured. Then, the ffmpeg source code is modified to have the function of returning to the parameter frame, and the ffmpeg source code is cross-coded. And finally, obtaining the ffmpeg decoding dynamic library and copying the dynamic library to a Haima development board.
11) First, ffmpeg compiling attribute is configured, and parameters are configured according to the platform type, the cpu type, the codec attribute, the format conversion attribute and the cross compiling attribute.
12) Modify acodec.h file, add variable: the parameter frame length SEI _ len and the array SEI _ BUF determine the parameter frame SIZE SEI _ BUF _ SIZE according to the application layer requirements;
13) adding a parameter frame acquiring function in a decode _ nal _ sei _ prefix function in a hevc _ sei.c file: obtaining a parameter SIZE in a function, assigning the SIZE to SEI _ len, judging whether the SIZE is smaller than or equal to SEI _ BUF _ SIZE, copying data in a context parameter array gb to SEI _ BUF if a condition is met, wherein the length of the copy is the SIZE, and an array subscript i of the copied SEI _ BUF is an index of the gb array divided by 8, namely SEI _ BUF [ i ] ═ gb _ BUF [ index/8 ];
14) then, executing a configure command, and generating decoding libraries libavcodec, libavformat, libavutil and libswscale under the subfolder lib of the configuration folder;
15) and finally, copying the dynamic library to a/usr/lib path of a decoding board card.
2) Creating a network receiving thread to receive original compressed image data sent by a host
Firstly, acquiring the information of ip and port number in transmission according to the configuration file. Then, image data waiting for network transmission is blocked.
21) First, the receiving ip and port in the configuration file are obtained.
22) Then, using the received ip address and the port number as parameters to create and bind the network socket, and after the creation is successful, performing step 23);
23) secondly, clearing the receiving buffer zone, and entering step 24) after waiting for receiving the image data sent by the network;
24) and judging whether the length of the image data received this time is greater than zero, if so, carrying out the next step, and otherwise, returning to the step 23).
25) Judging whether the protocol frame header meets the protocol requirement, if not, discarding the frame; and if the image frame is matched with the image frame, storing the image frame in a data receiving area.
3) Obtaining a complete compressed image data packet containing a parameter frame using an ffmpeg dynamic library
First, the ffmpeg usage environment is initialized. Second, an get packet thread is created. The full packet of the compressed image containing the parametric frame is then obtained using the ffmpeg dynamic library.
31) Initializing decoding library usage environment
First, a decoder type is set, and an HEVC (h265) type decoder is adopted. Then, the context environment of the decoder is initialized, and the image frame storage space is dynamically applied.
32) Creating an ffmpeg acquisition data packet thread, wherein the thread comprises the following specific flows:
321) acquiring an image data array first address pointer bufPtr and a length bufLen read from a network;
322) judging whether the current data length bufLen is larger than 0, if so, continuing the next step, and if not, exiting the decoding process of the data and waiting for the next data transmission of the network;
323) transmitting a data array first address pointer bufPtr and a length bufLen to a soft decoding module, segmenting data by frames by using a library function av _ parser _ parse2, recording and continuing the next step if a complete image frame data packet can be successfully obtained from the array, and quitting the decoding process if the complete image frame data packet cannot be obtained from the array, and waiting for the next image data array transmitted by the network;
324) storing the length ret of the image frame divided in the data array at the time, removing the length ret of the image frame divided at the time from the total length bufLen of the data array, and moving a first address pointer bufPtr forwards, wherein the moving times are equal to the length ret of the image frame data packet;
325) and putting the image complete data packet segmented this time into a queue to be decoded.
4) Filtering the complete data packet and sending the complete data packet to a decoding module
And establishing a filtering mechanism, analyzing the number and the type of frames in the data packet, and storing the number and the type of protocol frames required by different images according to a compression principle and image type classification. At the same time, the image data is reprocessed according to the error type, thereby combining decodable data packets to the hard decoding module. The method comprises the following specific steps:
41) judging the image type, analyzing the number of P frames contained in the image data packets of the slice number required by different types), and recording the data structure variable PSlicENumber in the channel;
42) judging whether the size of the current data packet is larger than the maximum number specified by the protocol frame, if so, exiting the filtering process of the data packet, returning to the step 41 after receiving a new image data packet, and otherwise, carrying out the next step;
43) circularly traversing the data in the data packet, wherein the traversal is from 0 to (size-3), the size is the size of the data packet, whether the current frame accords with the h265 protocol frame header (namely, the three continuous bytes are 000001) is judged in a classified mode, if so, the next step is carried out, and if not, the step 49 is skipped;
44) judging whether the current frame is a P frame or an I frame, if so, increasing the number temp _ PSlicENumber of the slots in the data packet transmitted by the channel at this time, and entering the next step; if not, jump to step 46);
45) if the temp _ psicenumber of the current data packet is equal to 2, judging whether the current slice is the first slice of the current data packet, namely whether p _ slice is 0, if so, recording the position i of the traversal byte in the ptr of the current data packet, namely setting p _ size to be i, updating and recording a slice flag p _ slice to be 1, and indicating that one slice already exists;
46) judging whether the current frame type is SPS, if the current frame type is SPS, setting the mark top to be 1 and entering step 47), otherwise, directly entering step 48);
47) judging whether top is 1, if so, continuously judging whether the current frame is a parameter frame (sei frame), if so, copying the i position in the current data packet array ptr to a data structure Idr _ buf, storing frame header information (vps, pps, sps frames) before the sei frame, recording the data Size Idr _ Size of the currently stored frame header as i, marking entryFlag as 1 when the sps appears for the first time, and resetting the top to be 0; step 48) after completion;
48) judging whether the image data of the current channel needs to be inserted with an I frame (identified by an insertiDR), and the entryFlag is 1, if the two conditions are satisfied, judging whether the current frame type is a P frame, if the current frame type is the P frame, changing the byte content of the frame type of the data packet from 0x02 to 0x26, namely changing the P frame to the I frame; otherwise, directly entering the step 49);
49) continuing to check whether the ptr in the data packet has other data to be traversed for query, if so, returning to the step 43), and if not, continuing to the next step;
410) judging whether the insertIDR mark and the entryFlag of the current channel are both 1, if so, continuing the next step, and if not, performing the step 413);
411) judging whether the current temp _ PSlicENumber is equal to the PSlicENumber which the image type of the channel should have, if so, then the next step, if not, discarding the data packet, and exiting the decoding process of the data packet; returning to step 41) after receiving a new image data packet;
412) copying data with the Size of a complete image frame in a current data packet into Idr _ Buf, starting from the Idr _ Size of the Idr _ Buf at the copying position, increasing the Size of the Size by the Idr _ Size (namely the Size of the current data packet needs to be inserted into frame header information stored before), copying the data of the Idr _ Buf into ptr again, wherein the copied data Size is the Size;
413) judging whether the current temp _ PSlicENumber is equal to the PSlicENumber which the channel image type should have, if yes, carrying out the next step, and if not, carrying out the step 415);
414) judging whether p _ slice is 1, if so, assigning p _ size to size (namely, taking only), clearing p _ slice and entering step 415); otherwise, go directly to step 415);
415) recording the size of the image parameter frame of the current channel in seiSize, copying the parameter frame of the current image to a decodeArray of an image data structure to be decoded of the current channel, and taking the parameter number seiCount as a time stamp u64PTS of the image to be decoded;
416) inquiring the decoding state of the current channel, and saving the frame number of the decoded image in the state structure body to u32 DecodeSetreamFrames;
417) acquiring a first address ptr and a data length size of image data to be decoded of a current channel and a time stamp u64PTS, and sending the data to a decoding module in a streaming form;
418) circularly inquiring the working state of the decoder until a new decoded image is obtained;
419) calling a Haesi library function to inquire the channel state of the decoder;
420) judging whether the number of bytes needing to be decoded in the channel is 0, if so, entering the next step, and if not, delaying 1ms and jumping to a step 419);
421) whether the number of decoded picture frames of the decoder channel is not equal to the previously saved u32 decoderstreamframes or the insertIDR (I frame needs to be inserted) of the channel is marked as 1, if yes, the next step is carried out, and if not, the step 423 is carried out;
422) clearing the insrtIDR identifier, increasing the number of the parameter frames (seiCount is increased by 1), judging if seiCount is larger than the maximum SEI _ NUM specified by the protocol, clearing the parameter frame number flag seiCount, and entering step 426); otherwise, directly entering the next step;
423) resetting the decoder, proceeding to step 424);
424) judging whether the decoder reset flag resetFlag is 0, if so, entering the next step, if not, continuing to wait for 1ms, and jumping to the step 423);
425) setting a reset state to 1, calling a library function to stop a video stream receiving task of a decoding channel, destroying the decoding channel, restarting the decoding channel, setting an insertIDR (idle IDR) inserted into an I frame to 1, setting the reset state to 0, and entering the next step;
426) ending the filtering process;
5) capturing images within a decoder channel
And establishing a task of acquiring a decoded image, reading the working state of a decoder in real time, calling a library function to acquire the decoded image in a decoding channel, and sending the decoded image data to a host through a network sending task. The method comprises the following specific steps:
51) creating a decoding image obtaining thread;
52) entering an image acquisition cycle;
53) inquiring whether the reset state in the channel is 1, if so, delaying for 1ms, continuing to inquire the reset state, and if not, carrying out the next step;
54) calling a library function HI _ MPI _ VDEC _ GetFrame to acquire a memory address of image storage, jumping to the step 53 if the library function HI _ MPI _ VDEC _ GetFrame fails, and entering the next step if the library function HI _ MPI _ VDEC _ GetFrame succeeds;
55) setting a reset flag resetFlag to 1;
56) traversing parameter frames in the decodeArray structure, comparing whether the stored timestamp u64PTS is consistent with the u64PTS of the current decoding image or not, and if so, sending the image data;
57) creating a network sending task, and sending the data stored in the memory address in the step 54) to the host.
6) Starting self-recovery process monitoring decoding program
And setting a self-recovery script under a self-running path of the system, and monitoring the running state of the decoding program in real time by the content of the script. And if the decoding program is monitored to be abnormally exited, automatically recovering the decoding program. And simultaneously, stopping the thread of the decoding channel for sending the video stream before the program is restarted, exiting the video buffer block, and releasing system resources. The method comprises the following specific steps:
61) setting a self-recovery script S09Decode under the system/etc/rcS/path;
62) acquiring a process number of decode _ video operation;
63) acquiring the running state of the decode _ video process by using a ps command;
64) if the running state of the decode _ video is the running state stillRunning, delaying for 5ms, and returning to step 62); executing the next step if decode _ video is in the exit state;
65) adding an exit decoding channel in an initialization function of decode, releasing a video cache block and releasing system resources;
66) entering a directory where the decode _ video is located by using a cd command, and executing a decode program;
67) return to step 62).
The step 1) of the invention carries out the configuration of the decoding environment on the board, transplants the ffmpeg of the soft decoding library to adapt to the Haisi platform, and modifies the ffmpeg at the same time, so that the returned image data is provided with the parameter frame. Step 2) performing frame header analysis on the original compressed data, and preliminarily filtering error frames which do not meet the protocol requirements; step 3) a usage environment is created for ffmpeg and a decoder with a start requirement is specified, and protocol analysis can be performed on the original data. Meanwhile, original data are circularly obtained from a data area received by the network, a library function is called to obtain the length of a data packet which can be combined into a complete image frame with a parameter frame, and the length is removed from the data area received by the network and the process is circularly repeated until the data area has no data. And 4) establishing a filtering mechanism, dynamically screening error frames which do not accord with the protocol regulation, analyzing the coding rule, adjusting the sequence of the data packet image frames, and dynamically modifying the frame types according to the characteristics of the coder to combine the error frames into a decodable image data packet. Meanwhile, the state of the decoder is monitored in real time, and the operation of resetting the decoding channel is started according to the requirement of the current data packet, so that the decoding module of the decoding channel continuously works. And 5) establishing a data acquisition thread, starting a library function according to the reset state to acquire the image in the decoding channel, modifying the reset state after the data so that the working state of the decoder can be updated in real time in the step 4, and sending the decoded image to the host through a network sending task. And 6) starting a script for monitoring the decoding program according to the self-running script path characteristics of the linux system, judging the running state of the process in which the decoding program is located in real time, releasing resources when the decoding program is abnormally exited, and starting the decoding program again, so that the decoding is continuously carried out.
By utilizing the technical scheme and adopting the operation steps, the video decoding and transmission problems under the linux system can be realized, and the method is verified by an algorithm and is tested experimentally. The result shows that the scheme can adopt a soft decoding method to form a complete data packet containing image frames and parameter frames for solving the problem of image decoding, adopts a filtering mechanism to dynamically screen error frames which do not conform to the protocol type to obtain decodable image data packets, and then uses a hard decoding module of a chip to complete the decoding process, thereby effectively reducing the decoding time.
Although the present invention has been described with reference to the preferred embodiments, it is not intended to limit the present invention, and those skilled in the art can make variations and modifications of the present invention without departing from the spirit and scope of the present invention by using the methods and technical contents disclosed above.
Those skilled in the art will appreciate that those matters not described in detail in the present specification are well known in the art.
Claims (10)
1. A method for self-healing video decoding, comprising the steps of:
1) the compiling attribute and parameter of the ffmpeg are configured, and the ffmpeg dynamic library is transplanted to the Haisi platform;
2) creating a network receiving task to receive original compressed data transmitted by a host;
3) creating a data packet acquiring thread, and acquiring an image data packet with a parameter-containing frame by using an ffmpeg dynamic library;
4) establishing a filtering mechanism, screening error frames, combining the error frames into a decodable data packet and sending the decodable data packet to a hard decoding module of the chip;
5) acquiring a decoding image in a decoding module;
6) and starting a self-recovery script monitoring decoding program.
2. A method for self-healing video decoding as recited in claim 1, wherein: step 6) the method for starting the self-recovery script monitoring decoding program specifically comprises the following steps:
61) setting a self-recovery script S09Decode under the system/etc/rcS/path;
62) acquiring a process number of decode _ video operation;
63) acquiring the running state of the decode _ video process by using a ps command;
64) if the running state of the decode _ video is the running state stillRunning, delaying for 5ms, and returning to step 62); executing the next step if decode _ video is in the exit state;
65) adding an exit decoding channel in an initialization function of decode, releasing a video cache block and releasing system resources;
66) entering a directory where the decode _ video is located by using a cd command, and executing a decode program;
67) return to step 62).
3. A method for self-healing video decoding as recited in claim 1, wherein: step 1) the ffmpeg dynamic library is stored in a Haisi platform, which specifically comprises the following steps:
11) firstly, configuring ffmpeg compiling attribute, and configuring parameters according to a platform type, a cpu type, a codec attribute, a format conversion attribute and a cross compiling attribute;
12) modify acodec.h file, add variable: the parameter frame length SEI _ len and the array SEI _ BUF are determined, and meanwhile, the parameter frame SIZE SEI _ BUF _ SIZE is determined according to the application layer requirements;
13) adding a parameter frame acquiring function in a decode _ nal _ sei _ prefix function in a hevc _ sei.c file: obtaining a parameter SIZE in a function, assigning the SIZE to SEI _ len, judging whether the SIZE is smaller than or equal to SEI _ BUF _ SIZE, copying data in a context parameter array gb to SEI _ BUF if a condition is met, wherein the copy length is the SIZE, and an array subscript i of the copied SEI _ BUF is an index of the gb array divided by 8, namely SEI _ BUF [ i ] ═ gb _ BUF [ index/8 ];
14) then, executing a configure command, and generating decoding libraries libavcodec, libavformat, libavutil and libswscale under the subfolder lib of the configuration folder;
15) and finally, copying the dynamic library to a/usr/lib path of a decoding board card.
4. A method for self-healing video decoding as recited in claim 1, wherein: step 2) the original compressed image transmitted by the receiving host specifically comprises:
21) firstly, acquiring a receiving ip and a port in a configuration file;
22) then, using the received ip address and the port number as parameters to create and bind the network socket, and after the creation is successful, performing step 23);
23) secondly, clearing the receiving buffer zone, and entering step 24) after waiting for receiving the image data sent by the network;
24) judging whether the length of the image data received this time is greater than zero, if so, carrying out the next step, otherwise, returning to the step 23);
25) judging whether the protocol frame header meets the protocol requirement, if not, discarding the frame; and if the image frame is matched with the image frame, storing the image frame in a data receiving area.
5. A method for self-healing video decoding as recited in claim 1, wherein: step 3) the creating and acquiring data packet thread specifically comprises the following steps:
321) acquiring an image data array first address pointer bufPtr and a length bufLen read from a network;
322) judging whether the current data length bufLen is larger than 0, if so, continuing the next step, and if not, exiting the decoding process of the data and waiting for the next data transmission of the network;
323) transmitting a data array first address pointer bufPtr and a length bufLen to a soft decoding module, segmenting data by frames by using a library function av _ parser _ parse2, recording and continuing the next step if a complete image frame data packet can be successfully obtained from the array, and quitting the decoding process if the complete image frame data packet cannot be obtained from the array, and waiting for the next image data array transmitted by the network;
324) storing the length ret of the image frame divided in the data array at the time, removing the length ret of the image frame divided at the time from the total length bufLen of the data array, and moving a first address pointer bufPtr forwards, wherein the moving times are equal to the length ret of the image frame data packet;
325) and putting the image complete data packet segmented this time into a queue to be decoded.
6. A method for self-healing video decoding as recited in claim 1, wherein: step 4) the method for establishing the filtering mechanism and screening the error frame specifically comprises the following steps:
41) judging the image type, analyzing the number of P frames contained in the image data packets of the slice number required by different types, and recording the data structure variable PSlicENumber of the channel;
42) judging whether the size of the current data packet is larger than the maximum number specified by the protocol frame, if so, exiting the filtering process of the data packet, returning to the step 41 after receiving a new image data packet, and otherwise, carrying out the next step;
43) circularly traversing the data in the data packet, sequentially judging whether the current frame accords with the h265 protocol frame header, if so, performing the next step, and if not, jumping to the step 49);
44) judging whether the current frame is a P frame or an I frame, if so, increasing the number temp _ PSlicENumber of the slots in the data packet transmitted by the channel at this time, and entering the next step; if not, jump to step 46);
45) if the temp _ PSlicENumber of the current data packet is equal to 2, judging whether p _ slice is 0 again, if so, setting p _ size to i, and updating and recording a slice flag p _ slice to 1;
46) judging whether the current frame type is SPS, if so, setting the mark top to be 1, otherwise, directly entering 48);
47) continuously judging whether the current frame is a parameter frame, if so, copying the position i in the current data packet array ptr to a data structure Idr _ buf, storing frame header information before the sei frame, recording the data Size Idr _ Size of the currently stored frame header as i, setting the first-appearing sps identifier entryFlag as 1, and resetting top to be 0; step 48) after completion;
48) judging whether the image data of the current channel needs to be inserted into an I frame, and the entryFlag is 1, if the two conditions are satisfied, judging whether the current frame type is a P frame, if the current frame type is the P frame, changing the byte content of the frame type of the data packet from 0x02 to 0x26, and changing the P frame to the I frame; otherwise, directly entering the step 49);
49) continuing to check whether the ptr in the data packet has other data to be traversed for query, if so, returning to the step 43), and if not, continuing to the next step;
410) judging whether the insertIDR mark and the entryFlag of the current channel are both 1, if so, continuing the next step, and if not, performing the step 413);
411) judging whether the current temp _ PSlicENumber is equal to the PSlicENumber which the image type of the channel should have, if so, then the next step, if not, discarding the data packet, and exiting the decoding process of the data packet; returning to step 41) after receiving a new image data packet;
412) copying the data of which the Size of the complete image frame in the current data packet is Size into Idr _ Buf, starting from the Idr _ Size of the Idr _ Buf at the copying position, increasing the Size of the Size by the Idr _ Size, copying the data of the Idr _ Buf into ptr again, wherein the copied data is Size;
413) judging whether the current temp _ PSlicENumber is equal to the PSlicENumber which the channel image type should have, if yes, carrying out the next step, and if not, carrying out the step 415);
414) judging whether p _ slice is 1, if so, assigning p _ size to size, and clearing p _ slice; otherwise, go directly to step 415);
415) recording the size of the image parameter frame of the current channel in seiSize, copying the parameter frame of the current image to a decodeArray of an image data structure to be decoded of the current channel, and taking the parameter number seiCount as a time stamp u64PTS of the image to be decoded;
416) inquiring the decoding state of the current channel, and saving the frame number of the decoded image in the state structure body to u32 DecodeSetreamFrames;
417) acquiring a first address ptr and a data length size of image data to be decoded of a current channel and a time stamp u64PTS, and sending the data to a hard decoding module in a streaming form;
418) circularly inquiring the working state of the decoder until a new decoded image is obtained;
419) calling a Haesi library function to inquire the channel state of the decoder;
420) judging whether the number of bytes needing to be decoded in the channel is 0, if so, entering the next step, and if not, delaying 1ms and jumping to a step 419);
421) whether the number of decoded picture frames of the decoder channel is not equal to the previously saved u32 decoderstreamframes or the insertIDR of the channel is marked as 1, if so, going to the next step, otherwise, going to step 423);
422) clearing the insrtIDR identifier, increasing the number of the parameter frames, judging whether seiCount is greater than the maximum SEI _ NUM specified by the protocol, clearing the parameter frame number identifier seiCount, and entering step 426); otherwise, directly entering the next step;
423) resetting the decoder, proceeding to step 424);
424) judging whether the decoder reset flag resetFlag is 0, if so, entering the next step, if not, continuing to wait for 1ms, and jumping to the step 423);
425) setting a reset state to 1, calling a library function to stop a video stream receiving task of a decoding channel, destroying the decoding channel, restarting the decoding channel, setting an insertIDR (idle IDR) inserted into an I frame to 1, setting the reset state to 0, and entering the next step;
426) and finishing the filtering process.
7. A method for self-healing video decoding as recited in claim 1, wherein: step 5) the method for obtaining the decoded image in the decoding module specifically comprises the following steps:
51) creating a decoding image obtaining thread;
52) entering an image acquisition cycle;
53) inquiring whether the reset state in the channel is 1, if so, delaying for 1ms, continuing to inquire the reset state, and if not, carrying out the next step;
54) calling a library function HI _ MPI _ VDEC _ GetFrame to acquire a memory address of image storage; if the failure is failed, jumping to the step 53), if the failure is successful, entering the next step;
55) setting a reset flag resetFlag to 1;
56) traversing parameter frames in the decodeArray structure, comparing whether the stored timestamp u64PTS is consistent with the u64PTS of the current decoding image or not, and if so, sending the image data;
57) creating a network sending task, and sending the data stored in the memory address in the step 54) to the host.
8. A processing apparatus, comprising:
a memory for storing a computer program;
a processor for calling and running the computer program from the memory to perform the method of any of claims 1 to 7.
9. A computer-readable storage medium, having stored thereon a computer program or instructions, which, when executed, implement the method of any one of claims 1 to 7.
10. A computer program product, characterized in that it comprises instructions which, when run on a computer, cause the computer to carry out the method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110903528.XA CN113747171B (en) | 2021-08-06 | 2021-08-06 | Self-recovery video decoding method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110903528.XA CN113747171B (en) | 2021-08-06 | 2021-08-06 | Self-recovery video decoding method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113747171A true CN113747171A (en) | 2021-12-03 |
| CN113747171B CN113747171B (en) | 2024-04-19 |
Family
ID=78730319
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110903528.XA Active CN113747171B (en) | 2021-08-06 | 2021-08-06 | Self-recovery video decoding method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113747171B (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101674465A (en) * | 2009-10-28 | 2010-03-17 | 广东威创视讯科技股份有限公司 | Method and system for multi-task monitoring process of videos |
| US20120213291A1 (en) * | 2011-02-23 | 2012-08-23 | Qualcomm Incorporated | Multi-metric filtering |
| CN106131563A (en) * | 2016-07-28 | 2016-11-16 | 武汉斗鱼网络科技有限公司 | The method and system of hardware decoding H264 video flowing are carried out based on DXVA |
| US20170238022A1 (en) * | 2016-02-15 | 2017-08-17 | Nvidia Corporation | Quality aware error concealment method for video and game streaming and a viewing device employing the same |
| CN108737182A (en) * | 2018-05-22 | 2018-11-02 | 平安科技（深圳）有限公司 | The processing method and system of system exception |
| CN110795264A (en) * | 2019-10-14 | 2020-02-14 | 杭州海兴电力科技股份有限公司 | Monitoring management method and system and intelligent management terminal |
| CN112261460A (en) * | 2020-10-19 | 2021-01-22 | 天津津航计算技术研究所 | PCIE-based multi-channel video decoding scheme design method |
| CN112672166A (en) * | 2020-12-24 | 2021-04-16 | 北京睿芯高通量科技有限公司 | Multi-code stream decoding acceleration system and method of video decoder |
| CN113064762A (en) * | 2021-04-09 | 2021-07-02 | 上海新炬网络信息技术股份有限公司 | Service self-recovery method based on multiple detection |
-
2021
- 2021-08-06 CN CN202110903528.XA patent/CN113747171B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101674465A (en) * | 2009-10-28 | 2010-03-17 | 广东威创视讯科技股份有限公司 | Method and system for multi-task monitoring process of videos |
| US20120213291A1 (en) * | 2011-02-23 | 2012-08-23 | Qualcomm Incorporated | Multi-metric filtering |
| US20170238022A1 (en) * | 2016-02-15 | 2017-08-17 | Nvidia Corporation | Quality aware error concealment method for video and game streaming and a viewing device employing the same |
| CN106131563A (en) * | 2016-07-28 | 2016-11-16 | 武汉斗鱼网络科技有限公司 | The method and system of hardware decoding H264 video flowing are carried out based on DXVA |
| CN108737182A (en) * | 2018-05-22 | 2018-11-02 | 平安科技（深圳）有限公司 | The processing method and system of system exception |
| CN110795264A (en) * | 2019-10-14 | 2020-02-14 | 杭州海兴电力科技股份有限公司 | Monitoring management method and system and intelligent management terminal |
| CN112261460A (en) * | 2020-10-19 | 2021-01-22 | 天津津航计算技术研究所 | PCIE-based multi-channel video decoding scheme design method |
| CN112672166A (en) * | 2020-12-24 | 2021-04-16 | 北京睿芯高通量科技有限公司 | Multi-code stream decoding acceleration system and method of video decoder |
| CN113064762A (en) * | 2021-04-09 | 2021-07-02 | 上海新炬网络信息技术股份有限公司 | Service self-recovery method based on multiple detection |
Non-Patent Citations (3)
| Title |
|---|
| P. -C. WANG ET AL.: "Enhanced Backward Error Concealment for H.264/AVC Videos on Error-Prone Networks", 《2013 INTERNATIONAL SYMPOSIUM ON BIOMETRICS AND SECURITY TECHNOLOGIES》, 16 September 2013 (2013-09-16), pages 62 - 66 * |
| 张准；崔朝霞；李小秀: "基于嵌入式Linux的操作系统故障自动恢复的研究", 《数字技术与应用》, no. 10, 18 December 2009 (2009-12-18), pages 40 - 41 * |
| 梁家翠: "基于 iOS 平台网络视频会话系统的研究与实现", 《中国优秀硕士学位论文全文数据库》信息科技辑, 15 February 2017 (2017-02-15) * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113747171B (en) | 2024-04-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113645490B (en) | Soft-hard combined multichannel video synchronous decoding method | |
| CN113727114B (en) | Transcoded video decoding method | |
| US9961398B2 (en) | Method and device for switching video streams | |
| US10476928B2 (en) | Network video playback method and apparatus | |
| CN113170239A (en) | Method, apparatus and computer program for encapsulating media data into a media file | |
| CN110996160B (en) | Video processing method and device, electronic equipment and computer readable storage medium | |
| CN114095784A (en) | H.265 format video stream transcoding playing method, system, device and medium | |
| CN113727116B (en) | Video decoding method based on filtering mechanism | |
| WO2021254375A1 (en) | Video partitioning method, transfer method, server, adaptor and storage medium | |
| CN113596469A (en) | Soft-hard combined and high-efficiency transmission video decoding method | |
| CN105379281B (en) | Picture reference control for video decoding using a graphics processor | |
| WO2024098836A1 (en) | Video alignment method and apparatus | |
| CN115802045A (en) | Data packet filtering method and decoding method based on data packet filtering method | |
| CN113747171B (en) | Self-recovery video decoding method | |
| US8543612B2 (en) | Mobile terminal | |
| CN113727115B (en) | Efficient transcoded video decoding method | |
| WO2023033300A1 (en) | Encoding and decoding video data | |
| CN112291483B (en) | Video pushing method and system, electronic equipment and readable storage medium | |
| CN113645467B (en) | Soft and hard combined video decoding method | |
| CN111447490A (en) | Streaming media file processing method and device | |
| CN113709518A (en) | RTSP (real time streaming protocol) -based video real-time transmission mode design method | |
| CN114125493B (en) | Distributed storage method, device and equipment for streaming media | |
| US9100717B2 (en) | Methods and systems for file based content verification using multicore architecture | |
| CN116647702A (en) | Self-recovery video decoding method and device | |
| US20230067994A1 (en) | Encoding and decoding video data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |