WO2019141193A1 - 对视频帧数据进行处理的方法和装置 - Google Patents

对视频帧数据进行处理的方法和装置 Download PDF

Info

Publication number
WO2019141193A1
WO2019141193A1 PCT/CN2019/072033 CN2019072033W WO2019141193A1 WO 2019141193 A1 WO2019141193 A1 WO 2019141193A1 CN 2019072033 W CN2019072033 W CN 2019072033W WO 2019141193 A1 WO2019141193 A1 WO 2019141193A1
Authority
WO
WIPO (PCT)
Prior art keywords
video frame
data
converted
frame data
neural network
Prior art date
Application number
PCT/CN2019/072033
Other languages
English (en)
French (fr)
Inventor
宋晓丹
周璐璐
姚佳宝
王莉
武晓阳
Original Assignee
杭州海康威视数字技术股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 杭州海康威视数字技术股份有限公司 filed Critical 杭州海康威视数字技术股份有限公司
Publication of WO2019141193A1 publication Critical patent/WO2019141193A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
    • G06N3/063—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/103—Selection of coding mode or of prediction mode
    • H04N19/107—Selection of coding mode or of prediction mode between spatial and temporal predictive coding, e.g. picture refresh
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/124—Quantisation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
    • H04N19/172—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a picture, frame or field
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/593—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving spatial prediction techniques
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/60—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding
    • H04N19/61—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding in combination with predictive coding
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/90—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using coding techniques not provided for in groups H04N19/10-H04N19/85, e.g. fractals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/90—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using coding techniques not provided for in groups H04N19/10-H04N19/85, e.g. fractals
    • H04N19/91—Entropy coding, e.g. variable length coding [VLC] or arithmetic coding

Definitions

  • the present disclosure relates to the field of video codec technology, and more particularly to a method and apparatus for processing video frame data.
  • the video frame image In the process of compression encoding a video frame image, it is necessary to process the video frame image, such as filtering processing. Specifically, the original video frame image is distorted, so the video frame image obtained during the decoding process is also a distorted video frame image. In order not to affect the use of the video frame image, it is necessary to filter the decoded distortion video frame image to obtain a de-distorted video frame image.
  • neural network can be used to filter distorted video frame images.
  • the data that is operated in the neural network is floating-point data, and the operation result of the floating-point data is related to the operation mode.
  • the number of decimal places of floating-point data is variable.
  • the result of the operation will inevitably exceed the range that the floating-point data can represent. Therefore, the result of the operation is always constrained, that is, the decimal of the result of the operation.
  • Bits are constrained to the extent that the data of the floating point type can be represented.
  • the data after the constraint is approximate data. Due to the existence of approximate data, the order of operations directly affects the results of the operation.
  • floating-point data A, B, C if the number of decimal places of A, B, C is inconsistent, to calculate the result of their addition, the first way, you can first calculate the result of A + B to constrain , plus C, and then constrained to get D1. In the second way, you can also calculate the result of B+C to constrain, add A, and then constrain to get D2.
  • the results obtained by the above two methods are different from D1 and D2.
  • a method of processing video frame data comprising:
  • the converted video frame data is input into a neural network loaded with the converted weight parameter to obtain video frame data after the target processing.
  • the neural network is a convolutional neural network
  • the weight parameters include a convolution kernel element and an offset.
  • converting the data type of the weight parameter in the pre-trained neural network to a fixed-point type, and obtaining the converted weight parameter including:
  • the data type of the video frame data to be subjected to the target processing is converted into a fixed-point type, and the converted video frame data is obtained, including:
  • Data of the video frame data to be subjected to the target processing according to the data bit width of the preset fixed-point type video frame data and the data with the largest absolute value among the feature data output from the input layer of the convolutional neural network The type is converted to a fixed point type, and the converted video frame data is obtained.
  • the method further includes:
  • the converted video frame data and the converted side information are input into a neural network loaded with the converted weight parameter to obtain the target processed video frame data.
  • the method further includes:
  • the video frame data after the target processing is rounded to obtain an integer video frame data.
  • the target processing is a de-distortion filtering process
  • Converting the data type of the video frame data to be subjected to the target processing into a fixed-point type, and obtaining the converted video frame data including:
  • the converted video frame data is input into a neural network loaded with the converted weight parameter, and subjected to de-distortion filtering processing to obtain a de-distorted video frame image.
  • the target processing is a coding intra prediction process
  • Converting the data type of the video frame data to be subjected to the target processing into a fixed-point type, and obtaining the converted video frame data including:
  • An image of a target area in the original unprocessed video frame image in the video encoding process, and an associated area corresponding to the target area in the video frame image obtained by the reconstruction processing corresponding to the original unprocessed video frame image Converting the data type of the image to a fixed-point type, and obtaining the converted video frame data;
  • the converted video frame data is input into a neural network loaded with the converted weight parameter, and the encoded intra prediction process is performed to obtain an intra prediction image and intra prediction related information.
  • the target processing is an encoding inter prediction process
  • Converting the data type of the video frame data to be subjected to the target processing into a fixed-point type, and obtaining the converted video frame data including:
  • the converted video frame data is input into a neural network loaded with the converted weight parameter, and the inter-frame prediction process is performed to obtain an inter-predicted image and inter-frame prediction related information.
  • the target processing is an entropy encoding process
  • Converting the data type of the video frame data to be subjected to the target processing into a fixed-point type, and obtaining the converted video frame data including:
  • the converted video frame data is input into a neural network loaded with the converted weight parameter, and entropy coding processing is performed to obtain entropy coding information.
  • the target processing is an entropy decoding process
  • Converting the data type of the video frame data to be subjected to the target processing into a fixed-point type, and obtaining the converted video frame data including:
  • the converted video frame data is input into a neural network loaded with the converted weight parameter, and entropy decoding processing is performed to obtain intra prediction related information, inter prediction related information, and quantized coefficients.
  • the target processing is decoding intra prediction processing
  • Converting the data type of the video frame data to be subjected to the target processing into a fixed-point type, and obtaining the converted video frame data including:
  • the converted video frame data is input to a neural network loaded with the converted weight parameter, and the decoded intra prediction process is performed to obtain an intra prediction image of the target region.
  • the target processing is decoding inter prediction processing
  • Converting the data type of the video frame data to be subjected to the target processing into a fixed-point type, and obtaining the converted video frame data including:
  • the converted video frame data is input to a neural network loaded with the converted weight parameter, and the inter-frame prediction process is performed to obtain an inter-predicted image.
  • an apparatus for processing video frame data comprising:
  • a first conversion module configured to convert a data type of a weight parameter in a pre-trained neural network to a fixed-point type, to obtain a converted weight parameter, wherein the neural network is used for a video frame in a video encoding and decoding process
  • An algorithm model for data processing
  • a second conversion module configured to convert a data type of the video frame data to be subjected to the target processing into a fixed point type, to obtain the converted video frame data
  • the input module is configured to input the converted video frame data into a neural network loaded with the converted weight parameter to obtain video frame data after the target processing.
  • the neural network is a convolutional neural network
  • the weight parameters include a convolution kernel element and an offset.
  • the first conversion module includes:
  • a first determining unit configured to determine, for each convolution kernel in the pre-trained convolutional neural network, a convolution kernel element having the largest absolute value in the convolution kernel
  • a second determining unit configured to determine, for the plurality of offsets in the convolutional neural network, an offset with an absolute maximum value among the plurality of offsets
  • a conversion unit for convolution kernel elements in each convolution kernel according to a convolution kernel element having the largest absolute value in each convolution kernel and a data bit width of a predetermined fixed-point convolution kernel element Converting the data type to a fixed-point type, converting the data types of the plurality of offsets to a fixed-point type according to the offset of the largest absolute value among the plurality of offsets, and the data width of the offset of the preset fixed-point type The converted weight parameter.
  • the second conversion module is configured to: according to a data bit width of the preset fixed-point type video frame data, and a pre-statisticd maximum value of the feature data output by the input layer of the convolutional neural network Data, the data type of the video frame data to be processed by the target is converted into a fixed point type, and the converted video frame data is obtained.
  • the device further includes:
  • a third conversion module configured to convert a data type of the side information of the preset video frame data into a fixed point type, to obtain converted side information
  • the input module is configured to input the converted video frame data and the converted side information into a neural network loaded with the converted weight parameter to obtain the target processed video frame data.
  • the device further includes:
  • the rounding module is configured to perform rounding processing on the video frame data after the target processing to obtain an integer video frame data.
  • the target processing is a de-distortion filtering process
  • the second conversion module is configured to convert a data type of a video frame image obtained by performing a reconstruction process in a video encoding and decoding process into a fixed point type, to obtain converted video frame data;
  • the input module is configured to input the converted video frame data into a neural network loaded with the converted weight parameter, perform de-distortion filtering processing, and obtain a de-distorted video frame image.
  • the target processing is a coding intra prediction process
  • the second conversion module is configured to: in an image of a target area in an original unprocessed video frame image in a video encoding process, and in a video frame image obtained by a reconstruction process corresponding to the original unprocessed video frame image Converting, by the fixed-point type, the data type of the image of the associated area corresponding to the target area, and obtaining the converted video frame data;
  • the input module is configured to input the converted video frame data into a neural network loaded with the converted weight parameter, perform encoding intra prediction processing, and obtain intra prediction images and intra prediction related information.
  • the target processing is an encoding inter prediction process
  • the second conversion module is configured to convert a data type of the original unprocessed video frame image in the video encoding process and the de-distorted filtered reference frame image corresponding to the original unprocessed video frame image into a fixed point Type, obtain converted video frame data;
  • the input module is configured to input the converted video frame data into a neural network loaded with the converted weight parameter, perform encoding inter prediction processing, and obtain inter prediction images and inter prediction related information.
  • the target processing is an entropy encoding process
  • the second conversion module is configured to convert the data types of the intra prediction related information, the inter prediction related information, and the quantized coefficients obtained in the video encoding process into a fixed point type, to obtain converted video frame data;
  • the input module is configured to input the converted video frame data into a neural network loaded with the converted weight parameter, and perform entropy coding processing to obtain entropy coding information.
  • the target processing is an entropy decoding process
  • the second conversion module is configured to convert a data type of the entropy coding information acquired in the video decoding process into a fixed point type, to obtain converted video frame data;
  • the input module is configured to input the converted video frame data into a neural network loaded with the converted weight parameter, perform entropy decoding processing, and obtain intra prediction related information, inter prediction related information, and quantized coefficients.
  • the target processing is decoding intra prediction processing
  • the second conversion module is configured to convert the data type of the image of the associated region and the intra prediction related information corresponding to the target region in the video frame image obtained by the reconstruction process in the video decoding process into a fixed point type, and obtain the converted Video frame data;
  • the input module is configured to input the converted video frame data into a neural network loaded with the converted weight parameter, perform decoding intra prediction processing, and obtain an intra prediction image of the target area.
  • the target processing is decoding inter prediction processing
  • the second conversion module is configured to convert the data type of the reference frame image and the inter prediction related information after the de-distortion filtering process in the video decoding process into a fixed-point type, to obtain the converted video frame data;
  • the input module is configured to input the converted video frame data into a neural network loaded with the converted weight parameter, and perform decoding inter prediction processing to obtain an inter prediction image.
  • a terminal including a processor, a communication interface, a memory, and a communication bus, wherein:
  • the processor, the communication interface, and the memory complete communication with each other through the communication bus;
  • the memory is configured to store a computer program
  • the processor is configured to execute a program stored on the memory to implement the foregoing method for processing video frame data.
  • a computer readable storage medium having stored therein a computer program that, when executed by a processor, implements the processing of video frame data as described above method.
  • the data type of the weight parameter in the pre-trained neural network is converted into a fixed-point type by the method provided by the embodiment of the present disclosure, and the converted weight parameter is obtained; and the data type of the video frame data to be subjected to the target processing is converted into a fixed-point type. And obtaining the converted video frame data; inputting the converted video frame data into the neural network loaded with the converted weight parameter to obtain the target processed video frame data.
  • the floating-point data is converted into fixed-point data, and the fixed-point data has a fixed decimal point position, and there is no need to constrain the results in the operation process, and there is no case where the same data is subjected to the same operation but different results occur.
  • the codec operation results are consistent, and the decoder can decode normally.
  • FIG. 1 is a schematic structural diagram of an encoding end of a video codec system according to an exemplary embodiment
  • FIG. 2 is a schematic structural diagram of a decoding end of a video codec system according to an exemplary embodiment
  • FIG. 3 is a flow chart showing a method of processing video frame data according to an exemplary embodiment
  • FIG. 4 is a flow chart showing a method for processing video frame data according to an exemplary embodiment
  • FIG. 5 is a schematic diagram of a neuron in a convolutional neural network, according to an exemplary embodiment
  • FIG. 6 is a flowchart illustrating a method of processing video frame data according to an exemplary embodiment
  • FIG. 7 is a flowchart illustrating a method of processing video frame data according to an exemplary embodiment
  • FIG. 8 is a flowchart showing a method of processing video frame data according to an exemplary embodiment
  • FIG. 9 is a flowchart illustrating a method of processing video frame data according to an exemplary embodiment
  • FIG. 10 is a flowchart showing a method of processing video frame data according to an exemplary embodiment
  • FIG. 11 is a flowchart diagram of a method for processing video frame data according to an exemplary embodiment
  • FIG. 12 is a schematic structural diagram of an apparatus for processing video frame data according to an exemplary embodiment
  • FIG. 13 is a schematic structural diagram of a terminal according to an exemplary embodiment.
  • Embodiments of the present disclosure provide a method of processing video frame data, which may be implemented by a terminal.
  • the terminal may be a set top box, a tablet computer, a desktop computer, a notebook computer, or the like.
  • the terminal can include components such as a processor, a memory, and the like.
  • the processor may be a CPU (Central Processing Unit) or the like, and may be used to convert a data type of a weight parameter in a pre-trained neural network into a fixed-point type, and the like.
  • the memory may be a RAM (Random Access Memory), a Flash (flash memory), etc., and may be used to store received data, data required for processing, data generated during processing, and the like, such as video frame data. Wait.
  • the terminal may also include a transceiver, an input component, a display component, an audio output component, and the like.
  • the transceiver can be used for data transmission with a server, and the transceiver can include a Bluetooth component, a WiFi (Wireless-Fidelity) component, an antenna, a matching circuit, a modem, and the like.
  • the input component can be a touch screen, a keyboard, a mouse, or the like.
  • the audio output unit can be a speaker, a headphone, or the like.
  • a method for processing video frame data provided by this embodiment can be applied to a video codec system.
  • the video codec mainly includes an encoding end and a decoding end.
  • the encoding end may include a coding intra prediction module, an encoding inter prediction module, a transform module, a quantization module, an entropy encoder, an inverse quantization module, an inverse transform module, a reconstruction module, and a filtering module. , reference image buffer, etc.
  • the coding intra prediction module and the coding inter prediction module may respectively determine an intra prediction image, an intra prediction related information, an inter prediction image, and a video frame image obtained by performing reconstruction processing in a video encoding and decoding process, Inter prediction related information.
  • a switch coupled to the coded intra prediction module and the coded inter prediction module is configured to select to use an encoded intra prediction module or a coded inter prediction module, and the selected module provides an intra prediction image or an inter prediction image to the adder.
  • the intra prediction image or the inter prediction image passes through the adder to obtain a prediction residual.
  • the prediction residual is transformed and quantized to become a quantized coefficient.
  • the quantized coefficients, the intra prediction related information, the inter prediction related information, and the side information of the preset video frame image are input to an entropy encoder for entropy encoding, and finally the code stream is output.
  • the side information may be a quantized coefficient used in the quantization process, and the quantized coefficient may be set by a user or may be calculated.
  • the basic unit corresponding to the side information may be a video frame image or an image block into which the video frame image is divided. If the encoding side uses side information, then the side stream information is also carried in the code stream, so that the decoding end can decode normally based on the side information.
  • the coded inter prediction module When the coded inter prediction module is used, it is necessary to acquire a reference frame image, that is, a de-distorted video frame image, and the reference frame image can be stored in the reference image buffer. Specifically, the quantized coefficients may be inverse quantized and inverse transformed to recover the prediction residual. In the reconstruction module, the prediction residual is added back to the corresponding intra prediction image or inter prediction image to obtain a distorted video frame image. The distorted video frame image is processed by de-distortion filtering and converted into a reference frame image.
  • the decoding end may include a decoding intra prediction module, a decoding inter prediction module, an entropy decoder, an inverse quantization module, an inverse transform module, a reconstruction module, a filtering module, a reference image buffer, Video playback buffer, etc.
  • a video can be encoded by the encoding end to obtain a code stream, and the code stream can be restored to a distorted video at the decoding end.
  • the decoding process is also required at the encoding end, because the video frame image can be restored after the decoding process, and the restored video frame image is used as a reference image, thereby Perform motion compensation and other operations. Since the restored video frame image is distorted, the restored video frame image can be filtered by the trained neural network to obtain a de-distorted video frame image, and the image can be processed by using the method provided in this embodiment. .
  • the coded intra prediction module, the coded inter prediction module, the entropy encoder, the entropy decoder, the decoded intra prediction module, and the decoded inter prediction module in the video codec system may respectively apply the respective trained neural networks. Encoding intra prediction, coding inter prediction, entropy coding, entropy decoding, decoding intra prediction, and decoding inter prediction processing are performed. Since the image or the data is processed using the neural network in the process of performing the corresponding processing, the image or the data can be processed through the neural network using the method provided in the embodiment.
  • modules in the video codec system such as a transform module, a quantization module, an inverse transform module, and an inverse quantization module, perform quantization, transform, back-conversion, and inverse quantization processing, if it involves processing a image or data using a neural network. All of the images or data can be processed by the neural network using the method provided in this embodiment.
  • a combination of two or more modules connected in series for example, a combination of a transform module and a quantization module, performs transform quantization processing, if it involves processing a image or data using a neural network
  • the present embodiment may also be used.
  • the entire encoding end or the decoding end can use a neural network to directly encode and decode the video data. Since this case also involves processing the image or data using the neural network, the image or data can also be processed by the neural network using the method provided by the embodiment.
  • An exemplary embodiment of the present disclosure provides a method for processing video frame data. As shown in FIG. 3, the processing flow of the method may include the following steps:
  • Step S310 converting the data type of the weight parameter in the pre-trained neural network into a fixed-point type, and obtaining the converted weight parameter.
  • the neural network is an algorithm model for performing target processing on video frame data in a video codec process.
  • the neural network may be a convolutional neural network, a cyclic neural network, an anti-generation network, a self-encoder, a deep neural network, or the like.
  • the weight parameter can be a parameter obtained by training during training. In the process of neural network, such as convolutional neural network training, since the floating-point data is continuous, the partial derivative can be obtained, and the fixed-point data is non-continuous, and the partial derivative cannot be directly obtained, so the trained neural network is The data type of the weight parameter is floating point.
  • the fixed-point type data can be a normal fixed point number or a dynamic fixed point number.
  • the neural network is a convolutional neural network
  • the weight parameters include a convolution kernel element and an offset.
  • Step S320 converting the data type of the video frame data to be subjected to the target processing into a fixed point type, and obtaining the converted video frame data.
  • the video frame data to be subjected to the target processing may include an original video frame image or data obtained by processing the original video frame image.
  • the video frame data to be subjected to the target processing may be a reconstructed video frame image. If the method provided by this embodiment is used in an inter prediction or intra prediction module, the video frame data to be subjected to the target processing may be the original video frame image. If the method provided in this embodiment is used in an entropy coder, the video frame data to be subjected to the target processing may be data obtained after the original video frame image is subjected to prediction, transformation, quantization, and the like.
  • the video frame data to be processed by the target network needs to be input to the converted neural network, and the video frame data to be subjected to the target processing and the weighted parameters of the fixed-point type in the converted neural network are calculated in the neural network, but the target is to be performed.
  • the data type of the processed video frame data is integer or floating point type, and the fixed point type data cannot be directly operated with the integer type and floating point type data. Therefore, it is necessary to convert the data type of the video frame data to be subjected to the target processing into a fixed point type.
  • Step S330 the converted video frame data is input into the neural network loaded with the converted weight parameter to obtain the target processed video frame data.
  • the converted video frame image may be segmented and encoded after the converted video frame image is input into the transformed neural network image, and the converted video frame image is sliced into block image blocks, and the block image is segmented. The block is input into the transformed neural network.
  • the method provided in this embodiment further includes: converting a data type of the side information of the preset video frame data into a fixed point type to obtain the converted side information; and step S330 may include: converting the converted video frame data. And the converted side information, input the neural network loaded with the converted weight parameter, and obtain the fixed-point target processed video frame data.
  • the side information may be a quantized coefficient used in the quantization process, and the quantized coefficient may be set by a user or may be calculated.
  • the side information corresponds to a video frame image, or an image block into which a video frame image is divided.
  • the code stream also carries side information, so that the decoding end can decode normally based on the side information.
  • the data type of the weight parameter in the pre-trained neural network is converted into a fixed-point type by the method provided by the embodiment of the present disclosure, and the converted weight parameter is obtained; and the data type of the video frame data to be subjected to the target processing is converted into a fixed-point type. And obtaining the converted video frame data; inputting the converted video frame data into the neural network loaded with the converted weight parameter to obtain the target processed video frame data.
  • the floating-point data is converted into fixed-point data, and the fixed-point data has a fixed decimal point position, and there is no need to constrain the results in the operation process, and there is no case where the same data is subjected to the same operation but different results occur.
  • the codec operation results are consistent, and the decoder can decode normally.
  • An exemplary embodiment of the present disclosure provides a method for processing video frame data. As shown in FIG. 4, the processing flow of the method may include the following steps:
  • Step S410 converting the data type of the weight parameter in the pre-trained neural network into a fixed-point type, and obtaining the converted weight parameter.
  • neural network is an algorithm model that imitates the behavior characteristics of animal neural networks and performs distributed parallel information processing.
  • This algorithm model relies on the complexity of the entire algorithm model, and adjusts the relationship between a large number of nodes within the algorithm model to achieve the purpose of processing images.
  • the neural network of this embodiment is an algorithm model for performing de-distortion filtering processing on video frame data in a video codec process.
  • the function of the neural network is to process the image, it is inevitable to input the image into the neural network, calculate the weight parameters in the image and the neural network, and output the processed image.
  • the data type of the image and the weight parameter need to be unified, for example, unified into a fixed point type, and the unified different data can be operated.
  • the fixed-point type data can be a normal fixed point number or a dynamic fixed point number.
  • the position of the decimal point of any data in the convention is fixed.
  • the position of the decimal point is not indicated in the processor, but is pre-agreed, that is, once the position of the decimal point is determined, In general, the position of the decimal point is no longer changed.
  • the sign bit and magnitude can be used to represent the normal fixed point number. Assuming that the data bit width of the normal fixed point number is n bits, the sign bit occupies 1 bit, and the magnitude accounts for n-1 bits. Since the position of the decimal point of the ordinary fixed point number does not change, it is not necessary to take an approximation to constrain it to the specified precision, thereby avoiding the operation of the same value due to the different constraints of the different processors on the precision.
  • n represents the data bit width of the fixed point number
  • FL represents the length of the fractional part
  • x i is the data of the i th bit of the mantissa portion.
  • a large number of video frame images with distortions of different degrees of distortion and corresponding undistorted video frame images can be used to perform multiple iterations of the neural network to obtain a video frame image that can be distorted.
  • a neural network for distortion processing The specific training methods are as follows:
  • the neural network is a convolutional neural network
  • the weight parameters include a convolution kernel element and an offset.
  • the convolutional neural network consists of neurons. As shown in Figure 5, it is a schematic diagram of a neuron. Where X 1 to X 4 , +1 are inputs, w 1 to w 4 are convolution kernel elements, may be matrices, b is an offset, f is an activation function, and y is an output.
  • the distinguishing feature of convolutional neural networks from other neural networks is that weight parameters can be shared. Compared with other neural networks, the space for storing weight parameters can be saved, and the number of weight parameters that need to be trained is also reduced.
  • step S410 may include: determining, for each convolution kernel in the pre-trained convolutional neural network, a convolution kernel element having the largest absolute value in the convolution kernel; for multiple offsets in the convolutional neural network Determining the offset with the largest absolute value among the plurality of offsets; each volume is based on the convolution kernel element of the largest absolute value in each convolution kernel and the data bit width of the predetermined fixed-point convolution kernel element
  • the data type of the convolution kernel element in the product core is converted to a fixed-point type, and the plurality of offset data are based on the offset of the largest absolute value among the plurality of offsets and the data width of the offset of the preset fixed-point type.
  • the type is converted to a fixed-point type, and the converted weight parameter is obtained.
  • W j ij and b ij are used to represent the jth convolution kernel of the i-th layer and the j-th bias of the i-th layer, respectively, in the convolutional neural network.
  • i 1, 2, ..., N.
  • j 1, 2, ..., C i .
  • N is the number of layers of the convolutional neural network that do not contain the input layer.
  • C i is the number of convolution kernels of the i-th layer.
  • the following operations are mainly for rounding the convolution kernel elements or offsets, so that in the subsequent calculation process, only the sign bit and the mantissa part of the dynamic fixed point number can be operated, and the fractional part is not calculated.
  • the operation of the sign bit and the mantissa part of the dynamic fixed-point number is obviously an integer operation, which is much more convenient than the operation with a decimal point, and only records the value of the fractional part, that is, the value of FL, regardless of how the integer operation is performed in the middle. Finally, you can divide it by 2 -FL to map it back to the actual value.
  • the data bit width of the i-th layer convolution kernel element of the set point type is Then there are:
  • ) represents the value of the convolution kernel element having the largest absolute value in the convolution kernel W ij . Symbol at both ends of the log Indicates that the data in the symbol is rounded down.
  • Equation 3 The purpose of the calculation of Equation 3 is to be all in the i-th layer. Choose one target This goal Make the other in the i-th layer Greater than and less than the target The number is relatively average, even equal, will target As
  • W i f For fixed-point types W i and B i , respectively, W i f can be expressed as:
  • round () is a rounding operation.
  • a rounding operation is also performed. Whether it is a rounding operation or a rounding operation, the purpose is to operate only on the sign and mantissa parts of the dynamic fixed point number. This part of the operation is obviously an integer operation, which is much more convenient than performing operations with a decimal point. As long as the value of the fractional part, that is, the value of FL, is recorded, regardless of how the integer operation is performed in the middle, it can still be divided by 2 -FL to map it back to the actual value. Therefore, the rounding below is to record only the fractional part of the dynamic fixed-point number and not to perform the operation.
  • the data types of the image and the weight parameter are unified, such as a unified method.
  • the method of converting the data type of the weight parameter to the fixed point type is introduced in detail. The following describes how to convert the data type of an image to a fixed-point type.
  • Step S420 converting the data type of the video frame image obtained by the reconstruction process in the video encoding and decoding process to a fixed-point type, and obtaining the converted video frame data.
  • the video frame image obtained by the reconstruction process in the video codec process It is necessary to input the video frame image obtained by the reconstruction process in the video codec process to the converted neural network, the video frame image obtained by the reconstruction process in the video codec process, and the fixed-point weight parameter in the converted neural network.
  • the operation is performed.
  • the data type of the video frame image obtained by the reconstruction process in the video encoding and decoding process is integer or floating point type
  • the fixed-point type data cannot be directly operated with the integer type and floating point type data. Therefore, it is necessary to convert the data type of the video frame image obtained by the reconstruction process in the video codec process into a fixed point type.
  • step S420 may include: according to the data bit width of the preset fixed-point type video frame data, and the data with the largest absolute value among the feature data outputted by the input layer of the pre-statistical convolutional neural network, the target to be performed is to be performed.
  • the data type of the processed video frame data is converted to a fixed point type, and the converted video frame data is obtained.
  • FL 0 is a parameter required in the process of converting the data type of the pixel value in the image frame image to be processed into a fixed-point type, and this parameter can be obtained by calculation:
  • a convolutional neural network which may be a convolutional neural network that has not been converted, or a convolutional neural network after conversion.
  • the pixel value of the pixel value of the feature image output by the input layer may be counted as the largest pixel value, Then there are:
  • the data bit width of the pixel value of the input layer for the fixed point type is the data bit width of the pixel value of the input layer for the fixed point type.
  • a parameter FL i is further calculated, which is used in the subsequent process of the fixed-point operation.
  • S is input into the convolutional neural network
  • the pixel value of the pixel value of the feature image outputted by each layer of the hidden layer can be counted, and the pixel value of the feature image output by the i-th layer hidden layer is absolutely The pixel value with the largest value is written as Then there are:
  • the convolutional neural network contains a total of N-1 hidden layers.
  • the data type of the pixel value in the video frame image to be processed can be converted into a fixed point type based on FL 0 , and specifically, the data type of the pixel value in the video frame image to be processed by Equation 13 can be converted.
  • For fixed-point type get I f :
  • I is an integer or floating point pixel value in a video frame image to be processed.
  • the method of converting the data type of the video frame image obtained by the reconstruction process in the video codec process into a fixed-point type is described above, that is, the non-fixed type I is converted into I f .
  • the parameter FL 0 needs to be determined, so the process of determining FL 0 is also introduced.
  • Step S430 the converted video frame data is input into a neural network loaded with the converted weight parameter, and subjected to de-distortion filtering processing to obtain a de-distorted video frame image.
  • the converted video frame data may be a transformed distortion video frame image, and the pixel value in the original distorted video frame image is non-fixed, and the pixel value in the converted distorted video frame image is fixed-point type. .
  • the converted distorted video frame image is input to a neural network for performing de-distortion filtering processing, and the de-distorted video frame image can be output, thereby performing de-distortion filtering processing on the distorted video frame image.
  • the converted distortion video frame image is segmented and encoded, and the entire replaced video frame image is sliced into blocks.
  • Block which inputs a block of image blocks into the transformed neural network.
  • the video frame image will be described below instead of a block of image blocks.
  • the convolution kernel element, the offset, and the converted video frame data in the fixed-point convolutional neural network can be operated by a fixed-point operation.
  • the specific algorithm is as follows:
  • the output data of the upper layer can be used as the input data of the current layer, and it is operated with the fixed-point convolution kernel element and the offset of the current layer to obtain the feature image F'' i (I f ).
  • i be the number of layers in the current layer
  • i-1 be the number of layers in the previous layer.
  • FL i-1 can be calculated by the formula 10.
  • Equation 14 represents the form of the sign bit and the mantissa part of the dynamic fixed point number, and the actual values are finally determined based on their fractional parts. That is, only the numerator is operated, only the size of the denominator is recorded, and the actual value is obtained by dividing by the denominator at the end. In this way, the operation of the numerator is an integer type operation, which is relatively simple and convenient.
  • Operation is to convert the magnitude of the fractional part B i with the same order of magnitude W i f * F i-1 (I f) is carried out.
  • F'' i (I f ) can be quantized. Since the data of the fixed-point type can be limited, it is possible to quantize F'' i (I f ) in order to prevent overflow during the operation.
  • the quantized F'' i (I f ) is denoted as F' i (I f ), then:
  • input F' i (I f ) into the activation function, and the activation function can be a nonlinear activation function.
  • a fixed-point operation is required.
  • the specific fixed-point operation refer to the above description to ensure that the de-distorted video frame image F N- is obtained after the operation. 1 (I f ).
  • the data type of the pixel value in the output video frame image is a fixed-point type.
  • the operation process such as motion compensation, it is necessary to use an integer pixel value, so it is necessary to convert the data type of the pixel value in the output video frame image into an integer type, or convert the fixed-point type target processed video frame data as needed.
  • the video frame data of the fixed-point target processing can be first converted into a floating point type, and then converted from a floating point type to a target type.
  • the method provided in this embodiment may further include: converting a data type of the target processed video frame data into a floating point type to obtain floating-point type video frame data.
  • Converting the data type of the target processed video frame data to a floating point type is a one-step intermediate medium step.
  • the data type of the data that the module after the neural network needs to receive is different, and the data type of the data output by the neural network needs to be converted into the data type of the data that the module needs to receive after the neural network.
  • the neural network and the modules behind the neural network can be seamlessly connected, that is, the data output by the neural network can be subsequently processed by the module behind the neural network.
  • the pixel value in the output video frame image of the integer is O.
  • the method provided by the embodiment of the present disclosure may further include:
  • the video frame data after the target processing is rounded to obtain an integer video frame data.
  • the convolutional neural network is applied to the filtering module, the encoding/decoding intra prediction module, and the encoding/decoding inter prediction module, it is necessary to replace the floating-point video frame data with the integer video frame data, that is, the integer type.
  • the pixel value constitute the video frame image.
  • the feature image output by the last layer of the hidden layer also needs the convolution kernel element of the output layer.
  • the offset B N performs a fixed-point operation to obtain a pixel value O′′ in the output video frame image of the fixed-point type, and there are:
  • the pixel value O" in the fixed-point output video frame image is converted into the pixel value O' in the floating-point output video frame image.
  • the pixel value O' in the floating-point output video frame image can be converted to the pixel value O in the integer output video frame image.
  • the data type of the weight parameter in the pre-trained neural network is converted into a fixed-point type by the method provided by the embodiment of the present disclosure, and the converted weight parameter is obtained; the video frame image obtained by the reconstruction process in the video encoding and decoding process is obtained.
  • the data type is converted into a fixed-point type, and the converted video frame data is obtained; the converted video frame data is input into a neural network loaded with the converted weight parameter, and de-distortion filtering processing is performed to obtain a de-distorted video frame image.
  • the floating-point data is converted into fixed-point data, and the fixed-point data has a fixed decimal point position, and there is no need to constrain the results in the operation process, and there is no case where the same data is subjected to the same operation but different results occur. Furthermore, the codec operation results are consistent, and the decoder can decode normally.
  • An exemplary embodiment of the present disclosure provides a method for processing video frame data. As shown in FIG. 6, the processing flow of the method may include the following steps:
  • Step S610 converting the data type of the weight parameter in the pre-trained neural network into a fixed-point type, and obtaining the converted weight parameter.
  • the neural network of this embodiment is an algorithm model for encoding intra frame prediction processing of video frame data in a video encoding and decoding process.
  • the method of converting the data type of the weight parameter to the fixed point type refer to the conversion method provided in the embodiment corresponding to step S410 to step S430.
  • step S610 may include: determining, for each convolution kernel in the pre-trained convolutional neural network, a convolution kernel element having the largest absolute value in the convolution kernel; for multiple offsets in the convolutional neural network Determining the offset with the largest absolute value among the plurality of offsets; each volume is based on the convolution kernel element of the largest absolute value in each convolution kernel and the data bit width of the predetermined fixed-point convolution kernel element
  • the data type of the convolution kernel element in the product core is converted to a fixed-point type, and the plurality of offset data are based on the offset of the largest absolute value among the plurality of offsets and the data width of the offset of the preset fixed-point type.
  • the type is converted to a fixed-point type, and the converted weight parameter is obtained.
  • Step S620 the image of the target area in the original unprocessed video frame image in the video encoding process, and the associated area corresponding to the target area in the video frame image obtained by the reconstruction process corresponding to the original unprocessed video frame image.
  • the data type of the image is converted to a fixed-point type, and the converted video frame data is obtained.
  • the original unprocessed video frame image can be taken by the video capture device.
  • the original unprocessed video frame image may be divided into a preset number of regions, and an region adjacent to the image of the target region may serve as an associated region corresponding to the target region. Since adjacent pixels or regions in one image have similarities, after the image of the associated region corresponding to the target region is known, the image of the target region can be restored based on other information.
  • the method of converting the data type of the image to the fixed point type refer to the conversion method provided in the embodiment corresponding to step S410 to step S430.
  • the step S620 may include: according to the data bit width of the preset fixed-point type video frame data, and the data with the largest absolute value among the feature data outputted by the input layer of the pre-statistical convolutional neural network, the target to be performed is to be performed.
  • the data type of the processed video frame data is converted to a fixed point type, and the converted video frame data is obtained.
  • Step S630 the converted video frame data is input into a neural network loaded with the converted weight parameter, and the encoded intra prediction process is performed to obtain an intra prediction image and intra prediction related information.
  • the intra prediction related information may be information that can recover the image of the target area based on the information and the image of the associated area corresponding to the target area.
  • the intra prediction image may be an image in which the image of the target area is restored based on the intra prediction related information and the image of the associated area corresponding to the target area.
  • the restored image is compressed after all, so it is impossible to completely match the original image, that is, the image of the target area. Therefore, the image of the target area can be predicted to obtain an intra prediction image, and the intra prediction image and the image of the target area can be compared to obtain a prediction residual, that is, difference information between the restored image and the original image.
  • the coded intra prediction module may output an intra prediction image to the adder, and may output intra prediction related information to the entropy encoder.
  • the process of using the neural network to perform the intra-frame prediction processing is similar to the process of the de-distortion filtering process.
  • the details refer to the description of the embodiment of the de-distortion filtering process, and details are not described herein again.
  • the data type of the weight parameter in the pre-trained neural network is converted into a fixed-point type by the method provided by the embodiment of the present disclosure, and the converted weight parameter is obtained;
  • the target in the original unprocessed video frame image in the video encoding process The image of the region and the data type of the image of the associated region corresponding to the target region in the video frame image obtained by the reconstruction process corresponding to the original unprocessed video frame image are converted into a fixed-point type, and the converted video frame data is obtained;
  • the subsequent video frame data is input to a neural network loaded with the converted weight parameter, and the encoded intra prediction process is performed to obtain an intra prediction image and intra prediction related information.
  • the floating-point data is converted into fixed-point data, and the fixed-point data has a fixed decimal point position, and there is no need to constrain the results in the operation process, and there is no case where the same data is subjected to the same operation but different results occur. Furthermore, the codec operation results are consistent, and the decoder can decode normally.
  • An exemplary embodiment of the present disclosure provides a method for processing video frame data. As shown in FIG. 7, the processing flow of the method may include the following steps:
  • Step S710 converting the data type of the weight parameter in the pre-trained neural network into a fixed-point type, and obtaining the converted weight parameter.
  • the neural network in this embodiment is an algorithm model for performing inter-frame prediction processing on video frame data in a video encoding and decoding process.
  • the method of converting the data type of the weight parameter to the fixed point type refer to the conversion method provided in the embodiment corresponding to step S410 to step S430.
  • step S710 may include: determining, for each convolution kernel in the pre-trained convolutional neural network, a convolution kernel element having the largest absolute value in the convolution kernel; for multiple offsets in the convolutional neural network Determining the offset with the largest absolute value among the plurality of offsets; each volume is based on the convolution kernel element of the largest absolute value in each convolution kernel and the data bit width of the predetermined fixed-point convolution kernel element
  • the data type of the convolution kernel element in the product core is converted to a fixed-point type, and the plurality of offset data are based on the offset of the largest absolute value among the plurality of offsets and the data width of the offset of the preset fixed-point type.
  • the type is converted to a fixed-point type, and the converted weight parameter is obtained.
  • Step S720 converting the original unprocessed video frame image in the video encoding process and the data type of the de-distorted filtered reference frame image corresponding to the original unprocessed video frame image into a fixed-point type, and obtaining the converted video frame. data.
  • the de-distortion filtering module inputs the de-distortion filtered processed reference frame image corresponding to the original unprocessed video frame image to the encoding inter prediction module. Since the adjacent images have similarities, after learning the reference frame image, based on other information, the original unprocessed video frame image corresponding to the reference frame image can be recovered.
  • the conversion method provided in the embodiment corresponding to step S410 to step S430 refer to the conversion method provided in the embodiment corresponding to step S410 to step S430.
  • step S720 may include: according to the data bit width of the preset fixed-point type video frame data, and the data with the largest absolute value among the feature data output by the input layer of the pre-statistical convolutional neural network, the target to be performed is to be performed.
  • the data type of the processed video frame data is converted to a fixed point type, and the converted video frame data is obtained.
  • Step S730 the converted video frame data is input into a neural network loaded with the converted weight parameter, and the inter-frame prediction process is performed to obtain an inter-prediction image and inter-frame prediction related information.
  • the inter prediction related information may be a motion vector in the motion compensation, that is, the original unprocessed video frame image corresponding to the reference frame image may be obtained by the displacement of the reference frame image.
  • the inter prediction image may be an image that is recovered based on the inter prediction related information and the reference frame image. However, the restored image is compressed after all, so it is impossible to completely match the original image, that is, the reference frame image. Therefore, the reference frame image can be predicted to obtain an inter prediction image, and the inter-predicted image and the original unprocessed video frame image corresponding to the reference frame image are compared to obtain a prediction residual, that is, the restored image and the original image.
  • the difference information of the image is a motion vector in the motion compensation, that is, the original unprocessed video frame image corresponding to the reference frame image may be obtained by the displacement of the reference frame image.
  • the inter prediction image may be an image that is recovered based on the inter prediction related information and the reference frame image. However, the restored image is compressed after all, so it is
  • the inter prediction image and the inter prediction related information are input to an adder and an entropy encoder, respectively.
  • the process of using the neural network to perform the inter-frame prediction process is similar to the process of the de-distortion filtering process.
  • the data type of the weight parameter in the pre-trained neural network is converted into a fixed-point type, and the converted weight parameter is obtained; the original unprocessed video frame image and the original will be in the video encoding process.
  • the data type of the de-distortion filter-processed reference frame image corresponding to the unprocessed video frame image is converted into a fixed-point type, and the converted video frame data is obtained; and the converted video frame data is input into the nerve loaded with the converted weight parameter.
  • the network performs coding inter prediction processing to obtain inter prediction images and inter prediction related information.
  • the floating-point data is converted into fixed-point data, and the fixed-point data has a fixed decimal point position, and there is no need to constrain the results in the operation process, and there is no case where the same data is subjected to the same operation but different results occur. Furthermore, the codec operation results are consistent, and the decoder can decode normally.
  • An exemplary embodiment of the present disclosure provides a method for processing video frame data. As shown in FIG. 8, the processing flow of the method may include the following steps:
  • Step S810 converting the data type of the weight parameter in the pre-trained neural network into a fixed-point type, and obtaining the converted weight parameter.
  • the neural network of this embodiment is an algorithm model for entropy encoding processing of video frame data in a video encoding and decoding process.
  • the method of converting the data type of the weight parameter to the fixed point type refer to the conversion method provided in the embodiment corresponding to step S410 to step S430.
  • step S810 may include: determining, for each convolution kernel in the pre-trained convolutional neural network, a convolution kernel element having the largest absolute value in the convolution kernel; for multiple offsets in the convolutional neural network Determining the offset with the largest absolute value among the plurality of offsets; each volume is based on the convolution kernel element of the largest absolute value in each convolution kernel and the data bit width of the predetermined fixed-point convolution kernel element
  • the data type of the convolution kernel element in the product core is converted to a fixed-point type, and the plurality of offset data are based on the offset of the largest absolute value among the plurality of offsets and the data width of the offset of the preset fixed-point type.
  • the type is converted to a fixed-point type, and the converted weight parameter is obtained.
  • Step S820 converting the data types of the intra prediction related information, the inter prediction related information, and the quantized coefficients obtained in the video encoding process into a fixed point type, to obtain the converted video frame data.
  • the quantization coefficient may be data output by the quantization module.
  • the coding intra prediction module, the coding inter prediction module, and the quantization module respectively input intra prediction related information, inter prediction related information, and quantization coefficients to the entropy encoder.
  • the method of converting the data type of the image to the fixed point type refer to the conversion method provided in the embodiment corresponding to step S410 to step S430.
  • the step S820 may include: according to the data bit width of the preset fixed-point type video frame data, and the data with the largest absolute value among the feature data output by the input layer of the pre-statistical convolutional neural network, the target to be performed is to be performed.
  • the data type of the processed video frame data is converted to a fixed point type, and the converted video frame data is obtained.
  • Step S830 the converted video frame data is input into a neural network loaded with the converted weight parameter, and entropy coding processing is performed to obtain entropy coding information.
  • the entropy encoded information is mapped to a code stream and output to the decoding end.
  • the method provided in this embodiment may further include: converting a data type of the target processed video frame data into a floating point type to obtain floating-point type video frame data.
  • the entropy encoding information may also be mapped to the code stream used for sending to the decoding end.
  • the correspondence between the range of the floating-point video frame data and the binary code stream may be pre-stored, and the target binary code corresponding to the range to which the floating-frame type video frame data belongs is pre-stored. flow.
  • Each floating point video frame data is mapped to a target binary code stream to obtain a code stream for transmission to the decoding end.
  • the process of using the neural network to perform the entropy coding process is similar to the process of the de-distortion filter process. For details, refer to the description of the embodiment of the de-distortion filter process, and details are not described herein again.
  • the data type of the weight parameter in the pre-trained neural network is converted into a fixed-point type by the method provided by the embodiment of the present disclosure, and the converted weight parameter is obtained; the intra-prediction related information and the inter-frame obtained in the video encoding process are obtained.
  • the data type of the prediction related information and the quantized coefficient is converted into a fixed point type, and the converted video frame data is obtained; the converted video frame data is input into a neural network loaded with the converted weight parameter, and entropy coding processing is performed to obtain entropy coding information. .
  • the floating-point data is converted into fixed-point data, and the fixed-point data has a fixed decimal point position, and there is no need to constrain the results in the operation process, and there is no case where the same data is subjected to the same operation but different results occur. Furthermore, the codec operation results are consistent, and the decoder can decode normally.
  • An exemplary embodiment of the present disclosure provides a method for processing video frame data. As shown in FIG. 9, the processing flow of the method may include the following steps:
  • Step S910 converting the data type of the weight parameter in the pre-trained neural network to a fixed-point type, and obtaining the converted weight parameter.
  • the neural network of this embodiment is an algorithm model for entropy decoding processing of video frame data in a video codec process.
  • For the method of converting the data type of the weight parameter to the fixed point type refer to the conversion method provided in the embodiment corresponding to step S410 to step S430.
  • step S910 may include: determining, for each convolution kernel in the pre-trained convolutional neural network, a convolution kernel element having the largest absolute value in the convolution kernel; for multiple offsets in the convolutional neural network Determining the offset with the largest absolute value among the plurality of offsets; each volume is based on the convolution kernel element of the largest absolute value in each convolution kernel and the data bit width of the predetermined fixed-point convolution kernel element
  • the data type of the convolution kernel element in the product core is converted to a fixed-point type, and the plurality of offset data are based on the offset of the largest absolute value among the plurality of offsets and the data width of the offset of the preset fixed-point type.
  • the type is converted to a fixed-point type, and the converted weight parameter is obtained.
  • Step S920 converting the data type of the entropy coding information acquired in the video decoding process to a fixed point type, and obtaining the converted video frame data.
  • the encoding end may input the entropy encoding information to the entropy decoder of the decoding end.
  • the method of converting the data type of the image to the fixed point type refer to the conversion method provided in the embodiment corresponding to step S410 to step S430.
  • step S920 may include: according to the data bit width of the preset fixed-point type video frame data, and the data with the largest absolute value among the feature data output by the input layer of the pre-statistical convolutional neural network, the target to be performed is to be performed.
  • the data type of the processed video frame data is converted to a fixed point type, and the converted video frame data is obtained.
  • Step S930 the converted video frame data is input into a neural network loaded with the converted weight parameter, and subjected to entropy decoding processing to obtain intra prediction related information, inter prediction related information, and quantized coefficients.
  • the entropy decoder may output intra prediction related information, inter prediction related information to the decoded intra prediction module, the decoded inter prediction module, and output the quantized coefficients to the inverse quantization module.
  • the process of performing the entropy decoding process using the neural network is similar to the process of the de-distortion filtering process. For details, refer to the description of the embodiment of the de-distortion filtering process, and details are not described herein again.
  • the data type of the weight parameter in the pre-trained neural network is converted into a fixed-point type by the method provided by the embodiment of the present disclosure, and the converted weight parameter is obtained; and the data type of the entropy encoded information acquired in the video decoding process is converted into
  • the fixed-point type obtains the converted video frame data; the converted video frame data is input into the neural network loaded with the converted weight parameter, and performs entropy decoding processing to obtain intra prediction related information, inter prediction related information, and quantization coefficient. .
  • the floating-point data is converted into fixed-point data, and the fixed-point data has a fixed decimal point position, and there is no need to constrain the results in the operation process, and there is no case where the same data is subjected to the same operation but different results occur. Furthermore, the codec operation results are consistent, and the decoder can decode normally.
  • An exemplary embodiment of the present disclosure provides a method for processing video frame data. As shown in FIG. 10, the processing procedure of the method may include the following steps:
  • step S1010 the data type of the weight parameter in the pre-trained neural network is converted into a fixed-point type, and the converted weight parameter is obtained.
  • the neural network of this embodiment is an algorithm model for performing decoding intra prediction processing on video frame data in a video encoding and decoding process.
  • the method of converting the data type of the weight parameter to the fixed point type refer to the conversion method provided in the embodiment corresponding to step S410 to step S430.
  • step S1010 may include: determining, for each convolution kernel in the pre-trained convolutional neural network, a convolution kernel element having the largest absolute value in the convolution kernel; for multiple offsets in the convolutional neural network Determining the offset with the largest absolute value among the plurality of offsets; each volume is based on the convolution kernel element of the largest absolute value in each convolution kernel and the data bit width of the predetermined fixed-point convolution kernel element
  • the data type of the convolution kernel element in the product core is converted to a fixed-point type, and the plurality of offset data are based on the offset of the largest absolute value among the plurality of offsets and the data width of the offset of the preset fixed-point type.
  • the type is converted to a fixed-point type, and the converted weight parameter is obtained.
  • Step S1020 Convert the data type of the image of the associated area and the intra prediction related information corresponding to the target area in the video frame image obtained by the reconstruction process in the video decoding process to a fixed point type, and obtain the converted video frame data.
  • the reconstruction module may input, to the decoded intra prediction module, an image of the associated region corresponding to the target region in the video frame image obtained by the reconstruction process.
  • the entropy decoder may input intra prediction related information to the decoded intra prediction module. For the method of converting the data type of the image to the fixed point type, refer to the conversion method provided in the embodiment corresponding to step S410 to step S430.
  • the step S1020 may include: according to the data bit width of the preset fixed-point type video frame data, and the data with the largest absolute value among the feature data output by the input layer of the pre-statistical convolutional neural network, the target to be performed is to be performed.
  • the data type of the processed video frame data is converted to a fixed point type, and the converted video frame data is obtained.
  • Step S1030 Input the converted video frame data into the neural network loaded with the converted weight parameter, perform decoding intra prediction processing, and obtain an intra prediction image of the target region.
  • the decoded intra prediction module may output an intra prediction image of the target area to the reconstruction module.
  • the process of using the neural network to perform the decoding of the intra prediction process is similar to the process of the de-distortion filtering process.
  • the data type of the weight parameter in the pre-trained neural network is converted into a fixed-point type by the method provided by the embodiment of the present disclosure, and the converted weight parameter is obtained; the video frame image obtained by the reconstruction process in the video decoding process and the target are obtained.
  • the data type of the image and the intra prediction related information corresponding to the region is converted into a fixed point type, and the converted video frame data is obtained; and the converted video frame data is input into a neural network loaded with the converted weight parameter for decoding.
  • the intra prediction process obtains an intra prediction image of the target area.
  • the floating-point data is converted into fixed-point data, and the fixed-point data has a fixed decimal point position, and there is no need to constrain the results in the operation process, and there is no case where the same data is subjected to the same operation but different results occur. Furthermore, the codec operation results are consistent, and the decoder can decode normally.
  • An exemplary embodiment of the present disclosure provides a method for processing video frame data. As shown in FIG. 11, the processing flow of the method may include the following steps:
  • step S1110 the data type of the weight parameter in the pre-trained neural network is converted into a fixed-point type, and the converted weight parameter is obtained.
  • the neural network of this embodiment is an algorithm model for performing inter-frame prediction processing on video frame data in a video codec process.
  • the method of converting the data type of the weight parameter to the fixed point type refer to the conversion method provided in the embodiment corresponding to step S410 to step S430.
  • step S1110 may include: determining, for each convolution kernel in the pre-trained convolutional neural network, a convolution kernel element having the largest absolute value in the convolution kernel; and a plurality of offsets in the convolutional neural network Determining the offset with the largest absolute value among the plurality of offsets; each volume is based on the convolution kernel element of the largest absolute value in each convolution kernel and the data bit width of the predetermined fixed-point convolution kernel element
  • the data type of the convolution kernel element in the product core is converted to a fixed-point type, and the plurality of offset data are based on the offset of the largest absolute value among the plurality of offsets and the data width of the offset of the preset fixed-point type.
  • the type is converted to a fixed-point type, and the converted weight parameter is obtained.
  • Step S1120 Convert the data type of the reference frame image and the inter prediction related information after the de-distortion filtering process in the video decoding process to a fixed-point type, and obtain the converted video frame data.
  • the filtering module may input the reference frame image after the de-distortion filtering process in the video decoding process to the decoding inter prediction module, and the entropy decoder may input the inter prediction related information to the decoding inter prediction module.
  • the method of converting the data type of the image to the fixed point type refer to the conversion method provided in the embodiment corresponding to step S410 to step S430.
  • step S1120 may include: according to the data bit width of the preset fixed-point type video frame data, and the data with the largest absolute value among the feature data output by the input layer of the pre-statistical convolutional neural network, the target to be performed is to be performed.
  • the data type of the processed video frame data is converted to a fixed point type, and the converted video frame data is obtained.
  • Step S1130 Input the converted video frame data into a neural network loaded with the converted weight parameter, and perform decoding inter prediction processing to obtain an inter prediction image.
  • the decoded inter prediction module may output an inter prediction image to the reconstruction module.
  • the process of using the neural network to perform the inter-frame prediction process is similar to the process of the de-distortion filtering process.
  • the data type of the weight parameter in the pre-trained neural network is converted into a fixed-point type by the method provided by the embodiment of the present disclosure, and the converted weight parameter is obtained; the reference frame image after the de-distortion filtering process in the video decoding process is The data type of the inter prediction related information is converted into a fixed point type, and the converted video frame data is obtained; the converted video frame data is input into a neural network loaded with the converted weight parameter, and the interframe prediction processing is performed to obtain an interframe. Predict the image.
  • the floating-point data is converted into fixed-point data, and the fixed-point data has a fixed decimal point position, and there is no need to constrain the results in the operation process, and there is no case where the same data is subjected to the same operation but different results occur. Furthermore, the codec operation results are consistent, and the decoder can decode normally.
  • the apparatus includes:
  • the first conversion module 1210 is configured to convert the data type of the weight parameter in the pre-trained neural network into a fixed-point type, where the converted weight parameter is obtained, where the neural network is used for video in the video encoding and decoding process.
  • a second conversion module 1220 configured to convert a data type of the video frame data to be subjected to the target processing into a fixed-point type, to obtain converted video frame data
  • the input module 1230 is configured to input the converted video frame data into the neural network loaded with the converted weight parameter to obtain the target processed video frame data.
  • the neural network is a convolutional neural network
  • the weight parameters include a convolution kernel element and an offset.
  • the first conversion module 1210 includes:
  • a first determining unit configured to determine, for each convolution kernel in the pre-trained convolutional neural network, a convolution kernel element having the largest absolute value in the convolution kernel
  • a second determining unit configured to determine, for the plurality of offsets in the convolutional neural network, an offset with an absolute maximum value among the plurality of offsets
  • a conversion unit for convolution kernel elements in each convolution kernel according to a convolution kernel element having the largest absolute value in each convolution kernel and a data bit width of a predetermined fixed-point convolution kernel element Converting the data type to a fixed-point type, converting the data types of the plurality of offsets to a fixed-point type according to the offset of the largest absolute value among the plurality of offsets, and the data width of the offset of the preset fixed-point type The converted weight parameter.
  • the second conversion module 1220 is configured to: according to a data bit width of the preset fixed-point type video frame data, and a pre-stated maximum value of the feature data output by the input layer of the convolutional neural network.
  • the data of the video frame data to be processed by the target is converted into a fixed-point type to obtain converted video frame data.
  • the device further includes:
  • a third conversion module configured to convert a data type of the side information of the preset video frame data into a fixed point type, to obtain converted side information
  • the input module is configured to input the converted video frame data and the converted side information into a neural network loaded with the converted weight parameter to obtain the target processed video frame data.
  • the device further includes:
  • the rounding module is configured to perform rounding processing on the video frame data after the target processing to obtain an integer video frame data.
  • the target processing is a de-distortion filtering process
  • the second conversion module 1220 is configured to convert a data type of a video frame image obtained by performing a reconstruction process in a video encoding and decoding process into a fixed-point type, to obtain converted video frame data;
  • the input module 1230 is configured to input the converted video frame data into a neural network loaded with the converted weight parameter, perform de-distortion filtering processing, and obtain a de-distorted video frame image.
  • the target processing is a coding intra prediction process
  • the second conversion module 1220 is configured to: image an image of a target area in an original unprocessed video frame image in a video encoding process, and a video frame image obtained by a reconstruction process corresponding to the original unprocessed video frame image
  • the data type of the image of the associated area corresponding to the target area is converted to a fixed point type, and the converted video frame data is obtained;
  • the input module 1230 is configured to input the converted video frame data into a neural network loaded with the converted weight parameter, perform encoding intra prediction processing, and obtain intra prediction images and intra prediction related information.
  • the target processing is an encoding inter prediction process
  • the second conversion module 1220 is configured to convert a data type of the original unprocessed video frame image in the video encoding process and the de-distorted filtered reference frame image corresponding to the original unprocessed video frame image into Fixed-point type, obtaining converted video frame data;
  • the input module 1230 is configured to input the converted video frame data into a neural network loaded with the converted weight parameter, perform encoding inter prediction processing, and obtain inter prediction images and inter prediction related information.
  • the target processing is an entropy encoding process
  • the second conversion module 1220 is configured to convert the data types of the intra prediction related information, the inter prediction related information, and the quantized coefficients obtained in the video encoding process into a fixed point type, to obtain converted video frame data;
  • the input module 1230 is configured to input the converted video frame data into a neural network loaded with the converted weight parameter, and perform entropy coding processing to obtain entropy coding information.
  • the target processing is an entropy decoding process
  • the second conversion module 1220 is configured to convert a data type of the entropy coding information acquired in the video decoding process into a fixed-point type, to obtain converted video frame data;
  • the input module 1230 is configured to input the converted video frame data into a neural network loaded with the converted weight parameter, perform entropy decoding processing, and obtain intra prediction related information, inter prediction related information, and quantized coefficients.
  • the target processing is decoding intra prediction processing
  • the second conversion module 1220 is configured to convert the data type of the image of the associated area and the intra prediction related information corresponding to the target area in the video frame image obtained by the reconstruction process in the video decoding process into a fixed point type, and obtain the converted Video frame data;
  • the input module 1230 is configured to input the converted video frame data into a neural network loaded with the converted weight parameter, perform decoding intra prediction processing, and obtain an intra prediction image of the target area.
  • the target processing is decoding inter prediction processing
  • the second conversion module 1220 is configured to convert a data type of the reference frame image and the inter prediction related information after the de-distortion filtering process in the video decoding process into a fixed-point type, to obtain converted video frame data;
  • the input module 1230 is configured to input the converted video frame data into a neural network loaded with the converted weight parameter, and perform decoding inter prediction processing to obtain an inter prediction image.
  • the fixed-point data has a fixed decimal point position, and there is no need to constrain the results in the operation process, and there will be no different results for the same data but different results. Furthermore, the codec operation results are consistent, and the decoder can decode normally.
  • the apparatus for processing the video frame data is only illustrated by the division of the foregoing functional modules. In actual applications, the foregoing functions may be performed as needed. The allocation is done by different functional modules, that is, the internal structure of the terminal is divided into different functional modules to complete all or part of the functions described above.
  • the device for processing the video frame data provided by the foregoing embodiment is the same as the method for processing the video frame data. For the specific implementation process, refer to the method embodiment, and details are not described herein again.
  • FIG. 13 is a schematic structural diagram of a terminal 1800 according to an exemplary embodiment of the present disclosure.
  • the terminal 1800 can be: a set top box, a smart phone, a tablet computer, an MP4 (Moving Picture Experts Group Audio Layer IV) player, a notebook computer or a desktop computer.
  • Terminal 1800 may also be referred to as a user device, a portable terminal, a laptop terminal, a desktop terminal, and the like.
  • the terminal 1800 includes a processor 1801 and a memory 1802.
  • the processor 1801 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like.
  • the processor 1801 may be configured by at least one of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), and a PLA (Programmable Logic Array). achieve.
  • the processor 1801 may also include a main processor and a coprocessor.
  • the main processor is a processor for processing data in an awake state, which is also called a CPU (Central Processing Unit); the coprocessor is A low-power processor for processing data in standby.
  • the processor 1801 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and rendering of the content that the display needs to display.
  • the processor 1801 may further include an AI (Artificial Intelligence) processor for processing computational operations related to machine learning.
  • AI Artificial Intelligence
  • Memory 1802 can include one or more computer readable storage media, which can be non-transitory. Memory 1802 can also include high speed random access memory, as well as non-volatile memory, such as one or more disk storage devices, flash storage devices. In some embodiments, the non-transitory computer readable storage medium in memory 1802 is for storing at least one instruction for execution by processor 1801 to implement the video provided by the method embodiments of the present application. The method of processing frame data.
  • the terminal 1800 optionally further includes: a peripheral device interface 1803 and at least one peripheral device.
  • the processor 1801, the memory 1802, and the peripheral device interface 1803 may be connected by a bus or a signal line.
  • Each peripheral device can be connected to the peripheral device interface 1803 via a bus, signal line or circuit board.
  • the peripheral device includes at least one of a radio frequency circuit 1804, a touch display screen 1805, a camera 1806, an audio circuit 1807, a positioning component 1808, and a power source 1809.
  • the peripheral device interface 1803 can be used to connect at least one peripheral device associated with an I/O (Input/Output) to the processor 1801 and the memory 1802.
  • the processor 1801, the memory 1802, and the peripheral device interface 1803 are integrated on the same chip or circuit board; in some other embodiments, any one of the processor 1801, the memory 1802, and the peripheral device interface 1803 or The two can be implemented on a separate chip or circuit board, which is not limited in this embodiment.
  • the RF circuit 1804 is configured to receive and transmit an RF (Radio Frequency) signal, also referred to as an electromagnetic signal.
  • the RF circuit 1804 communicates with the communication network and other communication devices via electromagnetic signals.
  • the RF circuit 1804 converts the electrical signal into an electromagnetic signal for transmission, or converts the received electromagnetic signal into an electrical signal.
  • radio frequency circuit 1804 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like.
  • Radio frequency circuitry 1804 can communicate with other terminals via at least one wireless communication protocol.
  • the wireless communication protocols include, but are not limited to, the World Wide Web, a metropolitan area network, an intranet, generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and/or WiFi (Wireless Fidelity) networks.
  • the RF circuit 1804 may also include NFC (Near Field Communication) related circuitry, which is not limited in this application.
  • the display 1805 is used to display a UI (User Interface).
  • the UI can include graphics, text, icons, video, and any combination thereof.
  • the display 1805 also has the ability to acquire touch signals over the surface or surface of the display 1805.
  • the touch signal can be input to the processor 1801 as a control signal for processing.
  • the display 1805 can also be used to provide virtual buttons and/or virtual keyboards, also referred to as soft buttons and/or soft keyboards.
  • the display screen 1805 can be one, and the front panel of the terminal 1800 is disposed; in other embodiments, the display screen 1805 can be at least two, respectively disposed on different surfaces of the terminal 1800 or in a folded design; In still other embodiments, display screen 1805 can be a flexible display screen disposed on a curved surface or a folded surface of terminal 1800. Even the display screen 1805 can be set to a non-rectangular irregular pattern, that is, a profiled screen.
  • the display 1805 can be made of a material such as an LCD (Liquid Crystal Display) or an OLED (Organic Light-Emitting Diode).
  • Camera component 1806 is used to capture images or video.
  • camera assembly 1806 includes a front camera and a rear camera.
  • the front camera is placed on the front panel of the terminal, and the rear camera is placed on the back of the terminal.
  • the rear camera is at least two, which are respectively a main camera, a depth camera, a wide-angle camera, and a telephoto camera, so as to realize the background blur function of the main camera and the depth camera, and the main camera Combine with a wide-angle camera for panoramic shooting and VR (Virtual Reality) shooting or other integrated shooting functions.
  • camera assembly 1806 can also include a flash.
  • the flash can be a monochrome temperature flash or a two-color temperature flash.
  • the two-color temperature flash is a combination of a warm flash and a cool flash that can be used for light compensation at different color temperatures.
  • the audio circuit 1807 can include a microphone and a speaker.
  • the microphone is used to collect sound waves of the user and the environment, and convert the sound waves into electrical signals for processing into the processor 1801 for processing, or input to the RF circuit 1804 for voice communication.
  • the microphones may be multiple, and are respectively disposed at different parts of the terminal 1800.
  • the microphone can also be an array microphone or an omnidirectional acquisition microphone.
  • the speaker is then used to convert electrical signals from the processor 1801 or the RF circuit 1804 into sound waves.
  • the speaker can be a conventional film speaker or a piezoelectric ceramic speaker.
  • audio circuit 1807 can also include a headphone jack.
  • the positioning component 1808 is configured to locate the current geographic location of the terminal 1800 to implement navigation or LBS (Location Based Service).
  • the positioning component 1808 can be a positioning component based on a US-based GPS (Global Positioning System), a Chinese Beidou system, or a Russian Galileo system.
  • a power supply 1809 is used to power various components in the terminal 1800.
  • the power source 1809 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery.
  • the rechargeable battery may be a wired rechargeable battery or a wireless rechargeable battery.
  • a wired rechargeable battery is a battery that is charged by a wired line
  • a wireless rechargeable battery is a battery that is charged by a wireless coil.
  • the rechargeable battery can also be used to support fast charging technology.
  • terminal 1800 also includes one or more sensors 1810.
  • the one or more sensors 1810 include, but are not limited to, an acceleration sensor 1811, a gyro sensor 1812, a pressure sensor 1813, a fingerprint sensor 1814, an optical sensor 1815, and a proximity sensor 1816.
  • the acceleration sensor 1811 can detect the magnitude of the acceleration on the three coordinate axes of the coordinate system established by the terminal 1800.
  • the acceleration sensor 1811 can be used to detect components of gravity acceleration on three coordinate axes.
  • the processor 1801 can control the touch display screen 1805 to display the user interface in a landscape view or a portrait view according to the gravity acceleration signal collected by the acceleration sensor 1811.
  • the acceleration sensor 1811 can also be used for the acquisition of game or user motion data.
  • the gyro sensor 1812 can detect the body direction and the rotation angle of the terminal 1800, and the gyro sensor 1812 can cooperate with the acceleration sensor 1811 to collect the 3D action of the user on the terminal 1800. Based on the data collected by the gyro sensor 1812, the processor 1801 can implement functions such as motion sensing (such as changing the UI according to the user's tilting operation), image stabilization at the time of shooting, game control, and inertial navigation.
  • functions such as motion sensing (such as changing the UI according to the user's tilting operation), image stabilization at the time of shooting, game control, and inertial navigation.
  • the pressure sensor 1813 can be disposed on a side border of the terminal 1800 and/or a lower layer of the touch display screen 1805.
  • the pressure sensor 1813 When the pressure sensor 1813 is disposed on the side frame of the terminal 1800, the user's holding signal to the terminal 1800 can be detected, and the processor 1801 performs left and right hand recognition or shortcut operation according to the holding signal collected by the pressure sensor 1813.
  • the operability control on the UI interface is controlled by the processor 1801 according to the user's pressure operation on the touch display screen 1805.
  • the operability control includes at least one of a button control, a scroll bar control, an icon control, and a menu control.
  • the fingerprint sensor 1814 is configured to collect the fingerprint of the user, and the processor 1801 identifies the identity of the user according to the fingerprint collected by the fingerprint sensor 1814, or the fingerprint sensor 1814 identifies the identity of the user according to the collected fingerprint. Upon identifying that the user's identity is a trusted identity, the processor 1801 authorizes the user to perform related sensitive operations including unlocking the screen, viewing encrypted information, downloading software, paying and changing settings, and the like.
  • Fingerprint sensor 1814 can be provided with the front, back or side of terminal 1800. When the physical button or vendor logo is set on the terminal 1800, the fingerprint sensor 1814 can be integrated with the physical button or the manufacturer logo.
  • Optical sensor 1815 is used to collect ambient light intensity.
  • the processor 1801 can control the display brightness of the touch display 1805 based on the ambient light intensity acquired by the optical sensor 1815. Specifically, when the ambient light intensity is high, the display brightness of the touch display screen 1805 is raised; when the ambient light intensity is low, the display brightness of the touch display screen 1805 is lowered.
  • the processor 1801 can also dynamically adjust the shooting parameters of the camera assembly 1806 based on the ambient light intensity acquired by the optical sensor 1815.
  • Proximity sensor 1816 also referred to as a distance sensor, is typically disposed on the front panel of terminal 1800.
  • Proximity sensor 1816 is used to capture the distance between the user and the front of terminal 1800.
  • the processor 1801 controls the touch display 1805 to switch from the bright screen state to the interest screen state; when the proximity sensor 1816 detects When the distance between the user and the front side of the terminal 1800 gradually becomes larger, the processor 1801 controls the touch display screen 1805 to switch from the state of the screen to the bright state.
  • terminal 1800 does not constitute a limitation to terminal 1800, may include more or fewer components than illustrated, or may combine certain components, or employ different component arrangements.

Abstract

本公开是关于一种对视频帧数据进行处理的方法和装置，属于视频编解码技术领域。所述方法包括：将预先训练的神经网络中的权重参数的数据类型转换为定点型，得到转换后的权重参数；将待进行目标处理的视频帧数据的数据类型转换为定点型，得到转换后的视频帧数据；将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，得到目标处理后的视频帧数据。这样，将浮点型的数据转换为定点型的数据，定点型的数据的小数点位置固定，无需对运算过程中的结果进行约束，不会出现对相同数据进行相同运算却出现不同结果的情况。进而编解码运算结果一致，解码端可以正常解码。

Description

对视频帧数据进行处理的方法和装置
本申请要求于2018年01月19日提交的申请号为201810054242.7、发明名称为“对视频帧数据进行处理的方法和装置”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本公开是关于视频编解码技术领域，尤其是关于一种对视频帧数据进行处理的方法和装置。
背景技术
在对视频帧图像进行压缩编码的过程中，需要对视频帧图像进行处理，如滤波处理。具体地，原始的视频帧图像会产生失真，因此在解码的过程得到的视频帧图像也是失真的视频帧图像。为了不影响视频帧图像的使用，需要对解码后的失真的视频帧图像进行滤波得到去失真的视频帧图像。
有研究表明，可以采用神经网络对失真的视频帧图像进行滤波。
在神经网络中进行运算的数据是浮点型的数据，浮点型的数据的运算结果与运算方式相关。浮点型的数据的小数位的位数可变，在运算过程中不可避免地运算结果会超出浮点型的数据可以表示的范围，因此总是要对运算结果进行约束，即将运算结果的小数位约束到浮点型的数据可以表示的范围之内。约束之后的数据是近似数据。由于近似数据的存在，运算的先后顺序会直接影响运算结果。
例如，浮点型的数据A、B、C，假如A、B、C的小数位的位数不一致，要计算它们相加的结果，第一种方式，可以先计算A+B的结果进行约束，再加C，再进行约束得到D1。第二种方式，也可以先计算B+C的结果进行约束，再加A，再进行约束得到D2。上述两种方式得到的结果D1和D2是不一样的。
在实现本公开的过程中，发明人发现至少存在以下问题：
不同编译器对浮点型数据的运算方式不一样，而且不同的运算方式对应不同的好处，不能直接规定所有编译器运算方式。如果编码端的编译器采用了第 一种方式对浮点型数据进行运算，而解码端的编译器采用了第二种方式对浮点型数据进行运算，它们两端得到的结果不一致，解码端无法正常解码。
发明内容
为了克服相关技术中存在的问题，本公开提供了以下技术方案：
根据本公开实施例的第一方面，提供一种对视频帧数据进行处理的方法，所述方法包括：
将预先训练的神经网络中的权重参数的数据类型转换为定点型，得到转换后的权重参数，其中，所述神经网络为用于在视频编解码过程中对视频帧数据进行目标处理的算法模型；
将待进行目标处理的视频帧数据的数据类型转换为定点型，得到转换后的视频帧数据；
将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，得到目标处理后的视频帧数据。
可选地，所述神经网络为卷积神经网络，所述权重参数包括卷积核元素和偏置。
可选地，所述将预先训练的神经网络中的权重参数的数据类型转换为定点型，得到转换后的权重参数，包括：
对于预先训练的卷积神经网络中的每个卷积核，确定所述卷积核中绝对值最大的卷积核元素；
对于所述卷积神经网络中的多个偏置，确定所述多个偏置中绝对值最大的偏置；
根据每个卷积核中绝对值最大的卷积核元素、以及预设的定点型的卷积核元素的数据位宽，将每个卷积核中的卷积核元素的数据类型转换为定点型，根据多个偏置中绝对值最大的偏置、以及预设的定点型的偏置的数据位宽，将所述多个偏置的数据类型转换为定点型，得到转换后的权重参数。
可选地，所述将待进行目标处理的视频帧数据的数据类型转换为定点型，得到转换后的视频帧数据，包括：
根据预设的定点型的视频帧数据的数据位宽、以及预先统计的所述卷积神经网络的输入层输出的特征数据中绝对值最大的数据，将待进行目标处理的视 频帧数据的数据类型转换为定点型，得到转换后的视频帧数据。
可选地，所述方法还包括：
将预设的所述视频帧数据的边信息的数据类型转换为定点型，得到转换后的边信息；
所述将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，得到目标处理后的视频帧数据，包括：
将转换后的视频帧数据和转换后的边信息，输入加载了转换后的权重参数的神经网络，得到目标处理后的视频帧数据。
可选地，在将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，得到目标处理后的视频帧数据之后，所述方法还包括：
对目标处理后的视频帧数据进行取整处理，得到整型的视频帧数据。
可选地，所述目标处理为去失真滤波处理；
所述将待进行目标处理的视频帧数据的数据类型转换为定点型，得到转换后的视频帧数据，包括：
将在视频编解码过程中进行重建处理得到的视频帧图像的数据类型转换为定点型，得到转换后的视频帧数据；
所述将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，得到目标处理后的视频帧数据，包括：
将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，进行去失真滤波处理，得到去失真的视频帧图像。
可选地，所述目标处理为编码帧内预测处理；
所述将待进行目标处理的视频帧数据的数据类型转换为定点型，得到转换后的视频帧数据，包括：
将在视频编码过程中原始未处理的视频帧图像中的目标区域的图像、以及在所述原始未处理的视频帧图像对应的重建处理得到的视频帧图像中与所述目标区域对应的关联区域的图像的数据类型转换为定点型，得到转换后的视频帧数据；
所述将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，得到目标处理后的视频帧数据，包括：
将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，进行编 码帧内预测处理，得到帧内预测图像和帧内预测相关信息。
可选地，所述目标处理为编码帧间预测处理；
所述将待进行目标处理的视频帧数据的数据类型转换为定点型，得到转换后的视频帧数据，包括：
将在视频编码过程中原始未处理的视频帧图像、以及所述原始未处理的视频帧图像对应的去失真滤波处理后的参考帧图像的数据类型转换为定点型，得到转换后的视频帧数据；
所述将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，得到目标处理后的视频帧数据，包括：
将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，进行编码帧间预测处理，得到帧间预测图像和帧间预测相关信息。
可选地，所述目标处理为熵编码处理；
所述将待进行目标处理的视频帧数据的数据类型转换为定点型，得到转换后的视频帧数据，包括：
将在视频编码过程中得到的帧内预测相关信息、帧间预测相关信息和量化系数的数据类型转换为定点型，得到转换后的视频帧数据；
所述将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，得到目标处理后的视频帧数据，包括：
将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，进行熵编码处理，得到熵编码信息。
可选地，所述目标处理为熵解码处理；
所述将待进行目标处理的视频帧数据的数据类型转换为定点型，得到转换后的视频帧数据，包括：
将在视频解码过程中获取的熵编码信息的数据类型转换为定点型，得到转换后的视频帧数据；
所述将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，得到目标处理后的视频帧数据，包括：
将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，进行熵解码处理，得到帧内预测相关信息、帧间预测相关信息和量化系数。
可选地，所述目标处理为解码帧内预测处理；
所述将待进行目标处理的视频帧数据的数据类型转换为定点型，得到转换后的视频帧数据，包括：
将在视频解码过程中重建处理得到的视频帧图像中与目标区域对应的关联区域的图像和帧内预测相关信息的数据类型转换为定点型，得到转换后的视频帧数据；
所述将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，得到目标处理后的视频帧数据，包括：
将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，进行解码帧内预测处理，得到目标区域的帧内预测图像。
可选地，所述目标处理为解码帧间预测处理；
所述将待进行目标处理的视频帧数据的数据类型转换为定点型，得到转换后的视频帧数据，包括：
将在视频解码过程中去失真滤波处理后的参考帧图像和帧间预测相关信息的数据类型转换为定点型，得到转换后的视频帧数据；
所述将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，得到目标处理后的视频帧数据，包括：
将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，进行解码帧间预测处理，得到帧间预测图像。
根据本公开实施例的第二方面，提供一种对视频帧数据进行处理的装置，所述装置包括：
第一转换模块，用于将预先训练的神经网络中的权重参数的数据类型转换为定点型，得到转换后的权重参数，其中，所述神经网络为用于在视频编解码过程中对视频帧数据进行目标处理的算法模型；
第二转换模块，用于将待进行目标处理的视频帧数据的数据类型转换为定点型，得到转换后的视频帧数据；
输入模块，用于将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，得到目标处理后的视频帧数据。
可选地，所述神经网络为卷积神经网络，所述权重参数包括卷积核元素和偏置。
可选地，所述第一转换模块包括：
第一确定单元，用于对于预先训练的卷积神经网络中的每个卷积核，确定所述卷积核中绝对值最大的卷积核元素；
第二确定单元，用于对于所述卷积神经网络中的多个偏置，确定所述多个偏置中绝对值最大的偏置；
转换单元，用于根据每个卷积核中绝对值最大的卷积核元素、以及预设的定点型的卷积核元素的数据位宽，将每个卷积核中的卷积核元素的数据类型转换为定点型，根据多个偏置中绝对值最大的偏置、以及预设的定点型的偏置的数据位宽，将所述多个偏置的数据类型转换为定点型，得到转换后的权重参数。
可选地，所述第二转换模块，用于根据预设的定点型的视频帧数据的数据位宽、以及预先统计的所述卷积神经网络的输入层输出的特征数据中绝对值最大的数据，将待进行目标处理的视频帧数据的数据类型转换为定点型，得到转换后的视频帧数据。
可选地，所述装置还包括：
第三转换模块，用于将预设的所述视频帧数据的边信息的数据类型转换为定点型，得到转换后的边信息；
所述输入模块，用于将转换后的视频帧数据和转换后的边信息，输入加载了转换后的权重参数的神经网络，得到目标处理后的视频帧数据。
可选地，所述装置还包括：
取整模块，用于对目标处理后的视频帧数据进行取整处理，得到整型的视频帧数据。
可选地，所述目标处理为去失真滤波处理；
所述第二转换模块，用于将在视频编解码过程中进行重建处理得到的视频帧图像的数据类型转换为定点型，得到转换后的视频帧数据；
所述输入模块，用于将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，进行去失真滤波处理，得到去失真的视频帧图像。
可选地，所述目标处理为编码帧内预测处理；
所述第二转换模块，用于将在视频编码过程中原始未处理的视频帧图像中的目标区域的图像、以及在所述原始未处理的视频帧图像对应的重建处理得到的视频帧图像中与所述目标区域对应的关联区域的图像的数据类型转换为定点型，得到转换后的视频帧数据；
所述输入模块，用于将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，进行编码帧内预测处理，得到帧内预测图像和帧内预测相关信息。
可选地，所述目标处理为编码帧间预测处理；
所述第二转换模块，用于将在视频编码过程中原始未处理的视频帧图像、以及所述原始未处理的视频帧图像对应的去失真滤波处理后的参考帧图像的数据类型转换为定点型，得到转换后的视频帧数据；
所述输入模块，用于将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，进行编码帧间预测处理，得到帧间预测图像和帧间预测相关信息。
可选地，所述目标处理为熵编码处理；
所述第二转换模块，用于将在视频编码过程中得到的帧内预测相关信息、帧间预测相关信息和量化系数的数据类型转换为定点型，得到转换后的视频帧数据；
所述输入模块，用于将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，进行熵编码处理，得到熵编码信息。
可选地，所述目标处理为熵解码处理；
所述第二转换模块，用于将在视频解码过程中获取的熵编码信息的数据类型转换为定点型，得到转换后的视频帧数据；
所述输入模块，用于将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，进行熵解码处理，得到帧内预测相关信息、帧间预测相关信息和量化系数。
可选地，所述目标处理为解码帧内预测处理；
所述第二转换模块，用于将在视频解码过程中重建处理得到的视频帧图像中与目标区域对应的关联区域的图像和帧内预测相关信息的数据类型转换为定点型，得到转换后的视频帧数据；
所述输入模块，用于将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，进行解码帧内预测处理，得到目标区域的帧内预测图像。
可选地，所述目标处理为解码帧间预测处理；
所述第二转换模块，用于将在视频解码过程中去失真滤波处理后的参考帧图像和帧间预测相关信息的数据类型转换为定点型，得到转换后的视频帧数据；
所述输入模块，用于将转换后的视频帧数据输入加载了转换后的权重参数 的神经网络，进行解码帧间预测处理，得到帧间预测图像。
根据本公开实施例的第三方面，提供一种终端，所述终端包括处理器、通信接口、存储器和通信总线，其中：
所述处理器、所述通信接口和所述存储器通过所述通信总线完成相互间的通信；
所述存储器，用于存放计算机程序；
所述处理器，用于执行所述存储器上所存放的程序，以实现上述对视频帧数据进行处理的方法。
根据本公开实施例的第四方面，提供一种计算机可读存储介质，所述计算机可读存储介质内存储有计算机程序，所述计算机程序被处理器执行时实现上述对视频帧数据进行处理的方法。
本公开的实施例提供的技术方案可以包括以下有益效果：
通过本公开实施例提供的方法，将预先训练的神经网络中的权重参数的数据类型转换为定点型，得到转换后的权重参数；将待进行目标处理的视频帧数据的数据类型转换为定点型，得到转换后的视频帧数据；将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，得到目标处理后的视频帧数据。这样，将浮点型的数据转换为定点型的数据，定点型的数据的小数点位置固定，无需对运算过程中的结果进行约束，不会出现对相同数据进行相同运算却出现不同结果的情况。进而编解码运算结果一致，解码端可以正常解码。
应当理解的是，以上的一般描述和后文的细节描述仅是示例性和解释性的，并不能限制本公开。
附图说明
此处的附图被并入说明书中并构成本说明书的一部分，示出了符合本公开的实施例，并与说明书一起用于解释本公开的原理。在附图中：
图1是根据一示例性实施例示出的一种视频编解码系统的编码端的结构示意图；
图2是根据一示例性实施例示出的一种视频编解码系统的解码端的结构示意图；
图3是根据一示例性实施例示出的一种对视频帧数据进行处理的方法的流 程图示意图；
图4是根据一示例性实施例示出的一种对视频帧数据进行处理的方法的流程图示意图；
图5是根据一示例性实施例示出的一种卷积神经网络中神经元的示意图；
图6是根据一示例性实施例示出的一种对视频帧数据进行处理的方法的流程图示意图；
图7是根据一示例性实施例示出的一种对视频帧数据进行处理的方法的流程图示意图；
图8是根据一示例性实施例示出的一种对视频帧数据进行处理的方法的流程图示意图；
图9是根据一示例性实施例示出的一种对视频帧数据进行处理的方法的流程图示意图；
图10是根据一示例性实施例示出的一种对视频帧数据进行处理的方法的流程图示意图；
图11是根据一示例性实施例示出的一种对视频帧数据进行处理的方法的流程图示意图；
图12是根据一示例性实施例示出的一种对视频帧数据进行处理的装置的结构示意图；
图13是根据一示例性实施例示出的一种终端的结构示意图。
通过上述附图，已示出本公开明确的实施例，后文中将有更详细的描述。这些附图和文字描述并不是为了通过任何方式限制本公开构思的范围，而是通过参考特定实施例为本领域技术人员说明本公开的概念。
具体实施方式
这里将详细地对示例性实施例进行说明，其示例表示在附图中。下面的描述涉及附图时，除非另有表示，不同附图中的相同数字表示相同或相似的要素。以下示例性实施例中所描述的实施方式并不代表与本公开相一致的所有实施方式。相反，它们仅是与如所附权利要求书中所详述的、本公开的一些方面相一致的装置和方法的例子。
本公开实施例提供了一种对视频帧数据进行处理的方法，该方法可以由终 端实现。其中，终端可以是机顶盒、平板电脑、台式计算机、笔记本计算机等。
终端可以包括处理器、存储器等部件。处理器，可以为CPU(Central Processing Unit，中央处理单元)等，可以用于将预先训练的神经网络中的权重参数的数据类型转换为定点型，等处理。存储器，可以为RAM(Random Access Memory，随机存取存储器)，Flash(闪存)等，可以用于存储接收到的数据、处理过程所需的数据、处理过程中生成的数据等，如视频帧数据等。
终端还可以包括收发器、输入部件、显示部件、音频输出部件等。收发器，可以用于与服务器进行数据传输，收发器可以包括蓝牙部件、WiFi(Wireless-Fidelity，无线高保真技术)部件、天线、匹配电路、调制解调器等。输入部件可以是触摸屏、键盘、鼠标等。音频输出部件可以是音箱、耳机等。
本实施例提供的一种对视频帧数据进行处理的方法可以应用于视频编解码系统中。视频编解码主要包括编码端和解码端。
下面对视频编解码系统中编码端的结构进行简单的介绍。在编码端中，原始的视频帧图像会被进行以下处理：预测、变换、量化、重建、滤波等。对应这些处理过程，如图1所示，编码端可以包括编码帧内预测模块、编码帧间预测模块、变换模块、量化模块、熵编码器、反量化模块、反变换模块、重建模块、滤波模块、参考图像缓存器等。
在图1中，编码帧内预测模块、编码帧间预测模块可以基于在视频编解码过程中进行重建处理得到的视频帧图像分别确定帧内预测图像、帧内预测相关信息、帧间预测图像、帧间预测相关信息。与编码帧内预测模块和编码帧间预测模块相连的开关用于选择使用编码帧内预测模块或者编码帧间预测模块，由被选择的模块向加法器提供帧内预测图像或者帧间预测图像。帧内预测图像或者帧间预测图像经过加法器之后，得到预测残差。预测残差经过变换、量化处理，变为量化系数。量化系数、帧内预测相关信息、帧间预测相关信息、预设的视频帧图像的边信息被输入到熵编码器中进行熵编码，最终输出码流。
其中，边信息可以是量化过程中使用的量化系数，该量化系数可以是用户设置的，也可以是通过计算得到的。边信息对应的基本单元可以是视频帧图像，或者是视频帧图像被切分成的图像块。如果编码端使用了边信息，那么码流中也要携带边信息，这样解码端才可以基于边信息正常进行解码。
在使用编码帧间预测模块时，需要获取参考帧图像即去失真的视频帧图像， 参考帧图像可以被存储在参考图像缓存器中。具体地，可以将量化系数进行反量化、反变换，以恢复预测残差。在重建模块，预测残差被加回到相应的帧内预测图像或者帧间预测图像上，得到失真的视频帧图像。失真的视频帧图像经过去失真滤波处理，就可以转换为参考帧图像。
下面对视频编解码系统中解码端的结构进行简单的介绍。在解码端中，如图2所示，解码端可以包括解码帧内预测模块、解码帧间预测模块、熵解码器、反量化模块、反变换模块、重建模块、滤波模块、参考图像缓存器、视频播放缓存器等。
在视频编解码系统中，一个视频可以经过编码端编码之后得到码流，码流在解码端可以被恢复成一个有失真的视频。需要说明的是，除了在解码端需要进行解码过程，在编码端也需要进行解码过程，这是因为经过解码过程可以将视频帧图像进行恢复，恢复后的视频帧图像作为参考图像，从而对其进行运动补偿等操作。由于恢复后的视频帧图像存在失真，因此可以通过训练好的神经网络对恢复后的视频帧图像进行滤波，得到去失真的视频帧图像，可以使用本实施例提供的方法对图像进行处理的操作。
除此以外，视频编解码系统中的编码帧内预测模块、编码帧间预测模块、熵编码器、熵解码器、解码帧内预测模块、解码帧间预测模块可以分别应用各自训练好的神经网络进行编码帧内预测、编码帧间预测、熵编码、熵解码、解码帧内预测、解码帧间预测处理。由于在进行相应处理的过程中，涉及到使用神经网络对图像或者数据进行处理，因此都可以使用本实施例提供的方法通过神经网络对图像或者数据进行处理。
另外，视频编解码系统中的其他模块如变换模块、量化模块、反变换模块、反量化模块在进行量化、变换、返变换、反量化处理时，如果涉及到使用神经网络对图像或者数据进行处理，全都可以使用本实施例提供的方法通过神经网络对图像或者数据进行处理。或者，两个或者两个以上串联的模块的组合，例如变换模块和量化模块的组合，进行变换量化处理时，如果涉及到使用神经网络对图像或者数据进行处理，也可以使用本实施例提供的方法通过神经网络对图像或者数据进行处理。再或者，整个编码端或者解码端都可以分别使用一个神经网络，直接对视频数据进行编解码处理。由于这种情况也涉及到使用神经网络对图像或者数据进行处理，也可以使用本实施例提供的方法通过神经网络 对图像或者数据进行处理。
本公开一示例性实施例提供了一种对视频帧数据进行处理的方法，如图3所示，该方法的处理流程可以包括如下的步骤：
步骤S310，将预先训练的神经网络中的权重参数的数据类型转换为定点型，得到转换后的权重参数。
其中，神经网络为用于在视频编解码过程中对视频帧数据进行目标处理的算法模型。
神经网络可以是卷积神经网络、循环神经网络、对抗生成网络、自编码器、深度神经网络等模型。权重参数可以是在训练过程中由训练得到的参数。在进行神经网络如卷积神经网络训练过程中，由于浮点型的数据是连续的，可以求偏导，而定点型的数据是非连续的，不能直接求偏导，因此训练好的神经网络中的权重参数的数据类型是浮点型的。
为了保证编解码一致以及方便运算，需要将浮点型的权重参数转换为定点型的权重参数。定点型的数据可以为普通定点数或者动态定点数等。
可选地，神经网络为卷积神经网络，权重参数包括卷积核元素和偏置。
步骤S320，将待进行目标处理的视频帧数据的数据类型转换为定点型，得到转换后的视频帧数据。
其中，待进行目标处理的视频帧数据可以包括原始的视频帧图像或者对原始的视频帧图像进行处理后得到的数据。
如果本实施例提供的方法被用在滤波模块中，则待进行目标处理的视频帧数据可以是重建的视频帧图像。如果本实施例提供的方法被用在帧间预测或者帧内预测模块中，则待进行目标处理的视频帧数据可以是原始的视频帧图像。如果本实施例提供的方法被用在熵编码器中，则待进行目标处理的视频帧数据可以是原始的视频帧图像经过预测、变换、量化等处理后得到的数据。
需要向转换后的神经网络输入待进行目标处理的视频帧数据，在神经网络中，待进行目标处理的视频帧数据和转换后的神经网络中的定点型的权重参数进行运算，然而待进行目标处理的视频帧数据的数据类型是整型或者浮点型的，定点型的数据无法和整型、浮点型的数据直接做运算。因此，需要将待进行目标处理的视频帧数据的数据类型转换为定点型。
步骤S330，将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，得到目标处理后的视频帧数据。
可以在将转换后的视频帧图像输入转换后的神经网络之前，对转换后的视频帧图像进行切分编码，将转换后的视频帧图像切分为一块块的图像块，将一块块的图像块输入到转换后的神经网络中。
可选地，本实施例提供的方法还包括：将预设的视频帧数据的边信息的数据类型转换为定点型，得到转换后的边信息；步骤S330可以包括：将转换后的视频帧数据和转换后的边信息，输入加载了转换后的权重参数的神经网络，得到定点型的目标处理后的视频帧数据。
可以只将转换后的视频帧图像输入到转换后的神经网络得到由定点型的像素值构成的输出视频帧图像。还可以将转换后的视频帧图像和边信息，输入转换后的神经网络，得到由定点型的像素值构成的输出视频帧图像。要求输入的边信息的数据类型是定点型的，因此需要将边信息的数据类型转换为定点型，得到转换后的边信息。
其中，边信息可以是量化过程中使用的量化系数，该量化系数可以是用户设置的，也可以是通过计算得到的。边信息对应视频帧图像，或者视频帧图像被切分成的图像块。码流中也携带有边信息，这样解码端可以基于边信息正常进行解码。
通过本公开实施例提供的方法，将预先训练的神经网络中的权重参数的数据类型转换为定点型，得到转换后的权重参数；将待进行目标处理的视频帧数据的数据类型转换为定点型，得到转换后的视频帧数据；将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，得到目标处理后的视频帧数据。这样，将浮点型的数据转换为定点型的数据，定点型的数据的小数点位置固定，无需对运算过程中的结果进行约束，不会出现对相同数据进行相同运算却出现不同结果的情况。进而编解码运算结果一致，解码端可以正常解码。
下面以视频编解码过程中去失真滤波处理为例进行本实施例的介绍：
本公开一示例性实施例提供了一种对视频帧数据进行处理的方法，如图4所示，该方法的处理流程可以包括如下的步骤：
步骤S410，将预先训练的神经网络中的权重参数的数据类型转换为定点型， 得到转换后的权重参数。
其中，神经网络是一种模仿动物神经网络行为特征，进行分布式并行信息处理的算法模型。这种算法模型依靠整个算法模型的复杂程度，通过调整算法模型内部大量节点之间相互连接的关系，从而达到处理图像的目的。本实施例的神经网络为用于在视频编解码过程中对视频帧数据进行去失真滤波处理的算法模型。
由于神经网络的功能是对图像进行处理，因此不可避免地需要向神经网络中输入图像，图像和神经网络中的权重参数进行运算，输出处理后的图像。在这个过程中，图像和权重参数进行运算时，需要将图像和权重参数的数据类型进行统一，如统一成定点型，统一后的不同数据可以进行运算。首先，介绍将权重参数的数据类型转换为定点型的方法。
定点型的数据可以为普通定点数或者动态定点数等。
对于普通定点数，约定处理器中任一数据的小数点的位置是固定不变的，小数点的位置在处理器中不予表示，而是靠预先约定好，即一旦确定了小数点的位置，则在一般的情况下不再改变小数点的位置。可以使用符号位和量值表示普通定点数。假设普通定点数的数据位宽为n比特，则符号位占1比特，量值占n-1比特。由于普通定点数的小数点位置不会发生改变，因此不需要取近似值将其约束到规定的精度，从而避免了由于不同处理器对于精度的约束规则不一样，而产生的对相同的数值进行运算却算出不同的结果的现象。在视频编解码系统中，如果编码端和解码端的处理器对于精度的约束规则不一样，则会导致编码端和解码端运算结果不一样，进而解码端无法正确解码出正确的视频帧图像。
对于动态定点数，其可以表示为：
Figure PCTCN2019072033-appb-000001
其中，n表示定点数的数据位宽，FL表示分数部分的长度，x i为尾数部分第i个比特位的数据。
在本实施例中，主要以将非定点数转化为动态定点数为示例进行说明，其他情况与之类似，在此不再赘述。
在训练神经网络时，可以使用大量的具有不同失真程度的失真的视频帧图像和与其对应的未失真的视频帧图像，对神经网络进行多次迭代训练，得到能够对失真的视频帧图像进行去失真处理的神经网络。具体训练方法如下：
(1)在同一编码端中，对大量的未失真的视频帧图像进行编码，得到失真的视频帧图像，将未失真的视频帧图像和对应的失真的视频帧图像组成对，多对视频帧图像组成训练集Ω。
(2)初始化卷积神经网络的网络参数Θ 0，对学习率、权重更新算法、模型结构等进行合理的设置。
(3)基于训练集Ω以及卷积神经网络中的参数Θ 0或者Θ i进行前向计算，获取卷积神经网络的输出F(Y)，使用均方误差公式作为损失函数，得到损失值L(Θ i)。
(4)利用反向传播算法对卷积神经网络中的参数Θ 0或者Θ i进行调整，获得调整后的Θ 0或者Θ i。
(5)重复步骤(3)至步骤(4)，直至反向传播函数收敛，输出参数Θ final。
可选地，神经网络为卷积神经网络，权重参数包括卷积核元素和偏置。
卷积神经网络由神经元构成。如图5所示，是一个神经元的示意图。其中，X 1至X 4、+1是输入，w 1至w 4是卷积核元素，可以是矩阵，b为偏置，f是激活函数，y为输出。卷积神经网络区别于其他神经网络的特点在于权重参数可以共享，相比于其他神经网络可以节省存储权重参数的空间，也减少了需要训练的权重参数的数量。
可选地，步骤S410可以包括：对于预先训练的卷积神经网络中的每个卷积核，确定卷积核中绝对值最大的卷积核元素；对于卷积神经网络中的多个偏置，确定多个偏置中绝对值最大的偏置；根据每个卷积核中绝对值最大的卷积核元素、以及预设的定点型的卷积核元素的数据位宽，将每个卷积核中的卷积核元素的数据类型转换为定点型，根据多个偏置中绝对值最大的偏置、以及预设的定点型的偏置的数据位宽，将多个偏置的数据类型转换为定点型，得到转换后的权重参数。
使用W ij和b ij分别表示卷积神经网络中第i层的第j个卷积核和第i层的第j个偏置。其中，i＝1，2，……，N。j＝1，2，……，C i。N为卷积神经网络的不包含输入层的层数。C i为第i层的卷积核的个数。
下面介绍将每个卷积核中的卷积核元素的数据类型转换为定点型，得到转换后的卷积神经网络的方法：
下面的运算主要是为了对卷积核元素或者偏置进行取整处理，这样后续的 计算过程中可以只对动态定点数的符号位和尾数部分进行运算，暂不对小数部分进行运算。对动态定点数的符号位和尾数部分进行运算显然是整数运算，相比带着小数点进行运算要便捷地多，而只要记录下分数部分的量级即FL的值，不管中间如何进行整数运算，最后依然可以除以2 -FL将其映射回实际数值。
对于卷积核元素，设定点型的第i层卷积核元素的数据位宽为
Figure PCTCN2019072033-appb-000002
则有：
Figure PCTCN2019072033-appb-000003
其中，max(|W ij(·)|)表示卷积核W ij中绝对值最大的卷积核元素的值。log两端的符号
Figure PCTCN2019072033-appb-000004
表示对符号中的数据进行向下取整。
接着，通过公式3可以确定
Figure PCTCN2019072033-appb-000005
Figure PCTCN2019072033-appb-000006
进行公式3的计算的目的是在第i层所有的
Figure PCTCN2019072033-appb-000007
中，选一个目标
Figure PCTCN2019072033-appb-000008
这个目标
Figure PCTCN2019072033-appb-000009
使得第i层中其他
Figure PCTCN2019072033-appb-000010
大于和小于目标
Figure PCTCN2019072033-appb-000011
的数量较为平均，甚至是相等，将目标
Figure PCTCN2019072033-appb-000012
作为
Figure PCTCN2019072033-appb-000013
其中，Cnt less表示当前层所有
Figure PCTCN2019072033-appb-000014
小于在所有
Figure PCTCN2019072033-appb-000015
中选定的一个
Figure PCTCN2019072033-appb-000016
(在下式表示为FL)的个数，可以写为：
Figure PCTCN2019072033-appb-000017
其中，Cnt large表示当前层所有
Figure PCTCN2019072033-appb-000018
大于在所有
Figure PCTCN2019072033-appb-000019
中选定的一个
Figure PCTCN2019072033-appb-000020
(在下式表示为FL)的个数，可以写为：
Figure PCTCN2019072033-appb-000021
另外，对于偏置，设定点型的第i层的偏置的数据位宽为
Figure PCTCN2019072033-appb-000022
则有：
Figure PCTCN2019072033-appb-000023
其中，
Figure PCTCN2019072033-appb-000024
表示多个偏置中绝对值最大的偏置的值。
假如第i层所有卷积核的集合为W i，所有偏置的集合为B i。W i f和
Figure PCTCN2019072033-appb-000025
分别为定点型的W i和B i，则W i f可表示为：
Figure PCTCN2019072033-appb-000026
则
Figure PCTCN2019072033-appb-000027
可表示为：
Figure PCTCN2019072033-appb-000028
其中，round()为取整操作。在上述求
Figure PCTCN2019072033-appb-000029
的过程中，也进行了向下取整运算操作。不管是取整操作还是向下取整操作，其目的是为了只对动态定点 数的符号位和尾数部分进行运算，这部分的运算显然是整数运算，相比带着小数点进行运算要便捷地多，而只要记录下分数部分的量级即FL的值，不管中间如何进行整数运算，最后依然可以除以2 -FL将其映射回实际数值。因此，下文中的取整都是将动态定点数的分数部分只记录不做运算。
上面介绍了，图像和权重参数进行运算时，将图像和权重参数的数据类型进行统一，如统一成定点型的方法。具体介绍了将权重参数的数据类型转换为定点型的方法。下面介绍将图像的数据类型转换为定点型的方法。
步骤S420，将在视频编解码过程中进行重建处理得到的视频帧图像的数据类型转换为定点型，得到转换后的视频帧数据。
需要向转换后的神经网络输入在视频编解码过程中进行重建处理得到的视频帧图像，在视频编解码过程中进行重建处理得到的视频帧图像和转换后的神经网络中的定点型的权重参数进行运算，然而在视频编解码过程中进行重建处理得到的视频帧图像的数据类型是整型或者浮点型的，定点型的数据无法和整型、浮点型的数据直接做运算。因此，需要将在视频编解码过程中进行重建处理得到的视频帧图像的数据类型转换为定点型。
可选地，步骤S420可以包括：根据预设的定点型的视频帧数据的数据位宽、以及预先统计的卷积神经网络的输入层输出的特征数据中绝对值最大的数据，将待进行目标处理的视频帧数据的数据类型转换为定点型，得到转换后的视频帧数据。
假设大量的待处理的视频帧图像的的集合为S，可以通过统计将S输入到卷积神经网络，并统计输入层和隐含层能够输出的数据的大小来确定FL 0以及FL i。FL 0是在将待处理的视频帧图像中的像素值的数据类型转换为定点型的运算过程中，需要的一个参数，这个参数可以通过计算获得：
假设将S输入到卷积神经网络中，这个卷积神经网络可以是没有经过转换后的卷积神经网络，也可以是转换后的卷积神经网络。在卷积神经网络没有经过转换的情况下，可以统计输入层输出的特征图像的像素值中绝对值最大的像素值，记为
Figure PCTCN2019072033-appb-000030
则有：
Figure PCTCN2019072033-appb-000031
其中，
Figure PCTCN2019072033-appb-000032
为定点型的输入层的像素值的数据位宽。
这里，再计算一个参数FL i，该参数在后续进行定点型运算的过程中要使用 到。同样，假设将S输入到卷积神经网络中，可以统计每层隐含层输出的特征图像的像素值中绝对值最大的像素值，第i层隐含层输出的特征图像的像素值中绝对值最大的像素值记为
Figure PCTCN2019072033-appb-000033
则有：
Figure PCTCN2019072033-appb-000034
其中，
Figure PCTCN2019072033-appb-000035
为定点型的第i层隐含层的像素值的数据位宽。该卷积神经网络中共包含N-1个隐含层。
需要说明的是，在卷积神经网络是经过转换的情况下，在统计
Figure PCTCN2019072033-appb-000036
和
Figure PCTCN2019072033-appb-000037
之前，由于转换后的卷积神经网络中的卷积核元素和偏置的数据类型由浮点型转换为定点型，在这个转换过程中，定点型的数据不能准确地一一对应浮点型的数据，因此它们之前存在误差，需要带着这个误差去统计
Figure PCTCN2019072033-appb-000038
和
Figure PCTCN2019072033-appb-000039
才能更准确地统计出
Figure PCTCN2019072033-appb-000040
和
Figure PCTCN2019072033-appb-000041
具体做法是将由公式7和公式8确定的W i f和
Figure PCTCN2019072033-appb-000042
的数据类型再转换为浮点型。记转换后的浮点型的卷积核权重和偏置为
Figure PCTCN2019072033-appb-000043
和
Figure PCTCN2019072033-appb-000044
则
Figure PCTCN2019072033-appb-000045
可以表示为：
Figure PCTCN2019072033-appb-000046
Figure PCTCN2019072033-appb-000047
可以表示为：
Figure PCTCN2019072033-appb-000048
上述公式11和公式12中的
Figure PCTCN2019072033-appb-000049
和
Figure PCTCN2019072033-appb-000050
的计算方法可参见前面的公式3和公式6，在此不再赘述。
在得到
Figure PCTCN2019072033-appb-000051
和
Figure PCTCN2019072033-appb-000052
之后，将
Figure PCTCN2019072033-appb-000053
和
Figure PCTCN2019072033-appb-000054
作为卷积神经网络的卷积核元素和偏置去统计
Figure PCTCN2019072033-appb-000055
和
Figure PCTCN2019072033-appb-000056
接下来，可以依然通过公式9和公式10去计算FL 0以及FL i。
在得到FL 0之后，就可以基于FL 0将待处理的视频帧图像中的像素值的数据类型转换为定点型了，具体可以通过公式13待处理的视频帧图像中的像素值的数据类型转换为定点型，得到I f：
Figure PCTCN2019072033-appb-000057
其中，I为整型或者浮点型的待处理的视频帧图像中的像素值。
上面介绍了，将在视频编解码过程中进行重建处理得到的视频帧图像的数据类型转换为定点型的方法，即将非定点类型的I转换为I f。在转换的过程中，需要确定参数FL 0，因此还介绍了确定FL 0的过程。
步骤S430，将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，进行去失真滤波处理，得到去失真的视频帧图像。
转换后的视频帧数据可以是转换后的失真的视频帧图像，原来的失真的视频帧图像中的像素值是非定点型的，而转换后的失真的视频帧图像中的像素值是定点型的。将转换后的失真的视频帧图像输入到用于进行去失真滤波处理的神经网络，可以输出去失真的视频帧图像，借此来对失真的视频帧图像进行去失真滤波处理。
可以在将转换后的失真的视频帧图像输入转换后的神经网络之前，将对转换后的失真的视频帧图像进行切分进行编码，将整个换后的视频帧图像切分为一块块的图像块，将一块块的图像块输入到转换后的神经网络中。为了方便说明，下文都用视频帧图像代替一块块的图像块进行说明。
可以通过定点型运算来对定点型的卷积神经网络中的卷积核元素、偏置和转换后的视频帧数据进行运算。具体算法如下所述：
首先，上一层的输出数据可以作为当前层的输入数据，将其与当前层的定点型的卷积核元素和偏置进行运算，得到特征图像F‘’ i(I f)。设i为当前层的层数，i-1则为上一层的层数，则有：
Figure PCTCN2019072033-appb-000058
其中，FL i-1可以通过公式10计算得到。需要说明的是，公式14表示的是动态定点数的符号位以及尾数部分进行运算的形式，实际的数值还要基于它们的分数部分最终确定。即只对分子进行运算，只记录分母的大小，在最终再除以分母得到实际数值。这样，分子的运算都是整数型的运算，较为简单便捷。公式中的
Figure PCTCN2019072033-appb-000059
是为了将B i的分数部分的量级转换为与W i f*F i-1(I f)的量级相同的量级而进行的运算。
因为当W i f*F i-1(I f)即进行卷积运算时，分数部分2 -FL在相乘的运算中会变为2 -nFL，而B i的分数部分为2 -FL，如果不将它们统一，则W i f*F i-1(I f)与B i的分数部分不一致，无法进行相加运算。
可选地，由于B i的分数部分在不经过转换的情况下就与W i f*F i-1(I f)的分数部分一致，故而就不需要进行转换了。因此，在进行
Figure PCTCN2019072033-appb-000060
转换之前，可以先确定B i的分数部分和W i f*F i-1(I f)的分数部分是否一致。在一致的情况下，直接进行W i f*F i-1(I f)+B i的运算。在不一致的情况下，再使用公式14提供的算法进行运算。
接着，可以对F‘’ i(I f)进行量化。由于定点型的数据可以表示的数据的有限，因此为了防止在运算过程中产生溢出，可以对F‘’ i(I f)进行量化。量化后的F‘’ i(I f)记 为F′ i(I f)，则有：
Figure PCTCN2019072033-appb-000061
最后，将F′ i(I f)输入到激活函数中，激活函数可以是非线性激活函数记为
g()，得到当前层的特征图像F i(I f)。
F i(I f)＝g(F′ i(I f))  (公式16)
在有了定点型的权重参数和待进行目标处理的视频帧数据之后，需要做定点型运算，具体定点型运算可以参见上面的介绍，以确保在运算后得到去失真的视频帧图像F N-1(I f)。
以上，介绍了通过定点型运算来对定点型的卷积神经网络中的卷积核元素、偏置和转换后的视频帧图像进行运算的方法。在通过定点型运算来对定点型的卷积神经网络中的卷积核元素、偏置和转换后的视频帧图像进行运算之后，输出视频帧图像中的像素值的数据类型为定点型。而在运动补偿等操作过程中需要使用整型的像素值，因此需要将输出视频帧图像中的像素值的数据类型转换为整型，或者根据需要将定点型的目标处理后的视频帧数据转换为目标类型。无论要将定点型的目标处理后的视频帧数据转换为何种目标类型，都可以先将定点型的目标处理后的视频帧数据转换为浮点型，再由浮点型转换为目标类型。
可选地，本实施例提供的方法还可以包括：将目标处理后的视频帧数据的数据类型转换为浮点型，得到浮点型的视频帧数据。
将目标处理后的视频帧数据的数据类型转换为浮点型是一步中间媒介步骤。在神经网络之后的模块需要接收的数据的数据类型不同，需要将神经网络输出的数据的数据类型转换为神经网络之后的模块需要接收的数据的数据类型。不管神经网络之后的模块需要接收的数据的数据类型是什么样的类型，都可以先将定点型转换为浮点型，再由浮点型转换为神经网络之后的模块需要接收的数据的数据类型。这样，神经网络和神经网络之后的模块之间，可以无缝隙连接，即神经网络输出的数据可以被神经网络之后的模块进行后续处理。
假设卷积神经网络中的最后一层隐含层输出的特征图像中的像素值为F N-1(I f)，整型的输出视频帧图像中的像素值为O。
可选地，在将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，得到目标处理后的视频帧数据之后，本公开实施例提供的方法还可以包括：
对目标处理后的视频帧数据进行取整处理，得到整型的视频帧数据。
在执行将目标处理后的视频帧数据的数据类型转换为浮点型之后，还可以：基于取整处理，将浮点型的视频帧数据转换为整型的视频帧数据，得到处理后的视频帧数据。
在卷积神经网络应用于滤波模块、编码/解码帧内预测模块、编码/解码帧间预测模块的情况下，需要将浮点型的视频帧数据换为整型的视频帧数据，即整型的像素值。整型的像素值构成了视频帧图像。
首先，最后一层隐含层输出的特征图像也需要和输出层的卷积核元素
Figure PCTCN2019072033-appb-000062
偏置B N进行定点型运算，得到定点型的输出视频帧图像中的像素值O″，则有：
Figure PCTCN2019072033-appb-000063
其中，可以根据公式3计算
Figure PCTCN2019072033-appb-000064
根据公式6计算
Figure PCTCN2019072033-appb-000065
根据公式10计算FL N-1。
接着，将定点型的输出视频帧图像中的像素值O″转换为浮点型的输出视频帧图像中的像素值O′。
Figure PCTCN2019072033-appb-000066
最后，可以将浮点型的输出视频帧图像中的像素值O′转换为整型的输出视频帧图像中的像素值O。
O＝round(O′)  (公式19)
由此，可以得到运动补偿等操作过程中需要使用的整型的像素值O。
通过本公开实施例提供的方法，将预先训练的神经网络中的权重参数的数据类型转换为定点型，得到转换后的权重参数；将在视频编解码过程中进行重建处理得到的视频帧图像的数据类型转换为定点型，得到转换后的视频帧数据；将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，进行去失真滤波处理，得到去失真的视频帧图像。这样，将浮点型的数据转换为定点型的数据，定点型的数据的小数点位置固定，无需对运算过程中的结果进行约束，不会出现对相同数据进行相同运算却出现不同结果的情况。进而编解码运算结果一致，解码端可以正常解码。
下面以视频编解码过程中编码帧内预测处理为例进行本实施例的介绍：
本公开一示例性实施例提供了一种对视频帧数据进行处理的方法，如图6所示，该方法的处理流程可以包括如下的步骤：
步骤S610，将预先训练的神经网络中的权重参数的数据类型转换为定点型， 得到转换后的权重参数。
其中，本实施例的神经网络为用于在视频编解码过程中对视频帧数据进行编码帧内预测处理的算法模型。具体将权重参数的数据类型转换为定点型的方法可以参见步骤S410-步骤S430对应的实施例中提供的转换方法。
可选地，步骤S610可以包括：对于预先训练的卷积神经网络中的每个卷积核，确定卷积核中绝对值最大的卷积核元素；对于卷积神经网络中的多个偏置，确定多个偏置中绝对值最大的偏置；根据每个卷积核中绝对值最大的卷积核元素、以及预设的定点型的卷积核元素的数据位宽，将每个卷积核中的卷积核元素的数据类型转换为定点型，根据多个偏置中绝对值最大的偏置、以及预设的定点型的偏置的数据位宽，将多个偏置的数据类型转换为定点型，得到转换后的权重参数。
步骤S620，将在视频编码过程中原始未处理的视频帧图像中的目标区域的图像、以及在原始未处理的视频帧图像对应的重建处理得到的视频帧图像中与目标区域对应的关联区域的图像的数据类型转换为定点型，得到转换后的视频帧数据。
可以通过视频拍摄装置拍摄原始未处理的视频帧图像。可以将原始未处理的视频帧图像分为预设的区域数量个，与目标区域的图像相邻的区域可以作为与目标区域对应的关联区域。由于一张图像中相邻的像素点或者区域具有相似性，因此在得知与目标区域对应的关联区域的图像之后，再基于其他信息，可以将目标区域的图像恢复出来。具体将图像的数据类型转换为定点型的方法可以参见步骤S410-步骤S430对应的实施例中提供的转换方法。
可选地，步骤S620可以包括：根据预设的定点型的视频帧数据的数据位宽、以及预先统计的卷积神经网络的输入层输出的特征数据中绝对值最大的数据，将待进行目标处理的视频帧数据的数据类型转换为定点型，得到转换后的视频帧数据。
步骤S630，将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，进行编码帧内预测处理，得到帧内预测图像和帧内预测相关信息。
其中，帧内预测相关信息可以是基于该信息和与目标区域对应的关联区域的图像可以将目标区域的图像恢复出来的信息。帧内预测图像可以是基于帧内预测相关信息和与目标区域对应的关联区域的图像将目标区域的图像恢复出来 的图像。但是恢复出来的图像毕竟是经过了压缩的，因此不可能完全与原来的图像即目标区域的图像一致。因此，可以将目标区域的图像预测出来得到帧内预测图像，再将帧内预测图像和目标区域的图像进行比对，得到预测残差，即恢复的图像和原来的图像的差别信息。
编码帧内预测模块可以向加法器输出帧内预测图像，可以向熵编码器输出帧内预测相关信息。
具体使用神经网络进行编码帧内预测处理的过程与去失真滤波处理的过程类似，可以参见去失真滤波处理的实施例的介绍，在此不再赘述。
通过本公开实施例提供的方法，将预先训练的神经网络中的权重参数的数据类型转换为定点型，得到转换后的权重参数；将在视频编码过程中原始未处理的视频帧图像中的目标区域的图像、以及在原始未处理的视频帧图像对应的重建处理得到的视频帧图像中与目标区域对应的关联区域的图像的数据类型转换为定点型，得到转换后的视频帧数据；将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，进行编码帧内预测处理，得到帧内预测图像和帧内预测相关信息。这样，将浮点型的数据转换为定点型的数据，定点型的数据的小数点位置固定，无需对运算过程中的结果进行约束，不会出现对相同数据进行相同运算却出现不同结果的情况。进而编解码运算结果一致，解码端可以正常解码。
下面以视频编解码过程中编码帧间预测处理为例进行本实施例的介绍：
本公开一示例性实施例提供了一种对视频帧数据进行处理的方法，如图7所示，该方法的处理流程可以包括如下的步骤：
步骤S710，将预先训练的神经网络中的权重参数的数据类型转换为定点型，得到转换后的权重参数。
其中，本实施例的神经网络为用于在视频编解码过程中对视频帧数据进行编码帧间预测处理的算法模型。具体将权重参数的数据类型转换为定点型的方法可以参见步骤S410-步骤S430对应的实施例中提供的转换方法。
可选地，步骤S710可以包括：对于预先训练的卷积神经网络中的每个卷积核，确定卷积核中绝对值最大的卷积核元素；对于卷积神经网络中的多个偏置，确定多个偏置中绝对值最大的偏置；根据每个卷积核中绝对值最大的卷积核元 素、以及预设的定点型的卷积核元素的数据位宽，将每个卷积核中的卷积核元素的数据类型转换为定点型，根据多个偏置中绝对值最大的偏置、以及预设的定点型的偏置的数据位宽，将多个偏置的数据类型转换为定点型，得到转换后的权重参数。
步骤S720，将在视频编码过程中原始未处理的视频帧图像、以及原始未处理的视频帧图像对应的去失真滤波处理后的参考帧图像的数据类型转换为定点型，得到转换后的视频帧数据。
去失真滤波模块向编码帧间预测模块输入原始未处理的视频帧图像对应的去失真滤波处理后的参考帧图像。由于相邻的几张图像具有相似性，因此在得知参考帧图像之后，再基于其他信息，可以将参考帧图像对应的原始未处理的视频帧图像恢复出来。具体将图像的数据类型转换为定点型的方法可以参见步骤S410-步骤S430对应的实施例中提供的转换方法。
可选地，步骤S720可以包括：根据预设的定点型的视频帧数据的数据位宽、以及预先统计的卷积神经网络的输入层输出的特征数据中绝对值最大的数据，将待进行目标处理的视频帧数据的数据类型转换为定点型，得到转换后的视频帧数据。
步骤S730，将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，进行编码帧间预测处理，得到帧间预测图像和帧间预测相关信息。
其中，帧间预测相关信息可以是运动补偿中的运动向量，即参考帧图像经过怎么样的位移就可以得到参考帧图像对应的原始未处理的视频帧图像。帧间预测图像可以是基于帧间预测相关信息和与参考帧图像恢复出来的图像。但是恢复出来的图像毕竟是经过了压缩的，因此不可能完全与原来的图像即参考帧图像一致。因此，可以将参考帧图像预测出来得到帧间预测图像，再将帧间预测图像和参考帧图像对应的原始未处理的视频帧图像进行比对，得到预测残差，即恢复的图像和原来的图像的差别信息。
帧间预测图像和帧间预测相关信息分别被输入到加法器、熵编码器中。
具体使用神经网络进行编码帧间预测处理的过程与去失真滤波处理的过程类似，可以参见去失真滤波处理的实施例的介绍，在此不再赘述。
通过本公开实施例提供的方法，将预先训练的神经网络中的权重参数的数据类型转换为定点型，得到转换后的权重参数；将在视频编码过程中原始未处 理的视频帧图像、以及原始未处理的视频帧图像对应的去失真滤波处理后的参考帧图像的数据类型转换为定点型，得到转换后的视频帧数据；将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，进行编码帧间预测处理，得到帧间预测图像和帧间预测相关信息。这样，将浮点型的数据转换为定点型的数据，定点型的数据的小数点位置固定，无需对运算过程中的结果进行约束，不会出现对相同数据进行相同运算却出现不同结果的情况。进而编解码运算结果一致，解码端可以正常解码。
下面以视频编解码过程中熵编码处理为例进行本实施例的介绍：
本公开一示例性实施例提供了一种对视频帧数据进行处理的方法，如图8所示，该方法的处理流程可以包括如下的步骤：
步骤S810，将预先训练的神经网络中的权重参数的数据类型转换为定点型，得到转换后的权重参数。
其中，本实施例的神经网络为用于在视频编解码过程中对视频帧数据进行熵编码处理的算法模型。具体将权重参数的数据类型转换为定点型的方法可以参见步骤S410-步骤S430对应的实施例中提供的转换方法。
可选地，步骤S810可以包括：对于预先训练的卷积神经网络中的每个卷积核，确定卷积核中绝对值最大的卷积核元素；对于卷积神经网络中的多个偏置，确定多个偏置中绝对值最大的偏置；根据每个卷积核中绝对值最大的卷积核元素、以及预设的定点型的卷积核元素的数据位宽，将每个卷积核中的卷积核元素的数据类型转换为定点型，根据多个偏置中绝对值最大的偏置、以及预设的定点型的偏置的数据位宽，将多个偏置的数据类型转换为定点型，得到转换后的权重参数。
步骤S820，将在视频编码过程中得到的帧内预测相关信息、帧间预测相关信息和量化系数的数据类型转换为定点型，得到转换后的视频帧数据。
其中，量化系数可以是量化模块输出的数据。
编码帧内预测模块、编码帧间预测模块、量化模块分别向熵编码器输入帧内预测相关信息、帧间预测相关信息和量化系数。具体将图像的数据类型转换为定点型的方法可以参见步骤S410-步骤S430对应的实施例中提供的转换方法。
可选地，步骤S820可以包括：根据预设的定点型的视频帧数据的数据位宽、 以及预先统计的卷积神经网络的输入层输出的特征数据中绝对值最大的数据，将待进行目标处理的视频帧数据的数据类型转换为定点型，得到转换后的视频帧数据。
步骤S830，将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，进行熵编码处理，得到熵编码信息。
熵编码信息被映射为码流，被输出到解码端。
可选地，本实施例提供的方法还可以包括：将目标处理后的视频帧数据的数据类型转换为浮点型，得到浮点型的视频帧数据。
可选地，在执行将目标处理后的视频帧数据的数据类型转换为浮点型之后，还可以：将熵编码信息映射为用于向解码端发送的码流。
在卷积神经网络应用于熵编码器的情况下，可以预先存储浮点型的视频帧数据的范围与二进制码流的对应关系，基于浮点型的视频帧数据所属的范围对应的目标二进制码流。将每个浮点型的视频帧数据都映射为目标二进制码流，得到用于向解码端发送的码流。
具体使用神经网络进行熵编码处理的过程与去失真滤波处理的过程类似，可以参见去失真滤波处理的实施例的介绍，在此不再赘述。
通过本公开实施例提供的方法，将预先训练的神经网络中的权重参数的数据类型转换为定点型，得到转换后的权重参数；将在视频编码过程中得到的帧内预测相关信息、帧间预测相关信息和量化系数的数据类型转换为定点型，得到转换后的视频帧数据；将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，进行熵编码处理，得到熵编码信息。这样，将浮点型的数据转换为定点型的数据，定点型的数据的小数点位置固定，无需对运算过程中的结果进行约束，不会出现对相同数据进行相同运算却出现不同结果的情况。进而编解码运算结果一致，解码端可以正常解码。
下面以视频编解码过程中熵解码处理为例进行本实施例的介绍：
本公开一示例性实施例提供了一种对视频帧数据进行处理的方法，如图9所示，该方法的处理流程可以包括如下的步骤：
步骤S910，将预先训练的神经网络中的权重参数的数据类型转换为定点型，得到转换后的权重参数。
其中，本实施例的神经网络为用于在视频编解码过程中对视频帧数据进行熵解码处理的算法模型。具体将权重参数的数据类型转换为定点型的方法可以参见步骤S410-步骤S430对应的实施例中提供的转换方法。
可选地，步骤S910可以包括：对于预先训练的卷积神经网络中的每个卷积核，确定卷积核中绝对值最大的卷积核元素；对于卷积神经网络中的多个偏置，确定多个偏置中绝对值最大的偏置；根据每个卷积核中绝对值最大的卷积核元素、以及预设的定点型的卷积核元素的数据位宽，将每个卷积核中的卷积核元素的数据类型转换为定点型，根据多个偏置中绝对值最大的偏置、以及预设的定点型的偏置的数据位宽，将多个偏置的数据类型转换为定点型，得到转换后的权重参数。
步骤S920，将在视频解码过程中获取的熵编码信息的数据类型转换为定点型，得到转换后的视频帧数据。
编码端可以向解码端的熵解码器输入熵编码信息。具体将图像的数据类型转换为定点型的方法可以参见步骤S410-步骤S430对应的实施例中提供的转换方法。
可选地，步骤S920可以包括：根据预设的定点型的视频帧数据的数据位宽、以及预先统计的卷积神经网络的输入层输出的特征数据中绝对值最大的数据，将待进行目标处理的视频帧数据的数据类型转换为定点型，得到转换后的视频帧数据。
步骤S930，将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，进行熵解码处理，得到帧内预测相关信息、帧间预测相关信息和量化系数。
熵解码器可以向解码帧内预测模块、解码帧间预测模块输出帧内预测相关信息、帧间预测相关信息，向反量化模块输出量化系数。
具体使用神经网络进行熵解码处理的过程与去失真滤波处理的过程类似，可以参见去失真滤波处理的实施例的介绍，在此不再赘述。
通过本公开实施例提供的方法，将预先训练的神经网络中的权重参数的数据类型转换为定点型，得到转换后的权重参数；将在视频解码过程中获取的熵编码信息的数据类型转换为定点型，得到转换后的视频帧数据；将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，进行熵解码处理，得到帧内预测相关信息、帧间预测相关信息和量化系数。这样，将浮点型的数据转换 为定点型的数据，定点型的数据的小数点位置固定，无需对运算过程中的结果进行约束，不会出现对相同数据进行相同运算却出现不同结果的情况。进而编解码运算结果一致，解码端可以正常解码。
下面以视频编解码过程中解码帧内预测处理为例进行本实施例的介绍：
本公开一示例性实施例提供了一种对视频帧数据进行处理的方法，如图10所示，该方法的处理流程可以包括如下的步骤：
步骤S1010，将预先训练的神经网络中的权重参数的数据类型转换为定点型，得到转换后的权重参数。
其中，本实施例的神经网络为用于在视频编解码过程中对视频帧数据进行解码帧内预测处理的算法模型。具体将权重参数的数据类型转换为定点型的方法可以参见步骤S410-步骤S430对应的实施例中提供的转换方法。
可选地，步骤S1010可以包括：对于预先训练的卷积神经网络中的每个卷积核，确定卷积核中绝对值最大的卷积核元素；对于卷积神经网络中的多个偏置，确定多个偏置中绝对值最大的偏置；根据每个卷积核中绝对值最大的卷积核元素、以及预设的定点型的卷积核元素的数据位宽，将每个卷积核中的卷积核元素的数据类型转换为定点型，根据多个偏置中绝对值最大的偏置、以及预设的定点型的偏置的数据位宽，将多个偏置的数据类型转换为定点型，得到转换后的权重参数。
步骤S1020，将在视频解码过程中重建处理得到的视频帧图像中与目标区域对应的关联区域的图像和帧内预测相关信息的数据类型转换为定点型，得到转换后的视频帧数据。
重建模块可以向解码帧内预测模块输入重建处理得到的视频帧图像中与目标区域对应的关联区域的图像。熵解码器可以向解码帧内预测模块输入帧内预测相关信息。具体将图像的数据类型转换为定点型的方法可以参见步骤S410-步骤S430对应的实施例中提供的转换方法。
可选地，步骤S1020可以包括：根据预设的定点型的视频帧数据的数据位宽、以及预先统计的卷积神经网络的输入层输出的特征数据中绝对值最大的数据，将待进行目标处理的视频帧数据的数据类型转换为定点型，得到转换后的视频帧数据。
步骤S1030，将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，进行解码帧内预测处理，得到目标区域的帧内预测图像。
其中，解码帧内预测模块可以向重建模块输出目标区域的帧内预测图像。
具体使用神经网络进行解码帧内预测处理的过程与去失真滤波处理的过程类似，可以参见去失真滤波处理的实施例的介绍，在此不再赘述。
通过本公开实施例提供的方法，将预先训练的神经网络中的权重参数的数据类型转换为定点型，得到转换后的权重参数；将在视频解码过程中重建处理得到的视频帧图像中与目标区域对应的关联区域的图像和帧内预测相关信息的数据类型转换为定点型，得到转换后的视频帧数据；将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，进行解码帧内预测处理，得到目标区域的帧内预测图像。这样，将浮点型的数据转换为定点型的数据，定点型的数据的小数点位置固定，无需对运算过程中的结果进行约束，不会出现对相同数据进行相同运算却出现不同结果的情况。进而编解码运算结果一致，解码端可以正常解码。
下面以视频编解码过程中解码帧间预测处理为例进行本实施例的介绍：
本公开一示例性实施例提供了一种对视频帧数据进行处理的方法，如图11所示，该方法的处理流程可以包括如下的步骤：
步骤S1110，将预先训练的神经网络中的权重参数的数据类型转换为定点型，得到转换后的权重参数。
其中，本实施例的神经网络为用于在视频编解码过程中对视频帧数据进行解码帧间预测处理的算法模型。具体将权重参数的数据类型转换为定点型的方法可以参见步骤S410-步骤S430对应的实施例中提供的转换方法。
可选地，步骤S1110可以包括：对于预先训练的卷积神经网络中的每个卷积核，确定卷积核中绝对值最大的卷积核元素；对于卷积神经网络中的多个偏置，确定多个偏置中绝对值最大的偏置；根据每个卷积核中绝对值最大的卷积核元素、以及预设的定点型的卷积核元素的数据位宽，将每个卷积核中的卷积核元素的数据类型转换为定点型，根据多个偏置中绝对值最大的偏置、以及预设的定点型的偏置的数据位宽，将多个偏置的数据类型转换为定点型，得到转换后的权重参数。
步骤S1120，将在视频解码过程中去失真滤波处理后的参考帧图像和帧间预测相关信息的数据类型转换为定点型，得到转换后的视频帧数据。
滤波模块可以向解码帧间预测模块输入在视频解码过程中去失真滤波处理后的参考帧图像，熵解码器可以向解码帧间预测模块输入帧间预测相关信息。具体将图像的数据类型转换为定点型的方法可以参见步骤S410-步骤S430对应的实施例中提供的转换方法。
可选地，步骤S1120可以包括：根据预设的定点型的视频帧数据的数据位宽、以及预先统计的卷积神经网络的输入层输出的特征数据中绝对值最大的数据，将待进行目标处理的视频帧数据的数据类型转换为定点型，得到转换后的视频帧数据。
步骤S1130，将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，进行解码帧间预测处理，得到帧间预测图像。
解码帧间预测模块可以向重建模块输出帧间预测图像。
具体使用神经网络进行解码帧间预测处理的过程与去失真滤波处理的过程类似，可以参见去失真滤波处理的实施例的介绍，在此不再赘述。
通过本公开实施例提供的方法，将预先训练的神经网络中的权重参数的数据类型转换为定点型，得到转换后的权重参数；将在视频解码过程中去失真滤波处理后的参考帧图像和帧间预测相关信息的数据类型转换为定点型，得到转换后的视频帧数据；将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，进行解码帧间预测处理，得到帧间预测图像。这样，将浮点型的数据转换为定点型的数据，定点型的数据的小数点位置固定，无需对运算过程中的结果进行约束，不会出现对相同数据进行相同运算却出现不同结果的情况。进而编解码运算结果一致，解码端可以正常解码。
本公开又一示例性实施例提供了一种对视频帧数据进行处理的装置，如图12所示，该装置包括：
第一转换模块1210，用于将预先训练的神经网络中的权重参数的数据类型转换为定点型，得到转换后的权重参数，其中，所述神经网络为用于在视频编解码过程中对视频帧数据进行目标处理的算法模型；
第二转换模块1220，用于将待进行目标处理的视频帧数据的数据类型转换 为定点型，得到转换后的视频帧数据；
输入模块1230，用于将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，得到目标处理后的视频帧数据。
可选地，所述神经网络为卷积神经网络，所述权重参数包括卷积核元素和偏置。
可选地，所述第一转换模块1210包括：
第一确定单元，用于对于预先训练的卷积神经网络中的每个卷积核，确定所述卷积核中绝对值最大的卷积核元素；
第二确定单元，用于对于所述卷积神经网络中的多个偏置，确定所述多个偏置中绝对值最大的偏置；
转换单元，用于根据每个卷积核中绝对值最大的卷积核元素、以及预设的定点型的卷积核元素的数据位宽，将每个卷积核中的卷积核元素的数据类型转换为定点型，根据多个偏置中绝对值最大的偏置、以及预设的定点型的偏置的数据位宽，将所述多个偏置的数据类型转换为定点型，得到转换后的权重参数。
可选地，所述第二转换模块1220，用于根据预设的定点型的视频帧数据的数据位宽、以及预先统计的所述卷积神经网络的输入层输出的特征数据中绝对值最大的数据，将待进行目标处理的视频帧数据的数据类型转换为定点型，得到转换后的视频帧数据。
可选地，所述装置还包括：
第三转换模块，用于将预设的所述视频帧数据的边信息的数据类型转换为定点型，得到转换后的边信息；
所述输入模块，用于将转换后的视频帧数据和转换后的边信息，输入加载了转换后的权重参数的神经网络，得到目标处理后的视频帧数据。
可选地，所述装置还包括：
取整模块，用于对目标处理后的视频帧数据进行取整处理，得到整型的视频帧数据。
可选地，所述目标处理为去失真滤波处理；
所述第二转换模块1220，用于将在视频编解码过程中进行重建处理得到的视频帧图像的数据类型转换为定点型，得到转换后的视频帧数据；
所述输入模块1230，用于将转换后的视频帧数据输入加载了转换后的权重 参数的神经网络，进行去失真滤波处理，得到去失真的视频帧图像。
可选地，所述目标处理为编码帧内预测处理；
所述第二转换模块1220，用于将在视频编码过程中原始未处理的视频帧图像中的目标区域的图像、以及在所述原始未处理的视频帧图像对应的重建处理得到的视频帧图像中与所述目标区域对应的关联区域的图像的数据类型转换为定点型，得到转换后的视频帧数据；
所述输入模块1230，用于将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，进行编码帧内预测处理，得到帧内预测图像和帧内预测相关信息。
可选地，所述目标处理为编码帧间预测处理；
所述第二转换模块1220，用于将在视频编码过程中原始未处理的视频帧图像、以及所述原始未处理的视频帧图像对应的去失真滤波处理后的参考帧图像的数据类型转换为定点型，得到转换后的视频帧数据；
所述输入模块1230，用于将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，进行编码帧间预测处理，得到帧间预测图像和帧间预测相关信息。
可选地，所述目标处理为熵编码处理；
所述第二转换模块1220，用于将在视频编码过程中得到的帧内预测相关信息、帧间预测相关信息和量化系数的数据类型转换为定点型，得到转换后的视频帧数据；
所述输入模块1230，用于将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，进行熵编码处理，得到熵编码信息。
可选地，所述目标处理为熵解码处理；
所述第二转换模块1220，用于将在视频解码过程中获取的熵编码信息的数据类型转换为定点型，得到转换后的视频帧数据；
所述输入模块1230，用于将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，进行熵解码处理，得到帧内预测相关信息、帧间预测相关信息和量化系数。
可选地，所述目标处理为解码帧内预测处理；
所述第二转换模块1220，用于将在视频解码过程中重建处理得到的视频帧 图像中与目标区域对应的关联区域的图像和帧内预测相关信息的数据类型转换为定点型，得到转换后的视频帧数据；
所述输入模块1230，用于将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，进行解码帧内预测处理，得到目标区域的帧内预测图像。
可选地，所述目标处理为解码帧间预测处理；
所述第二转换模块1220，用于将在视频解码过程中去失真滤波处理后的参考帧图像和帧间预测相关信息的数据类型转换为定点型，得到转换后的视频帧数据；
所述输入模块1230，用于将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，进行解码帧间预测处理，得到帧间预测图像。
关于上述实施例中的装置，其中各个模块执行操作的具体方式已经在有关该方法的实施例中进行了详细描述，此处将不做详细阐述说明。
将浮点型的数据转换为定点型的数据，定点型的数据的小数点位置固定，无需对运算过程中的结果进行约束，不会出现对相同数据进行相同运算却出现不同结果的情况。进而编解码运算结果一致，解码端可以正常解码。
需要说明的是：上述实施例提供的对视频帧数据进行处理的装置在对视频帧图像进行处理时，仅以上述各功能模块的划分进行举例说明，实际应用中，可以根据需要而将上述功能分配由不同的功能模块完成，即将终端的内部结构划分成不同的功能模块，以完成以上描述的全部或者部分功能。另外，上述实施例提供的对视频帧数据进行处理的装置与对视频帧数据进行处理的方法实施例属于同一构思，其具体实现过程详见方法实施例，这里不再赘述。
图13示出了本公开一个示例性实施例提供的终端1800的结构示意图。该终端1800可以是：机顶盒、智能手机、平板电脑、MP4(Moving Picture Experts Group Audio Layer IV，动态影像专家压缩标准音频层面4)播放器、笔记本电脑或台式电脑。终端1800还可能被称为用户设备、便携式终端、膝上型终端、台式终端等其他名称。
通常，终端1800包括有：处理器1801和存储器1802。
处理器1801可以包括一个或多个处理核心，比如4核心处理器、8核心处理器等。处理器1801可以采用DSP(Digital Signal Processing，数字信号处理)、 FPGA(Field－Programmable Gate Array，现场可编程门阵列)、PLA(Programmable Logic Array，可编程逻辑阵列)中的至少一种硬件形式来实现。处理器1801也可以包括主处理器和协处理器，主处理器是用于对在唤醒状态下的数据进行处理的处理器，也称CPU(Central Processing Unit，中央处理器)；协处理器是用于对在待机状态下的数据进行处理的低功耗处理器。在一些实施例中，处理器1801可以在集成有GPU(Graphics Processing Unit，图像处理器)，GPU用于负责显示屏所需要显示的内容的渲染和绘制。一些实施例中，处理器1801还可以包括AI(Artificial Intelligence，人工智能)处理器，该AI处理器用于处理有关机器学习的计算操作。
存储器1802可以包括一个或多个计算机可读存储介质，该计算机可读存储介质可以是非暂态的。存储器1802还可包括高速随机存取存储器，以及非易失性存储器，比如一个或多个磁盘存储设备、闪存存储设备。在一些实施例中，存储器1802中的非暂态的计算机可读存储介质用于存储至少一个指令，该至少一个指令用于被处理器1801所执行以实现本申请中方法实施例提供的对视频帧数据进行处理的方法。
在一些实施例中，终端1800还可选包括有：外围设备接口1803和至少一个外围设备。处理器1801、存储器1802和外围设备接口1803之间可以通过总线或信号线相连。各个外围设备可以通过总线、信号线或电路板与外围设备接口1803相连。具体地，外围设备包括：射频电路1804、触摸显示屏1805、摄像头1806、音频电路1807、定位组件1808和电源1809中的至少一种。
外围设备接口1803可被用于将I/O(Input/Output，输入/输出)相关的至少一个外围设备连接到处理器1801和存储器1802。在一些实施例中，处理器1801、存储器1802和外围设备接口1803被集成在同一芯片或电路板上；在一些其他实施例中，处理器1801、存储器1802和外围设备接口1803中的任意一个或两个可以在单独的芯片或电路板上实现，本实施例对此不加以限定。
射频电路1804用于接收和发射RF(Radio Frequency，射频)信号，也称电磁信号。射频电路1804通过电磁信号与通信网络以及其他通信设备进行通信。射频电路1804将电信号转换为电磁信号进行发送，或者，将接收到的电磁信号转换为电信号。可选地，射频电路1804包括：天线系统、RF收发器、一个或多个放大器、调谐器、振荡器、数字信号处理器、编解码芯片组、用户身份模 块卡等等。射频电路1804可以通过至少一种无线通信协议来与其它终端进行通信。该无线通信协议包括但不限于：万维网、城域网、内联网、各代移动通信网络(2G、3G、4G及5G)、无线局域网和/或WiFi(Wireless Fidelity，无线保真)网络。在一些实施例中，射频电路1804还可以包括NFC(Near Field Communication，近距离无线通信)有关的电路，本申请对此不加以限定。
显示屏1805用于显示UI(UserInterface，用户界面)。该UI可以包括图形、文本、图标、视频及其它们的任意组合。当显示屏1805是触摸显示屏时，显示屏1805还具有采集在显示屏1805的表面或表面上方的触摸信号的能力。该触摸信号可以作为控制信号输入至处理器1801进行处理。此时，显示屏1805还可以用于提供虚拟按钮和/或虚拟键盘，也称软按钮和/或软键盘。在一些实施例中，显示屏1805可以为一个，设置终端1800的前面板；在另一些实施例中，显示屏1805可以为至少两个，分别设置在终端1800的不同表面或呈折叠设计；在再一些实施例中，显示屏1805可以是柔性显示屏，设置在终端1800的弯曲表面上或折叠面上。甚至，显示屏1805还可以设置成非矩形的不规则图形，也即异形屏。显示屏1805可以采用LCD(Liquid Crystal Display，液晶显示屏)、OLED(Organic Light-Emitting Diode,有机发光二极管)等材质制备。
摄像头组件1806用于采集图像或视频。可选地，摄像头组件1806包括前置摄像头和后置摄像头。通常，前置摄像头设置在终端的前面板，后置摄像头设置在终端的背面。在一些实施例中，后置摄像头为至少两个，分别为主摄像头、景深摄像头、广角摄像头、长焦摄像头中的任意一种，以实现主摄像头和景深摄像头融合实现背景虚化功能、主摄像头和广角摄像头融合实现全景拍摄以及VR(Virtual Reality，虚拟现实)拍摄功能或者其它融合拍摄功能。在一些实施例中，摄像头组件1806还可以包括闪光灯。闪光灯可以是单色温闪光灯，也可以是双色温闪光灯。双色温闪光灯是指暖光闪光灯和冷光闪光灯的组合，可以用于不同色温下的光线补偿。
音频电路1807可以包括麦克风和扬声器。麦克风用于采集用户及环境的声波，并将声波转换为电信号输入至处理器1801进行处理，或者输入至射频电路1804以实现语音通信。出于立体声采集或降噪的目的，麦克风可以为多个，分别设置在终端1800的不同部位。麦克风还可以是阵列麦克风或全向采集型麦克风。扬声器则用于将来自处理器1801或射频电路1804的电信号转换为声波。 扬声器可以是传统的薄膜扬声器，也可以是压电陶瓷扬声器。当扬声器是压电陶瓷扬声器时，不仅可以将电信号转换为人类可听见的声波，也可以将电信号转换为人类听不见的声波以进行测距等用途。在一些实施例中，音频电路1807还可以包括耳机插孔。
定位组件1808用于定位终端1800的当前地理位置，以实现导航或LBS(Location Based Service，基于位置的服务)。定位组件1808可以是基于美国的GPS(Global Positioning System，全球定位系统)、中国的北斗系统或俄罗斯的伽利略系统的定位组件。
电源1809用于为终端1800中的各个组件进行供电。电源1809可以是交流电、直流电、一次性电池或可充电电池。当电源1809包括可充电电池时，该可充电电池可以是有线充电电池或无线充电电池。有线充电电池是通过有线线路充电的电池，无线充电电池是通过无线线圈充电的电池。该可充电电池还可以用于支持快充技术。
在一些实施例中，终端1800还包括有一个或多个传感器1810。该一个或多个传感器1810包括但不限于：加速度传感器1811、陀螺仪传感器1812、压力传感器1813、指纹传感器1814、光学传感器1815以及接近传感器1816。
加速度传感器1811可以检测以终端1800建立的坐标系的三个坐标轴上的加速度大小。比如，加速度传感器1811可以用于检测重力加速度在三个坐标轴上的分量。处理器1801可以根据加速度传感器1811采集的重力加速度信号，控制触摸显示屏1805以横向视图或纵向视图进行用户界面的显示。加速度传感器1811还可以用于游戏或者用户的运动数据的采集。
陀螺仪传感器1812可以检测终端1800的机体方向及转动角度，陀螺仪传感器1812可以与加速度传感器1811协同采集用户对终端1800的3D动作。处理器1801根据陀螺仪传感器1812采集的数据，可以实现如下功能：动作感应(比如根据用户的倾斜操作来改变UI)、拍摄时的图像稳定、游戏控制以及惯性导航。
压力传感器1813可以设置在终端1800的侧边框和/或触摸显示屏1805的下层。当压力传感器1813设置在终端1800的侧边框时，可以检测用户对终端1800的握持信号，由处理器1801根据压力传感器1813采集的握持信号进行左右手识别或快捷操作。当压力传感器1813设置在触摸显示屏1805的下层时，由处 理器1801根据用户对触摸显示屏1805的压力操作，实现对UI界面上的可操作性控件进行控制。可操作性控件包括按钮控件、滚动条控件、图标控件、菜单控件中的至少一种。
指纹传感器1814用于采集用户的指纹，由处理器1801根据指纹传感器1814采集到的指纹识别用户的身份，或者，由指纹传感器1814根据采集到的指纹识别用户的身份。在识别出用户的身份为可信身份时，由处理器1801授权该用户执行相关的敏感操作，该敏感操作包括解锁屏幕、查看加密信息、下载软件、支付及更改设置等。指纹传感器1814可以被设置终端1800的正面、背面或侧面。当终端1800上设置有物理按键或厂商Logo时，指纹传感器1814可以与物理按键或厂商Logo集成在一起。
光学传感器1815用于采集环境光强度。在一个实施例中，处理器1801可以根据光学传感器1815采集的环境光强度，控制触摸显示屏1805的显示亮度。具体地，当环境光强度较高时，调高触摸显示屏1805的显示亮度；当环境光强度较低时，调低触摸显示屏1805的显示亮度。在另一个实施例中，处理器1801还可以根据光学传感器1815采集的环境光强度，动态调整摄像头组件1806的拍摄参数。
接近传感器1816，也称距离传感器，通常设置在终端1800的前面板。接近传感器1816用于采集用户与终端1800的正面之间的距离。在一个实施例中，当接近传感器1816检测到用户与终端1800的正面之间的距离逐渐变小时，由处理器1801控制触摸显示屏1805从亮屏状态切换为息屏状态；当接近传感器1816检测到用户与终端1800的正面之间的距离逐渐变大时，由处理器1801控制触摸显示屏1805从息屏状态切换为亮屏状态。
本领域技术人员可以理解，图13中示出的结构并不构成对终端1800的限定，可以包括比图示更多或更少的组件，或者组合某些组件，或者采用不同的组件布置。
本领域技术人员在考虑说明书及实践这里公开的公开后，将容易想到本公开的其它实施方案。本申请旨在涵盖本公开的任何变型、用途或者适应性变化，这些变型、用途或者适应性变化遵循本公开的一般性原理并包括本公开未公开的本技术领域中的公知常识或惯用技术手段。说明书和实施例仅被视为示例性 的，本公开的真正范围和精神由权利要求指出。
应当理解的是，本公开并不局限于上面已经描述并在附图中示出的精确结构，并且可以在不脱离其范围进行各种修改和改变。本公开的范围仅由所附的权利要求来限制。

Claims (28)

  1. 一种对视频帧数据进行处理的方法，其特征在于，所述方法包括：
    将预先训练的神经网络中的权重参数的数据类型转换为定点型，得到转换后的权重参数，其中，所述神经网络为用于在视频编解码过程中对视频帧数据进行目标处理的算法模型；
    将待进行目标处理的视频帧数据的数据类型转换为定点型，得到转换后的视频帧数据；
    将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，得到目标处理后的视频帧数据。
  2. 根据权利要求1所述的方法，其特征在于，所述神经网络为卷积神经网络，所述权重参数包括卷积核元素和偏置。
  3. 根据权利要求2所述的方法，其特征在于，所述将预先训练的神经网络中的权重参数的数据类型转换为定点型，得到转换后的权重参数，包括：
    对于预先训练的卷积神经网络中的每个卷积核，确定所述卷积核中绝对值最大的卷积核元素；
    对于所述卷积神经网络中的多个偏置，确定所述多个偏置中绝对值最大的偏置；
    根据每个卷积核中绝对值最大的卷积核元素、以及预设的定点型的卷积核元素的数据位宽，将每个卷积核中的卷积核元素的数据类型转换为定点型，根据多个偏置中绝对值最大的偏置、以及预设的定点型的偏置的数据位宽，将所述多个偏置的数据类型转换为定点型，得到转换后的权重参数。
  4. 根据权利要求2所述的方法，其特征在于，所述将待进行目标处理的视频帧数据的数据类型转换为定点型，得到转换后的视频帧数据，包括：
    根据预设的定点型的视频帧数据的数据位宽、以及预先统计的所述卷积神经网络的输入层输出的特征数据中绝对值最大的数据，将待进行目标处理的视频帧数据的数据类型转换为定点型，得到转换后的视频帧数据。
  5. 根据权利要求1所述的方法，其特征在于，所述方法还包括：
    将预设的所述视频帧数据的边信息的数据类型转换为定点型，得到转换后的边信息；
    所述将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，得到目标处理后的视频帧数据，包括：
    将转换后的视频帧数据和转换后的边信息，输入加载了转换后的权重参数的神经网络，得到目标处理后的视频帧数据。
  6. 根据权利要求1所述的方法，其特征在于，在将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，得到目标处理后的视频帧数据之后，所述方法还包括：
    对目标处理后的视频帧数据进行取整处理，得到整型的视频帧数据。
  7. 根据权利要求1-6任一项所述的方法，其特征在于，所述目标处理为去失真滤波处理；
    所述将待进行目标处理的视频帧数据的数据类型转换为定点型，得到转换后的视频帧数据，包括：
    将在视频编解码过程中进行重建处理得到的视频帧图像的数据类型转换为定点型，得到转换后的视频帧数据；
    所述将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，得到目标处理后的视频帧数据，包括：
    将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，进行去失真滤波处理，得到去失真的视频帧图像。
  8. 根据权利要求1-6任一项所述的方法，其特征在于，所述目标处理为编码帧内预测处理；
    所述将待进行目标处理的视频帧数据的数据类型转换为定点型，得到转换后的视频帧数据，包括：
    将在视频编码过程中原始未处理的视频帧图像中的目标区域的图像、以及在所述原始未处理的视频帧图像对应的重建处理得到的视频帧图像中与所述目标区域对应的关联区域的图像的数据类型转换为定点型，得到转换后的视频帧数据；
    所述将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，得到目标处理后的视频帧数据，包括：
    将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，进行编码帧内预测处理，得到帧内预测图像和帧内预测相关信息。
  9. 根据权利要求1-6任一项所述的方法，其特征在于，所述目标处理为编码帧间预测处理；
    所述将待进行目标处理的视频帧数据的数据类型转换为定点型，得到转换后的视频帧数据，包括：
    将在视频编码过程中原始未处理的视频帧图像、以及所述原始未处理的视频帧图像对应的去失真滤波处理后的参考帧图像的数据类型转换为定点型，得到转换后的视频帧数据；
    所述将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，得到目标处理后的视频帧数据，包括：
    将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，进行编码帧间预测处理，得到帧间预测图像和帧间预测相关信息。
  10. 根据权利要求1-6任一项所述的方法，其特征在于，所述目标处理为熵编码处理；
    所述将待进行目标处理的视频帧数据的数据类型转换为定点型，得到转换后的视频帧数据，包括：
    将在视频编码过程中得到的帧内预测相关信息、帧间预测相关信息和量化系数的数据类型转换为定点型，得到转换后的视频帧数据；
    所述将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，得到目标处理后的视频帧数据，包括：
    将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，进行熵编码处理，得到熵编码信息。
  11. 根据权利要求1-6任一项所述的方法，其特征在于，所述目标处理为熵解码处理；
    所述将待进行目标处理的视频帧数据的数据类型转换为定点型，得到转换后的视频帧数据，包括：
    将在视频解码过程中获取的熵编码信息的数据类型转换为定点型，得到转换后的视频帧数据；
    所述将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，得到目标处理后的视频帧数据，包括：
    将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，进行熵 解码处理，得到帧内预测相关信息、帧间预测相关信息和量化系数。
  12. 根据权利要求1-6任一项所述的方法，其特征在于，所述目标处理为解码帧内预测处理；
    所述将待进行目标处理的视频帧数据的数据类型转换为定点型，得到转换后的视频帧数据，包括：
    将在视频解码过程中重建处理得到的视频帧图像中与目标区域对应的关联区域的图像和帧内预测相关信息的数据类型转换为定点型，得到转换后的视频帧数据；
    所述将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，得到目标处理后的视频帧数据，包括：
    将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，进行解码帧内预测处理，得到目标区域的帧内预测图像。
  13. 根据权利要求1-6任一项所述的方法，其特征在于，所述目标处理为解码帧间预测处理；
    所述将待进行目标处理的视频帧数据的数据类型转换为定点型，得到转换后的视频帧数据，包括：
    将在视频解码过程中去失真滤波处理后的参考帧图像和帧间预测相关信息的数据类型转换为定点型，得到转换后的视频帧数据；
    所述将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，得到目标处理后的视频帧数据，包括：
    将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，进行解码帧间预测处理，得到帧间预测图像。
  14. 一种对视频帧数据进行处理的装置，其特征在于，所述装置包括：
    第一转换模块，用于将预先训练的神经网络中的权重参数的数据类型转换为定点型，得到转换后的权重参数，其中，所述神经网络为用于在视频编解码过程中对视频帧数据进行目标处理的算法模型；
    第二转换模块，用于将待进行目标处理的视频帧数据的数据类型转换为定点型，得到转换后的视频帧数据；
    输入模块，用于将转换后的视频帧数据输入加载了转换后的权重参数的神 经网络，得到目标处理后的视频帧数据。
  15. 根据权利要求14所述的装置，其特征在于，所述神经网络为卷积神经网络，所述权重参数包括卷积核元素和偏置。
  16. 根据权利要求15所述的装置，其特征在于，所述第一转换模块包括：
    第一确定单元，用于对于预先训练的卷积神经网络中的每个卷积核，确定所述卷积核中绝对值最大的卷积核元素；
    第二确定单元，用于对于所述卷积神经网络中的多个偏置，确定所述多个偏置中绝对值最大的偏置；
    转换单元，用于根据每个卷积核中绝对值最大的卷积核元素、以及预设的定点型的卷积核元素的数据位宽，将每个卷积核中的卷积核元素的数据类型转换为定点型，根据多个偏置中绝对值最大的偏置、以及预设的定点型的偏置的数据位宽，将所述多个偏置的数据类型转换为定点型，得到转换后的权重参数。
  17. 根据权利要求15所述的装置，其特征在于，所述第二转换模块，用于根据预设的定点型的视频帧数据的数据位宽、以及预先统计的所述卷积神经网络的输入层输出的特征数据中绝对值最大的数据，将待进行目标处理的视频帧数据的数据类型转换为定点型，得到转换后的视频帧数据。
  18. 根据权利要求14所述的装置，其特征在于，所述装置还包括：
    第三转换模块，用于将预设的所述视频帧数据的边信息的数据类型转换为定点型，得到转换后的边信息；
    所述输入模块，用于将转换后的视频帧数据和转换后的边信息，输入加载了转换后的权重参数的神经网络，得到目标处理后的视频帧数据。
  19. 根据权利要求14所述的装置，其特征在于，所述装置还包括：
    取整模块，用于对目标处理后的视频帧数据进行取整处理，得到整型的视频帧数据。
  20. 根据权利要求14-19任一项所述的装置，其特征在于，所述目标处理为去失真滤波处理；
    所述第二转换模块，用于将在视频编解码过程中进行重建处理得到的视频帧图像的数据类型转换为定点型，得到转换后的视频帧数据；
    所述输入模块，用于将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，进行去失真滤波处理，得到去失真的视频帧图像。
  21. 根据权利要求14-19任一项所述的装置，其特征在于，所述目标处理为编码帧内预测处理；
    所述第二转换模块，用于将在视频编码过程中原始未处理的视频帧图像中的目标区域的图像、以及在所述原始未处理的视频帧图像对应的重建处理得到的视频帧图像中与所述目标区域对应的关联区域的图像的数据类型转换为定点型，得到转换后的视频帧数据；
    所述输入模块，用于将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，进行编码帧内预测处理，得到帧内预测图像和帧内预测相关信息。
  22. 根据权利要求14-19任一项所述的装置，其特征在于，所述目标处理为编码帧间预测处理；
    所述第二转换模块，用于将在视频编码过程中原始未处理的视频帧图像、以及所述原始未处理的视频帧图像对应的去失真滤波处理后的参考帧图像的数据类型转换为定点型，得到转换后的视频帧数据；
    所述输入模块，用于将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，进行编码帧间预测处理，得到帧间预测图像和帧间预测相关信息。
  23. 根据权利要求14-19任一项所述的装置，其特征在于，所述目标处理为熵编码处理；
    所述第二转换模块，用于将在视频编码过程中得到的帧内预测相关信息、帧间预测相关信息和量化系数的数据类型转换为定点型，得到转换后的视频帧数据；
    所述输入模块，用于将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，进行熵编码处理，得到熵编码信息。
  24. 根据权利要求14-19任一项所述的装置，其特征在于，所述目标处理为熵解码处理；
    所述第二转换模块，用于将在视频解码过程中获取的熵编码信息的数据类型转换为定点型，得到转换后的视频帧数据；
    所述输入模块，用于将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，进行熵解码处理，得到帧内预测相关信息、帧间预测相关信息和量化系数。
  25. 根据权利要求14-19任一项所述的装置，其特征在于，所述目标处理为 解码帧内预测处理；
    所述第二转换模块，用于将在视频解码过程中重建处理得到的视频帧图像中与目标区域对应的关联区域的图像和帧内预测相关信息的数据类型转换为定点型，得到转换后的视频帧数据；
    所述输入模块，用于将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，进行解码帧内预测处理，得到目标区域的帧内预测图像。
  26. 根据权利要求14-19任一项所述的装置，其特征在于，所述目标处理为解码帧间预测处理；
    所述第二转换模块，用于将在视频解码过程中去失真滤波处理后的参考帧图像和帧间预测相关信息的数据类型转换为定点型，得到转换后的视频帧数据；
    所述输入模块，用于将转换后的视频帧数据输入加载了转换后的权重参数的神经网络，进行解码帧间预测处理，得到帧间预测图像。
  27. 一种终端，其特征在于，所述终端包括处理器、通信接口、存储器和通信总线，其中：
    所述处理器、所述通信接口和所述存储器通过所述通信总线完成相互间的通信；
    所述存储器，用于存放计算机程序；
    所述处理器，用于执行所述存储器上所存放的程序，以实现权利要求1-13任一所述的方法步骤。
  28. 一种计算机可读存储介质，其特征在于，所述计算机可读存储介质内存储有计算机程序，所述计算机程序被处理器执行时实现权利要求1-13任一所述的方法步骤。
PCT/CN2019/072033 2018-01-19 2019-01-16 对视频帧数据进行处理的方法和装置 WO2019141193A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201810054242.7A CN110062246B (zh) 2018-01-19 2018-01-19 对视频帧数据进行处理的方法和装置
CN201810054242.7 2018-01-19

Publications (1)

Publication Number Publication Date
WO2019141193A1 true WO2019141193A1 (zh) 2019-07-25

Family

ID=67300944

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/072033 WO2019141193A1 (zh) 2018-01-19 2019-01-16 对视频帧数据进行处理的方法和装置

Country Status (2)

Country Link
CN (1) CN110062246B (zh)
WO (1) WO2019141193A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111598227A (zh) * 2020-05-20 2020-08-28 字节跳动有限公司 数据处理方法、装置、电子设备及计算机可读存储介质
CN112116010A (zh) * 2020-09-21 2020-12-22 中国科学院自动化研究所 基于膜电势预处理的ann-snn转换的分类方法

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110569961A (zh) * 2019-08-08 2019-12-13 合肥图鸭信息科技有限公司 一种神经网络训练方法、装置及终端设备
CN114365147A (zh) * 2019-10-22 2022-04-15 深圳鲲云信息科技有限公司 神经网络计算压缩方法、系统及存储介质
CN111064958B (zh) * 2019-12-28 2021-03-30 复旦大学 一种针对b帧和p帧的低复杂度神经网络滤波算法
CN113780523B (zh) * 2021-08-27 2024-03-29 深圳云天励飞技术股份有限公司 图像处理方法、装置、终端设备及存储介质
CN117501695A (zh) * 2021-12-10 2024-02-02 英特尔公司 用于基于深度学习的视频处理的增强体系结构

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6075884A (en) * 1996-03-29 2000-06-13 Sarnoff Corporation Method and apparatus for training a neural network to learn and use fidelity metric as a control mechanism
CN106502626A (zh) * 2016-11-03 2017-03-15 北京百度网讯科技有限公司 数据处理方法和装置
CN106575379A (zh) * 2014-09-09 2017-04-19 英特尔公司 用于神经网络的改进的定点整型实现方式
CN107197260A (zh) * 2017-06-12 2017-09-22 清华大学深圳研究生院 基于卷积神经网络的视频编码后置滤波方法
CN107239826A (zh) * 2017-06-06 2017-10-10 上海兆芯集成电路有限公司 在卷积神经网络中的计算方法及装置
CN107292334A (zh) * 2017-06-08 2017-10-24 北京深瞐科技有限公司 图像识别方法及装置
CN107368857A (zh) * 2017-07-24 2017-11-21 深圳市图芯智能科技有限公司 图像对象检测方法、系统及模型处理方法、设备、终端

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2537113B2 (ja) * 1992-03-30 1996-09-25 移動通信システム開発株式会社 音声符号化、復号化器における声道パラメ―タ情報の適応的圧縮方法
US6424737B1 (en) * 2000-01-24 2002-07-23 Sony Corporation Method and apparatus of compressing images using localized radon transforms
HUP0301368A3 (en) * 2003-05-20 2005-09-28 Amt Advanced Multimedia Techno Method and equipment for compressing motion picture data
US20060190960A1 (en) * 2005-02-14 2006-08-24 Barker Geoffrey T System and method for incorporating video analytics in a monitoring network
US10373050B2 (en) * 2015-05-08 2019-08-06 Qualcomm Incorporated Fixed point neural network based on floating point neural network quantization
CN107480770B (zh) * 2017-07-27 2020-07-28 中国科学院自动化研究所 可调节量化位宽的神经网络量化与压缩的方法及装置

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6075884A (en) * 1996-03-29 2000-06-13 Sarnoff Corporation Method and apparatus for training a neural network to learn and use fidelity metric as a control mechanism
CN106575379A (zh) * 2014-09-09 2017-04-19 英特尔公司 用于神经网络的改进的定点整型实现方式
CN106502626A (zh) * 2016-11-03 2017-03-15 北京百度网讯科技有限公司 数据处理方法和装置
CN107239826A (zh) * 2017-06-06 2017-10-10 上海兆芯集成电路有限公司 在卷积神经网络中的计算方法及装置
CN107292334A (zh) * 2017-06-08 2017-10-24 北京深瞐科技有限公司 图像识别方法及装置
CN107197260A (zh) * 2017-06-12 2017-09-22 清华大学深圳研究生院 基于卷积神经网络的视频编码后置滤波方法
CN107368857A (zh) * 2017-07-24 2017-11-21 深圳市图芯智能科技有限公司 图像对象检测方法、系统及模型处理方法、设备、终端

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111598227A (zh) * 2020-05-20 2020-08-28 字节跳动有限公司 数据处理方法、装置、电子设备及计算机可读存储介质
CN111598227B (zh) * 2020-05-20 2023-11-03 字节跳动有限公司 数据处理方法、装置、电子设备及计算机可读存储介质
CN112116010A (zh) * 2020-09-21 2020-12-22 中国科学院自动化研究所 基于膜电势预处理的ann-snn转换的分类方法
CN112116010B (zh) * 2020-09-21 2023-12-12 中国科学院自动化研究所 基于膜电势预处理的ann-snn转换的分类方法

Also Published As

Publication number Publication date
CN110062246B (zh) 2021-01-05
CN110062246A (zh) 2019-07-26

Similar Documents

Publication Publication Date Title
WO2019141193A1 (zh) 对视频帧数据进行处理的方法和装置
CN109086709B (zh) 特征提取模型训练方法、装置及存储介质
CN110097019B (zh) 字符识别方法、装置、计算机设备以及存储介质
CN108810538B (zh) 视频编码方法、装置、终端及存储介质
CN110795625B (zh) 推荐方法、装置、计算机设备及存储介质
EP3787291B1 (en) Method and device for video encoding, storage medium, and equipment
CN112907725B (zh) 图像生成、图像处理模型的训练、图像处理方法和装置
CN111107357B (zh) 一种图像处理的方法、装置、系统及存储介质
WO2024016611A1 (zh) 图像处理方法、装置、电子设备及计算机可读存储介质
CN110503160B (zh) 图像识别方法、装置、电子设备及存储介质
CN112614500A (zh) 回声消除方法、装置、设备及计算机存储介质
CN110572710B (zh) 视频生成方法、装置、设备及存储介质
CN114299306A (zh) 获取图像检索模型的方法、图像检索方法、装置和设备
CN110232417B (zh) 图像识别方法、装置、计算机设备及计算机可读存储介质
CN113822955B (zh) 图像数据处理方法、装置、计算机设备及存储介质
CN113891074B (zh) 视频编码方法和装置、电子装置和计算机可读存储介质
CN111698512B (zh) 视频处理方法、装置、设备及存储介质
CN114332709A (zh) 视频处理方法、装置、存储介质以及电子设备
CN110062226B (zh) 一种视频编码方法、视频解码方法、装置、系统及介质
CN110460856B (zh) 视频编码方法、装置、编码设备及计算机可读存储介质
CN111310701B (zh) 手势识别方法、装置、设备及存储介质
CN113079372B (zh) 帧间预测的编码方法、装置、设备及可读存储介质
CN114422782B (zh) 视频编码方法、装置、存储介质及电子设备
CN111641831B (zh) 帧内预测方法、装置、设备及存储介质
WO2019141255A1 (zh) 一种图片滤波的方法及装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19740925

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19740925

Country of ref document: EP

Kind code of ref document: A1