WO2022062388A1 - 一种视频数据处理方法、系统及相关组件 - Google Patents

一种视频数据处理方法、系统及相关组件 Download PDF

Info

Publication number
WO2022062388A1
WO2022062388A1 PCT/CN2021/089924 CN2021089924W WO2022062388A1 WO 2022062388 A1 WO2022062388 A1 WO 2022062388A1 CN 2021089924 W CN2021089924 W CN 2021089924W WO 2022062388 A1 WO2022062388 A1 WO 2022062388A1
Authority
WO
WIPO (PCT)
Prior art keywords
dimensional
matrix
video data
calculation
data processing
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2021/089924
Other languages
English (en)
French (fr)
Inventor
董刚
赵雅倩
李仁刚
杨宏斌
刘海威
蒋东东
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Suzhou Wave Intelligent Technology Co Ltd
Original Assignee
Suzhou Wave Intelligent Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Suzhou Wave Intelligent Technology Co Ltd filed Critical Suzhou Wave Intelligent Technology Co Ltd
Priority to US18/012,939 priority Critical patent/US20230267740A1/en
Publication of WO2022062388A1 publication Critical patent/WO2022062388A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/40—Scenes; Scene-specific elements in video content
    • G06V20/46—Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/213—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/7715—Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/94—Hardware or software architectures specially adapted for image or video understanding
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/40—Scenes; Scene-specific elements in video content
    • G06V20/41—Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/40—Scenes; Scene-specific elements in video content
    • G06V20/49—Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three-dimensional [3D] modelling for computer graphics
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G06V10/443—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components by matching or filtering
    • G06V10/449—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters
    • G06V10/451—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters with interaction between the filter responses, e.g. cortical complex cells
    • G06V10/454—Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/40—Scenes; Scene-specific elements in video content

Definitions

  • the present application relates to the field of video data processing, and in particular, to a video data processing method, system and related components.
  • Video feature extraction is a basic link in video data processing. Almost all video analysis and processing processes need to extract video features first.
  • 3D CNN Convolutional Neural Networks, Convolutional Neural Networks
  • the main calculation steps of 3D convolution can classify video data or extract features from it.
  • the method of 3D convolution calculation basically adopts dimension reduction, and converts 3D data into 2D data or even 1D data.
  • the running speed is very slow, resulting in low video data processing efficiency.
  • the purpose of this application is to provide a video data processing method, system, electronic device, and computer-readable storage medium, which can fully expand the parallelism of computation, and use multiple three-dimensional systolic arrays to construct a four-dimensional systolic computing architecture for three-dimensional eigenvalue matrices and three-dimensional weights.
  • the value matrix is calculated in parallel, which shortens the calculation time of the three-dimensional convolution and improves the video data processing efficiency.
  • the present application provides a video data processing method, including:
  • the three-dimensional feature data and the three-dimensional weight data are respectively preprocessed to obtain an eigenvalue matrix and a weight value matrix;
  • the eigenvalue matrix and the weight value matrix are input into a plurality of three-dimensional systolic arrays for parallel calculation to obtain a video data processing result.
  • the process of preprocessing the three-dimensional feature data to obtain an eigenvalue matrix specifically includes:
  • the three-dimensional feature data is cut according to the size of the convolution kernel to obtain a plurality of feature data groups, and each of the feature data groups is converted into a corresponding two-dimensional matrix according to a preset mapping relationship;
  • An eigenvalue matrix is obtained from all the two-dimensional matrices.
  • the process of preprocessing the three-dimensional weight data to obtain the weight value matrix specifically includes:
  • the process of inputting the eigenvalue matrix and the weight value matrix into multiple three-dimensional systolic arrays for parallel calculation to obtain the video data processing result specifically includes:
  • the i-th three-dimensional systolic array performs a calculation operation on the eigenvalue matrix and the weight value matrix in the i-th input channel according to the corresponding target intermediate value, and the process of obtaining the i-th calculation result is concrete.
  • the Cout weight value matrices corresponding to the eigenvalue matrix in the ith input channel are respectively stored in the Cout computing units of the ith three-dimensional systolic array, where Cout is the number of output channels;
  • Each sub-eigenvalue matrix corresponding to the eigenvalue matrix in the ith input channel is sequentially input into the ith three-dimensional systolic array according to the first preset time period;
  • the process of inputting each sub-eigenvalue matrix corresponding to the eigenvalue matrix in the ith input channel into the ith three-dimensional systolic array includes:
  • the q eigenvalues in the r+1th row in the sub-eigenvalue matrix are input to the jth computing unit, and the q eigenvalues in the rth row in the sub-eigenvalue matrix are input into the jth calculation unit
  • the process of obtaining the sub-calculation result corresponding to the calculation unit by performing the calculation operation according to the target intermediate value, the received eigenvalue matrix and the stored weight value matrix by each of the calculation units includes:
  • the calculation operation is performed by the q PEs in the rth row of each of the calculation units according to the first relational expression, and the calculation result of each PE is obtained;
  • the sub-computation result of the computation unit is obtained according to the sum of the computation results of all the PEs in the same column.
  • the process of obtaining the video data processing result according to the Cinth calculation result specifically includes:
  • the video data processing result is obtained according to the output results of the Cout calculation units.
  • the process of acquiring the output result of each of the computing units in the Cinth three-dimensional systolic array specifically includes:
  • the output result of each of the computing units in the Cin-th three-dimensional systolic array is obtained through a second relational expression, where the second relational expression is:
  • the application also provides a video data processing system, including:
  • an acquisition module for acquiring 3D feature data and 3D weight data corresponding to the video data
  • a preprocessing module configured to preprocess the three-dimensional feature data and the three-dimensional weight data respectively to obtain an eigenvalue matrix and a weight value matrix
  • the calculation module is used for inputting the eigenvalue matrix and the weight value matrix into a plurality of three-dimensional systolic arrays for parallel calculation to obtain a video data processing result.
  • an electronic device including:
  • the processor is configured to implement the steps of the video data processing method according to any one of the above when executing the computer program.
  • the present application also provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, any one of the above-mentioned methods is implemented.
  • the steps of a video data processing method are described in detail below.
  • the present application provides a video data processing method.
  • the three-dimensional feature value and three-dimensional weight value of the video data are preprocessed by first reducing the dimension and then increasing the dimension. Under feasible conditions, the parallelism of the calculation is fully expanded, and multiple three-dimensional pulses are used.
  • the array constructs a four-dimensional systolic computing architecture to perform parallel computing on the eigenvalue matrix and the weight value matrix, which shortens the calculation time of the three-dimensional convolution and improves the video data processing efficiency.
  • the present application also provides a three-dimensional convolution parallel computing system, an electronic device, and a computer-readable storage medium, which have the same beneficial effects as the three-dimensional convolution parallel computing method described above.
  • FIG. 3 is a schematic diagram of a three-dimensional feature value rearrangement provided by the application.
  • FIG. 5 is a schematic diagram of a four-dimensional systolic array architecture provided by the application.
  • FIG. 6 is a schematic diagram of data transmission of a three-dimensional systolic array computing module provided by the application;
  • FIG. 7 is a schematic structural diagram of a video data processing system provided by the present application.
  • the core of the present application is to provide a video data processing method, system, electronic device and computer-readable storage medium, which can fully expand the parallelism of computing, and use multiple three-dimensional systolic arrays to construct a four-dimensional systolic computing architecture.
  • the value matrix is calculated in parallel, which shortens the calculation time of the three-dimensional convolution and improves the video data processing efficiency.
  • the structural parameters of the convolutional layer mainly include the following:
  • KerCinel Size Defines the receptive field of the convolution operation. In three-dimensional convolution, it is usually set to 3, that is, the size of the convolution kernel is 3 ⁇ 3 ⁇ 3.
  • Stride defines the stride size when the convolution kernel traverses the image.
  • the default value is usually set to 1, or the image can be downsampled after the stride is set to 2, which is similar to max pooling .
  • Boundary expansion defines how the network layer handles sample boundaries. When the convolution kernel is greater than 1 and the boundary expansion is not performed, the output size will be reduced accordingly. When the convolution kernel performs boundary expansion in a standard way, the output data space The dimensions will be equal to the input.
  • the size of the input data is a1 ⁇ a2 ⁇ a3
  • the number of input channels is Cin
  • the size of the convolution kernel is f
  • a convolution kernel contains f ⁇ f ⁇ f weight values
  • the number of output channels is Cout
  • the final output size of the 3D convolution is , the formula is still valid for one-dimensional convolution and two-dimensional convolution, as long as the dimension of the input data is adjusted.
  • FIG. 1 is a flowchart of the steps of a video data processing method provided by the present application.
  • the video data processing method includes:
  • the input data of the three-dimensional convolution is composed of Cin three-dimensional eigenvalue matrices and several (Cin ⁇ Cout) three-dimensional weight value matrices, where the size of a single three-dimensional eigenvalue matrix is a1 ⁇ a2 ⁇ a3 , the size of a single three-dimensional weight value matrix is f ⁇ f ⁇ f. Therefore, the video data processing method based on three-dimensional convolution proposed in this embodiment needs to obtain the input data of three-dimensional convolution in advance, that is, the three-dimensional feature corresponding to the video data. data and 3D weight data for subsequent convolution operations on video data.
  • the video data in this embodiment may be the video data corresponding to the security monitoring video, the video data correspondingly collected during the automatic driving process, or the video data corresponding to the streaming media online video.
  • the application field of the video data in this application No specific limitation is made.
  • the three-dimensional feature data and the three-dimensional weight data corresponding to the video data may be acquired according to a preset acquisition cycle, or the three-dimensional feature data and the three-dimensional weight data corresponding to the video data may be acquired after receiving the acquisition instruction.
  • the triggering conditions for acquiring the three-dimensional feature data and three-dimensional weight data corresponding to the video data are not limited.
  • S102 Preprocess the three-dimensional feature data and the three-dimensional weight data respectively to obtain an eigenvalue matrix and a weight value matrix;
  • the process of preprocessing the three-dimensional feature data to obtain the feature value matrix specifically includes: cutting the three-dimensional feature data according to the size of the convolution kernel to obtain multiple feature data groups, and then converting the three-dimensional feature data according to the preset mapping relationship.
  • Each characteristic data group is converted into its corresponding two-dimensional matrix, and the characteristic value matrix is obtained according to all the two-dimensional matrices; as a preferred embodiment, the process of preprocessing the three-dimensional weight data to obtain the weight value matrix specifically includes: The data is rearranged according to the preset mapping relationship to obtain a weight value matrix.
  • the three-dimensional feature data is divided according to the size of the convolution kernel to obtain a three-dimensional eigenvalue matrix 1, a three-dimensional eigenvalue matrix 2 and a three-dimensional eigenvalue matrix 3.
  • the The three-dimensional eigenvalue matrix 1 forms a two-dimensional matrix 1'
  • the three-dimensional eigenvalue matrix 2 is formed into a two-dimensional matrix 2' according to the specific mapping relationship shown in Figure 2
  • the three-dimensional eigenvalue matrix is formed according to the specific mapping relationship shown in Figure 2.
  • FIG. 3 form a two-dimensional matrix 3', and combine the two-dimensional matrix 1', the two-dimensional matrix 2' and the two-dimensional matrix 3' into a new eigenvalue matrix according to the traversal order of a1, a2, and a3.
  • the three-dimensional weight value matrix Z1 is composed of a two-dimensional matrix Z1'
  • the three-dimensional weight value matrix Z2 is composed of a two-dimensional matrix according to the specific mapping relationship shown in Figure 2.
  • Matrix Z2' according to the specific mapping relationship shown in Figure 2, the three-dimensional weight value matrix Z3 is formed into a two-dimensional matrix Z3', and a new weight value is obtained according to the two-dimensional matrix Z1', the two-dimensional matrix Z2' and the two-dimensional matrix Z3' matrix.
  • the mapping relationship is reordered to obtain the two-dimensional matrix corresponding to the first feature data group, which is set as t1-1-1, t1-1-2, ..., t1-1-9, t1-2-1, t1-2 here -2, ..., t1-2-9, t1-3-1, t1-3-2, ..., t1-3-9, a total of 27 feature data, of which t1-1-1, t1-1-2, ..., t1-1-9 represent the nine values in the first row, t1-2-1, t1-2-2, ..., t1-2-9 represent the nine values in the second row, t1-3- 1.
  • t1-3-2, ..., t1-3-9 represent the nine values in the third row.
  • the second group of 3 ⁇ 3 ⁇ 3 can be selected from the three-dimensional feature data of the first input channel
  • the feature data is used as the second feature data group, and is reordered according to the mapping relationship shown in FIG. 2 to obtain a two-dimensional matrix corresponding to the second feature data group, and so on.
  • the three-dimensional weight data corresponding to the first input channel should also be reordered according to the mapping relationship shown in Figure 2 to obtain multiple weight value matrices.
  • the eigenvalue matrix and the weight value matrix use the same data structure for subsequent parallel calculations. Improve the efficiency of parallel computing.
  • S103 Input the eigenvalue matrix and the weight value matrix into a plurality of three-dimensional systolic arrays for parallel calculation, and obtain a video data processing result.
  • the three-dimensional feature data and the three-dimensional weight data of the video data are both preprocessed as described in S102, they are input into a four-dimensional systolic computing framework constructed by using multiple three-dimensional systolic arrays to perform parallel computation on the eigenvalue matrix and the weight value matrix. , thereby obtaining a three-dimensional convolution calculation result, and using the three-dimensional convolution calculation result as a video data processing result, where the video data processing result may be a classification result, a feature extraction result, and the like.
  • the computing efficiency is improved, especially for the characteristics of high real-time requirements in video processing, it can support various artificial intelligence processing of real-time 3D images.
  • the 3D eigenvalues and 3D weight values of the video data are preprocessed by first reducing the dimension and then increasing the dimension.
  • the parallelism of the calculation is fully expanded, and multiple three-dimensional pulsation arrays are used to construct a four-dimensional pulsation.
  • the computing architecture performs parallel computing on the eigenvalue matrix and the weight value matrix, which shortens the calculation time of the three-dimensional convolution and improves the video data processing efficiency.
  • the eigenvalue matrix and the weight value matrix are input into multiple three-dimensional systolic arrays for parallel calculation, and the process of obtaining the video data processing result specifically includes:
  • one input channel corresponds to one three-dimensional systolic array.
  • a four-dimensional parallel computing architecture can be constructed from multiple three-dimensional systolic arrays.
  • the eigenvalues and intermediate values can be pulsatingly propagated in a specific direction.
  • the solid line arrow Y Represents the transmission direction of eigenvalues
  • the dotted arrow V represents the transmission direction of intermediate values, forming a three-dimensional network pipeline operation.
  • the median value of the target corresponding to the first 3D systolic array is 0, the median value of the target corresponding to the second 3D systolic array is the calculation result of the first 3D systolic array, and the target corresponding to the third 3D systolic array is The intermediate value is the calculation result of the second three-dimensional systolic array, and so on, after obtaining the calculation result of the last three-dimensional systolic array (ie, the Cin-th three-dimensional systolic array), the video data processing result is obtained according to the calculation result.
  • the i-th three-dimensional systolic array performs a calculation operation on the eigenvalue matrix and the weight value matrix in the i-th input channel according to the corresponding target intermediate value, and the process of obtaining the i-th calculation result specifically includes: :
  • the Cout weight value matrices corresponding to the eigenvalue matrix in the i-th input channel are respectively stored in the Cout calculation units of the i-th three-dimensional systolic array, where Cout is the number of output channels;
  • Each sub-eigenvalue matrix corresponding to the eigenvalue matrix in the ith input channel is sequentially input into the ith three-dimensional systolic array according to the first preset time period;
  • the calculation operation is performed according to the target intermediate value, the received eigenvalue matrix and the stored weight value matrix, and the sub-calculation result corresponding to the calculation unit is obtained;
  • the first three-dimensional weight data matrix corresponding to the first input channel is reordered according to the arrangement relationship shown in Figure 2, and pre-stored in the left oblique stripe region (the first calculation unit) of the three-dimensional systolic array shown in Figure 4 , here set as q1-1-1, q1-1-2, ..., q1-1-9, q1-2-1, q1-2-2, ..., q1-2-9, q1-3-1, q1-3-2, ..., q1-3-9 have a total of 27 weight data.
  • the second three-dimensional weight data matrix corresponding to the first input channel is reordered according to the arrangement relationship shown in Figure 2, and pre-stored in the right oblique stripe area (second computing unit) of the three-dimensional systolic array shown in Figure 4
  • the Coutth three-dimensional weight data matrix corresponding to the first input channel is reordered according to the arrangement relationship shown in FIG. 2, and pre-stored in the horizontal stripe area of the three-dimensional systolic array shown in )middle.
  • the processing of the three-dimensional data of the remaining input channels is the same. It can be understood that all the three-dimensional weight data are pre-stored in the three-dimensional systolic array after preprocessing, and do not change during the parallel calculation process.
  • a plurality of sub-eigenvalue matrices can be obtained according to the eigenvalue matrix in each input channel, and the size of each sub-eigenvalue matrix is p ⁇ q, and each sub-eigenvalue matrix is input into the three-dimensional systolic array according to the first preset time period , if the eigenvalue matrix in the input channel includes 3 sub-eigenvalue matrices, the first sub-eigenvalue matrix is input into the three-dimensional systolic array in the first first preset time period, and in the second first The second sub-eigenvalue matrix is input into the three-dimensional systolic array in the preset time period, and the third sub-eigenvalue matrix is input into the three-dimensional systolic array in the third first preset time period.
  • the eigenvalue matrices in different input channels can be input into their corresponding three-dimensional systolic arrays at preset time intervals.
  • the processing scheme of the eigenvalue matrices in each input channel refer to the above.
  • Each calculation unit performs a calculation operation according to the target intermediate value, the received eigenvalue matrix and the stored weight value matrix, and obtains a sub-calculation result corresponding to the calculation unit, and the sub-calculation results of all calculation units in the three-dimensional systolic array constitute the Calculation results of a 3D systolic array.
  • the process of inputting each sub-eigenvalue matrix corresponding to the eigenvalue matrix in the ith input channel into the ith three-dimensional systolic array includes:
  • the following describes the input process of one of the sub-eigenvalue matrices.
  • the 9 characteristic data in the first row of the sub-characteristic matrix are input into the 9 PEs in the first row of the first computing unit in one-to-one correspondence, and in the second second preset period, the sub-characteristic matrix No.
  • the 9 characteristic data of one row are input into the 9 PEs of the first row of the second computing unit in one-to-one correspondence, and in the third second preset period, the 9 characteristic data of the first row of the sub-feature matrix are one-to-one.
  • the 9 characteristic data of the first row of the third computing unit are one-to-one.
  • the 9 characteristic data of the first row of the sub-characteristic matrix are input into the 9 PEs of the first row of the second computing unit in a one-to-one correspondence.
  • the 9 feature data in the second row of the sub-feature matrix are input into the 9 PEs in the second row of the first computing unit in a one-to-one correspondence.
  • each calculation unit performs a calculation operation according to the target intermediate value, the received eigenvalue matrix and the stored weight value matrix, and the process of obtaining the sub-calculation result corresponding to the calculation unit includes:
  • the calculation operation is performed by the q PEs in the rth row of each calculation unit according to the first relational expression, and the calculation result of each PE is obtained;
  • the sub-computation result of the computation unit is obtained according to the sum of the computation results of all PEs in the same column.
  • the process of obtaining the video data processing result according to the Cinth calculation result specifically includes:
  • the video data processing result is obtained according to the output results of the Cout computing units.
  • the process of acquiring the output result of each computing unit in the Cinth three-dimensional systolic array specifically includes:
  • the output result of each computing unit in the Cin-th three-dimensional systolic array is obtained through the second relational expression.
  • the second relational expression is:
  • t2-3-2, ..., t2-3-9 has a total of 27 eigenvalues, select the third sub-eigenvalue matrix from the eigenvalue matrix of the first input channel, including t3-1-1, t3-1 -2,...,t3-1-9,t3-2-1,t3-2-2,...,t3-2-9,t3-3-1,t3-3-2,...,t3-3-9 There are a total of 27 eigenvalues, and the fourth sub-eigenvalue matrix is selected from the eigenvalue matrix of the first input channel, including t4-1-1, t4-1-2, ..., t4-1-9, t4-2- 1.
  • the characteristic data t1-1-1, t1-1-2, ..., t1-1-9 are sent to the 9 PEs in the top row of the left diagonal stripe area shown in Figure 4, Correspondingly multiply with the weight values q1-1-1, q1-1-2, ..., q1-1-9, that is, multiply t1-1-1 and q1-1-1, t1-1-2 and q1- Multiplying 1-2, and so on, the obtained intermediate values are set as m1-1-1, m1-1-2, ..., m1-1-9, respectively, as the target intermediate values of each PE in the next row.
  • the characteristic data t1-2-1, t1-2-2, ..., t1-2-9 are sent to the 9 PEs in the middle row of the left diagonal stripe shown in Fig.
  • the weight values q1-2-1, q1-2-2, ..., q1-2-9 are correspondingly multiplied, and are multiplied with the input intermediate values m1-1-1, m1-1-2, ..., m1-1-9 Addition, that is, multiply t1-2-1 and q1-2-1 and add m1-1-1, multiply t1-2-2 and q1-2-2 and add m1-1-2, so that By analogy, the obtained intermediate values are set as m1-2-1, m1-2-2, ..., m1-2-9.
  • t2-1-1, t2-1-2, ..., t2-1-9 of the feature data are sent to the 9 PEs in the top row of the left diagonal stripe shown in Figure 4, and q1-1-1, q1 -1-2, ..., q1-1-9 are correspondingly multiplied, that is, t2-1-1 is multiplied by q1-1-1, t2-1-2 is multiplied by q1-1-2, and so on, to get The intermediate values of are set to m2-1-1, m2-1-2, ..., m2-1-9.
  • the characteristic data t1-1-1, t1-1-2, ..., t1-1-9 are sent to the 9 PEs in the top row of the right diagonal stripe shown in Figure 4, and q2-1-1, q2- 1-2, ..., q2-1-9 are multiplied correspondingly, that is, t1-1-1 and q2-1-1 are multiplied, t1-1-2 and q2-1-2 are multiplied, and so on, the obtained Intermediate values are set to n1-1-1, n1-1-2, ..., n1-1-9.
  • the characteristic data t1-3-1, t1-3-2, ..., t1-3-9 are sent to the 9 PEs in the bottom row of the left diagonal stripe shown in Figure 4, Correspondingly multiplied with q1-3-1, q1-3-2, ..., q1-3-9, and with the input intermediate value m1-2-1, m1-2-2, ..., m1-2-9 Add, that is, multiply t1-3-1 and q1-3-1 and add m1-2-1, multiply t1-3-2 and q1-3-2 and add m1-2-2, and so on , and the obtained intermediate values are set as m1-3-1, m1-3-2, ..., m1-3-9.
  • the feature data t2-2-1, t2-2-2, ..., t2-2-9 are sent to the 9 PEs in the middle row of the left diagonal stripe shown in Figure 4, and q1-2-1, q1- 2-2, ..., q1-2-9 are multiplied correspondingly and added to the input intermediate values m2-1-1, m2-1-2, ..., m2-1-9, that is, t2-2-1 and Multiply q1-2-1 and add m2-1-1, multiply t2-2-2 and q1-2-2 and add m2-1-2, and so on, the resulting intermediate value is set to m2- 2-1, m2-2-2, ..., m2-2-9.
  • the feature data t3-1-1, t3-1-2,..., t3-1-9 are sent to the 9 PEs in the top row of the left diagonal stripe shown in Figure 4, and q1-1-1, q1- 1-2, ..., q1-1-9 are multiplied correspondingly, that is, t3-1-1 and q1-1-1 are multiplied, t3-1-2 and q1-1-2 are multiplied, and so on, the obtained
  • the intermediate values are set to m3-1-1, m3-1-2, ..., m3-1-9.
  • the feature data t1-2-1, t1-2-2, ..., t1-2-9 are sent to the 9 PEs in the middle row of the right diagonal stripe shown in Figure 4, and q2-2-1, q2- 2-2, ..., q2-2-9 are multiplied correspondingly and added to the input intermediate values n1-1-1, n1-1-2, ..., n1-1-9, that is, t1-2-1 and Multiply q2-2-1 and add n1-1-1, multiply t1-2-2 and q2-2-2 and add n1-1-2, and so on, the resulting intermediate value is set to n1- 2-1, n1-2-2, ..., n1-2-9.
  • the characteristic data t2-1-1, t2-1-2, ..., t2-1-9 are sent to the 9 PEs in the top row of the right diagonal stripe shown in Figure 4, and q2-1-1, q2- 1-2, ..., q2-1-9 are multiplied correspondingly, that is, t2-1-1 and q2-1-1 are multiplied, t2-1-2 and q2-1-2 are multiplied, and so on, the obtained Intermediate values are set to n2-1-1, n2-1-2, ..., n2-1-9.
  • the feature data t1-1-1, t1-1-2, ..., t1-1-9 are sent to the top row of the group behind the right diagonal stripe shown in Figure 4 (in Figure 4, for the sake of display effect, it is not drawn)
  • the resulting intermediate values are set to o1-1-1, o1-1-2, ..., o1-1-9.
  • the characteristic data t1-1-1, t1-1-2, ..., t1-1-9 are sent to the second group behind the right diagonal stripe shown in Fig. effect, not shown), among the 9 PEs in the top row, are multiplied with q4-1-1, q4-1-2, ..., q4-1-9, that is, t1-1-1 and q4-1- Multiply by 1, multiply t1-1-2 and q4-1-2, and so on, the intermediate values obtained are set to k1-1-1, k1-1-2, ..., k1-1-9.
  • the feature data t1-2-1, t1-2-2,..., t1-2-9 are sent to the 9 PEs in the middle row of the group behind the right diagonal stripe shown in Figure 4, and q3-2-1 , q3-2-2, ..., q3-2-9 are multiplied correspondingly, and added to the input intermediate values o1-1-1, o1-1-2, ..., o1-1-9, that is, t1-2 Multiply -1 and q3-2-1 and add o1-1-1, multiply t1-2-2 and q3-2-2 and add o1-1-2, and so on, the obtained intermediate value is set as are o1-2-1, o1-2-2, ..., o1-2-9.
  • the feature data t1-3-1, t1-3-2, ..., t1-3-9 are sent to the 9 PEs in the bottom row of the right diagonal stripe shown in Figure 4, and q2-3-1, q2 -3-2, ..., q2-3-9 are multiplied correspondingly, and added to the input intermediate values n1-2-1, n1-2-2, ..., n1-2-9, that is, t1-3-1 Multiply with q1-3-1 and add m1-2-1, multiply t1-3-2 with q1-3-2 and add m1-2-2, and so on, the obtained intermediate value is set to m1 -3-1, m1-3-2, ..., m1-3-9.
  • the feature data t2-1-1, t2-1-2, ..., t2-1-9 are sent to the top row of the group behind the right diagonal stripe shown in Figure 4 (in Figure 4 for the sake of display effect, it is not drawn)
  • the resulting intermediate values are set to o2-1-1, o2-1-2, ..., o2-1-9.
  • the feature data t2-2-1, t2-2-2, ..., t2-2-9 are sent to the 9 PEs in the middle row of the right diagonal stripe shown in Figure 4, and q2-2-1, q2- 2-2, ..., q2-2-9 are multiplied correspondingly and added to the input intermediate values n2-1-1, n2-1-2, ..., n2-1-9, that is, t2-2-1 and Multiply q2-2-1 and add n2-1-1, multiply t2-2-2 and q2-2-2 and add n2-1-2, and so on, the resulting intermediate value is set to n2- 2-1, n2-2-2, ..., n2-2-9.
  • the feature data t2-3-1, t2-3-2,..., t2-3-9 are sent to the 9 PEs in the bottom row of the left diagonal stripe shown in Figure 4, and q1-3-1, q1 -3-2, ..., q1-3-9 are multiplied correspondingly, and added to the input intermediate values m2-2-1, m2-2-2, ..., m2-2-9, that is, t2-3-1 Multiply with q1-3-1 and add m2-2-1, multiply t2-3-2 with q1-3-2 and add m2-2-2, and so on, the obtained intermediate value is set to m2 -3-1, m2-3-2, ..., m2-3-9.
  • the characteristic data t3-1-1, t3-1-2, ..., t3-1-9 are sent to the 9 PEs in the top row of the right diagonal stripe shown in Figure 4, and q2-1-1, q2- 1-2, ..., q2-1-9 are multiplied correspondingly, that is, t3-1-1 and q2-1-1 are multiplied, t3-1-2 and q2-1-2 are multiplied, and so on, the obtained
  • the intermediate values are set to n3-1-1, n3-1-2, ..., n3-1-9.
  • the characteristic data t3-2-1, t3-2-2, ..., t3-2-9 are sent to the 9 PEs in the middle row of the left diagonal stripe shown in Figure 4, and q1-2-1, q1- 2-2, ..., q1-2-9 are multiplied correspondingly and added to the input intermediate values m3-1-1, m3-1-2, ..., m3-1-9, that is, t3-2-1 and Multiply q1-2-1 and add m3-1-1, multiply t3-2-2 and q1-2-2 and add m3-1-2, and so on, the resulting intermediate value is set to m3- 2-1, m3-2-2, ..., m3-2-9.
  • the characteristic data t4-1-1, t4-1-2, ..., t4-1-9 are sent to the 9 PEs in the top row of the left diagonal stripe shown in Figure 4, and q1-1-1, q1- 1-2, ..., q1-1-9 are multiplied correspondingly, that is, t4-1-1 and q1-1-1 are multiplied, t4-1-2 and q1-1-2 are multiplied, and so on, the obtained
  • the intermediate values are set to m4-1-1, m4-1-2, ..., m4-1-9.
  • the characteristic data t'1-1-1, t'1-1-2, ..., t'1-1-9 of the second input channel are sent to the left side corresponding to the characteristic value of input channel 2 shown in Figure 5.
  • the 9 PEs in the uppermost row of the diagonal stripe they are multiplied by q'1-1-1, q'1-1-2, ..., q'1-1-9 corresponding to the input intermediate value m1-3 -1, m1-3-2, ..., m1-3-9 are added, that is, t'1-1-1 and q'1-1-1 are multiplied and added to m1-3-1, t'1-1 Multiply -2 by q1-1-2 and add m1-3-2, and so on, the resulting intermediate values are set to m'1-1-1, m'1-1-2, ..., m'1 -1-9, the subsequent calculation steps are basically the superposition and repetition of the above.
  • the characteristic data will be transmitted horizontally in these budget units, as shown by the solid arrow Y in FIG. 6 .
  • the intermediate values of multiplication and accumulation at all levels are transferred among multiple groups of three-dimensional operation arrays, as shown by the vertical dotted arrow V in FIG. 6 .
  • the three-dimensional systolic array results of the Cin group are vertically accumulated together, it is necessary to additionally accumulate the results of the 9 columns in this array into a single value, that is, by adding the products of 3 ⁇ 9 ⁇ Cin, the obtained The result is one value in one output channel, and the above process is repeated until all calculations are complete.
  • FIG. 7 is a schematic structural diagram of a video data processing system provided by the application.
  • the video data processing system includes:
  • an acquisition module 11 configured to acquire three-dimensional feature data and three-dimensional weight data corresponding to the video data
  • the preprocessing module 12 is used for preprocessing the three-dimensional feature data and the three-dimensional weight data respectively to obtain the feature value matrix and the weight value matrix;
  • the calculation module 13 is used for inputting the eigenvalue matrix and the weight value matrix into a plurality of three-dimensional systolic arrays for parallel calculation to obtain the video data processing result.
  • the 3D eigenvalues and 3D weight values of the video data are preprocessed by first reducing the dimension and then increasing the dimension.
  • the parallelism of the calculation is fully expanded, and multiple three-dimensional pulsation arrays are used to construct a four-dimensional pulsation.
  • the computing architecture performs parallel computing on the eigenvalue matrix and the weight value matrix, which shortens the calculation time of the three-dimensional convolution and improves the video data processing efficiency.
  • the preprocessing module 12 includes:
  • the first preprocessing unit is used to cut the three-dimensional feature data according to the size of the convolution kernel to obtain a plurality of sub-eigenvalue matrices, and convert each of the sub-eigenvalue matrices into their corresponding sub-eigenvalues according to a preset mapping relationship matrix, and an eigenvalue matrix is obtained according to all the sub-eigenvalue matrices.
  • the preprocessing module 12 further includes:
  • the second preprocessing unit is used for rearranging the three-dimensional weight data according to a preset mapping relationship to obtain a weight value matrix.
  • the calculation module 3 is specifically used for:
  • the i-th three-dimensional systolic array performs a calculation operation on the eigenvalue matrix and the weight value matrix in the i-th input channel according to the corresponding target intermediate value, and the process of obtaining the i-th calculation result specifically includes: :
  • the Cout weight value matrices corresponding to the eigenvalue matrix in the i-th input channel are respectively stored in the Cout calculation units of the i-th three-dimensional systolic array, where Cout is the number of output channels;
  • Each sub-eigenvalue matrix corresponding to the eigenvalue matrix in the ith input channel is sequentially input into the ith three-dimensional systolic array according to the first preset time period;
  • the calculation operation is performed according to the target intermediate value, the received eigenvalue matrix and the stored weight value matrix, and the sub-calculation result corresponding to the calculation unit is obtained;
  • the process of inputting each sub-eigenvalue matrix corresponding to the eigenvalue matrix in the ith input channel into the ith three-dimensional systolic array includes:
  • each calculation unit performs a calculation operation according to the target intermediate value, the received eigenvalue matrix and the stored weight value matrix, and the process of obtaining the sub-calculation result corresponding to the calculation unit includes:
  • the calculation operation is performed by the q PEs in the rth row of each calculation unit according to the first relational expression, and the calculation result of each PE is obtained;
  • the sub-computation result of the computation unit is obtained according to the sum of the computation results of all PEs in the same column.
  • the process of obtaining the video data processing result according to the Cinth calculation result specifically includes:
  • the video data processing result is obtained according to the output results of the Cout computing units.
  • the process of acquiring the output result of each computing unit in the Cinth three-dimensional systolic array specifically includes:
  • the output result of each computing unit in the Cin-th three-dimensional systolic array is obtained through the second relational expression.
  • the second relational expression is:
  • an electronic device comprising:
  • the processor is configured to implement the steps of the video data processing method described in any one of the above embodiments when executing the computer program.
  • the processor may be a central processing unit (Central Processing Unit, CPU), an application-specific integrated circuit, a digital signal processor, an off-the-shelf programmable gate array or other programmable logic devices, and the like.
  • CPU Central Processing Unit
  • application-specific integrated circuit an application-specific integrated circuit
  • digital signal processor an off-the-shelf programmable gate array or other programmable logic devices, and the like.
  • the electronic device provided by the present application has the same beneficial effects as the above-mentioned video data processing method.
  • the present application also provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the video data processing method described in any one of the above embodiments is implemented. step.
  • the computer-readable storage medium provided by the present application has the same beneficial effects as the above-mentioned video data processing method.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Software Systems (AREA)
  • Evolutionary Computation (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Artificial Intelligence (AREA)
  • Computational Linguistics (AREA)
  • Computing Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Data Mining & Analysis (AREA)
  • Biomedical Technology (AREA)
  • General Engineering & Computer Science (AREA)
  • Molecular Biology (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • Biophysics (AREA)
  • Mathematical Physics (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Biodiversity & Conservation Biology (AREA)
  • Image Processing (AREA)
  • Image Analysis (AREA)

Abstract

一种视频数据处理方法，包括：获取视频数据对应的三维特征数据和三维权重数据(S101)；对三维特征数据和三维权重数据分别进行预处理，得到特征值矩阵及权重值矩阵(S102)；将特征值矩阵和权重值矩阵输入多个三维脉动阵列并行计算，得到视频数据处理结果(S103)。本方法能够充分扩展计算的并行度，采用多重三维脉动阵列构造四维脉动计算架构对三维特征值矩阵和三维权重值矩阵进行并行计算，缩短了三维卷积的计算时间，提高了视频数据处理效率。一种三维卷积并行计算系统、电子设备及计算机可读存储介质，具有以上有益效果。

Description

一种视频数据处理方法、系统及相关组件
本申请要求于2020年09月25日提交中国专利局、申请号为202011026282.4、发明名称为“一种视频数据处理方法、系统及相关组件”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及视频数据处理领域，特别涉及一种视频数据处理方法、系统及相关组件。
背景技术
视频特征提取是视频数据处理的一个基本环节，几乎所有视频分析和处理的过程，都需要先对视频特征进行提取。三维CNN(Convolutional Neural Networks，卷积神经网络)以其能更好的捕获视频中的时间和空间的特征信息，在视频分类、动作识别等领域发挥着巨大的优势，三维卷积是三维CNN中的主要计算步骤，通过三维卷积可以对视频数据进行分类或从中提取特征，目前对于三维卷积计算的方法基本都是采用降低维度，将三维数据转化映射为二维数据甚至是一维数据，进行局部的并行计算，由于计算量巨大，运行速度非常缓慢，导致视频数据处理效率低。
因此，如何提供一种解决上述技术问题的方案是本领域技术人员目前需要解决的问题。
发明内容
本申请的目的是提供一种视频数据处理方法、系统、电子设备及计算机可读存储介质，能够充分扩展计算的并行度，采用多重三维脉动阵列构造四维脉动计算架构对三维特征值矩阵和三维权重值矩阵进行并行计算，缩短了三维卷积的计算时间，提高了视频数据处理效率。
为解决上述技术问题，本申请提供了一种视频数据处理方法，包括：
获取视频数据对应的三维特征数据和三维权重数据；
对所述三维特征数据和所述三维权重数据分别进行预处理，得到特征 值矩阵及权重值矩阵；
将所述特征值矩阵和所述权重值矩阵输入多个三维脉动阵列并行计算，得到视频数据处理结果。
优选的，对所述三维特征数据进行预处理得到特征值矩阵的过程具体包括：
对所述三维特征数据按卷积核尺寸进行切割，得到多个特征数据组，按预设映射关系将各个所述特征数据组转换成各自对应的二维矩阵；
根据所有所述二维矩阵得到特征值矩阵。
优选的，对所述三维权重数据进行预处理得到权重值矩阵的过程具体包括：
将所述三维权重数据按所述预设映射关系进行重排，得到权重值矩阵。
优选的，所述将所述特征值矩阵和所述权重值矩阵输入多个三维脉动阵列并行计算，得到视频数据处理结果的过程具体包括：
通过第i个三维脉动阵列根据对应的目标中间值对第i个输入通道中的所述特征值矩阵及所述权重值矩阵执行计算操作，得到第i个计算结果，i＝1，2，…，Cin；
根据第Cin个计算结果得到视频数据处理结果；
其中，当i＝1时，所述目标中间值为0，当1＜i≤Cin时，所述目标中间值为第i-1个计算结果。
优选的，所述通过第i个三维脉动阵列根据对应的目标中间值对第i个输入通道中的所述特征值矩阵及所述权重值矩阵执行计算操作，得到第i个计算结果的过程具体包括：
将第i个输入通道中的所述特征值矩阵对应的Cout个权重值矩阵分别存储至第i个三维脉动阵列的Cout个计算单元中，Cout为输出通道个数；
将所述第i个输入通道中特征值矩阵对应的各个子特征值矩阵按第一预设时间周期依次输入到所述第i个三维脉动阵列中；
通过每个所述计算单元根据目标中间值、接收到的特征值矩阵及存储的权重值矩阵执行计算操作，得到该计算单元对应的子计算结果；
通过所有所述子计算结果，得到第i个计算结果。
优选的，将所述第i个输入通道中特征值矩阵对应的每个子特征值矩阵输入到所述第i个三维脉动阵列中的过程包括：
将所述第i个输入通道中特征值矩阵对应的每个子特征值矩阵的第r行的q个特征值按第二预设时间周期分别输入到所述第i个三维脉动阵列的Cout个计算单元的第r行的q个PE中，所述子特征值矩阵的大小为p×q，p、q均为正整数，r＝1，2，…，p-1；
其中，所述子特征值矩阵中第r+1行的q个特征值输入到第j个计算单元与所述子特征值矩阵中第r行的q个特征值输入到所述第j个计算单元的时间间隔为所述第二预设时间周期，其中，j＝1，2，…，Cout。
优选的，所述通过每个所述计算单元根据目标中间值、接收到的特征值矩阵及存储的权重值矩阵执行计算操作，得到该计算单元对应的子计算结果的过程包括：
通过每个所述计算单元的第r行的q个PE按第一关系式执行计算操作，得到每个PE的计算结果；
所述第一关系式为h rw＝t rw×q rw+c rw，h w为第r行、第w个PE的计算结果，t w为第r行、第w个PE接收到的特征值，q w为第r行、第w个PE的权重值，c w为第r行、第w个PE对应的目标中间值，w＝1，2，…，q；
根据相同列的所有所述PE的计算结果的和得到该计算单元的子计算结果。
优选的，所述根据第Cin个计算结果得到视频数据处理结果的过程具体包括：
获取第Cin个三维脉动阵列中每个所述计算单元的输出结果；
根据Cout个所述计算单元的输出结果得到视频数据处理结果。
优选的，所述获取第Cin个三维脉动阵列中每个所述计算单元的输出结果的过程具体包括：
通过第二关系式获取第Cin个三维脉动阵列中每个所述计算单元的输出结果，所述第二关系式为
Figure PCTCN2021089924-appb-000001
为解决上述技术问题，本申请还提供了一种视频数据处理系统，包括：
获取模块，用于获取视频数据对应的三维特征数据和三维权重数据；
预处理模块，用于对所述三维特征数据和所述三维权重数据分别进行预处理，得到特征值矩阵及权重值矩阵；
计算模块，用于将所述特征值矩阵和所述权重值矩阵输入多个三维脉动阵列并行计算，得到视频数据处理结果。
为解决上述技术问题，本申请还提供了一种电子设备，包括：
存储器，用于存储计算机程序；
处理器，用于执行所述计算机程序时实现如上文任意一项所述的视频数据处理方法的步骤。
为解决上述技术问题，本申请还提供了一种计算机可读存储介质，所述计算机可读存储介质上存储有计算机程序，所述计算机程序被处理器执行时实现如上文任意一项所述的视频数据处理方法的步骤。
本申请提供了一种视频数据处理方法，对视频数据的三维特征值和三维权重值采取先降维再升维的预处理，在可行的条件下，充分扩展计算的并行度，采用多重三维脉动阵列构造四维脉动计算架构对特征值矩阵和权重值矩阵进行并行计算，缩短了三维卷积的计算时间，提高了视频数据处理效率。本申请还提供了一种三维卷积并行计算系统、电子设备及计算机可读存储介质，具有和上述三维卷积并行计算方法相同的有益效果。
附图说明
为了更清楚地说明本申请实施例，下面将对实施例中所需要使用的附图做简单的介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1为本申请所提供的一种视频数据处理方法的步骤流程图；
图2为本申请所提供的一种映射关系示意图；
图3为本申请所提供的一种三维特征值重排示意图；
图4为本申请所提供的一种三维权重值重排示意图；
图5为本申请所提供的一种四维脉动阵列架构示意图；
图6为本申请所提供的一种三维脉动阵列计算模块的数据传输示意 图；
图7为本申请所提供的一种视频数据处理系统的结构示意图。
具体实施方式
本申请的核心是提供一种视频数据处理方法、系统、电子设备及计算机可读存储介质，能够充分扩展计算的并行度，采用多重三维脉动阵列构造四维脉动计算架构对三维特征值矩阵和三维权重值矩阵进行并行计算，缩短了三维卷积的计算时间，提高了视频数据处理效率。
为使本申请实施例的目的、技术方案和优点更加清楚，下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。
为便于理解本申请所提供的一种基于三维卷积的视频数据处理方法，下面对卷积层的结构参数进行介绍，卷积层的结构参数主要有以下几种：
卷积核尺寸(KerCinel Size)：定义了卷积操作的感受野，在三维卷积中，通常设置为3，即卷积核大小为3×3×3。
步幅(Stride)：定义了卷积核遍历图像时的步幅大小，其默认值通常设置为1，也可将步幅设置为2后对图像进行下采样，这种方式与最大池化类似。
边界扩充(Padding)：定义了网络层处理样本边界的方式，当卷积核大于1且不进行边界扩充，输出尺寸将相应缩小，当卷积核以标准方式进行边界扩充，则输出数据的空间尺寸将与输入相等。
假设输入数据的大小为a1×a2×a3，输入通道个数为Cin，卷积核大小为f，即一个卷积核中包含了f×f×f个权重值，输出通道个数为Cout，因此权重值的总数目为f×f×f×Cin×Cout。
基于上述情况，三维卷积最终的输出尺寸为
Figure PCTCN2021089924-appb-000002
，该公式对于一维卷积、二维卷积仍然有效，只要调整输入数据的维度。
下面对本申请所提供的一种视频数据处理方法进行详细说明。
请参照图1，图1为本申请所提供的一种视频数据处理方法的步骤流程图，该视频数据处理方法包括：
S101：获取视频数据对应的三维特征数据和三维权重数据；
首先需要说明的是，三维卷积的输入数据是由Cin个三维特征值矩阵和若干个(Cin×Cout个)三维权重值矩阵组成的，其中，单个三维特征值矩阵尺寸为a1×a2×a3，单个三维权重值矩阵尺寸为f×f×f，因此，本实施例所提出的一种基于三维卷积的视频数据处理方法需要预先获取三维卷积的输入数据，即视频数据对应的三维特征数据和三维权重数据，以便后续对视频数据进行卷积运算。本实施例中的视频数据可以为安防监控录像对应的视频数据、也可以为自动驾驶过程中对应采集的视频数据、还可以为流媒体在线视频对应的视频数据，本申请对于视频数据的应用领域不作具体的限定。
本步骤中，可以按预设获取周期获取视频数据对应的三维特征数据和三维权重数据，也可以在接收到获取指令后获取视频数据对应的三维特征数据和三维权重数据，本实施例在此对获取视频数据对应的三维特征数据和三维权重数据的触发条件不作限定。
S102：对三维特征数据和三维权重数据分别进行预处理，得到特征值矩阵及权重值矩阵；
具体的，在获取到三维特征数据和三维权重数据后，先对二者进行降维预处理，使其符合三维脉动阵列的规模以及时序上的要求。作为一种优选的实施例，对三维特征数据进行预处理得到特征值矩阵的过程具体包括：对三维特征数据按卷积核尺寸进行切割，得到多个特征数据组，然后按预 设映射关系将各个特征数据组转换成各自对应的二维矩阵，根据所有二维矩阵得到特征值矩阵；作为一种优选的实施例，对三维权重数据进行预处理得到权重值矩阵的过程具体包括：将三维权重数据按预设映射关系进行重排，得到权重值矩阵。
具体的，参照图3所示，对三维特征数据按卷积核尺寸分割，得到三维特征值矩阵1、三维特征值矩阵2及三维特征值矩阵3，按如图2所示的特定映射关系将三维特征值矩阵1组成二维矩阵1’，按如图2所示的特定映射关系将三维特征值矩阵2组成二维矩阵2’，按如图2所示的特定映射关系将三维特征值矩阵3组成二维矩阵3’，将二维矩阵1’，二维矩阵2’和二维矩阵3’按照按a1、a2、a3的遍历顺序组合成新的特征值矩阵。其中，图2是以f＝3为例的根据数据循环的方式确定的映射关系，便于在硬件电路中实现。
具体的，对于三维卷积计算所需的三维权重值的预处理参照图4所示，将同一个输入通道对应的Cout个卷积核分别按图2所示的对应关系展开，整合成一个新的权重值矩阵，具体的，按照如图2所示的特定映射关系将三维权重值矩阵Z1组成二维矩阵Z1’，按照如图2所示的特定映射关系将三维权重值矩阵Z2组成二维矩阵Z2’，按照如图2所示的特定映射关系将三维权重值矩阵Z3组成二维矩阵Z3’，根据二维矩阵Z1’、二维矩阵Z2’及二维矩阵Z3’得到新的权重值矩阵。
具体的，以f＝3为例，首先从第一个输入通道的三维特征数据中选取第一组3×3×3个特征数据作为第一特征数据组，并将其按照图2所示的映射关系进行重新排序，得到第一特征数据组对应的二维矩阵，这里设为t1-1-1、t1-1-2、…、t1-1-9、t1-2-1、t1-2-2、…、t1-2-9、t1-3-1、t1-3-2、…、t1-3-9共27个特征数据，其中，t1-1-1、t1-1-2、…、t1-1-9代表第一行中的九个数值，t1-2-1、t1-2-2、…、t1-2-9代表第二行中的九个数值，t1-3-1、t1-3-2、…、t1-3-9代表第三行中的九个数值，同理，可从第一个输入通道的三维特征数据中选取第二组3×3×3个特征数据作为第二特征数据组，并将其按照图2所示的映射关系进行重新排序，得到第二特征数据组对应的二维矩阵，以此类推。第一个输入通道所对应的三维权重数据也要按照图 2所示的映射关系进行重新排序得到多个权重值矩阵，特征值矩阵和权重值矩阵采用相同的数据结构，以便后续进行并行计算，提高并行运算效率。
S103：将特征值矩阵和权重值矩阵输入多个三维脉动阵列并行计算，得到视频数据处理结果。
具体的，当对视频数据的三维特征数据和三维权重数据都进行了如S102所描述的预处理之后，输入到采用多重三维脉动阵列构造四维脉动计算架构对特征值矩阵和权重值矩阵进行并行计算，从而得到三维卷积计算结果，将三维卷积计算结果作为视频数据处理结果，这里的视频数据处理结果可以是分类结果、特征提取结果等。通过扩展三维卷积计算的并行度，提高了运算效率，尤其是针对视频处理方面对于实时性要求较高的特点，可以支持各种对实时三维图像的人工智能处理。
可见，本实施例中，对视频数据的三维特征值和三维权重值采取先降维再升维的预处理，在可行的条件下，充分扩展计算的并行度，采用多重三维脉动阵列构造四维脉动计算架构对特征值矩阵和权重值矩阵进行并行计算，缩短了三维卷积的计算时间，提高了视频数据处理效率。
在上述实施例的基础上：
作为一种优选的实施例，将特征值矩阵和权重值矩阵输入多个三维脉动阵列并行计算，得到视频数据处理结果的过程具体包括：
通过第i个三维脉动阵列根据对应的目标中间值对第i个输入通道中的特征值矩阵及权重值矩阵执行计算操作，得到第i个计算结果，i＝1，2，…，Cin；
根据第Cin个计算结果得到视频数据处理结果；
其中，当i＝1时，目标中间值为0，当1＜i≤Cin时，目标中间值为第i-1个计算结果。
可以理解的是，一个输入通道对应一个三维脉动阵列，本实施例中，参照图5所示，存在多个三维脉动阵列，其中，每个三维脉动阵列均用于实现其对应的输入通道中的三维数据的卷积计算，由多个三维脉动阵列可以构造四维并行运算架构。在进行卷积计算时，需要用到中间值、权重值 和特征值，采用本实施例的方案可以将特征值和中间值按照特定的方向进行脉动传播，参照图6所示，实线箭头Y代表特征值的传输方向，虚线箭头V代表中间值的传输方向，形成立体网状的流水线操作。
可以理解的是，第一个三维脉动阵列对应的目标中间值为0，第二个三维脉动阵列对应的目标中间值为第一个三维脉动阵列的计算结果，第三个三维脉动阵列对应的目标中间值为第二个三维脉动阵列的计算结果，以此类推，得到最后一个三维脉动阵列(即第Cin个三维脉动阵列)的计算结果后，根据该计算结果得到视频数据处理结果。
作为一种优选的实施例，通过第i个三维脉动阵列根据对应的目标中间值对第i个输入通道中的特征值矩阵及权重值矩阵执行计算操作，得到第i个计算结果的过程具体包括：
将第i个输入通道中的特征值矩阵对应的Cout个权重值矩阵分别存储至第i个三维脉动阵列的Cout个计算单元中，Cout为输出通道个数；
将第i个输入通道中特征值矩阵对应的各个子特征值矩阵按第一预设时间周期依次输入到第i个三维脉动阵列中；
通过每个计算单元根据目标中间值、接收到的特征值矩阵及存储的权重值矩阵执行计算操作，得到该计算单元对应的子计算结果；
通过所有子计算结果，得到第i个计算结果。
具体的，假设f＝3，以第一个输入通道对应的特征值矩阵和权重值矩阵为例对三维脉动阵列执行的计算操作进行说明，其余输入通道，同理。将第一个输入通道所对应的第一个三维权重数据矩阵按照图2所示的排列关系进行重新排序，预存至图4所示的三维脉动阵列的左斜条纹区域(第一计算单元)中，这里设为q1-1-1、q1-1-2、…、q1-1-9、q1-2-1、q1-2-2、…、q1-2-9、q1-3-1、q1-3-2、…、q1-3-9共27个权重数据。第一个输入通道所对应的第二个三维权重数据矩阵按照图2所示的排列关系进行重新排序，并预存至图4所示的三维脉动阵列的右斜条纹区域(第二计算单元)中，该第一个输入通道所对应的第Cout个三维权重数据矩阵按照图2所示的排列关系进行重新排序，并预存至图4所示的三维脉动阵列的横条条纹区域(第Cout计算单元)中。对其余输入通道的三维数据的处理，同理。可以 理解的是，所有的三维权重数据经预处理后预存到三维脉动阵列中，在并行计算过程中，不发生变化。
具体的，根据每个输入通道中的特征值矩阵可以得到多个子特征值矩阵，每个子特征值矩阵大小为p×q，将各个子特征值矩阵按第一预设时间周期输入到三维脉动阵列中，如该输入通道中的特征值矩阵包括3个子特征值矩阵，则在第一个第一预设时间周期将第一个子特征值矩阵输入到三维脉动阵列中，在第二个第一预设时间周期将第二个子特征值矩阵输入到三维脉动阵列中，在第三个第一预设时间周期将第三个子特征值矩阵输入到三维脉动阵列中。
进一步的，不同输入通道中的特征值矩阵可以间隔预设时间段输入到各自对应的三维脉动阵列中，对每个输入通道中的特征值矩阵的处理方案，参照上文所述。
每个计算单元根据目标中间值、接收到的特征值矩阵及存储的权重值矩阵执行计算操作，得到该计算单元对应的子计算结果，该三维脉动阵列中的所有计算单元的子计算结果构成该三维脉动阵列的计算结果。
作为一种优选的实施例，将第i个输入通道中特征值矩阵对应的每个子特征值矩阵输入到第i个三维脉动阵列中的过程包括：
将第i个输入通道中特征值矩阵对应的每个子特征值矩阵的第r行的q个特征值按第二预设时间周期分别输入到第i个三维脉动阵列的Cout个计算单元的第r行的q个PE中，子特征值矩阵的大小为p×q，p、q均为正整数，r＝1，2，…，p-1；
其中，子特征值矩阵中第r+1行的q个特征值输入到第j个计算单元与子特征值矩阵中第r行的q个特征值输入到第j个计算单元的时间间隔为第二预设时间周期，其中，j＝1，2，…，Cout。
具体的，本实施例对各个子特征值矩阵输入到三维脉动阵列的过程进行限定，假设f＝3，可以理解的是，在经过上述S102的预处理后，三维脉动阵列中的每一计算单元均包括3×9个PE(Processing Elements，处理单元)，每个PE会以预定的步骤和它相邻的PE进行数据传输。下面以其中一个子特征值矩阵的输入过程进行说明，子特征值矩阵的大小为p×q， 本实施例中，p＝3，q＝9，参照图4所示，在第一个第二预设周期，将子特征矩阵第一行的9个特征数据一一对应输入到第一计算单元的第一行的9个PE中，在第二个第二预设周期，将子特征矩阵第一行的9个特征数据一一对应输入到第二计算单元的第一行的9个PE中，在第三个第二预设周期，将子特征矩阵第一行的9个特征数据一一对应输入到第三计算单元的第一行的9个PE中，以此类推。进一步的，为提高计算并行度，在第二个第二预设周期，在将子特征矩阵第一行的9个特征数据一一对应输入到第二计算单元的第一行的9个PE中的同时，将子特征矩阵第二行的9个特征数据一一对应输入到第一计算单元的第二行的9个PE中。
作为一种优选的实施例通过每个计算单元根据目标中间值、接收到的特征值矩阵及存储的权重值矩阵执行计算操作，得到该计算单元对应的子计算结果的过程包括：
通过每个计算单元的第r行的q个PE按第一关系式执行计算操作，得到每个PE的计算结果；
第一关系式为h rw＝t rw×q rw+c rw，h w为第r行、第w个PE的计算结果，t w为第r行、第w个PE接收到的特征值，q w为第r行、第w个PE的权重值，c w为第r行、第w个PE对应的目标中间值，w＝1，2，…，q；
根据相同列的所有PE的计算结果的和得到该计算单元的子计算结果。作为一种优选的实施例，根据第Cin个计算结果得到视频数据处理结果的过程具体包括：
获取第Cin个三维脉动阵列中每个计算单元的输出结果；
根据Cout个计算单元的输出结果得到视频数据处理结果。
作为一种优选的实施例，获取第Cin个三维脉动阵列中每个计算单元的输出结果的过程具体包括：
通过第二关系式获取第Cin个三维脉动阵列中每个计算单元的输出结果，第二关系式为
Figure PCTCN2021089924-appb-000003
具体的，下面对三维脉动阵列的具体计算过程进行解释说明：
首先从第一个输入通道的特征值矩阵中选取第一子特征值矩阵，包括 t1-1-1、t1-1-2、…、t1-1-9、t1-2-1、t1-2-2、…、t1-2-9、t1-3-1、t1-3-2、…、t1-3-9共27个特征数据，从第一个输入通道的特征值矩阵中选取第二子特征值矩阵，包括t2-1-1、t2-1-2、…、t2-1-9、t2-2-1、t2-2-2、…、t2-2-9、t2-3-1、t2-3-2、…、t2-3-9共27个特征数据，从第一个输入通道的特征值矩阵中选取第三子特征值矩阵，包括t3-1-1、t3-1-2、…、t3-1-9、t3-2-1、t3-2-2、…、t3-2-9、t3-3-1、t3-3-2、…、t3-3-9共27个特征数据，从第一个输入通道的特征值矩阵中选取第四子特征值矩阵，包括t4-1-1、t4-1-2、…、t4-1-9、t4-2-1、t4-2-2、…、t4-2-9、t4-3-1、t4-3-2、…、t4-3-9共27个特征数据，以此类推，在第二个输入通道的特征值矩阵中选取第一子特征值矩阵，包括t’1-1-1、t’1-1-2、…、t’1-1-9、t’1-2-1、t’1-2-2、…、t’1-2-9、t’1-3-1、t’1-3-2、…、t’1-3-9共27个特征数据。
在第一个预设时间周期内，特征数据t1-1-1、t1-1-2、…、t1-1-9送到图4所示左斜条纹区域的最上一行的9个PE中，分别与权重值q1-1-1、q1-1-2、…、q1-1-9对应相乘，即t1-1-1和q1-1-1相乘，t1-1-2和q1-1-2相乘，以此类推，分别得到的中间值设为m1-1-1、m1-1-2、…、m1-1-9，分别作为下一行各个PE的目标中间值。
在第二个预设时间周期内，特征数据t1-2-1、t1-2-2、…、t1-2-9送到图4所示左斜条纹的中间一行的9个PE中，与权重值q1-2-1、q1-2-2、…、q1-2-9对应相乘，并与输入的中间值m1-1-1、m1-1-2、…、m1-1-9相加，即t1-2-1和q1-2-1相乘并加上m1-1-1，t1-2-2和q1-2-2相乘并加上m1-1-2，以此类推，得到的中间值设为m1-2-1、m1-2-2、…、m1-2-9。同时，特征数据的t2-1-1、t2-1-2、…、t2-1-9送到图4所示左斜条纹的最上一行的9个PE中，与q1-1-1、q1-1-2、…、q1-1-9对应相乘，即t2-1-1和q1-1-1相乘，t2-1-2和q1-1-2相乘，以此类推，得到的中间值设为m2-1-1、m2-1-2、…、m2-1-9。同时，特征数据t1-1-1、t1-1-2、…、t1-1-9送到图4所示右斜条纹的最上一行的9个PE中，与q2-1-1、q2-1-2、…、q2-1-9对应相乘，即t1-1-1和q2-1-1相乘，t1-1-2和q2-1-2相乘，以此类推，得到的中间值设为n1-1-1、n1-1-2、…、n1-1-9。
在第三个预设时间周期内，特征数据t1-3-1、t1-3-2、…、t1-3-9送到 图4所示左斜条纹的最下一行的9个PE中，与q1-3-1、q1-3-2、…、q1-3-9对应相乘，并与输入的中间值m1-2-1、m1-2-2、…、m1-2-9相加，即t1-3-1和q1-3-1相乘并加上m1-2-1，t1-3-2和q1-3-2相乘并加上m1-2-2，以此类推，得到的中间值设为m1-3-1、m1-3-2、…、m1-3-9。同时，特征数据t2-2-1、t2-2-2、…、t2-2-9送到图4所示左斜条纹的中间一行的9个PE中，与q1-2-1、q1-2-2、…、q1-2-9对应相乘，并与输入的中间值m2-1-1、m2-1-2、…、m2-1-9相加，即t2-2-1和q1-2-1相乘并加上m2-1-1，t2-2-2和q1-2-2相乘并加上m2-1-2，以此类推，得到的中间值设为m2-2-1、m2-2-2、…、m2-2-9。同时，特征数据t3-1-1、t3-1-2、…、t3-1-9送到图4所示左斜条纹的最上一行的9个PE中，与q1-1-1、q1-1-2、…、q1-1-9对应相乘，即t3-1-1和q1-1-1相乘，t3-1-2和q1-1-2相乘，以此类推，得到的中间值设为m3-1-1、m3-1-2、…、m3-1-9。同时，特征数据t1-2-1、t1-2-2、…、t1-2-9送到图4所示右斜条纹的中间一行的9个PE中，与q2-2-1、q2-2-2、…、q2-2-9对应相乘，并与输入的中间值n1-1-1、n1-1-2、…、n1-1-9相加，即t1-2-1和q2-2-1相乘并加上n1-1-1，t1-2-2和q2-2-2相乘并加上n1-1-2，以此类推，得到的中间值设为n1-2-1、n1-2-2、…、n1-2-9。同时，特征数据t2-1-1、t2-1-2、…、t2-1-9送到图4所示右斜条纹的最上一行的9个PE中，与q2-1-1、q2-1-2、…、q2-1-9对应相乘，即t2-1-1和q2-1-1相乘，t2-1-2和q2-1-2相乘，以此类推，得到的中间值设为n2-1-1、n2-1-2、…、n2-1-9。同时，特征数据t1-1-1、t1-1-2、…、t1-1-9送到图4所示右斜条纹后面一组(图4中为了显示效果，没有画出)的最上一行的9个PE中，与q3-1-1、q3-1-2、…、q3-1-9对应相乘，即t1-1-1和q3-1-1相乘，t1-1-2和q3-1-2相乘，以此类推，得到的中间值设为o1-1-1、o1-1-2、…、o1-1-9。
在第四个预设时间周期内，特征数据t1-1-1、t1-1-2、…、t1-1-9送到图4所示右斜条纹后面第二组(图4中为了显示效果，没有画出)的最上一行的9个PE中，与q4-1-1、q4-1-2、…、q4-1-9对应相乘，即t1-1-1和q4-1-1相乘，t1-1-2和q4-1-2相乘，以此类推，得到的中间值设为k1-1-1、k1-1-2、…、k1-1-9。同时，特征数据t1-2-1、t1-2-2、…、t1-2-9送到图4所示右斜条纹后面一组的中间一行的9个PE中，与q3-2-1、q3-2-2、…、 q3-2-9对应相乘，并与输入的中间值o1-1-1、o1-1-2、…、o1-1-9相加，即t1-2-1和q3-2-1相乘并加上o1-1-1，t1-2-2和q3-2-2相乘并加上o1-1-2，以此类推，得到的中间值设为o1-2-1、o1-2-2、…、o1-2-9。同时，特征数据t1-3-1、t1-3-2、…、t1-3-9送到图4所示右斜条纹的最下一行的9个PE中，与q2-3-1、q2-3-2、…、q2-3-9对应相乘，并与输入的中间值n1-2-1、n1-2-2、…、n1-2-9相加，即t1-3-1和q1-3-1相乘并加上m1-2-1，t1-3-2和q1-3-2相乘并加上m1-2-2，以此类推，得到的中间值设为m1-3-1、m1-3-2、…、m1-3-9。同时，特征数据t2-1-1、t2-1-2、…、t2-1-9送到图4所示右斜条纹后面一组(图4中为了显示效果，没有画出)的最上一行的9个PE中，与q3-1-1、q3-1-2、…、q3-1-9对应相乘，即t2-1-1和q3-1-1相乘，t2-1-2和q3-1-2相乘，以此类推，得到的中间值设为o2-1-1、o2-1-2、…、o2-1-9。同时，特征数据t2-2-1、t2-2-2、…、t2-2-9送到图4所示右斜条纹的中间一行的9个PE中，与q2-2-1、q2-2-2、…、q2-2-9对应相乘，并与输入的中间值n2-1-1、n2-1-2、…、n2-1-9相加，即t2-2-1和q2-2-1相乘并加上n2-1-1，t2-2-2和q2-2-2相乘并加上n2-1-2，以此类推，得到的中间值设为n2-2-1、n2-2-2、…、n2-2-9。同时，特征数据t2-3-1、t2-3-2、…、t2-3-9送到图4所示左斜条纹的最下一行的9个PE中，与q1-3-1、q1-3-2、…、q1-3-9对应相乘，并与输入的中间值m2-2-1、m2-2-2、…、m2-2-9相加，即t2-3-1和q1-3-1相乘并加上m2-2-1，t2-3-2和q1-3-2相乘并加上m2-2-2，以此类推，得到的中间值设为m2-3-1、m2-3-2、…、m2-3-9。同时，特征数据t3-1-1、t3-1-2、…、t3-1-9送到图4所示右斜条纹的最上一行的9个PE中，与q2-1-1、q2-1-2、…、q2-1-9对应相乘，即t3-1-1和q2-1-1相乘，t3-1-2和q2-1-2相乘，以此类推，得到的中间值设为n3-1-1、n3-1-2、…、n3-1-9。同时，特征数据t3-2-1、t3-2-2、…、t3-2-9送到图4所示左斜条纹的中间一行的9个PE中，与q1-2-1、q1-2-2、…、q1-2-9对应相乘，并与输入的中间值m3-1-1、m3-1-2、…、m3-1-9相加，即t3-2-1和q1-2-1相乘并加上m3-1-1，t3-2-2和q1-2-2相乘并加上m3-1-2，以此类推，得到的中间值设为m3-2-1、m3-2-2、…、m3-2-9。同时，特征数据t4-1-1、t4-1-2、…、t4-1-9送到图4所示左斜条纹的最上一行的9个PE中，与q1-1-1、q1-1-2、…、 q1-1-9对应相乘，即t4-1-1和q1-1-1相乘，t4-1-2和q1-1-2相乘，以此类推，得到的中间值设为m4-1-1、m4-1-2、…、m4-1-9。同时，第二个输入通道的特征数据t’1-1-1、t’1-1-2、…、t’1-1-9送到图5所示输入通道2特征值所对的左斜条纹的最上一行的9个PE中，与q’1-1-1、q’1-1-2、…、q’1-1-9对应相乘，并与输入的中间值m1-3-1、m1-3-2、…、m1-3-9相加即t’1-1-1和q’1-1-1相乘并加上m1-3-1，t’1-1-2和q1-1-2相乘并加上m1-3-2，以此类推，得到的中间值设为m’1-1-1、m’1-1-2、…、m’1-1-9，后面的计算步骤基本就是上面的叠加和重复。
由于是随着时间周期逐步分批启动各级运算单元，特征数据将在这些预算单元内水平传送，如图6中的实线箭头Y所示。各级乘累加的中间值在多组三维运算阵列之中进行传递，如图6中的纵向虚线箭头V所示。当Cin组的三维脉动阵列结果都纵向累加到了一起，就还需要额外将这个排列中的9列的结果再进行累加成一个数值，也就是由3×9×Cin乘积相加得来，得到的结果是一个输出通道中的一个数值，然后重复上述的过程，直到完成全部的计算。
这里再做进一步的扩展，由于深度学习网络中往往输入通道和输出通道数目比较大，达到1000以上的规模，因此完全串行计算很慢。这里就重复上面的三维计算阵列，实现多路并行计算。但多个三维计算阵列中的权重值是互不相同的，而且这些不同的三维计算阵列得到的结果需要再次累加才是最终的结果。
请参照图7，图7为本申请所提供的一种视频数据处理系统的结构示意图，该视频数据处理系统包括：
获取模块11，用于获取视频数据对应的三维特征数据和三维权重数据；
预处理模块12，用于对三维特征数据和三维权重数据分别进行预处理，得到特征值矩阵及权重值矩阵；
计算模块13，用于将特征值矩阵和权重值矩阵输入多个三维脉动阵列 并行计算，得到视频数据处理结果。
可见，本实施例中，对视频数据的三维特征值和三维权重值采取先降维再升维的预处理，在可行的条件下，充分扩展计算的并行度，采用多重三维脉动阵列构造四维脉动计算架构对特征值矩阵和权重值矩阵进行并行计算，缩短了三维卷积的计算时间，提高了视频数据处理效率。
作为一种优选的实施例，预处理模块12包括：
第一预处理单元，用于对所述三维特征数据按卷积核尺寸进行切割，得到多个子特征值矩阵，按预设映射关系将各个所述子特征值矩阵转换成各自对应的子特征值矩阵，根据所有所述子特征值矩阵得到特征值矩阵。
作为一种优选的实施例，预处理模块12还包括：
第二预处理单元，用于将三维权重数据按预设映射关系进行重排，得到权重值矩阵。
作为一种优选的实施例，计算模块3具体用于：
通过第i个三维脉动阵列根据对应的目标中间值对第i个输入通道中的特征值矩阵及权重值矩阵执行计算操作，得到第i个计算结果，i＝1，2，…，Cin；
根据第Cin个计算结果得到视频数据处理结果；
其中，当i＝1时，目标中间值为0，当1＜i≤Cin时，目标中间值为第i-1个计算结果。
作为一种优选的实施例，通过第i个三维脉动阵列根据对应的目标中间值对第i个输入通道中的特征值矩阵及权重值矩阵执行计算操作，得到第i个计算结果的过程具体包括：
将第i个输入通道中的特征值矩阵对应的Cout个权重值矩阵分别存储至第i个三维脉动阵列的Cout个计算单元中，Cout为输出通道个数；
将第i个输入通道中特征值矩阵对应的各个子特征值矩阵按第一预设时间周期依次输入到第i个三维脉动阵列中；
通过每个计算单元根据目标中间值、接收到的特征值矩阵及存储的权重值矩阵执行计算操作，得到该计算单元对应的子计算结果；
通过所有子计算结果，得到第i个计算结果。
作为一种优选的实施例，将第i个输入通道中特征值矩阵对应的每个子特征值矩阵输入到第i个三维脉动阵列中的过程包括：
将第i个输入通道中特征值矩阵对应的每个子特征值矩阵的第r行的q个特征值按第二预设时间周期分别输入到第i个三维脉动阵列的Cout个计算单元的第r行的q个PE中，子特征值矩阵的大小为p×q，p、q均为正整数，r＝1，2，…，p-1；
其中，子特征值矩阵中第r+1行的q个特征值输入到第j个计算单元与子特征值矩阵中第r行的q个特征值输入到第j个计算单元的时间间隔为第二预设时间周期，其中，j＝1，2，…，Cout。
作为一种优选的实施例通过每个计算单元根据目标中间值、接收到的特征值矩阵及存储的权重值矩阵执行计算操作，得到该计算单元对应的子计算结果的过程包括：
通过每个计算单元的第r行的q个PE按第一关系式执行计算操作，得到每个PE的计算结果；
第一关系式为h rw＝t rw×q rw+c rw，h w为第r行、第w个PE的计算结果，t w为第r行、第w个PE接收到的特征值，q w为第r行、第w个PE的权重值，c w为第r行、第w个PE对应的目标中间值，w＝1，2，…，q；
根据相同列的所有PE的计算结果的和得到该计算单元的子计算结果。
作为一种优选的实施例，根据第Cin个计算结果得到视频数据处理结果的过程具体包括：
获取第Cin个三维脉动阵列中每个计算单元的输出结果；
根据Cout个计算单元的输出结果得到视频数据处理结果。
作为一种优选的实施例，获取第Cin个三维脉动阵列中每个计算单元的输出结果的过程具体包括：
通过第二关系式获取第Cin个三维脉动阵列中每个计算单元的输出结果，第二关系式为
Figure PCTCN2021089924-appb-000004
另一方面，本申请还提供了一种电子设备，包括：
存储器，用于存储计算机程序；
处理器，用于执行所述计算机程序时实现如上文任意一个实施例所描述的视频数据处理方法的步骤。
在本申请实施例中，该处理器，可以为中央处理器(Central Processing Unit，CPU)，特定应用集成电路，数字信号处理器，现成可编程门阵列或者其他可编程逻辑器件等。
对于本申请所提供的一种电子设备的介绍请参照上述实施例，本申请在此不再赘述。
本申请所提供的一种电子设备具有和上述视频数据处理方法相同的有益效果。
另一方面，本申请还提供了一种计算机可读存储介质，计算机可读存储介质上存储有计算机程序，计算机程序被处理器执行时实现如上文任意一个实施例所描述的视频数据处理方法的步骤。
对于本申请所提供的一种计算机可读存储介质的介绍请参照上述实施例，本申请在此不再赘述。
本申请所提供的一种计算机可读存储介质具有和上述视频数据处理方法相同的有益效果。
还需要说明的是，在本说明书中，诸如第一和第二等之类的关系术语仅仅用来将一个实体或者操作与另一个实体或操作区分开来，而不一定要求或者暗示这些实体或操作之间存在任何这种实际的关系或者顺序。而且，术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者设备不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者设备所固有的要素。在没有更多限制的状况下，由语句“包括一个…”限定的要素，并不排除在包括所述要素的过程、方法、物品或者设备中还存在另外的相同要素。
对所公开的实施例的上述说明，使本领域专业技术人员能够实现或使用本申请。对这些实施例的多种修改对本领域的专业技术人员来说将是显 而易见的，本文中所定义的一般原理可以在不脱离本申请的精神或范围的情况下，在其他实施例中实现。因此，本申请将不会被限制于本文所示的这些实施例，而是要符合与本文所公开的原理和新颖特点相一致的最宽的范围。

Claims (12)

  1. 一种视频数据处理方法，其特征在于，包括：
    获取视频数据对应的三维特征数据和三维权重数据；
    对所述三维特征数据和所述三维权重数据分别进行预处理，得到特征值矩阵及权重值矩阵；
    将所述特征值矩阵和所述权重值矩阵输入多个三维脉动阵列并行计算，得到视频数据处理结果。
  2. 根据权利要求1所述的视频数据处理方法，其特征在于，对所述三维特征数据进行预处理得到特征值矩阵的过程具体包括：
    对所述三维特征数据按卷积核尺寸进行切割，得到多个特征数据组，按预设映射关系将各个所述特征数据组转换成各自对应的二维矩阵；
    根据所有所述二维矩阵得到特征值矩阵。
  3. 根据权利要求2所述的视频数据处理方法，其特征在于，对所述三维权重数据进行预处理得到权重值矩阵的过程具体包括：
    将所述三维权重数据按所述预设映射关系进行重排，得到权重值矩阵。
  4. 根据权利要求1-3任意一项所述的视频数据处理方法，其特征在于，所述将所述特征值矩阵和所述权重值矩阵输入多个三维脉动阵列并行计算，得到视频数据处理结果的过程具体包括：
    通过第i个三维脉动阵列根据对应的目标中间值对第i个输入通道中的所述特征值矩阵及所述权重值矩阵执行计算操作，得到第i个计算结果，i＝1，2，…，Cin；
    根据第Cin个计算结果得到视频数据处理结果；
    其中，当i＝1时，所述目标中间值为0，当1＜i≤Cin时，所述目标中间值为第i-1个计算结果。
  5. 根据权利要求4所述的视频数据处理方法，其特征在于，所述通过第i个三维脉动阵列根据对应的目标中间值对第i个输入通道中的所述特征值矩阵及所述权重值矩阵执行计算操作，得到第i个计算结果的过程具体包括：
    将第i个输入通道中的所述特征值矩阵对应的Cout个权重值矩阵分别 存储至第i个三维脉动阵列的Cout个计算单元中，Cout为输出通道个数；
    将所述第i个输入通道中特征值矩阵对应的各个子特征值矩阵按第一预设时间周期依次输入到所述第i个三维脉动阵列中；
    通过每个所述计算单元根据目标中间值、接收到的特征值矩阵及存储的权重值矩阵执行计算操作，得到该计算单元对应的子计算结果；
    通过所有所述子计算结果，得到第i个计算结果。
  6. 根据权利要求5所述的视频数据处理方法，其特征在于，将所述第i个输入通道中特征值矩阵对应的每个子特征值矩阵输入到所述第i个三维脉动阵列中的过程包括：
    将所述第i个输入通道中特征值矩阵对应的每个子特征值矩阵的第r行的q个特征值按第二预设时间周期分别输入到所述第i个三维脉动阵列的Cout个计算单元的第r行的q个PE中，所述子特征值矩阵的大小为p×q，p、q均为正整数，r＝1，2，…，p-1；
    其中，所述子特征值矩阵中第r+1行的q个特征值输入到第j个计算单元与所述子特征值矩阵中第r行的q个特征值输入到所述第j个计算单元的时间间隔为所述第二预设时间周期，其中，j＝1，2，…，Cout。
  7. 根据权利要求6所述的视频数据处理方法，其特征在于，所述通过每个所述计算单元根据目标中间值、接收到的特征值矩阵及存储的权重值矩阵执行计算操作，得到该计算单元对应的子计算结果的过程包括：
    通过每个所述计算单元的第r行的q个PE按第一关系式执行计算操作，得到每个PE的计算结果；
    所述第一关系式为h rw＝t rw×q rw+c rw，h w为第r行、第w个PE的计算结果，t w为第r行、第w个PE接收到的特征值，q w为第r行、第w个PE的权重值，c w为第r行、第w个PE对应的目标中间值，w＝1，2，…，q；
    根据相同列的所有所述PE的计算结果的和得到该计算单元的子计算结果。
  8. 根据权利要求7所述的视频数据处理方法，其特征在于，所述根据第Cin个计算结果得到视频数据处理结果的过程具体包括：
    获取第Cin个三维脉动阵列中每个所述计算单元的输出结果；
    根据Cout个所述计算单元的输出结果得到视频数据处理结果。
  9. 根据权利要求8所述的视频数据处理方法，其特征在于，所述获取第Cin个三维脉动阵列中每个所述计算单元的输出结果的过程具体包括：
    通过第二关系式获取第Cin个三维脉动阵列中每个所述计算单元的输出结果，所述第二关系式为
    Figure PCTCN2021089924-appb-100001
  10. 一种视频数据处理系统，其特征在于，包括：
    获取模块，用于获取视频数据对应的三维特征数据和三维权重数据；
    预处理模块，用于对所述三维特征数据和所述三维权重数据分别进行预处理，得到特征值矩阵及权重值矩阵；
    计算模块，用于将所述特征值矩阵和所述权重值矩阵输入多个三维脉动阵列并行计算，得到视频数据处理结果。
  11. 一种电子设备，其特征在于，包括：
    存储器，用于存储计算机程序；
    处理器，用于执行所述计算机程序时实现如权利要求1-9任意一项所述的视频数据处理方法的步骤。
  12. 一种计算机可读存储介质，其特征在于，所述计算机可读存储介质上存储有计算机程序，所述计算机程序被处理器执行时实现如权利要求1-9任意一项所述的视频数据处理方法的步骤。
PCT/CN2021/089924 2020-09-25 2021-04-26 一种视频数据处理方法、系统及相关组件 Ceased WO2022062388A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US18/012,939 US20230267740A1 (en) 2020-09-25 2021-04-26 Video data processing method and system, and relevant assemblies

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202011026282.4A CN112016522B (zh) 2020-09-25 2020-09-25 一种视频数据处理方法、系统及相关组件
CN202011026282.4 2020-09-25

Publications (1)

Publication Number Publication Date
WO2022062388A1 true WO2022062388A1 (zh) 2022-03-31

Family

ID=73528240

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/089924 Ceased WO2022062388A1 (zh) 2020-09-25 2021-04-26 一种视频数据处理方法、系统及相关组件

Country Status (3)

Country Link
US (1) US20230267740A1 (zh)
CN (1) CN112016522B (zh)
WO (1) WO2022062388A1 (zh)

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112016522B (zh) * 2020-09-25 2022-06-07 苏州浪潮智能科技有限公司 一种视频数据处理方法、系统及相关组件
CN113011566B (zh) * 2021-03-30 2025-02-14 北京深演智能科技股份有限公司 数据处理方法、电子设备及计算机可读存储介质
WO2023097423A1 (en) * 2021-11-30 2023-06-08 Intel Corporation Apparatus and method for dynamic quadruple convolution in 3d cnn
CN117275058B (zh) * 2023-08-29 2026-04-17 深圳云天励飞技术股份有限公司 人脸姿态预估方法、装置、系统及电子设备
CN119091446B (zh) * 2024-11-06 2025-03-25 中科南京智能技术研究院 图像特征提取方法、装置及系统

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190065903A1 (en) * 2017-08-22 2019-02-28 Northrop Grumman Systems Corporation System and method for distributive training and weight distribution in a neural network
CN110785778A (zh) * 2018-08-14 2020-02-11 深圳市大疆创新科技有限公司 基于脉动阵列的神经网络处理装置
CN111695672A (zh) * 2019-03-14 2020-09-22 百度（美国）有限责任公司 用于提高ai引擎mac利用率的方法
CN112016522A (zh) * 2020-09-25 2020-12-01 苏州浪潮智能科技有限公司 一种视频数据处理方法、系统及相关组件

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP6771018B2 (ja) * 2015-07-23 2020-10-21 マイヤプリカ テクノロジー エルエルシー 二次元配列プロセッサの性能向上
WO2019136761A1 (zh) * 2018-01-15 2019-07-18 深圳鲲云信息科技有限公司 一种用于识别人为动作的三维卷积装置
CN108416427A (zh) * 2018-02-22 2018-08-17 重庆信络威科技有限公司 卷积核堆积数据流、压缩编码以及深度学习算法
US10572770B2 (en) * 2018-06-15 2020-02-25 Intel Corporation Tangent convolution for 3D data
CN109902821B (zh) * 2019-03-06 2021-03-16 苏州浪潮智能科技有限公司 一种数据处理方法、装置及相关组件
CN110147880A (zh) * 2019-05-22 2019-08-20 苏州浪潮智能科技有限公司 一种神经网络数据处理结构、方法、系统及相关装置

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190065903A1 (en) * 2017-08-22 2019-02-28 Northrop Grumman Systems Corporation System and method for distributive training and weight distribution in a neural network
CN110785778A (zh) * 2018-08-14 2020-02-11 深圳市大疆创新科技有限公司 基于脉动阵列的神经网络处理装置
CN111695672A (zh) * 2019-03-14 2020-09-22 百度（美国）有限责任公司 用于提高ai引擎mac利用率的方法
CN112016522A (zh) * 2020-09-25 2020-12-01 苏州浪潮智能科技有限公司 一种视频数据处理方法、系统及相关组件

Also Published As

Publication number Publication date
CN112016522A (zh) 2020-12-01
US20230267740A1 (en) 2023-08-24
CN112016522B (zh) 2022-06-07

Similar Documents

Publication Publication Date Title
WO2022062388A1 (zh) 一种视频数据处理方法、系统及相关组件
CN109886400B (zh) 基于卷积核拆分的卷积神经网络硬件加速器系统及其计算方法
US10896354B2 (en) Target detection method and device, computing device and readable storage medium
CN107229598B (zh) 一种面向卷积神经网络的低功耗电压可调卷积运算模块
Garipov et al. Ultimate tensorization: compressing convolutional and fc layers alike
CN112819910B (zh) 基于双鬼注意力机制网络的高光谱图像重建方法
WO2021218469A1 (zh) 影像数据检测方法、装置、计算机设备和存储介质
CN110020639B (zh) 视频特征提取方法及相关设备
CN110717527A (zh) 结合空洞空间金字塔结构的目标检测模型确定方法
CN106709441B (zh) 一种基于卷积定理的人脸验证加速方法
CN109871830A (zh) 基于三维深度残差网络的空谱融合高光谱图像分类方法
CN109086244A (zh) 一种基于向量处理器的矩阵卷积向量化实现方法
US11481994B2 (en) Method and apparatus for extracting image data in parallel from multiple convolution windows, device, and computer-readable storage medium
CN109360193A (zh) 一种基于深度学习的原代肿瘤细胞分割识别方法及系统
CN110796235A (zh) 卷积神经网络Valid卷积的向量化实现方法
US20230065725A1 (en) Parallel depth-wise processing architectures for neural networks
CN110147252A (zh) 一种卷积神经网络的并行计算方法及装置
CN117975086A (zh) 一种基于度量元学习的少样本图像分类方法及系统
CN108629405B (zh) 提高卷积神经网络计算效率的方法和装置
WO2023122896A1 (zh) 一种数据处理方法和装置
CN117851742A (zh) 数据存储方法、数据处理方法、数据存储器、数据处理器
Niu et al. Spec2: Spectral sparse cnn accelerator on fpgas
Tida et al. Kernel-segregated transpose convolution operation
WO2019136761A1 (zh) 一种用于识别人为动作的三维卷积装置
Chen et al. Swin transformer with local aggregation

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21870777

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21870777

Country of ref document: EP

Kind code of ref document: A1