WO2020134703A1 - Procédé de traitement d'image à base de système de réseau neuronal et système de réseau neuronal - Google Patents

Procédé de traitement d'image à base de système de réseau neuronal et système de réseau neuronal Download PDF

Info

Publication number
WO2020134703A1
WO2020134703A1 PCT/CN2019/119211 CN2019119211W WO2020134703A1 WO 2020134703 A1 WO2020134703 A1 WO 2020134703A1 CN 2019119211 W CN2019119211 W CN 2019119211W WO 2020134703 A1 WO2020134703 A1 WO 2020134703A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
image data
image frame
layer
neural network
Prior art date
Application number
PCT/CN2019/119211
Other languages
English (en)
Chinese (zh)
Inventor
祝夭龙
何伟
郝轶
Original Assignee
北京灵汐科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 北京灵汐科技有限公司 filed Critical 北京灵汐科技有限公司
Publication of WO2020134703A1 publication Critical patent/WO2020134703A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition

Definitions

  • the invention relates to the technical field of image processing, in particular to an image processing method and a neural network system based on a neural network system.
  • the present invention provides a neural network system-based image processing method and neural network system that overcome the above problems or at least partially solve the above problems.
  • an image processing method based on a neural network system for video analysis.
  • the method includes:
  • the neural network system recognizes the currently input image frame and judges whether the currently input image frame is the basic image frame
  • the N-1 convolutional layer of the neural network system sequentially performs calculation processing on the first image data, and outputs the processed second image data to the Nth layer convolutional layer, where N is greater than or equal to 2 An integer of
  • the Nth layer convolutional layer of the neural network system uses the second image data to replace the first intermediate complete data stored in the Nth layer convolutional layer corresponding to the location of the second image data Data to obtain third image data, and the Nth convolutional layer performs calculation processing on the third image data and outputs; wherein, the first intermediate complete data is the basic image frame in the Nth layer The calculation result of the convolution layer.
  • the N-1 convolutional layer of the neural network system sequentially performs calculation processing on the first image data, specifically including:
  • the i-th convolution layer of the neural network system compares the input image data with the second intermediate complete data stored in the i-th convolution layer, and extracts the input image data and the Two fourth image data having different intermediate complete data, and performing calculation processing on the fourth image data.
  • the N-1 convolutional layer of the neural network system sequentially performs calculation processing on the first image data, specifically including:
  • the k-th convolution layer of the neural network system directly performs calculation processing on the first image data.
  • the currently input image frame is not a basic image frame
  • extract first image data in the currently input image frame that is different from the basic image frame and convert the first image data
  • the output to the neural network system specifically includes:
  • the i-th convolution layer of the neural network system compares the input image data with the second intermediate complete data stored in the i-th convolution layer to extract the input image data
  • the fourth image data different from the second intermediate complete data specifically includes:
  • the i-th convolutional layer of the neural network system divides a frame of image into multiple data blocks based on preset image data block division rules
  • the first image Before the data is output to the neural network system it also includes:
  • the i-th convolutional layer of the neural network system compares the input image data with the second intermediate complete data, including:
  • the i-th convolution layer of the neural network system compares the input image data with the image data at the position corresponding to the identifier in the second intermediate complete data.
  • the identification includes: the starting address of the data block where the input image data is located in the entire image, and the size of each data block.
  • the Nth layer convolutional layer of the neural network system uses the second image data to replace the first intermediate complete data stored in the Nth layer convolutional layer with the second image data
  • the data corresponding to the location obtains the third image data, including:
  • the Nth layer convolutional layer of the neural network system replaces the image data of the position corresponding to the identifier in the first intermediate complete data of the Nth layer convolutional layer with the second image data.
  • the N-1 convolutional layer of the neural network system sequentially performs calculation processing on the first image data, including:
  • the nth layer convolutional layer of the neural network system replaces the image data at the position corresponding to the identifier in the intermediate complete data stored in the nth layer with the image data obtained by the calculation process to obtain the completeness of the nth layer Image data, the complete image data is used as the output data of the next convolution layer.
  • the neural network system recognizes the currently input image frame and determines whether the currently input image frame is a basic image frame, including:
  • the currently input image frame is the basic image frame, and in response to the currently input image frame The number of data frames spaced from the last input basic image frame is not equal to the preset number, then the currently input image frame is a non-base image frame; or
  • the currently input image frame In response to the image of the currently input image frame containing a predetermined file identifier, the currently input image frame is a basic image frame, and the image in response to the currently input image frame does not contain a predetermined file Flag, the currently input image frame is a non-base image frame.
  • the method further includes:
  • a neural network system including: an input layer and a multi-layer convolution layer;
  • the input layer is used to identify the currently input image frame and determine whether the currently input image frame is a basic image frame; in response to the currently input image frame not being a basic image frame, extract the currently input image First image data in a frame different from the basic image frame, output the first image data to the multi-layer convolutional layer;
  • the N-1 convolutional layer of the neural network system is used to sequentially perform calculation processing on the first image data, and output the processed second image data to the Nth layer convolutional layer, where N is Integer greater than or equal to 2;
  • the Nth layer convolution layer of the neural network system is used to replace the first intermediate complete data stored in the Nth layer convolution layer with the second image data corresponding to the location of the second image data
  • the Nth convolutional layer performs calculation processing on the third image data and outputs; wherein, the first intermediate complete data is the basic image frame in the Nth The calculation result of the layer convolution layer.
  • the i-th convolution layer of the neural network system is also used to compare the input image data with the second intermediate complete data stored in the i-th convolution layer to extract the input
  • the fourth image data having different image data from the second intermediate complete data is subjected to calculation processing on the fourth image data; wherein, i is an integer less than N.
  • the k-th layer convolution of the neural network system is also used to directly perform calculation processing on the first image data.
  • the input layer is also used to determine that the currently input image frame is not a basic image frame, and divide a frame of image into multiple data blocks based on a preset image data block division rule;
  • the i-th convolutional layer of the neural network system is further used to divide a frame of image into multiple data blocks based on preset image data block division rules;
  • the input layer is also used to identify the location of the data block where the first image data is located;
  • the i-th convolution layer in the multi-layer convolution layer is also used to compare the input image data with the image data at the position of the mark in the second intermediate complete data.
  • the identification information includes: the starting address of the data block where the input image data is located in the entire image, and the size of each data block.
  • the N-th convolution layer in the multi-layer convolution layer is also used to replace the first intermediate complete data of the N-th convolution layer with the second image data.
  • the image data identifying the corresponding position is described.
  • the n-th convolutional layer of the neural network system is also used to replace the intermediate complete data stored in the n-th layer with image data obtained by calculation before outputting data to the next convolutional layer.
  • the image data at the position corresponding to the input identification information obtains the complete image data of the nth layer, and uses the complete image data as the output data of the next convolution layer.
  • the input layer is further configured to respond to that the number of data frames between the currently input image frame and the last input basic image frame is equal to a preset number, then the current input image frame is based Image frames, in response to that the number of data frames between the currently input image frame and the last input basic image frame is not equal to the preset number, the current input image frame is a non-base image frame; or
  • the currently input image frame In response to the image of the currently input image frame containing a predetermined file identifier, the currently input image frame is a basic image frame, and the image in response to the currently input image frame does not contain a predetermined file Flag, the currently input image frame is a non-base image frame.
  • the input layer is also used to determine that the current input image frame is the basic image frame, and input the entire image data of the currently input image frame to the N-layer convolution of the neural network system.
  • the input image data is sequentially calculated by the N-layer convolutional layer, and the image data obtained after the calculation process is stored in each layer of the N-layer convolutional layer.
  • a storage device in which a computer program is stored, and when the computer program runs in an electronic device, the processor of the electronic device loads and executes any of the above Image processing method based on neural network system.
  • an electronic device including:
  • a processor for running computer programs and
  • the storage device is used to store a computer program, and when the computer program is running in the electronic device, the processor loads and executes the image processing method based on any one of the neural network systems described above.
  • the present invention provides an image processing method and neural network system based on a neural network system, which is applied to video analysis.
  • the system extracts difference data by comparing changes between the current frame and the basic image frame, and the difference data only retains the changes
  • the process of image processing only the change information retained in the difference data is input into the neural network system for data analysis and processing; in this way, in each layer of the neural network, the repeated information in the previous and subsequent frames can be hidden, and Only the changed information is calculated, thereby saving a lot of computing resources, reducing unnecessary information transmission, and speeding up the processing speed.
  • each convolutional layer of the neural network system will save a complete set of image data generated during the processing of the basic image frame. Using the stored image data, it can be restored to a complete integer after each layer of processing The image data is compared again as the input data of the next layer, so as to ensure the processing accuracy of the image by the neural network system.
  • FIG. 1 shows a schematic diagram of a conventional neural network-based image processing process
  • FIG. 2 shows a schematic flowchart of an image processing method based on a neural network system according to an embodiment of the present invention
  • FIG. 3 shows a schematic diagram of data block division according to an embodiment of the present invention
  • FIG. 4 shows a schematic diagram of an image processing process based on a neural network system according to another preferred embodiment of the present invention.
  • FIG. 5 shows a schematic flowchart of an image processing method based on a neural network system according to another preferred embodiment of the present invention
  • FIG. 6 shows a schematic structural diagram of an electronic device according to an embodiment of the present invention.
  • Fig. 1 shows a schematic diagram of a conventional neural network-based image processing process.
  • Fig. 1 shows a schematic diagram of a conventional neural network-based image processing process.
  • the basic image frame data is input to the neural network, for a neural network system with three convolutional layers of L1, L2, and L3, for For each frame of image data received, regardless of whether there is a change in the information between each frame of data, each layer of convolution will perform convolution calculation on its entire image data, resulting in the need to process many duplicate data, and it is not effective Discovering the differential information of the data stream itself has low processing efficiency, high energy consumption, and poor economy.
  • FIG. 2 shows a processing flowchart of an image processing method based on a neural network system according to an embodiment of the present invention, which is applied to the analysis of video.
  • the method includes at least step S201 to step S206.
  • Step S201 the neural network system recognizes the currently input image frame and determines whether the currently input image frame is a basic image frame; if the currently input image frame is a basic image frame, step S202 is executed; if the current input Is not the basic image frame, then step S203 is executed.
  • the basic image frame can be determined by, for example, two methods. First, the neural network system determines whether the number of data frames between the currently input image frame and the last input basic image frame is equal to the preset number If it is, the currently input image frame is a basic image frame, if not, the currently input image frame is a non-basic image frame.
  • the preset number is, for example, 3-5 frames, or other number of frames set based on different situations.
  • another judgment method can be adopted for the judgment of the basic image frame, that is, the neural network system judges whether the image of the currently input image frame contains a predetermined file identifier, and if so, the current input image frame is Basic image frame, if not, the currently input image frame is a non-basic image frame. For example, use the frame data when there is no person in the image data as the basic image frame, or use the frame data of a certain object as the basic image frame, etc.
  • the currently input image frame is a basic image frame according to other methods, which is not limited in the present invention.
  • Step S202 the entire image data of the currently input image frame is sequentially input to the N-layer convolutional layer of the neural network system, and the N-layer convolutional layer of the neural network system sequentially calculates the currently input image data Process and store the intermediate complete data after the calculation process in each of the N convolutional layers.
  • the basic image frame is input into the neural network system, and the basic image frame data is calculated in turn by each convolution layer Process and store the corresponding processing results.
  • the storage method may be, for example, storing the image data after the calculation of the specified convolutional layer in the N-layer convolutional layer in a preset storage device, or storing the intermediate complete data of all convolutional layers in the preset storage device.
  • the intermediate complete data in the embodiment of the present invention refers to the data after the basic image frame is calculated and processed in each convolutional layer, that is, the data after the basic image frame is calculated and processed by any convolutional layer of the neural network system can be used as The intermediate complete data of this layer is stored.
  • the preset storage device in the embodiment of the present invention is mainly used to store data, including the weight of each convolutional layer in the neural network system and the calculated intermediate data.
  • the preset storage device may be a self-use memory of each processor core in the many-core system, or may be a common memory provided on a chip carrying each processing core.
  • the many-core system may be a neural network system that allocates a processor core to each layer of convolutional layers, and each processing core has its own memory.
  • Each convolutional layer can calculate intermediate complete data based on the calculation of the basic image frame. Directly stored on the corresponding core storage.
  • Storage devices include, but are not limited to: static random access memory, dynamic random access memory, flash memory, magnetic change memory, resistance change memory, phase change storage, memristor and other non-volatile memories.
  • the calculation processing for each convolutional layer in the neural network system may include one or more of calculation operations such as convolution, pooling, and relu, which are not limited in the present invention.
  • Step S203 Compare the image data of the currently input image frame with the image data of the basic image frame, extract the first image data different from the basic image frame in the currently input image frame, and input the first image data to the neural network system in.
  • the neural network system When the neural network system recognizes that the currently input image frame is not the basic image frame, it compares it with the basic image frame, and uses the image data of the currently input image frame different from the basic image frame as the first image data.
  • the preset image data block division rule may be used to divide the current input image frame into multiple data blocks; compare the data of the corresponding position of the current input image frame with the basic image frame. If the blocks are the same, extract the data blocks in the currently input image frame that are different from the image data at the corresponding positions of the basic image frame, and output the different data blocks as the first image data to the neural network system.
  • the preset image data block in the embodiment of the present invention may be based on the characteristics of the type and pixel size of the frame data included in the video to be analyzed. To divide, set the size of each data block at the same time, and then use data blocks with different image data as the first image data.
  • Fig. 3 schematically shows the block diagram of the image frame data input to the neural network system.
  • the image frame data is equally divided into 56 data blocks.
  • the currently input image frame and basis The difference data K1 of the image frame spans three data blocks.
  • the three data blocks are input as the first image data to the first layer convolution layer of the neural network for convolution processing.
  • Step S204 the N-1 layer convolution layer of the neural network system sequentially performs calculation processing on the first image data, and outputs the processed second image data to the Nth layer convolution layer, where N is an integer greater than or equal to 2, N is the number of convolutional layers included in the neural network system.
  • the first image data may also be directly calculated and processed by the k-th convolution layer of the neural network system.
  • the k-th convolutional layer is any one of the N-1 convolutional layers in the neural network system, where i and k are all integers less than N.
  • step S205 at least one of the N-1 convolutional layers of the neural network system
  • the first layer compares the input image data of this layer with the saved intermediate complete data, extracts the difference data between the input image data of this layer and the saved intermediate complete data, performs the corresponding convolution operation on the difference data and outputs The next convolutional layer.
  • the i-th convolutional layer of the N-1 convolutional layer can compare the input image data with all The intermediate complete data stored in the i-th convolutional layer is compared, fourth image data that is different from the input image data and the stored intermediate complete data is extracted, and calculation processing is performed on the fourth image data.
  • the image frame data input to the neural network system can be partitioned to determine the first image data that is different from the basic image frame.
  • the i-th convolutional layer of the neural network system it can also be based on a preset image
  • the data block division rule divides a frame of image into multiple data blocks; compare whether the data block where the input image data is located is the same as the data block at the corresponding position of the intermediate complete data stored in the current layer, extract the input image data and store the middle
  • the complete data corresponds to data blocks with different image data at positions, and the different data blocks are used as fourth image data.
  • the N-1 convolutional layer of the neural network system can use the preset image data block division rule shown in step S203 to divide the image frame input to the layer into multiple data blocks, and then determine it by the comparison number A data block in which the currently input image data is different from the corresponding position image data in the intermediate complete data stored in the layer, and the data block is used as difference data for calculation processing.
  • the N-1 convolutional layers of the neural network system can all use the preset image data block division rule shown in step S203 to obtain difference data.
  • the neural network system can also identify the position of the data block where the first image data is located, and then the N-1 convolution layer of the neural network system, Based on the comparison between the input data block with the mark and the image data in the stored intermediate complete data corresponding to the mark. Specifically, the input image data and the image data at the position corresponding to the mark in the intermediate complete data stored in the i-th layer can be performed through the i-th convolution layer in the N-1 convolution layer of the neural network system Compare.
  • the identification of the data block is preferably the starting address of the data block and the size of each data block.
  • the calculation of the input image data by each convolutional layer of the neural network system includes but is not limited to convolution operations, Relu, sigmoid, pooling and other operations.
  • the start address can determine the starting position of the data block in the entire image, and the size of the data block determines the data range of the calculation process. The above information can be given when determining the data block. While performing the data calculation process, a And calculate the address information of the image data input to the next layer after processing, and then pack the address information with the output data of the current layer and transmit it to the next convolutional layer together.
  • the address information when calculating the address information, since the length and width of the data block are known parameters, when the coordinate information of the starting point of the data block is given, the addresses of all the data blocks can be calculated.
  • the difference data K1 spans three data blocks.
  • the starting address information of the three data blocks will also enter the convolutional layer of the neural network system with the difference data, and then calculate the corresponding output data address information.
  • the next convolutional layer can obtain the area of the area to be compared, and then quickly match the data block where the input image data identification is located and the intermediate complete data stored in this layer to the identification Compare the image data of the location.
  • the Nth layer convolution layer of the neural network system replaces the image data at the position corresponding to the identifier in the intermediate complete data with the input image data.
  • the image data input by this layer is compared with the saved intermediate complete data, and after obtaining the difference data, the difference data Perform calculations.
  • n-1 convolution layer of the N-1 convolution layer of the neural network system before outputting data to the next convolution layer, you can also use the calculation process to obtain Replaces the image data at the position corresponding to the input identifier in the stored intermediate complete data, and obtains the complete intermediate data of the convolutional layer as the output data of the next convolutional layer, where n is less than N Integer.
  • the data after the calculation and processing of the convolutional layer of this layer can also be replaced by the input intermediate identification in the corresponding stored intermediate complete data
  • the image data corresponding to the position is restored to the complete image data based on the difference data after the calculation process, and then output to the next layer, and then the complete image data is used as the output data of the layer, thereby meeting the requirements for image accuracy.
  • step S204 may only include step S204, or only include step S205, and may also perform step S204 in some layers of the N-1 convolutional layer, and perform step S205 in some layers.
  • S204 and/or S205 may be set and executed according to actual needs.
  • the method provided by the embodiment of the present invention may further include:
  • Step S206 the Nth layer convolutional layer of the neural network system replaces the data corresponding to the position of the second image data in the intermediate complete data stored in the Nth layer convolutional layer with the second image data, thereby obtaining the first Three image data, the third image data is a complete image data, and the third image data is calculated and output.
  • the last convolutional layer replace the data with the difference between the current input image frame and the basic image frame with the intermediate complete data stored in the last convolutional layer to restore a complete image data. Perform the final convolution operation to obtain the complete data of the neural network data processing of the currently input image frame.
  • the second image data can be used to replace the image of the position corresponding to the input identifier in the intermediate complete data stored in the Nth layer convolutional layer Data, get complete image data quickly, and then calculate and output it.
  • An embodiment of the present invention provides a more efficient image processing method based on a neural network system.
  • the image frame that responds to the current input is the basic image frame, and then the convolution process is performed through the neural network system first, and the image that is currently input
  • the frame is not a basic image frame, compare it with the basic image frame to extract the first image data that is different from the current input image frame and the basic image frame, and then input the first image data into the neural network to perform convolution layer operations in sequence.
  • the last layer of the convolutional layer of the neural network system replace the data at the corresponding position in the intermediate complete data stored in this layer with the image data input from this layer to obtain the image data of the new complete image, and calculate the image data And output.
  • the subsequent image data when processing the subsequent image data, information that has not changed in the preceding and subsequent frames is hidden, and only the changed information is processed, which can save a lot of computing resources, reduce unnecessary information transmission, and speed up the processing speed.
  • any intermediate layer of the neural network system can compare the input image data with the intermediate complete data stored in the current layer, extract the difference data, and perform calculation processing on the difference data, therefore, based on the embodiment of the present invention provides The image processing method can further improve the accuracy of image processing, and then meet different image processing needs.
  • the neural network can be based on specific needs. Any layer of the neural network replaces the data at the corresponding position in the stored intermediate complete data according to the difference data after the calculation process, and outputs the replaced entire image data as the output data of this layer to the next layer of convolution layer, thus Does not affect the accuracy of processing. This not only improves efficiency and reduces system power consumption, but also ensures the processing accuracy of the system.
  • the overall image processing flow based on the embodiment of the present invention may include:
  • Step 1 The input layer receives the currently input image frame, determines that the currently input image frame is the basic image frame data (that is, the frame data corresponding to the time t in FIG. 4), then stores the basic image frame and enters it into L1-
  • the L4 convolutional layer performs calculation processing layer by layer in the neural network, and stores the intermediate complete data after calculation processing in each layer;
  • Step 2 The input layer receives the currently input image frame, determines that the currently input image frame is not the basic image frame (that is, the frame data at time t+1 in FIG. 4), and compares the currently input image frame with the basic image frame , Extract the difference data in the current input image frame and the basic image frame, that is, the data in the box in the figure, determine the data block where the difference data is located, and output the difference data block to the L1 convolution layer for calculation processing;
  • Step 3 After the L1 convolution layer calculates and processes the difference data block, the calculated data is input into the L2 convolution layer;
  • Step 4 The L2 convolutional layer compares the image data input by the L1 convolutional layer with the intermediate complete data stored in L2 to obtain the difference data, and determines the data block where the difference data is located, performs calculation processing on the difference data block, and processes the calculation
  • the data is output as L2 output data to the L3 convolution layer;
  • Step 4 the L3 convolutional layer compares the image data input by L2 with the intermediate complete data stored in L3 to obtain the difference data, and determines the data block where the difference data is located, performs calculation processing on the difference data block, and takes the calculated data as
  • the output data of the third layer is output to L4; when the L2 and L3 convolutional layers are calculated, only relevant data that is different from the image data saved in the previous basic image frame data will be input to the system for processing.
  • the so-called relevant data here , Refers to the data associated with the changed data, that is, the data belonging to the same data block;
  • Step 5 fuse the difference data output by L3 with the intermediate complete data of L4, and replace the data at the same position in the intermediate complete data of L4 with the difference data output by L3 to obtain the image data of a new complete image.
  • the image data of the complete image is subjected to calculation processing, and the image data obtained by the calculation processing is output.
  • step 3 or 4 above after L2 or L3 completes the calculation of the difference data block, by replacing the changed data in the image data of the corresponding layer, the convolutional layer can be restored to the image data of the complete image And input to the next layer, and in the next layer can use the image data of the complete image as input data, so as to ensure the accuracy of the algorithm network.
  • an embodiment of the present invention also provides a neural network system, which may include: an input layer and multiple convolution layers;
  • the input layer is used to identify the currently input image frame, and in response to the currently input image frame not being a basic image frame, extract first image data in the currently input image frame that is different from the basic image frame, Output the first image data to the multi-layer convolution layer;
  • the N-1 convolutional layer of the neural network system is used to sequentially perform calculation processing on the first image data and output the processed second image data to the Nth layer convolutional layer, where N is greater than or equal to An integer of 2;
  • the Nth layer convolutional layer of the neural network system is used to replace the data corresponding to the location of the second image data in the first intermediate complete data stored in the Nth layer convolutional layer with the second image data To obtain third image data, and the Nth convolutional layer performs calculation processing on the third image data and outputs; wherein, the first intermediate complete data is that the basic image frame is in the Nth layer volume The calculation result of the layer.
  • the i-th convolution layer of the neural network system is also used to compare the input image data with the second intermediate complete data stored in the i-th convolution layer, Extracting fourth image data that is different from the input image data and the second intermediate complete data, and performing calculation processing on the fourth image data; wherein, i is an integer less than N.
  • the k-th layer convolution of the neural network system is also used to directly perform calculation processing on the first image data.
  • the input layer is also used to determine that the currently input image frame is not a basic image frame, and divide a frame of image into multiple data based on a preset image data block division rule Piece;
  • the i-th convolution layer of the neural network system is further used to divide a frame of image into multiple data blocks based on a preset image data block division rule;
  • the input layer is also used to identify the location of the data block where the first image data is located;
  • the i-th convolution layer in the multi-layer convolution layer is also used to compare the input image data with the image data at the position of the mark in the second intermediate complete data.
  • the identification information includes: the start address of the data block where the input image data is located in the entire image, and the size of each data block.
  • the N-th convolution layer in the multi-layer convolution layer is also used to replace the first intermediate complete data of the N-th convolution layer with the second image data Image data at the position corresponding to the logo in.
  • the n-th convolution layer of the neural network system is also used to replace the n-th convolution layer with image data obtained by calculation before outputting data to the next convolution layer.
  • the image data at the position corresponding to the input identification information in the intermediate complete data stored in the layer obtains the complete image data of the nth layer, and uses the complete image data as the output data of the next convolution layer.
  • the input layer is also used to respond to that the number of data frames between the currently input image frame and the last input basic image frame is equal to the preset number, then the currently input image frame is Basic image frames, in response to the number of data frames between the currently input image frame and the last input basic image frame being not equal to the preset number, the currently input image frame is a non-basic image frame; or
  • the currently input image frame In response to the image of the currently input image frame containing a predetermined file identification, the currently input image frame is the basic image frame, and in response to the image of the currently input image frame not containing a predetermined file identification, the current input The image frame is a non-base image frame.
  • the input layer is also used to determine that the current input image frame is the basic image frame, and all the entire image data of the currently input image frame is input to the neural network system.
  • the input image data is sequentially processed by the N-level convolutional layer, and the image data obtained after the calculation process is stored in each layer of the N-level convolutional layer.
  • an embodiment of the present invention further provides a storage device in which a computer program is stored, and when the computer program is run in an electronic device, the processor of the electronic device loads and executes any of the above embodiments
  • the image processing method based on neural network system.
  • an embodiment of the present invention also provides an electronic device, including:
  • a processor for running computer programs and
  • the storage device is used to store a computer program, and when the computer program is running in the electronic device, the processor loads and executes the image processing method based on the neural network system described in any of the above embodiments.
  • the electronic device of this embodiment includes processing cores 111, 112-11N and a network on chip 121.
  • the convolutional layers in the convolutional neural network are mapped to the processing cores 111-11N, respectively. It should be understood that one convolutional layer may be mapped to multiple processing cores, or multiple convolutional layers may be mapped to one processing core.
  • the processing cores 111-11N are all connected to the on-chip network 121.
  • the on-chip network 121 is used to exchange data between the N processing cores and external data.
  • At least one processing core of the N processing cores executes the image processing method based on the neural network system described in any of the above embodiments.
  • the processing core 111 may further include a memory 111a, an operator 111b, and a controller 111c.
  • the memory 111a can be used to store program codes
  • the operator 111b is used to perform various calculations on the program codes in the electronic device
  • the controller 111c is coupled to the memory 111a and the operator 111b, and controls the data storage of the memory 111a and the operator 111b to the memory 111a, respectively The data in is calculated according to different calculation logics.
  • the embodiments of the present invention can achieve the following beneficial effects:
  • Embodiments of the present invention provide an image processing method and neural network system based on a neural network system, which are applied to video analysis.
  • the system extracts difference data by comparing changes between the current frame and the basic image frame.
  • the difference data is only Retain the changed data information.
  • image processing only the changed information retained in the difference data is input to the neural network system for data analysis and processing; at the same time, in order to maintain the processing accuracy of the system, a complete basic image frame processing process will be saved
  • the intermediate complete data generated in the process is used to perform comparison at each layer after the intermediate complete data is processed, and the affected and changed intermediate data is extracted again as the input data of the next layer.
  • the repeated information in the previous and subsequent frames can be hidden, and only the changed information is calculated, thereby saving a lot of computing resources, reducing unnecessary information transmission, and speeding up the processing speed.
  • modules in the device in the embodiment can be adaptively changed and set in one or more devices different from the embodiment.
  • the modules or units or components in the embodiments may be combined into one module or unit or component, and in addition, they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and/or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstract and drawings) and any methods so disclosed or All processes or units of equipment are combined. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose.
  • the various component embodiments of the present invention may be implemented in hardware, or implemented in software modules running on one or more processors, or implemented in a combination thereof.
  • a microprocessor or a digital signal processor (DSP) may be used to implement some or all functions of some or all components in the neural network system according to an embodiment of the present invention.
  • DSP digital signal processor
  • the present invention may also be implemented as a device or device program (eg, computer program and computer program product) for performing part or all of the method described herein.
  • Such a program implementing the present invention may be stored on a computer-readable medium, or may have the form of one or more signals.
  • Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Evolutionary Biology (AREA)
  • Evolutionary Computation (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Artificial Intelligence (AREA)
  • Image Analysis (AREA)

Abstract

La présente invention concerne un procédé de traitement d'image basé sur un système de réseau neuronal et un système de réseau neuronal. Selon le procédé fourni par la présente invention, le système de réseau neuronal reconnaît une trame d'image en cours d'entrée, et détermine si la trame d'image en cours d'entrée est une trame d'image de base ; si c'est le cas, une couche de convolution multicouche du réseau neuronal réalise un traitement de calcul sur la trame d'image en cours d'entrée ; si ce n'est pas le cas, premièrement, comparer la trame d'image en cours d'entrée avec des données d'image de la trame d'image de base, extraire des données d'image différentes de la trame d'image de base, puis la couche de convolution multicouche effectue un traitement de calcul en fonction des données d'image, enfin, la dernière couche de la couche de convolution du réseau neuronal remplace les données calculées et fournies par la couche précédente par les données d'image de la position correspondante de données complètes intermédiaires stockées dans la couche pour obtenir les données d'image d'une nouvelle image complète. Selon le procédé fourni par la présente invention, lorsque le système de réseau neuronal traite les données, les informations sans changement dans les trames précédentes et suivantes peuvent être cachées, et seules les informations modifiées sont traitées, ce qui peut économiser beaucoup de ressources informatiques et accélérer la vitesse de traitement.
PCT/CN2019/119211 2018-12-29 2019-11-18 Procédé de traitement d'image à base de système de réseau neuronal et système de réseau neuronal WO2020134703A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201811634090.4 2018-12-29
CN201811634090.4A CN109740508B (zh) 2018-12-29 2018-12-29 一种基于神经网络系统的图像处理方法及神经网络系统

Publications (1)

Publication Number Publication Date
WO2020134703A1 true WO2020134703A1 (fr) 2020-07-02

Family

ID=66362235

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/119211 WO2020134703A1 (fr) 2018-12-29 2019-11-18 Procédé de traitement d'image à base de système de réseau neuronal et système de réseau neuronal

Country Status (2)

Country Link
CN (1) CN109740508B (fr)
WO (1) WO2020134703A1 (fr)

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109740508B (zh) * 2018-12-29 2021-07-23 北京灵汐科技有限公司 一种基于神经网络系统的图像处理方法及神经网络系统
CN113673664B (zh) * 2020-05-14 2023-09-12 杭州海康威视数字技术股份有限公司 数据溢出检测方法、装置、设备及存储介质
CN111831354B (zh) * 2020-07-09 2023-05-16 北京灵汐科技有限公司 数据精度配置方法、装置、芯片、芯片阵列、设备及介质
CN111831356B (zh) * 2020-07-09 2023-04-07 北京灵汐科技有限公司 权重精度配置方法、装置、设备及存储介质
WO2022007879A1 (fr) 2020-07-09 2022-01-13 北京灵汐科技有限公司 Procédé et appareil de configuration de précision de poids, dispositif informatique et support de stockage
CN111737193B (zh) * 2020-08-03 2020-12-08 深圳鲲云信息科技有限公司 数据存储方法、装置、设备和存储介质

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150170338A1 (en) * 2013-12-18 2015-06-18 Huawei Technologies Co., Ltd. Image processing method and apparatus
CN105787867A (zh) * 2016-04-21 2016-07-20 华为技术有限公司 基于神经网络算法的处理视频图像的方法和装置
CN107993217A (zh) * 2017-12-22 2018-05-04 北京奇虎科技有限公司 视频数据实时处理方法及装置、计算设备
CN109740508A (zh) * 2018-12-29 2019-05-10 北京灵汐科技有限公司 一种基于神经网络系统的图像处理方法及神经网络系统

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103440612B (zh) * 2013-08-27 2016-12-28 华为技术有限公司 一种gpu虚拟化中图像处理方法和装置
CN107492115B (zh) * 2017-08-30 2021-01-01 北京小米移动软件有限公司 目标对象的检测方法及装置
CN108876812B (zh) * 2017-11-01 2021-11-19 北京旷视科技有限公司 用于视频中物体检测的图像处理方法、装置及设备

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150170338A1 (en) * 2013-12-18 2015-06-18 Huawei Technologies Co., Ltd. Image processing method and apparatus
CN105787867A (zh) * 2016-04-21 2016-07-20 华为技术有限公司 基于神经网络算法的处理视频图像的方法和装置
CN107993217A (zh) * 2017-12-22 2018-05-04 北京奇虎科技有限公司 视频数据实时处理方法及装置、计算设备
CN109740508A (zh) * 2018-12-29 2019-05-10 北京灵汐科技有限公司 一种基于神经网络系统的图像处理方法及神经网络系统

Also Published As

Publication number Publication date
CN109740508A (zh) 2019-05-10
CN109740508B (zh) 2021-07-23

Similar Documents

Publication Publication Date Title
WO2020134703A1 (fr) Procédé de traitement d'image à base de système de réseau neuronal et système de réseau neuronal
US11263520B2 (en) Instruction generation process multiplexing method and device
CN112990297A (zh) 多模态预训练模型的训练方法、应用方法及装置
US20210089871A1 (en) Processing system and method for binary weight convolutional neural network
CN110781819A (zh) 一种图像目标检测方法、系统、电子设备及存储介质
JP2018116469A (ja) 演算システムおよびニューラルネットワークの演算方法
JP7087825B2 (ja) 学習装置および学習方法
JP7087695B2 (ja) 学習装置および学習方法
WO2022077907A1 (fr) Procédé, système, et dispositif de détection d'attaque antagoniste, et support de stockage lisible par ordinateur
US11948069B2 (en) Compression of neural network activation data
CN112799599B (zh) 一种数据存储方法、计算核、芯片和电子设备
US10031947B2 (en) Method and apparatus for performing a search operation on heterogeneous computing systems
EP3161793A1 (fr) Mécanisme de partition adaptatif avec forme de carreau arbitraire pour architecture gpu de reproduction par carreaux
KR102305575B1 (ko) 이미지 간 유사도를 이용한 유사 영역 강조 방법 및 시스템
CN114445336A (zh) 配电设备缺陷检测方法、装置、计算机设备和存储介质
CN115344805A (zh) 素材审核方法、计算设备及存储介质
Tan et al. Saliency detection by deep network with boundary refinement and global context
US10997497B2 (en) Calculation device for and calculation method of performing convolution
JP7069897B2 (ja) 学習識別装置および学習識別方法
JP7069898B2 (ja) 学習識別装置および学習識別方法
CN115438658A (zh) 一种实体识别方法、识别模型的训练方法和相关装置
CN115984671A (zh) 模型在线更新方法、装置、电子设备及可读存储介质
CN106502775B (zh) 一种分时调度dsp算法的方法和系统
CN113840169B (zh) 一种视频处理方法、装置、计算设备和存储介质
CN113989601A (zh) 特征融合网络、样本选择方法、目标检测方法及装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19903362

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19903362

Country of ref document: EP

Kind code of ref document: A1