WO2020134703A1

WO2020134703A1 - Neural network system-based image processing method and neural network system

Info

Publication number: WO2020134703A1
Application number: PCT/CN2019/119211
Authority: WO
Inventors: 祝夭龙; 何伟; 郝轶
Original assignee: 北京灵汐科技有限公司
Priority date: 2018-12-29
Filing date: 2019-11-18
Publication date: 2020-07-02
Also published as: CN109740508B; CN109740508A

Abstract

The present invention provides a neural network system-based image processing method and a neural network system, according to the method provided by the present invention, the neural network system recognizes a currently input image frame, and determines whether the currently input image frame is a basic image frame; if yes, a multi-layer convolutional layer of the neural network performs computing processing on the currently input image frame; if not, firstly comparing the currently input image frame with image data of the basic image frame, extracting image data different from the basic image frame, then the multi-layer convolutional layer performs computing processing based on the image data, finally, the last layer of the convolutional layer of the neural network replaces the data calculated and output by the previous layer with the image data of the corresponding position of intermediate complete data stored in the layer to obtain the image data of a new complete image. According to the method provided by the present invention, when the neural network system processes the data, the information with no change in the previous and next frames can be hidden, and only the changed information is processed, which can save a lot of computing resources and speed up the processing speed.

Description

An image processing method and neural network system based on neural network system

This application claims the priority of the Chinese patent application filed on December 29, 2018 with the application number 201811634090.4 and the invention titled "An Image Processing Method and Neural Network System Based on Neural Network System", the entire contents of which are cited by reference Incorporated in this application.

Technical field

The invention relates to the technical field of image processing, in particular to an image processing method and a neural network system based on a neural network system.

Background technique

In today's era, with the surging development of artificial intelligence technology, it not only affects people's production and life from all aspects, but also promotes the development and progress of the world. In order to effectively use the information collected from each terminal, the requirements for real-time processing of information are also constantly increasing. Especially in the field of security, the input of video information is large and the amount of calculation is large, resulting in high power consumption. However, in most cases, there is not much difference between the information before and after the frame of the video information. For the traditional scheme, when processing the video frame, regardless of whether the information between frames changes, all the data in each frame will be input into the neural network Processing by algorithms such as systems results in the need to process many duplicate data.

Summary of the invention

In view of the above problems, the present invention provides a neural network system-based image processing method and neural network system that overcome the above problems or at least partially solve the above problems.

According to an aspect of the present invention, an image processing method based on a neural network system is provided for video analysis. The method includes:

The neural network system recognizes the currently input image frame and judges whether the currently input image frame is the basic image frame;

In response to that the currently input image frame is not a basic image frame, extract first image data different from the basic image frame in the currently input image frame, and output the first image data to the neural network system;

The N-1 convolutional layer of the neural network system sequentially performs calculation processing on the first image data, and outputs the processed second image data to the Nth layer convolutional layer, where N is greater than or equal to 2 An integer of

The Nth layer convolutional layer of the neural network system uses the second image data to replace the first intermediate complete data stored in the Nth layer convolutional layer corresponding to the location of the second image data Data to obtain third image data, and the Nth convolutional layer performs calculation processing on the third image data and outputs; wherein, the first intermediate complete data is the basic image frame in the Nth layer The calculation result of the convolution layer.

Optionally, the N-1 convolutional layer of the neural network system sequentially performs calculation processing on the first image data, specifically including:

The i-th convolution layer of the neural network system compares the input image data with the second intermediate complete data stored in the i-th convolution layer, and extracts the input image data and the Two fourth image data having different intermediate complete data, and performing calculation processing on the fourth image data.

The k-th convolution layer of the neural network system directly performs calculation processing on the first image data.

Optionally, in response to that the currently input image frame is not a basic image frame, extract first image data in the currently input image frame that is different from the basic image frame, and convert the first image data The output to the neural network system specifically includes:

In response to that the currently input image frame is not a basic image frame, divide a frame of image into multiple data blocks based on a preset image data block division rule;

Comparing whether each data block of the currently input image frame is the same as the data block corresponding to the basic image frame, and extracting data blocks in the current input image frame that are different from the image data of the corresponding position of the basic image frame , Output the different data blocks as the first image data to the neural network system.

Optionally, the i-th convolution layer of the neural network system compares the input image data with the second intermediate complete data stored in the i-th convolution layer to extract the input image data The fourth image data different from the second intermediate complete data specifically includes:

The i-th convolutional layer of the neural network system divides a frame of image into multiple data blocks based on preset image data block division rules;

Comparing whether the data block where the input image data is located is the same as the data block corresponding to the second intermediate complete data, and extracting data in the input image data that is different from the image data corresponding to the second intermediate complete data Block, using the different data block as the fourth image data.

Optionally, in response to the currently input image frame not being a basic image frame, after extracting first image data in the currently input image frame that is different from the basic image frame, the first image Before the data is output to the neural network system, it also includes:

Identify the location of the data block where the first image data is located;

The i-th convolutional layer of the neural network system compares the input image data with the second intermediate complete data, including:

The i-th convolution layer of the neural network system compares the input image data with the image data at the position corresponding to the identifier in the second intermediate complete data.

Optionally, the identification includes: the starting address of the data block where the input image data is located in the entire image, and the size of each data block.

Optionally, the Nth layer convolutional layer of the neural network system uses the second image data to replace the first intermediate complete data stored in the Nth layer convolutional layer with the second image data The data corresponding to the location obtains the third image data, including:

The Nth layer convolutional layer of the neural network system replaces the image data of the position corresponding to the identifier in the first intermediate complete data of the Nth layer convolutional layer with the second image data.

Optionally, the N-1 convolutional layer of the neural network system sequentially performs calculation processing on the first image data, including:

The nth layer convolutional layer of the neural network system replaces the image data at the position corresponding to the identifier in the intermediate complete data stored in the nth layer with the image data obtained by the calculation process to obtain the completeness of the nth layer Image data, the complete image data is used as the output data of the next convolution layer.

Optionally, the neural network system recognizes the currently input image frame and determines whether the currently input image frame is a basic image frame, including:

In response to that the number of data frames between the currently input image frame and the last input basic image frame is equal to a preset number, the currently input image frame is the basic image frame, and in response to the currently input image frame The number of data frames spaced from the last input basic image frame is not equal to the preset number, then the currently input image frame is a non-base image frame; or

In response to the image of the currently input image frame containing a predetermined file identifier, the currently input image frame is a basic image frame, and the image in response to the currently input image frame does not contain a predetermined file Flag, the currently input image frame is a non-base image frame.

Optionally, the method further includes:

In response to the currently input image frame being the basic image frame, all the entire image data of the currently input image frame is input to the N-layer convolutional layer of the neural network system, and the N-layer convolutional layer pairs The input image data is sequentially calculated, and the image data obtained after the calculation process is stored in each of the N layers of the convolutional layer.

According to another aspect of the present invention, a neural network system is also provided, including: an input layer and a multi-layer convolution layer;

The input layer is used to identify the currently input image frame and determine whether the currently input image frame is a basic image frame; in response to the currently input image frame not being a basic image frame, extract the currently input image First image data in a frame different from the basic image frame, output the first image data to the multi-layer convolutional layer;

The N-1 convolutional layer of the neural network system is used to sequentially perform calculation processing on the first image data, and output the processed second image data to the Nth layer convolutional layer, where N is Integer greater than or equal to 2;

The Nth layer convolution layer of the neural network system is used to replace the first intermediate complete data stored in the Nth layer convolution layer with the second image data corresponding to the location of the second image data To obtain third image data, the Nth convolutional layer performs calculation processing on the third image data and outputs; wherein, the first intermediate complete data is the basic image frame in the Nth The calculation result of the layer convolution layer.

Optionally, the i-th convolution layer of the neural network system is also used to compare the input image data with the second intermediate complete data stored in the i-th convolution layer to extract the input The fourth image data having different image data from the second intermediate complete data is subjected to calculation processing on the fourth image data; wherein, i is an integer less than N.

Optionally, the k-th layer convolution of the neural network system is also used to directly perform calculation processing on the first image data.

Optionally, the input layer is also used to determine that the currently input image frame is not a basic image frame, and divide a frame of image into multiple data blocks based on a preset image data block division rule;

Optionally, the i-th convolutional layer of the neural network system is further used to divide a frame of image into multiple data blocks based on preset image data block division rules;

Optionally, the input layer is also used to identify the location of the data block where the first image data is located;

The i-th convolution layer in the multi-layer convolution layer is also used to compare the input image data with the image data at the position of the mark in the second intermediate complete data.

Optionally, the identification information includes: the starting address of the data block where the input image data is located in the entire image, and the size of each data block.

Optionally, the N-th convolution layer in the multi-layer convolution layer is also used to replace the first intermediate complete data of the N-th convolution layer with the second image data. The image data identifying the corresponding position is described.

Optionally, the n-th convolutional layer of the neural network system is also used to replace the intermediate complete data stored in the n-th layer with image data obtained by calculation before outputting data to the next convolutional layer. The image data at the position corresponding to the input identification information obtains the complete image data of the nth layer, and uses the complete image data as the output data of the next convolution layer.

Optionally, the input layer is further configured to respond to that the number of data frames between the currently input image frame and the last input basic image frame is equal to a preset number, then the current input image frame is based Image frames, in response to that the number of data frames between the currently input image frame and the last input basic image frame is not equal to the preset number, the current input image frame is a non-base image frame; or

Optionally, the input layer is also used to determine that the current input image frame is the basic image frame, and input the entire image data of the currently input image frame to the N-layer convolution of the neural network system. In the layer, the input image data is sequentially calculated by the N-layer convolutional layer, and the image data obtained after the calculation process is stored in each layer of the N-layer convolutional layer.

According to another aspect of the present invention, there is also provided a storage device in which a computer program is stored, and when the computer program runs in an electronic device, the processor of the electronic device loads and executes any of the above Image processing method based on neural network system.

According to another aspect of the present invention, an electronic device is also provided, including:

A processor for running computer programs; and

The storage device is used to store a computer program, and when the computer program is running in the electronic device, the processor loads and executes the image processing method based on any one of the neural network systems described above.

The present invention provides an image processing method and neural network system based on a neural network system, which is applied to video analysis. The system extracts difference data by comparing changes between the current frame and the basic image frame, and the difference data only retains the changes In the process of image processing, only the change information retained in the difference data is input into the neural network system for data analysis and processing; in this way, in each layer of the neural network, the repeated information in the previous and subsequent frames can be hidden, and Only the changed information is calculated, thereby saving a lot of computing resources, reducing unnecessary information transmission, and speeding up the processing speed. In addition, each convolutional layer of the neural network system will save a complete set of image data generated during the processing of the basic image frame. Using the stored image data, it can be restored to a complete integer after each layer of processing The image data is compared again as the input data of the next layer, so as to ensure the processing accuracy of the image by the neural network system.

The above description is only an overview of the technical solutions of the present invention. In order to understand the technical means of the present invention more clearly, it can be implemented in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more obvious and understandable The specific embodiments of the present invention are listed below.

According to the following detailed description of specific embodiments of the present invention in conjunction with the accompanying drawings, those skilled in the art will understand more about the above and other objects, advantages, and features of the present invention.

BRIEF DESCRIPTION

By reading the detailed description of the preferred embodiments below, various other advantages and benefits will become clear to those of ordinary skill in the art. The drawings are only for the purpose of showing the preferred embodiments, and are not considered to limit the present invention. Furthermore, throughout the drawings, the same reference symbols are used to denote the same components. In the drawings:

FIG. 1 shows a schematic diagram of a conventional neural network-based image processing process;

2 shows a schematic flowchart of an image processing method based on a neural network system according to an embodiment of the present invention;

FIG. 3 shows a schematic diagram of data block division according to an embodiment of the present invention;

4 shows a schematic diagram of an image processing process based on a neural network system according to another preferred embodiment of the present invention;

5 shows a schematic flowchart of an image processing method based on a neural network system according to another preferred embodiment of the present invention;

FIG. 6 shows a schematic structural diagram of an electronic device according to an embodiment of the present invention.

detailed description

Hereinafter, exemplary embodiments of the present disclosure will be described in more detail with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

Fig. 1 shows a schematic diagram of a conventional neural network-based image processing process. As can be seen from Fig. 1, after the basic image frame data is input to the neural network, for a neural network system with three convolutional layers of L1, L2, and L3, for For each frame of image data received, regardless of whether there is a change in the information between each frame of data, each layer of convolution will perform convolution calculation on its entire image data, resulting in the need to process many duplicate data, and it is not effective Discovering the differential information of the data stream itself has low processing efficiency, high energy consumption, and poor economy.

FIG. 2 shows a processing flowchart of an image processing method based on a neural network system according to an embodiment of the present invention, which is applied to the analysis of video. Referring to FIG. 2, the method includes at least step S201 to step S206.

Step S201, the neural network system recognizes the currently input image frame and determines whether the currently input image frame is a basic image frame; if the currently input image frame is a basic image frame, step S202 is executed; if the current input Is not the basic image frame, then step S203 is executed.

In this embodiment, the basic image frame can be determined by, for example, two methods. First, the neural network system determines whether the number of data frames between the currently input image frame and the last input basic image frame is equal to the preset number If it is, the currently input image frame is a basic image frame, if not, the currently input image frame is a non-basic image frame. The preset number is, for example, 3-5 frames, or other number of frames set based on different situations.

Optionally, another judgment method can be adopted for the judgment of the basic image frame, that is, the neural network system judges whether the image of the currently input image frame contains a predetermined file identifier, and if so, the current input image frame is Basic image frame, if not, the currently input image frame is a non-basic image frame. For example, use the frame data when there is no person in the image data as the basic image frame, or use the frame data of a certain object as the basic image frame, etc.

In practical applications, it can also be determined whether the currently input image frame is a basic image frame according to other methods, which is not limited in the present invention.

Step S202, the entire image data of the currently input image frame is sequentially input to the N-layer convolutional layer of the neural network system, and the N-layer convolutional layer of the neural network system sequentially calculates the currently input image data Process and store the intermediate complete data after the calculation process in each of the N convolutional layers. It should be noted that during the analysis of the video, when it is judged that the currently input image frame is the basic image frame, the basic image frame is input into the neural network system, and the basic image frame data is calculated in turn by each convolution layer Process and store the corresponding processing results. The storage method may be, for example, storing the image data after the calculation of the specified convolutional layer in the N-layer convolutional layer in a preset storage device, or storing the intermediate complete data of all convolutional layers in the preset storage device. The intermediate complete data in the embodiment of the present invention refers to the data after the basic image frame is calculated and processed in each convolutional layer, that is, the data after the basic image frame is calculated and processed by any convolutional layer of the neural network system can be used as The intermediate complete data of this layer is stored.

The preset storage device in the embodiment of the present invention is mainly used to store data, including the weight of each convolutional layer in the neural network system and the calculated intermediate data. The preset storage device may be a self-use memory of each processor core in the many-core system, or may be a common memory provided on a chip carrying each processing core. Among them, the many-core system may be a neural network system that allocates a processor core to each layer of convolutional layers, and each processing core has its own memory. Each convolutional layer can calculate intermediate complete data based on the calculation of the basic image frame. Directly stored on the corresponding core storage. Storage devices include, but are not limited to: static random access memory, dynamic random access memory, flash memory, magnetic change memory, resistance change memory, phase change storage, memristor and other non-volatile memories. The calculation processing for each convolutional layer in the neural network system may include one or more of calculation operations such as convolution, pooling, and relu, which are not limited in the present invention.

Step S203: Compare the image data of the currently input image frame with the image data of the basic image frame, extract the first image data different from the basic image frame in the currently input image frame, and input the first image data to the neural network system in.

When the neural network system recognizes that the currently input image frame is not the basic image frame, it compares it with the basic image frame, and uses the image data of the currently input image frame different from the basic image frame as the first image data.

For specific comparison, for example, the preset image data block division rule may be used to divide the current input image frame into multiple data blocks; compare the data of the corresponding position of the current input image frame with the basic image frame. If the blocks are the same, extract the data blocks in the currently input image frame that are different from the image data at the corresponding positions of the basic image frame, and output the different data blocks as the first image data to the neural network system.

For the same video to be analyzed, the pixel size of each frame of data is the same. Therefore, the preset image data block in the embodiment of the present invention may be based on the characteristics of the type and pixel size of the frame data included in the video to be analyzed. To divide, set the size of each data block at the same time, and then use data blocks with different image data as the first image data.

Fig. 3 schematically shows the block diagram of the image frame data input to the neural network system. Referring to Fig. 3, it can be seen that the image frame data is equally divided into 56 data blocks. In Fig. 3, the currently input image frame and basis The difference data K1 of the image frame spans three data blocks. At this time, the three data blocks are input as the first image data to the first layer convolution layer of the neural network for convolution processing.

Step S204, the N-1 layer convolution layer of the neural network system sequentially performs calculation processing on the first image data, and outputs the processed second image data to the Nth layer convolution layer, where N is an integer greater than or equal to 2, N is the number of convolutional layers included in the neural network system. In addition, the first image data may also be directly calculated and processed by the k-th convolution layer of the neural network system. The k-th convolutional layer is any one of the N-1 convolutional layers in the neural network system, where i and k are all integers less than N.

In addition, in order to further improve the processing accuracy of the image difference data between the currently input image frame and the basic image frame, optionally, it may further include step S205, at least one of the N-1 convolutional layers of the neural network system The first layer compares the input image data of this layer with the saved intermediate complete data, extracts the difference data between the input image data of this layer and the saved intermediate complete data, performs the corresponding convolution operation on the difference data and outputs The next convolutional layer.

That is to say, when the N-1 convolutional layer of the neural network system sequentially calculates the first image data, the i-th convolutional layer of the N-1 convolutional layer can compare the input image data with all The intermediate complete data stored in the i-th convolutional layer is compared, fourth image data that is different from the input image data and the stored intermediate complete data is extracted, and calculation processing is performed on the fourth image data.

As mentioned above, the image frame data input to the neural network system can be partitioned to determine the first image data that is different from the basic image frame. For the i-th convolutional layer of the neural network system, it can also be based on a preset image The data block division rule divides a frame of image into multiple data blocks; compare whether the data block where the input image data is located is the same as the data block at the corresponding position of the intermediate complete data stored in the current layer, extract the input image data and store the middle The complete data corresponds to data blocks with different image data at positions, and the different data blocks are used as fourth image data. Further, the N-1 convolutional layer of the neural network system can use the preset image data block division rule shown in step S203 to divide the image frame input to the layer into multiple data blocks, and then determine it by the comparison number A data block in which the currently input image data is different from the corresponding position image data in the intermediate complete data stored in the layer, and the data block is used as difference data for calculation processing.

Further, the N-1 convolutional layers of the neural network system can all use the preset image data block division rule shown in step S203 to obtain difference data.

Optionally, in order to more accurately locate and calculate the difference data, the neural network system can also identify the position of the data block where the first image data is located, and then the N-1 convolution layer of the neural network system, Based on the comparison between the input data block with the mark and the image data in the stored intermediate complete data corresponding to the mark. Specifically, the input image data and the image data at the position corresponding to the mark in the intermediate complete data stored in the i-th layer can be performed through the i-th convolution layer in the N-1 convolution layer of the neural network system Compare.

In this embodiment, the identification of the data block is preferably the starting address of the data block and the size of each data block. When identifying each data block, you can first select the starting point in the entire image frame data and set the starting coordinate information; combine the starting coordinate information and the size of the data block to determine the data block where the difference data is located in the entire image frame Address information. That is, when the data block where the difference data is located is identified, the data block where the difference data is located is identified, and then the address of the data block in the image frame to be processed is determined, wherein the selected data block includes the difference data, That is, the difference data is used as a subset of the identified data blocks, so that the calculation accuracy can be improved.

The calculation of the input image data by each convolutional layer of the neural network system includes but is not limited to convolution operations, Relu, sigmoid, pooling and other operations. The start address can determine the starting position of the data block in the entire image, and the size of the data block determines the data range of the calculation process. The above information can be given when determining the data block. While performing the data calculation process, a And calculate the address information of the image data input to the next layer after processing, and then pack the address information with the output data of the current layer and transmit it to the next convolutional layer together.

When calculating the address information, since the length and width of the data block are known parameters, when the coordinate information of the starting point of the data block is given, the addresses of all the data blocks can be calculated. In FIG. 3, the difference data K1 spans three data blocks. The starting address information of the three data blocks will also enter the convolutional layer of the neural network system with the difference data, and then calculate the corresponding output data address information.

After receiving the input image data, the next convolutional layer can obtain the area of the area to be compared, and then quickly match the data block where the input image data identification is located and the intermediate complete data stored in this layer to the identification Compare the image data of the location. Finally, the Nth layer convolution layer of the neural network system replaces the image data at the position corresponding to the identifier in the intermediate complete data with the input image data.

In the above embodiment, after determining that the currently input image frame is the basic image frame, it will be input to each convolutional layer in the neural network system for calculation processing, and correspondingly stored each convolutional layer to process the basic image frame After the intermediate complete data, therefore, for at least one of the N-1 convolutional layers of the neural network system, the image data input by this layer is compared with the saved intermediate complete data, and after obtaining the difference data, the difference data Perform calculations.

In other words, in addition to the above, for the n-1 convolution layer of the N-1 convolution layer of the neural network system, before outputting data to the next convolution layer, you can also use the calculation process to obtain Replaces the image data at the position corresponding to the input identifier in the stored intermediate complete data, and obtains the complete intermediate data of the convolutional layer as the output data of the next convolutional layer, where n is less than N Integer. In other words, for the neural network system, in addition to any convolutional layer of the last convolutional layer, the data after the calculation and processing of the convolutional layer of this layer can also be replaced by the input intermediate identification in the corresponding stored intermediate complete data The image data corresponding to the position is restored to the complete image data based on the difference data after the calculation process, and then output to the next layer, and then the complete image data is used as the output data of the layer, thereby meeting the requirements for image accuracy.

It should be noted that the embodiment of the present invention may only include step S204, or only include step S205, and may also perform step S204 in some layers of the N-1 convolutional layer, and perform step S205 in some layers. For limitation, S204 and/or S205 may be set and executed according to actual needs.

Finally, referring to FIG. 2, the method provided by the embodiment of the present invention may further include:

Step S206, the Nth layer convolutional layer of the neural network system replaces the data corresponding to the position of the second image data in the intermediate complete data stored in the Nth layer convolutional layer with the second image data, thereby obtaining the first Three image data, the third image data is a complete image data, and the third image data is calculated and output.

In the last convolutional layer, replace the data with the difference between the current input image frame and the basic image frame with the intermediate complete data stored in the last convolutional layer to restore a complete image data. Perform the final convolution operation to obtain the complete data of the neural network data processing of the currently input image frame. Similarly, for the Nth layer convolutional layer of the neural network system when recovering complete data, the second image data can be used to replace the image of the position corresponding to the input identifier in the intermediate complete data stored in the Nth layer convolutional layer Data, get complete image data quickly, and then calculate and output it.

An embodiment of the present invention provides a more efficient image processing method based on a neural network system. First, the image frame that responds to the current input is the basic image frame, and then the convolution process is performed through the neural network system first, and the image that is currently input If the frame is not a basic image frame, compare it with the basic image frame to extract the first image data that is different from the current input image frame and the basic image frame, and then input the first image data into the neural network to perform convolution layer operations in sequence. , In the last layer of the convolutional layer of the neural network system, replace the data at the corresponding position in the intermediate complete data stored in this layer with the image data input from this layer to obtain the image data of the new complete image, and calculate the image data And output. In the embodiment of the present invention, when processing the subsequent image data, information that has not changed in the preceding and subsequent frames is hidden, and only the changed information is processed, which can save a lot of computing resources, reduce unnecessary information transmission, and speed up the processing speed.

Further, since any intermediate layer of the neural network system can compare the input image data with the intermediate complete data stored in the current layer, extract the difference data, and perform calculation processing on the difference data, therefore, based on the embodiment of the present invention provides The image processing method can further improve the accuracy of image processing, and then meet different image processing needs.

In addition to the above introduction, since the intermediate complete data calculated and processed based on the basic image frame will also be stored corresponding to the convolutional layers of the neural network system, therefore, in this embodiment, the neural network can be based on specific needs. Any layer of the neural network replaces the data at the corresponding position in the stored intermediate complete data according to the difference data after the calculation process, and outputs the replaced entire image data as the output data of this layer to the next layer of convolution layer, thus Does not affect the accuracy of processing. This not only improves efficiency and reduces system power consumption, but also ensures the processing accuracy of the system.

4 and 5 respectively show schematic diagrams of the image processing process and method based on the neural network system of the present invention. It can be known with reference to FIGS. 4 and 5 that the neural network system in this embodiment includes a total of four convolutional layers L1, L2, L3, and L4, where the L1 convolutional layer can be used as an input layer at the same time. The overall image processing flow based on the embodiment of the present invention may include:

Step 1: The input layer receives the currently input image frame, determines that the currently input image frame is the basic image frame data (that is, the frame data corresponding to the time t in FIG. 4), then stores the basic image frame and enters it into L1- The L4 convolutional layer performs calculation processing layer by layer in the neural network, and stores the intermediate complete data after calculation processing in each layer;

Step 2: The input layer receives the currently input image frame, determines that the currently input image frame is not the basic image frame (that is, the frame data at time t+1 in FIG. 4), and compares the currently input image frame with the basic image frame , Extract the difference data in the current input image frame and the basic image frame, that is, the data in the box in the figure, determine the data block where the difference data is located, and output the difference data block to the L1 convolution layer for calculation processing;

Step 3: After the L1 convolution layer calculates and processes the difference data block, the calculated data is input into the L2 convolution layer;

Step 4. The L2 convolutional layer compares the image data input by the L1 convolutional layer with the intermediate complete data stored in L2 to obtain the difference data, and determines the data block where the difference data is located, performs calculation processing on the difference data block, and processes the calculation The data is output as L2 output data to the L3 convolution layer;

Step 4, the L3 convolutional layer compares the image data input by L2 with the intermediate complete data stored in L3 to obtain the difference data, and determines the data block where the difference data is located, performs calculation processing on the difference data block, and takes the calculated data as The output data of the third layer is output to L4; when the L2 and L3 convolutional layers are calculated, only relevant data that is different from the image data saved in the previous basic image frame data will be input to the system for processing. The so-called relevant data here , Refers to the data associated with the changed data, that is, the data belonging to the same data block;

Step 5, fuse the difference data output by L3 with the intermediate complete data of L4, and replace the data at the same position in the intermediate complete data of L4 with the difference data output by L3 to obtain the image data of a new complete image. The image data of the complete image is subjected to calculation processing, and the image data obtained by the calculation processing is output.

In addition, in step 3 or 4 above, after L2 or L3 completes the calculation of the difference data block, by replacing the changed data in the image data of the corresponding layer, the convolutional layer can be restored to the image data of the complete image And input to the next layer, and in the next layer can use the image data of the complete image as input data, so as to ensure the accuracy of the algorithm network.

Based on the same inventive concept, an embodiment of the present invention also provides a neural network system, which may include: an input layer and multiple convolution layers;

The input layer is used to identify the currently input image frame, and in response to the currently input image frame not being a basic image frame, extract first image data in the currently input image frame that is different from the basic image frame, Output the first image data to the multi-layer convolution layer;

The N-1 convolutional layer of the neural network system is used to sequentially perform calculation processing on the first image data and output the processed second image data to the Nth layer convolutional layer, where N is greater than or equal to An integer of 2;

The Nth layer convolutional layer of the neural network system is used to replace the data corresponding to the location of the second image data in the first intermediate complete data stored in the Nth layer convolutional layer with the second image data To obtain third image data, and the Nth convolutional layer performs calculation processing on the third image data and outputs; wherein, the first intermediate complete data is that the basic image frame is in the Nth layer volume The calculation result of the layer.

In an optional embodiment of the present invention, the i-th convolution layer of the neural network system is also used to compare the input image data with the second intermediate complete data stored in the i-th convolution layer, Extracting fourth image data that is different from the input image data and the second intermediate complete data, and performing calculation processing on the fourth image data; wherein, i is an integer less than N.

In an optional embodiment of the present invention, the k-th layer convolution of the neural network system is also used to directly perform calculation processing on the first image data.

In an optional embodiment of the present invention, the input layer is also used to determine that the currently input image frame is not a basic image frame, and divide a frame of image into multiple data based on a preset image data block division rule Piece;

In an optional embodiment of the present invention, the i-th convolution layer of the neural network system is further used to divide a frame of image into multiple data blocks based on a preset image data block division rule;

In an optional embodiment of the present invention, the input layer is also used to identify the location of the data block where the first image data is located;

In an optional embodiment of the present invention, the identification information includes: the start address of the data block where the input image data is located in the entire image, and the size of each data block.

In an optional embodiment of the present invention, the N-th convolution layer in the multi-layer convolution layer is also used to replace the first intermediate complete data of the N-th convolution layer with the second image data Image data at the position corresponding to the logo in.

In an optional embodiment of the present invention, the n-th convolution layer of the neural network system is also used to replace the n-th convolution layer with image data obtained by calculation before outputting data to the next convolution layer. The image data at the position corresponding to the input identification information in the intermediate complete data stored in the layer obtains the complete image data of the nth layer, and uses the complete image data as the output data of the next convolution layer.

In an optional embodiment of the present invention, the input layer is also used to respond to that the number of data frames between the currently input image frame and the last input basic image frame is equal to the preset number, then the currently input image frame is Basic image frames, in response to the number of data frames between the currently input image frame and the last input basic image frame being not equal to the preset number, the currently input image frame is a non-basic image frame; or

In response to the image of the currently input image frame containing a predetermined file identification, the currently input image frame is the basic image frame, and in response to the image of the currently input image frame not containing a predetermined file identification, the current input The image frame is a non-base image frame.

In an optional embodiment of the present invention, the input layer is also used to determine that the current input image frame is the basic image frame, and all the entire image data of the currently input image frame is input to the neural network system. In the N-level convolutional layer, the input image data is sequentially processed by the N-level convolutional layer, and the image data obtained after the calculation process is stored in each layer of the N-level convolutional layer.

Based on the same inventive concept, an embodiment of the present invention further provides a storage device in which a computer program is stored, and when the computer program is run in an electronic device, the processor of the electronic device loads and executes any of the above embodiments The image processing method based on neural network system.

Based on the same inventive concept, an embodiment of the present invention also provides an electronic device, including:

A processor for running computer programs; and

The storage device is used to store a computer program, and when the computer program is running in the electronic device, the processor loads and executes the image processing method based on the neural network system described in any of the above embodiments.

6 is a schematic diagram of an electronic device according to an embodiment of the invention. As shown in the figure, the electronic device of this embodiment includes processing cores 111, 112-11N and a network on chip 121. Among them, the convolutional layers in the convolutional neural network are mapped to the processing cores 111-11N, respectively. It should be understood that one convolutional layer may be mapped to multiple processing cores, or multiple convolutional layers may be mapped to one processing core.

The processing cores 111-11N are all connected to the on-chip network 121. The on-chip network 121 is used to exchange data between the N processing cores and external data. At least one processing core of the N processing cores executes the image processing method based on the neural network system described in any of the above embodiments.

In addition, referring to FIG. 6 continuously, the processing core 111 may further include a memory 111a, an operator 111b, and a controller 111c. The memory 111a can be used to store program codes, the operator 111b is used to perform various calculations on the program codes in the electronic device, the controller 111c is coupled to the memory 111a and the operator 111b, and controls the data storage of the memory 111a and the operator 111b to the memory 111a, respectively The data in is calculated according to different calculation logics.

According to any one of the foregoing optional embodiments or a combination of multiple optional embodiments, the embodiments of the present invention can achieve the following beneficial effects:

Embodiments of the present invention provide an image processing method and neural network system based on a neural network system, which are applied to video analysis. The system extracts difference data by comparing changes between the current frame and the basic image frame. The difference data is only Retain the changed data information. During image processing, only the changed information retained in the difference data is input to the neural network system for data analysis and processing; at the same time, in order to maintain the processing accuracy of the system, a complete basic image frame processing process will be saved The intermediate complete data generated in the process is used to perform comparison at each layer after the intermediate complete data is processed, and the affected and changed intermediate data is extracted again as the input data of the next layer. In this way, not only in each layer of the neural network, the repeated information in the previous and subsequent frames can be hidden, and only the changed information is calculated, thereby saving a lot of computing resources, reducing unnecessary information transmission, and speeding up the processing speed. On the other hand, according to specific requirements, it is possible to restore complete image data at any layer of the neural network, and input these complete image data into subsequent layers for processing and analysis, so as to ensure the processing accuracy of the image by the neural network system.

The specification provided here explains a lot of specific details. However, it can be understood that the embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this description.

Similarly, it should be understood that in order to streamline the disclosure and help understand one or more of the various inventive aspects, in the above description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together into a single embodiment, Figure, or its description. However, the disclosed method should not be interpreted as reflecting the following intention: that is, the claimed invention requires more features than those explicitly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Therefore, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, where each claim itself serves as a separate embodiment of the present invention.

Those skilled in the art can understand that the modules in the device in the embodiment can be adaptively changed and set in one or more devices different from the embodiment. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition, they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and/or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstract and drawings) and any methods so disclosed or All processes or units of equipment are combined. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose.

In addition, those skilled in the art can understand that although some of the embodiments described herein include certain features included in other embodiments instead of other features, the combination of features of different embodiments is meant to be within the scope of the present invention And form different embodiments. For example, in the claims, any one of the claimed embodiments can be used in any combination.

The various component embodiments of the present invention may be implemented in hardware, or implemented in software modules running on one or more processors, or implemented in a combination thereof. Those skilled in the art should understand that, in practice, a microprocessor or a digital signal processor (DSP) may be used to implement some or all functions of some or all components in the neural network system according to an embodiment of the present invention. The present invention may also be implemented as a device or device program (eg, computer program and computer program product) for performing part or all of the method described herein. Such a program implementing the present invention may be stored on a computer-readable medium, or may have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

It should be noted that the above-mentioned embodiments illustrate the present invention rather than limit the present invention, and those skilled in the art can design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs between parentheses should not be constructed as limitations on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "one" before an element does not exclude the presence of multiple such elements. The invention can be realized by means of hardware including several different elements and by means of a suitably programmed computer. In the unit claims enumerating several devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third does not indicate any order. These words can be interpreted as names.

By now, those skilled in the art should realize that although a number of exemplary embodiments of the present invention have been shown and described in detail herein, they can still be disclosed according to the present invention without departing from the spirit and scope of the present invention. The content directly determines or derives many other variations or modifications that conform to the principles of the present invention. Therefore, the scope of the present invention should be understood and deemed to cover all these other variations or modifications.

Claims

An image processing method based on a neural network system is applied to video analysis. The method includes:

The neural network system recognizes the currently input image frame and judges whether the currently input image frame is the basic image frame;

In response to that the currently input image frame is not a basic image frame, extract first image data different from the basic image frame in the currently input image frame, and output the first image data to the neural network system;

The N-1 convolutional layer of the neural network system sequentially performs calculation processing on the first image data, and outputs the processed second image data to the Nth layer convolutional layer, where N is greater than or equal to 2 An integer of

The Nth layer convolutional layer of the neural network system uses the second image data to replace the first intermediate complete data stored in the Nth layer convolutional layer corresponding to the location of the second image data Data to obtain third image data, and the Nth convolutional layer performs calculation processing on the third image data and outputs; wherein, the first intermediate complete data is the basic image frame in the Nth layer The calculation result of the convolution layer.
The method according to claim 1, wherein the N-1 convolutional layer of the neural network system sequentially performs calculation processing on the first image data, specifically including:

The i-th convolution layer of the neural network system compares the input image data with the second intermediate complete data stored in the i-th convolution layer, and extracts the input image data and the Two fourth image data having different intermediate complete data, and performing calculation processing on the fourth image data.
The method according to claim 1 or 2, wherein the N-1 convolutional layer of the neural network system sequentially performs calculation processing on the first image data, specifically including:

The k-th convolution layer of the neural network system directly performs calculation processing on the first image data.
The method according to any one of claims 1 to 3, wherein, in response to the currently input image frame is not a basic image frame, extracting the current input image frame that is different from the basic image frame The first image data, outputting the first image data to the neural network system, specifically includes:

In response to that the currently input image frame is not a basic image frame, divide a frame of image into multiple data blocks based on a preset image data block division rule;

Comparing whether each data block of the currently input image frame is the same as the data block corresponding to the basic image frame, and extracting data blocks in the current input image frame that are different from the image data of the corresponding position of the basic image frame , Output the different data blocks as the first image data to the neural network system.
The method according to claim 2, wherein the i-th convolutional layer of the neural network system compares the input image data with the second intermediate complete data stored in the i-th convolutional layer to extract all The fourth image data in which the input image data is different from the second intermediate complete data specifically includes:

The i-th convolutional layer of the neural network system divides a frame of image into multiple data blocks based on preset image data block division rules;

Comparing whether the data block where the input image data is located is the same as the data block corresponding to the second intermediate complete data, and extracting data in the input image data that is different from the image data corresponding to the second intermediate complete data Block, using the different data block as the fourth image data.
The method according to claim 5, wherein the image frame responsive to the current input is not a basic image frame, after extracting first image data different from the basic image frame in the currently input image frame Before outputting the first image data to the neural network system, further including:

Identify the location of the data block where the first image data is located;

The i-th convolutional layer of the neural network system compares the input image data with the second intermediate complete data, including:

The i-th convolutional layer of the neural network system compares the input image data with the image data of the position corresponding to the identifier in the second intermediate complete data.
The method according to claim 6, wherein the identification comprises: a start address of a data block where the input image data is located in the entire image, and a size of each data block.
The method according to claim 6 or 7, wherein the Nth layer convolutional layer of the neural network system replaces the first intermediate completeness stored in the Nth layer convolutional layer with the second image data The data corresponding to the location of the second image data in the data to obtain the third image data includes:

The Nth layer convolutional layer of the neural network system replaces the image data of the position corresponding to the identifier in the first intermediate complete data of the Nth layer convolutional layer with the second image data.
The method according to any one of claims 2, 4-8, wherein the N-1 convolutional layer of the neural network system sequentially performs calculation processing on the first image data, including:

The nth layer convolutional layer of the neural network system replaces the image data at the position corresponding to the identifier in the intermediate complete data stored in the nth layer with the image data obtained by the calculation process to obtain the completeness of the nth layer , The complete image data is used as the output data of the next convolution layer, and n is an integer less than N.
The method according to any one of claims 1-9, wherein the neural network system recognizes the currently input image frame to determine whether the currently input image frame is a basic image frame, including:

In response to that the number of data frames between the currently input image frame and the last input basic image frame is equal to a preset number, the currently input image frame is the basic image frame, and in response to the currently input image frame The number of data frames spaced from the last input basic image frame is not equal to the preset number, then the currently input image frame is a non-base image frame; or

In response to the image of the currently input image frame containing a predetermined file identifier, the currently input image frame is a basic image frame, and the image in response to the currently input image frame does not contain a predetermined file Flag, the currently input image frame is a non-base image frame.
The method according to any one of claims 1-10, wherein the method further comprises:

In response to the currently input image frame being the basic image frame, all the entire image data of the currently input image frame is input to the N-layer convolutional layer of the neural network system, and the N-layer convolutional layer pairs The input image data is sequentially calculated, and the image data obtained after the calculation process is stored in each of the N layers of the convolutional layer.
A neural network system, including: an input layer and multiple convolution layers;

The input layer is used to identify the currently input image frame and determine whether the currently input image frame is a basic image frame; in response to the currently input image frame not being a basic image frame, extract the currently input image First image data in a frame different from the basic image frame, output the first image data to the multi-layer convolutional layer;

The N-1 convolutional layer of the neural network system is used to sequentially perform calculation processing on the first image data, and output the processed second image data to the Nth layer convolutional layer, where N is Integer greater than or equal to 2;

The Nth layer convolution layer of the neural network system is used to replace the first intermediate complete data stored in the Nth layer convolution layer with the second image data corresponding to the location of the second image data To obtain third image data, the Nth convolutional layer performs calculation processing on the third image data and outputs; wherein, the first intermediate complete data is the basic image frame in the Nth The calculation result of the layer convolution layer.
A storage device in which a computer program is stored, and when the computer program runs in an electronic device, the processor of the electronic device loads and executes the image based on the neural network system of any one of claims 1-11 Approach.
An electronic device, including:

A processor for running computer programs; and

A storage device is used to store a computer program, and when the computer program is running in the electronic device, the processor loads and executes the image processing method based on the neural network system of any one of claims 1-11.