CN108830377B - Neural network circuit and self-circulation multi-stage iteration method thereof - Google Patents
Neural network circuit and self-circulation multi-stage iteration method thereof Download PDFInfo
- Publication number
- CN108830377B CN108830377B CN201810642686.2A CN201810642686A CN108830377B CN 108830377 B CN108830377 B CN 108830377B CN 201810642686 A CN201810642686 A CN 201810642686A CN 108830377 B CN108830377 B CN 108830377B
- Authority
- CN
- China
- Prior art keywords
- neural network
- image
- unit
- network circuit
- identification
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
Abstract
The invention provides a neural network circuit and a self-circulation multistage iteration method thereof, wherein the neural network circuit comprises an image reduction unit, a sensitive area image access unit, a multi-path selection unit, a configurable neural network circuit unit, a neural network structure configuration unit, a sensitive area coordinate information storage unit and a coordinate restoration unit; the method comprises the steps that firstly, an original image is reduced by an image reducing unit, a sensitive object area in the image is divided by a configurable neural network circuit unit, the original image is reduced by a coordinate reducing unit, and then local reading is carried out by a sensitive area image access unit; and then the data are sent to a configurable neural network circuit unit for classification and identification, and the identification process adopts a self-circulation multi-stage iteration mode to perform time-line operation for multiple rounds, so that a continuously refined identification result is obtained.
Description
Technical Field
The invention relates to a neural network circuit structure, which has a self-circulation multi-stage iteration function.
Background
The image sensitive area identification technology is the prior art, a common area division network algorithm is FCN (https:// blog.csdn.net/tag/area/details/51401448), and the FCN firstly converts a full connection layer in the traditional CNN into a convolution layer according to a segmentation network structure. After multiple convolutions (and posing), the obtained feature map is smaller and lower, the resolution ratio is lower and lower, the value of each point of the feature map represents the probability of one classification, when the probability value of the point is larger than a certain threshold value (the threshold value is configurable), the point is considered to have a certain sensitive object, and then the coordinates of the sensitive area where the feature point is located are restored through sampling, so that the sensitive area of the image is identified.
The Convolutional Neural Network (CNN) is a configurable neural network circuit, and the CNN is strong in that its multilayer structure can automatically learn features and can learn features of multiple layers, so that since 2012, the Convolutional Neural Network (CNN) has been used in image classification and image detection. However, with the rapid popularization and application of deep learning neural networks, more and more scenes use the neural network recognition capability. However, due to the complexity of the algorithm (including the image segmentation algorithm) of the deep learning neural network, the hardware circuit and the power consumption are greatly consumed, and particularly, the mobile device terminal has difficulty in supporting the huge deep learning network. The current terminal device can only be used for recognition processing of calculating smaller images, for example, (320x240, 256x256, etc.), when large image recognition is encountered, only the cloud server can be used for calculation, or the calculation time is too long, so that certain applications are limited.
Therefore, the invention provides a neural network circuit which has a self-circulation multi-stage iteration function, can process a high-resolution image in real time under the condition of limited calculation power and effectively reduces the power consumption of a system.
Disclosure of Invention
The technical problem to be solved by the invention is to provide a neural network circuit and a self-circulation multi-stage iterative electrical method thereof, which can process a large-resolution image in real time under the condition of limited calculation power and effectively reduce the power consumption of a system.
The circuit of the invention is realized as follows: a neural network circuit comprises an image reduction unit, a sensitive area image access unit, a multi-path selection unit, a configurable neural network circuit unit, a neural network structure configuration unit, a sensitive area coordinate information storage unit and a coordinate restoration unit;
the image reducing unit is connected with the sensitive area image acquisition unit through the coordinate restoring unit; the image reducing unit and the sensitive area image access unit are connected with the configurable neural network circuit unit through the multi-path selection unit; the configurable neural network circuit unit is also connected with the coordinate restoring unit through the sensitive area coordinate information storage unit, and the neural network structure configuration unit is connected with the configurable neural network circuit unit;
the image reducing unit is responsible for reducing the original image to the size of the image which can be processed by the configurable neural network circuit unit, and sending the reduced proportion of the image to the coordinate restoring unit;
the sensitive region image data acquisition unit is responsible for locally reading the image data of the sensitive object region in the original image according to the coordinates sent by the coordinate restoration unit and then sending the image data to the multi-path selection unit;
the multi-path selection unit gates the image reduction unit to the configurable neural network circuit unit in a neural network image segmentation stage, and gates the sensitive area image acquisition unit to the configurable neural network circuit unit in a neural network image identification stage;
the configurable neural network circuit unit reconstructs the configuration information according to the configuration unit of the neural network structure to generate various identified neural network structures; in the neural network image segmentation stage, the neural network image segmentation unit is responsible for segmenting the sensitive object region in the image and sending the coordinates of the segmented sensitive object region to the coordinate restoration unit; in the neural network image identification stage, the neural network image identification stage is responsible for classifying and identifying the sensitive objects in the regional image to obtain an identification result;
and the coordinate reduction unit reduces the coordinates of the sensitive object area in the reduced image into the coordinates of the original image according to the image reduction proportion and then sends the coordinates to the sensitive area image acquisition unit.
Furthermore, the circuit of the invention further comprises a coordinate reduction judging unit which is respectively connected with the image reducing unit, the sensitive area image access unit and the configurable neural network circuit unit;
the coordinate restoration judging unit is responsible for judging whether the image of the sensitive object area needs to be restored or not according to the coordinate and size information of the sensitive object area and the processing capacity of the neural network circuit.
Furthermore, the circuit of the invention also comprises an original image storage unit and an area identification result storage unit;
the original image storage unit is respectively connected with the image reducing unit and the sensitive area image access unit and is used for storing original image data, sending the data to the image reducing unit in a neural network image segmentation stage and sending the data to the sensitive area image access unit in a neural network image identification stage;
the area identification result storage unit is connected with the configurable neural network circuit unit and used for storing the identification result.
Furthermore, the sensitive area coordinate information storage unit stores the coordinates of the sensitive object area in a manner that pixel coordinate information of the upper left corner and the lower right corner of the image block of each sensitive object area in the image is stored.
Further, the neural network structure configuration unit is responsible for reading different neural network structure information to perform neural network structure configuration on the configurable neural network circuit unit; in a neural network image segmentation stage, configuring the configurable neural network circuit unit into a neural network structure for sensitive object region image segmentation, and in a neural network image identification stage, configuring the configurable neural network circuit unit into a neural network structure for sensitive object region image content identification;
further, the device also comprises a neural network structure information storage unit connected with the neural network structure configuration unit and used for storing different neural network structure information.
Furthermore, the configurable neural network circuit unit further comprises a characteristic data reading unit, an internal characteristic data caching unit, a convolution kernel data reading unit, an internal convolution kernel caching unit, a multiplication and addition array convolution operation unit, an activation function operation unit, a pooling operation unit and a full-connection operation unit;
the characteristic data reading unit, the internal characteristic data caching unit, the multiply-add array convolution operation unit, the activation function operation unit, the pooling operation unit and the full-connection operation unit are connected in sequence; the convolution kernel data reading unit and the internal convolution kernel cache unit are sequentially connected with the multiplication and addition array convolution operation unit; the multi-path selection unit is connected with the characteristic data reading unit, and the neural network structure configuration unit is connected with the convolution kernel data reading unit.
The method of the invention is realized as follows: a self-loop multistage iteration method of a neural network circuit comprises a neural network image segmentation stage and a neural network image identification stage:
the neural network image segmentation stage is as follows:
step 11, reducing the original image to the size of an image which can be processed by a configurable neural network circuit unit in the neural network circuit, and obtaining the reduced image and the image reduction proportion;
step 12, the configurable neural network circuit unit is reconfigured after configuration, all sensitive object areas in the reduced image are identified after the identification network structure is generated, all sensitive object areas are segmented, and the coordinates of the segmented sensitive object areas are stored;
step 13, sequentially restoring the coordinates of the sensitive object areas in the reduced image into the coordinates in the original image according to the image reduction proportion and the coordinates of each sensitive object area;
step 14, reading image data of a sensitive object area in an original image, obtaining a local image, sending the local image to the configurable neural network circuit unit, and entering a neural network image identification stage to identify a sensitive object area;
the neural network image identification stage is as follows:
step 21, reconstructing the configurable neural network circuit unit after configuration, performing general type identification on a sensitive object of a local image after generating a new identification neural network structure, storing or outputting an identification result to complete the identification of the current round, and then finishing the identification or performing the next round so as to enter the next round of identification;
step 22, restoring the coordinates of the same sensitive object area in the reduced image into the coordinates in the original image according to the image reduction proportion and the coordinates of the sensitive object area;
step 23, reading image data of the sensitive object area in the original image to obtain a local image, and then sending the local image to the configurable neural network circuit unit; go back to step 21
In each next round of identification, the configurable neural network circuit unit in step 21 is reconstructed, and a more detailed identification neural network structure is generated, so that a self-loop multistage iterative identification function is realized.
Further, the step 23 may be replaced by:
step 23, judging whether the restored image size of the sensitive object area exceeds the processing capacity image size of the configurable neural network circuit; if not, reading the image data of the sensitive object area in the original image, and directly sending the image data to the configurable neural network circuit unit after obtaining a local image; returning to step 21; and if so, reading the image data of the sensitive object area in the original image to obtain a local image, reducing the local image to the size of an image which can be processed by a configurable neural network circuit unit in the neural network circuit, and then sending the image to the configurable neural network circuit unit.
Further, the identifying the hierarchical relationship of the neural network structure is:
performing first round identification, and reconstructing to generate a sensitive area identification neural network;
performing second round of identification, and reconstructing to generate a general object identification neural network;
the third round and its subsequent identification, reconstruction produces object classification and its branch recognition neural network.
The invention has the following advantages: the circuit of the invention firstly reduces the large image pooling, configures the large image pooling into an interest area identification network (segmentation neural network) through a configurable neural network circuit unit, segments the interest areas (namely sensitive object areas), configures the configurable neural network circuit unit into an image classification network, identifies the sensitive objects in the interest areas one by one, and can deeply identify each interest area for multiple rounds, thereby obtaining an identification result which is continuously refined. The method can greatly reduce the operation amount of the neural network, and simultaneously, the reconfigurable circuit identifies the configuration of the neural network structure to the configurable neural network circuit unit, thereby saving the circuit loss of the two neural networks.
Drawings
The invention will be further described with reference to the following examples with reference to the accompanying drawings.
FIG. 1 is a flow chart of a self-loop multi-stage iteration method of a neural network circuit according to the present invention.
Fig. 2 is a schematic structural diagram of a neural network circuit according to an embodiment of the present invention.
FIG. 3 is a schematic structural diagram of a configurable neural network circuit unit according to the present invention.
FIG. 4 is a schematic structural diagram of a neural network circuit according to another embodiment of the present invention.
FIG. 5 is a schematic diagram of the hierarchical relationship of the neural network structures of the neural network circuit according to the present invention.
Detailed Description
Referring to fig. 1, the self-loop multistage iteration method of the neural network circuit of the present invention includes a neural network image segmentation stage and a neural network image recognition stage:
the neural network image segmentation stage is as follows:
step 11, reducing the original image to the size of an image which can be processed by a configurable neural network circuit unit in the neural network circuit, and obtaining the reduced image and the image reduction proportion; the original image can be directly acquired by a camera, for example, the original image with the resolution of 1024 × 720 pixels is acquired by the camera, and if the size of the image which can be processed by the configurable neural network circuit unit is 320 × 240 pixels, the original image is reduced to 320 × 240 pixels, and a reduced image with 320 × 240 pixels and an image reduction ratio of 0.3 are obtained.
Step 12, the configurable neural network circuit unit is configured and then reconstructed to generate a recognition network structure, then all sensitive object regions in the reduced image are recognized, such as vehicles, animals or human objects in the image, all the sensitive object regions are segmented to respectively segment the vehicles, the animals or the human objects, and the coordinates of the segmented sensitive object regions are stored, namely the coordinates of the vehicle regions, the animal regions or the human regions are stored; the configurable neural network circuit unit can be a configurable CNN circuit, and can be reconstructed according to configuration information of a CNN structure to generate CNN networks of various structures, so that areas with sensitive objects in an image can be divided according to the CNN networks.
Step 13, restoring the coordinates of the sensitive object areas in the reduced image into the coordinates of the original image in sequence according to the image reduction proportion and the coordinates of each sensitive object area, so that the original image is still identified in the subsequent neural network image identification stage instead of the reduced image, and the identification accuracy is higher;
step 14, reading image data of a sensitive object area in an original image, obtaining a local image, sending the local image to the configurable neural network circuit unit, and entering a neural network image identification stage to identify a sensitive object area; because the original image is large, the whole image is not suitable to be sent to the configurable neural network circuit unit for identification, but the local image of the sensitive object area is identified so as to adapt to the processing capability of the configurable neural network circuit unit.
The neural network image identification stage is as follows:
step 21, at this time, the configurable neural network circuit unit needs to be reconfigured and then reconstructed again, so that a new recognition neural network structure is generated, the general type recognition is carried out on the sensitive object of the local image, the recognition result is stored or output to complete the recognition of the current round, if the recognition of the current round meets the requirement, the recognition is finished, and if the further recognition is needed, the next step is continued; it should be noted that: in the neural network image recognition stage, the content recognized by the configurable neural network circuit unit is different from that in the neural network image segmentation stage, the configurable neural network circuit unit only needs to judge the sensitive area, but in the neural network image recognition stage, the image content in the sensitive area needs to be judged, such as specific people, vehicles or animals, and the like, and the people, the vehicles or the animals can be further recognized according to the needs.
Step 22, restoring the coordinates of the same sensitive object area in the reduced image into the coordinates in the original image according to the image reduction proportion and the coordinates of the sensitive object area;
step 23, reading image data of the sensitive object area in the original image to obtain a local image, and then sending the local image to the configurable neural network circuit unit; go back to step 21
In each next round of identification, the configurable neural network circuit unit in step 21 is reconstructed, and a more detailed identification neural network structure is generated, so that a self-loop multistage iterative identification function is realized. The hierarchical relationship of the identified neural network structure is: performing first round identification, and reconstructing to generate a sensitive area identification neural network; performing second round of identification, and reconstructing to generate a general object identification neural network; the third round and its subsequent identification, reconstruction produces object classification and its branch recognition neural network.
As mentioned above, since the neural network image recognition stage is to recognize the original image to ensure a higher recognition accuracy, although the partial image of the original image is referred to in a larger range, the processing capability of the configurable neural network circuit unit may be exceeded, and therefore, the step 23 may be replaced by:
step 23, judging whether the restored image size of the sensitive object area exceeds the processing capacity image size of the configurable neural network circuit; if not, reading the image data of the sensitive object area in the original image, and directly sending the image data to the configurable neural network circuit unit after obtaining a local image; returning to step 21; and if so, reading the image data of the sensitive object area in the original image to obtain a local image, reducing the local image to the size of an image which can be processed by a configurable neural network circuit unit in the neural network circuit, and then sending the image to the configurable neural network circuit unit.
According to the idea of the above method, the present invention provides a neural network circuit, as shown in fig. 2, the neural network circuit of the present invention includes an image reduction unit 1, a sensitive area image capture unit 2, a multiplexing unit 3, a configurable neural network circuit unit 4, a neural network structure configuration unit 5, a sensitive area coordinate information storage unit 61, and a coordinate restoration unit 7;
the image reducing unit 1 is connected with the sensitive area image taking unit 2 through the coordinate restoring unit 7; the image reducing unit 1 and the sensitive area image access unit 2 are connected with the configurable neural network circuit unit 4 through the multi-path selection unit 3; the configurable neural network circuit unit 4 is further connected with the coordinate restoring unit 7 through the sensitive area coordinate information storage unit 61, and the neural network structure configuration unit 5 is connected with the configurable neural network circuit unit 4.
As shown in fig. 3, the configurable neural network circuit unit 4 further includes a feature data reading unit 41, an internal feature data buffer unit 42, a convolution kernel data reading unit 43, an internal convolution kernel buffer unit 44, a multiplication and addition array convolution operation unit 45, an activation function operation unit 46, a pooling operation unit 47, and a full-connection operation unit 48; the feature data reading unit 41, the internal feature data caching unit 42, the multiplication and addition array convolution operation unit 45, the activation function operation unit 46, the pooling operation unit 47, and the full-connection operation unit 48 are connected in sequence; the convolution kernel data reading unit 43 and the internal convolution kernel buffer unit 44 are sequentially connected to the multiplication and addition array convolution operation unit 45; the multi-path selection unit 3 is connected with the characteristic data reading unit 41, and the neural network structure configuration unit 5 is connected with the convolution kernel data reading unit 43.
Wherein the content of the first and second substances,
the image reducing unit 1 is responsible for reducing an original image (which can be an image shot by a camera) to the size of an image which can be processed by the configurable neural network circuit unit 4, and sending the reduced proportion of the image to the coordinate restoring unit 7;
the sensitive region image data taking unit 2 is responsible for locally reading the image data of the sensitive object region in the original image according to the coordinates sent by the coordinate restoring unit 7 and then sending the image data to the multi-path selecting unit 3;
the multi-path selection unit 3 gates the image reduction unit 1 to the configurable neural network circuit unit 4 in the neural network image segmentation stage so as to send the reduced image to the configurable neural network circuit unit 4, and gates the sensitive area image taking unit 2 to the configurable neural network circuit unit 4 in the neural network image identification stage so as to send the segmented and restored image of the sensitive area to the configurable neural network circuit unit 4;
the configurable neural network circuit unit 4 reconstructs the configuration information according to the configuration information of the neural network structure configuration unit 5 to generate various identified neural network structures (such as CNN network structures); in the neural network image segmentation stage, the neural network image segmentation stage is responsible for segmenting sensitive object regions in the image, and sending coordinates of the segmented sensitive object regions to the sensitive region coordinate information storage unit 61 for storage, wherein the number of the sensitive object regions can be multiple, and the storage mode is that the coordinate information of each sensitive region image is respectively stored; in the neural network image identification stage, the neural network image identification stage is responsible for classifying and identifying the sensitive objects in the regional image to obtain an identification result;
the neural network structure configuration unit 5 is responsible for reading different neural network structure information to perform neural network structure configuration on the configurable neural network circuit unit 4; in the stage of neural network image segmentation, the configurable neural network circuit unit 4 is configured into a neural network structure for sensitive object region image segmentation, and at this time, only the region of a possible sensitive object in an image needs to be segmented without judging what sensitive object is specifically, in the stage of neural network image identification, the configurable neural network circuit unit 4 is configured into the neural network structure for sensitive object region image content identification, including a general object identification neural network structure, so that the configurable neural network circuit unit 4 can be used for identifying a person, a vehicle or an animal, or the neural network structure for further identifying image content details, and the configurable neural network circuit unit 4 can be further identified on the basis of general object identification;
the sensitive region coordinate information storage unit 61 is configured to store coordinates of the sensitive object region; the storage mode is that the pixel coordinate information of the upper left corner and the lower right corner of the image block of each sensitive object area in the image is stored.
The coordinate restoring unit 7 restores the coordinates of the sensitive object region in the reduced image into the coordinates of the original image according to the image reduction ratio, and sends the coordinates to the sensitive region image taking unit 2.
As shown in fig. 2, fig. 4 and fig. 5, in order to facilitate the configuration of the neural network structure configuration unit 5, the neural network structure information storage unit 8 connected to the neural network structure configuration unit 5 is further included, and is configured to store different neural network structure information, and the neural network structure information may have a category and a hierarchy, and may be stored separately, so that refined identification may be implemented. For example, the neural network structure information storage unit 8 may be divided into a sensitive area recognition neural network structure storage unit 81, a general object neural network structure storage unit 82, a character recognition neural network structure storage unit 83, a dog recognition neural network structure storage unit 84, a human recognition neural network structure storage unit 85, and the like, wherein the dog recognition neural network structure storage unit 84 may further be divided into a dog emotion recognition neural network structure storage unit 841, and the human recognition neural network structure storage unit 85 may further be divided into a human gender recognition neural network structure storage unit 851 and a human age recognition neural network structure storage unit 852.
In addition, in the preferred embodiment, the circuit of the present invention further includes an original image storage unit 62 and an area identification result storage unit 63;
the original image storage unit 62 is respectively connected to the image reduction unit 1 and the sensitive area image access unit 2, and is configured to store original image data, send the data to the image reduction unit 1 in a neural network image segmentation stage, and send the data to the sensitive area image access unit 2 in a neural network image identification stage;
the area identification result storage unit 63 is connected to the configurable neural network circuit unit 4, and is configured to store the identification result of the configurable neural network circuit unit 4.
As shown in fig. 4, in a preferred embodiment, the circuit of the present invention further includes the coordinate reduction determination unit 9, where the coordinate reduction determination unit 9 is respectively connected to the image reduction unit 1, the sensitive area image capture unit 2, and the configurable neural network circuit unit 4 (where the coordinate reduction determination unit 9 is connected to the configurable neural network circuit unit 4 through the sensitive area coordinate information storage unit 61); the image processing device is used for judging whether the image of the sensitive object area needs to be restored or not according to the coordinate and size information of the sensitive object area and the processing capacity of the neural network circuit, and the situation that the restored image of the sensitive area exceeds the processing capacity of the neural network circuit is avoided.
As shown in fig. 5, the neural network image recognition stage is subdivided into multiple loop recognition processes, and in the multiple loop recognition processes, the network structure information read by the neural network structure configuration unit 5 is successively refined.
The following examples are given to better illustrate the method and circuit of the present invention:
after the circuit starts to work, firstly, a neural network image segmentation stage is carried out:
1. the camera collects an original image with a resolution of 1024 × 720, and sends image data to the original image storage unit 62 for storage.
2. The original image storage unit 62 stores the original image data, and then sends the original image data to the image reduction unit 1, the image reduction unit 1 reduces the original image to a reduced image of 320 × 240, so that the resolution of the configurable neural network circuit unit 4 capable of real-time operation is achieved, and the reduction ratio is sent to the coordinate restoration unit 7.
3. At this time, the configurable neural network circuit 4 is configured by the neural network structure configuration unit 5 as a neural network structure for sensitive object region image segmentation, and is responsible for segmenting regions in the reduced image where there may be sensitive objects, and sending coordinates of the sensitive object regions to the sensitive region coordinate information storage unit 61, where the storage mode is pixel coordinate information of the upper left corner and the lower right corner of the image block of each sensitive object region in the reduced image, and there may be a plurality of sensitive object regions.
4. Then, the coordinate restoring unit 7 restores the coordinates of the sensitive object region in the reduced image to the coordinates in the original image, based on the image reduction ratio and the coordinates of each sensitive object region.
5. The sensitive area image data taking unit 2 reads the image data of the sensitive object area in the original image according to the coordinates of the sensitive object area in the original image sent by the coordinate restoring unit 7, sends the read image data to the multi-path selecting unit 3, and sends the read image data to the configurable neural network circuit unit 4 through the multi-path selecting unit 3, so that the first round of sensitive image segmentation work is completed.
Then, a neural network image recognition stage is carried out:
1. the multiplexing unit 3 gates the data stream of the sensitive area image fetching unit 2 at this time, so that the image data read by the sensitive area image fetching unit 2 can be really sent to the configurable neural network circuit unit 4 through the multiplexing unit 3.
2. Since the configurable neural network circuit unit 4 is in the neural network image recognition stage at this time, the neural network structure configured for the specific sensitive object region recognition is responsible for classifying and recognizing the sensitive object, and sending the recognition result to the region recognition result storage unit 63. The task type of the neural network image recognition is changed according to the change of the configured neural network structure, so that the neural network image recognition can be subdivided into multiple deep recognition, and various different special object recognition networks can be read for fine recognition.
For example, in an image with a vehicle, a sensitive object region after being segmented and restored is a local image in a red frame in the image, in a first round of sensitive image content identification stage, the configurable neural network circuit unit 4 is configured as a general object identification neural network structure, the identification result is "vehicle", then, based on the identification result of "vehicle", a second round of identification is performed, at this time, the configurable neural network circuit unit 4 is configured as a brand vehicle identification neural network structure, the identification result is "popular polo", then, a third round of identification is performed, the license plate identification neural network structure is called, and license plate content is identified. If necessary, the identification can be carried out infinitely deeply, such as the continuous identification of the car age, the state of the passengers, the number of the passengers, the sex of the passengers and the like. And continuously and circularly iterating to perform multiple rounds of deep content identification and analysis to obtain a continuously refined identification and analysis result.
In the subsequent deep recognition and analysis process, the same image is analyzed, namely, the image recognized in the previous round is taken out from the sensitive area coordinate information storage unit 61, and the coordinate in the original image is restored by the coordinate restoration unit 7; then the sensitive area image taking unit 2 reads the image data of the sensitive object area in the original image according to the coordinates of the sensitive object area in the original image sent by the coordinate restoring unit 7, sends the read image data to the multi-path selecting unit 3, and sends the image data to the configurable neural network circuit unit 4 for the next round of identification after being gated by the multi-path selecting unit 3.
In addition, since the coordinate restoring unit 7 restores the divided image to the coordinates in the original image so that the recognition is performed in a part of the original image, there is a possibility that the part of the original image may exceed the range of the resolution in which the configurable neural network circuit unit 4 can operate. Therefore, the coordinate restoration judging unit 9 can be arranged to judge whether the restored image size of the sensitive object area exceeds the processing capacity image size of the configurable neural network circuit; if not, reading the image data of the sensitive object area in the original image according to the mode, obtaining a local image and then directly sending the local image to the configurable neural network circuit unit 4; and if so, reading the image data of the sensitive object area in the original image to obtain a local image, reducing the local image to the size of an image which can be processed by a configurable neural network circuit unit in the neural network circuit, and then sending the image to the configurable neural network circuit unit 4.
Although specific embodiments of the invention have been described above, it will be understood by those skilled in the art that the specific embodiments described are illustrative only and are not limiting upon the scope of the invention, and that equivalent modifications and variations can be made by those skilled in the art without departing from the spirit of the invention, which is to be limited only by the appended claims.
Claims (10)
1. A neural network circuit, characterized by: the system comprises an image reducing unit, a sensitive area image data taking unit, a multi-path selecting unit, a configurable neural network circuit unit, a neural network structure configuration unit, a sensitive area coordinate information storage unit and a coordinate restoring unit;
the image reducing unit is connected with the sensitive area image acquisition unit through the coordinate restoring unit; the image reducing unit and the sensitive area image access unit are connected with the configurable neural network circuit unit through the multi-path selection unit; the configurable neural network circuit unit is also connected with the coordinate restoring unit through the sensitive area coordinate information storage unit, and the neural network structure configuration unit is connected with the configurable neural network circuit unit;
the image reducing unit is responsible for reducing the original image to the size of the image which can be processed by the configurable neural network circuit unit, and sending the reduced proportion of the image to the coordinate restoring unit;
the sensitive region image data acquisition unit is responsible for locally reading the image data of the sensitive object region in the original image according to the coordinates sent by the coordinate restoration unit and then sending the image data to the multi-path selection unit;
the multi-path selection unit gates the image reduction unit to the configurable neural network circuit unit in a neural network image segmentation stage, and gates the sensitive area image acquisition unit to the configurable neural network circuit unit in a neural network image identification stage;
the configurable neural network circuit unit reconstructs the configuration information according to the configuration unit of the neural network structure to generate various identified neural network structures; in the neural network image segmentation stage, the neural network image segmentation unit is responsible for segmenting the sensitive object region in the image and sending the coordinates of the segmented sensitive object region to the coordinate restoration unit; in the neural network image identification stage, the neural network image identification stage is responsible for classifying and identifying the sensitive objects in the regional image to obtain an identification result;
and the coordinate reduction unit reduces the coordinates of the sensitive object area in the reduced image into the coordinates of the original image according to the image reduction proportion and then sends the coordinates to the sensitive area image acquisition unit.
2. The neural network circuit of claim 1, wherein: the coordinate reduction judging unit is respectively connected with the image reducing unit, the sensitive area image access unit and the configurable neural network circuit unit;
the coordinate restoration judging unit is responsible for judging whether the image of the sensitive object area needs to be restored or not according to the coordinate and size information of the sensitive object area and the processing capacity of the neural network circuit.
3. The neural network circuit of claim 2, wherein: the system also comprises an original image storage unit and an area identification result storage unit;
the original image storage unit is respectively connected with the image reducing unit and the sensitive area image access unit and is used for storing original image data, sending the data to the image reducing unit in a neural network image segmentation stage and sending the data to the sensitive area image access unit in a neural network image identification stage;
the area identification result storage unit is connected with the configurable neural network circuit unit and used for storing the identification result.
4. The neural network circuit of claim 3, wherein: the sensitive area coordinate information storage unit stores the coordinates of the sensitive object areas in a manner that the pixel coordinate information of the upper left corner and the lower right corner of the image block of each sensitive object area in the image is stored.
5. The neural network circuit of claim 1, wherein: the neural network structure configuration unit is responsible for reading different neural network structure information to carry out neural network structure configuration on the configurable neural network circuit unit; in the stage of neural network image segmentation, the configurable neural network circuit unit is configured into a neural network structure for sensitive object area image segmentation, and in the stage of neural network image identification, the configurable neural network circuit unit is configured into a neural network structure for sensitive object area image content identification.
6. The neural network circuit of claim 5, wherein: the device also comprises a neural network structure information storage unit connected with the neural network structure configuration unit and used for storing different neural network structure information.
7. The neural network circuit of claim 1, wherein: the configurable neural network circuit unit further comprises a characteristic data reading unit, an internal characteristic data caching unit, a convolution kernel data reading unit, an internal convolution kernel caching unit, a multiplication and addition array convolution operation unit, an activation function operation unit, a pooling operation unit and a full-connection operation unit;
the characteristic data reading unit, the internal characteristic data caching unit, the multiply-add array convolution operation unit, the activation function operation unit, the pooling operation unit and the full-connection operation unit are connected in sequence; the convolution kernel data reading unit and the internal convolution kernel cache unit are sequentially connected with the multiplication and addition array convolution operation unit;
the multi-path selection unit is connected with the characteristic data reading unit, and the neural network structure configuration unit is connected with the convolution kernel data reading unit.
8. A self-loop multistage iteration method of a neural network circuit is characterized in that: the method comprises a neural network image segmentation stage and a neural network image identification stage:
the neural network image segmentation stage is as follows:
step 11, reducing the original image to the size of an image which can be processed by a configurable neural network circuit unit in the neural network circuit, and obtaining the reduced image and the image reduction proportion;
step 12, the configurable neural network circuit unit is reconfigured after configuration, all sensitive object areas in the reduced image are identified after the identification network structure is generated, all sensitive object areas are segmented, and the coordinates of the segmented sensitive object areas are stored;
step 13, sequentially restoring the coordinates of the sensitive object areas in the reduced image into the coordinates in the original image according to the image reduction proportion and the coordinates of each sensitive object area;
step 14, reading image data of a sensitive object area in an original image, obtaining a local image, sending the local image to the configurable neural network circuit unit, and entering a neural network image identification stage to identify a sensitive object area;
the neural network image identification stage is as follows:
step 21, reconstructing the configurable neural network circuit unit after configuration, performing general type identification on a sensitive object of a local image after generating a new identification neural network structure, storing or outputting an identification result to complete the identification of the current round, and then finishing the identification or performing the next round so as to enter the next round of identification;
step 22, restoring the coordinates of the same sensitive object area in the reduced image into the coordinates in the original image according to the image reduction proportion and the coordinates of the sensitive object area;
step 23, reading image data of the sensitive object area in the original image to obtain a local image, and then sending the local image to the configurable neural network circuit unit; returning to step 21;
in each next round of identification, the configurable neural network circuit unit in step 21 is reconstructed, and a more detailed identification neural network structure is generated, so that a self-loop multistage iterative identification function is realized.
9. The method of claim 8, wherein the method comprises: step 23 is replaced by:
step 23, judging whether the restored image size of the sensitive object area exceeds the processing capacity image size of the configurable neural network circuit; if not, reading the image data of the sensitive object area in the original image, and directly sending the image data to the configurable neural network circuit unit after obtaining a local image; returning to step 21; and if so, reading the image data of the sensitive object area in the original image to obtain a local image, reducing the local image to the size of an image which can be processed by a configurable neural network circuit unit in the neural network circuit, and then sending the image to the configurable neural network circuit unit.
10. A method of self-looping multi-stage iteration of a neural network circuit as claimed in claim 8 or 9, wherein: the hierarchical relationship of the identified neural network structure is:
performing first round identification, and reconstructing to generate a sensitive area identification neural network;
performing second round of identification, and reconstructing to generate a general object identification neural network;
the third round and its subsequent identification, reconstruction produces object classification and its branch recognition neural network.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810642686.2A CN108830377B (en) | 2018-06-21 | 2018-06-21 | Neural network circuit and self-circulation multi-stage iteration method thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810642686.2A CN108830377B (en) | 2018-06-21 | 2018-06-21 | Neural network circuit and self-circulation multi-stage iteration method thereof |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108830377A CN108830377A (en) | 2018-11-16 |
| CN108830377B true CN108830377B (en) | 2020-12-15 |
Family
ID=64142926
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810642686.2A Active CN108830377B (en) | 2018-06-21 | 2018-06-21 | Neural network circuit and self-circulation multi-stage iteration method thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108830377B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109800868B (en) * | 2018-12-25 | 2022-12-16 | 瑞芯微电子股份有限公司 | Data coding chip and method based on deep learning |
| CN109840590A (en) * | 2019-01-31 | 2019-06-04 | 福州瑞芯微电子股份有限公司 | A kind of scene classification circuit framework neural network based and method |
| KR20210004702A (en) * | 2019-07-05 | 2021-01-13 | 삼성전자주식회사 | Artificial intelligence processor and performing neural network operation thereof |
| CN111429458B (en) * | 2020-03-20 | 2021-06-08 | 北京创世云科技股份有限公司 | Image restoration method and device and electronic equipment |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105701541A (en) * | 2016-01-13 | 2016-06-22 | 哈尔滨工业大学深圳研究生院 | Circuit structure based on memristor pulse nerve network |
| CN107644418A (en) * | 2017-09-26 | 2018-01-30 | 山东大学 | Optic disk detection method and system based on convolutional neural networks |
| CN108171723A (en) * | 2017-12-22 | 2018-06-15 | 湖南源信光电科技股份有限公司 | Based on more focal length lens of Vibe and BP neural network algorithm linkage imaging camera machine system |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9811775B2 (en) * | 2012-12-24 | 2017-11-07 | Google Inc. | Parallelizing neural networks during training |
| US10902343B2 (en) * | 2016-09-30 | 2021-01-26 | Disney Enterprises, Inc. | Deep-learning motion priors for full-body performance capture in real-time |
-
2018
- 2018-06-21 CN CN201810642686.2A patent/CN108830377B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105701541A (en) * | 2016-01-13 | 2016-06-22 | 哈尔滨工业大学深圳研究生院 | Circuit structure based on memristor pulse nerve network |
| CN107644418A (en) * | 2017-09-26 | 2018-01-30 | 山东大学 | Optic disk detection method and system based on convolutional neural networks |
| CN108171723A (en) * | 2017-12-22 | 2018-06-15 | 湖南源信光电科技股份有限公司 | Based on more focal length lens of Vibe and BP neural network algorithm linkage imaging camera machine system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108830377A (en) | 2018-11-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108830377B (en) | Neural network circuit and self-circulation multi-stage iteration method thereof | |
| CN109697434B (en) | Behavior recognition method and device and storage medium | |
| CN109472191B (en) | Pedestrian re-identification and tracking method based on space-time context | |
| CN111368636B (en) | Object classification method, device, computer equipment and storage medium | |
| CN109714526B (en) | Intelligent camera and control system | |
| CN112329702B (en) | Method and device for rapid face density prediction and face detection, electronic equipment and storage medium | |
| CN112200041B (en) | Video motion recognition method and device, storage medium and electronic equipment | |
| CN111723773B (en) | Method and device for detecting carryover, electronic equipment and readable storage medium | |
| CN110837786B (en) | Density map generation method and device based on spatial channel, electronic terminal and medium | |
| CN112861575A (en) | Pedestrian structuring method, device, equipment and storage medium | |
| CN111401196A (en) | Method, computer device and computer readable storage medium for self-adaptive face clustering in limited space | |
| CN111401215A (en) | Method and system for detecting multi-class targets | |
| CN111414910B (en) | Small target enhancement detection method and device based on double convolution neural network | |
| CN111079519A (en) | Multi-posture human body detection method, computer storage medium and electronic device | |
| CN114037640A (en) | Image generation method and device | |
| CN113496208A (en) | Video scene classification method and device, storage medium and terminal | |
| CN114238904A (en) | Identity recognition method, and training method and device of two-channel hyper-resolution model | |
| Qiu et al. | A fully convolutional encoder–decoder spatial–temporal network for real-time background subtraction | |
| Venkatesvara Rao et al. | Real-time video object detection and classification using hybrid texture feature extraction | |
| CN111178495A (en) | Lightweight convolutional neural network for detecting very small objects in images | |
| CN112766232A (en) | Road risk target identification method based on reconfigurable convolutional neural network | |
| CN112070035A (en) | Target tracking method and device based on video stream and storage medium | |
| CN112805723B (en) | Image processing system and method and automatic driving vehicle comprising system | |
| CN112348011B (en) | Vehicle damage assessment method and device and storage medium | |
| CN110826469B (en) | Person detection method and device and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: 350000 building, No. 89, software Avenue, Gulou District, Fujian, Fuzhou 18, China Applicant after: Ruixin Microelectronics Co., Ltd Address before: 350000 building, No. 89, software Avenue, Gulou District, Fujian, Fuzhou 18, China Applicant before: Fuzhou Rockchips Electronics Co.,Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |