TWI805005B - Method for detecting images, electronic device and storage medium - Google Patents

Abstract

The present application provides a method for detecting images, an electronic device and a storage medium. The method includes: obtaining a neural network model, which includes n operators; mapping the neural network model to a singly linked list structure, which includes n nodes; establishing a data pair vector corresponding to the singly linked list structure based on an output and an input of each node; determining operator subgroups in the neural network model based on the data pair vector; inputting a target image to the neural network model, recording input data and output data of the operator subgroups; detecting the target image based on the input data and the output data, and outputting a detection result. By utilizing the present application, an image detection result can be quickly obtained.

Description

Image detection method, electronic device and storage medium

The present application relates to the technical field of image processing, and in particular to an image detection method, electronic equipment and a storage medium.

In practical applications, when an image is detected by a neural network model installed in an electronic device, the characteristic data of the image will be transmitted between the operating units. In general, it is necessary to record the input data and output data of each operation element, and perform further detection on the image based on the input data and output data of each operation element. However, some complex neural network models have many operands, and the input data and output data of the operands that need to be counted are huge. This will inevitably cause the electronic device to have excessive memory usage, slow CPU processing, and slow image detection speed.

In view of the above, it is necessary to provide an image detection method, an electronic device and a storage medium, which can quickly output image detection results.

The present application provides an image detection method, the method comprising: obtaining a neural network model, wherein the neural network model includes n operands; based on the logical operation relationship between the n operands, the The neural network model is mapped to a single-linked list structure, wherein the single-linked list structure includes n nodes; each node in the single-linked list structure is scanned, and statistics are made on each node in the neural network. The output and input in the network model; based on the output and input of each node, the data pair vector corresponding to the single linked list structure is established, wherein the data pair vector is [[a1, b1], [ a2, b2]...[ai, bi]...[an, bn]], ai is the output of each node, and bi is the input of each node; determine the neural network based on the data pair vector A subgroup of operands in the network model; inputting a bullseye image to the neural network model, recording input data and output data of the subgroup of operands; utilizing the neural network based on the input data and output data The model detects the bull's-eye image and outputs the detection result.

In a possible implementation, the n nodes in the singly linked list are OP1, OP2, ... OPi, ..., OPn, and the output ai of the node OPi indicates that the operand corresponding to the node OPi is in When the data is transmitted in the neural network model, the number of operands receiving the data; the input quantity bi of the node OPi indicates that the operand corresponding to the node OPi is in the neural network model When receiving the data transferred by other operands, the quantity of the other operands that transfer the data.

In a possible implementation manner, determining the operator subset in the neural network model based on the data pair vector includes: determining a plurality of node subsets according to the data pair vector; A subset of operands in the neural network model is determined.

In a possible implementation manner, determining multiple node subsets according to the data pair vector includes: S51, traversing the data pairs in the data pair vector starting from [a1, b1]; S52, judging whether the data pair Satisfy the push-in stacking condition, if the data pair does not meet the push-in stacking condition, execute S54; S53, if the data pair meets the push-in stacking condition, push the data pair into the stack; S54, judge the data Whether the pop-up stacking condition is satisfied, if the data pair does not meet the pop-up stacking condition, execute S57; S55, if the data pair meets the pop-up stacking condition, pop up the topmost data pair in the current stack, and count the remaining The number of data pairs m; S56, determine the start node corresponding to the topmost data pair in the stack, and determine the end node corresponding to the data pair that meets the pop-up condition, set Determine the subset formed by all nodes from the start node to the end node as the m+1 level node subset; S57, continue to traverse the data pairs in the data pair vector, and repeatedly execute S52 to S57 until all The data pairs in the data pair vector are all traversed; S58, end the traverse.

In a possible implementation manner, the determining the operator subset in the neural network model according to the multiple node subsets includes: traversing the multiple node subsets; if the multiple node subsets The nodes in any one of the node subsets are not repeated with the nodes in other node subsets, and each node subset is determined to be a subset of operands in the neural network model.

In a possible implementation manner, determining the operator subset in the neural network model according to the multiple node subsets further includes: traversing the multiple node subsets to obtain multiple first node subsets and A plurality of second subsets of nodes, wherein all nodes in the first subset of nodes are actually included in the second subset of nodes; removing all nodes of the first subset of nodes in the second subset of nodes ; Set the remaining node set in the second node subset as the operator subset of the neural network model.

In a possible implementation manner, determining the operator subset in the neural network model according to the multiple node subsets further includes: setting the multiple first node subsets as the neural network model's Subgroup of operands.

In a possible implementation, if bi 2 in the data pair [ai, bi], determine that the data pair [ai, bi] satisfies the pop-up stacking condition; if ai 2 in the data pair [ai, bi], determine The data pair [ai, bi] satisfies the push-in stacking condition.

The present application also provides an electronic device, the electronic device includes a processor and a memory, and the processor is configured to implement the image detection method when executing a computer program stored in the memory.

The present application also provides a computer-readable storage medium, the computer-readable storage medium stores A computer program is stored, and when the computer program is executed by the processor, the image detection method is realized.

The image detection method, electronic equipment and storage medium disclosed in the present application can quickly output image detection results.

S1~S7: steps

S51~S58: steps

5: Electronic equipment

51: memory

52: Processor

53: Computer program

54: communication bus

OP1~OP15: Node

Fig. 1 is a flowchart of a preferred embodiment of an image detection method disclosed in this application.

FIG. 2 is a detailed flow chart of step S5 of an image detection method disclosed in the present application.

FIG. 3 is an exemplary singly linked list structure disclosed in this application.

Fig. 4 is an exemplary node data transmission diagram disclosed in the present application.

Fig. 5 is a schematic structural diagram of an electronic device implementing a preferred embodiment of the image detection method of the present application.

In order to make the purpose, technical solution and advantages of the present application clearer, the present application will be described in detail below in conjunction with the accompanying drawings and specific embodiments.

As shown in FIG. 1 , it is a flow chart of a preferred embodiment of the image detection method of the present application. According to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted.

The image detection method is applied to one or more electronic devices 5, and the electronic device 5 is a device capable of automatically performing numerical calculation and/or information processing according to preset or stored instructions, and its hardware includes but Not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital signal processors (Digital Signal Processor, DSP), embedded devices, etc. .

The electronic device 5 can be any electronic product capable of man-machine interaction with the user, for example, a personal computer, a tablet computer, a smart phone, a personal digital assistant (Personal Digital Assistant, PDA), game consoles, interactive Internet TV (Internet Protocol Television, IPTV), smart wearable devices, etc.

The electronic equipment 5 may also include network equipment and/or user equipment. Wherein, the network device includes, but is not limited to, a single network server, a server group composed of multiple network servers, or a cloud composed of a large number of hosts or network servers based on Cloud Computing.

The network where the electronic device 5 is located includes, but is not limited to, the Internet, a wide area network, a metropolitan area network, a local area network, a virtual private network (Virtual Private Network, VPN) and the like.

Step S1. Obtain a neural network model, wherein the neural network model includes n operands, where n represents a positive integer.

In this embodiment, the neural network model may be a neural network model obtained directly from the network end, or may be a neural network model after optimization of the neural network model obtained from the network end. Model. Optimizing the neural network model can be understood as performing operations such as operator fusion, network pruning, model quantization, and network cutting on the neural network model. In this embodiment, the neural network model includes n operands, which are respectively op1, op2, . . . , opn.

Step S2: Map the neural network model into a singly linked list structure based on the logical operation relationship between the n operands, wherein the singly linked list structure includes n nodes.

In order to subsequently count the input and output data of each operator, it is necessary to map the neural network model into a singly linked list structure.

In this implementation manner, the singly linked list structure includes a plurality of nodes, and the plurality of nodes are arranged in a unidirectional manner. The node arranged at the start position of the singly linked list structure is the head of the singly linked list structure, and the node arranged at the end position of the singly linked list structure is the tail of the singly linked list structure. When accessing the singly linked list structure, it is necessary to read from the head to the tail according to the unidirectional order of the singly linked list.

In this embodiment, the logical operation relationship between the n operands can be understood as a data transfer relationship between the operands. For example, the neural network model includes an operator A, an operator B, an operator C, and an operator D. Wherein, the output data of the operator A is the input data of the operator B and the operator C, and the output data of the operator B and the output data of the operator C are the input data of the operator D. In this way, the single linked list structure that can be obtained includes A─>B─>C─>D, or A─>C─>B─>D.

In this embodiment, the singly linked list structure includes n nodes, and the n nodes are respectively OP1, OP2, ..., OPn, and the n nodes correspond one by one to the each operand of . Based on the dependency relationship between the n operands, multiple singly linked list structures can be obtained. In this embodiment, one of the singly linked list structures OP1─>OP2─>...─>OPn is selected for subsequent description. For example, the singly linked list structure shown in Figure 3.

Step S3, scanning each node in the singly linked list structure, and counting the output and input of each node in the neural network model.

In order to subsequently obtain the subgroup of operands, it is necessary to count the output and input of each node in the neural network model.

In this embodiment, the output amount represents the number of operation elements that receive the data when the operation element corresponding to the node transmits the data to the outside in the neural network model; the input amount represents the number of operation elements corresponding to the node. When the operation unit receives the data delivered by other operation units in the neural network model, the number of the operation unit that transmits the data.

Exemplarily, the input quantity of the node OP4 in FIG. 4 is 1, and the output quantity is 4.

Step S4, based on the output and input of each node, establish a data pair vector corresponding to the singly linked list structure, wherein the data pair vector is [[a1, b1], [a2, b2]...[ ai, bi]...[an, bn]], ai is the output of each node, bi is the input of each node, where ai represents a positive integer, bi represents a positive integer.

In this embodiment, the data pair vector is a two-dimensional array established according to the output and input of the nodes OP1, OP2, ..., and OPn, where [ai, bi] represents the output and input of the node OPi Quantitative data.

Step S5, determining a subset of operands in the neural network model based on the data pair vector.

In this embodiment, when the electronic device performs image detection through the neural network model, it needs to record the input data and output data of each operation unit. In addition, the data to be recorded by the electronic device also includes some invalid data. In this way, the electronic device needs to record a huge amount of data, which may lead to excessive memory usage and slow CPU processing of the electronic device. In order to improve the processing capability of electronic equipment and quickly obtain image detection results, the method provided by this application can be used to search for the subgroup of operands in the neural network model. In this way, only the input data and output data of the operator subgroup can be recorded, and the next step of image detection can be performed.

Specifically, determining the operator subset in the neural network model based on the data pair vector includes: determining a plurality of node subsets according to the data pair vector; determining the neural network model according to the plurality of node subsets. A subgroup of operands in the path model.

Referring to FIG. 2 , it is a detailed flow chart of step S5 of an image detection method disclosed in the present application. According to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted.

S51. Start traversing the data pairs in the data pair vector from [a1, b1]; S52. Determine whether the data pairs meet the push-in stacking conditions, wherein the data pairs [ai, bi] satisfy the push-in stacking conditions ai>=2, if the data pair does not meet the push stacking condition, execute S54; S53. If the data pair satisfies the push-in stacking condition, push the data pair into the stack; S54. Determine whether the data pair satisfies the pop-up stacking condition, wherein the data pair [ai, bi ] in bi>=2, if the data pair does not meet the pop-up stacking condition, execute S57; S55, if the data pair meets the pop-up stacking condition, pop up the topmost data pair in the current stack, and count the data in the stack The number m of the remaining data pairs, wherein m represents a positive integer; S56, determine the start node corresponding to the topmost data pair in the stack, and determine the end node corresponding to the data pair that satisfies the pop-up stacking condition, set from The subset formed by all nodes from the start node to the end node is a subset of m+1 level nodes; S57, continue to traverse the data pairs in the data pair vector, and repeatedly execute S52 to S57 until the data The data pairs in the pair vector are all traversed; S58, end the traverse.

After the above steps S51 to S58 are processed, there are two situations in the multiple node subsets obtained. One case is that there are no duplicate nodes in any two node subsets among the plurality of node subsets; the other case is that a certain subset among the plurality of node subsets is really included in other subsets. For different situations, the method for determining the subgroup of operands in the neural network model is also different.

In an embodiment, the determining the operator subset in the neural network model according to the plurality of node subsets includes: traversing the plurality of node subsets; if any one of the plurality of node subsets Nodes in the node subset are not repeated with nodes in other node subsets, and each node subset is determined as a subset of operands in the neural network model.

In another embodiment, the determining the operator subsets in the neural network model according to the multiple node subsets includes: traversing the multiple node subsets to obtain multiple first node subsets and multiple a second subset of nodes, wherein the first subset of nodes is actually included in the second subset of nodes; Except all the nodes in the first node subset in the second node subset, the remaining node sets in the second node subset are used as a subset of operands of the neural network model.

It should be noted that there are no duplicate nodes between the first node subset and the remaining node sets in the second node subset. Therefore, it is necessary to set the plurality of first node subsets as the operator subsets of the neural network model.

Exemplarily, if the structure of the singly linked list is OP1─>OP2─>...─>OP15, wherein the nodes OP1 to OP11 respectively correspond to the operational elements op1 to op15 of the neural network model, in the singly linked list The data pair vector corresponding to the node is [[2, 1], [1, 1], [1, 2], [4, 1], [1, 1], [1, 1], [1, 1] , [1,1], [1,1], [1,1], [1,1], [2,1], [1,1], [1,2], [1,4]]; Start traversing the data pair vector from [2, 1]; since the data pair [2, 1] satisfies the push-in stacking condition, push the data pair [2, 1] into the stack; because the data pair [1, 2] satisfies the pop-up Stacking conditions, pop up the topmost data pair [2, 1] in the current stack; count the number of data pairs in the stack as 0, and determine that nodes OP1, OP2, and OP3 are the first-level node subsets; continue to traverse the Data pair vector; since the data pair [4, 1] satisfies the push-in stacking condition, push the data pair [4, 1] into the stack; because the data pair [2, 1] satisfies the push-in stacking condition, push the data pair [2, 1] into the stack 1] Push into the stack; since the data pair [1, 2] satisfies the pop-up stacking condition, pop the topmost data pair [2, 1] in the current stack; count the number of data pairs in the stack as 1, and determine the node OP12, OP13, and OP14 are a subset of level 2 nodes; Continue traversing the data pair vector; since the data pair [1, 4] satisfies the pop-up stacking condition, pop the topmost data pair [4, 1] in the current stack; count the number of data pairs in the stack as 0, Determine that the nodes OP4, OP5, ..., OP15 are a first-level node subset; end the traversal, and determine that the operator subgroups in the neural network model include two first-level operator subgroups and one second-level operator subgroup Group. Wherein, one of the 1-level operator subgroups includes the operator op1, op2, and op3; the other 1-level operator subgroup includes the operator op4, op5, op6, op7, op8, op9, op10, op11, and op15. The 2-level operand subset includes op12, op13, and op14.

Step S6 , inputting the bull's-eye diagram image to the neural network model, and recording the input data and output data of the operator subgroup.

When the neural network model performs image detection, the feature data of the image will be transmitted between the operators, and the input data and output data of the operator subgroup need to be recorded, and then the next step of image processing and calculation can be performed.

In this embodiment, the input data of the operand subgroup is the output data of the initial operand corresponding to the start node, and the output data of the operand subgroup is the last operand corresponding to the end node input data. The method provided by the present application can only record the input data and output data of the sub-group of operation elements in the neural network model, and does not need to record the input data and output data of all the operation elements in the neural network model, thus The amount of recorded data can be greatly reduced.

Step S7 , using the neural network model to detect the bull's-eye image based on the input data and output data, and output a detection result.

In this embodiment, if it is necessary to use the neural network model to When detecting the face in the image, it is necessary to extract the features in the bull's-eye image. The features mainly include color, texture and edge. Therefore, the input data and output data mainly include color data, texture data and edge data. Through the transmission of the input data and output data among the operator subgroups in the neural network model, the detection of the bull's-eye image is completed, and the face detection result is output.

It should be noted that, if the integrated modules/units of the electronic device 5 are realized in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, all or part of the processes in the methods of the above embodiments of the present application can also be completed by instructing related hardware through computer programs, and the computer programs can be stored in a computer-readable storage medium. When the computer program is executed by the processor, it can realize the steps of the above-mentioned various method embodiments. Wherein, the computer program code may be in the form of original code, object code, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, flash drive, removable hard disk, magnetic disk, optical disk, computer memory, read-only memory (Read-Only Memory) , ROM).

In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical units, that is, they may be located in one place, or may also be distributed to multiple networks on the unit. Part or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional module in each embodiment of the present application may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one processing unit. in units. The above-mentioned integrated units can be implemented not only in the form of hardware, but also in the form of hardware plus software function modules.

It will be apparent to those skilled in the art that the present application is not limited to the details of the exemplary embodiments described above, but that the present application can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application. Therefore, no matter from any point of view, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the application is defined by the appended claims rather than the above description, so it is intended to All changes within the meaning and range of equivalents of the elements are embraced in this application. Any attached reference mark in a claim shall not be deemed to limit the claim to which it relates. In addition, it is obvious that the word "comprising" does not exclude other elements or steps, and the singular does not exclude the plural. A plurality of units or devices stated in the system requirements can also be implemented by one unit or device by software or hardware. Secondary terms are used to denote names without implying any particular order.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application without limitation. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be Make modifications or equivalent replacements without departing from the spirit and scope of the technical solutions of the present application.

S1~S7: steps

Claims (9)

An image detection method used in electronic equipment, wherein the image detection method includes: acquiring a neural network model, wherein the neural network model includes n operands, wherein n represents a positive integer; based on The logical operation relationship between the n operands maps the neural network model into a singly linked list structure, wherein the singly linked list structure includes n nodes, and the n nodes in the singly linked list are respectively OP1, OP2, ...OPi, ..., OPn, and the output ai of the node OPi indicates that when the operator corresponding to the node OPi transmits the data in the neural network model, the statistics The number of operands that receive the data; the input quantity bi of the node OPi represents the number of statistics obtained when the operand corresponding to the node OPi receives the data delivered by other operands in the neural network model The number of other operands, wherein ai represents a positive integer, and bi represents a positive integer; scan each node in the singly linked list structure, and count the output of each node in the neural network model and the input amount; establishing a data pair vector corresponding to the single linked list structure based on the output amount and input amount of each node, wherein the data pair vector is [[a1, b1], [a2, b2]... [ai, bi]...[an, bn]]; determine the operator subgroup in the neural network model based on the data pair vector; input the bullseye image to the neural network model, record the The input data and the output data of the operation element subgroup; based on the input data and the output data, the neural network model is used to detect the bull's-eye diagram image, and output the detection result. The image detection method according to claim 1, wherein determining the operator subset in the neural network model based on the data pair vector includes: determining a plurality of node subsets according to the data pair vector; A subset of operands in the neural network model is determined according to the plurality of node subsets. The image detection method according to claim 2, wherein determining a plurality of node subsets according to the data pair vector includes: Step 1: starting from [a1, b1] to traverse the data pairs in the data pair vector; step 2: Determine whether the data pair satisfies the push-in stacking condition, if the data pair does not meet the push-in stacking condition, perform step 4; Step 3: if the data pair meets the push-in stacking condition, press the data pair into the stack; the step 4: judge whether the data pair meets the pop-up stacking condition, if the data pair does not meet the pop-up stacking condition, perform step 7; step 5: if the data pair meets the pop-up stacking condition, pop up the current The topmost data pair in the stack, and counting the number m of the remaining data pairs in the stack, where m represents a positive integer; Step 6: Determine the starting node corresponding to the topmost data pair in the stack, and determine that satisfies The data of the pop-up stacking condition corresponds to the end node, and the subset formed by all the nodes from the start node to the end node is set as the m+1 level node subset; the step 7: judge the data Check whether there are data pairs that have not been traversed in the vector, and if there are data pairs that have not been traversed in the data pair vector, return to step 1 and continue to traverse the next data pair in the data pair vector; If all the data pairs in the data pair vector have been traversed, the traversal ends. The image detection method according to claim 2, wherein said determining the operator subset in the neural network model according to the plurality of node subsets includes: traversing the plurality of node subsets; if the The nodes in any one of the plurality of node subsets are not repeated with the nodes in other node subsets, and any one of the node subsets is determined as a subset of operands in the neural network model. The image detection method according to claim 2, wherein, according to the multiple node subsets, determining the operator subset in the neural network model further includes: traversing the multiple node subsets to obtain multiple A first node subset and a plurality of second node subsets, wherein the second node subset really contains the first node subset; removing the first node subset from the second node subset All nodes; setting the set of remaining nodes in the second node subset as the operator subset of the neural network model. The image detection method according to claim 5, wherein, according to the plurality of node subsets, determining the operator subset in the neural network model further includes: setting the plurality of first node subsets as all A subgroup of operands for a neural network model. The image detection method as described in claim 3, wherein: if the bi in the data pair [ai, bi] is greater than or equal to 2, it is determined that the data pair [ai, bi] meets the pop-up stacking condition; if the data pair [ai, bi] ai in bi] is greater than or equal to 2, it is determined that the data pair [ai, bi] satisfies the push-in stacking condition. An electronic device, wherein, the electronic device includes: a memory storing at least one instruction; and a processor executing the instruction stored in the memory to implement any one of claim 1 to claim 7. Image detection method. A computer-readable storage medium, wherein: at least one instruction is stored in the computer-readable storage medium, and the at least one instruction is executed by a processor in an electronic device to implement any one of request item 1 to request item 7 The image detection method described above.

Images