CN117392698A - Method, device, equipment and storage medium for identifying hand-drawn circuit diagram - Google Patents

Method, device, equipment and storage medium for identifying hand-drawn circuit diagram Download PDF

Info

Publication number
CN117392698A
CN117392698A CN202311407184.9A CN202311407184A CN117392698A CN 117392698 A CN117392698 A CN 117392698A CN 202311407184 A CN202311407184 A CN 202311407184A CN 117392698 A CN117392698 A CN 117392698A
Authority
CN
China
Prior art keywords
hand
drawn
image
module
circuit
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202311407184.9A
Other languages
Chinese (zh)
Inventor
李俊峰
黄进
孙伟
田丰
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shenzhen Honghe Innovation Information Technology Co Ltd
Original Assignee
Shenzhen Honghe Innovation Information Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shenzhen Honghe Innovation Information Technology Co Ltd filed Critical Shenzhen Honghe Innovation Information Technology Co Ltd
Priority to CN202311407184.9A priority Critical patent/CN117392698A/en
Publication of CN117392698A publication Critical patent/CN117392698A/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
    • G06V30/40—Document-oriented image-based pattern recognition
    • G06V30/42—Document-oriented image-based pattern recognition based on the type of document
    • G06V30/422—Technical drawings; Geographical maps
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/048—Activation functions
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • Software Systems (AREA)
  • Multimedia (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Molecular Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Medical Informatics (AREA)
  • Databases & Information Systems (AREA)
  • Image Analysis (AREA)

Abstract

The application relates to the technical field of image recognition, and provides a method, a device, equipment and a storage medium for recognizing a hand-drawn circuit diagram. The recognition method of the hand-drawn circuit diagram comprises the following steps: acquiring a hand-drawn circuit image of the hand-drawn circuit diagram; identifying element types of each hand-drawn circuit element in the hand-drawn circuit image and element areas thereof in the hand-drawn circuit image through a neural network; identifying the connection line position of each hand-drawn connection line from the remaining area excluding each element area in the hand-drawn circuit image; and generating a standard circuit diagram according to the identified element types, the corresponding element areas and the positions of the connecting lines, wherein standard circuit elements in the standard circuit diagram are determined according to the corresponding element types, and standard connecting lines are determined according to the positions of the corresponding connecting lines. The method and the device realize quick and accurate generation of the standard circuit diagram.

Description

Method, device, equipment and storage medium for identifying hand-drawn circuit diagram
Technical Field
The present disclosure relates to the field of image recognition technologies, and in particular, to a method, an apparatus, a device, and a storage medium for recognizing a hand-drawn circuit diagram.
Background
In many scenarios, for example: the circuit principle demonstration of the electronic product, the physical teaching of junior high school and other scenes need to generate a standard circuit diagram. Taking the physical teaching of junior high school as an example, in the process of using a large screen for teaching, a user generally needs to select required elements (such as an ammeter, a voltmeter, a resistor and the like) from software icons of physical disciplines in order to build a standard circuit diagram, generate corresponding elements on the large screen in a clicking or dragging mode, and then form a standard circuit diagram capable of teaching by a plurality of elements in a complex element connection or element editing mode.
Because a circuit diagram needs more elements, a user needs to repeatedly search in an element list to find element icons corresponding to a plurality of elements respectively, and the user needs to click on the icon, drag, connect and other operations for a plurality of times to form a final circuit diagram, the operation is complex, inflexible and time-consuming, and the whole process of generating the circuit diagram is time-consuming.
Therefore, how to quickly and accurately generate a standard circuit diagram is a urgent problem to be solved.
Disclosure of Invention
Based on the above-mentioned current situation, a main object of the present application is to provide a method, a device and a storage medium for identifying a hand-drawn circuit diagram, which are used for solving the problems of long time consumption and low accuracy of generating a standard circuit diagram in the prior art.
In order to achieve the above purpose, the technical scheme adopted in the application is as follows:
a method of identifying a hand-drawn circuit diagram, the method comprising: the acquisition step: acquiring a hand-drawn circuit image of a hand-drawn circuit diagram, wherein the hand-drawn circuit image comprises hand-drawn circuit elements and hand-drawn connecting wires; a first identification step: identifying element types of each hand-drawn circuit element in the hand-drawn circuit image and element areas thereof in the hand-drawn circuit image through a neural network; a second identification step: identifying the connection line position of each hand-drawn connection line from the remaining area excluding each element area in the hand-drawn circuit image; generating: and generating a standard circuit diagram according to the identified element types, the corresponding element areas and the positions of the connecting lines, wherein standard circuit elements in the standard circuit diagram are determined according to the corresponding element types, and standard connecting lines are determined according to the positions of the corresponding connecting lines.
Preferably, the neural network comprises a first convolution module and a second convolution module, wherein the first convolution module comprises a first characteristic sub-module and a first identification sub-module; the second convolution module comprises a second characteristic sub-module and a second identification sub-module; a first identifying step, comprising: a first judging sub-step: inputting the hand-painted circuit image into a first feature sub-module for feature extraction to obtain a first feature image, and identifying the first feature image through a first identification sub-module to obtain a first result vector; a second judging sub-step: inputting the first feature map to a second feature sub-module for feature extraction to obtain a second feature map, and identifying the second feature map through a second identification sub-module to obtain a second result vector; the first result vector and the second result vector both comprise an element area, a type and a classification probability belonging to the type of each hand-drawn circuit element; the length and the width of the input image, the first feature image and the second feature image input to the first feature sub-module are sequentially decreased, and the depth is sequentially increased; the comparison substep: and respectively comparing the classification probabilities of the hand-painted circuit elements in the first result vector and the second result vector, and taking the type corresponding to the highest classification probability as the element type of the hand-painted circuit element.
Preferably, the neural network further comprises a third convolution module, wherein the third convolution module comprises a third feature sub-module and a third identification sub-module; the first identifying step further comprises, prior to the comparing sub-step: and a third judging sub-step: inputting the second feature map to a third feature sub-module for feature extraction to obtain a third feature map, and identifying the third feature map through a third identification sub-module to obtain a third result vector; the third result vector comprises an element area, a type and a classification probability belonging to the type of each hand-drawn circuit element; the length and width of the images input to the first feature sub-module, the first feature image, the second feature image and the third feature image are sequentially decreased, and the depth is sequentially increased; the comparison substep: and respectively comparing the classification probabilities of the hand-drawn circuit elements in the first result vector, the second result vector and the third result vector, and taking the type corresponding to the highest classification probability as the element type of the hand-drawn circuit element.
Preferably, at least one of the first, second and third feature sub-modules comprises at least two convolution layers, the output of the previous convolution layer being the input of the next convolution layer in the sub-module comprising at least two convolution layers.
Preferably, the first feature sub-module comprises six convolutions, the second feature sub-module comprises two convolutions and the third feature sub-module comprises two convolutions.
Preferably, the first identifying step includes: according to the input image size of the neural network, the hand-painted circuit image is processed by taking the area where the hand-painted circuit image is located as the center, and the hand-painted circuit image which comprises the hand-painted circuit image and accords with the input image size is obtained; inputting the hand-drawn circuit image which accords with the input image size into a neural network to obtain the element types of all hand-drawn circuit elements in the hand-drawn circuit image and the element areas of the hand-drawn circuit elements in the hand-drawn circuit image.
Preferably, the neural network is trained by the following steps: acquiring first training data, wherein the first training data comprises a plurality of hand-drawn circuit diagram samples, and the hand-drawn circuit diagram samples comprise hand-drawn circuit elements; processing the plurality of hand-drawn circuit diagram samples to enable the hand-drawn circuit elements on the hand-drawn circuit diagram samples to perform at least one of rotation, scaling and mirroring operations to obtain second training data; training a preset convolutional neural network by using the first training data and the second training data, and obtaining the neural network after training is completed.
According to the embodiment of the application, the standard circuit diagram is obtained by identifying the hand-drawn circuit diagram, so that the time required by user reaction and the time required by searching for multiple operations such as element icons, clicking icons, dragging and connecting which are respectively corresponding to multiple elements can be reduced, and the generation efficiency of the standard circuit diagram is improved; further, the hand-drawn connecting lines and the hand-drawn circuit elements are separately identified, and after the hand-drawn circuit elements are identified through the trained neural network, the hand-drawn connecting lines are identified from the remaining area except the element area in the hand-drawn circuit image, so that the circuit elements are prevented from being identified as the hand-drawn connecting lines, and the accuracy of the finally generated standard circuit diagram is improved.
The application also provides the following technical scheme:
an apparatus for identifying a hand-drawn circuit diagram, the apparatus comprising: the device comprises an acquisition module, a display module and a display module, wherein the acquisition module is used for acquiring a hand-painted circuit image of the hand-painted circuit image, and the hand-painted circuit image comprises hand-painted circuit elements and hand-painted connecting wires; a first identifying module for identifying the element type of each hand-drawn circuit element in the hand-drawn circuit image and the element area thereof in the hand-drawn circuit image through a neural network; the second identification module is used for identifying the connecting wire position of each hand-drawn connecting wire in the residual area except for each element area in the hand-drawn circuit image; the generating module is used for generating a standard circuit diagram according to the identified element types, the corresponding element areas and the positions of the connecting lines, wherein standard circuit elements in the standard circuit diagram are determined according to the corresponding element types, and standard connecting lines are determined according to the positions of the corresponding connecting lines.
The embodiment of the application also provides a device for identifying the hand-drawn circuit diagram, which comprises: the processor and the memory store executable instructions, and the processor can execute the executable instructions to realize the method for identifying the hand-drawn circuit diagram in any embodiment.
The application also provides a computer readable storage medium, such as a chip, an optical disc, etc., on which an execution program is stored, where the execution program is executed to implement the method for identifying a hand-drawn circuit diagram according to any one of the embodiments.
Other advantages of the present application will be described in the detailed description by way of description of specific technical features and technical solutions, and those skilled in the art should understand the advantages brought by the technical features and technical solutions by way of description of the technical features and technical solutions.
Drawings
Preferred embodiments of the method, apparatus, device and computer-readable storage medium for identifying hand-drawn circuit diagrams of the present application will be described below with reference to the accompanying drawings. In the figure:
fig. 1 is an application scenario schematic diagram of a method for identifying a hand-drawn circuit diagram according to an embodiment of the present application;
fig. 2 is a flow chart of a method for identifying a hand-drawn circuit diagram according to an embodiment of the present application;
Fig. 3 is a flowchart illustrating a sub-step of S120 in the method for identifying a hand-drawn circuit diagram according to the embodiment of the present application;
fig. 4 is a schematic diagram of a neural network model according to an embodiment of the present application;
fig. 5 is a block diagram of a hand-drawn circuit diagram recognition device according to an embodiment of the present application;
fig. 6 is a block diagram of a hand-drawn circuit diagram recognition device according to an embodiment of the present application.
Detailed Description
The present application is described below based on examples, but the present application is not limited to only these examples. In the following detailed description of the present application, certain specific details are set forth in order to avoid obscuring the essence of the present application, well-known methods, procedures, flows, components are not described in detail.
Moreover, those of ordinary skill in the art will appreciate that the drawings are provided herein for illustrative purposes and that the drawings are not necessarily drawn to scale.
Unless the context clearly requires otherwise, throughout the description and the claims, the words "comprise", "comprising", and the like are to be construed in an inclusive sense as opposed to an exclusive or exhaustive sense; that is, it is the meaning of "including but not limited to".
In the description of the present application, it should be understood that the terms "first," "second," and the like are used for descriptive purposes only and are not to be construed as indicating or implying relative importance. Furthermore, in the description of the present application, unless otherwise indicated, the meaning of "a plurality" is two or more.
In many scenarios, for example: the circuit principle demonstration of the electronic product, the physical teaching of junior high school and other scenes need to generate a standard circuit diagram. Taking the physical teaching of junior high school as an example, in the process of using a large screen for teaching, a user generally needs to select required elements (such as an ammeter, a voltmeter, a resistor and the like) from software icons of physical disciplines in order to build a standard circuit diagram, generate corresponding elements on the large screen in a clicking or dragging mode, and then form a standard circuit diagram capable of teaching by a plurality of elements in a complex element connection or element editing mode.
Because a circuit diagram needs more elements, a user needs to repeatedly search in an element list to find element icons corresponding to a plurality of elements respectively, and the user needs to click on the icon, drag, connect and other operations for a plurality of times to form a final circuit diagram, the operation is complex, inflexible and time-consuming, and the whole process of generating the circuit diagram is time-consuming.
In the research process, the inventor discovers that the standard circuit diagram can be generated by enabling a user to manually draw the circuit diagram and then identifying the manually drawn circuit diagram, so that the generation efficiency of the standard circuit diagram is improved; the plurality of circuit elements and the connecting lines of the plurality of circuit elements in the hand-drawn circuit diagram can be identified by training the neural network model, and finally the circuit elements with standard shapes and the connecting lines with standard shapes are output. However, identifying circuit elements together with connection lines may result in some circuit elements being identified as connection lines, which may not be well identified. Therefore, further, the hand-drawn circuit element and the hand-drawn connecting wire can be separately identified, and the hand-drawn circuit element can be identified firstly and then the hand-drawn connecting wire can be identified, so that the hand-drawn circuit element is prevented from being identified as the hand-drawn connecting wire, and the accuracy of the hand-drawn circuit diagram identification is improved.
The method and the device are suitable for identifying the hand-painted circuit diagram. The recognition of the hand-drawn circuit diagram can be realized by recognizing the hand-drawn circuit diagram on the interaction large screen in real time and outputting the recognition result to the interaction large screen in real time; the method can also be used for identifying the circuit diagram of the hand drawing on the interactive screen in real time and outputting the identification result to other screens in real time; the method can also be used for identifying the hand-drawn circuit diagram in the stored shooting image and storing the identification result.
Fig. 1 is an application scenario schematic diagram of a method for identifying a hand-drawn circuit diagram according to an embodiment of the present application. Referring to fig. 1, P1 is a hand-drawn circuit image including a hand-drawn circuit diagram, P1 includes a hand-drawn power supply, a capacitor and a resistor, and hand-drawn connection lines between hand-drawn circuit elements, and P2 is an image including a standard circuit diagram obtained by identifying the hand-drawn circuit diagram in P1.
Fig. 2 is a flowchart of a method for identifying a hand-drawn circuit diagram according to an embodiment of the present application. The method is executed by the identification equipment of the hand-drawn circuit diagram, wherein the identification equipment of the hand-drawn circuit diagram can be intelligent writing equipment such as an electronic whiteboard and an interactive large screen, can be a control unit in the intelligent writing equipment, and can also be intelligent terminal equipment such as a computer, a tablet personal computer and a mobile phone. Referring to fig. 2, the method includes the steps of:
s110, acquiring: a hand-drawn circuit image of the hand-drawn circuit diagram is obtained, wherein the hand-drawn circuit image comprises hand-drawn circuit elements and hand-drawn connecting lines.
As shown in fig. 1, P1 in fig. 1 is a hand-drawn circuit image including a hand-drawn circuit diagram, wherein the hand-drawn circuit diagram includes a hand-drawn power supply, a hand-drawn capacitor and a hand-drawn resistor, and a hand-drawn connection line between the hand-drawn power supply, the hand-drawn capacitor and the hand-drawn resistor.
The hand-painted circuit image drawn on the interactive screen can be obtained in real time; the hand-drawn circuit image stored on the recognition device of the hand-drawn circuit diagram can also be obtained; the hand-drawn circuit image may also be obtained from a third party device.
S120, a first identification step: the element type of each hand-drawn circuit element in the hand-drawn circuit image and the element region thereof in the hand-drawn circuit image are identified by the neural network.
Before the neural network is used for executing the first recognition step, an original model needs to be trained, namely, a marked hand-drawn circuit diagram needs to be used as a training data set to be input into the original model, and parameters of the original model are adjusted according to model output results so that the trained neural network has a better recognition result on hand-drawn circuit elements, namely, the trained neural network can accurately classify the hand-drawn circuit elements in the hand-drawn circuit image. When labeling hand-drawn circuit diagrams, it is necessary to collect several hand-drawn circuit diagrams and label the positions and types of circuit elements in the hand-drawn circuit diagrams.
When the neural network is used to identify the hand-drawn circuit image (i.e., when the first identifying step is performed), the hand-drawn circuit image needs to be processed to meet the input image size requirement of the neural network before the acquired hand-drawn circuit image is input to the trained neural network. If the hand-painted circuit image is smaller than the input image size of the neural network, black edge filling is needed to be carried out on the hand-painted circuit image so as to meet the requirement of the input image size of the neural network; if the hand-drawn circuit image size is larger than the input image size of the neural network, the hand-drawn circuit image needs to be compressed to meet the input image size requirement of the neural network. For example: the size of the hand-drawn circuit image is 960×540, and the input size of the neural network is 640×640, so that the size of the hand-drawn circuit image needs to be compressed to 640×640.
When the circuit diagram drawn on the interactive screen is identified in real time, the circuit diagram drawn on the screen is redrawn to an off-screen bitmap, the off-screen bitmap is processed into the input image size of the neural network, and the off-screen bitmap meeting the requirement of the input image size is input into the neural network for identification.
Preferably, in the practical application scene, the colors of the elements and the connecting lines in the circuit diagram are not required, the generated standard circuit diagram can be a black-and-white image, namely, no color identification requirement exists, so that the input image of the input neural network can be a black-and-white image or a gray image, and the parameter of the number of input channels of the neural network can be set to be 1, so that the training efficiency of the model is improved. For example, a black-and-white or gray-scale hand-drawn circuit image is directly obtained, or the obtained color hand-drawn circuit image is converted into a black-and-white image or a gray-scale image, and the pixel values of the three RGB channels of the black-and-white image or the gray-scale image are the same, so that the requirement that the number of input channels of the neural network is 1 is satisfied.
Inputting the obtained hand-drawn circuit images into a trained neural network, and identifying the element types of the hand-drawn circuit elements in the hand-drawn circuit images and the positions of the hand-drawn circuit elements in the hand-drawn circuit images by the neural network (namely, the element areas can be represented by coordinates of two diagonal vertexes of rectangular areas). Preferably, since the aspect ratio of the circuit element is relatively fixed, in order to achieve a relatively fast recognition speed, the element region obtained by the neural network recognition is set to a region of a fixed aspect ratio, for example, a fixed aspect ratio of 1:2, the element area aspect ratio is 1:2 or 2:1. the element region may be represented by the upper left corner, width and height (e.g., in the form of a bounding box) of the rectangular region, or may be represented directly by the upper left corner, lower right corner, of the rectangular region.
After the neural network finishes the recognition, the recognition device of the hand-drawn circuit diagram can retain the element type data and the position data of a plurality of hand-drawn circuit elements, and can also convert the hand-drawn circuit elements into standard circuit elements according to the element types of the hand-drawn circuit elements, and generate the standard circuit elements at the element areas thereof to replace or cover the original hand-drawn circuit elements in the element areas.
S130, a second identification step: the connection line position of each hand-drawn connection line is identified from the remaining area excluding each element area in the hand-drawn circuit image.
After the first recognition step is performed, whether the hand-drawn circuit element remains in the hand-drawn circuit image or the original hand-drawn circuit element is replaced by the corresponding standard circuit element, the hand-drawn connection line is recognized directly from the complete hand-drawn circuit image, and the circuit element is easily recognized as the connection line. As shown in P1 in fig. 1, the hand-drawn power supply in P1 is easily identified as a long and a short two-wire connection; as shown in P2 in fig. 1, the capacitance in P2 is easily identified as two connecting lines of equal length.
Therefore, it is necessary to determine the remaining area except for each element area in the hand-drawn circuit image, and then identify the connection line position of each hand-drawn connection line from the remaining area, so as to avoid identifying the hand-drawn circuit element or the standard circuit element as the connection line, and improve the identification accuracy.
The connection line position of each hand-drawn connection line is the position (for example, the start point coordinate and the end point coordinate) of the start point and the end point of each hand-drawn connection line in the hand-drawn circuit image.
S140, generating: and generating a standard circuit diagram according to the identified element types, the corresponding element areas and the positions of the connecting lines, wherein standard circuit elements in the standard circuit diagram are determined according to the corresponding element types, and standard connecting lines are determined according to the positions of the corresponding connecting lines.
And searching a standard circuit element corresponding to the element type according to the element type obtained by the first identification step, displaying the circuit element on a corresponding position in the hand-painted circuit image according to the element area obtained by the first identification step, displaying the standard connection line on a corresponding position in the hand-painted circuit image according to each connection line position obtained by the second identification step, and finally obtaining the standard circuit diagram.
The standard circuit diagram can be obtained after the original hand-painted circuit element and the hand-painted connecting wire are replaced in the original hand-painted circuit image, or can be obtained by drawing the standard circuit element and the standard connecting wire in a blank image with the same size as the original hand-painted circuit image.
The standard circuit element is a circuit element with a preset icon; the standard connecting line is a line with a preset pattern.
It should be noted that, in the embodiment of the present application, the determining sequence of the standard circuit element and the standard connection line is not limited, and the standard circuit element may be determined first and then the standard connection line may be determined, the standard connection line may be determined first and then the standard circuit element may be determined, and the standard circuit element and the standard connection line may be determined simultaneously.
Compared with the method that a standard circuit diagram is generated by dragging standard circuit elements by a user, the method and the device for generating the standard circuit diagram can reduce time required by user reaction and time required by searching for element icons, clicking icons, dragging, connecting and other operations corresponding to a plurality of elements respectively by identifying the hand-drawn circuit diagram, so that the generation efficiency of the standard circuit diagram is improved; further, the hand-drawn connecting lines and the hand-drawn circuit elements are separately identified, and after the hand-drawn circuit elements are identified through the trained neural network, the hand-drawn connecting lines are identified from the remaining area except the element area in the hand-drawn circuit image, so that the circuit elements are prevented from being identified as the hand-drawn connecting lines, and the accuracy of the finally generated standard circuit diagram is improved.
In real-time recognition of hand-drawn circuit diagrams, there is generally a very high requirement for recognition speed, and therefore, in a preferred manner, S130 includes the steps of:
s131, identifying the connecting line position of each hand-drawn connecting line from the rest area except the element area in the hand-drawn circuit image by using a Hough straight line detection algorithm.
The hough straight line detection algorithm is a classical line detection algorithm, and is used for detecting straight lines in an image. The basic principle is to map points in the image space into the parameter space and determine straight lines in the image by finding intersections in the parameter space.
Compared with other detection algorithms, such as a Canny edge detection algorithm, an edge detector and an LSD algorithm (Line Segment Detector, straight line segment detection algorithm), the Hough straight line detection algorithm has better robustness to noise and partial shielding in an image, can detect straight lines in any direction, is not limited by the inclination angle of the straight lines, namely the Hough straight line detection algorithm can accurately detect the straight lines; the Hough straight line detection algorithm uses parameterization to represent straight lines, namely, the positions of a plurality of straight lines in a parameter space are represented through parameters, the representation form is convenient for detecting and extracting the straight lines, and the positions of connecting lines of all hand-drawn connecting lines can be recognized relatively quickly, so that the requirement on recognition speed when a hand-drawn circuit diagram is recognized in real time is met.
Furthermore, the connecting lines in the hand-drawn circuit diagram are usually straight lines, so that the Hough straight line detection algorithm is used for guaranteeing the identification accuracy and meeting the requirement on the identification speed when the hand-drawn circuit diagram is identified in real time.
In a preferred manner, the neural network comprises a first convolution module and a second convolution module, the first convolution module comprising a first feature sub-module and a first identification sub-module; the second convolution module includes a second feature sub-module and a second identification sub-module.
Each of the first feature sub-module and the second feature sub-module may include one or more layers of convolution units (the convolution units include a convolution layer, a nonlinear activation layer, a max-pooling layer, etc.), where a calculation formula of the convolution units is as follows:
in the equation, k represents a kth layer convolution unit, k=1.2.3 …; c (C) k-1 The number of the characteristic diagrams output by the k-1 layer is represented,an ith feature map representing the output of the kth-1 layer,>the j-th feature map representing the k-th layer output, max_pooling (x) representing the maximum pooling operation, delta (x) representing the activation function, +.> Representing convolution operations +.>Representation ofFeature map and->Convolution kernel with connected feature maps>Representation->Bias parameters of the feature map.
When training the neural network, iterative training can be carried out on the original model by using a batch random gradient descent method and a back propagation algorithm until the loss cost E of the model k And (3) reducing the precision to a fixed precision, stopping training, and storing the trained model to obtain the neural network used by the recognition method of the hand-drawn circuit diagram in the embodiment of the application.
The calculation formula of the loss cost of the model is as follows:
in the equation, C k Representing the kth sample data x in the sample data of the hand-drawn circuit diagram k Class loss of corresponding circuit element, B k Representing sample x k Corresponding position loss, α represents the adjustment parameter, and N represents the number of samples.
Fig. 3 is a flowchart illustrating a sub-step of S120 in the method for identifying a hand-drawn circuit diagram according to an embodiment of the present application. Referring to fig. 3, S120 includes the following sub-steps:
s121, a first judging sub-step: and inputting the hand-drawn circuit image into a first characteristic submodule to perform characteristic extraction to obtain a first characteristic image, and identifying the first characteristic image through a first identification submodule to obtain a first result vector.
Inputting the hand-painted circuit image into a first characteristic sub-module, and operating the hand-painted circuit image by a convolution unit in the first characteristic sub-module to extract the characteristics which can be used for identifying the region where the hand-painted circuit element is located and the type of the circuit element to obtain a first characteristic diagram; the first feature map is input to a first identification sub-module, a convolution unit in the first identification sub-module carries out operation on the first feature map, the area where the hand-painted circuit element is located is detected, and the type of the circuit element where the hand-painted circuit element is located is identified to obtain a first result vector. The first result vector includes an element area where each detected hand-drawn circuit element is located, and a probability that each detected hand-drawn circuit element belongs to each circuit element type.
S123, a second judging sub-step: and inputting the first feature map to a second feature sub-module for feature extraction to obtain a second feature map, and identifying the second feature map through a second identification sub-module to obtain a second result vector. The first result vector and the second result vector both comprise an element area, a type and a classification probability belonging to the type of each hand-drawn circuit element; the length and width of the input image, the first feature image and the second feature image input to the first feature sub-module are sequentially decreased, and the depth is sequentially increased.
Inputting the first feature map to a second feature sub-module, and operating the first feature map by a convolution unit in the second feature sub-module to obtain a second feature map; and inputting the second characteristic diagram into a second identification sub-module, and operating the second characteristic diagram by a convolution unit in the second identification sub-module, detecting the region where the hand-painted circuit element is positioned, and identifying the type of the circuit element to which the hand-painted circuit element belongs to obtain a second result vector. The second result vector includes an element area where each detected hand-drawn circuit element is located, and a probability that each detected hand-drawn circuit element belongs to each circuit element type.
In the whole neural network, the first feature map belongs to a shallow feature map compared with the second feature map, and the second feature map belongs to a deep feature map compared with the first feature map. The first characteristic diagram is subjected to convolution operation of the neural network to obtain a second characteristic diagram, the size of the second characteristic diagram is smaller than that of the first characteristic diagram, namely the length and the width of images of the first characteristic diagram and the second characteristic diagram are sequentially decreased, and the receptive field of the second characteristic diagram is larger than that of the first characteristic diagram, so that the second characteristic diagram is more suitable for detecting circuit elements with larger sizes, the first characteristic diagram is more suitable for detecting circuit elements with smaller sizes, the classification probability of the hand-drawn circuit elements with larger sizes in a second result vector obtained by identifying the second characteristic diagram is more accurate, and the classification probability of the hand-drawn circuit elements with smaller sizes in the first result vector obtained by identifying the first characteristic diagram is more accurate.
The image depths of the first feature map and the second feature map are sequentially increased, namely the number of channels is increased, and the increase of the number of channels is beneficial to the convolution unit to extract more feature types of the hand-painted circuit elements, so that the more the number of channels is, the more accurate the identification result is.
S125a, comparing sub-steps: and respectively comparing the classification probabilities of the hand-painted circuit elements in the first result vector and the second result vector, and taking the type corresponding to the highest classification probability as the element type of the hand-painted circuit element.
The first feature map contains outline detail information of more hand-drawn circuit elements, the second feature map is obtained by convolution operation of the first feature map, and due to the effect of the neural network convolution unit, the second feature map contains overall outline information of more hand-drawn circuit elements, namely, the scale of the second feature map is smaller than that of the first feature map, so that the probability that the hand-drawn circuit elements in a first recognition result obtained according to the first feature map containing more outline detail information are classified as circuit element types is different from the probability that the hand-drawn circuit elements in a second recognition result obtained according to the second feature map containing more overall outline information are classified as circuit element types. The classification probabilities of the hand-painted circuit elements in the first result vector and the second result vector are compared respectively, and the type corresponding to the highest classification probability is used as the element type of the hand-painted circuit element, so that the accuracy of the recognition of the hand-painted circuit element can be improved.
For example, there are four element types, namely, power, resistance, capacitance, and diode, and the probability that a certain hand-drawn circuit element is classified as power, resistance, capacitance, and diode in the first result vector is 0.3, 0.4, 0.1, and 0.2, respectively, and the probability that it is classified as power, resistance, capacitance, and diode in the second result vector is 0.5, 0.2, and 0.1, respectively, the highest classification probability of the hand-drawn circuit element is 0.5, and therefore, the hand-drawn circuit element is considered to be the power corresponding to 0.5.
Compared with a neural network which only recognizes the feature diagram output by the last layer of convolution unit to obtain a result vector, the neural network is divided into the first convolution module and the second convolution module when in use because the scale change of the hand-painted circuit elements in the hand-painted physical circuit diagram is larger, the first convolution module and the second convolution module respectively output the first result vector and the second result vector, the classification probability of each hand-painted circuit element in the first result vector and the second result vector is compared, the type corresponding to the highest classification probability is used as the element type of the hand-painted circuit element, and the circuit elements with different scale changes can be recognized better, so that the accuracy of neural network recognition is improved.
In order to further identify circuit elements of different scale variations, in an alternative manner, the neural network further comprises a third convolution module comprising a third feature sub-module and a third identification sub-module.
Prior to S125a, S120 further includes: s124, a third judging sub-step: and inputting the second feature map to a third feature sub-module for feature extraction to obtain a third feature map, and identifying the third feature map through a third identification sub-module to obtain a third result vector. The third result vector comprises an element area, a type and a classification probability belonging to the type of each hand-drawn circuit element; the length and width of the images input to the first feature sub-module, the first feature image, the second feature image and the third feature image are sequentially decreased, and the depth is sequentially increased.
Because the pixels of the hand-drawn circuit diagram are too sparse, a third convolution module can be arranged for the neural network, the length and the width of an image of the third feature diagram obtained by extracting the features of the second feature diagram by the third convolution module can be reduced, namely the size of the third feature diagram can be reduced, and therefore the third convolution module can identify the hand-drawn circuit diagram with larger size and sparse pixel distribution.
And because the similarity of the hand-drawn handwriting is higher, the depth of the image of the third feature map obtained by the feature extraction of the second feature map by the third convolution module is increased by setting the number of convolution kernels of the third feature sub-module in the third convolution module, namely the number of channels is increased, so that the third convolution module can more accurately identify the types of the hand-drawn circuit elements.
S125a further includes: s125b, i.e. in this implementation, the comparing sub-step comprises: and respectively comparing the classification probabilities of the hand-drawn circuit elements in the first result vector, the second result vector and the third result vector, and taking the type corresponding to the highest classification probability as the element type of the hand-drawn circuit element.
The classification probabilities of the hand-painted circuit elements in the first result vector, the second result vector and the third result vector are compared, and the type corresponding to the highest classification probability is used as the element type of the hand-painted circuit element, so that the circuit elements with different scale changes and different painting styles can be better identified, and the accuracy of neural network identification is improved.
In order to improve the recognition efficiency of the neural network, in a preferred manner, at least one of the first feature sub-module, the second feature sub-module and the third feature sub-module includes at least two convolution layers, and in the sub-module including at least two convolution layers, the output of the previous convolution layer serves as the input of the next convolution layer.
Preferably, the first feature sub-module comprises six convolutions, the second feature sub-module comprises two convolutions and the third feature sub-module comprises two convolutions. Fig. 4 is a schematic diagram of a neural network according to an embodiment of the present application. Referring to fig. 4, the neural network includes shallow convolutional layers G1 to G6 (i.e., first feature sub-modules), middle convolutional layers G7 to G8 (i.e., second feature sub-modules) and deep convolutional layers G9 to G10 (i.e., third feature sub-modules) of the backbone, and further includes a shallow detect and identify convolutional layer (i.e., first identification sub-modules), a middle detect and identify convolutional layer (i.e., second identification sub-modules) and a deep detect and identify convolutional layer (i.e., third identification sub-modules), wherein the shallow detect and identify convolutional layer outputs a shallow detect and identify result (i.e., first result vector), the middle detect and identify convolutional layer outputs a middle detect and identify result (i.e., second result vector), and the deep detect and identify convolutional layer outputs a deep detect and identify result (i.e., third result vector). The output of the previous convolution layer between G1 and G10 is taken as the input of the next convolution layer, i.e. G1 receives the hand-drawn circuit diagram of the input neural network and takes the output characteristic diagram as the input of G2, … … and the characteristic diagram of G9 as the input of G10. The first feature map output by the main shallow convolution layer G6 is used as the input of the middle convolution layer G7 of the main and also used as the input of the shallow detection and identification convolution layer; the second feature map output by the middle layer convolution layer G8 of the trunk is used as the input of the deep layer convolution layer G9 of the trunk and also used as the input of the middle layer detection and identification convolution layer; the third feature map output by the main deep convolutional layer G10 serves as an input to the deep detection and recognition convolutional layer to output a deep detection and recognition result.
Compared with the method that each layer of convolution layer is provided with an identification sub-module for identifying the feature map output by each layer of convolution layer, the identification sub-module is arranged when the feature map extracts the features suitable for identification, so that the identification efficiency of the neural network can be improved; by arranging a plurality of convolution layers in the first feature sub-module, the second feature sub-module and the third feature sub-module, higher-level semantic information of the features can be effectively detected, and therefore identification accuracy is improved.
In order to further improve the recognition efficiency of the hand-drawn circuit diagram, in a preferred manner, S120 further includes the following sub-steps:
and S127, processing the hand-painted circuit image by taking the region where the hand-painted circuit image is positioned as the center according to the input image size of the neural network, and obtaining the hand-painted circuit image which comprises the hand-painted circuit image and accords with the input image size.
In identifying the hand-drawn circuit image using the neural network (i.e., in performing the first identifying step), the acquired hand-drawn circuit image also needs to be cropped before being input into the trained neural network. For example, when a hand-drawn circuit diagram is drawn on a large screen, the hand-drawn circuit diagram does not occupy the whole screen, and more often, the hand-drawn circuit diagram occupies only a part of the large screen, and the circuit diagram drawn on the large screen is redrawn on an off-screen bitmap, where the area of the hand-drawn circuit diagram occupies a very low proportion of the total area of the off-screen bitmap. In order to improve the recognition efficiency of the neural network and avoid nonsensical operation on the blank area, at this time, the hand-drawn circuit image including the hand-drawn circuit image needs to be cut out by taking the area where the hand-drawn circuit image is located as the center, and then the hand-drawn circuit image is compressed or filled to be in accordance with the size of the input image.
S129, inputting the hand-drawn circuit image which accords with the input image size into a neural network to obtain the element types of the hand-drawn circuit elements in the hand-drawn circuit image and the element areas of the hand-drawn circuit elements in the hand-drawn circuit image.
When the hand-painted circuit image is processed, the area of the hand-painted circuit image occupies a relatively high area in the total area of the hand-painted circuit image by taking the area of the hand-painted circuit image as the center, so that the recognition efficiency of the neural network is improved, namely the recognition efficiency of the hand-painted circuit image is improved; and the hand-drawn circuit image can be ensured to contain a complete hand-drawn circuit diagram, so that the condition of inaccurate identification result caused by the defect of circuit elements in the hand-drawn circuit diagram is avoided.
Because of the difference in drawing styles, there may be a large difference between a plurality of hand-drawn circuit elements obtained by drawing the same circuit element by different people, so in order to improve the generalization capability of the neural network, in a preferred manner, the neural network used in the embodiment of the present application is obtained by training the following steps:
s210, acquiring first training data, wherein the first training data comprises a plurality of hand-drawn circuit diagram samples, and the hand-drawn circuit diagram samples comprise hand-drawn circuit elements.
And manually drawing a hand-drawn circuit diagram sample, and taking the hand-drawn circuit elements in the hand-drawn circuit diagram sample and the types of the circuit elements obtained by manually marking the circuit elements as first training data.
S220, processing the plurality of hand-drawn circuit diagram samples, so that the hand-drawn circuit elements on the hand-drawn circuit diagram samples perform at least one of rotation, scaling and mirroring operations to obtain second training data.
In an actual application scene, when a user draws a hand-drawn circuit diagram, the user has the condition of oblique drawing. For example: the drawn circuit elements may be inclined by 0 deg. to 30 deg. compared to the horizontal line of the screen. Therefore, in order to improve the accuracy of identifying the inclined hand-drawn circuit elements, the hand-drawn circuit pattern sample is rotated to obtain the second training data. Preferably, the second training data is obtained by rotating the hand-drawn circuit diagram sample by-10 °, -20 °, -30 °, 10 °, -20 ° and-30 °. The model is trained by using the training data, so that the recognition requirement of the hand-drawn circuit element drawn obliquely can be met.
When drawing a hand-drawn circuit diagram, there are cases where the sizes of a plurality of hand-drawn circuit elements in the same hand-drawn circuit diagram are not uniform, and there are cases where the sizes of hand-drawn circuit elements in different hand-drawn circuit diagrams are not uniform. Therefore, in order to improve the accuracy of identifying the hand-drawn circuit elements with inconsistent sizes, the second training data is obtained by performing scaling operation on the hand-drawn circuit diagram samples.
In drawing a circuit diagram, there are cases where one hand-drawn circuit element has different placement directions, for example, the positive electrode of the power supply is on the left, the positive electrode of the power supply is on the right, and whether the positive electrode of the hand-drawn power supply is on the left or the positive electrode is on the right, the power supply is indicated. Therefore, in order to improve the accuracy of identifying the hand-drawn circuit elements in different placement directions, the second training data is obtained by performing mirror image operation on the hand-drawn circuit pattern samples.
It should be noted that, in the embodiment of the present application, three operations including rotation, scaling and mirroring are preferably three operations, at least one operation of the three operations may be selected according to practical situations, and in order to improve generalization capability of the neural network, other operations may be selected to be performed on the first training data to obtain second training data, for example, translation may be performed on a hand-drawn circuit element in the first training data to obtain second training data that is partially overlapped between the hand-drawn circuit elements.
And S230, training a preset convolutional neural network by using the first training data and the second training data, and obtaining the neural network after training is completed.
When the model is trained, the marked hand-drawn circuit diagram (i.e. the hand-drawn circuit diagram sample) is used as a training data set to optimize the parameters of the original model to obtain a trained neural network. The first training data and the second training data are used as training data sets to train the original model, so that the generalization capability of the trained neural network can be effectively improved, and the accuracy of neural network identification is improved.
Based on the above-described concept, according to an aspect of the embodiments of the present application, there is provided an apparatus for recognizing a hand-drawn circuit diagram, which is capable of performing the method for recognizing a hand-drawn circuit diagram of any of the embodiments described above. Referring specifically to fig. 5, fig. 5 is a block diagram illustrating a structure of a device for identifying a hand-drawn circuit diagram according to an embodiment of the present application. As shown in fig. 5, the apparatus 300 includes an acquisition module 310, a first identification module 320, a second identification module 330, and a generation module 340.
An obtaining module 310, configured to obtain a hand-drawn circuit image of a hand-drawn circuit diagram, where the hand-drawn circuit image includes hand-drawn circuit elements and hand-drawn connection lines;
a first identifying module 320, configured to identify, through a neural network, an element type of each hand-drawn circuit element in the hand-drawn circuit image and an element area thereof in the hand-drawn circuit image;
a second identifying module 330 for identifying the connection line position of each hand-drawn connection line from the remaining area excluding each element area in the hand-drawn circuit image;
the generating module 340 is configured to generate a standard circuit diagram according to the identified element types and the corresponding element areas, and the positions of the connecting lines, where the standard circuit elements in the standard circuit diagram are determined according to the corresponding element types, and the standard connecting lines are determined according to the positions of the corresponding connecting lines.
In an alternative manner, the second identifying module 330 is further configured to identify the connection line position of each hand-drawn connection line from the remaining area except for each element area in the hand-drawn circuit image using a hough line detection algorithm.
In an alternative manner, the neural network includes a first convolution module and a second convolution module, the first convolution module including a first feature sub-module and a first identification sub-module; the second convolution module comprises a second characteristic sub-module and a second identification sub-module; the first recognition module 320 is further configured to input the hand-painted circuit image to the first feature sub-module for feature extraction to obtain a first feature map, and recognize the first feature map through the first recognition sub-module to obtain a first result vector; inputting the first feature map to a second feature sub-module for feature extraction to obtain a second feature map, and identifying the second feature map through a second identification sub-module to obtain a second result vector; the first result vector and the second result vector both comprise an element area, a type and a classification probability belonging to the type of each hand-drawn circuit element; the length and the width of the input image, the first feature image and the second feature image input to the first feature sub-module are sequentially decreased, and the depth is sequentially increased; and respectively comparing the classification probabilities of the hand-painted circuit elements in the first result vector and the second result vector, and taking the type corresponding to the highest classification probability as the element type of the hand-painted circuit element.
In an alternative manner, the neural network further comprises a third convolution module comprising a third feature sub-module and a third identification sub-module; the first recognition module 320 is further configured to input the second feature map to a third feature sub-module for feature extraction, obtain a third feature map, and recognize the third feature map through the third recognition sub-module, so as to obtain a third result vector; the third result vector comprises an element area, a type and a classification probability belonging to the type of each hand-drawn circuit element; the length and width of the images input to the first feature sub-module, the first feature image, the second feature image and the third feature image are sequentially decreased, and the depth is sequentially increased; and respectively comparing the classification probabilities of the hand-drawn circuit elements in the first result vector, the second result vector and the third result vector, and taking the type corresponding to the highest classification probability as the element type of the hand-drawn circuit element.
In an alternative way, at least one of the first, second and third feature sub-modules comprises at least two convolution layers, the output of the previous convolution layer being the input of the next convolution layer in the sub-module comprising at least two convolution layers.
In an alternative, the first feature sub-module comprises six convolutions, the second feature sub-module comprises two convolutions, and the third feature sub-module comprises two convolutions.
In an alternative manner, the first recognition module 320 is further configured to process the hand-drawn circuit image with the area where the hand-drawn circuit image is located as a center according to the size of the input image of the neural network, so as to obtain the hand-drawn circuit image that includes the hand-drawn circuit image and conforms to the size of the input image; inputting the hand-drawn circuit image which accords with the size of the input image into a neural network to obtain the element types of each hand-drawn circuit element in the hand-drawn circuit image and the element areas of each hand-drawn circuit element in the hand-drawn circuit image.
In an optional manner, the recognition device further includes a training module, and the training module is configured to obtain first training data, where the first training data includes a plurality of hand-drawn circuit diagram samples, and the hand-drawn circuit diagram samples include hand-drawn circuit elements; processing the plurality of hand-drawn circuit diagram samples to enable the hand-drawn circuit elements on the hand-drawn circuit diagram samples to perform at least one of rotation, scaling and mirroring operations to obtain second training data; training a preset convolutional neural network by using the first training data and the second training data, and obtaining the neural network after training is completed.
According to the embodiment of the application, the standard circuit diagram is obtained by identifying the hand-drawn circuit diagram through the identification device 300 of the hand-drawn circuit diagram, so that the reaction time of a user can be reduced, and the generation efficiency of the standard circuit diagram is improved; further, the first recognition module 320 and the second recognition module 330 are used for recognizing the hand-drawn circuit element and the hand-drawn connection line respectively, and after the hand-drawn circuit element is recognized by the neural network trained in the first recognition module 320, the second recognition module 330 is used for recognizing the hand-drawn connection line from the rest area except the element area in the hand-drawn circuit image, so that the circuit element is prevented from being recognized as the hand-drawn connection line, and the accuracy of the finally generated standard circuit diagram is improved.
The present application further provides an apparatus for identifying a hand-drawn circuit diagram, please refer to fig. 6, wherein the apparatus 400 for identifying a hand-drawn circuit diagram includes: a processor (processor) 402 and a memory (memory) 404.
The processor 402 is configured to execute the program 406, and may specifically perform the relevant steps in the above-described embodiment of the method for identifying a hand-drawn circuit diagram.
In particular, program 406 may include computer-executable instructions.
The processor 402 may be a central processing unit, CPU, or an application specific integrated circuit, ASIC, or one or more integrated circuits configured to implement embodiments of the present application. The one or more processors included in the recognition device of the hand-drawn circuit diagram may be the same type of processor, such as one or more CPUs; but may also be different types of processors such as one or more CPUs and one or more ASICs.
A memory 404 for storing a program 406. Memory 404 may comprise high-speed RAM memory or may further comprise non-volatile memory (non-volatile memory), such as at least one disk memory.
The application further provides a chip, which is suitable for the recognition equipment of the hand-drawn circuit diagram, wherein an instruction set is stored in the chip, and when the instruction set is executed, the recognition equipment of the hand-drawn circuit diagram can be instructed to realize the operation of the recognition method of the hand-drawn circuit diagram according to any embodiment.
In addition, the application further provides a computer readable storage medium, such as a chip, an optical disc, and the like, on which an execution program is stored, where the execution program is executed to implement the method for identifying a hand-drawn circuit diagram according to any one of the embodiments.
The computer readable storage medium according to the embodiments of the present disclosure is not limited to the above-described embodiments, and may be, for example, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the above. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In an embodiment of the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
Those skilled in the art will appreciate that the above-described preferred embodiments can be freely combined and stacked without conflict. In which the flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures, for example, two blocks shown in succession may in fact be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions. The numbering of the steps herein is for convenience of illustration and reference only and is not intended to limit the order in which the steps are performed, the particular order of execution being determined by the technology itself, and the skilled artisan can determine various allowable, reasonable orders based on the technology itself.
It should be noted that step numbers (letter or number numbers) are used in this application to refer to certain specific method steps for convenience and brevity only, and are in no way intended to limit the order of the method steps by letter or number. It will be apparent to those skilled in the art that the sequence of steps of the relevant method should be determined by the technique itself, should not be unduly limited by the presence of step numbers, and that one skilled in the art can determine various allowable, reasonable sequences of steps based on the technique itself.
Those skilled in the art will appreciate that the above-described preferred embodiments can be freely combined and stacked without conflict.
It should be understood that the above-described embodiments are exemplary only and not limiting, and that various obvious or equivalent modifications and substitutions to the details described above may be made by those skilled in the art without departing from the underlying principles of the present application, and are intended to be included within the scope of the claims of the present application.

Claims (10)

1. A method for identifying a hand-drawn circuit diagram, the method comprising:
the acquisition step: acquiring a hand-drawn circuit image of a hand-drawn circuit diagram, wherein the hand-drawn circuit image comprises hand-drawn circuit elements and hand-drawn connecting wires;
A first identification step: identifying, by a neural network, a component type of each of the hand-drawn circuit components in the hand-drawn circuit image and a component area in the hand-drawn circuit image;
a second identification step: identifying a connection line position of each hand-drawn connection line from a remaining region excluding each element region in the hand-drawn circuit image;
generating: and generating a standard circuit diagram according to the identified element types, the identified element areas and the identified connecting line positions, wherein standard circuit elements in the standard circuit diagram are determined according to the corresponding element types, and standard connecting lines are determined according to the corresponding connecting line positions.
2. The method of claim 1, wherein the neural network comprises a first convolution module and a second convolution module, the first convolution module comprising a first feature sub-module and a first identification sub-module; the second convolution module comprises a second characteristic sub-module and a second identification sub-module; the first identifying step includes:
a first judging sub-step: inputting the hand-drawn circuit image into the first feature sub-module for feature extraction to obtain a first feature image, and identifying the first feature image through the first identification sub-module to obtain a first result vector;
A second judging sub-step: inputting the first feature map to the second feature sub-module for feature extraction to obtain a second feature map, and identifying the second feature map through the second identification sub-module to obtain a second result vector;
the first result vector and the second result vector both comprise an element area, a type and a classification probability belonging to the type of each hand-drawn circuit element; the length and the width of the input image input to the first characteristic sub-module, the first characteristic image and the second characteristic image are sequentially decreased, and the depth is sequentially increased;
the comparison substep: and respectively comparing the classification probabilities of the hand-painted circuit elements in the first result vector and the second result vector, and taking the type corresponding to the highest classification probability as the element type of the hand-painted circuit element.
3. The method of claim 2, wherein the neural network further comprises a third convolution module comprising a third feature sub-module and a third identification sub-module; before the comparing sub-step, the first identifying step further includes:
and a third judging sub-step: inputting the second feature map to the third feature sub-module for feature extraction to obtain a third feature map, and identifying the third feature map by the third identification sub-module to obtain a third result vector;
The third result vector comprises an element area, a type and a classification probability of the type of each hand-drawn circuit element; the length and the width of the input image input to the first characteristic sub-module, the first characteristic image, the second characteristic image and the third characteristic image are sequentially decreased, and the depth is sequentially increased;
in the comparing sub-step: and respectively comparing the classification probabilities of the hand-drawn circuit elements in the first result vector, the second result vector and the third result vector, and taking the type corresponding to the highest classification probability as the element type of the hand-drawn circuit element.
4. A method according to claim 3, wherein at least one of the first, second and third feature sub-modules comprises at least two convolution layers, the output of the last convolution layer being the input of the next convolution layer in the sub-module comprising at least two convolution layers.
5. The method of claim 4, wherein the first feature sub-module comprises six convolutional layers, the second feature sub-module comprises two convolutional layers, and the third feature sub-module comprises two convolutional layers.
6. The method of claim 1, wherein the first identifying step comprises:
processing the hand-drawn circuit image by taking the area where the hand-drawn circuit image is positioned as the center according to the size of the input image of the neural network, so as to obtain the hand-drawn circuit image which comprises the hand-drawn circuit image and accords with the size of the input image;
inputting a hand-drawn circuit image conforming to the size of the input image into the neural network to obtain the element type of each hand-drawn circuit element in the hand-drawn circuit image and the element area of each hand-drawn circuit element in the hand-drawn circuit image.
7. The method according to claim 1, wherein the neural network is trained by:
acquiring first training data, wherein the first training data comprises a plurality of hand-drawn circuit diagram samples, and the hand-drawn circuit diagram samples comprise hand-drawn circuit elements;
processing a plurality of hand-drawn circuit diagram samples, so that the hand-drawn circuit elements on the hand-drawn circuit diagram samples perform at least one of rotation, scaling and mirroring operations to obtain second training data;
training a preset convolutional neural network by using the first training data and the second training data, and obtaining the neural network after training is completed.
8. An apparatus for identifying a hand-drawn circuit diagram, the apparatus comprising:
the device comprises an acquisition module, a display module and a display module, wherein the acquisition module is used for acquiring a hand-painted circuit image of the hand-painted circuit image, and the hand-painted circuit image comprises hand-painted circuit elements and hand-painted connecting wires;
a first identifying module for identifying an element type of each hand-drawn circuit element in the hand-drawn circuit image and an element region thereof in the hand-drawn circuit image through a neural network;
a second identifying module for identifying a connection line position of each hand-drawn connection line from a remaining area excluding each element area in the hand-drawn circuit image;
the generating module is used for generating a standard circuit diagram according to the identified element types, the identified element areas and the identified connecting line positions, wherein standard circuit elements in the standard circuit diagram are determined according to the corresponding element types, and standard connecting lines are determined according to the corresponding connecting line positions.
9. An identification device of a hand-drawn circuit diagram, comprising: a processor and a memory having stored therein executable instructions that are executable by the processor to implement the method of any one of claims 1 to 7.
10. A computer readable storage medium having stored therein at least one executable instruction, wherein the executable instructions when executed implement the method of any one of claims 1 to 7.
CN202311407184.9A 2023-10-27 2023-10-27 Method, device, equipment and storage medium for identifying hand-drawn circuit diagram Pending CN117392698A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202311407184.9A CN117392698A (en) 2023-10-27 2023-10-27 Method, device, equipment and storage medium for identifying hand-drawn circuit diagram

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202311407184.9A CN117392698A (en) 2023-10-27 2023-10-27 Method, device, equipment and storage medium for identifying hand-drawn circuit diagram

Publications (1)

Publication Number Publication Date
CN117392698A true CN117392698A (en) 2024-01-12

Family

ID=89462772

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202311407184.9A Pending CN117392698A (en) 2023-10-27 2023-10-27 Method, device, equipment and storage medium for identifying hand-drawn circuit diagram

Country Status (1)

Country Link
CN (1) CN117392698A (en)

Similar Documents

Publication Publication Date Title
CN110163198B (en) Table identification reconstruction method and device and storage medium
WO2022148192A1 (en) Image processing method, image processing apparatus, and non-transitory storage medium
CN110689037B (en) Method and system for automatic object annotation using deep networks
CN106447721B (en) Image shadow detection method and device
US9235759B2 (en) Detecting text using stroke width based text detection
JP6089886B2 (en) Region dividing method and inspection apparatus
CN109343920B (en) Image processing method and device, equipment and storage medium thereof
US10902053B2 (en) Shape-based graphics search
WO2020133442A1 (en) Text recognition method and terminal device
WO2015074521A1 (en) Devices and methods for positioning based on image detection
CN112101386B (en) Text detection method, device, computer equipment and storage medium
CN111460782A (en) Information processing method, device and equipment
CN113609984A (en) Pointer instrument reading identification method and device and electronic equipment
CN113436222A (en) Image processing method, image processing apparatus, electronic device, and storage medium
JP2016103759A (en) Image processing apparatus, image processing method, and program
CN111738252B (en) Text line detection method, device and computer system in image
CN112132750A (en) Video processing method and device
CN117392698A (en) Method, device, equipment and storage medium for identifying hand-drawn circuit diagram
WO2022125127A1 (en) Detection of image space suitable for overlaying media content
CN112215266A (en) X-ray image contraband detection method based on small sample learning
CN114202665A (en) Image similarity determining method and device, equipment and storage medium
CN113408502B (en) Gesture recognition method and device, storage medium and electronic equipment
WO2023098487A1 (en) Target detection method and apparatus, electronic device, and computer storage medium
CN116797657A (en) Attitude estimation method, electronic device, and storage medium
CN116152844A (en) Curve data extraction method, device, computer equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination