CN117392427A

CN117392427A - Molecular structure identification method, device, electronic equipment and storage medium

Info

Publication number: CN117392427A
Application number: CN202311136951.7A
Authority: CN
Inventors: 胡金水; 吴浩; 陈明军; 刘辰宇; 殷实; 吴嘉嘉; 殷保才; 殷兵; 刘聪
Original assignee: iFlytek Co Ltd
Current assignee: iFlytek Co Ltd
Priority date: 2023-09-04
Filing date: 2023-09-04
Publication date: 2024-01-12

Abstract

The invention provides a molecular structure identification method, a device, electronic equipment and a storage medium, wherein the method comprises the following steps: acquiring a molecular image; initializing an empty angle set, and storing the branch angle into the angle set under the condition of carrying out molecular structure decoding based on the image characteristics of the molecular image and decoding to the branch angle for the first time; taking out one branch angle from the angle set, taking the branch angle as a guide, decoding a molecular structure under the branch angle based on the image characteristics of the molecular image, and updating the angle set based on the decoding to obtain a new branch angle for decoding the molecular structure under the next branch angle until the angle set is empty; and determining a molecular structure corresponding to the molecular image based on the decoding result under each branch angle. The method, the device, the electronic equipment and the storage medium provided by the invention improve the reliability and the accuracy of decoding the molecular structure.

Description

Molecular structure identification method, device, electronic equipment and storage medium

Technical Field

The present invention relates to the field of image recognition technologies, and in particular, to a molecular structure recognition method, a molecular structure recognition device, an electronic device, and a storage medium.

Background

The chemical molecular structure identification can be widely applied to the fields of pharmaceutical research and development, man-machine interaction, biochemistry, education, organic synthesis and the like.

At present, aiming at the research of handwriting chemical molecular structure recognition, a post-processing method based on rules is excessively relied on, so that the complexity of molecular structure recognition is simplified, and the analysis recognition of the handwriting complex molecular layout is difficult to complete under the existing scheme.

Disclosure of Invention

The invention provides a molecular structure identification method, a device, electronic equipment and a storage medium, which are used for solving the defect that the identification of a complex handwritten molecular structure is difficult in the prior art.

The invention provides a molecular structure identification method, which comprises the following steps:

acquiring a molecular image;

initializing an empty angle set, and storing the branch angle into the angle set under the condition of carrying out molecular structure decoding based on the image characteristics of the molecular image and decoding to the branch angle for the first time; taking out one branch angle from the angle set, taking the branch angle as a guide, decoding a molecular structure under the branch angle based on the image characteristics of the molecular image, and updating the angle set based on the decoding to obtain a new branch angle for decoding the molecular structure under the next branch angle until the angle set is empty;

And determining a molecular structure corresponding to the molecular image based on the decoding result under each branch angle.

According to the molecular structure identification method provided by the invention, the updating of the angle set based on the new branch angle obtained by decoding comprises the following steps:

the new branch angles are respectively detected with the existing branch angles in the angle set by chemical bonds;

and updating the angle set based on the detection result of the chemical bond detection.

According to the molecular structure identification method provided by the invention, the updating of the angle set based on the detection result of the chemical bond detection comprises the following steps:

storing the new branch angle into the angle set under the condition that the detection result indicates that a chemical bond does not exist;

and deleting the branch angles forming the chemical bonds with the new branch angles from the angle set when the detection result indicates that the chemical bonds exist.

According to the molecular structure identification method provided by the invention, the molecular structure under the branch angle is decoded based on the image characteristics of the molecular image by taking the branch angle as a guide, and the molecular structure identification method comprises the following steps:

determining a visual context feature of a current decoding moment under the branch angle based on the decoding feature of the branch angle and a decoding state of a previous decoding moment under the branch angle;

And based on the visual context characteristics, performing molecular structure decoding of the current decoding moment on the image characteristics to obtain a decoding state of the current decoding moment, and returning the current decoding moment as the previous decoding moment to be decoded until the decoding under the branch angle is finished.

According to the molecular structure identification method provided by the invention, one branch angle is taken out from an angle set, the branch angle is taken as a guide, a molecular structure under the branch angle is decoded based on the image characteristics of the molecular image, a new branch angle is obtained based on the decoding, the angle set is updated for decoding of the molecular structure under the next branch angle until the angle set is empty, and the method comprises the following steps:

based on an identification model, taking out a branch angle from an angle set, taking the branch angle as a guide, decoding a molecular structure under the branch angle based on the image characteristics of the molecular image, and updating the angle set based on the decoding to obtain a new branch angle for decoding the molecular structure under the next branch angle until the angle set is empty;

the identification model is obtained based on sample images and molecular structure labels corresponding to the sample images through training;

The molecular structure label is obtained by connecting a atomic group and a chemical bond in a molecular formula corresponding to the sample image into a molecular structure diagram and then performing diagram traversal.

According to the molecular structure identification method provided by the invention, the molecular structure label determining step comprises the following steps:

connecting a group of atoms in a molecular formula corresponding to the sample image and a chemical bond to a structure diagram of the atoms;

traversing the molecular structure diagram, and generating the molecular structure label based on the atomic group, the chemical bond, the angle, the nested symbol and the reconnection mark obtained by traversing.

According to the molecular structure recognition method provided by the invention, the training steps of the recognition model comprise:

based on an initial model, carrying out molecular structure recognition on the sample image to obtain a chemical bond detection result between a sample branch angle decoded in the molecular structure recognition process and an existing branch angle in a sample angle set and a structure recognition result of the sample image;

and carrying out parameter iteration on the initial model based on the structure identification result, the molecular structure label, the chemical bond detection and the reconnection mark in the molecular structure label to obtain the identification model.

The invention also provides a molecular structure recognition device, which comprises:

an acquisition unit configured to acquire a molecular image;

the recognition unit is used for initializing an empty angle set and storing the branch angle into the angle set under the condition that the molecular structure is decoded based on the image characteristics of the molecular image and the branch angle is decoded for the first time; taking out one branch angle from the angle set, taking the branch angle as a guide, decoding a molecular structure under the branch angle based on the image characteristics of the molecular image, and updating the angle set based on the decoding to obtain a new branch angle for decoding the molecular structure under the next branch angle until the angle set is empty;

and the output unit is used for determining the molecular structure corresponding to the molecular image based on the decoding result under each branch angle.

The invention also provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the molecular structure identification method as described in any one of the above when executing the program.

The invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements a molecular structure identification method as described in any of the above.

The invention also provides a computer program product comprising a computer program which when executed by a processor implements a molecular structure identification method as described in any one of the above.

The molecular structure identification method, the device, the electronic equipment and the storage medium provided by the invention increase the decoding of each branch angle in the chemical molecular structure and the maintenance mechanism of the branch angle to be explored, and take the branch angle in the chemical molecular structure as the guiding condition during decoding, so as to enrich the information of decoding of the molecular structure, improve the reliability of decoding of the molecular structure and improve the decoding accuracy facing to the complex chemical molecular structure.

Drawings

In order to more clearly illustrate the invention or the technical solutions of the prior art, the following description will briefly explain the drawings used in the embodiments or the description of the prior art, and it is obvious that the drawings in the following description are some embodiments of the invention, and other drawings can be obtained according to the drawings without inventive effort for a person skilled in the art.

FIG. 1 is a schematic flow chart of a molecular structure identification method provided by the invention;

FIG. 2 is a schematic diagram of decoding of molecular structures provided by the present invention;

FIG. 3 is a handwritten molecular image provided by the present invention;

FIG. 4 is a molecular structural diagram provided by the present invention;

FIG. 5 is a molecular structure tag provided by the present invention;

FIG. 6 is a schematic diagram of a molecular structure recognition device according to the present invention;

fig. 7 is a schematic structural diagram of an electronic device provided by the present invention.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the present invention more apparent, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is apparent that the described embodiments are some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

In recent years, due to advances in the computer vision technology in object detection and semantic segmentation, satisfactory recognition performance has been achieved for simply controllable printed molecular images. Meanwhile, recognition research oriented to handwriting chemical molecular structures has made great progress. However, these recognition methods for handwriting-oriented chemical molecular structures may ignore the importance of recognition of handwriting molecular images in human-computer interaction, or excessively simplify the complexity of molecular structure recognition, and excessively rely on rule-based post-processing methods, resulting in recognition of large and complex molecular structures remains a major challenge.

Fig. 1 is a schematic flow chart of a molecular structure identification method provided by the present invention, as shown in fig. 1, the method includes:

step 110, a molecular image is acquired.

The molecular image here, i.e. the image in which the chemical molecular structure is plotted. Further, the chemical molecular structure in the molecular image may be printed or handwritten, which is not particularly limited in the embodiment of the present invention.

Step 120, initializing an empty angle set, and storing the branch angle into the angle set under the condition of decoding a molecular structure based on the image characteristics of the molecular image and decoding the branch angle for the first time; and taking out one branch angle from the angle set, taking the branch angle as a guide, decoding a molecular structure under the branch angle based on the image characteristics of the molecular image, and updating the angle set based on the decoding to obtain a new branch angle for decoding the molecular structure under the next branch angle until the angle set is empty.

In consideration of the complexity of the chemical molecular structure, when molecular structure identification is performed on a molecular image, the angle decoding and maintenance mechanism of each branch in the chemical molecular structure can be increased under the framework of a conventional encoder and a conventional decoder, and the branch angle in the chemical molecular structure is used as a guiding condition during decoding, so that the reliability and accuracy of decoding for the complex chemical molecular structure are improved.

In a specific implementation, the decoding may be performed branch by branch based on image features of the molecular image. It will be appreciated that here the branch-by-branch decoding, i.e. one graph traversal for a molecular image, is performed.

Fig. 2 is a schematic diagram of decoding a molecular structure according to the present invention, as shown in fig. 2, in fig. 2 (a), an empty angle set Memory may be initialized at the beginning of the graph traversal, and in the traversal process, a branching point is encountered, where two branching angles, namely, a branching angle (1) and a branching angle (2), may be decoded, where the branching angle (1) points upward, the branching angle (2) points downward and the branching angle (1) and the branching angle (2) are stored as the first decoded branching angle in the angle set Memory.

Subsequently, in fig. 2 (b), the branch angle (1) is taken out of the angle set Memory, traversed up along the branch angle (1) until a new branch point is encountered. In fig. 2 (c), two branch angles, namely, branch angle (3) and branch angle (4), are decoded, wherein branch angle (3) points to the right and branch angle (4) points to the bottom, and both branch angle (3) and branch angle (4) are stored in an angle set Memory. Next, in fig. 2 (d), the branch angles (1), (3), (5), and (7) are sequentially taken out from the angle set Memory, and the graph traversal is performed until one branch traversal is completed.

After the traversal of one branch has been completed in fig. 2 (e), the branch angle (6) is fetched from the angle set Memory, and the traversal is continued along the branch angle (6), where the traversed atom is found to be connected to the atom previously accessed, i.e. the atom marked as two filled circles in fig. 2 (e). In this case, in fig. 2 (f), it can be predicted whether the branch angle (8) already stored in the angle set Memory is a loop key, and if so, it can be considered that the branch angle (8) has completed the traversal, and the deletion is taken out from the angle set Memory. The loop key of this time is understood as whether the two branch angles form a reconnection relationship (reconnection), i.e. a loop forming a branch angle.

Based on the above traversal mode, after completing the decoding traversal of the branches under all the branch angles in the angle set Memory, the graph traversal for the molecular image can be considered to be completed.

It will be appreciated that the angle set is a set for storing the branch angles to be traversed, and that at the beginning of the graph traversal, an angle set may be initialized first, where the angle set is empty, i.e., at the beginning of the graph traversal, there are no branch angles to be traversed in the angle set. In the process of graph traversal, the first traversal reaches a branching point, namely when the first decoding is performed on the branch angles, the first decoded branch angles can be stored in an angle set, at the moment, the angle set is not empty any more, the branch angles to be traversed are stored in the angle set, and the subsequent traversal is performed again, and the branch angles can be taken out from the angle set for exploration. Therefore, the angle set is used for storing and maintaining the branch angles to be traversed, and in the decoding process of the branch angles, the angle set can be correspondingly updated and adjusted based on the decoded new branch angles so as to facilitate the decoding of the next branch angle.

Further, in decoding for any branch angle, the branch angle needs to be taken out from the angle set first, that is, the branch angle is confirmed to be the branch angle being explored or being explored, so that the angle set can always maintain the branch angle which is not explored.

After the branch angle is taken out, the molecular structure under the branch angle can be decoded, and the molecular structure can be decoded along the branch angle during decoding, namely, the branch angle is used as condition information to guide the decoding of the molecular structure, so that the information of the decoding of the molecular structure is enriched, and the reliability of the decoding of the molecular structure is improved.

In the process of decoding the molecular structure under the branch angle, if a new branch angle is decoded, the angle set may be updated based on the new branch angle, and specifically, the new branch angle may be added to the angle set as a branch angle that has not been explored, so as to be decoded later. In addition, before the new branch angle is added to the angle set as the branch angle which is not yet explored, the relationship between the new branch angle and each existing branch angle in the angle set can be judged, namely, whether the new branch angle and the existing branch angle form a reconnection relationship in the molecular structure or not is judged, if the reconnection relationship is formed, the new branch angle and the existing branch angle forming the reconnection relationship with the new branch angle can be considered to be explored, the new branch angle does not need to be stored in the angle set, and the existing branch angle forming the reconnection relationship with the new branch angle can also be deleted from the angle set, so that the angle set can always maintain the branch angle which is not yet explored.

And 130, determining a molecular structure corresponding to the molecular image based on the decoding result under each branch angle.

Specifically, after decoding is performed for the molecular structure at each branch angle in the molecular image, the decoding result at each branch angle can be obtained. The decoding results herein may include chemical bonds, angles, population of atoms, etc. at the branching angle. On the basis, the molecular structure corresponding to the molecular image can be obtained by combining the decoding results under each branch angle, wherein the molecular structure can be a character string recorded based on a markup language of an organic chemical structural formula, a representation diagram of the molecular structure or a vector representation of the molecular structure under a specific coding language, and the embodiment of the invention is not particularly limited.

The method provided by the embodiment of the invention increases the decoding of each branch angle in the chemical molecular structure and the maintenance mechanism of the branch angle to be explored, takes the branch angle in the chemical molecular structure as the guiding condition during decoding, enriches the information of decoding the molecular structure, improves the reliability of decoding the molecular structure, and improves the decoding accuracy facing the complex chemical molecular structure.

Based on the above embodiment, in step 120, the updating the angle set based on the new branch angle obtained by decoding includes:

Specifically, considering that there may be a reconnection relationship in the molecular structure, for example, single bonds of 6 in the benzene ring are connected end to end, if it is not considered whether a chemical bond is formed between a new branch angle obtained by decoding and a branch angle which is not yet explored in the angle set, repeated traversal is likely to be caused, that is, the molecular structures under the two branch angles are decoded respectively, so that the molecular structure under one branch should be decoded into two molecular structures by mistake, and the accuracy of molecular structure identification is affected.

Therefore, after the new branch angles are obtained by decoding, the new branch angles can be respectively subjected to chemical bond detection with the existing branch angles in the angle set. Here, for any one of the new branch angle and the angle set, it may be detected whether or not a chemical bond is formed between the two branch angles, and the detection result obtained may include whether or not a chemical bond is present, and may further include the category of the formed chemical bond, such as a single bond, a double bond, or the like, in the case where it is determined that the chemical bond is included.

After the new branch angle and the detection result of the chemical bond detection between the existing branch angles in the angle set are obtained, the angle set can be updated based on this. It can be understood that if no chemical bond exists between the new branch angle and each existing branch angle, which indicates that the reconnection relationship is not decoded temporarily, the new branch angle can be put into the angle set as the branch angle which is not yet explored; if a chemical bond exists between the new branch angle and one of the existing branch angles, indicating that the reconnection relation is decoded, deleting the existing branch angle with the chemical bond exists between the new branch angle from the angle set, and not storing the new branch angle into the angle set.

Based on any of the above embodiments, in step 120, the updating the angle set based on the detection result of the chemical bond detection includes:

Specifically, for a new branch angle, in the detection results of chemical bond detection between the branch angle and each existing branch angle, if all the detection results indicate that no chemical bond exists, that is, no chemical bond exists between the new branch angle and each existing branch angle, the new branch angle needs to be explored, and at the moment, the new branch angle can be stored in the angle set, so that the angle set can continuously maintain all the branch angles which are not explored;

if a detection result indicates that a chemical bond exists, that is, a chemical bond exists between a new branch angle and an existing branch angle in the angle set, and a reconnection relationship exists between the new branch angle and the existing branch angle, the new branch angle and the existing branch angle forming the reconnection relationship with the new branch angle can be considered to be completely explored, the new branch angle does not need to be stored in the angle set, and the existing branch angle forming the reconnection relationship with the new branch angle can also be deleted from the angle set, so that the angle set can always maintain the branch angle which is not explored.

Based on any of the foregoing embodiments, in step 120, the decoding the molecular structure at the branching angle based on the image feature of the molecular image with the branching angle as a guide includes:

Specifically, in decoding a molecular structure at one branch angle, the branch angle may be used as a guide condition for decoding the molecular structure. Upon encountering a branch, decoding may continue along the direction of the branch angle taken from the angle set, and the branch angle applied at that time directs the generation of visual context features when decoding for the molecular structure at that branch angle.

That is, when generating the visual context feature of one decoding time, the visual context feature of the decoding time at that time is determined in comparison with the prior art in which only the decoding state of the previous decoding time is applied, and the decoding feature of the branching angle is additionally considered. It is to be understood that, here, the decoding characteristics of the branch angle are stored when the branch angle is decoded in the decoding process of the molecular structure, and the decoding characteristics of the branch angle may include attention weight information and status characteristics of the branch angle.

For example, the calculation of the visual context characteristics for the current decoding moment can be expressed as the following formula:

in the formula e _t,i Energy, W, W, of the ith position in the feature map at the t decoding time ^x 、W ^y 、W ^S 、W ^a 、W ^sp 、W ^ap Are projection parameters of the attention module. X is x _i Is the image feature of the molecular image at the ith position,s is the decoding result of the t-1 st decoding time _t-1 A is the decoding status of the t-1 st decoding moment _t-1 、a _t Attention weights, a, for the t-1 st decoding time and the t decoding time, respectively _b 、s _b Attention weight information and status characteristics at the time of branch angle decoding, respectively. When decoding is performed without a division angle, a _b Sum s _b All are zero vectors, at this time e _t,i E in the calculation formula of (2) and the visual context characteristic acquisition mode in the conventional character string decoder _t,i Is consistent with the calculation formula of (2).

a _t,i The attention weight of the ith decoding moment at the ith position, and h and w are the height and width of the molecular image respectively; c _t Is a visual context feature at the t-th decoding moment.

After obtaining the visual context feature of the current decoding moment, decoding the molecular structure of the current decoding moment based on the visual context feature, thereby obtaining the decoding state of the current decoding moment, which can be expressed as the following formula:

p _t ＝softmax(W ^c s _t )；

Wherein s is _te I.e. the decoding status at the current decoding moment, the GRU represents the computational unit of the recurrent neural network. P is p _te Representing probability distribution of decoding result outputted at current decoding time, W ^c Is a parameter of the classification layer.

It can be understood that after the decoding of the molecular structure at the current decoding time is completed, the decoding state at the current decoding time can be used as the decoding state at the previous decoding time, and the decoding of the molecular structure at the next decoding time can be performed. Also, in this process, at different times when decoding the molecular structure at one branch angle, the decoding feature of the same branch angle is applied.

In the embodiment of the invention, the neural network structure for realizing the decoding of the molecular structure is recorded as a random condition guided decoder. The random conditional pilot decoder is significantly superior to the conventional string decoder in terms of generalization of the complex structure without adding significant additional parameters or computational burden, and the accuracy of multipath decoding can be effectively improved by adopting the conditional pilot mechanism and path selection.

Based on any of the above embodiments, step 120 includes:

Specifically, step 120 may be implemented by a pre-trained recognition model. Here, the recognition model may be in the form of an encoder-decoder. Training for the recognition model may be achieved by supervised learning. It can be understood that the sample with supervised learning is a sample image drawn with chemical molecular structures, and the label with supervised learning is a molecular structure label corresponding to the sample image, where the molecular structure label reflects the molecular structure in a language form.

The conventional markup language for organic chemical formulas may be Chemfig, SMILES, etc. Compared with the most commonly used structure mark language SMILES, chemfig requires little abstract chemical knowledge, and the design purpose is to improve the correlation and accuracy by utilizing the chemical structure graph. The syntax of Chemfig also provides a description of "angles" that can be used to specify the orientation of bonds in a molecule. In the handwritten molecular image shown in fig. 3, the atom "N" is connected to two double bonds, one pointing in the upward right direction and the other pointing in the downward right direction. The corner codes in Chemfig syntax, such as "[1]" and "[ -1]", can be used to represent the approximate orientation of these two double bonds, whereby the visual appearance of the molecule appears more accurate and comprehensive.

However, if the original Chemfig markup language is directly used as a tag for recognition model training, the following problems still remain:

in one, chemfig has grammatical ambiguity: since the starting point and the traversal order are different, there are multiple correct Chemfig sequences that can represent the same molecular image. Such ambiguity is unavoidable and increases the difficulty of model learning.

Secondly, chemfig syntax has complexity: a priori rules and domain knowledge need to be incorporated into the grammar to preserve its simplicity. For example, rules such as grouping atoms using brackets, and the need to delete redundant chemical symbols and price dashes to reduce grammar complexity. Furthermore, knowledge of the domain like IUPAC (International Union of Pure and Applied Chemistry )) naming rules is necessary to ensure that the compound structure is correctly labelled. The use of these rules and domain knowledge increases the complexity of Chemfig syntax. For example, the benzene ring is simply denoted as "×6 (- -)", and the direction of the six single bonds needs to be automatically deduced according to the rules of the benzene ring. These prior rules are complex and increase the learning difficulty of the model.

In order to overcome the above problems, in the embodiment of the present invention, it is proposed to analyze a population of atoms and chemical bonds in a molecular formula corresponding to a sample image, and connect the population of atoms and chemical bonds to form a molecular structure diagram, and then perform diagram traversal to obtain a molecular structure label.

Here, considering that the molecular formulas corresponding to the sample image may be represented by different character strings, in order to solve the ambiguity on the grammar level, in the embodiment of the present invention, the atomic groups and chemical bonds are manually parsed and recovered from the character strings of the molecular formulas, and then all the obtained atomic groups and chemical bonds are connected to generate a graph, which is herein referred to as a molecular structure diagram.

For example, "HO" - - (-COOH) - - -) "," COOH- [4] - (-OH) - - -) "and" - [ 6 ] ([: 30] - (-OH) - - (-COOH) - -) "are different character string representations of the same formula, and by resolving and recovering the subgroup and the chemical bond, a molecular structure diagram as shown in fig. 4 can be constructed, so that the uniformity of the representation is achieved in the form of the structure diagram.

In fig. 4, the character strings are represented as atom groups such as "HO", "COOH". The ring-shaped benzene molecular structure is regarded as a special group of atoms. The lines between the population of atoms represent chemical bonds and may be single bond "-", double bond "=" or triple bond. One group of atoms may be linked to another group of atoms by a single or multiple chemical bonds. By using the atomic group as the vertex and the chemical bond as the side, a molecular structure diagram can be obtained, whereby the graphic representation of the same molecule is identical even if different Chemfig tag strings are used, thereby eliminating possible ambiguity in the tag.

The molecular structure diagram obtained by the method is a complex data structure, and the molecular structure diagram is converted into a label form suitable for model training through diagram traversal. Here, the molecular structure diagram is specifically required to be converted into an equivalent one-dimensional text representation through graph traversal, where the conversion rule may be a fixed rule, specifically, traversing the molecular structure diagram from a specific point, and attaching the traversed node (atom group) and edge (chemical bond) to the end of the character string, thereby obtaining a molecular structure label that is concise and free from grammatical ambiguity.

According to the method provided by the embodiment of the invention, the atomic groups and the chemical bonds in the molecular formula are connected into the molecular structure diagram, and then the diagram traversal is carried out, so that a simple molecular structure label which does not have grammar ambiguity is obtained, the difficulty of supervised learning of the recognition model can be reduced, and the reliability of molecular structure recognition based on the recognition model is improved.

Based on any of the above embodiments, the step of determining the molecular structure tag includes:

Specifically, the molecular structure diagram obtained by molecular structural analysis and connection is a complex data structure, which must be converted into a form suitable for model training. For the recognition model of the encoder-decoder structure, the form suitable for training is typically a one-dimensional string or a standardized tag. Standardized labels involve converting a molecular structure diagram into an equivalent one-dimensional text representation.

In order to ensure that the rules of the converted molecular structure labels remain consistent and are not ambiguous, the specific rules may be preset fixed rules. For the traversal of the molecular structure diagram, the molecular structure diagram may be traversed starting from a specific point, with nodes (atom clusters) and edges (chemical bonds) appended to the end of the string. And after the traversing is finished, obtaining the standardized label. For example, traversing the molecular structure diagram shown in FIG. 4, one can get the following standardized tags: HO- [:0]? [a] (- [:60] - [:0] - [:300] (- [:0] COOH) - [:240] - [:180]

Wherein the cells are separated by spaces. The graph traversal may use Depth-First Search (DFS). The above standardized tags may contain four classes of elements, namely "atom group", "chemical bond", "angle", "nested symbol" and "reconnection label". Specifically, in the standardized label, "HO" represents a group of atoms, "? [a] "? [ a, { - } ] "is a reconnection label," - "represents a chemical bond," [:0] "represents an angle," ("represents a nesting symbol).

On this basis, the standardized tag can be integrated, thereby obtaining a molecular structure tag specific to the structure. Wherein if the atomic group does not contain any characters, it may be omitted. While the entity characters may be represented using a LaTeX representation. Since the "angle" and the "chemical bond" are closely related in visual information, a corresponding angle can be appended to each chemical bond in the standardized label as "[: ]", to represent a "bond angle" modeling unit. The angle values may be calculated from the format during the graphic construction and output process. Further, nested branches may be represented using the nested symbol "()". Furthermore, the original Chemfig syntax can be preserved for reconnection labeling, by "? [a] "representation, the symbol represents quilt"? [ a, - ] "represents a unit surrounded by a cyclic unit. When a reconnection occurs, the reconstructed angle value is not represented. Finally, it is also possible to indicate a specific atom in benzene with "- [: ]", and construct a virtual bond angle unit "- [: ]", to indicate the connection between "Μ" and any atom on the ring.

For example, for the standardized tags shown above, the molecular structure tags shown in FIG. 5 can be integrally formed, with nested relationships at different angles, as well as reconnection relationships, marked in the molecular structure tags shown in FIG. 5. Wherein, decoding under a branching angle is completed by "\ eob", a nesting relationship is represented by a line with an arrow between "\angle [:60]" and "- [:60]", and a reconnection mark is represented by a double arrow line between "\angle [:300]" and "\angle [:120 ]".

According to the method provided by the embodiment of the invention, the atomic group and the chemical bond in the molecular formula are connected into the component structure diagram, and the molecular structure diagram is traversed to form the molecular structure label, so that the formed molecular structure label has the advantage of higher consistency with the image compared with the mainstream language SMILES, and is more beneficial to model learning. Furthermore, the syntax of the molecular structure tag is not limited by chemical knowledge, which enables it to represent erroneous or non-existent molecular structures and makes them more versatile. The method provides an innovative method for the characterization of the molecular structure, and has potential application prospects in various fields of chemical structure identification and analysis.

Based on any of the above embodiments, the training step of the recognition model comprises:

In particular, for identifying a model, an initial model of the encoder+decoder structure may be obtained, where model parameters of the initial model may be initialized. In the training process, molecular structure identification can be performed on a sample image based on an initial model, specifically, image feature extraction can be performed on the sample image based on the initial model, then a sample branch angle is taken out of a sample angle set, the sample branch angle is taken as a guide, the molecular structure under the sample branch angle is decoded based on the image feature of the sample image, and a new sample branch angle is obtained based on the decoding, so that the sample angle set is updated for decoding the molecular structure under the next sample branch angle until the sample angle set is empty.

It can be understood that when the initial model is decoded to obtain a new sample branch angle, the sample angle set needs to be updated based on the new sample branch angle obtained by decoding, and when the sample angle set is updated, the new sample branch angle needs to be detected by chemical bonds with each existing branch angle in the sample angle set, so that a chemical bond detection result between the sample branch angle and the existing branch angles in the sample angle set can be obtained.

Here, the presence or absence of a chemical bond in the chemical bond detection result is used to reflect whether or not there is a reconnection relationship between the two sample branch angles. The chemical bond detection result can be compared with the reconnection marks in the molecular structure label, so that the training loss of the initial model is measured from the detection angle of the reconnection relation.

In addition, the structure recognition result and the molecular structure label can be compared, so that the training loss of the initial model is measured from the angle of molecular structure recognition.

Further, the loss L of molecular structure recognition can be expressed based on the following formula _ce ：

In the method, in the process of the invention,L _ce for cross entropy loss, V represents the number of model unit characters, y _ti Whether the molecular structure label is the ith character at the t decoding moment or not, p _ti The probability that the structure recognition result is the ith character at the t decoding time is represented.

The detection loss L of the reconnection relation can be expressed based on the following formula _bc I.e. losses caused by cycling:

q _tb ＝softmax(W ^m s _b +W ^o s _t )；

wherein s is _t And decoding the obtained branch angle for the current decoding time t. W (W) ^m And W is ^o Are all parameters for chemical bond detection. State features s stored in the angle set for existing branch angles b _b The probability distribution of chemical bond detection with the branch angle t is expressed as Where N is the total number of types of chemical bonds, n+1 includes N types of chemical bonds and where no chemical bonds are present.

N is the total number of branch angles in the angle set, z _tbi Detecting whether the chemical bond type between the molecular structure label and the branch angle b is i and q at the t decoding moment _tb The probability that the type of chemical bond between the detection result of the chemical bond and the branch angle b is i is detected at t decoding moments is represented.

Combining the two losses can obtain the overall loss for training the recognition model, wherein the overall loss L can be expressed as L _ce +L _bc May also be denoted as L _ce And L _bc The result of the weighted summation is not particularly limited in this regard by the embodiments of the present invention. After the overall loss is obtained, parameter iteration can be performed on the initial model based on the overall loss, so that a completely trained recognition model is obtained.

The method provided by the embodiment of the invention realizes molecular structure identification oriented to the molecular image through end-to-end modeling and training.

Based on any of the above embodiments, the initial model is the structure of the encoder+decoder. The encoder is DenseNet and comprises three dense blocks for converting an input RGB three-channel image into high-dimensional features. The growth rate and depth of each dense block were set to 24 and 32, respectively. Furthermore, both the encoder and decoder employ a GRU with a hidden state dimension of 256 as the cyclic unit of the RNN, with the dimension of the attention projection parameter set to 128. In addition, the dimension of the embedding parameter is set to 256, and a 15% drop-out rate is applied. For the decoder, the projection parameter dimension of the chemical bond detection class is set to 256.

The optimizer applied for parameter iteration of the initial model may be Adam, the initial learning rate is 2e-4, the learning rate attenuation strategy is multi-step attenuation, the learning rate is adjusted by using multi steplr of Pytorch, and the attenuation factor gamma is set to 0.5.

Based on any one of the above embodiments, the embodiment of the present invention provides a molecular structure identification method, where the molecular structure identification method is implemented based on an identification model. The recognition model here includes a random condition decoder, different from the string decoder, which combines three mechanisms to solve the problems in the prior art: the conditions direct attention, circulation and path selection.

Wherein the conditional directed attention mechanism can utilize the natural graph structure of the molecular structural formula, and consider the identification process thereof as a graph traversal problem. When identifying a model traversal map, a number of branch angles are encountered, which follow a fixed counter-clockwise direction in the order in the proposed modeling unit. However, if decoding is performed only according to a fixed angular order, the model may forget which angular units have not been decoded due to the extension of the late decoding time. To solve this problem, under the conditional directing attention mechanism, the decoding process may be directed using the branch angle as conditional information. Decoding continues along the specified branch angle direction as the recognition model encounters the branch. The updated decoder no longer uses "(" and ")" to indicate the start and end of the branches. Instead, the branch angle of each branch, i.e., "\angle [: < angle value > ]", is first predicted, and the decoded features of each branch angle obtained by decoding are stored separately to an angle set Memory, i.e., the context and attention weight information of each branch angle.

Then in the decoding process for the molecular structure at one branch angle, when "\ eob" is decoded, it indicates that no additional branch angle needs to be predicted, and the decoding feature of one branch angle can be selected from the angle set Memory as a condition for continuing the decoding process.

Furthermore, the main difference between the graph structure and the tree structure is that the graph structure has a cyclic characteristic. In the molecular structure identification method provided by the embodiment of the invention, the branch angles corresponding to the loops are already stored in the angle set Memory and are not yet decoded. Therefore, a simple multi-label classification module can be constructed for chemical bond detection to determine the direction corresponding to the loop angle, and meanwhile, the types of the loop bonds (such as single bonds, double bonds and the like) are classified. The result of the chemical bond detection may be either a null or a bond type, the null representing none of the loops having a corresponding branch angle that has not yet been decoded. If the detection result has a cyclic chemical bond, deleting the corresponding branch angle from the angle set Memory, and not storing the branch angle decoded at the current moment. In contrast, if there is no loop, i.e. the detection result is null, the branch angle at the current moment is saved in the angle set Memory for future decoding selection.

Finally, the identification process of the random condition decoder may be regarded as a graph traversal problem, wherein different traversals of the graph may produce multiple target sequences. Thus, a single graph may have multiple training labels. Although the conditional directed attention mechanism still decodes according to a fixed counterclockwise order, it can lead to over-fitting and recognition errors in complex or unusual structures. Therefore, the embodiment of the invention can decode by combining a path selection mechanism, wherein the mechanism randomly selects different paths in the training process so as to improve the alignment between visual information and decoded characters by using a conditional directed attention mechanism. During the inference process, the recognition model may be caused to attempt to decode all branch angles stored in the angle set and to participate in calculating the beam search path score PK, thereby selecting the path with the highest score to continue decoding.

By the mode, the recognition model can have better universality and recognition efficiency.

And, for training of the recognition model, the embodiment of the invention establishes a handwriting data set to be used as a sample image by taking the publicly accessed CASIA-CSDB data set as a reference. In particular, the real world handwritten molecular structure may be derived from educational environments. The data set includes a number of instances of structural data with writing errors and absence.

Further, the handwritten data set consists of 52,987 handwritten molecular structure images collected in an educational scene. These images are obtained using various devices such as cameras, scanners, and screens, and are labeled as native Chemfig strings. And, there is a mixture of molecular structure and regular molecular formula in the dataset. In addition to isolated molecular structures, many examples in the dataset also contain combinations of formulas and molecular structures. In addition, the dataset integrates different structural writing styles, such as with/without abbreviations, type of kegler ring representation of benzene, and inclusion/exclusion of hydrogen atoms, etc. The dataset also contains a large number of instances of structures that violate the principles of chemistry, even where human writing errors do not exist, and recognition of such structures based on the model trained therefrom can potentially be applied to correct and modify handwritten answers. In addition, to examine the generalization performance of the model, the molecular structure in the dataset needs to maintain a certain level of complexity, specifically the sum of atoms and bonds can be used to evaluate the molecular structure complexity level, with a complexity level of about 10% of the dataset exceeding that of the most complex sample in the conventional training set.

Based on any of the above embodiments, after the recognition model training is completed, experimental tests may be performed on the recognition model.

Specifically, considering that the molecular structure tag itself is sufficiently compact, it is possible to directly compare whether the character string predicted to obtain the recognition result and the character string of the tag are identical. The character string of the tag and the character string of the recognition result may both be converted into a molecular structural diagram, and then the molecular structural diagrams of the tag and the recognition result are compared for matching. Assuming that T is the number of samples and R represents the number of recognition results that Match the tag, an Exact Match (EM) score can be calculated using the following formula:

furthermore, a single image may contain multiple molecular structures, considering that a mixture of formulas and molecular structures exist in the handwritten data set. Therefore, two auxiliary indexes are defined in the embodiment of the invention as follows:

structural exact match (Structure EM): for a sample containing a mixed molecular structure and a regular formula, when all molecular structure recognition results are matched with the marker pattern, the structure in the sample is determined to be correctly recognized. Let T be the number of samples, R _struct Representing the number of correctly identified structural samples, the resulting structural EM score is as follows:

Structural EM fraction EM _struct And a single EM score EM can measure recognition performance of the recognition model on the molecular structure in the handwriting dataset in the mixed mode.

From the experimental results, the molecular structure recognition method provided by the embodiment of the invention has the advantage that compared with the recognition method based on SMILES, the recognition performance is obviously improved. This is mainly due to the reduced ambiguity of molecular structure tags and the stronger consistency between images and tags. In addition, the random condition decoder provided by the embodiment of the invention has obvious advantages over a character string decoder. It should be emphasized that the calculation and parameter costs of the random conditional decoder are almost the same as the string decoder.

Based on any of the above embodiments, fig. 6 is a schematic structural diagram of a molecular structure identification device according to the present invention, as shown in fig. 6, the device includes:

an acquisition unit 610 for acquiring a molecular image;

an identifying unit 620, configured to initialize an empty angle set, and store a branch angle into the angle set when decoding a molecular structure based on an image feature of the molecular image and decoding the branch angle for the first time; taking out one branch angle from the angle set, taking the branch angle as a guide, decoding a molecular structure under the branch angle based on the image characteristics of the molecular image, and updating the angle set based on the decoding to obtain a new branch angle for decoding the molecular structure under the next branch angle until the angle set is empty;

And an output unit 630, configured to determine a molecular structure corresponding to the molecular image based on the decoding result at each branch angle.

The device provided by the embodiment of the invention increases the decoding of each branch angle in the chemical molecular structure and the maintenance mechanism of the branch angle to be explored, takes the branch angle in the chemical molecular structure as the guiding condition during decoding, enriches the information of decoding the molecular structure, improves the reliability of decoding the molecular structure, and improves the decoding accuracy facing the complex chemical molecular structure.

Based on any of the above embodiments, the identifying unit 620 includes a set updating unit configured to:

Based on any of the foregoing embodiments, the set updating unit is specifically configured to:

Based on any of the above embodiments, the identifying unit 620 includes a branch decoding unit for:

Based on any of the above embodiments, the identifying unit 620 is specifically configured to:

Based on any of the above embodiments, the apparatus further includes a tag conversion unit configured to:

Based on any of the above embodiments, the apparatus further comprises a model training unit configured to:

Fig. 7 illustrates a physical schematic diagram of an electronic device, as shown in fig. 7, which may include: processor 710, communication interface (Communications Interface) 720, memory 730, and communication bus 740, wherein processor 710, communication interface 720, memory 730 communicate with each other via communication bus 740. Processor 710 may invoke logic instructions in memory 730 to perform a molecular structure identification method comprising:

Acquiring a molecular image;

Further, the logic instructions in the memory 730 described above may be implemented in the form of software functional units and may be stored in a computer readable storage medium when sold or used as a stand alone product. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.

In another aspect, the present invention also provides a computer program product comprising a computer program, the computer program being storable on a non-transitory computer readable storage medium, the computer program, when executed by a processor, being capable of performing the molecular structure recognition method provided by the above methods, the method comprising:

acquiring a molecular image;

In yet another aspect, the present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, is implemented to perform the molecular structure identification method provided by the above methods, the method comprising:

Acquiring a molecular image;

The apparatus embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.

From the above description of the embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus necessary general hardware platforms, or of course may be implemented by means of hardware. Based on this understanding, the foregoing technical solution may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a computer readable storage medium, such as ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method described in the respective embodiments or some parts of the embodiments.

Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A molecular structure identification method, comprising:

acquiring a molecular image;

2. The molecular structure identification method according to claim 1, wherein updating the angle set based on the decoding to obtain a new branch angle comprises:

3. The method according to claim 2, wherein updating the angle set based on the detection result of the chemical bond detection comprises:

4. The molecular structure identification method according to claim 1, wherein the decoding the molecular structure at the branching angle based on the image feature of the molecular image guided by the branching angle includes:

5. The method according to any one of claims 1 to 4, wherein the extracting a branch angle from the angle set, guided by the branch angle, decodes a molecular structure at the branch angle based on an image feature of the molecular image, and updates the angle set for decoding a molecular structure at a next branch angle based on the decoded new branch angle until the angle set is empty, includes:

based on an identification model, taking out a branch angle from the angle set, taking the branch angle as a guide, decoding a molecular structure under the branch angle based on the image characteristics of the molecular image, and updating the angle set based on the decoding to obtain a new branch angle for decoding the molecular structure under the next branch angle until the angle set is empty;

6. The method of claim 5, wherein the step of determining the molecular structure tag comprises:

7. The method of claim 5, wherein the training step of the recognition model comprises:

8. A molecular structure recognition device, comprising:

an acquisition unit configured to acquire a molecular image;

9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the molecular structure identification method of any one of claims 1 to 7 when the program is executed by the processor.

10. A non-transitory computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when executed by a processor, implements the molecular structure identification method according to any one of claims 1 to 7.