CN114529917A - Zero-sample Chinese single character recognition method, system, device and storage medium - Google Patents

Abstract

The invention discloses a zero-sample Chinese single character recognition method, a system, a device and a storage medium, wherein the method comprises the following steps: extracting visual characteristics of the Chinese single character image; the method comprises the steps of performing learnable class coding on Chinese single character classes, decomposing the component structure of the Chinese single characters, and calculating to obtain learnable class coding; mapping the category codes of the Chinese single characters into a visual space, and constraining semantic consistency of the category codes before and after mapping through a reconstruction loss function; matching the category code of the Chinese single character with the visual characteristics of the image through a transform-based decoder, acquiring the characteristics related to the category code from the visual characteristics of the image, and finally decoding and outputting the recognition result of the Chinese single character. The invention realizes the Chinese single character recognition of zero samples by a learnable category coding method, and solves the problem that the traditional Chinese single character recognition method depends on a large amount of labeled data. The invention can be widely applied to the technical field of pattern recognition and artificial intelligence.

Description

A zero-sample Chinese word recognition method, system, device and storage medium

technical field

The invention relates to the technical field of pattern recognition and artificial intelligence, and in particular, to a zero-sample Chinese word recognition method, system, device and storage medium.

Background technique

Chinese is one of the oldest languages in the world, with a history of thousands of years. The study of Chinese word recognition is of great value and significance to the electronic preservation of ancient books. The current Chinese word recognition methods mainly rely on data-driven deep learning methods, which need to label a large number of training samples. However, there are a huge number of Chinese categories. According to the GB18030-2005 standard, there are 70,224 Chinese words. It is a difficult and time-consuming problem to label sufficient data for each Chinese word. In recent years, some related works try to solve the above problems by adopting component decoding-based or component-coding-based zero-sample recognition methods. However, the component-based decoding method requires long decoding time and post-processing operations, while the component-based coding method adopts artificially designed codes and cannot be flexibly adjusted according to different data.

SUMMARY OF THE INVENTION

In order to solve one of the technical problems existing in the prior art at least to a certain extent, the purpose of the present invention is to provide a zero-sample Chinese word recognition method, system, device and storage medium.

The technical scheme adopted in the present invention is:

A zero-sample Chinese word recognition method, comprising the following steps:

Extract visual features of Chinese single-character images;

Carry out learnable category coding for Chinese word categories, use the depth-first search algorithm to decompose the component structure of Chinese single words, and calculate the learnable category coding;

The category encoding of the Chinese word is mapped to the visual space, based on the mapping module of the fully connected layer, so that the dimension of the category encoding of the Chinese word is equal to the dimension of the visual space, and the reconstruction loss function is used to constrain the category encoding before and after the mapping. semantic consistency;

Through the transformer-based decoder, the category encoding of the Chinese word and the visual features of the image are matched, and the features related to the category encoding are obtained from the visual features of the image, and finally the recognition result of the Chinese word is decoded and output.

Further, the described extraction of the visual features of Chinese single-character images includes:

An image encoder based on a densely connected convolutional neural network is used to extract the visual features of the Chinese single-character images.

Further, the image encoder adopts the DenseNet121 model as the backbone network for extracting the visual features of the image;

The backbone network adopts an 8-fold downsampling method. In order to make the output visual features better match the category codes, the backbone network removes the last output activation layer and global average pooling layer.

Further, the learnable category coding is carried out to the Chinese single-character category, and the algorithm of depth-first search is used to decompose the component structure of the Chinese single-character, and the learnable category coding is obtained by calculation, including:

According to the Chinese ideographic sequence dictionary, the component sequence of the decomposed Chinese word is obtained through the depth-first search algorithm, and the component sequence is represented as a tree data structure, and the depth information and relative position information of each component are obtained; wherein, the depth information It represents the depth of the component in the tree, and the relative position information represents the location of the component relative to its parent node;

The learnable category code corresponding to each Chinese word is calculated, and the calculation process is expressed as formula (1):

Among them, _i represents a component in the component sequence R, li represents the depth information of the component, γ _i represents the relative position information of the component, α and β are learnable parameters, _yi is the one of the component -hot encoding;

The learnable category code obtained by calculation and the depth information and relative position information of each component are spliced in dimension to obtain the final learnable category code. The calculation process is expressed as formula (2):

和

表示的是归一化后的深度信息和相对位置信息，

表示的是拼接操作。in,

and

Represents the normalized depth information and relative position information,

Represents the concatenation operation.

Further, the fully-connected layer-based mapping module is composed of a fully-connected layer; the output elements of the fully-connected layer are all input elements obtained through linear operations;

The fully connected layer maps the category codes of Chinese words into the visual space, so that the dimension of the category codes is equal to the dimension of the visual space.

Further, the reconstruction loss function is used to calculate the mean square error of the category coding before and after the mapping, and the calculation process is expressed as formula (3):

表示的是映射后的类别编码，φ(y_i)表示的是映射前的类别编码，b和w^T是全连接层的偏置和权重的转置，N是类别编码的数量。where L _re is the reconstruction loss function,

represents the class code after mapping, φ(y _i ) represents the class code before mapping, b and w ^T are the transposition of the bias and weight of the fully connected layer, and N is the number of class codes.

Further, the specific operations based on the transformer decoder include:

The multi-head attention mechanism is used to match the category encoding of Chinese words and the visual features of the image, and obtain the features related to the category encoding from the visual features of the image. The calculation process is expressed as formula (4):

MultiHead(Q, K, V)=Concat(head ₁ ,...,head _h )W ^O

head _i =Attention(QW _i ^Q , KW _i ^K , VW _i ^V )

Among them, multi-head attention is realized by MultiHead(Q, K, V), attention is calculated by Attention(Q, K, V), Q represents the category encoding of Chinese words, and K and V represent the visual features of images , Wi ^Q , Wi ^K , Wi _V are all ^learnable projection matrices, d _k represents the dimensions of Q, _K , and _V , and ^WO represents the parameters of multi-head attention;

After obtaining the features related to the category coding, the features are decoded by the feedforward neural network. The feedforward neural network consists of three fully connected layers, and finally outputs the recognition results of Chinese words; in the feedforward neural network training stage, The cross-entropy loss function is adopted as the optimization target of the network, and the expression of the cross-entropy loss function is:

where pi is the label probability for class _i , qi is the predicted probability for class _i , and k is the total number of classes.

Another technical scheme adopted by the present invention is:

A zero-sample Chinese word recognition system, comprising:

The feature extraction module is used to extract the visual features of Chinese single-character images;

The category encoding module is used to perform learnable category encoding for Chinese word categories, using a depth-first search algorithm to decompose the component structure of Chinese words, and calculate the learnable category encoding;

The information mapping module is used to map the category encoding of the Chinese word into the visual space, based on the mapping module of the fully connected layer, so that the dimension of the category encoding of the Chinese word is equal to the dimension of the visual space, and is constrained by the reconstruction loss function Semantic consistency of category encoding before and after mapping;

The information matching module is used to match the category encoding of Chinese words and the visual features of the image through the transformer-based decoder, obtain the features related to the category encoding from the visual features of the image, and finally decode and output the recognition result of the Chinese word.

Another technical scheme adopted by the present invention is:

A zero-sample Chinese word recognition device, comprising:

at least one processor;

at least one memory for storing at least one program;

When the at least one program is executed by the at least one processor, the at least one processor implements the above method.

Another technical scheme adopted by the present invention is:

A computer-readable storage medium in which a processor-executable program is stored, the processor-executable program, when executed by the processor, is used to perform the method as described above.

The beneficial effects of the present invention are as follows: the present invention realizes zero-sample Chinese word recognition through a learnable category coding method, and solves the problem that the existing Chinese word recognition method relies on a large amount of labeled data, and requires time and money to label data. question.

Description of drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following descriptions are given to the accompanying drawings of the embodiments of the present invention or the related technical solutions in the prior art. It should be understood that the drawings in the following introduction are only In order to facilitate and clearly express some embodiments of the technical solutions of the present invention, for those skilled in the art, other drawings can also be obtained from these drawings without creative work.

1 is a flow chart of steps of a zero-sample Chinese word recognition method based on learnable category coding in an embodiment of the present invention;

2 is a schematic diagram of an image encoder in an embodiment of the present invention;

3 is a schematic diagram of a learnable category code in an embodiment of the present invention;

4 is a schematic diagram of a mapping module based on a fully connected layer in an embodiment of the present invention;

FIG. 5 is a schematic diagram of a transformer-based decoder in an embodiment of the present invention.

Detailed ways

The following describes in detail the embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals refer to the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary, only used to explain the present invention, and should not be construed as a limitation of the present invention. The numbers of the steps in the following embodiments are only set for the convenience of description, and the sequence between the steps is not limited in any way, and the execution sequence of each step in the embodiments can be adapted according to the understanding of those skilled in the art Sexual adjustment.

In the description of the present invention, it should be understood that the azimuth description, such as the azimuth or position relationship indicated by up, down, front, rear, left, right, etc., is based on the azimuth or position relationship shown in the drawings, only In order to facilitate the description of the present invention and simplify the description, it is not indicated or implied that the indicated device or element must have a particular orientation, be constructed and operated in a particular orientation, and therefore should not be construed as limiting the present invention.

In the description of the present invention, the meaning of several is one or more, the meaning of multiple is two or more, greater than, less than, exceeding, etc. are understood as not including this number, above, below, within, etc. are understood as including this number. If it is described that the first and the second are only for the purpose of distinguishing technical features, it cannot be understood as indicating or implying relative importance, or indicating the number of the indicated technical features or the order of the indicated technical features. relation.

In the description of the present invention, unless otherwise clearly defined, words such as setting, installation, connection should be understood in a broad sense, and those skilled in the art can reasonably determine the specific meanings of the above words in the present invention in combination with the specific content of the technical solution.

As shown in FIG. 1 , the present embodiment provides a zero-sample Chinese word recognition method based on learnable category coding. The zero-sample Chinese single-character recognition method based on learnable category coding is adopted, and the learnable category coding is used to replace the artificially designed one. way so that it can be flexibly adjusted according to different data. The method includes the following steps:

S1, extract the visual feature of Chinese single-character image, adopt the image encoder based on densely connected convolutional neural network, extract the visual feature of described Chinese single-character image, specifically:

The DenseNet121 model is used as the backbone network to extract the visual features of Chinese single-character images. As shown in Figure 2, the image encoder takes the single-character RGB three-channel image as input, and outputs the visual feature map after downsampling by 8 times. The DenseNet121 model is composed of dense connection blocks and transition modules. In order to make the output visual features better match the category encoding, the backbone network removes the final output activation layer and global average pooling layer.

S2. Carry out learnable category coding for Chinese word categories, use a depth-first search algorithm to decompose the component structure of Chinese single words, and calculate to obtain learnable category codes, specifically:

First, according to the Chinese ideographic sequence dictionary, the decomposed Chinese word component sequence is obtained through the depth-first search algorithm. The component sequence can be represented as a tree data structure, as shown in Figure 3, so the depth of each component can also be obtained. information and relative position information; the depth information represents the depth of the component in the tree, and the relative position information represents the location of the component relative to its parent node; then the learnable category code corresponding to each Chinese word is calculated, and the calculation process It is expressed as formula (1):

Among them, _i represents a component in the component sequence R, li represents the depth information of the component, γ _i represents the relative position information of the component, α and β are learnable parameters, and the initial values are respectively set to 0.5 and 0.001, _yi is the one-hot encoding of the part.

By setting two learnable parameters, α and β, the network can continuously adjust the encoding of categories during training to better match visual features. Finally, the learnable category code is spliced with the depth information and relative position information of each component in the dimension to obtain the final learnable category code. The calculation process is expressed as formula (2):

和

表示的是归一化后的深度信息和相对位置信息，

表示的是拼接操作。在拼接了深度信息和相对位置信息后，类别编码不仅能够表示所包含的部件信息，也能够表示每个部件的深度和相对位置，从而使得类别编码能够包含更丰富的信息，提升网络的识别准确性。in

and

Represents the normalized depth information and relative position information,

Represents the concatenation operation. After splicing the depth information and relative position information, the category coding can not only represent the included component information, but also the depth and relative position of each component, so that the category coding can contain richer information and improve the recognition accuracy of the network. sex.

S3. Based on the mapping module of the fully connected layer, the category encoding of the Chinese word is mapped to the visual space, specifically:

The fully-connected layer-based mapping module, as shown in FIG. 4 , is composed of a fully-connected layer, and its output elements are all input elements obtained through linear operations. The mapping module is used to convert the category code to the same dimension as the visual feature of the image, and at the same time fuse the depth information and the relative position information. In order to make the semantic consistency of the category encoding before and after the mapping module, the reconstruction loss function is used to constrain it. The method is to calculate the mean square error of the category encoding before and after the mapping. The calculation process is expressed as formula (3):

表示的是映射后的类别编码，φ(y_i)表示的是映射前的类别编码，b和w^T是所述全连接层的偏置和权重的转置。where L _re is the reconstruction loss function,

represents the class code after mapping, φ(y _i ) represents the class code before mapping, b and w ^T are the transposition of the bias and weight of the fully connected layer.

S4. Use a transformer-based decoder to match the category encoding of Chinese words and the visual features of images, and decode and output the recognition results of Chinese words, specifically:

The transformer-based decoder, as shown in Figure 5, adopts a multi-head attention mechanism to match the category encoding of Chinese words and the visual features of the image, and obtain the features related to the category encoding from the visual features of the image. The calculation process It is expressed as formula (4):

MultiHead(Q, K, V)=Concat(head ₁ ,...,head _h )W ^O

head _i =Attention(QW _i ^Q , KW _i ^K , VW _i ^V )

Among them, multi-head attention is realized by MultiHead (Q, K, V), and attention is calculated by Attention (Q, K, V), where Q represents the category encoding of Chinese words, and K and V represent the visual features of images. Wi ^Q , Wi ^K , and Wi _V are all learnable projection matrices, d _k represents the dimensions of Q, _K , and ^V , and head _{i represents} the attention of a head. In the calculation of Attention(Q, K, V), the Q representing the category code and the K representing the image feature are multiplied by the matrix, which is equivalent to matching the category code and the visual feature, and then the attention matrix is calculated by the softmax function. , and finally the attention matrix is multiplied by V representing visual features to obtain features related to category encoding.

After obtaining the features related to the category encoding, the features are decoded using a feedforward neural network. The feedforward neural network is composed of three fully connected layers. In addition, residual connections and layer normalization operations are used. Finally, Output the recognition results of Chinese characters. In the network training stage, in order to fit the distribution of network prediction results and real labels, the cross-entropy loss function is used as the optimization goal of the network. The expression of the cross-entropy loss function is shown in formula (5):

where pi is the label probability of class _i , qi is the predicted probability of class _i , and K is the total number of classes.

The invention realizes zero-sample Chinese word recognition through the method of learning category coding, and the method can adaptively adjust the coding of categories according to different data. In addition, the implementation process of the present invention is simple and flexible, and can be transplanted into the mainstream character recognition framework. In general, the method provided by the embodiments of the present invention, compared with the prior art, at least has the following beneficial effects:

(1) The present invention designs a zero-sample recognition model for Chinese words, solves the problem that the existing Chinese word recognition method relies on a large amount of labeled data, and requires time and money to label data, so that the recognition model has better Generalization ability, and the implementation process of the present invention is simple and flexible, and can be transplanted into the mainstream character recognition framework.

(2) The present invention focuses on the problem of zero-sample recognition of Chinese words. Compared with the existing zero-sample Chinese word recognition methods, a learnable category code is adopted, thereby replacing the manual design method, so that it can be based on different data. Adjust flexibly.

(3) The present invention adopts a transformer-based decoder, which can decode quickly and does not require post-processing operations, so that it can be easily applied in practical scenarios.

This embodiment also provides a zero-sample Chinese word recognition system, including:

The feature extraction module is used to extract the visual features of Chinese single-character images;

The category encoding module is used to perform learnable category encoding for Chinese word categories, using a depth-first search algorithm to decompose the component structure of Chinese words, and calculate the learnable category encoding;

The information mapping module is used to map the category encoding of the Chinese word into the visual space, based on the mapping module of the fully connected layer, so that the dimension of the category encoding of the Chinese word is equal to the dimension of the visual space, and is constrained by the reconstruction loss function Semantic consistency of category encoding before and after mapping;

The information matching module is used to match the category encoding of Chinese words and the visual features of the image through the transformer-based decoder, obtain the features related to the category encoding from the visual features of the image, and finally decode and output the recognition result of the Chinese word.

A zero-sample Chinese word recognition system in this embodiment can execute a zero-sample Chinese word recognition method provided by the method embodiment of the present invention, can perform any combination of implementation steps of the method embodiment, and has the corresponding functions and beneficial effect.

This embodiment also provides a zero-sample Chinese word recognition device, including:

at least one processor;

at least one memory for storing at least one program;

When the at least one program is executed by the at least one processor, the at least one processor implements the method shown in FIG. 1 .

A zero-sample Chinese word recognition device in this embodiment can execute a zero-sample Chinese word recognition method provided by the method embodiment of the present invention, can perform any combination of implementation steps of the method embodiment, and has the corresponding functions and beneficial effect.

Embodiments of the present application further disclose a computer program product or computer program, where the computer program product or computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of the computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the method shown in FIG. 1 .

This embodiment also provides a storage medium, which stores an instruction or program for executing a zero-sample Chinese word recognition method provided by the method embodiment of the present invention. When the instruction or program is executed, the method embodiment can be executed. Any combination of implementation steps has corresponding functions and beneficial effects of the method.

In some alternative implementations, the functions/operations noted in the block diagrams may occur out of the order noted in the operational diagrams. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality/operations involved. Furthermore, the embodiments presented and described in the flowcharts of the present invention are provided by way of example in order to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of the various operations are altered and in which sub-operations described as part of larger operations are performed independently.

Furthermore, while the invention is described in the context of functional modules, it is to be understood that, unless stated to the contrary, one or more of the described functions and/or features may be integrated in a single physical device and/or or software modules, or one or more functions and/or features may be implemented in separate physical devices or software modules. It will also be appreciated that a detailed discussion of the actual implementation of each module is not necessary to understand the present invention. Rather, given the attributes, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the modules will be within the routine skill of the engineer. Accordingly, those skilled in the art, using ordinary skill, can implement the invention as set forth in the claims without undue experimentation. It is also to be understood that the specific concepts disclosed are illustrative only and are not intended to limit the scope of the invention, which is to be determined by the appended claims along with their full scope of equivalents.

The functions, if implemented in the form of software functional units and sold or used as independent products, may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention can be embodied in the form of a software product in essence, or the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including Several instructions are used to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, Read-Only Memory (ROM, Read-Only Memory), Random Access Memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program codes .

The logic and/or steps represented in flowcharts or otherwise described herein, for example, may be considered an ordered listing of executable instructions for implementing the logical functions, may be embodied in any computer-readable medium, For use with, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other system that can fetch instructions from and execute instructions from an instruction execution system, apparatus, or apparatus) or equipment. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport the program for use by or in connection with an instruction execution system, apparatus, or apparatus.

More specific examples (non-exhaustive list) of computer readable media include the following: electrical connections with one or more wiring (electronic devices), portable computer disk cartridges (magnetic devices), random access memory (RAM), Read Only Memory (ROM), Erasable Editable Read Only Memory (EPROM or Flash Memory), Fiber Optic Devices, and Portable Compact Disc Read Only Memory (CDROM). In addition, the computer readable medium may even be paper or other suitable medium on which the program may be printed, as the paper or other medium may be optically scanned, for example, followed by editing, interpretation, or other suitable medium as necessary process to obtain the program electronically and then store it in computer memory.

It should be understood that various parts of the present invention may be implemented in hardware, software, firmware or a combination thereof. In the above-described embodiments, various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented by any one or a combination of the following techniques known in the art: Discrete logic circuits, application specific integrated circuits with suitable combinational logic gates, Programmable Gate Arrays (PGA), Field Programmable Gate Arrays (FPGA), etc.

In the above description of the present specification, reference to the description of the terms "one embodiment/example", "another embodiment/example" or "certain embodiments/examples" etc. means the description in conjunction with the embodiment or example. Particular features, structures, materials, or characteristics are included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

Although embodiments of the present invention have been shown and described, it will be understood by those of ordinary skill in the art that various changes, modifications, substitutions and alterations can be made in these embodiments without departing from the principles and spirit of the invention, The scope of the invention is defined by the claims and their equivalents.

The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above-mentioned embodiments, and those skilled in the art can also make various equivalent deformations or replacements on the premise of not violating the spirit of the present invention. Equivalent modifications or substitutions are included within the scope defined by the claims of the present application.

Claims (10)

1. a zero-sample Chinese word recognition method, is characterized in that, comprises the following steps: Extract visual features of Chinese single-character images; Carry out learnable category coding for Chinese word categories, use the depth-first search algorithm to decompose the component structure of Chinese single words, and calculate the learnable category coding; The category encoding of the Chinese word is mapped to the visual space, based on the mapping module of the fully connected layer, so that the dimension of the category encoding of the Chinese word is equal to the dimension of the visual space, and the reconstruction loss function is used to constrain the category encoding before and after the mapping. semantic consistency; Through the transformer-based decoder, the category encoding of the Chinese word and the visual features of the image are matched, and the features related to the category encoding are obtained from the visual features of the image, and finally the recognition result of the Chinese word is decoded and output. 2. a kind of zero-sample Chinese word recognition method according to claim 1, is characterized in that, the visual feature of described extraction Chinese word image, comprises: An image encoder based on a densely connected convolutional neural network is used to extract the visual features of the Chinese single-character images. 3. a kind of zero-sample Chinese word recognition method according to claim 2, is characterized in that, described image encoder adopts DenseNet121 model as backbone network, is used for extracting the visual feature of image; The backbone network adopts an 8-fold downsampling method. In order to make the output visual features better match the category codes, the backbone network removes the last output activation layer and global average pooling layer. 4. a kind of zero-sample Chinese word recognition method according to claim 1, it is characterised in that the described Chinese word category is carried out to learnable category coding, adopts the algorithm of depth-first search, the component structure of Chinese word is decomposed , and calculates the learnable category codes, including: According to the Chinese ideographic sequence dictionary, the component sequence of the decomposed Chinese word is obtained through the depth-first search algorithm, and the component sequence is represented as a tree data structure, and the depth information and relative position information of each component are obtained; wherein, the depth information It represents the depth of the component in the tree, and the relative position information represents the location of the component relative to its parent node; The learnable category code corresponding to each Chinese word is calculated, and the calculation process is expressed as formula (1):

Among them, _i represents a component in the component sequence R, li represents the depth information of the component, γ _i represents the relative position information of the component, α and β are learnable parameters, _yi is the one of the component -hot encoding; The learnable category code obtained by calculation and the depth information and relative position information of each component are spliced in dimension to obtain the final learnable category code. The calculation process is expressed as formula (2):

in,

and

Represents the normalized depth information and relative position information,

Represents the concatenation operation. 5. a kind of zero-sample Chinese word recognition method according to claim 1, is characterized in that, described mapping module based on fully connected layer is formed by a fully connected layer; The output element of described fully connected layer is input element Obtained by linear operation; The fully connected layer maps the category codes of Chinese words into the visual space, so that the dimension of the category codes is equal to the dimension of the visual space. 6. A zero-sample Chinese word recognition method according to claim 5, wherein the reconstruction loss function is used to calculate the mean square error of the category coding before and after the mapping, and the calculation process is expressed as shown in formula (3) :

where L _re is the reconstruction loss function,

represents the class code after mapping, φ(y _i ) represents the class code before mapping, b and w ^T are the transposition of the bias and weight of the fully connected layer, and N is the number of class codes. 7. a kind of zero-sample Chinese word recognition method according to claim 1, is characterized in that, described concrete operation based on transformer decoder comprises: The multi-head attention mechanism is used to match the category encoding of Chinese words and the visual features of the image, and obtain the features related to the category encoding from the visual features of the image. The calculation process is expressed as formula (4): MultiHead(Q,K,V)=Concat(head ₁ ,...,head _h )W ^O

Among them, multi-head attention is realized by MultiHead(Q,K,V), attention is calculated by Attention(Q,K,V), Q represents the category encoding of Chinese words, K and V represent the visual features of the image ,

are all learnable projection matrices, d _k represents the dimensions of Q, K, and V, and W ^O represents the parameters of multi-head attention; After obtaining the features related to the category coding, the features are decoded by the feedforward neural network. The feedforward neural network consists of three fully connected layers, and finally outputs the recognition results of Chinese words; in the feedforward neural network training stage, The cross-entropy loss function is adopted as the optimization target of the network, and the expression of the cross-entropy loss function is:

where pi is the label probability for class _i , qi is the predicted probability for class _i , and k is the total number of classes. 8. A zero-sample Chinese word recognition system, characterized in that, comprising: The feature extraction module is used to extract the visual features of Chinese single-character images; The category encoding module is used to perform learnable category encoding for Chinese word categories, using a depth-first search algorithm to decompose the component structure of Chinese words, and calculate the learnable category encoding; The information mapping module is used to map the category encoding of the Chinese word into the visual space, based on the mapping module of the fully connected layer, so that the dimension of the category encoding of the Chinese word is equal to the dimension of the visual space, and is constrained by the reconstruction loss function Semantic consistency of category encoding before and after mapping; The information matching module is used to match the category encoding of Chinese words and the visual features of the image through the transformer-based decoder, obtain the features related to the category encoding from the visual features of the image, and finally decode and output the recognition result of the Chinese word. 9. A zero-sample Chinese word recognition device, characterized in that, comprising: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method of any one of claims 1-7. 10. A computer-readable storage medium, wherein a program executable by a processor is stored, wherein the program executable by the processor, when executed by the processor, is used to execute any one of claims 1-7 the method.

Images