CN113610080B - Cross-modal perception-based sensitive image identification method, device, equipment and medium - Google Patents

Abstract

The invention discloses a sensitive image recognition method, device, device and medium based on cross-modal perception. The method includes: acquiring image information to be recognized in a network community; inputting the image information into a preset sensitive image recognition The cross-modal perception module in the model obtains the cross-modal text description of the image information; inputs the cross-modal text description of the image information into the sensitive information identification module in the sensitive image recognition model to obtain the sensitive information containing sensitive images. According to the sensitive image recognition method provided by the embodiments of the present disclosure, it aims to express the semantic information content of online community images across modalities, and integrate a large amount of prior knowledge of sensitive text content in online communities to analyze and judge the content of community images more accurately. By obtaining cross-modal text descriptions of images, it is possible to disseminate and trace sensitive image information.

Description

Sensitive image recognition method, device, equipment and medium based on cross-modal perception

technical field

The present invention relates to the technical field of image recognition, in particular to a sensitive image recognition method, device, equipment and medium based on cross-modal perception.

Background technique

With the development of the multimedia era, the graphic network community environment is an important feature of network information dissemination and development. Paying attention to the image-based network community environment, accurately judging and identifying image information, and timely and necessary intervention on sensitive content are conducive to maintaining the stability of the network community environment and maintaining social order.

In the existing technology, the information carried by the image is expressed in an intuitive and visual way, and its semantic information cannot be obtained directly but needs to be "read" and "understood" by the brain. In recent years, with the rapid development of image processing technology, image content recognition technology has also been improved to a certain extent. At present, image classification technology is basically used in the identification and analysis of sensitive image content at home and abroad. Sensitive image recognition technology based on image classification technology extracts image features through neural network, and uses the feature vector as the input of the fully connected layer at the end of the neural network, and the output of the fully connected layer is the image content analysis result. Low-level features such as subjects, lines, and colors in sensitive images are captured through this process and based on this, discriminant analysis is performed. However, image semantic content such as the relationship between image subjects and subject behavior cannot be obtained in this identification and classification process, and there is a formal separation of a large amount of network community knowledge closely related to the identification of sensitive image content, which makes it difficult to combine the results of image content discrimination with a large number of networks. The prior knowledge of community text information is used to discriminate, the recognition accuracy is low, and the comprehensibility is poor. The network community environment is complex, and the dissemination and fermentation of image information in the time dimension is the focus of maintaining network security. Sensitive image recognition technology based on image classification cannot trace the dissemination of image information.

Contents of the invention

Embodiments of the present disclosure provide a sensitive image recognition method, device, device and medium based on cross-modal perception. In order to provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is presented below. This summary is not an overview, nor is it intended to identify key/critical elements or delineate the scope of these embodiments. Its sole purpose is to present some concepts in a simplified form as a prelude to the more detailed description that is presented later.

In the first aspect, the embodiment of the present disclosure provides a sensitive image recognition method based on cross-modal perception, including:

Obtain image information to be identified in the online community;

Input the image information into the cross-modal perception module in the preset sensitive image recognition model to obtain the cross-modal text description of the image information;

The cross-modal text description of image information is input into the sensitive information identification module in the sensitive image identification model to obtain sensitive images containing sensitive information.

In one embodiment, before inputting the image information into the preset sensitive image recognition model, it also includes:

A training data set is constructed, and a sensitive image recognition model is trained based on the training data set, wherein the sensitive image recognition model includes a cross-modal perception module and a sensitive information recognition module.

In one embodiment, the image information is input into the cross-modal perception module in the preset sensitive image recognition model, and the cross-modal text description of the image information is obtained, including:

Identify salient subjects within an image based on a cross-modal perception module;

Determine the cross-modal description model corresponding to the recognized image subject in the pre-trained cross-modal description model group;

According to the cross-modal description model, the subject, inter-subject relationship and high-level semantic information of the subject's behavior in the image are generalized to the content text mode transformation, and the cross-modal text description of the image information is obtained.

In one embodiment, identifying salient subjects within an image according to a cross-modal perception module includes:

According to the subject capture unit in the cross-modal perception module to identify the salient subject in the image, the subject capture unit includes a DenseNet-Block network structure, and its calculation formula is as follows:

X _l ＝H _l ([X ₀ ,X ₁ …X _l-1 ])

Among them, [X ₀ ,X ₁ ...X _l-1 ] means that the feature maps of layers 0 to l-1 are merged on the channel, and H _l means that the merged features are normalized, activated, and convoluted operation, xl represents the result after the l-th layer convolution calculation.

In one embodiment, according to the cross-modal description model, the subject in the image, the relationship between the subjects and the high-level semantic information of the subject's behavior are generalized to the content text mode conversion, and the cross-modal text description of the image information is obtained, including:

Extract image features through the VGGNET network structure in the cross-modal description model;

The extracted image features are input into the long-short-term memory recurrent neural network including the attention mechanism, and the cross-modal text description of the subject in the image, the relationship between subjects and the high-level semantic information of the subject's behavior is obtained.

In one embodiment, the cross-modal text description of image information is input into the sensitive information identification module in the sensitive image identification model to obtain sensitive images containing sensitive information, including:

Train the TextCNN convolutional neural network according to the pre-built training set to obtain the trained sensitive information identification module;

Input the cross-modal text description of the image information into the sensitive information identification module to obtain the identified sensitive text information;

The image corresponding to the sensitive text information is regarded as a sensitive image.

In the second aspect, an embodiment of the present disclosure provides a sensitive image recognition device based on cross-modal perception, including:

An acquisition module, configured to acquire image information to be identified in the online community;

The cross-modal description module is used to input the image information into the cross-modal perception module in the preset sensitive image recognition model to obtain the cross-modal text description of the image information;

The recognition module is used to input the cross-modal text description of image information into the sensitive information recognition module in the sensitive image recognition model to obtain sensitive images containing sensitive information.

In one embodiment, also includes:

The training module is used to construct a training data set, and train a sensitive image recognition model based on the training data set, wherein the sensitive image recognition model includes a cross-modal perception module and a sensitive information recognition module.

In a third aspect, an embodiment of the present disclosure provides a sensitive image recognition device based on cross-modal perception, including a processor and a memory storing program instructions, and the processor is configured to execute the above-mentioned embodiments when executing the program instructions. Sensitive image recognition method based on cross-modal perception.

In a fourth aspect, the embodiments of the present disclosure provide a computer-readable medium on which computer-readable instructions are stored, and the computer-readable instructions can be executed by a processor to implement the cross-modal perception-based Sensitive Image Recognition Methods.

The technical solutions provided by the embodiments of the present disclosure may include the following beneficial effects:

The embodiment of the present disclosure provides a network community-sensitive image recognition model (SIR-CM) based on cross-modal content awareness. The model is mainly composed of image content cross-modal perception module and network community sensitive text information recognition module. SIR-CM implements a generalized network community image content text conversion module on the MSCOCO dataset and online community sensitive image annotation dataset, which can perform fine-grained cross-modal expression of community image content text. Integrating the prior knowledge of sensitive content text sets in the network community environment in the network community sensitive text information identification module, it has the ability to analyze and identify sensitive information in the network community environment, and obtains more accurate and understandable sensitive information. Image recognition results. In addition, the content of information dissemination based on the time dimension, such as the addition of comments and the fermentation of topics, after obtaining the text modal content of the image, makes the image information and its related follow-up information such as additional comments and topic texts unified in form. The dissemination of information can be further traced.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention.

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description serve to explain the principles of the invention.

Fig. 1 is a schematic flowchart of a sensitive image recognition method based on cross-modal perception shown according to an exemplary embodiment;

Fig. 2 is a schematic structural diagram of a sensitive image recognition model shown according to an exemplary embodiment;

Fig. 3 is a schematic diagram of a subject capturing unit according to an exemplary embodiment;

Fig. 4 is a process diagram showing a content text context generation process according to an exemplary embodiment;

Fig. 5 is a process diagram showing a hidden state generation process according to an exemplary embodiment;

Fig. 6 is a diagram showing a gate variable generation process according to an exemplary embodiment;

Fig. 7 is a kind of current word generation process diagram shown according to an exemplary embodiment;

Fig. 8 is a structural diagram of a TextCNN convolutional neural network shown according to an exemplary embodiment;

Fig. 9 is a schematic diagram showing a description of image content according to an exemplary embodiment;

Fig. 10 is a schematic structural diagram of a sensitive image recognition device based on cross-modal perception according to an exemplary embodiment;

Fig. 11 is a schematic structural diagram of a sensitive image recognition device based on cross-modal perception according to an exemplary embodiment;

Fig. 12 is a schematic diagram of a computer storage medium according to an exemplary embodiment.

Detailed ways

The following description and drawings illustrate specific embodiments of the invention sufficiently to enable those skilled in the art to practice them.

It should be clear that the described embodiments are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

When the following description refers to the accompanying drawings, the same numerals in different drawings refer to the same or similar elements unless otherwise indicated. The implementations described in the following exemplary examples do not represent all implementations consistent with the present invention. Rather, they are merely examples of systems and methods consistent with aspects of the invention as recited in the appended claims.

In the description of the present invention, it should be understood that the terms "first", "second" and so on are used for descriptive purposes only, and cannot be interpreted as indicating or implying relative importance. Those of ordinary skill in the art can understand the specific meanings of the above terms in the present invention in specific situations. In addition, in the description of the present invention, unless otherwise specified, "plurality" means two or more. "And/or" describes the association relationship of associated objects, indicating that there may be three types of relationships, for example, A and/or B may indicate: A exists alone, A and B exist simultaneously, and B exists independently. The character "/" generally indicates that the contextual objects are an "or" relationship.

Fig. 1 shows a sensitive image recognition method based on cross-modal perception according to an exemplary embodiment. As shown in Fig. 1 , the method specifically includes the following steps.

S101 Acquire image information to be identified in the online community.

This embodiment uses image description technology to propose a network community-sensitive image recognition model based on cross-modal content perception, aiming at expressing the semantic information content of perceptual network community images across modalities, and integrating a large amount of prior knowledge of network community-sensitive text content, The content of community images can be analyzed and judged more accurately and understandably, and the dissemination and traceability of sensitive image information can be made possible by obtaining the cross-modal content text of network community images.

First, obtain the image information to be recognized in the network community, and then input the obtained image into the trained sensitive image recognition model for recognition.

S102 Input the image information into the cross-modal perception module in the preset sensitive image recognition model to obtain a cross-modal text description of the image information.

In one embodiment, before inputting the image information into the preset sensitive image recognition model, it further includes: constructing an online community sensitive image annotation dataset, and training the sensitive image recognition model based on the online community sensitive image annotation dataset and the MSCOCO dataset.

In an exemplary scenario, the manually annotated online community image dataset and the MSCOCO dataset are used as training datasets for the cross-modal perception module. The final training data set is 25,000 images, the verification set is 2,500 images, and the experimental results are verified on a total of 1,000 images in the test set to obtain a trained sensitive image recognition model. The sensitive image recognition model includes cross-modal perception modules and sensitive information. The recognition module obtains the cross-modal text description of the image information through the cross-modal perception module.

In one embodiment, obtaining the cross-modal text description of the image information according to the cross-modal perception module includes: identifying the salient subject in the image according to the cross-modal perception module, and determining the pre-trained cross-modal description model group and recognition The cross-modal description model corresponding to the obtained image subject, according to the cross-modal description model, generalizes the content text modal transformation of the subject, the relationship between subjects and the high-level semantic information of the subject's behavior in the image, and obtains the cross-modality of the image information. Modal text description.

Specifically, the cross-modal perception module includes a subject capture unit, which is used to identify prominent subjects in the image, perform pre-analysis for the next step across modalities, and improve the accuracy of image information text modal conversion.

Fig. 3 is a schematic diagram of a subject capturing unit shown according to an exemplary embodiment. As shown in Fig. 3, the subject capturing unit includes a 3-layer DenseNet-Block network structure, and its calculation formula is as follows:

X _l ＝H _l ([X ₀ ,X ₁ …X _l-1 ])

Among them, [X ₀ ,X ₁ ...X _l-1 ] means that the feature maps of layers 0 to l-1 are merged on the channel, and H _l means that the merged features are normalized, activated, and convoluted operation, xl represents the result after the l-th layer convolution calculation.

According to the formula, the results of each layer of convolution calculation are obtained, and the results of each layer of convolution calculation are combined to obtain the recognized image subject features.

The DenseNet network structure merges the output features of the 0-l layer on the channel, which alleviates the gradient disappearance problem in the training process of the CNN network and makes the training parameters less, so that it is outstanding in the training process of the main body capture unit.

Further, in the training process of the model, after the subjects of different images are obtained, multiple cross-modal description models are trained based on the identified multiple image subjects, and a group of trained cross-modal description models is obtained.

Then, according to the identified image subject, the cross-modal description model that best matches the subject is selected from the cross-modal description model group, and the cross-modal text description of the image information is obtained based on the selected cross-modal description model.

Further, the cross-modal text description of image information is obtained according to the cross-modal description model, including:

According to the feature extraction unit in the cross-modal description model, image features are extracted, and then the extracted image features are input into the image content cross-modal unit to obtain a text description of the image content.

In an optional embodiment, feature extraction is performed using VGGNET pre-trained on ImageNet, a standard data set of tens of millions of images, because VGGNET's smaller convolution kernel and deeper network perform better than Other networks are more suitable for our model. The network takes a 224*224 picture as input, and uses a convolution kernel to extract global features and local features. The activation function is used to increase the nonlinearity of the network structure, and the pooling layer is used to compress the input feature map to simplify the complexity of network calculations. The commonly used maximum pooling method also has the effect of highlighting the main features.

In the VGGNET structure, only the minimum convolution kernel of 3*3 is used. This can increase the mesh depth within the same receptive field range. First, the first layer uses 64 3*3 convolution kernels with a step size of 1*1, and the input is 224*224*64. The second convolution layer still uses 64 3*3 convolution kernels, and the input size It is 224*224*64, followed by the maximum pooling layer 2*2, and the output is 112*112*64. This is a whole convolution segment. The entire VGGNET has 5-segment convolutions, and this embodiment uses the 14*14*512-dimensional output of the conv5_3 layer in the VGGNET network as a feature representation. Finally, it is flattened into feature vectors with the number of regions L and dimension D. Expressed as {a ₁ ...a _i ...a _L }, where L is 14*14=196, and dimension D is 512. The convolution calculation formula is as follows:

B(i,j)= _∑m= 1∑n ₌₁ K(m,n)×A(i-m+1,j-n+1);

Among them, A is the convolution matrix, K is the convolution kernel, and B is the convolution result.

The formula for the activation function is as follows:

tanh(x)=2σ(2x)-1;

Where σ(x) is a sigmoid function, the activation function is used to improve the nonlinearity of feature extraction, and the MaxPooling function is used to retain the maximum value of the region after activation.

Further, the extracted image features are input into the image content cross-modal unit to obtain the text description of the image content.

Specifically, the input of the image content cross-modal unit is the image feature vector obtained through the feature extraction unit, and the long short-term memory cycle neural network LSTM model injected with the attention mechanism is used to generate the content text description of the network community image.

The long-short-term memory recurrent neural network (LSTM) injected with the attention mechanism uses the focus mechanism to perform adaptive attention on the input image to different degrees through the self-learning of the attention weight, so as to achieve more accurate image features based on the distribution of the attention weight. text description. LSTM can obtain natural language text description output that expresses important features and forgets irrelevant features under continuous memory.

Fig. 4 is a diagram showing a content text context generation process according to an exemplary embodiment. As shown in Fig. 4 , image feature a is subjected to the action of the attention module to obtain C contexts {Z ₁ ... Z _t with dimension D ...Z _c }, where C is understood as the word length of the output content text, and Z _t is understood as the D-dimensional feature representing the context of each word. The process of context generation is the generation process of word y _t one by one.

Among them, the context Z _t is the weighted sum of the original features A, and the weight is Right now:

The weight is L=196, which corresponds to the attention degree of each image feature region. The weight is obtained from the hidden variable h _t-1 of the network in the previous step through the fully connected layer, as shown in Figure 5, which is a diagram of a hidden state generation process shown in an exemplary embodiment. In addition, the weight of the first step has no Hidden state, generated entirely from image features.

Further, Fig. 6 is a diagram showing a gate variable generation process according to an exemplary embodiment. As shown in Fig. 6, the hidden variable h _t simulates the memory function, and generates input it and output _o for the context according to the hidden state of the previous step _t , forget _ft "gate variable", generate candidate _gt controls the current input strength, generate storage _ct controls the storage of the previous word. The hidden state h _t is jointly controlled by the storage c _t and the output gate o _t .

Further, FIG. 7 is a diagram showing a current word generation process according to an exemplary embodiment. As shown in FIG. 7 , the current hidden variable generates the current output word y _t through a fully connected network.

From the generated word-by-word, a textual description of the image content is obtained.

S103 Input the cross-modal text description of the image information into the sensitive information identification module in the sensitive image identification model to obtain a sensitive image containing sensitive information.

In one embodiment, a training data set is constructed first. In this embodiment, posts from 50 popular websites are crawled, and 35,000 sensitive texts and 50,000 non-sensitive texts are classified according to network community sensitivity as a training set.

Further, the TextCNN convolutional neural network is trained according to the pre-built training set, and the trained sensitive information recognition module is obtained.

TextCNN is a convolutional neural network on text, which represents the text as a word vector matrix, and each row of the input matrix represents a word. The convolution kernel is used as a local feature extractor, and its length is the same as the word length. When the Filter-size is different, its function It is a feature of different dimensions of n-gram. Fig. 8 is a structural diagram of a TextCNN convolutional neural network according to an exemplary embodiment. As shown in Figure 8, the TextCNN convolutional neural network includes an input sentence composed of n k-dimensional words, and then outputs the result through a multi-dimensional feature extraction convolutional layer, pooling layer, and fully connected layer.

Input the cross-modal text description of the image information into the sensitive information identification module to obtain the identified sensitive text information, and the image corresponding to the sensitive text information is the sensitive image.

The sensitive information identification module in the embodiment of the present disclosure integrates a large number of sensitive text knowledge bases in the network community environment, and based on this, more accurate and credible sensitive information identification can be performed on image content texts.

Fig. 2 is a schematic structural diagram of a sensitive image recognition model according to an exemplary embodiment. As shown in Fig. 2 , the sensitive image recognition model includes: a cross-modal perception module for image content in online communities and sensitive text information recognition in online communities module.

In the cross-modal perception module of online community image content, firstly, the online community image is input, and then image subjects are captured based on DenseNet. During the model training process, corresponding cross-modal description models are trained for different types of subjects, and Describe model populations across modalities. Among them, the cross-modal description model includes a feature extraction unit based on VGGNET for feature extraction, including an image content description unit, and a long-term short-term memory cycle neural network LSTM model based on an injection attention mechanism to generate content text descriptions of network community images.

In the network community sensitive text information recognition module, the TextCNN convolutional neural network is trained based on the network community prior knowledge base, and the trained sensitive information text recognition model is obtained, and the image content text description generated by the cross-modal perception module is input into the sensitive text information Identify the model and get the identified sensitive image.

In an exemplary scenario, an embodiment of the present disclosure performs experimental verification on the above method. First, obtain the experimental data, use the manually labeled network community description data and MSCOCO data as the cross-modal perception module training data set, the final training data set is 25,000 images, the verification set is 2,500 images, and a total of 1,000 images are used in the test set Verify the experimental results. A total of 35,000 sensitive training texts and 50,000 non-sensitive training texts were used to publish articles from the self-crawled university network communities to conduct sensitive text information recognition module training, and the experimental results were verified on 1,000 test texts.

Then construct the experimental environment, in the Tensorflow training environment, the maximum Epoch is set to 100, the Batch-Size is set to 64, and the learning rate is set to 0.001 to train the DenseNet main body capture unit.

In the Tensorflow environment, the Rmsprop optimizer is used, the cross-entropy loss function is used, the Batch-Size is set to 16, the maximum Epoch is set to 500, and the learning rate is 0.001 to train the cross-modal description model group.

In the Keras environment, the text length is set to 50, the dictionary length is 5000, the Batch-Size is set to 64, the maximum Epoch is set to 100, and the Filter-size is set to 2, 3, 4, 5, 7, 10 respectively to train network community sensitive text information identification module.

In the Tensorflow training environment, the maximum Epoch is set to 100, the Batch-Size is set to 64, and the learning rate is set to 0.001, the VGGNet and DenseNet sensitive image classification recognition models are trained as control models. As one of the representatives of deep neural networks, VGGNET forms a deeper network due to smaller convolution kernels, making the error rate of the network in the field of image recognition and classification lower than 7.3%. Due to its outstanding performance as a classic neural network in the field of image classification and recognition, it is selected as the comparison model 1 in this embodiment. In the development process of the classic neural network, DenseNet, as a complex version of ResNet, its tight cross-layer connection and simplified memory consumption make it the latest representative of the classic neural network. This paper selects it as the comparison model 2.

Further, to start the experiment, first conduct DenseNet subject capture unit training on the above data set, and use the marked 12 types of sensitive subjects and MSCOCO non-sensitive subjects as 13 categories of classification labels. The subject capture module trains the model with the highest classification accuracy as the catcher.

For the training of each subject category, the model with the best performance of the BLEU standard in the description text on the verification set image is selected as the network community cross-modal description model. All proprietary models form a cross-modal description model group for image content in online communities.

TextCNN content text recognition model training selects the model with the highest recognition sensitivity accuracy on the test text set as the final network community image sensitive text information recognition model.

The VGGNet and DenseNet image recognition classification models are trained on the above datasets to verify the best performing model on the image set as a control model.

The training results of the DenseNet subject capture unit are shown in the following table:

loss Acc_val Acc_test Score 0.225 0.9206 0.917

The average performance parameters of the cross-modal description model for each subject category in the model group on the verification set are shown in the table below:

Bleu_1 0.7727 Bleu_2 0.6809 Bleu_3 0.6143 Bleu_4 0.5834

The semantic information representation of the cross-modal perception network community image is shown in Figure 9. The extracted text is described as "ablurry view of a building covered with fire", that is, "a blurry view of a building covered with fire".

The average score of the network community image content cross-modal description model group performance is better, mainly because this embodiment conducts network community-sensitive image description training in the training image set. For the specific application field of the network community environment, the image content description model It has better adaptability.

The image content text obtained in this process enables this paper to grasp the information content in the form of text in the process of image information dissemination, and has stronger interpretability for the discrimination results of sensitive image recognition in the next step, making it possible to integrate the network community text knowledge base for sensitive recognition become possible.

The accuracy of TextCNN sensitive text recognition accuracy on 1000 test text sets is shown in the following table:

loss Acc_val Acc_test Score 0.104 0.981 0.97

The experimental results of sensitive recognition accuracy comparison between the content text-based network community sensitive image recognition model and the comparison model on the verification set and test set are shown in the following table:

The experimental results show that based on cross-modal perception and fusion of a large number of network community text-sensitive discriminative prior knowledge to identify sensitive images in the network community, the accuracy of the discriminant prediction in this paper has been significantly improved compared with the effect of sensitive recognition through neural networks.

In the discrimination process of the network community-sensitive image recognition model based on cross-modal content awareness, the cross-modal text of the image content can be obtained, and a large amount of prior knowledge of network community text-sensitive recognition can be integrated to obtain more accurate and understandable images. Sensitive image recognition results. In addition, the content of information dissemination based on the time dimension, such as the addition of comments and the fermentation of topics, after obtaining the text modal content of the image, makes the image information and its related follow-up information such as additional comments and topic texts unified in form. The dissemination of information can be further traced.

An embodiment of the present disclosure also provides a sensitive image recognition device based on cross-modal perception, which is used to implement the sensitive image recognition method based on cross-modal perception in the above embodiment, as shown in FIG. 10 , the device includes:

An acquisition module 1001, configured to acquire image information to be identified in the online community;

The cross-modal description module 1002 is used to input the image information into the cross-modal perception module in the preset sensitive image recognition model to obtain the cross-modal text description of the image information;

The recognition module 1003 is configured to input the cross-modal text description of the image information into the sensitive information recognition module in the sensitive image recognition model to obtain sensitive images containing sensitive information.

In one embodiment, also includes:

The training module is used to construct a training data set, and train a sensitive image recognition model based on the training data set, wherein the sensitive image recognition model includes a cross-modal perception module and a sensitive information recognition module.

It should be noted that when the sensitive image recognition device based on cross-modal perception provided by the above-mentioned embodiments executes the sensitive image recognition method based on cross-modal perception, it only uses the division of the above-mentioned functional modules as an example. In practical applications, The above function allocation can be completed by different functional modules as required, that is, the internal structure of the device is divided into different functional modules, so as to complete all or part of the functions described above. In addition, the sensitive image recognition device based on cross-modal perception provided by the above embodiment and the sensitive image recognition method embodiment based on cross-modal perception belong to the same concept, and its implementation process is detailed in the method embodiment, and will not be repeated here.

Embodiments of the present disclosure also provide an electronic device corresponding to the sensitive image recognition method based on cross-modal perception provided in the foregoing embodiments, so as to execute the above-mentioned sensitive image recognition method based on cross-modal perception.

Please refer to FIG. 11 , which shows a schematic diagram of an electronic device provided by some embodiments of the present application. As shown in Figure 11, electronic equipment comprises: processor 1100, memory 1101, bus 1102 and communication interface 1103, processor 1100, communication interface 1103 and memory 1101 are connected through bus 1102; running computer program, the processor 1100 executes the sensitive image recognition method based on cross-modal perception provided by any one of the foregoing embodiments of the present application when running the computer program.

Wherein, the memory 1101 may include a high-speed random access memory (RAM: Random Access Memory), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 1103 (which may be wired or wireless), and the Internet, wide area network, local network, metropolitan area network, etc. can be used.

The bus 1102 can be an ISA bus, a PCI bus, or an EISA bus, etc. The bus can be divided into address bus, data bus, control bus and so on. Wherein, the memory 1101 is used to store the program, and the processor 1100 executes the program after receiving the execution instruction, and the sensitive image recognition method based on cross-modal perception disclosed in any of the above-mentioned embodiments of the present application can be applied to the processor 1100 , or implemented by the processor 1100.

The processor 1100 may be an integrated circuit chip with signal processing capability. In the implementation process, each step of the above method may be implemented by an integrated logic circuit of hardware in the processor 1100 or instructions in the form of software. The above-mentioned processor 1100 can be a general-purpose processor, including a central processing unit (Central Processing Unit, referred to as CPU), a network processor (Network Processor, referred to as NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), off-the-shelf programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. Various methods, steps, and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, and the like. The steps of the method disclosed in the embodiments of the present application may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the field such as random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, register. The storage medium is located in the memory 1101, and the processor 1100 reads the information in the memory 1101, and completes the steps of the above method in combination with its hardware.

The electronic device provided in the embodiment of the present application is based on the same inventive concept as the sensitive image recognition method based on cross-modal perception provided in the embodiment of the present application, and has the same beneficial effect as the method adopted, operated or implemented.

The embodiment of the present application also provides a computer-readable storage medium corresponding to the sensitive image recognition method based on cross-modal perception provided in the foregoing embodiments, please refer to FIG. 12 , the computer-readable storage medium shown in it is an optical disc 1200 A computer program (that is, a program product) is stored thereon, and when the computer program is run by the processor, it will execute the sensitive image recognition method based on cross-modal perception provided by any of the foregoing embodiments.

It should be noted that examples of computer-readable storage media may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access Memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media will not be repeated here.

The computer-readable storage medium provided by the above-mentioned embodiments of the present application is based on the same inventive concept as the sensitive image recognition method based on cross-modal perception provided by the embodiments of the present application. The same beneficial effect of the method.

The above examples only express several implementations of the present invention, and the description thereof is relatively specific and detailed, but should not be construed as limiting the patent scope of the present invention. It should be noted that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent for the present invention should be based on the appended claims.

Claims (8)

1. A sensitive image recognition method based on cross-modal perception, characterized in that, comprising: Obtain image information to be identified in the online community; Inputting the image information into a cross-modal perception module in a preset sensitive image recognition model to obtain a cross-modal text description of the image information; including: identifying a prominent subject in the image according to the cross-modal perception module; Determine the cross-modal description model corresponding to the identified image subject in the pre-trained cross-modal description model group; according to the cross-modal description model, the high-level semantic information of the subject, inter-subject relationship and subject behavior in the image performing generalized content text modal conversion to obtain a cross-modal text description of the image information; Wherein, the image features are extracted through the VGGNET network structure in the cross-modal description model; the extracted image features are input into the long-short-term memory cycle neural network including the attention mechanism, and the subject, inter-subject relationship and subject behavior in the image are obtained Cross-modal textual description of high-level semantic information of ; Inputting the cross-modal text description of the image information into a sensitive information identification module in the sensitive image identification model to obtain a sensitive image containing sensitive information. 2. The method according to claim 1, wherein, before inputting the image information into a preset sensitive image recognition model, further comprising: A training data set is constructed, and the sensitive image recognition model is trained based on the training data set, wherein the sensitive image recognition model includes a cross-modal perception module and a sensitive information recognition module. 3. The method according to claim 1, wherein identifying a salient subject in an image according to the cross-modal perception module comprises: According to the subject capture unit in the cross-modal perception module to identify the salient subject in the image, the subject capture unit includes a DenseNet-Block network structure, and its calculation formula is as follows: X _l = H _l ([X ₀ , X ₁ ... X _l-1 ]) Among them, [X ₀ , X ₁ ... X _l-1 ] means that the feature maps of layers 0 to l-1 are merged on the channel, and H _l means that the merged features are normalized, activated, and Convolution operation, X _l represents the result after the l-th layer convolution calculation. 4. The method according to claim 1, wherein the cross-modal text description of the image information is input into the sensitive information identification module in the sensitive image identification model to obtain a sensitive image containing sensitive information, comprising: Train the TextCNN convolutional neural network according to the pre-built training set to obtain the trained sensitive information identification module; Inputting the cross-modal text description of the image information into the sensitive information identification module to obtain the identified sensitive text information; The image corresponding to the sensitive text information is used as a sensitive image. 5. A sensitive image recognition device based on cross-modal perception, characterized in that it comprises: An acquisition module, configured to acquire image information to be identified in the online community; The cross-modal description module is used to input the image information into the cross-modal perception module in the preset sensitive image recognition model to obtain the cross-modal text description of the image information; including: according to the cross-modal perception The module recognizes the significant subject in the image; determines the cross-modal description model corresponding to the identified image subject in the pre-trained cross-modal description model group; The generalized content text modal conversion is performed on the high-level semantic information of the relationship and the subject's behavior, and the cross-modal text description of the image information is obtained; Wherein, the image features are extracted through the VGGNET network structure in the cross-modal description model; the extracted image features are input into the long-short-term memory cycle neural network including the attention mechanism, and the subject, inter-subject relationship and subject behavior in the image are obtained Cross-modal textual description of high-level semantic information of ; The recognition module is configured to input the cross-modal text description of the image information into the sensitive information recognition module in the sensitive image recognition model to obtain a sensitive image containing sensitive information. 6. The device according to claim 5, further comprising: The training module is used to construct a training data set, and train the sensitive image recognition model based on the training data set, wherein the sensitive image recognition model includes a cross-modal perception module and a sensitive information recognition module. 7. A sensitive image recognition device based on cross-modal perception, characterized in that it includes a processor and a memory storing program instructions, and the processor is configured to execute the program according to claim 1 when executing the program instructions. The sensitive image recognition method based on cross-modal perception described in any one of to 4. 8. A computer-readable medium, characterized in that computer-readable instructions are stored thereon, and the computer-readable instructions can be executed by a processor to implement a method based on any one of claims 1 to 4. A Sensitive Image Recognition Approach for Cross-Modal Perception.