CN112199564A - Information filtering method and device and terminal equipment - Google Patents

Information filtering method and device and terminal equipment Download PDF

Info

Publication number
CN112199564A
CN112199564A CN201910614219.3A CN201910614219A CN112199564A CN 112199564 A CN112199564 A CN 112199564A CN 201910614219 A CN201910614219 A CN 201910614219A CN 112199564 A CN112199564 A CN 112199564A
Authority
CN
China
Prior art keywords
information
neural network
processed
type
inputting
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201910614219.3A
Other languages
Chinese (zh)
Inventor
陈炫言
霰心培
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
TCL Corp
TCL Research America Inc
Original Assignee
TCL Research America Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by TCL Research America Inc filed Critical TCL Research America Inc
Priority to CN201910614219.3A priority Critical patent/CN112199564A/en
Publication of CN112199564A publication Critical patent/CN112199564A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90Details of database functions independent of the retrieved data types
    • G06F16/95Retrieval from the web
    • G06F16/951Indexing; Web crawling techniques
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90Details of database functions independent of the retrieved data types
    • G06F16/95Retrieval from the web
    • G06F16/953Querying, e.g. by the use of web search engines
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90Details of database functions independent of the retrieved data types
    • G06F16/95Retrieval from the web
    • G06F16/958Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/04Architecture, e.g. interconnection topology

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Biomedical Technology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Health & Medical Sciences (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Image Analysis (AREA)

Abstract

The application is applicable to the technical field of data processing, and provides an information filtering method, an information filtering device and terminal equipment, wherein the method comprises the following steps: acquiring information to be processed, wherein the information to be processed at least comprises information of two data types; identifying the data types included in the information to be processed, inputting the information of different data types in the information to be processed into corresponding neural networks in a neural network group for processing, and obtaining tags of the information of different data types in the information to be processed; and filtering the information of which the label is a preset filtering label in the information to be processed. The method and the device can solve the problems that the existing information filtering method can only filter data of a single type and the filtering effect is not good.

Description

Information filtering method and device and terminal equipment
Technical Field
The application belongs to the technical field of data processing, and particularly relates to an information filtering method and device and terminal equipment.
Background
With the development of science and technology, the internet enters thousands of households, and people can look up and issue various information on the internet.
Among the mass information of the internet, a part of information belongs to bad information, and the bad information can influence the internet surfing experience of people and can influence the physical and mental health development of minors. To reduce such objectionable information, IT practitioners wish to filter such objectionable information.
In the current information filtering method, information can be identified and filtered only aiming at specific types of data, but most of information on the internet is mixed type data, such as combination of characters and images, combination of characters and videos, and the like. If only specific types of data can be filtered, information identification and filtering cannot be performed on other types of data in mixed types of data, and the filtering effect is poor.
In summary, the existing information filtering method can only filter a single type of bad data, and the filtering effect is not good.
Disclosure of Invention
In view of this, embodiments of the present application provide an information filtering method, an information filtering apparatus, and a terminal device, so as to solve the problem that the existing information filtering method can only filter a single type of data, and the filtering effect is not good.
A first aspect of an embodiment of the present application provides an information filtering method, including:
acquiring information to be processed, wherein the information to be processed at least comprises information of two data types;
identifying the data types included in the information to be processed, inputting the information of different data types in the information to be processed into corresponding neural networks in a neural network group for processing, and obtaining tags of the information of different data types in the information to be processed;
and filtering the information of which the label is a preset filtering label in the information to be processed.
A second aspect of an embodiment of the present application provides an information filtering apparatus, including:
the information acquisition module is used for acquiring information to be processed, wherein the information to be processed at least comprises information of two data types;
the tag identification module is used for identifying the data types included in the information to be processed, inputting the information of different data types in the information to be processed into corresponding neural networks in a neural network group for processing, and obtaining tags of the information of different data types in the information to be processed;
and the information filtering module is used for filtering the information of which the label is a preset filtering label in the information to be processed.
A third aspect of the embodiments of the present application provides a terminal device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the method when executing the computer program.
A fourth aspect of embodiments of the present application provides a computer-readable storage medium, in which a computer program is stored, which, when executed by a processor, implements the steps of the method as described above.
Compared with the prior art, the embodiment of the application has the advantages that:
according to the information filtering method, when the received information to be processed contains information of at least two data types, the data types contained in the information to be processed can be identified, the information of different data types in the information to be processed is input into the corresponding neural networks in the neural network group to be processed, the neural network group contains multiple neural networks, the information of multiple data types can be identified through the division and cooperation of the multiple neural networks, the information of various data types in the information to be processed is comprehensively identified and filtered, the problem that the existing information filtering method can only filter data of a single type is solved, and the filtering effect is poor.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or the prior art descriptions will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings without creative efforts.
Fig. 1 is a schematic flow chart illustrating an implementation of an information filtering method according to an embodiment of the present application;
FIG. 2 is a schematic diagram of an information filtering apparatus according to an embodiment of the present disclosure;
fig. 3 is a schematic diagram of a terminal device provided in an embodiment of the present application;
FIG. 4 is a diagram of an example of an application of the LSTM-Jump network provided by an embodiment of the present application;
FIG. 5 is a diagram illustrating an example of an application of a convolutional neural network provided in an embodiment of the present application;
fig. 6 is a diagram of an application example of a 3D convolutional neural network provided in an embodiment of the present application.
Detailed Description
In the following description, for purposes of explanation and not limitation, specific details are set forth, such as particular system structures, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. It will be apparent, however, to one skilled in the art that the present application may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.
In order to explain the technical solution described in the present application, the following description will be given by way of specific examples.
It will be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It is also to be understood that the terminology used in the description of the present application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in the specification of the present application and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
It should be further understood that the term "and/or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.
As used in this specification and the appended claims, the term "if" may be interpreted contextually as "when", "upon" or "in response to a determination" or "in response to a detection". Similarly, the phrase "if it is determined" or "if a [ described condition or event ] is detected" may be interpreted contextually to mean "upon determining" or "in response to determining" or "upon detecting [ described condition or event ]" or "in response to detecting [ described condition or event ]".
The first embodiment is as follows:
referring to fig. 1, an information filtering method provided in an embodiment of the present application is described below, where the information filtering method in the embodiment of the present application includes:
s101, obtaining information to be processed, wherein the information to be processed at least comprises information of two data types;
the information to be processed is obtained, and since most of the information on the current network is mixed information, the information to be processed may include information of two or more data types, for example, the information to be processed may include information of two or more data types of a text type, an image type, and a video type.
Step S102, identifying data types included in the information to be processed, inputting information of different data types in the information to be processed into corresponding neural networks in a neural network group for processing, and obtaining labels of information of different data types in the information to be processed;
after the information to be processed is acquired, the data type included in the information to be processed may be identified, for example, the data type may be identified according to a filename suffix of a file.
Classifying the information in the information to be processed according to the data types, inputting the information of different data types in the information to be processed into corresponding neural networks in a neural network group for processing, and outputting the labels of the information of the corresponding data types by each neural network.
And S103, filtering the information of which the label is a preset filtering label in the information to be processed.
The label of the information to be processed may include a non-preset filtering label and a preset filtering label, and the information that the label in the information to be processed is the preset filtering label is filtered, for example, the information to be processed includes information of a text type and information of an image type, and if the label of the information of the image type is the preset filtering label, the information of the image type in the information to be processed is filtered.
The preset filtering label can be set according to actual requirements, for example, the bad information label can be set to be the preset filtering label.
Further, the identifying the data type included in the information to be processed, and inputting the information of different data types in the information to be processed into a corresponding neural network in a neural network group for processing, so as to obtain the tags of the information of different data types in the information to be processed specifically includes:
a1, identifying the data type included in the information to be processed;
a2, when the information to be processed contains the information of the text type, inputting the information of the text type into a text recognition neural network in a neural network group to obtain a label of the information of the text type;
when the information to be processed contains the information of the text type, the information of the text type can be input into the text recognition neural network in the neural network group to obtain the label of the information of the text type. The type of the text recognition neural network can be selected according to actual requirements, for example, one of an RNN network, an LSTM network and an LSTM-Jump network can be selected as the text recognition neural network.
When the LSTM-Jump network is adopted as the text recognition neural network, the processing speed of the text recognition neural network on long texts can be improved, and a large amount of noise can be filtered due to word skipping, so that the recognition accuracy of the LSTM-Jump network is improved.
The user can set the hyper-parameters of the LSTM-Jump network, such as the number of words R that need to be read between each Jump, the maximum number of words K allowed to be skipped, and the total number of jumps N allowed.
After the framework of the LSTM-Jump network is constructed, the LSTM-Jump network is trained using the training samples.
After the training is finished, the LSTM-Jump network can be used for identifying the information of the text type, embedding the information of the text type into the LSTM-Jump network in the form of word vectors, the calculation is performed according to the number R of words to be read between each Jump as the initial value, for example, the LSTM-Jump network in fig. 4, the value of R is set to 2, then, first, 2 words, i.e. "To" and "be", are read in, the hidden layer is transferred in, and then the softmax layer is transferred in, and the step that needs To jump next time is calculated, for example, the step is 3, and jumping for 3 steps, then entering a next cycle, continuously reading 2 words which are respectively 'not' and 'to', repeating the steps for propagation until a preset stop condition is met, and finishing the propagation, wherein the preset stop condition can be that the total jumping number exceeds the allowed total jumping number N, or softmax outputs 0, which indicates that the text is completely read.
After propagation is finished, the trunk of the LSTM-Jump network outputs the probability that the data of the text type corresponds to the preset filtering tag, a text determination threshold may be set when the tag determination is performed, when the probability of the preset filtering tag output by the LSTM-Jump network is greater than or equal to the text determination threshold, the tag of the information of the text type is the preset filtering tag, and the specific value of the text determination threshold may be set in an actual situation, for example, may be set to 0.5, 0.6, 0.7, and the like.
A3, when the information to be processed contains the information of the image type, inputting the information of the image type into an image recognition neural network in a neural network group to obtain a label of the information of the image type;
when the information to be processed includes the information of the image type, the information of the image type can be input into the image recognition neural network in the neural network group to obtain the label of the information of the image type.
The type of the image recognition neural network can be selected according to actual requirements, for example, a conventional convolutional neural network can be selected as the image recognition neural network.
When a conventional convolutional neural network is adopted as the image recognition neural network, image feature extraction can be performed through a convolutional layer, a pooling layer and the like, and the probability of each label is classified and output through a classifier in a full-link layer.
When the probability of the label of the bad information output by the conventional convolutional neural network is greater than or equal to the image judgment threshold, the label of the information of the image type is the bad information, and the specific value of the image judgment threshold can be set in actual conditions, for example, can be set to 0.5, 0.6, 0.7, and the like.
When the information of the image type includes a plurality of images, the label of each image is identified.
A4, when the information to be processed contains the information of the video type, inputting the information of the video type into a video recognition neural network in a neural network group to obtain the label of the information of the video type.
When the information to be processed contains the information of the video type, the information of the video type can be input into the video identification neural network in the neural network group to obtain the label of the information of the video type. The category of the video recognition neural network can be selected according to actual requirements.
Further, when the information to be processed includes information of a text type, inputting the information of the text type into a text recognition neural network in a neural network group, and obtaining the label of the information of the text type specifically includes:
b1, when the information to be processed contains text type information, performing sentence segmentation processing and word segmentation processing on the text type information;
when the information to be processed contains the information of the text type, word segmentation processing can be carried out on the information of the text type, so that the information of the text type can be conveniently input into the text recognition neural network.
Meanwhile, if each recognition is to recognize the information of the whole text type, the condition of excessive filtering may exist during the filtering of bad information, for example, only a few sensitive words exist in an article, but the whole article is directly filtered as bad information, so that the sentence division processing can be performed on the information of the text type, for example, the article is divided into a plurality of text sentences according to the periods, the filtering of the information is performed by taking the text sentences as units, and the occurrence of the excessive filtering is reduced.
And B2, inputting the text sentences after word segmentation into a text recognition neural network in the neural network group to obtain the labels of the text sentences in the information of the text types.
After that, the text sentence after word segmentation processing may be input to the text recognition neural network in the neural network group to obtain the label of each text sentence in the information of the text type, and the text sentence judged as the preset filtering label is filtered according to the label.
The way of filtering the text sentence can be selected according to the actual situation, for example, the text sentence judged as the preset filtering label can be replaced by the shielding character when the information of the text type is displayed.
Further, when the information to be processed includes information of an image type, the information of the image type is input to an image recognition neural network in a neural network group, and the tag for obtaining the information of the image type specifically includes:
c1, when the information to be processed contains the information of the image type, inputting the information of the image type into a convolution layer of a convolution neural network in a neural network group for first feature extraction processing to obtain first feature information;
in some embodiments, a convolutional neural network may be employed as the image recognition neural network, and may include a convolutional layer, a pooling layer, a full-link layer, a Softmax layer, and an output layer.
Taking the convolutional neural network in fig. 5 as an example, when the information to be processed includes information of an image type, the information of the image type may be input into a convolutional layer of the convolutional neural network to perform first feature extraction, so as to obtain first feature information.
C2, inputting the first feature information into the pooling layer of the convolutional neural network for second feature extraction processing to obtain second feature information;
and then inputting the first feature information into the pooling layer for second feature extraction to obtain second feature information.
C3, inputting the second characteristic information into a first full-connection layer of the convolutional neural network for first classification processing to obtain a first classification result;
and inputting the second characteristic information into the first full-connection layer to carry out first classification processing to obtain a first classification result.
And C4, inputting the first classification result into a Softmax layer of the convolutional neural network for second classification processing to obtain the probability that the information of the image type corresponds to each label, and taking the label with the maximum probability as the label of the information of the image type.
Inputting the first classification result into a Softmax layer to perform a second classification process, in some embodiments, only two tags may be set, where tag 0 represents a non-preset filtering tag, and tag 1 represents a preset filtering tag, and then the Softmax layer performs a first-time second classification to obtain probabilities of tag 0 and tag 1 and output the probabilities to the outside of the convolutional neural network through an output layer, and at this time, the tag with the highest probability may be used as the tag of the information of the image type, for example, the probability of tag 1 is 0.6, which is greater than the probability of tag 0, and tag 1 is used as the tag of the information of the image type, and at this time, the information of the image type is information that needs to be filtered.
It should be understood that the convolutional neural network structure in fig. 5 is only an illustrative example, in an actual application process, other types of neural networks or convolutional neural networks with other structures may be selected as the image recognition neural network, and a specific setting manner may be selected according to actual requirements.
The manner of filtering the information of the image type may be selected according to actual situations, for example, when the information of the image type is presented, a blank image may be substituted for the image determined as the preset filtering label.
Further, when the information to be processed includes information of a video type, the inputting the information of the video type into a video recognition neural network in a neural network group, and obtaining the tag of the information of the video type specifically includes:
d1, when the information to be processed contains the information of the video type, carrying out segmentation processing on the information of the video type according to a preset frame number threshold value to obtain at least one video segment;
when the information to be processed includes information of a video type, in order to avoid an oversize video, the information of the video type may be segmented according to a preset frame number threshold to obtain at least one video segment, for example, the preset frame number threshold may be set to 7 frames, and the information of the video type may be divided into a plurality of video segments of 7 frames or less than 7 frames.
And D2, inputting the video segments into the video recognition neural network in the neural network group, and obtaining the label of each video segment in the information of the video type.
And inputting the video segments into a video identification neural network in the neural network group to obtain the labels of all the video segments in the video type information, and filtering the video segments judged as preset filtering labels according to the labels.
The mode of filtering the video segment can be selected according to actual situations, for example, a blank frame can be used to replace a video frame determined as a preset filtering label when the information of the video type is displayed.
Further, the video identification neural network is specifically a 3D convolutional neural network, and when the information to be processed includes information of a video type, the information of the video type is input to the video identification neural networks in the neural network group, and the tag for obtaining the information of the video type specifically includes:
e1, when the information to be processed contains the information of the video type, inputting the information of the video type into a hard connection layer of a 3D convolutional neural network in a neural network group for multi-channel feature extraction processing to obtain third feature information;
in some embodiments, a 3D convolutional neural network (3D CNN network) may be selected as the video recognition neural network. When the 3D CNN network is adopted as the video recognition neural network, the 3D CNN network adds the dimension of time in the network and endows the neural network with a function of behavior recognition, so that the 3D CNN network can be adopted to recognize behaviors in the video, for example, whether fighting behaviors exist in the video can be recognized, and whether the information of the video type is filtered or not can be judged.
Taking the 3D CNN network in fig. 6 as an example, the 3D CNN network may include a hard-wired layer, a convolutional layer, a downsampling layer, and a full-connected layer, where the number of the convolutional layer and the downsampling layer may be set according to actual needs, and 3 convolutional layers and 2 downsampling layers are set in the 3D CNN network in fig. 6.
Inputting the video type information into a hard connection layer, wherein in some embodiments, the hard connection layer divides the video type information into 5 channels for parallel processing to obtain characteristic information of each channel, and combines the information output by all the channels to obtain third characteristic information;
e2, inputting the third feature information into a first 3D convolutional layer of the 3D convolutional neural network for first 3D feature extraction processing to obtain fourth feature information, wherein a convolution kernel in the first 3D convolutional layer is a 3D convolution kernel;
and taking the third feature information output by the hard connecting layer as an input of the first 3D convolutional layer, wherein the first 3D convolutional layer can use a 3D convolutional kernel to perform first 3D feature extraction processing to obtain fourth feature information, and in order to increase the number of feature information obtained by convolution, the first convolutional layer can use two different 3D convolutional kernels to perform feature extraction processing.
E3, inputting the fourth feature information into a first downsampling layer of the 3D convolutional neural network for first downsampling processing to obtain fifth feature information;
the fourth feature information output by the first 3D convolutional layer is input to the first downsampling layer, the downsampling method of the first downsampling layer may be set according to actual needs, for example, a method of maximizing pooling may be selected for downsampling, and the first downsampling process may be performed on the fourth feature information to obtain the fifth feature information.
E4, inputting the fifth feature information into a second 3D convolutional layer of the 3D convolutional neural network for second 3D feature extraction processing to obtain sixth feature information, wherein a convolution kernel in the second 3D convolutional layer is a 3D convolution kernel;
and taking the fifth feature information output by the first downsampling layer as the input of the second 3D convolutional layer, wherein the second 3D convolutional layer can use 3 different 3D convolutional kernels to perform second 3D feature extraction processing to obtain sixth feature information, and the number of feature information obtained by convolution is further increased.
E5, inputting the sixth feature information into a second down-sampling layer of the 3D convolutional neural network for second down-sampling processing to obtain seventh feature information;
and taking the sixth characteristic information output by the second 3D convolutional layer as the input of a second down-sampling layer, and performing second down-sampling processing on the sixth characteristic information output by the second 3D convolutional layer through the second down-sampling layer to obtain seventh characteristic information.
E6, inputting the seventh feature information into the 2D convolutional layer of the 3D convolutional neural network for 2D feature extraction processing to obtain eighth feature information, wherein a convolutional kernel in the 2D convolutional layer is a 2D convolutional kernel;
and taking the seventh feature information output by the second down-sampling layer as the input of the 2D convolution layer, wherein the 2D convolution layer can adopt a 2D convolution kernel to carry out 2D feature extraction processing, and the seventh feature information output by the second down-sampling layer is converted into a form of 1 x N, wherein N is the length of the feature information, so as to obtain eighth feature information.
E7, inputting the eighth characteristic information into a second full-link layer of the 3D convolutional neural network for third classification processing, so as to obtain the probability that the information of the video type corresponds to each label, and using the label with the highest probability as the label of the information of the video type.
And taking the eighth feature information output by the 2D convolutional layer as the input of a second full-connection layer, fully connecting all feature information output by the 2D convolutional layer by using the second full-connection layer to obtain the feature information of the full-connection layer, and classifying the feature information of the full-connection layer by using a linear classifier to obtain the probability that the information of the video type corresponds to each label.
And after the 3D CNN network outputs the probability of each label, taking the label with the highest probability as the label of the video type information.
Further, the method further comprises:
f1, storing the information to be processed with the label as a preset filtering label into a sample information base;
in addition, after the label of the information to be processed is identified, the information to be processed with the label as a preset filtering label can be stored in the sample information base, and training samples in the sample information base are enriched.
And F2, when the preset training condition is met, retraining the neural networks in the neural network group by using the information in the sample information base.
When the preset training condition is met, the information in the sample information base can be used for retraining the neural networks in the neural network group.
The preset training condition can be set according to the actual situation, for example, the filter can be set to filter bad information for a preset number of times, and the filter can also be set to preset conditions such as duration. When the preset training condition is set to be the bad information filtering for the preset times, the preset times can be set according to the actual situation, for example, the preset times is set to be 1, and the retraining of the neural network is performed after the information of the preset filtering label is filtered every time.
The identification precision and the processing speed of the neural network can be improved by storing the information to be processed with the labels as the preset filtering labels as training samples and a retraining mechanism of the neural network.
In the information filtering method provided by this embodiment, when the received information to be processed includes information of at least two data types, the data types included in the information to be processed may be identified, and information of different data types in the information to be processed is input into corresponding neural networks in a neural network group for processing, where the neural network group includes multiple neural networks, and information of multiple data types may be identified through division and cooperation of the multiple neural networks, so as to comprehensively identify and filter information to be filtered of various data types in the information to be processed, thereby solving the problem that the existing information filtering method can only filter data of a single type, and the filtering effect is poor.
When the information to be processed contains the information of the text type, sentence division processing and word division processing can be carried out on the information of the text type, information filtering is carried out by taking the text sentence as a unit, and the occurrence of an excessive filtering condition can be reduced.
When the information to be processed contains the information of the video type, the information of the video type can be segmented according to the preset frame number threshold, and the information is identified by taking the video segment as a unit, so that the processing speed can be improved, and the occurrence of an excessive filtering condition can be reduced.
The information to be processed with the labels of the bad information can be stored in a sample information base as a training sample, and when the preset training condition is met, the neural network can be retrained, so that the recognition accuracy and the processing speed of the neural network are improved.
It should be understood that, the sequence numbers of the steps in the foregoing embodiments do not imply an execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.
Example two:
the second embodiment of the present application provides an information filtering apparatus, which is only shown in relevant parts for convenience of description, and as shown in fig. 2, the information filtering apparatus includes,
an information obtaining module 201, configured to obtain information to be processed, where the information to be processed at least includes information of two data types;
a tag identification module 202, configured to identify a data type included in the to-be-processed information, and input information of different data types in the to-be-processed information into a corresponding neural network in a neural network group for processing, so as to obtain tags of information of different data types in the to-be-processed information;
the information filtering module 203 is configured to filter information in which a tag in the information to be processed is a preset filtering tag.
Further, the tag identification module specifically includes:
the type submodule is used for identifying the data type included in the information to be processed;
the text submodule is used for inputting the information of the text type into a text recognition neural network in a neural network group to obtain a label of the information of the text type when the information to be processed contains the information of the text type;
the picture submodule is used for inputting the information of the image type into an image recognition neural network in a neural network group to obtain a label of the information of the image type when the information to be processed contains the information of the image type;
and the video submodule is used for inputting the information of the video type into a video identification neural network in a neural network group to obtain a label of the information of the video type when the information to be processed contains the information of the video type.
Further, the text sub-module specifically includes:
the processing submodule is used for performing sentence segmentation processing and word segmentation processing on the information of the text type when the information to be processed contains the information of the text type;
and the recognition submodule is used for inputting the text sentences after word segmentation into a text recognition neural network in the neural network group to obtain the labels of the text sentences in the information of the text types.
Further, the picture sub-module specifically includes:
the first characteristic submodule is used for inputting the information of the image type into a convolution layer of a convolution neural network in a neural network group for first characteristic extraction processing when the information to be processed contains the information of the image type, so as to obtain first characteristic information;
the second characteristic submodule is used for inputting the first characteristic information into the pooling layer of the convolutional neural network for second characteristic extraction processing to obtain second characteristic information;
the first classification submodule is used for inputting the second characteristic information into a full connection layer of the convolutional neural network to perform first classification processing to obtain a first classification result;
and the second classification submodule is used for inputting the first classification result into a Softmax layer of the convolutional neural network for second classification processing to obtain the probability that the information of the image type corresponds to each label, and taking the label with the maximum probability as the label of the information of the image type.
Further, the video sub-module specifically includes:
the segmentation submodule is used for carrying out segmentation processing on the information of the video type according to a preset frame number threshold value when the information to be processed contains the information of the video type to obtain at least one video segment;
and the label submodule is used for inputting the video segments into the video identification neural network in the neural network group to obtain the labels of the video segments in the information of the video types.
Further, the video identification neural network is specifically a 3D convolutional neural network, and the video sub-modules specifically include:
the hard connection layer submodule is used for inputting the information of the video type into a hard connection layer of a 3D convolutional neural network in a neural network group for multi-channel feature extraction processing when the information to be processed contains the information of the video type, so as to obtain third feature information;
the first convolution submodule is used for inputting the third feature information into a first 3D convolution layer of the 3D convolutional neural network to perform first 3D feature extraction processing to obtain fourth feature information, wherein a convolution kernel in the first 3D convolution layer is a 3D convolution kernel;
the first sampling submodule is used for inputting the fourth characteristic information into a first downsampling layer of the 3D convolutional neural network for first downsampling processing to obtain fifth characteristic information;
the second convolution submodule is used for inputting the fifth feature information into a second 3D convolution layer of the 3D convolutional neural network to perform second 3D feature extraction processing to obtain sixth feature information, wherein a convolution kernel in the second 3D convolution layer is a 3D convolution kernel;
the second sampling submodule is used for inputting the sixth feature information into a second down-sampling layer of the 3D convolutional neural network to perform second down-sampling processing to obtain seventh feature information;
the third convolution submodule is used for inputting the seventh feature information into a 2D convolution layer of the 3D convolution neural network to perform 2D feature extraction processing to obtain eighth feature information, wherein a convolution kernel in the 2D convolution layer is a 2D convolution kernel;
and the classification output sub-module is used for inputting the eighth characteristic information into a second full connection layer of the 3D convolutional neural network to perform third classification processing, so that the probability that the information of the video type corresponds to each label is obtained, and the label with the maximum probability is used as the label of the information of the video type.
Further, the apparatus further comprises:
the storage module is used for storing the information to be processed with the label as a preset filtering label into a sample information base;
and the retraining module is used for retraining the neural network in the neural network group by using the information in the sample information base when the preset training condition is met.
It should be noted that, for the information interaction, execution process, and other contents between the above-mentioned devices/units, the specific functions and technical effects thereof are based on the same concept as those of the embodiment of the method of the present application, and specific reference may be made to the part of the embodiment of the method, which is not described herein again.
Example three:
fig. 3 is a schematic diagram of a terminal device provided in the third embodiment of the present application. As shown in fig. 3, the terminal device 3 of this embodiment includes: a processor 30, a memory 31 and a computer program 32 stored in said memory 31 and executable on said processor 30. The processor 30, when executing the computer program 32, implements the steps in the above-described information filtering method embodiments, such as the steps S101 to S103 shown in fig. 1. Alternatively, the processor 30, when executing the computer program 32, implements the functions of each module/unit in each device embodiment described above, for example, the functions of the modules 201 to 203 shown in fig. 2.
Illustratively, the computer program 32 may be partitioned into one or more modules/units that are stored in the memory 31 and executed by the processor 30 to accomplish the present application. The one or more modules/units may be a series of computer program instruction segments capable of performing specific functions, which are used to describe the execution process of the computer program 32 in the terminal device 3. For example, the computer program 32 may be divided into an information acquisition module, a tag identification module, and an information filtering module, and each module specifically functions as follows:
the information acquisition module is used for acquiring information to be processed, wherein the information to be processed at least comprises information of two data types;
the tag identification module is used for identifying the data types included in the information to be processed, inputting the information of different data types in the information to be processed into corresponding neural networks in a neural network group for processing, and obtaining tags of the information of different data types in the information to be processed;
and the information filtering module is used for filtering the information of which the label is a preset filtering label in the information to be processed.
The terminal device 3 may be a desktop computer, a notebook, a palm computer, a cloud server, or other computing devices. The terminal device may include, but is not limited to, a processor 30, a memory 31. It will be understood by those skilled in the art that fig. 3 is only an example of the terminal device 3, and does not constitute a limitation to the terminal device 3, and may include more or less components than those shown, or combine some components, or different components, for example, the terminal device may also include an input-output device, a network access device, a bus, etc.
The Processor 30 may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic device, discrete hardware component, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The memory 31 may be an internal storage unit of the terminal device 3, such as a hard disk or a memory of the terminal device 3. The memory 31 may also be an external storage device of the terminal device 3, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like, which are provided on the terminal device 3. Further, the memory 31 may also include both an internal storage unit and an external storage device of the terminal device 3. The memory 31 is used for storing the computer program and other programs and data required by the terminal device. The memory 31 may also be used to temporarily store data that has been output or is to be output.
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-mentioned division of the functional units and modules is illustrated, and in practical applications, the above-mentioned function distribution may be performed by different functional units and modules according to needs, that is, the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-mentioned functions. Each functional unit and module in the embodiments may be integrated in one processing unit, or each unit may exist alone physically, or two or more units are integrated in one unit, and the integrated unit may be implemented in a form of hardware, or in a form of software functional unit. In addition, specific names of the functional units and modules are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present application. The specific working processes of the units and modules in the system may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and reference may be made to the related descriptions of other embodiments for parts that are not described or illustrated in a certain embodiment.
Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus/terminal device and method may be implemented in other ways. For example, the above-described embodiments of the apparatus/terminal device are merely illustrative, and for example, the division of the modules or units is only one logical division, and there may be other divisions when actually implemented, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated modules/units, if implemented in the form of software functional units and sold or used as separate products, may be stored in a computer readable storage medium. Based on such understanding, all or part of the flow in the method of the embodiments described above can be realized by a computer program, which can be stored in a computer-readable storage medium and can realize the steps of the embodiments of the methods described above when the computer program is executed by a processor. Wherein the computer program comprises computer program code, which may be in the form of source code, object code, an executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, usb disk, removable hard disk, magnetic disk, optical disk, computer Memory, Read-Only Memory (ROM), Random Access Memory (RAM), electrical carrier wave signals, telecommunications signals, software distribution medium, and the like. It should be noted that the computer readable medium may contain content that is subject to appropriate increase or decrease as required by legislation and patent practice in jurisdictions, for example, in some jurisdictions, computer readable media does not include electrical carrier signals and telecommunications signals as is required by legislation and patent practice.
The above-mentioned embodiments are only used for illustrating the technical solutions of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not substantially depart from the spirit and scope of the embodiments of the present application and are intended to be included within the scope of the present application.

Claims (10)

1. An information filtering method, comprising:
acquiring information to be processed, wherein the information to be processed at least comprises information of two data types;
identifying the data types included in the information to be processed, inputting the information of different data types in the information to be processed into corresponding neural networks in a neural network group for processing, and obtaining tags of the information of different data types in the information to be processed;
and filtering the information of which the label is a preset filtering label in the information to be processed.
2. The information filtering method according to claim 1, wherein the identifying the data type included in the information to be processed, and inputting information of different data types in the information to be processed into corresponding neural networks in a neural network group for processing, and obtaining the labels of information of different data types in the information to be processed specifically includes:
identifying the data type included in the information to be processed;
when the information to be processed contains the information of the text type, inputting the information of the text type into a text recognition neural network in a neural network group to obtain a label of the information of the text type;
when the information to be processed contains information of the image type, inputting the information of the image type into an image recognition neural network in a neural network group to obtain a label of the information of the image type;
and when the information to be processed contains the information of the video type, inputting the information of the video type into a video identification neural network in a neural network group to obtain a label of the information of the video type.
3. The information filtering method according to claim 2, wherein when the information to be processed contains information of a text type, inputting the information of the text type into a text recognition neural network in a neural network group, and obtaining the label of the information of the text type specifically includes:
when the information to be processed contains the information of the text type, performing sentence segmentation processing and word segmentation processing on the information of the text type;
and inputting the text sentences after word segmentation into a text recognition neural network in the neural network group to obtain the labels of the text sentences in the information of the text types.
4. The information filtering method according to claim 2, wherein when the information to be processed includes information of an image type, inputting the information of the image type into an image recognition neural network in a neural network group, and obtaining the label of the information of the image type specifically includes:
when the information to be processed contains information of an image type, inputting the information of the image type into a convolution layer of a convolution neural network in a neural network group for first feature extraction processing to obtain first feature information;
inputting the first feature information into a pooling layer of the convolutional neural network for second feature extraction processing to obtain second feature information;
inputting the second characteristic information into a first full-connection layer of the convolutional neural network to perform first classification processing to obtain a first classification result;
and inputting the first classification result into a Softmax layer of the convolutional neural network for second classification processing to obtain the probability that the information of the image type corresponds to each label, and taking the label with the maximum probability as the label of the information of the image type.
5. The information filtering method according to claim 2, wherein when the information to be processed includes information of a video type, the inputting the information of the video type into a video recognition neural network in a neural network group, and the obtaining the label of the information of the video type specifically includes:
when the information to be processed contains video type information, performing segmentation processing on the video type information according to a preset frame number threshold to obtain at least one video segment;
and inputting the video segments into a video identification neural network in the neural network group to obtain the labels of all the video segments in the information of the video types.
6. The information filtering method according to claim 2, wherein the video recognition neural network is specifically a 3D convolutional neural network, and the entering the information of the video type into the video recognition neural networks in the neural network group when the information to be processed contains the information of the video type specifically includes:
when the information to be processed contains video type information, inputting the video type information into a hard connection layer of a 3D convolutional neural network in a neural network group for multi-channel feature extraction processing to obtain third feature information;
inputting the third feature information into a first 3D convolutional layer of the 3D convolutional neural network for first 3D feature extraction processing to obtain fourth feature information, wherein a convolution kernel in the first 3D convolutional layer is a 3D convolution kernel;
inputting the fourth feature information into a first downsampling layer of the 3D convolutional neural network for first downsampling processing to obtain fifth feature information;
inputting the fifth feature information into a second 3D convolutional layer of the 3D convolutional neural network for second 3D feature extraction processing to obtain sixth feature information, wherein a convolution kernel in the second 3D convolutional layer is a 3D convolution kernel;
inputting the sixth feature information into a second down-sampling layer of the 3D convolutional neural network for second down-sampling processing to obtain seventh feature information;
inputting the seventh feature information into a 2D convolutional layer of the 3D convolutional neural network for 2D feature extraction processing to obtain eighth feature information, wherein a convolution kernel in the 2D convolutional layer is a 2D convolution kernel;
inputting the eighth characteristic information into a second full-connection layer of the 3D convolutional neural network for third classification processing to obtain the probability that the information of the video type corresponds to each label, and taking the label with the maximum probability as the label of the information of the video type.
7. The information filtering method of any one of claims 1 to 6, wherein the method further comprises:
storing the information to be processed with the label as a preset filtering label into a sample information base;
and when the preset training condition is met, retraining the neural networks in the neural network group by using the information in the sample information base.
8. An information filtering device, comprising:
the information acquisition module is used for acquiring information to be processed, wherein the information to be processed at least comprises information of two data types;
the tag identification module is used for identifying the data types included in the information to be processed, inputting the information of different data types in the information to be processed into corresponding neural networks in a neural network group for processing, and obtaining tags of the information of different data types in the information to be processed;
and the information filtering module is used for filtering the information of which the label is a preset filtering label in the information to be processed.
9. A terminal device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the steps of the method according to any of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 7.
CN201910614219.3A 2019-07-08 2019-07-08 Information filtering method and device and terminal equipment Pending CN112199564A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910614219.3A CN112199564A (en) 2019-07-08 2019-07-08 Information filtering method and device and terminal equipment

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910614219.3A CN112199564A (en) 2019-07-08 2019-07-08 Information filtering method and device and terminal equipment

Publications (1)

Publication Number Publication Date
CN112199564A true CN112199564A (en) 2021-01-08

Family

ID=74004640

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910614219.3A Pending CN112199564A (en) 2019-07-08 2019-07-08 Information filtering method and device and terminal equipment

Country Status (1)

Country Link
CN (1) CN112199564A (en)

Citations (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101276417A (en) * 2008-04-17 2008-10-01 上海交通大学 Method for filtering internet cartoon medium rubbish information based on content
CN101281521A (en) * 2007-04-05 2008-10-08 中国科学院自动化研究所 Method and system for filtering sensitive web page based on multiple classifier amalgamation
CN101340308A (en) * 2008-08-19 2009-01-07 翁时锋 Network rubbish information filtering architecture, Network rubbish information cleaning system and method thereof
US20100246960A1 (en) * 2008-12-31 2010-09-30 Bong Gyoune Kim Image Based Spam Blocking
CN105335856A (en) * 2014-08-10 2016-02-17 高康 Radio-frequency identification based smart city realization method and system
CN107547555A (en) * 2017-09-11 2018-01-05 北京匠数科技有限公司 A kind of web portal security monitoring method and device
CN107809410A (en) * 2016-09-09 2018-03-16 腾讯科技(深圳)有限公司 Information filtering method and device
CN108170845A (en) * 2018-01-17 2018-06-15 腾讯音乐娱乐科技(深圳)有限公司 Multimedia data processing method, device and storage medium
CN108235003A (en) * 2018-03-19 2018-06-29 天津大学 Three-dimensional video quality evaluation method based on 3D convolutional neural networks
CN108920642A (en) * 2018-06-29 2018-11-30 武汉圣大东高科技有限公司 Campus information delivery system
CN109829443A (en) * 2019-02-23 2019-05-31 重庆邮电大学 Video behavior recognition methods based on image enhancement Yu 3D convolutional neural networks
CN109902223A (en) * 2019-01-14 2019-06-18 中国科学院信息工程研究所 A kind of harmful content filter method based on multi-modal information feature

Patent Citations (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101281521A (en) * 2007-04-05 2008-10-08 中国科学院自动化研究所 Method and system for filtering sensitive web page based on multiple classifier amalgamation
CN101276417A (en) * 2008-04-17 2008-10-01 上海交通大学 Method for filtering internet cartoon medium rubbish information based on content
CN101340308A (en) * 2008-08-19 2009-01-07 翁时锋 Network rubbish information filtering architecture, Network rubbish information cleaning system and method thereof
US20100246960A1 (en) * 2008-12-31 2010-09-30 Bong Gyoune Kim Image Based Spam Blocking
CN105335856A (en) * 2014-08-10 2016-02-17 高康 Radio-frequency identification based smart city realization method and system
CN107809410A (en) * 2016-09-09 2018-03-16 腾讯科技(深圳)有限公司 Information filtering method and device
CN107547555A (en) * 2017-09-11 2018-01-05 北京匠数科技有限公司 A kind of web portal security monitoring method and device
CN108170845A (en) * 2018-01-17 2018-06-15 腾讯音乐娱乐科技(深圳)有限公司 Multimedia data processing method, device and storage medium
CN108235003A (en) * 2018-03-19 2018-06-29 天津大学 Three-dimensional video quality evaluation method based on 3D convolutional neural networks
CN108920642A (en) * 2018-06-29 2018-11-30 武汉圣大东高科技有限公司 Campus information delivery system
CN109902223A (en) * 2019-01-14 2019-06-18 中国科学院信息工程研究所 A kind of harmful content filter method based on multi-modal information feature
CN109829443A (en) * 2019-02-23 2019-05-31 重庆邮电大学 Video behavior recognition methods based on image enhancement Yu 3D convolutional neural networks

Similar Documents

Publication Publication Date Title
CN109471938B (en) Text classification method and terminal
CN110188202B (en) Training method and device of semantic relation recognition model and terminal
CN108874776B (en) Junk text recognition method and device
CN112561080B (en) Sample screening method, sample screening device and terminal equipment
CN109684476B (en) Text classification method, text classification device and terminal equipment
CN112288018B (en) Training method of character recognition network, character recognition method and device
CN110019882B (en) Advertisement creative classification method and system
CN111126514A (en) Image multi-label classification method, device, equipment and medium
US11966455B2 (en) Text partitioning method, text classifying method, apparatus, device and storage medium
KR20200029351A (en) Sample processing method and device, related apparatus and storage medium
CN109697231A (en) A kind of display methods, system, storage medium and the processor of case document
CN113435499A (en) Label classification method and device, electronic equipment and storage medium
CN113780365A (en) Sample generation method and device
CN113704508A (en) Multimedia information identification method and device, electronic equipment and storage medium
CN111198943B (en) Resume screening method and device and terminal equipment
CN112199564A (en) Information filtering method and device and terminal equipment
CN116503596A (en) Picture segmentation method, device, medium and electronic equipment
CN107071553B (en) Method, device and computer readable storage medium for modifying video and voice
CN112818984B (en) Title generation method, device, electronic equipment and storage medium
CN113033682B (en) Video classification method, device, readable medium and electronic equipment
CN112633394A (en) Intelligent user label determination method, terminal equipment and storage medium
CN111767710B (en) Indonesia emotion classification method, device, equipment and medium
CN114186140A (en) Social interaction information processing method and device, electronic equipment and storage medium
CN111242449A (en) Enterprise information loss prediction method
CN116758565B (en) OCR text restoration method, equipment and storage medium based on decision tree

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination