CN116955934A - Network transmission data noise reduction method and device, computing equipment and storage medium - Google Patents
Network transmission data noise reduction method and device, computing equipment and storage medium Download PDFInfo
- Publication number
- CN116955934A CN116955934A CN202311222183.7A CN202311222183A CN116955934A CN 116955934 A CN116955934 A CN 116955934A CN 202311222183 A CN202311222183 A CN 202311222183A CN 116955934 A CN116955934 A CN 116955934A
- Authority
- CN
- China
- Prior art keywords
- data
- noise reduction
- noise
- outputting
- voice
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000005540 biological transmission Effects 0.000 title claims abstract description 51
- 238000000034 method Methods 0.000 title claims abstract description 49
- 238000012545 processing Methods 0.000 claims abstract description 25
- 238000013136 deep learning model Methods 0.000 claims abstract description 15
- 238000007405 data analysis Methods 0.000 claims abstract description 10
- 238000003058 natural language processing Methods 0.000 claims description 16
- 238000004891 communication Methods 0.000 claims description 15
- 238000005516 engineering process Methods 0.000 claims description 12
- 238000007781 pre-processing Methods 0.000 claims description 8
- 238000002372 labelling Methods 0.000 claims description 3
- 238000013135 deep learning Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 4
- 238000010801 machine learning Methods 0.000 description 4
- 241000282326 Felis catus Species 0.000 description 3
- 241000699666 Mus <mouse, genus> Species 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 2
- 238000004140 cleaning Methods 0.000 description 2
- 238000004590 computer program Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 239000004973 liquid crystal related substance Substances 0.000 description 2
- 241000699670 Mus sp. Species 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 238000011022 operating instruction Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/10—Pre-processing; Data cleansing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/23—Clustering techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2218/00—Aspects of pattern recognition specially adapted for signal processing
- G06F2218/02—Preprocessing
- G06F2218/04—Denoising
Abstract
The invention discloses a network transmission data noise reduction method, a device, a computing device and a storage medium, wherein the method comprises the following steps: acquiring data to be transmitted, and performing data analysis processing on the data to be transmitted to divide data attributes; inputting the picture data into a deep learning model for noise identification aiming at the picture data attribute, and outputting first noise reduction data after data noise reduction according to the identification result; for the voice data attribute, dividing noise signals according to the clustering operation result, and outputting second noise reduction data after removing noise according to the division result; aiming at the character data attribute, carrying out data noise reduction processing on the character data and then outputting third noise reduction data; and integrating and outputting the first noise reduction data, the second noise reduction data and the third noise reduction data, and then carrying out data network transmission. According to the data transmission method and the data transmission device, the data to be transmitted is preprocessed, the types are divided, and the corresponding noise reduction methods are applied according to different types, so that the data noise reduction is realized more accurately for network transmission, and the accuracy of data transmission is improved.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a method, an apparatus, a computing device, and a storage medium for noise reduction of network transmission data.
Background
Along with the development of the internet, the network data transmission application is more and more widespread, and the variety of transmission data is also five-in-eight, in order to guarantee transmission quality, noise processing is usually performed through a noise reduction algorithm in the network data transmission process, namely interference data in a data set is removed, so that the data transmission quality is improved.
However, in the prior art, for network data transmission, a single algorithm is generally applied to perform data denoising. The data to be transmitted generally includes multiple types of data, and if the same algorithm is applied to different types of data, a misprocessing or unprocessed phenomenon usually occurs, resulting in poor transmission quality.
Disclosure of Invention
The present invention has been made in view of the above problems, and it is an object of the present invention to provide a network transmission data noise reduction method, apparatus, computing device and storage medium that overcomes or at least partially solves the above problems.
According to one aspect of the present invention, there is provided a network transmission data denoising method, including:
acquiring data to be transmitted, and dividing data attributes after data analysis processing is carried out on the data to be transmitted; wherein the data attributes include picture data attributes, voice data attributes and/or text data attributes;
inputting the picture data into a deep learning model for noise recognition according to the object to be recognized according to the picture data attribute, and outputting first noise reduction data after data noise reduction according to the recognition result;
aiming at the voice data attribute, converting voice data into a signal data set, inputting the signal data into a clustering model for clustering operation, dividing noise signals according to a clustering operation result, and outputting second noise reduction data after noise is removed according to the division result;
for the character data attribute, applying a natural language processing technology to the character data to perform data noise reduction processing and then outputting third noise reduction data;
and integrating and outputting the first noise reduction data, the second noise reduction data and the third noise reduction data, and then carrying out data network transmission.
According to another aspect of the present invention, there is provided a network transmission data noise reduction apparatus, including:
the data preprocessing module is used for acquiring data to be transmitted, and dividing data attributes after data analysis processing is carried out on the data to be transmitted; wherein the data attributes include picture data attributes, voice data attributes and/or text data attributes;
the first noise reduction module is used for inputting the picture data into the deep learning model according to the attribute of the picture data, carrying out noise recognition according to the object to be recognized, carrying out data noise reduction according to the recognition result, and outputting first noise reduction data;
the second noise reduction module is used for converting voice data into a signal data set aiming at the voice data attribute, inputting the signal data into a clustering model for clustering operation, dividing noise signals according to a clustering operation result, and outputting second noise reduction data after removing noise according to a division result;
the third noise reduction module is used for applying a natural language processing technology to the text data for data noise reduction aiming at the text data attribute and outputting third noise reduction data;
and the transmission module is used for carrying out data network transmission after integrating and outputting the first noise reduction data, the second noise reduction data and the third noise reduction data.
According to yet another aspect of the present invention, there is provided a computing device comprising: the device comprises a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface are communicated with each other through the communication bus;
the memory is used for storing at least one executable instruction, and the executable instruction enables the processor to execute the operation corresponding to the network transmission data noise reduction method.
According to still another aspect of the present invention, there is provided a computer storage medium having stored therein at least one executable instruction for causing a processor to perform operations corresponding to a network transmission data denoising method as described above.
According to the network transmission data noise reduction method, the device, the computing equipment and the storage medium, data attributes are divided after data analysis processing is carried out on the data to be transmitted by acquiring the data to be transmitted; wherein the data attributes include picture data attributes, voice data attributes and/or text data attributes; inputting the picture data into a deep learning model for noise recognition according to the object to be recognized according to the picture data attribute, and outputting first noise reduction data after data noise reduction according to the recognition result; aiming at the voice data attribute, converting voice data into a signal data set, inputting the signal data into a clustering model for clustering operation, dividing noise signals according to a clustering operation result, and outputting second noise reduction data after noise is removed according to the division result; for the character data attribute, applying a natural language processing technology to the character data to perform data noise reduction processing and then outputting third noise reduction data; and integrating and outputting the first noise reduction data, the second noise reduction data and the third noise reduction data, and then carrying out data network transmission. According to the data preprocessing method, the data to be transmitted is preprocessed and classified, and the corresponding noise reduction method is applied according to different types, so that the network transmission is performed after the data is more accurately and accurately noise reduction, the accuracy of the data transmission is improved, and the user experience is improved.
The foregoing description is only an overview of the present invention, and is intended to be implemented in accordance with the teachings of the present invention in order that the same may be more clearly understood and to make the same and other objects, features and advantages of the present invention more readily apparent.
Drawings
Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention. Also, like reference numerals are used to designate like parts throughout the figures. In the drawings:
fig. 1 shows a flowchart of a network transmission data denoising method according to an embodiment of the present invention;
fig. 2 is a schematic structural diagram of a network transmission data noise reduction device according to an embodiment of the present invention;
FIG. 3 illustrates a schematic diagram of a computing device provided by an embodiment of the present invention.
Detailed Description
Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.
Fig. 1 shows a flowchart of an embodiment of a network transmission data denoising method according to the present invention, as shown in fig. 1, the method includes the following steps:
step S110: and acquiring data to be transmitted, and dividing data attributes after data analysis processing is carried out on the data to be transmitted.
In the step, data preprocessing is firstly carried out on data to be transmitted, the data to be transmitted is divided according to data attributes, wherein the data attributes comprise picture data attributes, voice data attributes and/or text data attributes; it should be noted that, if the data to be transmitted may include one or more of the three data attributes, if the data to be transmitted is only one or two data attributes, only the processing mode of the corresponding data attribute may be applied, and in this embodiment, only an application scenario in which the data to be transmitted includes all three data is given.
Step S120: and inputting the picture data into a deep learning model for noise recognition according to the object to be recognized according to the picture data attribute, and outputting first noise reduction data after data noise reduction according to the recognition result.
Deep Learning (DL) is a new research direction in the field of Machine Learning (ML), which was introduced to Machine Learning to bring it closer to the original goal-artificial intelligence (AI, artificial Intelligence). Deep learning is the inherent regularity and presentation hierarchy of learning sample data, and the information obtained during such learning is helpful in interpreting data such as text, images and sounds. Its final goal is to have the machine have analytical learning capabilities like a person, and to recognize text, image, and sound data. Deep learning is a complex machine learning algorithm that achieves far greater results in terms of speech and image recognition than prior art. Deep learning does not require us to extract features by himself, but automatically filters data, and automatically extracts high-dimensional features of the data, i.e., high-dimensional features of the picture data in this embodiment.
In an alternative manner, step S120 further includes: inputting data to be transmitted into a deep learning model, learning and extracting objects to be identified, labeling the objects to be identified one by one, defining data outside the labels as noise data, applying a noise reduction algorithm to the noise data to carry out data noise reduction, and outputting the picture data subjected to noise reduction as first noise reduction data; wherein the object to be identified is determined according to the user demand.
Specifically, the picture data itself carries some information which does not help the task; for example, a picture contains a cat and a mouse, the user needs the cat, the cat is an object to be identified, and the mouse and other backgrounds are noise data; the deep learning device needs to learn and identify the object to be identified, but sometimes the deep learning classifier considers mice as important information, and data noise reduction is affected; the noise reduction algorithm can thus be incorporated into a deep learning model, in particular, a soft-broad function can be added to the residual network res net, which is the core step of many noise reduction algorithms.
In an alternative manner, step S120 further includes: the soft threshold function applies a noise reduction algorithm with the formula:
；
wherein Y is first noise reduction data, and X is original picture data; z is noise data after the label.
Step S130: aiming at the voice data attribute, converting the voice data into a signal data set, inputting the signal data into a clustering model for clustering operation, dividing noise signals according to a clustering operation result, and outputting second noise reduction data after noise is removed according to the division result.
In an alternative manner, step S130 furtherComprising the following steps: after carrying out feature vectorization on voice data, abstracting the voice data into a plurality of voice signals; constructing a signal data set according to a plurality of voice signals) The method comprises the steps of carrying out a first treatment on the surface of the Wherein (1)>Representing the speech signal.
In an alternative manner, step S130 further includes: inputting the signal data into a clustering model for clustering operation to form a plurality of clustering clusters;
defining points outside the cluster as outliers;
determining signal data corresponding to the outliers as noise signals;
and after removing the noise signal, recombining and outputting the signal data to obtain second noise reduction data.
In an alternative manner, step S130 further includes: the clustering operation is specifically as follows:
step 1: randomly initializing k voice signals as initial cluster centroids;
step 2: signal data set @) Each point in (a) is assigned to one cluster;
step 3: the Euclidean distance between each point and the cluster centroid is calculated, and is distributed to the cluster corresponding to the cluster centroid closest to the Euclidean distance;
step 4: updating cluster mass centers; wherein, each cluster centroid is updated as the average value of all points of the cluster;
step 5: and repeatedly iterating the step 2-3 until the specified termination condition is reached.
In general, the usual termination conditions are: reaching the appointed iteration times; the cluster centroid no longer changes significantly, i.e., converges and/or reaches a minimum error square.
Step S140: and aiming at the character data attribute, applying a natural language processing technology to the character data to perform data noise reduction processing, and outputting third noise reduction data.
In an alternative manner, step S140 further includes: and (3) applying a natural language processing technology to the text data, respectively deleting useless data and correcting error data, and outputting to obtain third noise reduction data.
In this step, for text data, a Natural Language Processing (NLP) technique is applied to perform data preprocessing, where the NLP is mainly used for cleaning chinese text data, and the deletion of useless data may include: deleting useless texts with shorter length, removing redundant blank spaces in characters, deleting punctuation marks which appear continuously, deleting data lines without Chinese characters, deleting special symbols, and deleting according to the condition of each text data without manually inputting a plurality of characters each time and deleting after matching; for example, for txt format data, the special symbols appearing in each txt file are different, so that each txt file is stored in a txt file with the same name as the original text data, each special symbol is stored row by row, and for the symbols needing to be deleted in the subsequent cleaning process, reservation is given in the txt file storing the special symbols, and the subsequent reading and processing are facilitated; the method supports batch operation, reads and stores txt files of special symbols to be deleted, and deletes the txt files in text data; further, the method also comprises the step of correcting data of wrongly written characters and the like obvious in the text data.
Step S150: and integrating and outputting the first noise reduction data, the second noise reduction data and the third noise reduction data, and then carrying out data network transmission.
By adopting the method of the embodiment, the data to be transmitted is obtained, and after the data analysis processing is carried out on the data to be transmitted, the data attribute is divided; wherein the data attributes include picture data attributes, voice data attributes and/or text data attributes; inputting the picture data into a deep learning model for noise recognition according to the object to be recognized according to the picture data attribute, and outputting first noise reduction data after data noise reduction according to the recognition result; aiming at the voice data attribute, converting voice data into a signal data set, inputting the signal data into a clustering model for clustering operation, dividing noise signals according to a clustering operation result, and outputting second noise reduction data after noise is removed according to the division result; for the character data attribute, applying a natural language processing technology to the character data to perform data noise reduction processing and then outputting third noise reduction data; and integrating and outputting the first noise reduction data, the second noise reduction data and the third noise reduction data, and then carrying out data network transmission. According to the method, the data to be transmitted is preprocessed, the types are divided, and the corresponding noise reduction methods are applied according to different types, so that the network transmission is performed after the data noise reduction is more accurately and accurately realized, the accuracy of the data transmission is improved, and the user experience is improved.
Fig. 2 is a schematic structural diagram of an embodiment of a network transmission data noise reduction device according to the present invention. As shown in fig. 2, the apparatus includes: a data preprocessing module 210, a first noise reduction module 220, a second noise reduction module 230, a third noise reduction module 240, and a transmission module 250;
the data preprocessing module 210 is configured to obtain data to be transmitted, and divide data attributes after performing data analysis processing on the data to be transmitted; wherein the data attributes include picture data attributes, voice data attributes and/or text data attributes;
the first noise reduction module 220 is configured to input, for the attribute of the picture data, the picture data into the deep learning model, perform noise recognition according to the object to be recognized, perform data noise reduction according to the recognition result, and output first noise reduction data;
the second noise reduction module 230 is configured to convert the voice data into a signal data set according to the voice data attribute, input the signal data into a clustering model for clustering operation, divide the noise signal according to the clustering operation result, and output second noise reduction data after removing noise according to the division result;
the third noise reduction module 240 is configured to apply a natural language processing technique to the text data for the text data attribute, and output third noise reduction data after performing data noise reduction processing;
the transmission module 250 is configured to perform data network transmission according to the integrated output of the first noise reduction data, the second noise reduction data, and the third noise reduction data.
In an alternative manner, the first noise reduction module 220 is further configured to: inputting data to be transmitted into a deep learning model, learning and extracting objects to be identified, labeling the objects to be identified one by one, defining data outside the labels as noise data, applying a noise reduction algorithm to the noise data to carry out data noise reduction, and outputting the picture data subjected to noise reduction as first noise reduction data; wherein the object to be identified is determined according to the user demand.
In an alternative approach, the formula for the noise reduction algorithm is:
；
wherein Y is first noise reduction data, and X is original picture data; z is noise data after the label.
In an alternative manner, the second noise reduction module 230 is further configured to: after carrying out feature vectorization on voice data, abstracting the voice data into a plurality of voice signals; constructing a signal data set according to a plurality of voice signals) The method comprises the steps of carrying out a first treatment on the surface of the Wherein, the liquid crystal display device comprises a liquid crystal display device,representing the speech signal.
In an alternative manner, the second noise reduction module 230 is further configured to: inputting the signal data into a clustering model for clustering operation to form a plurality of clustering clusters; defining points outside the cluster as outliers; determining signal data corresponding to the outliers as noise signals; and after removing the noise signal, recombining and outputting the signal data to obtain second noise reduction data.
In an alternative way, the clustering operation is specifically:
step 1: randomly initializing k voice signals as initial cluster centroids;
step 2: signal data set @) Each point in (a) is assigned to one cluster;
step 3: the Euclidean distance between each point and the cluster centroid is calculated, and is distributed to the cluster corresponding to the cluster centroid closest to the Euclidean distance;
step 4: updating cluster mass centers; wherein, each cluster centroid is updated as the average value of all points of the cluster;
step 5: and repeatedly iterating the step 2-3 until the specified termination condition is reached.
In an alternative manner, the third noise reduction module 240 is further configured to: and (3) applying a natural language processing technology to the text data, respectively deleting useless data and correcting error data, and outputting to obtain third noise reduction data.
By adopting the device of the embodiment, the data to be transmitted is obtained, and after the data analysis processing is carried out on the data to be transmitted, the data attribute is divided; wherein the data attributes include picture data attributes, voice data attributes and/or text data attributes; inputting the picture data into a deep learning model for noise recognition according to the object to be recognized according to the picture data attribute, and outputting first noise reduction data after data noise reduction according to the recognition result; aiming at the voice data attribute, converting voice data into a signal data set, inputting the signal data into a clustering model for clustering operation, dividing noise signals according to a clustering operation result, and outputting second noise reduction data after noise is removed according to the division result; for the character data attribute, applying a natural language processing technology to the character data to perform data noise reduction processing and then outputting third noise reduction data; and integrating and outputting the first noise reduction data, the second noise reduction data and the third noise reduction data, and then carrying out data network transmission. The device performs data preprocessing on the data to be transmitted, divides the types, and applies the corresponding noise reduction method according to different types, so that the network transmission is performed after the data noise reduction is more accurately and accurately realized, the accuracy of the data transmission is improved, and the user experience is improved.
The embodiment of the invention provides a nonvolatile computer storage medium, which stores at least one executable instruction, and the computer executable instruction can execute a network transmission data denoising method in any method embodiment.
FIG. 3 illustrates a schematic diagram of an embodiment of a computing device of the present invention, and the embodiments of the present invention are not limited to a particular implementation of the computing device.
As shown in fig. 3, the computing device may include:
a processor (processor), a communication interface (Communications Interface), a memory (memory), and a communication bus.
Wherein: the processor, communication interface, and memory communicate with each other via a communication bus. A communication interface for communicating with network elements of other devices, such as clients or other servers, etc. The processor is configured to execute a program, and may specifically execute relevant steps in the foregoing embodiment of a network transmission data denoising method.
In particular, the program may include program code including computer-operating instructions.
The processor may be a central processing unit, CPU, or specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement embodiments of the present invention. The one or more processors included by the server may be the same type of processor, such as one or more CPUs; but may also be different types of processors such as one or more CPUs and one or more ASICs.
And the memory is used for storing programs. The memory may comprise high-speed RAM memory or may further comprise non-volatile memory, such as at least one disk memory.
The program may be specifically operative to cause the processor to:
the algorithms or displays presented herein are not inherently related to any particular computer, virtual system, or other apparatus. Various general-purpose systems may also be used with the teachings herein. The required structure for a construction of such a system is apparent from the description above. In addition, embodiments of the present invention are not directed to any particular programming language. It will be appreciated that the teachings of the present invention described herein may be implemented in a variety of programming languages, and the above description of specific languages is provided for disclosure of enablement and best mode of the present invention.
In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.
Similarly, it should be appreciated that in the above description of exemplary embodiments of the invention, various features of the embodiments of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. However, the disclosed method should not be construed as reflecting the intention that: i.e., the claimed invention requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of this invention.
Those skilled in the art will appreciate that the modules in the apparatus of the embodiments may be adaptively changed and disposed in one or more apparatuses different from the embodiments. The modules or units or components of the embodiments may be combined into one module or unit or component and, furthermore, they may be divided into a plurality of sub-modules or sub-units or sub-components. Any combination of all features disclosed in this specification (including any accompanying claims, abstract and drawings), and all of the processes or units of any method or apparatus so disclosed, may be used in combination, except insofar as at least some of such features and/or processes or units are mutually exclusive. Each feature disclosed in this specification (including any accompanying claims, abstract and drawings), may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.
Furthermore, those skilled in the art will appreciate that while some embodiments herein include some features but not others included in other embodiments, combinations of features of different embodiments are meant to be within the scope of the invention and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.
Various component embodiments of the invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art will appreciate that some or all of the functionality of some or all of the components according to embodiments of the present invention may be implemented in practice using a microprocessor or Digital Signal Processor (DSP). The present invention can also be implemented as an apparatus or device program (e.g., a computer program and a computer program product) for performing a portion or all of the methods described herein. Such a program embodying the present invention may be stored on a computer readable medium, or may have the form of one or more signals. Such signals may be downloaded from an internet website, provided on a carrier signal, or provided in any other form.
It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by one and the same item of hardware. The use of the words first, second, third, etc. do not denote any order. These words may be interpreted as names. The steps in the above embodiments should not be construed as limiting the order of execution unless specifically stated.
Claims (10)
1. A method for noise reduction of network transmission data, comprising:
acquiring data to be transmitted, and dividing data attributes after data analysis processing is carried out on the data to be transmitted; wherein the data attributes include at least: one or more of picture data attributes, voice data attributes, and/or text data attributes;
inputting the picture data into a deep learning model for noise identification according to an object to be identified according to the picture data attribute, and outputting first noise reduction data after data noise reduction according to an identification result;
aiming at the voice data attribute, converting the voice data into a signal data set, inputting the signal data into a clustering model for clustering operation, dividing noise signals according to the clustering operation result, and outputting second noise reduction data after removing noise according to the division result;
for the character data attribute, performing data noise reduction processing on the character data by using a natural language processing technology, and outputting third noise reduction data;
and integrating and outputting the first noise reduction data, the second noise reduction data and the third noise reduction data, and then carrying out data network transmission.
2. The method according to claim 1, wherein inputting the data to be transmitted into a deep learning model for noise recognition according to the object to be recognized for the picture data attribute, and outputting the first noise reduction data after data noise reduction according to the recognition result further comprises:
inputting the data to be transmitted into a deep learning model, learning and extracting objects to be identified, labeling the objects to be identified one by one, defining the data outside the labels as noise data, applying a noise reduction algorithm to the noise data to reduce the data, and outputting the image data subjected to noise reduction as first noise reduction data; wherein the object to be identified is determined according to the user demand.
3. The method of claim 2, wherein the noise reduction algorithm is formulated as:
；
wherein Y is first noise reduction data, and X is original picture data; z is noise data after the label.
4. The method of claim 1, wherein said converting said voice data into a signal data set for said voice data attribute further comprises:
after carrying out feature vectorization on the voice data, abstracting the voice data into a plurality of voice signals;
constructing a signal data set according to the plurality of voice signals) The method comprises the steps of carrying out a first treatment on the surface of the Wherein (1)>Representing the speech signal.
5. The method of claim 4, wherein inputting the signal data into a clustering model for clustering, dividing the noise signal according to the clustering result, and removing noise according to the dividing result to output second noise reduction data further comprises:
inputting the signal data into a clustering model for clustering operation to form a plurality of clusters;
defining points outside the cluster as outliers;
determining the signal data corresponding to the outliers as noise signals;
and after the noise signal is removed, recombining and outputting the signal data to obtain second noise reduction data.
6. The method according to claim 5, wherein the clustering operation is specifically:
step 1: randomly initializing k voice signals as initial cluster centroids;
step 2: signal data set @) Each point in (a) is assigned to one cluster;
step 3: the Euclidean distance between each point and the cluster centroid is calculated, and is distributed to the cluster corresponding to the cluster centroid closest to the Euclidean distance;
step 4: updating cluster mass centers; wherein, each cluster centroid is updated as the average value of all points of the cluster;
step 5: and repeatedly iterating the step 2-3 until the specified termination condition is reached.
7. The method of claim 1, wherein outputting third noise reduction data after applying a natural language processing technique to the text data for the text data attribute to perform data noise reduction processing further comprises:
and (3) applying a natural language processing technology to the text data, respectively deleting useless data and correcting error data, and outputting to obtain third noise reduction data.
8. A network transmission data noise reduction device, comprising:
the data preprocessing module is used for acquiring data to be transmitted, and dividing data attributes after data analysis processing is carried out on the data to be transmitted; wherein the data attributes comprise picture data attributes, voice data attributes and/or text data attributes;
the first noise reduction module is used for inputting the picture data into a deep learning model according to the attribute of the picture data, carrying out noise recognition according to the object to be recognized, carrying out data noise reduction according to the recognition result, and then outputting first noise reduction data;
the second noise reduction module is used for converting the voice data into a signal data set aiming at the voice data attribute, inputting the signal data into a clustering model for clustering operation, dividing noise signals according to the clustering operation result, and outputting second noise reduction data after removing noise according to the division result;
the third noise reduction module is used for applying a natural language processing technology to the text data aiming at the text data attribute to perform data noise reduction processing and then outputting third noise reduction data;
and the transmission module is used for carrying out data network transmission after integrating and outputting the first noise reduction data, the second noise reduction data and the third noise reduction data.
9. A computing device, comprising: the device comprises a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface complete communication with each other through the communication bus;
the memory is configured to store at least one executable instruction, where the executable instruction causes the processor to perform an operation corresponding to a network transmission data noise reduction method according to any one of claims 1 to 7.
10. A computer storage medium having stored therein at least one executable instruction for causing a processor to perform operations corresponding to a network transmission data denoising method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311222183.7A CN116955934A (en) | 2023-09-21 | 2023-09-21 | Network transmission data noise reduction method and device, computing equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311222183.7A CN116955934A (en) | 2023-09-21 | 2023-09-21 | Network transmission data noise reduction method and device, computing equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116955934A true CN116955934A (en) | 2023-10-27 |
Family
ID=88449703
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311222183.7A Pending CN116955934A (en) | 2023-09-21 | 2023-09-21 | Network transmission data noise reduction method and device, computing equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116955934A (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102087707A (en) * | 2009-12-03 | 2011-06-08 | 索尼株式会社 | Image processing equipment and image processing method |
| CN113593597A (en) * | 2021-08-27 | 2021-11-02 | 中国电信股份有限公司 | Voice noise filtering method and device, electronic equipment and medium |
| CN113722178A (en) * | 2021-08-11 | 2021-11-30 | 云南电网有限责任公司信息中心 | System for intelligent noise reduction and alarm convergence of alarm information |
| CN114241340A (en) * | 2021-12-16 | 2022-03-25 | 北京工业大学 | Image target detection method and system based on double-path depth residual error network |
-
2023
- 2023-09-21 CN CN202311222183.7A patent/CN116955934A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102087707A (en) * | 2009-12-03 | 2011-06-08 | 索尼株式会社 | Image processing equipment and image processing method |
| CN113722178A (en) * | 2021-08-11 | 2021-11-30 | 云南电网有限责任公司信息中心 | System for intelligent noise reduction and alarm convergence of alarm information |
| CN113593597A (en) * | 2021-08-27 | 2021-11-02 | 中国电信股份有限公司 | Voice noise filtering method and device, electronic equipment and medium |
| CN114241340A (en) * | 2021-12-16 | 2022-03-25 | 北京工业大学 | Image target detection method and system based on double-path depth residual error network |
Non-Patent Citations (1)
| Title |
|---|
| 刘强: "《构建企业级推荐系统 算法 工程实现与案例分析》", 机械工业出版社, pages: 243 - 244 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110377759B (en) | Method and device for constructing event relation graph | |
| WO2020232861A1 (en) | Named entity recognition method, electronic device and storage medium | |
| EP3869385B1 (en) | Method for extracting structural data from image, apparatus and device | |
| CN110765759B (en) | Intention recognition method and device | |
| CN109559363B (en) | Image stylization processing method and device, medium and electronic equipment | |
| CN110222330B (en) | Semantic recognition method and device, storage medium and computer equipment | |
| EP3620982B1 (en) | Sample processing method and device | |
| CN113656547B (en) | Text matching method, device, equipment and storage medium | |
| CN111859093A (en) | Sensitive word processing method and device and readable storage medium | |
| CN111177375A (en) | Electronic document classification method and device | |
| CN111475651B (en) | Text classification method, computing device and computer storage medium | |
| CN114612921B (en) | Form recognition method and device, electronic equipment and computer readable medium | |
| CN114639096A (en) | Text recognition method and device, electronic equipment and storage medium | |
| CN113205047A (en) | Drug name identification method and device, computer equipment and storage medium | |
| CN112883736A (en) | Medical entity relationship extraction method and device | |
| CN115130437B (en) | Intelligent document filling method and device and storage medium | |
| CN113627186B (en) | Entity relation detection method based on artificial intelligence and related equipment | |
| CN115565186A (en) | Method and device for training character recognition model, electronic equipment and storage medium | |
| CN116955934A (en) | Network transmission data noise reduction method and device, computing equipment and storage medium | |
| CN111401069A (en) | Intention recognition method and intention recognition device for conversation text and terminal | |
| CN115858776A (en) | Variant text classification recognition method, system, storage medium and electronic equipment | |
| CN114565751A (en) | OCR recognition model training method, OCR recognition method and related device | |
| US11335108B2 (en) | System and method to recognise characters from an image | |
| JP7430625B2 (en) | Version verification device, version verification system, and version verification method | |
| CN111767710A (en) | Indonesia emotion classification method, device, equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |