CN117076613A - Electric digital data processing system based on Internet big data - Google Patents

Electric digital data processing system based on Internet big data Download PDF

Info

Publication number
CN117076613A
CN117076613A CN202311323526.9A CN202311323526A CN117076613A CN 117076613 A CN117076613 A CN 117076613A CN 202311323526 A CN202311323526 A CN 202311323526A CN 117076613 A CN117076613 A CN 117076613A
Authority
CN
China
Prior art keywords
data
digital data
module
electric digital
neural network
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202311323526.9A
Other languages
Chinese (zh)
Other versions
CN117076613B (en
Inventor
张文韬
陈超群
向强铭
张鹏
艾远高
杨之圣
刘松林
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
China Yangtze Power Co Ltd
Original Assignee
China Yangtze Power Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by China Yangtze Power Co Ltd filed Critical China Yangtze Power Co Ltd
Priority to CN202311323526.9A priority Critical patent/CN117076613B/en
Publication of CN117076613A publication Critical patent/CN117076613A/en
Application granted granted Critical
Publication of CN117076613B publication Critical patent/CN117076613B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/33Querying
    • G06F16/3331Query processing
    • G06F16/334Query execution
    • G06F16/3344Query execution using natural language analysis
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/35Clustering; Classification
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00Handling natural language data
    • G06F40/20Natural language analysis
    • G06F40/279Recognition of textual entities
    • G06F40/289Phrasal analysis, e.g. finite state techniques or chunking
    • G06F40/295Named entity recognition
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/04Architecture, e.g. interconnection topology
    • G06N3/0464Convolutional networks [CNN, ConvNet]
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/04Architecture, e.g. interconnection topology
    • G06N3/048Activation functions
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/08Learning methods
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06QINFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00Commerce
    • G06Q30/02Marketing; Price estimation or determination; Fundraising
    • G06Q30/0201Market modelling; Market analysis; Collecting market data
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02DCLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • Artificial Intelligence (AREA)
  • Computational Linguistics (AREA)
  • General Health & Medical Sciences (AREA)
  • Business, Economics & Management (AREA)
  • Health & Medical Sciences (AREA)
  • Biophysics (AREA)
  • Finance (AREA)
  • Biomedical Technology (AREA)
  • Strategic Management (AREA)
  • Evolutionary Computation (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Accounting & Taxation (AREA)
  • Development Economics (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Databases & Information Systems (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Game Theory and Decision Science (AREA)
  • Economics (AREA)
  • Marketing (AREA)
  • General Business, Economics & Management (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

An internet big data based electronic digital data processing system comprising: the system comprises a data acquisition module, a data storage module, a data processing module, a data analysis module and a data visualization module; the invention collects and stores electric digital data through a data collection module, carries out end-to-end learning and deducing by a deep learning unit by utilizing a deep neural network technology, extracts high-level abstract features and expressions, trains and identifies images in the electric digital data through a convolutional neural network, realizes automatic image classification or object identification, carries out tasks such as feature extraction, classification and prediction of the electric digital data, and carries out analysis on the extracted data and visualizes the electric digital data into the forms of charts, graphs, maps and the like through a data analysis module and a data visualization module, so that a user can intuitively understand the data and find potential modes and trends.

Description

Electric digital data processing system based on Internet big data
Technical Field
The invention relates to the technical field of electric data processing, in particular to an electric digital data processing system based on Internet big data.
Background
Electronic data processing (EDP, electronic data processing) evolved from "data processing" which was generated when most of the calculations were entered into the computer in the form of a punched card and output in the form of a punched card or paper report. Computers are used to replace manual processing of routine data and report forms are generated to support the work activities of the organization. The content is focused on replacing repetitive manual operations to support basic level managers and operators, etc., such as accounting applications, so that it is a re-efficient information system.
An existing electronic data processing technology, for example, chinese patent document CN113794725a describes an electric digital data transmission method and system, comprising: acquiring initial information of current electric digital data, and comparing the electric digital data at the same time with the initial information through an encryption algorithm to obtain differentiated data; performing encryption diagnosis on the differentiated data by adopting a multi-element projection dimension reduction algorithm, and then decoding the diagnosed data; preprocessing the decoded data, marking the preprocessed data by adopting a ciphertext setting technology, and then transmitting the electric digital data. The method can be used for carrying out on-line monitoring, analysis and transmission on the electric digital data by improving the traditional analysis mode. In the scheme, the electric digital parameters of different types are difficult to extract and divide when being collected, processed and analyzed, and the object identification is inconvenient.
Disclosure of Invention
The invention aims to solve the technical problem of providing an electric digital data processing system based on Internet big data, which is used for solving the problems that different kinds of electric digital parameters are difficult to extract and divide characteristics and inconvenient to identify objects when electric digital is collected, processed and analyzed in the existing scheme.
In order to solve the technical problems, the invention adopts the following technical scheme:
an electric digital data processing system based on Internet big data comprises a data acquisition module, a data storage module, a data processing module, a data analysis module and a data visualization module which are connected in sequence;
the data acquisition module is responsible for acquiring electric digital data from external equipment or a sensor, and preprocessing and converting the electric digital data;
the data storage module is used for storing electric digital data and comprises a database, a file system and a cloud storage unit so as to ensure the safety and accessibility of the data;
the data processing module is used for executing data processing tasks, including data cleaning, conversion, aggregation, calculation and convolutional neural network model training, so as to obtain useful information and results, and comprises a natural language processing unit, a deep learning unit and an anomaly detection unit;
the data analysis module is used for analyzing the electric digital data by utilizing statistical analysis, machine learning and data mining technologies, extracting modes, relevance and data insight, and supporting decision making and prediction;
and the data visualization module is used for visualizing the electric digital data into the forms of charts, graphs and maps.
The natural language processing unit is used for processing word segmentation, part-of-speech tagging, syntactic analysis, entity identification, text classification and emotion analysis.
The deep learning unit performs data collection and preprocessing, constructs a convolutional neural network model, data training and optimization, model evaluation and testing, and model application and prediction, and trains and identifies images in the electric digital data through the convolutional neural network.
The construction convolutional neural network comprises a convolutional layer, an activation function and a pooling layer.
The above-mentioned convolution layer is used for extracting the characteristic of the picture or other two-dimensional data, the expression of the convolution layer is:
wherein y is 1 (i, j) represents the value of the position (i, j) of the output feature map after convolution, x (i, j) represents the value of the position (i, j) of the input feature map, W represents the weight parameter of the convolution kernel, b represents the bias parameter, and f represents the activation function.
The activation function is used for introducing nonlinear characteristics and enhancing the expression capacity of the neural network, and the formula of the activation function is as follows:
wherein y represents the output value after activation, x represents the input value, and activation represents the activation function.
The pooling layer is used for reducing the space size of the feature map and simultaneously retaining the feature information, and the pooling operation has the following formula:
wherein y is 2 (i, j) represents the value of the position (i, j) of the pooled output feature map, x (i, j) represents the value of the position (i, j) of the input feature map, pooling represents the pooling operation, and is the maximum value or the average value; feature extraction, classification and prediction of the deep learning model on the electric digital data are realized through convolution, and automatic image classification or object recognition is realized.
The invention provides an electric digital data processing system based on Internet big data, which has the beneficial effects that:
(1) The invention collects and stores electric digital data through the data collection module and the data storage module, and performs end-to-end learning and deducing by using a deep learning unit and a deep neural network technology, processes complex electric digital data, extracts high-level abstract features and expressions, trains and identifies images in the electric digital data through a convolutional neural network, and realizes automatic image classification or object identification, feature extraction, classification, prediction and other tasks of the electric digital data.
(2) According to the invention, the data analysis module and the data visualization module are used for analyzing the extracted data and visualizing the electric digital data into the forms of charts, graphs, maps and the like, so that a user can intuitively understand the data and find potential modes and trends, and an anomaly detection algorithm is used for carrying out real-time monitoring and anomaly detection on the electric digital data and identifying an anomaly mode, an outlier or a fault so as to improve the safety and stability of the system.
Drawings
The invention is further illustrated by the following examples in conjunction with the accompanying drawings:
FIG. 1 is a block diagram of an Internet big data based electrical digital data processing system of the present invention;
FIG. 2 is a block diagram of the components of the data processing module of the present invention;
fig. 3 is a block diagram of the convolutional neural network of the present invention.
Detailed Description
The technical scheme of the invention is described in detail below with reference to the accompanying drawings and examples.
As shown in fig. 1, 2 and 3, an electric digital data processing system based on internet big data comprises a data acquisition module, a data storage module, a data processing module, a data analysis module and a data visualization module which are connected in sequence;
the data acquisition module is responsible for acquiring electric digital data from external equipment or a sensor, and preprocessing and converting the electric digital data;
the data storage module is used for storing electric digital data and comprises a database, a file system and a cloud storage unit so as to ensure the safety and accessibility of the data;
the data processing module is used for executing data processing tasks, including data cleaning, conversion, aggregation, calculation and convolutional neural network model training, so as to obtain useful information and results, and comprises a natural language processing unit, a deep learning unit and an anomaly detection unit;
the data analysis module is used for analyzing the electric digital data by utilizing statistical analysis, machine learning and data mining technologies, extracting modes, relevance and data insight, and supporting decision making and prediction;
and the data visualization module is used for visualizing the electric digital data into the forms of charts, graphs and maps, so that a user can intuitively understand the data and find potential modes and trends.
The natural language processing unit is used for processing word segmentation, part-of-speech tagging, syntactic analysis, entity identification, text classification and emotion analysis.
Specifically, word segmentation divides text into words, phrases or other meaningful language units for text preprocessing and text analysis tasks;
the part of speech label is that each word in the text labels part of speech, such as nouns, verbs, adjectives and the like, and can be used for tasks such as grammar analysis, semantic analysis, information extraction and the like;
syntactic analysis is to analyze the grammar structure of sentences, identify phrases, clauses, dependency relationships and the like in the sentences, and analyze the composition and grammar rules which are helpful for understanding the sentences;
the entity identification is used for identifying named entities in the text, such as person names, place names, organization and the like, and can be used for tasks such as information extraction, relation extraction, entity linking and the like;
text classification classifies texts according to predefined categories, and is widely applied to the fields of emotion analysis, spam filtering, topic classification and the like;
emotion analysis is used to identify emotional trends in text, such as positive, negative, or neutral, and is commonly used in the field of social media analysis, etc.
The deep learning unit performs data collection and preprocessing, constructs a convolutional neural network model, data training and optimization, model evaluation and testing, and model application and prediction, and trains and identifies images in the electric digital data through the convolutional neural network.
The process of constructing the convolutional neural network model, training the data and optimizing the data comprises a convolutional layer, an activation function and a pooling layer.
The above convolution layer is used to extract the features of an image or other two-dimensional data, and the formula is as follows:
wherein y is 1 (i, j) represents the value of the position (i, j) of the output feature map after convolution, x (i, j) represents the value of the position (i, j) of the input feature map, W represents the weight parameter of the convolution kernel, b represents the bias parameter, and f represents the activation function.
The activation function is used for introducing nonlinear characteristics and enhancing the expression capacity of the neural network, and the common activation function comprises ReLU, sigmoid, tanh, reLU, namely Rectified Linear Unit, and the formula of the activation function is as follows:
wherein y represents the output value after activation, x represents the input value, and activation represents the activation function.
The Pooling layer is used for reducing the space size of the feature map and simultaneously retaining the feature information, and common Pooling operations comprise Max Pooling and Average Pooling, and the formulas of the Pooling operations are as follows:
wherein y is 2 (i, j) represents the value of the position (i, j) of the pooled output feature map, and x (i, j) represents the inputThe value of the position (i, j) of the feature map, pooling, represents a pooling operation, being a maximum or average; feature extraction, classification and prediction of the deep learning model on the electric digital data can be realized through convolution, and automatic image classification or object recognition is realized.
It can be seen from the above that, through the data acquisition module, the data storage module acquires and stores the electric digital data, and through the deep learning unit, the deep neural network technology is utilized to learn and infer end to end, process complex electric digital data, extract high-level abstract features and expressions, train and identify images in the electric digital data through the convolutional neural network, and realize automatic image classification or object identification, and perform tasks such as feature extraction, classification and prediction on the electric digital data;
the data analysis module and the data visualization module are used for analyzing the extracted data and visualizing the electric digital data into the forms of charts, graphs, maps and the like, so that a user can intuitively understand the data and find potential modes and trends, and the abnormal detection unit is used for monitoring and detecting the electric digital data in real time and identifying abnormal modes, outliers or faults so as to improve the safety and stability of the system.

Claims (7)

1. An electric digital data processing system based on Internet big data is characterized by comprising a data acquisition module, a data storage module, a data processing module, a data analysis module and a data visualization module which are connected in sequence;
the data acquisition module is responsible for acquiring electric digital data from external equipment or a sensor, and preprocessing and converting the electric digital data;
the data storage module is used for storing electric digital data and comprises a database, a file system and a cloud storage unit, and the data storage module is used for ensuring the safety and accessibility of the data;
the data processing module is used for executing data processing tasks, including data cleaning, conversion, aggregation, calculation and convolutional neural network model training, and comprises a natural language processing unit, a deep learning unit and an anomaly detection unit;
the data analysis module is used for analyzing the electric digital data by utilizing statistical analysis, machine learning and data mining technologies, extracting modes, relevance and data insight, and supporting decision making and prediction;
and the data visualization module is used for visualizing the electric digital data into the forms of charts, graphs and maps.
2. The system of claim 1, wherein the natural language processing unit is configured to process word segmentation, part-of-speech tagging, syntactic analysis, entity recognition, text classification, and emotion analysis.
3. The internet big data-based electric digital data processing system of claim 2, wherein the deep learning unit performs data collection and preprocessing, construction of a convolutional neural network model, data training and optimization, model evaluation and testing, and model application and prediction, and trains and identifies images in the electric digital data through the convolutional neural network.
4. An internet big data based digital data processing system according to claim 3, wherein said constructing a convolutional neural network comprises a convolutional layer, an activation function, and a pooling layer.
5. The system of claim 4, wherein the convolution layer is used to extract the features of the image or other two-dimensional data, and the expression of the convolution layer is:
wherein y is 1 (i, j) represents the value of the position (i, j) of the convolved output feature map, and x (i, j) represents the position (i) of the input feature mapThe value of j) is calculated,weight parameters representing convolution kernels, +.>Representing the bias parameters +.>Representing an activation function.
6. The system of claim 5, wherein the activation function is used to introduce nonlinear characteristics to enhance the expressive power of the neural network, and the formula of the activation function is as follows:
wherein,representing the output value after activation,/->Representing the input value +.>Representing an activation function.
7. An internet big data based digital data processing system according to claim 6, wherein the pooling layer is configured to reduce the spatial size of the feature map while preserving the feature information, and the pooling operation is formulated as follows:
wherein y is 2 (i, j) represents poolingThe value of the position (i, j) of the output feature map after that, x (i, j) represents the value of the position (i, j) of the input feature map,representing the pooling operation as a maximum or average; feature extraction, classification and prediction of the deep learning model on the electric digital data can be realized through convolution, and automatic image classification or object recognition is realized.
CN202311323526.9A 2023-10-13 2023-10-13 Electric digital data processing system based on Internet big data Active CN117076613B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202311323526.9A CN117076613B (en) 2023-10-13 2023-10-13 Electric digital data processing system based on Internet big data

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202311323526.9A CN117076613B (en) 2023-10-13 2023-10-13 Electric digital data processing system based on Internet big data

Publications (2)

Publication Number Publication Date
CN117076613A true CN117076613A (en) 2023-11-17
CN117076613B CN117076613B (en) 2024-07-16

Family

ID=88704569

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202311323526.9A Active CN117076613B (en) 2023-10-13 2023-10-13 Electric digital data processing system based on Internet big data

Country Status (1)

Country Link
CN (1) CN117076613B (en)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108764268A (en) * 2018-04-02 2018-11-06 华南理工大学 A kind of multi-modal emotion identification method of picture and text based on deep learning
CN109597891A (en) * 2018-11-26 2019-04-09 重庆邮电大学 Text emotion analysis method based on two-way length Memory Neural Networks in short-term
AU2020100710A4 (en) * 2020-05-05 2020-06-11 Chen, Dadu Mr A method for sentiment analysis of film reviews based on deep learning and natural language processing
KR102325022B1 (en) * 2020-09-22 2021-11-11 김백기 On-line image and review integrated analysis method and system using deep learning-based hybrid analysis method
CN116841262A (en) * 2023-06-27 2023-10-03 北京红山信息科技研究院有限公司 Intelligent factory production on-line monitoring analysis system based on machine vision

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108764268A (en) * 2018-04-02 2018-11-06 华南理工大学 A kind of multi-modal emotion identification method of picture and text based on deep learning
CN109597891A (en) * 2018-11-26 2019-04-09 重庆邮电大学 Text emotion analysis method based on two-way length Memory Neural Networks in short-term
AU2020100710A4 (en) * 2020-05-05 2020-06-11 Chen, Dadu Mr A method for sentiment analysis of film reviews based on deep learning and natural language processing
KR102325022B1 (en) * 2020-09-22 2021-11-11 김백기 On-line image and review integrated analysis method and system using deep learning-based hybrid analysis method
CN116841262A (en) * 2023-06-27 2023-10-03 北京红山信息科技研究院有限公司 Intelligent factory production on-line monitoring analysis system based on machine vision

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
刘小明;张英;郑秋生;: "基于卷积神经网络模型的互联网短文本情感分类", 计算机与现代化, no. 04, 15 April 2017 (2017-04-15) *

Also Published As

Publication number Publication date
CN117076613B (en) 2024-07-16

Similar Documents

Publication Publication Date Title
CN106886580B (en) Image emotion polarity analysis method based on deep learning
CN116663664A (en) Customer marketing scene data analysis system and method based on NLP algorithm
CN117009524B (en) Internet big data analysis method and system based on public opinion emotion analysis
CN111669368B (en) End-to-end network sensing abnormity detection and analysis method, system, device and medium
CN118013963B (en) Method and device for identifying and replacing sensitive words
CN116756688A (en) Public opinion risk discovery method based on multi-mode fusion algorithm
Mahgoub et al. Sentiment analysis: Amazon electronics reviews using bert and textblob
CN117076613B (en) Electric digital data processing system based on Internet big data
CN112989058A (en) Information classification method, test question classification method, device, server and storage medium
Schwenker et al. EXSCLAIM!--An automated pipeline for the construction of labeled materials imaging datasets from literature
CN112784601A (en) Key information extraction method and device, electronic equipment and storage medium
CN116864128A (en) Psychological state assessment system and method based on physical activity behavior pattern monitoring
CN112698897A (en) Method and system for arranging visual big data operator
CN116881395A (en) Public opinion information detection method and device
CN115330423A (en) Advertisement analysis method and system for webpage image-text data
CN110674269A (en) Cable information management and control method and system
CN116049385B (en) Method, device, equipment and platform for generating information and create industry research report
CN117172248B (en) Text data labeling method, system and medium
CN113220850B (en) Case image mining method for court trial and reading
CN114494958A (en) Examinee return abnormal behavior detection method based on video understanding model
Prabhu et al. A Unified Approach for Detecting Emotions through Sentimental and Emotional Analysis
CN118312606A (en) Session type recommendation method and system based on semantic similarity and clustering model
CN117556047A (en) Medical record information classification method, medical record information classification device, equipment and storage medium
Cena Cost-Effective Classification of Financial Documents with FinEmbedDiff: Leveraging Multi-Modal Embedding Models and Vector Sampling
CN118114663A (en) Automatic extraction method, system, equipment and medium for entities in power failure field

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant