CN116361520A - Data processing method and system applied to cloud platform - Google Patents

Data processing method and system applied to cloud platform Download PDF

Info

Publication number
CN116361520A
CN116361520A CN202310158432.4A CN202310158432A CN116361520A CN 116361520 A CN116361520 A CN 116361520A CN 202310158432 A CN202310158432 A CN 202310158432A CN 116361520 A CN116361520 A CN 116361520A
Authority
CN
China
Prior art keywords
data
audio
data set
image
video
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
CN202310158432.4A
Other languages
Chinese (zh)
Inventor
林江明
孙成建
哈愉晴
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Individual
Original Assignee
Individual
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Individual filed Critical Individual
Priority to CN202310158432.4A priority Critical patent/CN116361520A/en
Publication of CN116361520A publication Critical patent/CN116361520A/en
Withdrawn legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/904—Browsing; Visualisation therefor
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/901—Indexing; Data structures therefor; Storage structures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/906—Clustering; Classification
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/10—Pre-processing; Data cleansing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

The invention belongs to the technical field of data processing of cloud server platforms, and particularly relates to a data processing method and system applied to a cloud platform. The method comprises the following steps: the method comprises the steps of collecting unstructured data of an Internet of things platform, and carrying out data cleaning on the unstructured data so as to obtain a cleaning data set; partitioning the cleaning data set to obtain a block file of the data set, and performing data processing on the block file of the data set to generate a block file of a standard data set; storing the block files of the standard data set, thereby generating data storage files; and carrying out data analysis and visual display on the data storage file. According to the data cloud platform data processing method, data acquisition, processing, storage and analysis are carried out on the data of the Internet of things platform, so that the problem of data processing of the existing cloud computing platform is solved.

Description

Data processing method and system applied to cloud platform
Technical Field
The invention relates to the technical field of data processing of cloud server platforms, in particular to a data processing method and system applied to a cloud platform.
Background
Cloud computing is a model that enables computing resources (networks, servers, storage, applications, and services) to be acquired in a convenient, on-demand manner over a network, and from a shared, configurable pool of resources, and can be quickly acquired and released. The method overturns the consumption mode and service mode of the traditional IT industry, realizes the transition from purchasing software and hardware products to purchasing service in the past, has the effect mainly reflected in the aspect of virtualized data information processing and calculation, and is a novel business mode. Three concepts of cloud computing include: the increasing number of platforms and applications, infrastructure as a service (IaaS), platform as a service (PaaS) and software as a service (SaaS), also have led to an increasing proliferation of data volumes on the internet at a staggering rate, both as producers and consumers. Data processing computing is an important ring of cloud computing infrastructure, and aiming at visual analysis of data by a cloud server platform, the technical problem to be solved is urgent.
Disclosure of Invention
The invention provides a processing method applied to a cloud platform, which aims to solve at least one of the technical problems.
In order to achieve the above object, the data processing method applied to the cloud platform of the present invention includes the following steps:
Step S1: the method comprises the steps of collecting unstructured data of an Internet of things platform, and carrying out data cleaning on the unstructured data so as to obtain a cleaning data set;
step S2: partitioning the cleaning data set to obtain a block file of the data set, and performing data processing on the block file of the data set to generate a block file of a standard data set;
step S3: storing the block files of the standard data set, thereby generating data storage files;
step S4: and carrying out data analysis and visual display on the data storage file.
According to the embodiment, unstructured data such as audio class data, image class data and video class data on an Internet of things platform are collected, the collected audio class data, image class data and video class data are subjected to data cleaning, abnormal values in a data set are cleared and missing values are complemented, so that the data set can be subjected to data processing, storage, analysis and visualization, the data set is divided into a plurality of equal block files, the block files of the data set are obtained, the subsequent difficulty in processing the data is reduced, the data processing speed is increased, the total amount and time of resources occupied during data processing are reduced, the data processing work can be efficiently completed in a block processing mode by using the data, the data noise reduction processing is performed on preliminary noise points in the block files, the preliminary noise points in the data are reduced, the data are more close to ideal data, the data are extracted, calculated, grouped and converted into the block files of the standard data set after the data set are processed, the data is stored in a data mode, the data cannot be found back when the data is called, the data in the standard data set is called, the data in the storage block files are more visual mode, the data in the standard data set is more visual analysis is performed, and the data content of the data in the storage is more visual and the visual data is more clearly analyzed.
In one embodiment of the present specification, step S1 includes the steps of:
step S101, acquiring gateway communication connection;
step S102, acquiring an audio project data acquisition instruction set;
step S103, generating an audio project data acquisition instruction according to the gateway communication connection and the audio project data acquisition instruction set;
step S104, unstructured data of the Internet of things platform are collected according to the audio project data collection instruction, so that a preliminary audio data set is obtained;
step 105, acquiring an image item data acquisition instruction set;
step S106, generating an image item data acquisition instruction according to the gateway communication connection and the image item data acquisition instruction set;
step S107, unstructured data of the platform of the Internet of things are acquired according to the image item data acquisition instruction, so that a preliminary image data set is obtained;
step S108, acquiring a video project data acquisition instruction set;
step S109, generating a video project data acquisition instruction according to the gateway communication connection and the video project data acquisition instruction set;
step S110, unstructured data of the platform of the Internet of things are collected according to a video project data collection instruction, so that a preliminary video data set is obtained;
step S111, performing data cleaning processing on the audio data set, the image data set and the video data set, thereby generating a cleaning data set, wherein the cleaning data set comprises the audio cleaning data set, the image cleaning data set and the video cleaning data set.
According to the method, unstructured data are collected, a cloud server is in communication connection with an Internet of things platform through gateway communication connection, an audio item data collection instruction set, an image item data collection instruction set and a video item data collection instruction set are obtained through the audio item data collection instruction set, the image item data collection instruction set and the video item data collection instruction, data collection is controlled through the audio item data collection instruction, the image item data collection instruction set and the video item data collection instruction, a preliminary audio class data set, a preliminary image class data set and a preliminary video class data set are obtained, and data cleaning is conducted on the preliminary audio class data set, the preliminary image class data set and the preliminary video class data set, and an audio class cleaning data set, an image class cleaning data set and a video class cleaning data set are obtained.
In one embodiment of the present specification, step S111 includes the steps of:
performing data cleaning processing on the preliminary audio data set, converting the preliminary audio data set into a unified format, screening, judging whether screening data meet preset audio data set standards, marking the screening data as audio outliers if the screening data do not meet the preset audio data set standards, deleting the data of the audio outliers, and filling the data of the missing values with an average value to generate an audio cleaning data set;
Performing data cleaning processing on the preliminary image data set, converting the preliminary image data set into a unified format, screening, judging whether screening data meet the preset image data set standard, marking the screening data as image outliers if the screening data do not meet the preset image data set standard, deleting the data of the image outliers, and filling the data of the missing values with a mean value to generate an image cleaning data set;
and carrying out data cleaning processing on the preliminary video data set, converting the preliminary video data set into a unified format, screening, judging whether screening data meet the preset video data set standard, marking the screening data as video outliers if the screening data do not meet the preset video data set standard, deleting the data of the video outliers, and filling the data of the missing values with a mean value, so as to generate the video cleaning data set.
The embodiment teaches the process of performing data cleaning on the collected preliminary data set to obtain a cleaning data set, converting the preliminary data set into a uniform format, screening the preliminary data set, judging whether the data set meets the preset data set standard, marking the data set which does not meet the format requirement as outlier data, deleting the outlier data, and uniformly formatting the data so as to cause data processing problems due to outliers during subsequent data processing, and filling the missing values in the data set by using a mean value, thereby generating the cleaning data set.
In one embodiment of the present specification, step S2 includes the steps of:
step S201: performing data division processing on the video cleaning data set to obtain a video type audio cleaning data set and a video type image cleaning data set, and numbering the video type audio cleaning data set and the video type image cleaning data set correspondingly;
step S202: dividing an audio cleaning data set, an image cleaning data set, a video type audio cleaning data set and a video type image cleaning data set into discrete block files with equal quantity, and classifying the discrete block files to generate block files of the audio data set, block files of the image data set, block files of the video type audio data set and block files of the video type image data set;
step S203: carrying out noise reduction processing on the block files of the audio data set and the block files of the video type audio data set by utilizing an audio type data noise reduction processing formula, so as to generate the block files of the audio noise reduction data set and the block files of the video type audio noise reduction data set;
step S204: carrying out noise reduction processing on the block files of the image data set and the block files of the video type image data set by utilizing an image type data noise reduction processing formula, so as to generate the block files of the image noise reduction data set and the block files of the video type image noise reduction data set;
Step S205: carrying out data integration processing on the block files of the video type audio noise reduction data set and the block files of the video type image noise reduction data set, thereby obtaining the block files of the video noise reduction data set;
step S206: and carrying out information extraction, calculation, grouping and conversion data processing on the block files of the audio noise reduction data set, the block files of the image noise reduction data set and the block file data field preprocessing of the video noise reduction data set so as to generate a block file of a standard data set, wherein the standard data set block file comprises the block file of the audio standard data set, the block file of the image standard data set and the block file of the video standard data set.
In the embodiment, the video cleaning data set is divided into a video audio cleaning data set and a video image cleaning data set, the video audio cleaning data set and the video image cleaning data set are correspondingly numbered, the video audio cleaning data set and the video image cleaning data set obtained by separation can be processed together with the audio cleaning data set and the image cleaning data set, then the audio cleaning data set, the image cleaning data set, the video audio cleaning data set and the video image cleaning data set are divided into equal discrete block files, so that the subsequent processing difficulty of data is reduced, the data processing speed is accelerated, the total amount and time of resources are reduced during data processing, and the data processing work can be efficiently completed by a data blocking processing mode, thereby generating a block file of an audio data set, a block file of an image data set, a block file of a video type audio data set and a block file of a video type image data set, performing noise reduction processing on the block file of the audio data set and the block file of the video type audio data set by utilizing an audio type noise reduction formula, performing noise reduction processing on the block file of the image data set and the block file of the video type image data set by utilizing the image type data noise reduction processing formula, thereby generating a block file of the audio noise reduction data set, a block file of the video type audio noise reduction data set, a block file of the image noise reduction data set and a block file of the video type image noise reduction data set, correspondingly integrating the block file of the video type audio noise reduction data set and the block file of the video type image noise reduction data set through numbers, thereby generating a block file of the video noise reduction data set, and performing noise reduction on the block file of the audio noise reduction data set, the block files of the image noise reduction data set and the block file data field of the video noise reduction data set are preprocessed to extract information, calculate, group and process conversion data, so that the block files of the audio standard data set, the block files of the image standard data set and the block files of the video standard data set are generated, and data can be directly called to conduct data analysis.
In one embodiment of the present specification, the audio class data noise reduction processing formula in step S23 is:
the noise reduction processing formula of the audio class data is as follows:
Figure BDA0004093372520000041
wherein P is represented as an audio noise reduction index, n is represented as the last frame of audio data, α i Weight information expressed as audio anomaly noise of the first through i frames,
Figure BDA0004093372520000042
expressed as constant term, x i Expressed as the degree of change of the audio anomaly noise from the first frame to the i-th frame, z is expressed as denoising node audio information, u is expressed as the average value of the audio anomaly noise, beta i Correction terms expressed as first to i-th frames of audio anomaly noise, and epsilon expressed as adjustment terms of anomaly indexes;
in the embodiment, each frame of audio in the audio information is utilized to perform calculation processing in the audio data noise reduction processing formula, so that audio noise reduction is obtainedAn index, wherein the change degree x of the audio abnormal noise from the first frame to the i frame is added i For each frame of normal noise and white noise variation degree collected in the audio data, the correction term beta of the audio abnormal noise from the first frame to the i frame is utilized i Repairing the accumulated error of each frame of audio generated by white noise,
Figure BDA0004093372520000043
the audio data is noise-reduced by the noise node audio information z for the proportion information/quantization information of the audio noise.
In one embodiment of the present specification, the image class data noise reduction processing formula in step S24 is:
the noise reduction processing formula of the image data is as follows:
Figure BDA0004093372520000051
wherein T is expressed as an image noise reduction index, n is expressed as a pixel point of the last point of the horizontal axis of the image block, m is expressed as a pixel point of the last point of the vertical axis of the image block, y ij Represented as a matrix of image blocks and image blocks similar thereto, x ij Represented as a corresponding denoised matrix of image blocks,
Figure BDA0004093372520000052
expressed as constant term, z expressed as denoising node image information, μ ij Weight information of image anomaly noise represented as an image block composition matrix, k being represented as an average value of the image anomaly noise,/->
Figure BDA0004093372520000053
The correction term expressed as an image anomaly noise point and epsilon expressed as an adjustment term of an anomaly index.
In the embodiment, matrix y formed by image blocks and image blocks similar to the image blocks is utilized in image information in the image noise reduction processing formula ij Corresponding to image blocksIs a noise-reduced matrix x of (2) ij The square value of the square value is only the difference multiplied by the function
Figure BDA0004093372520000054
Figure BDA0004093372520000055
Obtaining an image noise reduction index, wherein for denoising node image information z, locking noise points to be removed in an image by utilizing the denoising node image information, removing the whole noise points of an image block by using the average value k of abnormal image noise points, and correcting terms of the abnormal image noise points >
Figure BDA0004093372520000056
And repairing accumulated errors generated by noise points and shadows of each image block by correction items of the abnormal noise points of the images.
In one embodiment of the present specification, step S3 includes the steps of:
and the block files of the audio standard data set, the block files of the image standard data set and the block files of the video standard data set are stored in a scattered manner in a multi-copy mode, so that a data storage file is generated, wherein the data storage file comprises an audio data storage file, an image data storage file and a video data storage file, and a data basis is provided when data is called.
In the embodiment, the block files of the audio standard data set, the block files of the image standard data set and the block files of the video standard data set are stored in a scattered manner in a multi-copy manner, so that data can be prevented from being missed and returned when the data is called, and data is stored or called to cause data blocking when the data amount is too large, thereby generating the data storage file.
In one embodiment of the present specification, step S4 includes the steps of:
step S401: performing data analysis and visual display on the audio data storage file, the image data storage file and the video data storage file, so as to generate audio visual data, image visual data and video visual data;
Step S402: and constructing a data visualization comprehensive page according to the audio visualization data, the image visualization data and the video visualization data so as to display a data visualization interface.
According to the embodiment, the audio data storage file, the image data storage file and the video data storage file are subjected to data analysis and visual display respectively, audio visual data, image visual data and video visual data are generated, and a comprehensive page is constructed to display the generated visual data.
In one embodiment of the present specification, the step of constructing the data visualization comprehensive page in step S402 includes the steps of:
carrying out different-level service inquiry processing on the audio data storage file, the image data storage file and the video data storage file to obtain service data contents of audio class, image class and video class;
carrying out data statistical analysis processing on the audio, image and video business data content data, thereby obtaining data statistical analysis results of the audio, image and video;
and carrying out data visualization processing according to the data statistics analysis results of the audio class, the picture class and the video class, converting the data statistics analysis results into a line graph, a column graph, an area graph and a dynamic graph, and carrying out data visualization display so as to generate audio visualization data, image visualization data and video visualization data.
In the implementation, different functions of data analysis are described for the data in the data storage file, the data is subjected to different levels of query processing, a user can obtain the data to be queried, the data is subjected to statistical analysis, the data can be subjected to statistical analysis processing according to service requirements, the user can obtain the required data analysis content, and the obtained data analysis content is subjected to visual display, so that the data analysis content is more concise and understandable.
In one embodiment of the present specification, a data processing system applied to a cloud platform includes:
at least one processor; the method comprises the steps of,
a memory communicatively coupled to the at least one processor; wherein, the liquid crystal display device comprises a liquid crystal display device,
the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the method of cloud platform data processing as described above.
The embodiment provides a data processing method and a system applied to a cloud platform, wherein the system can realize any data processing method applied to the cloud platform, is used for combining operation and data transmission media among all devices, acquires data through the Internet of things device, performs data cleaning, blocking and noise reduction on the data to obtain a data set to be processed, processes the data of the data set to be processed, sends the data set to a data storage module for data storage, performs data analysis and visualization through calling the data in the data storage module, and the internal structures of the systems are mutually cooperated, so that the data processing method of the cloud platform is perfected.
In the embodiment of the application, unstructured data of an internet of things platform are acquired, the acquired unstructured data comprise audio data, image data and video data, the acquired data are subjected to data cleaning, a disordered data set is cleaned into a processable data set, a cleaned data set is obtained, then the data set is subjected to block processing, the pressure of a cloud server can be reduced through the data block processing, the data is accelerated, the data set subjected to the block processing is subjected to noise reduction processing, data noise points existing in the data set are removed, the data are subjected to data processing to obtain a block file of a standard data set capable of carrying out data analysis, and the data in the block file is subjected to visual display and is designed into a comprehensive page of a data visual interface.
Drawings
FIG. 1 is a schematic flow chart of steps of a data processing method applied to a cloud platform;
FIG. 2 is a detailed process flow diagram of one of the steps of FIG. 1;
FIG. 3 is a detailed process flow diagram of one of the steps of FIG. 1;
fig. 4 is a schematic diagram of a method for collecting internet of things platform data by performing network communication connection through gateway equipment by using a cloud server;
FIG. 5 is a schematic diagram of control modules of each device of the cloud platform data processing system of the present invention;
the achievement of the objects, functional features and advantages of the present invention will be further described with reference to the accompanying drawings, in conjunction with the embodiments.
Detailed Description
It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
The embodiment of the application provides a data processing method and system applied to a cloud platform. The execution main body of the data processing method applied to the cloud platform comprises, but is not limited to, a system: mechanical devices, internet of things platforms, cloud server nodes, network transmission devices, and the like may be considered general purpose computing nodes of the present application. The internet of things platform includes, but is not limited to: at least one of a voice recognition system, a color classification management system and a traffic monitoring system.
The invention provides a data processing method applied to a cloud platform, which comprises the following steps:
step S1: the method comprises the steps of collecting unstructured data of an Internet of things platform, and carrying out data cleaning on the unstructured data so as to obtain a cleaning data set;
step S2: partitioning the cleaning data set to obtain a block file of the data set, and performing data processing on the block file of the data set to generate a block file of a standard data set;
Step S3: storing the block files of the standard data set, thereby generating data storage files;
step S4: and carrying out data analysis and visual display on the data storage file.
According to the embodiment, unstructured data such as audio class data, image class data and video class data on an Internet of things platform are collected, the collected audio class data, image class data and video class data are subjected to data cleaning, abnormal values in a data set are cleared and missing values are complemented, so that the data set can be subjected to data processing, storage, analysis and visualization, the data set is divided into a plurality of equal block files, the block files of the data set are obtained, the subsequent difficulty in processing the data is reduced, the data processing speed is increased, the total amount and time of resources occupied during data processing are reduced, the data processing work can be efficiently completed in a block processing mode by using the data, the data noise reduction processing is performed on preliminary noise points in the block files, the preliminary noise points in the data are reduced, the data are more close to ideal data, the data are extracted, calculated, grouped and converted into the block files of the standard data set after the data set are processed, the data is stored in a data mode, the data cannot be found back when the data is called, the data in the standard data set is called, the data in the storage block files are more visual mode, the data in the standard data set is more visual analysis is performed, and the data content of the data in the storage is more visual and the visual data is more clearly analyzed.
In the embodiment of the present invention, as described with reference to fig. 1, a flow chart of steps of a data processing method applied to a cloud platform of the present invention is shown, where in this example, the steps of the data processing method applied to the cloud platform include:
step S1: the method comprises the steps of collecting unstructured data of an Internet of things platform, and carrying out data cleaning on the unstructured data so as to obtain a cleaning data set;
in the embodiment of the invention, the unstructured data of the platform of the Internet of things can be unstructured data such as audio data, image data, video data and the like on a voice recognition system, a color classification management system and a traffic monitoring system, and the unstructured data is subjected to data cleaning, and the disordered data in the data is filtered, so that a cleaning data set is obtained.
Step S2: partitioning the cleaning data set to obtain a block file of the data set, and performing data processing on the block file of the data set to generate a block file of a standard data set;
in the embodiment of the invention, the cleaning data set is subjected to block processing to obtain the block file of the data set, so that the audio data, the image data and the video data in the data are conveniently divided into equal and different block files, the complexity of the data can be reduced, the data processing time can be reduced, and the data processing is more convenient. The data processing of the block files of the data set is to remove preliminary noise points in audio class data, image class data and video class data, and extract, calculate, group and convert the data to obtain the block files of the standard data set which can be directly used after being processed.
Step S3: storing the block files of the standard data set, thereby generating data storage files;
in the embodiment of the invention, the block files of the standard data set are stored, and the block files of the audio class data, the block files of the image class data and the block files of the video class data are respectively stored according to the block files of the audio class data, the block files of the image class data so as to prevent data loss or make the cloud server call the data more convenient.
Step S4: and carrying out data analysis and visual display on the data storage file.
In the embodiment of the invention, the data in the data storage file is called for data analysis, and the audio class data, the image class data and the video class data are respectively analyzed, for example, different types of sounds are classified according to tone color data in a voice recognition system, colors are classified according to similarity of colors in a color classification management system, whether the behavior of violating traffic rules of vehicles is analyzed in a traffic management system, and the analyzed data analysis content is visually displayed, so that a user can more intuitively and efficiently see the data analysis content.
Step S101, acquiring gateway communication connection;
step S102, acquiring an audio project data acquisition instruction set;
step S103, generating an audio project data acquisition instruction according to the gateway communication connection and the audio project data acquisition instruction set;
Step S104, unstructured data of the Internet of things platform are collected according to the audio project data collection instruction, so that a preliminary audio data set is obtained;
step 105, acquiring an image item data acquisition instruction set;
step S106, generating an image item data acquisition instruction according to the gateway communication connection and the image item data acquisition instruction set;
step S107, unstructured data of the platform of the Internet of things are acquired according to the image item data acquisition instruction, so that a preliminary image data set is obtained;
step S108, acquiring a video project data acquisition instruction set;
step S109, generating a video project data acquisition instruction according to the gateway communication connection and the video project data acquisition instruction set;
step S110, unstructured data of the platform of the Internet of things are collected according to a video project data collection instruction, so that a preliminary video data set is obtained;
step S111, performing data cleaning processing on the audio data set, the image data set and the video data set, thereby generating a cleaning data set, wherein the cleaning data set comprises the audio cleaning data set, the image cleaning data set and the video cleaning data set.
According to the method, unstructured data are collected, a cloud server is in communication connection with an Internet of things platform through gateway communication connection, an audio item data collection instruction set, an image item data collection instruction set and a video item data collection instruction set are obtained through the audio item data collection instruction set, the image item data collection instruction set and the video item data collection instruction, data collection is controlled through the audio item data collection instruction, the image item data collection instruction set and the video item data collection instruction, a preliminary audio class data set, a preliminary image class data set and a preliminary video class data set are obtained, and data cleaning is conducted on the preliminary audio class data set, the preliminary image class data set and the preliminary video class data set, and an audio class cleaning data set, an image class cleaning data set and a video class cleaning data set are obtained.
According to the embodiment of the invention, the cloud server platform is enabled to acquire network communication connection through the external gateway equipment, so that the cloud server platform can perform network communication with the Internet of things, an audio item data acquisition instruction set, an image item data acquisition instruction set and a video item data acquisition instruction set are acquired through a data control module of the Internet of things platform, the cloud server platform is enabled to generate an audio item data acquisition instruction, an image item data acquisition instruction set and a video item data acquisition instruction through the first two steps, and then data acquisition is performed on unstructured data of the Internet of things platform through the audio item data acquisition instruction, the image item data acquisition instruction set and the video item data acquisition instruction, such as unstructured data of a voice recognition system is acquired and marked as an audio data set; collecting unstructured data of a color classification management system and marking the unstructured data as an image data set; unstructured data of the traffic management system is collected and labeled as a video dataset. And (3) carrying out data cleaning on the collected audio data set, image data set and video data set, and filtering the disordered data in the data such as the audio data mixed with other text data to obtain an audio cleaning data set, an image cleaning data set and a video cleaning data set.
In the embodiment of the present invention, as described with reference to fig. 3, a method for collecting internet of things platform data by a cloud server through network communication connection of gateway equipment according to the present invention is shown in the embodiment, where the method includes:
the network communication module of the cloud server acquires gateway communication connection through external gateway equipment, performs network communication on the cloud server platform and the Internet of things platform through the gateway communication connection, the data acquisition module of the cloud server platform acquires a data acquisition instruction set through the Internet of things platform data control module, generates a data acquisition instruction through the data acquisition instruction set, acquires data in the Internet of things platform data storage module, and for the Internet of things platform, receives the data acquisition instruction of the cloud server platform, invokes the data in the data storage module and then uploads the data to the data acquisition module of the cloud server.
In one embodiment of the present specification, step S111 includes the steps of:
performing data cleaning processing on the preliminary audio data set, converting the preliminary audio data set into a unified format, screening, judging whether screening data meet preset audio data set standards, marking the screening data as audio outliers if the screening data do not meet the preset audio data set standards, deleting the data of the audio outliers, and filling the data of the missing values with an average value to generate an audio cleaning data set;
Performing data cleaning processing on the preliminary image data set, converting the preliminary image data set into a unified format, screening, judging whether screening data meet the preset image data set standard, marking the screening data as image outliers if the screening data do not meet the preset image data set standard, deleting the data of the image outliers, and filling the data of the missing values with a mean value to generate an image cleaning data set;
and carrying out data cleaning processing on the preliminary video data set, converting the preliminary video data set into a unified format, screening, judging whether screening data meet the preset video data set standard, marking the screening data as video outliers if the screening data do not meet the preset video data set standard, deleting the data of the video outliers, and filling the data of the missing values with a mean value, so as to generate the video cleaning data set.
The embodiment teaches the process of performing data cleaning on the collected preliminary data set to obtain a cleaning data set, converting the preliminary data set into a uniform format, screening the preliminary data set, judging whether the data set meets the preset data set standard, marking the data set which does not meet the format requirement as outlier data, deleting the outlier data, and uniformly formatting the data so as to cause data processing problems due to outliers during subsequent data processing, and filling the missing values in the data set by using a mean value, thereby generating the cleaning data set.
In the embodiment of the invention, the data are marked as preliminary data, and the data are cleaned to obtain a cleaning data set. Specifically, if the audio data collected from the voice recognition system is marked as preliminary audio data, the preliminary audio data is screened, whether the data in the preliminary audio data meet the standard of an audio data format is observed, if the collected data are not audio data with the problems of audio segments or audio segments being blank audio segments and the like, the collected data are marked as audio outliers, the data marked as the audio outliers are deleted, and the data missing in the preliminary audio data are filled with the average value of the data with the missing value, so that an audio cleaning data set is generated.
In one embodiment of the present specification, step S2 includes the steps of:
step S201: performing data division processing on the video cleaning data set to obtain a video type audio cleaning data set and a video type image cleaning data set, and numbering the video type audio cleaning data set and the video type image cleaning data set correspondingly;
step S202: dividing an audio cleaning data set, an image cleaning data set, a video type audio cleaning data set and a video type image cleaning data set into discrete block files with equal quantity, and classifying the discrete block files to generate block files of the audio data set, block files of the image data set, block files of the video type audio data set and block files of the video type image data set;
Step S203: carrying out noise reduction processing on the block files of the audio data set and the block files of the video type audio data set by utilizing an audio type data noise reduction processing formula, so as to generate the block files of the audio noise reduction data set and the block files of the video type audio noise reduction data set;
step S204: carrying out noise reduction processing on the block files of the image data set and the block files of the video type image data set by utilizing an image type data noise reduction processing formula, so as to generate the block files of the image noise reduction data set and the block files of the video type image noise reduction data set;
step S205: carrying out data integration processing on the block files of the video type audio noise reduction data set and the block files of the video type image noise reduction data set, thereby obtaining the block files of the video noise reduction data set;
step S206: and carrying out information extraction, calculation, grouping and conversion data processing on the block files of the audio noise reduction data set, the block files of the image noise reduction data set and the block file data field preprocessing of the video noise reduction data set so as to generate a block file of a standard data set, wherein the standard data set block file comprises the block file of the audio standard data set, the block file of the image standard data set and the block file of the video standard data set.
In the embodiment, the video cleaning data set is divided into a video audio cleaning data set and a video image cleaning data set, the video audio cleaning data set and the video image cleaning data set are correspondingly numbered, the video audio cleaning data set and the video image cleaning data set obtained by separation can be processed together with the audio cleaning data set and the image cleaning data set, then the audio cleaning data set, the image cleaning data set, the video audio cleaning data set and the video image cleaning data set are divided into equal discrete block files, so that the subsequent processing difficulty of data is reduced, the data processing speed is accelerated, the total amount and time of resources are reduced during data processing, and the data processing work can be efficiently completed by a data blocking processing mode, thereby generating a block file of an audio data set, a block file of an image data set, a block file of a video type audio data set and a block file of a video type image data set, performing noise reduction processing on the block file of the audio data set and the block file of the video type audio data set by utilizing an audio type noise reduction formula, performing noise reduction processing on the block file of the image data set and the block file of the video type image data set by utilizing the image type data noise reduction processing formula, thereby generating a block file of the audio noise reduction data set, a block file of the video type audio noise reduction data set, a block file of the image noise reduction data set and a block file of the video type image noise reduction data set, correspondingly integrating the block file of the video type audio noise reduction data set and the block file of the video type image noise reduction data set through numbers, thereby generating a block file of the video noise reduction data set, and performing noise reduction on the block file of the audio noise reduction data set, the block files of the image noise reduction data set and the block file data field of the video noise reduction data set are preprocessed to extract information, calculate, group and process conversion data, so that the block files of the audio standard data set, the block files of the image standard data set and the block files of the video standard data set are generated, and data can be directly called to conduct data analysis.
As an example of the present invention, referring to fig. 2, a flowchart illustrating a detailed implementation of one step in fig. 1 is shown, where the content in this example includes:
step S201: performing data division processing on the video cleaning data set to obtain a video type audio cleaning data set and a video type image cleaning data set, and numbering the video type audio cleaning data set and the video type image cleaning data set correspondingly;
in the embodiment of the invention, for example, aiming at the video cleaning data set which is collected by the traffic management system and is subjected to data cleaning, the video cleaning data set is divided and processed to obtain the video audio cleaning data set and the video image cleaning data set, and the data sets of the video audio cleaning data set and the video image cleaning data set are correspondingly numbered.
Step S202: dividing an audio cleaning data set, an image cleaning data set, a video type audio cleaning data set and a video type image cleaning data set into discrete block files with equal quantity, and classifying the discrete block files to generate block files of the audio data set, block files of the image data set, block files of the video type audio data set and block files of the video type image data set;
in the embodiment of the invention, for example, the collected audio cleaning data in the voice recognition system, the collected image cleaning data set in the color classification management system, the collected video type audio cleaning data set and the collected video type image cleaning data set in the traffic management system are divided into equal amounts of discrete block files, and the discrete block files are classified, so that the block files of the audio data set, the block files of the image data set, the block files of the video type audio data set and the block files of the video type image data set are generated.
Step S203: carrying out noise reduction processing on the block files of the audio data set and the block files of the video type audio data set by utilizing an audio type data noise reduction processing formula, so as to generate the block files of the audio noise reduction data set and the block files of the video type audio noise reduction data set;
in the embodiment of the invention, the block files of the audio data set and the block files of the video type audio data set are subjected to audio noise reduction processing by utilizing an audio type data noise reduction processing formula, so that the block files of the audio noise reduction data set and the block files of the video type audio noise reduction data set are generated.
Step S204: carrying out noise reduction processing on the block files of the image data set and the block files of the video type image data set by utilizing an image type data noise reduction processing formula, so as to generate the block files of the image noise reduction data set and the block files of the video type image noise reduction data set;
in the embodiment of the invention, the block files of the image dataset and the block files of the video image dataset are subjected to image noise reduction processing by utilizing an image data noise reduction processing formula, so that the block files of the image noise reduction dataset and the block files of the video image noise reduction dataset are generated.
Step S205: carrying out data integration processing on the block files of the video type audio noise reduction data set and the block files of the video type image noise reduction data set, thereby obtaining the block files of the video noise reduction data set;
In the embodiment of the invention, according to the corresponding number divided before the block file of the video type audio noise reduction data set and the block file of the video type image noise reduction data set, the block file of the video type audio noise reduction data set and the block file of the video type image noise reduction data set are subjected to data integration processing through the corresponding number, so that the block file of the video noise reduction data set is obtained.
Step S206: and carrying out information extraction, calculation, grouping and conversion data processing on the block files of the audio noise reduction data set, the block files of the image noise reduction data set and the block file data field preprocessing of the video noise reduction data set so as to generate a block file of a standard data set, wherein the standard data set block file comprises the block file of the audio standard data set, the block file of the image standard data set and the block file of the video standard data set.
In the embodiment of the invention, the block files of the audio noise reduction data set are subjected to audio information data extraction, the information such as tone, loudness and the like of the audio are extracted, and the information is calculated, grouped and converted, so that the block files of the audio standard data set are generated; extracting image information data from a block file of an image noise reduction dataset, extracting information such as pixel points, colors, saturation, contrast and the like on an image block, and calculating, grouping and converting the information to generate an image standard dataset; extracting video data from a block file of video noise reduction data, extracting audio information and image information from the video data, extracting information such as tone, loudness and the like of the audio from the audio information, calculating, grouping and converting the information, extracting information such as pixel points, color, saturation, contrast and the like on an image block from the image information, calculating, grouping and converting the information, integrating the information and the information to generate a block file of a video standard data set, specifically, carrying out data processing on the video data in a traffic management system, and counting vehicles running red lights, overspeed, illegal traffic regulations such as illegal traffic regulations through the video data.
In one embodiment of the present specification, the audio class data noise reduction processing formula in step S23 is:
the noise reduction processing formula of the audio class data is as follows:
Figure BDA0004093372520000131
wherein P is represented as an audio noise reduction index, n is represented as the last frame of audio data, α i Weight information expressed as audio anomaly noise of the first through i frames,
Figure BDA0004093372520000132
expressed as constant term, x i Expressed as the degree of change of the audio anomaly noise from the first frame to the i-th frame, z is expressed as denoising node audio information, u is expressed as the average value of the audio anomaly noise, beta i Correction terms expressed as first to i-th frames of audio anomaly noise, and epsilon expressed as adjustment terms of anomaly indexes;
in this embodiment, each frame of audio in the audio information is used to calculate the noise reduction index in the audio data noise reduction formula, and the degree x of variation of the abnormal noise of the audio from the first frame to the i frame is added i For each frame of normal noise and white noise variation degree collected in the audio data, the correction term beta of the audio abnormal noise from the first frame to the i frame is utilized i For a pair ofEach frame of audio is repaired due to accumulated errors generated by white noise,
Figure BDA0004093372520000133
the audio data is noise-reduced by the noise node audio information z for the proportion information/quantization information of the audio noise.
In one embodiment of the present specification, the image class data noise reduction processing formula in step S24 is:
in one embodiment of the present specification, the image class data noise reduction processing formula in step S24 is:
the noise reduction processing formula of the image data is as follows:
Figure BDA0004093372520000141
wherein T is expressed as an image noise reduction index, n is expressed as a pixel point of the last point of the horizontal axis of the image block, m is expressed as a pixel point of the last point of the vertical axis of the image block, y ij Represented as a matrix of image blocks and image blocks similar thereto, x ij Represented as a corresponding denoised matrix of image blocks,
Figure BDA0004093372520000142
expressed as constant term, z expressed as denoising node image information, μ ij Weight information of image anomaly noise represented as an image block composition matrix, k being represented as an average value of the image anomaly noise,/->
Figure BDA0004093372520000143
The correction term expressed as an image anomaly noise point and epsilon expressed as an adjustment term of an anomaly index.
In the embodiment, matrix y formed by image blocks and image blocks similar to the image blocks is utilized in image information in the image noise reduction processing formula ij Noise-reduced matrix x corresponding to an image block ij The square value of the square value is only the difference multiplied by the function
Figure BDA0004093372520000144
Figure BDA0004093372520000145
Obtaining an image noise reduction index, wherein for denoising node image information z, locking noise points to be removed in an image by utilizing the denoising node image information, removing the whole noise points of an image block by using the average value k of abnormal image noise points, and correcting terms of the abnormal image noise points >
Figure BDA0004093372520000146
And repairing accumulated errors generated by noise points and shadows of each image block by correction items of the abnormal noise points of the images.
In one embodiment of the present specification, step S3 includes the steps of:
and the block files of the audio standard data set, the block files of the image standard data set and the block files of the video standard data set are stored in a scattered manner in a multi-copy mode, so that a data storage file is generated, wherein the data storage file comprises an audio data storage file, an image data storage file and a video data storage file, and a data basis is provided when data is called.
In the embodiment, the block files of the audio standard data set, the block files of the image standard data set and the block files of the video standard data set are stored in a scattered manner in a multi-copy manner, so that data can be prevented from being missed and returned when the data is called, and data is stored or called to cause data blocking when the data amount is too large, thereby generating the data storage file.
In the embodiment of the invention, the block files of the audio standard data set, the block files of the image standard data set and the block files of the video standard data set are stored in a scattered manner in a multi-copy manner, so that a data storage file is generated, a data basis is provided when data is called, specifically, the data is backed up in a plurality of copies, the backed up data is marked as copies, the data can not be found after the data is lost, the copies can be directly called in a copy manner to enable the data to return to normalization, and the data and the copies are stored in a scattered manner, because the data volume is huge, the scattered storage can reach a very high aggregation bandwidth, the performance is reduced instead after the traditional storage reaches the maximum expansion capacity, and the distributed storage cost is very low, so that the cloud server is required to be directly added by expansion.
In one embodiment of the present specification, step S4 includes the steps of:
step S401: performing data analysis and visual display on the audio data storage file, the image data storage file and the video data storage file, so as to generate audio visual data, image visual data and video visual data;
step S402: and constructing a data visualization comprehensive page according to the audio visualization data, the image visualization data and the video visualization data so as to display a data visualization interface.
According to the embodiment, the audio data storage file, the image data storage file and the video data storage file are subjected to data analysis and visual display respectively, audio visual data, image visual data and video visual data are generated, and a comprehensive page is constructed to display the generated visual data.
As an example of the present invention, referring to fig. 3, a flowchart illustrating a detailed implementation of one step in fig. 1 is shown, where the content in this example includes:
step S401: performing data analysis and visual display on the audio data storage file, the image data storage file and the video data storage file, so as to generate audio visual data, image visual data and video visual data;
In the embodiment of the invention, the audio data storage file acquired in the voice recognition system is subjected to visual processing to display the total duration, loudness, tone color and the like of the sound, and the sound of what species is judged through the audio; and carrying out visual processing on the image data storage file acquired in the color classification management system to display the color, pixel point information, contrast, saturation and the like of the image, and judging which color class the image is classified into. And carrying out visual processing on the collected video data storage file in the traffic management system to display information, etc. of a certain vehicle, which violates the traffic rules and is shot by traffic monitoring.
Step S402: and constructing a data visualization comprehensive page according to the audio visualization data, the image visualization data and the video visualization data so as to display a data visualization interface.
In the embodiment of the invention, audio visual data, image visual data and video visual data are constructed into a data visual comprehensive page so as to display the generated visual data, the data visual comprehensive page is used for selecting different functions, such as displaying business data content, data statistics analysis results, data analysis visualizations and the like, and data tables, charts and text conclusions are displayed together around a category in the data analysis visualizations.
In one embodiment of the present specification, the step of constructing the data visualization comprehensive page in step S402 includes the steps of: carrying out different-level service inquiry processing on the audio data storage file, the image data storage file and the video data storage file to obtain service data contents of audio class, image class and video class;
carrying out data statistical analysis processing on the audio, image and video business data content data, thereby obtaining data statistical analysis results of the audio, image and video;
and carrying out data visualization processing according to the data statistics analysis results of the audio class, the picture class and the video class, converting the data statistics analysis results into a line graph, a column graph, an area graph and a dynamic graph, and carrying out data visualization display so as to generate audio visualization data, image visualization data and video visualization data.
In the implementation, different functions of data analysis are described for the data in the data storage file, the data is subjected to different levels of query processing, a user can obtain the data to be queried, the data is subjected to statistical analysis, the data can be subjected to statistical analysis processing according to service requirements, the user can obtain the required data analysis content, and the obtained data analysis content is subjected to visual display, so that the data analysis content is more concise and understandable.
In the embodiment of the invention, the service data content query function can be selected on the comprehensive page of the data visualization interface to query the service data content of the audio class, the image class and the video class, the data statistics analysis function can be selected to query the data statistics analysis result of the audio class, the image class and the video class, and the data visualization display function can be selected to display the data visualization images such as the line graph, the bar graph, the area graph, the dynamic graph and the like of the data of the audio class, the image class and the video class.
In one embodiment of the present specification, a data processing system applied to a cloud platform includes:
at least one processor; the method comprises the steps of,
a memory communicatively coupled to the at least one processor; wherein, the liquid crystal display device comprises a liquid crystal display device,
the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the method of cloud platform data processing as described above.
The embodiment provides a data processing method and a system applied to a cloud platform, wherein the system can realize any data processing method applied to the cloud platform, is used for combining operation and data transmission media among all devices, acquires data through the Internet of things device, performs data cleaning, blocking and noise reduction on the data to obtain a data set to be processed, processes the data of the data set to be processed, sends the data set to a data storage module for data storage, performs data analysis and visualization through calling the data in the data storage module, and the internal structures of the systems are mutually cooperated, so that the data processing method of the cloud platform is perfected.
In the embodiment of the present invention, referring to fig. 5, a schematic diagram of each device control module of the cloud platform data processing system of the present invention is shown, and in this example, the device control module includes:
the cloud platform data processing system comprises a data acquisition module, a data processing module, a data storage module and a data processing module;
the data acquisition module comprises a data acquisition function and a data cleaning function;
the data processing module comprises a data blocking function, a data noise reduction function and a data processing function;
the data storage module comprises data storage and data backup;
the data analysis module comprises data analysis and data visualization;
wherein the data acquired by the data acquisition module of the cloud server are obtained from the internet of things platform;
wherein, each functional plate cooperates with each other to finish the data processing method of the cloud server;
the data processing module is a core of the cloud server platform and is used for receiving the data in the data acquisition module to process the data, and then the processed data is put into the data storage module to be processed by the data analysis module.
The method comprises the steps of collecting unstructured data of an Internet of things platform, wherein the collected unstructured data comprises audio data, image data and video data, cleaning the collected data into a processable data set, obtaining a cleaned data set, conducting block processing on the data set, reducing the pressure of a cloud server through the data block processing, enabling data to be quickened, conducting noise reduction on the data set subjected to the block processing, removing data noise existing in the data set, conducting data processing on the data to obtain a block file of a standard data set capable of conducting data analysis, conducting visual display on the data in the block file, and designing a comprehensive page of a data visual interface.
The present embodiments are, therefore, to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference signs in the claims shall not be construed as limiting the claim concerned.
It should be noted that in this document, relational terms such as "first" and "second" and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
The foregoing is only a specific embodiment of the invention to enable those skilled in the art to understand or practice the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims (10)

1. A data processing method applied to a cloud platform, which is characterized by being applied to a cloud server, the method comprising the following steps:
step S1: the method comprises the steps of collecting unstructured data of an Internet of things platform, and carrying out data cleaning on the unstructured data so as to obtain a cleaning data set;
step S2: partitioning the cleaning data set to obtain a block file of the data set, and performing data processing on the block file of the data set to generate a block file of a standard data set;
step S3: storing the block files of the standard data set, thereby generating data storage files;
step S4: and carrying out data analysis and visual display on the data storage file.
2. The data processing method applied to the cloud platform according to claim 1, wherein the step S1 includes the steps of:
step S101, acquiring gateway communication connection;
step S102, acquiring an audio project data acquisition instruction set;
step S103, generating an audio project data acquisition instruction according to the gateway communication connection and the audio project data acquisition instruction set;
step S104, unstructured data of the Internet of things platform are collected according to the audio project data collection instruction, so that a preliminary audio data set is obtained;
step 105, acquiring an image item data acquisition instruction set;
step S106, generating an image item data acquisition instruction according to the gateway communication connection and the image item data acquisition instruction set;
step S107, unstructured data of the platform of the Internet of things are acquired according to the image item data acquisition instruction, so that a preliminary image data set is obtained;
step S108, acquiring a video project data acquisition instruction set;
step S109, generating a video project data acquisition instruction according to the gateway communication connection and the video project data acquisition instruction set;
step S110, unstructured data of the platform of the Internet of things are collected according to a video project data collection instruction, so that a preliminary video data set is obtained;
Step S111, performing data cleaning processing on the audio data set, the image data set and the video data set, thereby generating a cleaning data set, wherein the cleaning data set comprises the audio cleaning data set, the image cleaning data set and the video cleaning data set.
3. The data processing method applied to the cloud platform as claimed in claim 2, wherein step S111 includes:
performing data cleaning processing on the preliminary audio data, converting the preliminary audio data into a unified format, screening, judging whether the screened data meets the preset audio data set standard, marking the screened data as audio outliers if the screened data does not meet the preset audio data set standard, deleting the data of the audio outliers, filling the data of the missing values with an average value, and thus generating an audio cleaning data set;
performing data cleaning processing on the preliminary image data, converting the preliminary image data into a unified format, screening, judging whether the screened data meets the preset image data set standard, marking the screened data as an image abnormal value if the screened data does not meet the preset image data set standard, deleting the data of the image abnormal value, filling the data of the missing value with a mean value, and thus generating an image cleaning data set;
and carrying out data cleaning treatment on the preliminary video data, converting the preliminary video data into a unified format, screening, judging whether the screened data meets the preset video data set standard, marking the screened data as video outliers if the screened data does not meet the preset video data set standard, deleting the data of the video outliers, and filling the data of the missing values with an average value, so as to generate a video cleaning data set.
4. The data processing method applied to the cloud platform according to claim 2, wherein the step S2 includes the steps of:
step S201: performing data division processing on the video cleaning data set to obtain a video type audio cleaning data set and a video type image cleaning data set, and numbering the video type audio cleaning data set and the video type image cleaning data set correspondingly;
step S202: dividing an audio cleaning data set, an image cleaning data set, a video type audio cleaning data set and a video type image cleaning data set into discrete block files with equal quantity, and classifying the discrete block files to generate block files of the audio data set, block files of the image data set, block files of the video type audio data set and block files of the video type image data set;
step S203: carrying out noise reduction processing on the block files of the audio data set and the block files of the video type audio data set by utilizing an audio type data noise reduction processing formula, so as to generate the block files of the audio noise reduction data set and the block files of the video type audio noise reduction data set;
step S204: carrying out noise reduction processing on the block files of the image data set and the block files of the video type image data set by utilizing an image type data noise reduction processing formula, so as to generate the block files of the image noise reduction data set and the block files of the video type image noise reduction data set;
Step S205: carrying out data integration processing on the block files of the video type audio noise reduction data set and the block files of the video type image noise reduction data set, thereby obtaining the block files of the video noise reduction data set;
step S206: and carrying out information extraction, calculation, grouping and conversion data processing on the block files of the audio noise reduction data set, the block files of the image noise reduction data set and the block file data field preprocessing of the video noise reduction data set so as to generate a block file of a standard data set, wherein the standard data set block file comprises the block file of the audio standard data set, the block file of the image standard data set and the block file of the video standard data set.
5. The data processing method applied to the cloud platform as claimed in claim 4, wherein the audio class data noise reduction processing formula in step S203 is:
the noise reduction processing formula of the audio class data is as follows:
Figure QLYQS_1
wherein P is represented as an audio noise reduction index, n is represented as the last frame of audio data, α i Weight information expressed as audio anomaly noise of the first through i frames,
Figure QLYQS_2
expressed as constant term, x i Expressed as the degree of variation of the audio anomaly noise from the first frame to the i-th frame, z is expressed as denoising node audio information, and u is expressed as audio anomaly Average value of constant noise, beta i Correction terms expressed as first to i-th frames of audio anomaly noise, and epsilon expressed as adjustment terms of anomaly indexes.
6. The data processing method applied to the cloud platform as claimed in claim 4, wherein the image class data noise reduction processing formula in step S204 is:
the noise reduction processing formula of the image data is as follows:
Figure QLYQS_3
wherein T is expressed as an image noise reduction index, n is expressed as a pixel point of the last point of the horizontal axis of the image block, m is expressed as a pixel point of the last point of the vertical axis of the image block, y ij Represented as a matrix of image blocks and image blocks similar thereto, x ij Represented as a corresponding denoised matrix of image blocks,
Figure QLYQS_4
expressed as constant term, z expressed as denoising node image information, μ ij Weight information of image anomaly noise represented as an image block composition matrix, k being represented as an average value of the image anomaly noise,/->
Figure QLYQS_5
The correction term expressed as an image anomaly noise point and epsilon expressed as an adjustment term of an anomaly index.
7. The data processing method applied to the cloud platform as claimed in claim 4, wherein the step S3 includes the steps of:
and the block files of the audio standard data set, the block files of the image standard data set and the block files of the video standard data set are stored in a scattered manner in a multi-copy mode, so that a data storage file is generated, wherein the data storage file comprises an audio data storage file, an image data storage file and a video data storage file, and a data basis is provided when data is called.
8. The data processing method applied to the cloud platform as claimed in claim 7, wherein the step S4 includes the steps of:
step S401: performing data analysis and visual display on the audio data storage file, the image data storage file and the video data storage file, so as to generate audio visual data, image visual data and video visual data;
step S402: and constructing a data visualization comprehensive page according to the audio visualization data, the image visualization data and the video visualization data so as to display a data visualization interface.
9. The data processing method applied to the cloud platform as claimed in claim 8, wherein the step of constructing the data visualization comprehensive page in step S402 includes the steps of:
carrying out different-level service inquiry processing on the audio data storage file, the image data storage file and the video data storage file to obtain service data contents of audio class, image class and video class;
carrying out data statistical analysis processing on the audio, image and video business data content data, thereby obtaining data statistical analysis results of the audio, image and video;
and carrying out data visualization processing according to the data statistics analysis results of the audio class, the picture class and the video class, converting the data statistics analysis results into a line graph, a column graph, an area graph and a dynamic graph, and carrying out data visualization display so as to generate audio visualization data, image visualization data and video visualization data.
10. A data processing system for a cloud platform, comprising:
at least one processor; the method comprises the steps of,
a memory communicatively coupled to the at least one processor; wherein, the liquid crystal display device comprises a liquid crystal display device,
the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the data processing method applied to a cloud platform as claimed in any one of claims 1 to 9.
CN202310158432.4A 2023-02-23 2023-02-23 Data processing method and system applied to cloud platform Withdrawn CN116361520A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310158432.4A CN116361520A (en) 2023-02-23 2023-02-23 Data processing method and system applied to cloud platform

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310158432.4A CN116361520A (en) 2023-02-23 2023-02-23 Data processing method and system applied to cloud platform

Publications (1)

Publication Number Publication Date
CN116361520A true CN116361520A (en) 2023-06-30

Family

ID=86917816

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310158432.4A Withdrawn CN116361520A (en) 2023-02-23 2023-02-23 Data processing method and system applied to cloud platform

Country Status (1)

Country Link
CN (1) CN116361520A (en)

Similar Documents

Publication Publication Date Title
CN110020122B (en) Video recommendation method, system and computer readable storage medium
CN105808604B (en) Data compliance management method and system
CN104572976B (en) Website data update method and system
CN110347888B (en) Order data processing method and device and storage medium
CN109564567B (en) Data storage method, device, electronic equipment and computer readable storage medium
CN104462096A (en) Public opinion monitoring and analysis method and device
CN107609172B (en) Cross-system multi-dimensional data retrieval processing method and device
CN111291028A (en) High-speed industrial field oriented data acquisition system and method
CN106557483B (en) Data processing method, data query method, data processing equipment and data query equipment
CN116361520A (en) Data processing method and system applied to cloud platform
CN114187463A (en) Electronic archive generation method and device, terminal equipment and storage medium
CN106708876B (en) Similar video retrieval method and system based on Lucene
CN110019152A (en) A kind of big data cleaning method
CN111061790A (en) Information acquisition system and method for customer data management
CN112613396B (en) Task emergency degree processing method and system
CN114969089A (en) Data obtaining method and device, electronic equipment and storage medium
CN109828968B (en) Data deduplication processing method, device, equipment, cluster and storage medium
CN109657098B (en) Video fingerprint extraction method and device
CN104980750A (en) Collection method, device and system for video transcoding logs
CN107948738B (en) Method and device for calculating number of users for internet protocol television reuse
CN104951869A (en) Workflow-based public opinion monitoring method and workflow-based public opinion monitoring device
CN111209284A (en) Metadata-based table dividing method and device
CN112487039B (en) Data processing method, device, equipment and readable storage medium
CN112287186B (en) Intelligent classification method and system for city management
CN114357054B (en) Method and device for processing unstructured data based on ClickHouse

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
WW01 Invention patent application withdrawn after publication
WW01 Invention patent application withdrawn after publication

Application publication date: 20230630