CN110134711A - Processing method, device, equipment and the computer readable storage medium of big data - Google Patents
Processing method, device, equipment and the computer readable storage medium of big data Download PDFInfo
- Publication number
- CN110134711A CN110134711A CN201910272512.6A CN201910272512A CN110134711A CN 110134711 A CN110134711 A CN 110134711A CN 201910272512 A CN201910272512 A CN 201910272512A CN 110134711 A CN110134711 A CN 110134711A
- Authority
- CN
- China
- Prior art keywords
- data
- user
- traffic
- high amount
- processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24553—Query execution of query operations
- G06F16/24554—Unary operations; Data partitioning operations
- G06F16/24556—Aggregation; Duplicate elimination
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24568—Data stream processing; Continuous queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2462—Approximate or statistical queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
- G06F16/285—Clustering or classification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/903—Querying
- G06F16/9035—Filtering based on additional data, e.g. user or group profiles
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/953—Querying, e.g. by the use of web search engines
- G06F16/9535—Search customisation based on user profiles and personalisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/955—Retrieval from the web using information identifiers, e.g. uniform resource locators [URL]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/30—Authentication, i.e. establishing the identity or authorisation of security principals
- G06F21/31—User authentication
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/604—Tools and structures for managing or administering access control systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
- G06F21/6218—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database
- G06F21/6245—Protecting personal data, e.g. for financial or medical purposes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/55—Push-based network services
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Computer Security & Cryptography (AREA)
- Computer Hardware Design (AREA)
- General Health & Medical Sciences (AREA)
- Bioethics (AREA)
- Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Probability & Statistics with Applications (AREA)
- Mathematical Physics (AREA)
- Computing Systems (AREA)
- Evolutionary Computation (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Artificial Intelligence (AREA)
- Automation & Control Theory (AREA)
- Fuzzy Systems (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Transfer Between Computers (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of processing method of big data, device, equipment and computer readable storage medium, which includes: acquisition high amount of traffic associated with user terminal, and high amount of traffic includes at least: user behavior data and/or equipment state；High amount of traffic is decoded, generates general format data, and count to general format data；Polymerization calculating is carried out to general format data according to count results, generates polymerization calculated result；User behavior is monitored according to polymerization calculated result, executes interactive operation.By applying the present invention, more open, hommization Data Share System can be provided for user, meanwhile, it is capable to guarantee the safety of user's data by the authentication of user；Also, corresponding flow chart of data processing or Processing Algorithm are provided based on shared platform, respective data processing service can be made to measure for user, improve the use feeling of user.
Description
Technical field
The present invention relates to big data processing technology fields, and in particular to a kind of processing method of big data, device, equipment and
Computer readable storage medium.
Background technique
In the information age, the development of database was formd using the data modeling induction and conclusion past and will be given a forecast to future
" computational science "；And internet/big data age is consumed, by collecting a large amount of data, forms and computer is allowed to go rule
Study and Optimized model " data science ".Big data (Bigdata) is commonly used to describe that a company creates a large amount of non-
Structural data and semi-structured data, with the arriving in industry internet AI intelligent epoch, big data (Bigdata) needs more
Effectively it is evolved into data-intensive scientific discovery (ScientificDiscovery).And any one scientific discovery, it requires
Meet the requirement of Reproducible Result (repeatable result).Because it is based on data as a result, be all from data -> from
Therefore reason -> result (Data- > Process- > Result) such a process will meet Reproducible Result's
It is required that a kind of more intelligent, more flexible real-time (not single only in face of user) data processing system is then needed to handle data.
In the epoch for consuming internet, data processing system used in each large size technology platform, is by providing data
Technology of sharing by data collection and is stored on themselves platform (centralization and publicly-owned cloud), and applies super id machine
System, has more permission to freely using for user by platform provider centralized control, these are brought using limitation to user
Inconvenience has dragged slowly industry to interconnect the development progress of technology required for network process decentralization more significantly.
Summary of the invention
In view of this, the embodiment of the invention provides a kind of processing method of big data, device, equipment and computer-readable
Storage medium, to solve the above problems.
According in a first aspect, the embodiment of the invention provides a kind of processing methods of big data, comprising: acquisition and user terminal
Associated high amount of traffic, the high amount of traffic include at least: user behavior data and/or equipment state；Decode the big number
According to stream, general format data is generated, and the general format data is counted；According to count results to the general format
Data carry out polymerization calculating, generate polymerization calculated result；User behavior is monitored according to the polymerization calculated result, executes interaction behaviour
Make.
With reference to first aspect, in first aspect first embodiment, user's row is monitored according to the polymerization calculated result
To execute interactive operation, comprising: whether reach default threshold according to the quantity that the polymerization calculated result detects the user behavior
Value；If the quantity of the user behavior reaches the preset threshold, interactive action is initiated to object is performed.
With reference to first aspect, in first aspect second embodiment, the processing method further include: obtain user's input
Solicited message；Preset flow chart of data processing and/or default machine are downloaded from data sharing platform according to the solicited message
Learning algorithm.
Second embodiment with reference to first aspect, in first aspect third embodiment, the processing method further include: sentence
It is disconnected whether to get data processing notification information；If getting the data processing notification information, the preset number is used
The data in the data processing notification information are handled according to process flow and/or default machine learning algorithm, according to treated
The user data and application scenarios of user terminal described in data configuration.
With reference to first aspect, in the 4th embodiment of first aspect, the high amount of traffic is decoded, generates general format number
According to, comprising: the high amount of traffic is decoded using flume* or kafka* technology, the high amount of traffic is decoded as the general lattice
Formula data.
According to second aspect, the embodiment of the invention provides a kind of processing units of big data, comprising: high amount of traffic obtains
Module, for obtaining associated with user terminal high amount of traffic, the high amount of traffic is included at least: user behavior data and/or
Equipment state；Data processing module generates general format data, and to the general format for decoding the high amount of traffic
Data are counted；It polymerize computing module, for carrying out polymerization calculating to the general format data according to count results, generates
It polymerize calculated result；Interactive module executes interactive operation for monitoring user behavior according to the polymerization calculated result.
In conjunction with second aspect, in second aspect first embodiment, the interactive module includes: counting statistics submodule
Whether block, the quantity for detecting the user behavior according to the polymerization calculated result reach preset threshold；Interactive action is held
Row submodule, if the quantity of the user behavior reaches the preset threshold, the interactive action implementation sub-module for pair
It is performed object and initiates interactive action.
In conjunction with second aspect, in second aspect second embodiment, the processing unit further include: solicited message obtains mould
Block, for obtaining the solicited message of user's input；Data download module, for according to the solicited message from data sharing platform
The preset flow chart of data processing of middle downloading and/or default machine learning algorithm.
In conjunction with second aspect second embodiment, in second aspect third embodiment, the device further include: first sentences
Disconnected module, gets data processing notification information for judging whether；Configuration module, if getting the data processing notice
Information, the configuration module handle the data using the preset flow chart of data processing and/or default machine learning algorithm
Data in processing notification information, the user data and application scenarios of the user terminal according to treated data configuration.
In conjunction with second aspect first embodiment, in the 4th embodiment of second aspect, the data decoder module tool
Body is used for: decoding the high amount of traffic using flume* or kafka* technology, the high amount of traffic is decoded as the general lattice
Formula data.
In conjunction with second aspect first embodiment, in second aspect second embodiment, the device further include: downloading is asked
Acquisition module is sought, for obtaining the downloading request of user's input；Data download module, for being requested according to the downloading from data
Preset flow chart of data processing and/or the default machine learning algorithm are downloaded in shared platform.
According to the third aspect, the embodiment of the invention provides a kind of big data processing equipments, comprising: memory and processing
Device communicates with each other connection between the memory and the processor, computer instruction, the place are stored in the memory
Device is managed by executing the computer instruction, thereby executing institute in any one of first aspect or first aspect embodiment
The big data processing method stated.
It is described computer-readable the embodiment of the invention provides a kind of computer readable storage medium according to fourth aspect
Storage medium stores computer instruction, and the computer instruction is for making the computer execute first aspect or first aspect
Any one embodiment described in big data processing method.
The beneficial effect that the embodiment of the present invention has is, more open, hommization data can be provided for user
Shared mechanism, meanwhile, it is capable to guarantee the safety of user's data by the authentication of user；Relative to existing use
Technical treatment full dose data, the common practice of streaming computing processing real-time incremental data of batch processing, the embodiment of the present invention provide
User data label processing method and device, obtain user data when only need according to the service logic of oneself manage it is of interest
User behavior data, either full dose data or incremental data also or in real time handle, can implement through the invention
Example is realized.Also, corresponding flow chart of data processing or Processing Algorithm are provided based on shared platform, can be made to measure respectively for user
From data processing service, improve the use feeling of user.
Detailed description of the invention
The features and advantages of the present invention will be more clearly understood by referring to the accompanying drawings, and attached drawing is schematically without that should manage
Solution is carries out any restrictions to the present invention, in the accompanying drawings:
Fig. 1 shows the flow chart of the high amount of traffic processing method of the embodiment of the present invention；
Fig. 2 shows the structural schematic diagrams of the high amount of traffic processing unit of the embodiment of the present invention；
Fig. 3 shows the hardware structural diagram of the high amount of traffic processing equipment of the embodiment of the present invention.
Specific embodiment
In order to make the object, technical scheme and advantages of the embodiment of the invention clearer, below in conjunction with the embodiment of the present invention
In attached drawing, technical scheme in the embodiment of the invention is clearly and completely described, it is clear that described embodiment is
A part of the embodiment of the present invention, instead of all the embodiments.Based on the embodiments of the present invention, those skilled in the art are not having
Every other embodiment obtained under the premise of creative work is made, shall fall within the protection scope of the present invention.
The embodiment of the present invention provides a kind of processing method of big data, as shown in Figure 1, mainly comprising the steps that
Step S11: obtaining high amount of traffic associated with user terminal, which includes at least: user behavior data
And/or equipment state；
The processing method of big data provided in an embodiment of the present invention, it is intended to help business processes high amount of traffic, the big data
Stream from Internet (outside enterprise firewall), and from Internet of Things (IOT) equipment generate sensing data or from
The high amount of traffic etc. that the forms such as the consumer behaviour data that mobile device generates obtain.These high amount of traffic represent Digital Business with
The cross reaction of physical world, and be made of user behavior data, equipment state or the operation data that different scenes provide.
The high amount of traffic generated from Internet can come from many different data sources, or a series of number
According to object.Sequence in data flow is unbounded, it means that can continuously generate data flow with any rate.
Step S12: decoding the high amount of traffic of acquisition, generates general format data, and count to general format data；
Specifically, it can be the flow data progress realized using flume*/kafka* technology to thering is user and equipment to generate
Processing in real time, when data access, is decoded as general format for input traffic.And it is possible to utilize Cassandra counter
Technology counts the data for being converted to general format.
Step S13: polymerization calculating is carried out to the general format data according to count results, generates polymerization calculated result；
Step S14: user behavior is monitored according to the polymerization calculated result, executes interactive operation.
Specifically whether preset threshold is reached according to the quantity of polymerization calculated result detection user behavior, the default threshold
Value can be the pre-set threshold value of user；If the quantity of user behavior reaches preset threshold, send out object is performed
Play interactive action (such as sending short message, Email, APPLE Message information, wechat, LINE etc.).
The processing method of big data through the embodiment of the present invention can provide more open, hommization number for user
According to shared mechanism, meanwhile, it is capable to guarantee the safety of user's data by the authentication of user；Also, based on shared
Platform provides corresponding flow chart of data processing or Processing Algorithm, can make respective data processing service to measure for user, mention
The use feeling of high user.
Optionally, in some embodiments of the invention, the processing method of the big data can be applied particularly to a user terminal
In equipment, through the ustomer premises access equipment other than above-mentioned big data treatment process can be achieved, it can also be achieved complicated, customization
Algorithm process process and automaton study, but the volume algorithm process process and machine learning algorithm of these customizations, need to be by this
Ustomer premises access equipment is downloaded with data sharing platform interconnection.It is the solicited message (example for obtaining user's input in practical application
Downloading request etc. in this way)；Preset flow chart of data processing and/or default machine are downloaded from data sharing platform according to solicited message
Device learning algorithm is used for subsequent processing data.
Optionally, in some embodiments of the invention, it can also receive what external equipment was sent by the ustomer premises access equipment
Data notification information, in which case, the processing method of the executable big data of the ustomer premises access equipment may also include that judgement is
It is no to get data processing notification information；If getting data processing notification information, using preset flow chart of data processing and/
Or the data in default machine learning algorithm processing data processing notification information, according to the use of treated data configuration user terminal
User data and application scenarios.
In practical applications, the ustomer premises access equipment is with the following functions and advantage:
1. providing very humanized workbench
Data analysis and process personnel can go up hand by short-term training and carry out visual workflow development.It pulls
The realization data analysis stream exploitation of formula.Business and data analyst can be flexible and be effectively performed including data preparation, number
Knowledge discovery end to end and forecast analysis according to conversion, including data modeling and data scoring.The ustomer premises access equipment can be right
Data are directly operated, and are not limited to the quantity of independent variable and the complexity of data type.
2. the integrality of ustomer premises access equipment big data analysis process deployment
Very easily program export function, user can be straight by the analysis process sequence developed for ustomer premises access equipment offer
Export is connect, preassigned catalogue is placed into.Configure clocked flip or manual triggering mode operation.Derived model can be with
Independent execution on the database, thereby further ensures that the reliability and integrality of data, accelerates model and service application
Integration.
3. ustomer premises access equipment big data analysis flow nodesization deployment makes custom algorithm model import standardization
The algorithm of many users has industrial characteristic, and user's custom algorithm is essential, what which provided
(Process as a Orchestrated Series of Operators (node) leads custom algorithm for nodeization deployment
Standard functions can be had by entering.
By the cloud AI algorithm shared platform communicated with the ustomer premises access equipment, user can issue and be total to analysis process sequence
It enjoys, issuing function uses B/S framework, and user directly the operation such as carry out checking for model, modify and run by webpage mode, right
Shared business model promotes standardized support.
In addition, the ustomer premises access equipment can help enterprise be compiled into GDPR strategy (" general data conservation regulation ",
General Data Protection Regulation, abbreviation GDPR), which takes all necessary measure to store
With protection data, these data are the digital copies of physical asset.Ustomer premises access equipment design allows enterprise to be compiled as GDPR,
By establishing closed-loop system with the cloud control centre of management high amount of traffic event handling, enable the enterprise to real-time as Digital Business
Service and action.
Enterprise can be helped to abide by the requirement in " GDPR " rule to user data secrecy.Specific of any identification will not be issued
The information of people, only exchange combing process complies fully with the requirement of GDPR rule, provides for enterprise low without exchanging any data
Delay, the high data interaction service handled up.
The embodiment of the present invention also provides a kind of processing unit of big data, as shown in Fig. 2, the processing unit specifically includes that
High amount of traffic obtains module 11, and for obtaining high amount of traffic associated with user terminal, high amount of traffic is included at least:
User behavior data and/or equipment state；Detailed content can be found in the associated description of the step S11 of above method embodiment.
Data processing module 12 generates general format data, and carry out to general format data for decoding high amount of traffic
It counts；Detailed content can be found in the associated description of the step S12 of above method embodiment.
It polymerize computing module 13, for carrying out polymerization calculating to general format data according to count results, generates polymerization meter
Calculate result；Detailed content can be found in the associated description of the step S13 of above method embodiment.
Interactive module 14 executes interactive operation for monitoring user behavior according to polymerization calculated result；The interactive module packet
It includes: counting statistics submodule, for whether reaching preset threshold according to the quantity of polymerization calculated result detection user behavior；Interaction
Implementation sub-module is acted, if the quantity of user behavior reaches preset threshold, interactive action implementation sub-module is used for being performed
Object initiates interactive action.Detailed content can be found in the associated description of the step S14 of above method embodiment.
Optionally, in some embodiments of the invention, the processing unit of the big data can be applied particularly to a user terminal
In equipment, through the ustomer premises access equipment other than above-mentioned big data treatment process can be achieved, it can also be achieved complicated, customization
Algorithm process process and automaton study, but the volume algorithm process process and machine learning algorithm of these customizations, need to be by this
Ustomer premises access equipment is downloaded with data sharing platform interconnection.In practical application, the processing unit further include: solicited message obtains
Modulus block, for obtaining the solicited message (e.g. downloading request etc.) of user's input；Data download module, for according to request
Information downloads preset flow chart of data processing and/or default machine learning algorithm from data sharing platform, for subsequent processing number
According to being used.
Optionally, in some embodiments of the invention, it can also receive what external equipment was sent by the ustomer premises access equipment
Data notification information, in which case, the processing unit of the executable big data of the ustomer premises access equipment may also include that first sentences
Disconnected module, gets data processing notification information for judging whether；Configuration module, if getting data processing notice letter
Breath, the configuration module handle data processing notification information using preset flow chart of data processing and/or default machine learning algorithm
In data, according to the user data and application scenarios of treated data configuration user terminal.
The embodiment of the present invention also provides a kind of big data processing equipment, as shown in figure 3, the big data processing equipment can wrap
Processor 31 and memory 32 are included, wherein processor 31 can be connected with memory 32 by bus or other modes, in Fig. 3
For being connected by bus.
Processor 31 can be central processing unit (Central Processing Unit, CPU).Processor 31 can be with
For other general processors, digital signal processor (Digital Signal Processor, DSP), specific integrated circuit
(Application Specific Integrated Circuit, ASIC), field programmable gate array (Field-
Programmable Gate Array, FPGA) either other programmable logic device, discrete gate or transistor logic,
The combination of the chips such as discrete hardware components or above-mentioned all kinds of chips.
Memory 32 is used as a kind of non-transient computer readable storage medium, can be used for storing non-transient software program, non-
Transient computer executable program and module, as the corresponding program of the processing method of the big data in the embodiment of the present invention refers to
Order/module is (for example, high amount of traffic shown in Fig. 2 obtains module 11, data processing module 12, polymerization computing module 13 and interaction
Module 14).Non-transient software program, instruction and module of the processor 31 by operation storage in memory 32, to hold
The various function application and data processing of row processor, the i.e. processing method of big data in realization above method embodiment.
Memory 32 may include storing program area and storage data area, wherein storing program area can storage program area,
Application program required at least one function；It storage data area can the data etc. that are created of storage processor 31.In addition, storage
Device 32 may include high-speed random access memory, can also include non-transient memory, for example, at least a magnetic disk storage
Part, flush memory device or other non-transient solid-state memories.In some embodiments, it includes relative to place that memory 32 is optional
The remotely located memory of device 31 is managed, these remote memories can pass through network connection to processor 31.The reality of above-mentioned network
Example includes but is not limited to internet, intranet, local area network, mobile radio communication and combinations thereof.
One or more of modules are stored in the memory 32, when being executed by the processor 31, are executed
The processing method of big data in embodiment as shown in Figure 1.
Above-mentioned big data processing equipment detail can correspond to refering to fig. 1 shown in corresponding associated description in embodiment
Understood with effect, details are not described herein again.
It is that can lead to it will be understood by those skilled in the art that realizing all or part of the process in above-described embodiment method
Computer program is crossed to instruct relevant hardware and complete, the program can be stored in a computer-readable storage medium
In, the program is when being executed, it may include such as the process of the embodiment of above-mentioned each method.Wherein, the storage medium can for magnetic disk,
CD, read-only memory (Read-Only Memory, ROM), random access memory (Random Access
Memory, RAM), flash memory (Flash Memory), hard disk (Hard Disk Drive, abbreviation: HDD) or solid state hard disk
(Solid-State Drive, SSD) etc.；The storage medium can also include the combination of the memory of mentioned kind.
Although being described in conjunction with the accompanying the embodiment of the present invention, those skilled in the art can not depart from the present invention
Spirit and scope in the case where various modifications and variations can be made, such modifications and variations are each fallen within by appended claims institute
Within the scope of restriction.
Claims (12)
1. a kind of processing method of big data characterized by comprising
High amount of traffic associated with user terminal is obtained, the high amount of traffic includes at least: user behavior data and/or equipment shape
State；
The high amount of traffic is decoded, generates general format data, and count to the general format data；
Polymerization calculating is carried out to the general format data according to count results, generates polymerization calculated result；
User behavior is monitored according to the polymerization calculated result, executes interactive operation.
2. the processing method of big data according to claim 1, which is characterized in that monitored according to the polymerization calculated result
User behavior executes interactive operation, comprising:
Whether reach preset threshold according to the quantity that the polymerization calculated result detects the user behavior；
If the quantity of the user behavior reaches the preset threshold, interactive action is initiated to object is performed.
3. the processing method of big data according to claim 1, which is characterized in that further include:
Obtain the solicited message of user's input；
Preset flow chart of data processing is downloaded from data sharing platform according to the solicited message and/or default machine learning is calculated
Method.
4. the processing method of big data according to claim 3, which is characterized in that further include:
Judge whether to get data processing notification information；
If getting the data processing notification information, the preset flow chart of data processing and/or default engineering are used
The data described in algorithm process in data processing notification information are practised, the number of users of the user terminal according to treated data configuration
According to and application scenarios.
5. the processing method of big data according to claim 1, which is characterized in that decode the high amount of traffic, generate logical
With formatted data, comprising:
The high amount of traffic is decoded using flume* or kafka* technology, the high amount of traffic is decoded as the general format number
According to.
6. a kind of processing unit of big data characterized by comprising
High amount of traffic obtains module, and for obtaining high amount of traffic associated with user terminal, the high amount of traffic is included at least: being used
Family behavioral data and/or equipment state；
Data processing module generates general format data for decoding the high amount of traffic, and to the general format data into
Row counts；
It polymerize computing module, for carrying out polymerization calculating to the general format data according to count results, generates polymerization and calculate
As a result；
Interactive module executes interactive operation for monitoring user behavior according to the polymerization calculated result.
7. the processing unit of big data according to claim 6, which is characterized in that the interactive module includes:
Whether counting statistics submodule, the quantity for detecting the user behavior according to the polymerization calculated result reach default
Threshold value；
Interactive action implementation sub-module, if the quantity of the user behavior reaches the preset threshold, the interactive action is held
Row submodule be used for be performed object initiate interactive action.
8. the processing unit of big data according to claim 6, which is characterized in that further include:
Solicited message obtains module, for obtaining the solicited message of user's input；
Data download module, for downloading preset flow chart of data processing from data sharing platform according to the solicited message
And/or default machine learning algorithm.
9. the processing unit of big data according to claim 8, which is characterized in that further include:
First judgment module gets data processing notification information for judging whether；
Configuration module, if getting the data processing notification information, the configuration module is used at the preset data
Reason process and/or default machine learning algorithm handle the data in the data processing notification information, according to treated data
Configure the user data and application scenarios of the user terminal.
10. the processing unit of big data according to claim 6, which is characterized in that the data decoder module is specifically used
In:
The high amount of traffic is decoded using flume* or kafka* technology, the high amount of traffic is decoded as the general format number
According to.
11. a kind of big data processing equipment characterized by comprising
Memory and processor communicate with each other connection, are stored in the memory between the memory and the processor
Computer instruction, the processor is by executing the computer instruction, thereby executing as described in any one of claim 1-5
Big data processing method.
12. a kind of computer readable storage medium, which is characterized in that the computer-readable recording medium storage has computer to refer to
It enables, the processing that the computer instruction is used to that the computer to be made to execute big data according to any one of claims 1 to 5
Method.
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2019100373090 | 2019-01-15 | ||
| CN201910037309 | 2019-01-15 |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110134711A true CN110134711A (en) | 2019-08-16 |
Family
ID=67569442
Family Applications (5)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910273168.2A Pending CN110188144A (en) | 2019-01-15 | 2019-04-04 | A kind of user data digitalized processing method, device and equipment |
| CN201910274370.7A Pending CN110188264A (en) | 2019-01-15 | 2019-04-04 | A kind of user data exchange method and device |
| CN201910273174.8A Pending CN110175281A (en) | 2019-01-15 | 2019-04-04 | A kind of user data processing, exchange method, apparatus and system |
| CN201910274466.3A Pending CN110196933A (en) | 2019-01-15 | 2019-04-04 | A kind of user data label processing method and device |
| CN201910272512.6A Pending CN110134711A (en) | 2019-01-15 | 2019-04-04 | Processing method, device, equipment and the computer readable storage medium of big data |
Family Applications Before (4)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910273168.2A Pending CN110188144A (en) | 2019-01-15 | 2019-04-04 | A kind of user data digitalized processing method, device and equipment |
| CN201910274370.7A Pending CN110188264A (en) | 2019-01-15 | 2019-04-04 | A kind of user data exchange method and device |
| CN201910273174.8A Pending CN110175281A (en) | 2019-01-15 | 2019-04-04 | A kind of user data processing, exchange method, apparatus and system |
| CN201910274466.3A Pending CN110196933A (en) | 2019-01-15 | 2019-04-04 | A kind of user data label processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (5) | CN110188144A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114520815A (en) * | 2021-12-31 | 2022-05-20 | 乐爱创新(北京)科技有限公司 | Method, device, equipment and storage medium for processing big data of water environment |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111046382B (en) * | 2019-12-30 | 2024-04-02 | 武汉英迈信息科技有限公司 | Database auditing method, equipment, storage medium and device |
| CN111241081A (en) * | 2020-01-09 | 2020-06-05 | 杭州涂鸦信息技术有限公司 | IOT platform data collection method and system, readable storage medium and computer equipment |
| CN111291267B (en) * | 2020-02-17 | 2024-04-12 | 中国农业银行股份有限公司 | APP user behavior analysis method and device |
| CN112073367B (en) * | 2020-07-28 | 2022-12-09 | 智维云图(上海)智能科技有限公司 | Protocol analysis method and system for visual fire-fighting Internet of things data |
| CN112184103A (en) * | 2020-09-15 | 2021-01-05 | 苏州牧星智能科技有限公司 | Order processing method, device and system |
| CN112287015B (en) * | 2020-10-14 | 2024-03-01 | 北京易观数智科技股份有限公司 | Image generation system, image generation method, electronic device, and storage medium |
| CN112632597A (en) * | 2020-12-08 | 2021-04-09 | 国家计算机网络与信息安全管理中心 | Data desensitization method and device readable storage medium |
-
2019
- 2019-04-04 CN CN201910273168.2A patent/CN110188144A/en active Pending
- 2019-04-04 CN CN201910274370.7A patent/CN110188264A/en active Pending
- 2019-04-04 CN CN201910273174.8A patent/CN110175281A/en active Pending
- 2019-04-04 CN CN201910274466.3A patent/CN110196933A/en active Pending
- 2019-04-04 CN CN201910272512.6A patent/CN110134711A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114520815A (en) * | 2021-12-31 | 2022-05-20 | 乐爱创新(北京)科技有限公司 | Method, device, equipment and storage medium for processing big data of water environment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110188144A (en) | 2019-08-30 |
| CN110175281A (en) | 2019-08-27 |
| CN110196933A (en) | 2019-09-03 |
| CN110188264A (en) | 2019-08-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110134711A (en) | Processing method, device, equipment and the computer readable storage medium of big data | |
| US10740096B2 (en) | Meta-indexing, search, compliance, and test framework for software development | |
| CN109067890B (en) | CDN node edge computing system based on docker container | |
| CN103403699B (en) | Load in estimation flow database application cuts down the system and method for data | |
| CN109709811A (en) | A kind of edge calculations controller and its working method towards intelligence manufacture | |
| CN106355319A (en) | Safety risk assessment system and method | |
| CN109685089A (en) | The system and method for assessment models performance | |
| CN108737243A (en) | Conversation message quality detecting method and device | |
| CN110225104A (en) | Data capture method, device and terminal device | |
| WO2017071341A1 (en) | Application programming interface (api) sharing method and device | |
| CN108255936A (en) | A kind of edit methods of webpage, system and editing machine | |
| CN108492137A (en) | Telephone outbound call control and analysis, electronic device and readable storage medium storing program for executing | |
| CN112463422A (en) | Internet of things fault operation and maintenance method and device, computer equipment and storage medium | |
| CN108985461A (en) | A kind of method, apparatus and terminal device of autonomous machine study | |
| CN110781180A (en) | Data screening method and data screening device | |
| CN107231647B (en) | Network detecting method, network detection device and intelligent terminal | |
| CN112528108B (en) | Model training system, gradient aggregation method and device in model training | |
| CN107276856B (en) | Network detecting method, network detection device and intelligent terminal | |
| CN107332739B (en) | Network detecting method, network detection device and intelligent terminal | |
| CN109992957A (en) | The methods, devices and systems that templating calculates | |
| CN107360059B (en) | Network detecting method, network detection device and intelligent terminal | |
| CN115480843A (en) | Service processing method and device, electronic equipment and nonvolatile storage medium | |
| CN107294812B (en) | Network detecting method, network detection device and intelligent terminal | |
| CN109685628A (en) | Information sharing apparatus and system | |
| KR20200063343A (en) | System and method for managing operaiton in trust reality viewpointing networking infrastucture |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |