CN110263079A - Data distribution enters library processing method and system - Google Patents
Data distribution enters library processing method and system Download PDFInfo
- Publication number
- CN110263079A CN110263079A CN201910470399.2A CN201910470399A CN110263079A CN 110263079 A CN110263079 A CN 110263079A CN 201910470399 A CN201910470399 A CN 201910470399A CN 110263079 A CN110263079 A CN 110263079A
- Authority
- CN
- China
- Prior art keywords
- data
- storage
- persistence
- need
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Abstract
The embodiment of the present invention discloses a kind of data distribution and enters library processing method and system, wherein method includes the following steps: to determine whether data to be put in storage need persistence according to the business demand of data to be put in storage, to need the data to be put in storage of persistence that storage configuration is arranged, data to be put in storage are stored to corresponding database according to storage configuration.Using the present invention, the concurrent request of mass data being supported, realizing the function of data distribution statistics, corresponding processing is done by preset configuration, the flexibility to database manipulation can be improved.
Description
Technical field
The present invention relates to field of computer technology more particularly to a kind of data distribution to enter library processing method and system.
Background technique
When different databases are written in real time data by computer data processing system, usually directly by data distribution extremely
Different databases when database is concurrently written, needs a powerful data resource interface to handle a large amount of concurrent request.
Summary of the invention
The embodiment of the present invention provides a kind of data distribution and enters library processing method and system, can support the concurrent of mass data
The function of data distribution statistics is realized in request.
First aspect of the embodiment of the present invention provides a kind of data distribution and enters library processing method, it may include:
Determine whether data to be put in storage need persistence according to the business demand of data to be put in storage；
To need the data to be put in storage of persistence that storage configuration is arranged；
Data to be put in storage are stored to corresponding database according to storage configuration.
Further, the above method further include:
The corresponding business demand of data to be put in storage is determined according to acquired data loading request.
Further, the above method further include:
The data to be put in storage for not needing persistence are stored into Distributed Message Queue kafka.
Further, the above method further include:
Monitor the data volume of data to be put in storage；
When the variation abnormality of data volume, output abnormality prompt information, at the time of abnormal prompt information instruction exception occurs
The location of with abnormal data.
Further, the above method further include:
Storage data, which are treated, based on user demand carries out data filtering.
Second aspect of the embodiment of the present invention provides a kind of data distribution storage processing system, it may include:
It is put in storage data detection module, for determining whether data to be put in storage need to hold according to the business demand of data to be put in storage
Longization；
It is put in storage configuration settings module, for configuring to need the data to be put in storage of persistence that storage is arranged；
First data memory module, for being stored data to be put in storage to corresponding database according to storage configuration.
Further, above system further include:
Business demand determining module, for determining the corresponding business of data to be put in storage according to acquired data loading request
Demand.
Further, above system further include:
Second data memory module, the data to be put in storage for that will not need persistence are stored to Distributed Message Queue
In kafka.
Further, above system further include:
Data monitoring module, for monitoring the data volume of data to be put in storage；
Abnormal prompt module, for when the variation abnormality of data volume, output abnormality prompt information, abnormal prompt information to refer to
At the time of showing abnormal occur and the location of abnormal data.
Further, above system further include:
Data filtering module carries out data filtering for treating storage data based on user demand.
In embodiments of the present invention, determine whether storage data need persistence by the business demand of data to be put in storage,
To need the data to be put in storage of persistence that storage configuration is arranged, data to be put in storage are stored to corresponding data according to storage configuration
Library.It supports the concurrent request of mass data and the distribution statistics of data, corresponding processing is done by preset configuration, is improved
Flexibility to database manipulation.
Detailed description of the invention
In order to more clearly explain the embodiment of the invention or the technical proposal in the existing technology, to embodiment or will show below
There is attached drawing needed in technical description to be briefly described.
Fig. 1 is the flow diagram that a kind of data distribution provided in an embodiment of the present invention enters library processing method；
Fig. 2 is a kind of structural schematic diagram of data distribution storage processing system provided in an embodiment of the present invention.
Specific embodiment
Following will be combined with the drawings in the embodiments of the present invention, and technical solution in the embodiment of the present invention carries out clear, complete
Site preparation description.
Data distribution provided in an embodiment of the present invention, which enters library processing method, can be applied to processing mass data concurrent request
Application scenarios.
Below in conjunction with attached drawing 1, library processing method is entered to data distribution provided in an embodiment of the present invention and is described in detail.
Referring to Figure 1, for the embodiment of the invention provides the flow diagrams that a kind of data distribution is put in storage facture.Such as figure
Shown in 1, the embodiment of the present invention the method may include following steps S101- step S103.
S101 determines whether data to be put in storage need persistence according to the business demand of data to be put in storage.
It should be noted that the available data loading request of above system, and can be determined according to the request wait be put in storage
The corresponding business demand of data, so-called business demand can indicate the corresponding business process of Data Data to be put in storage.
In the specific implementation, above system can provide unified entrance to all data to be put in storage, can according to
The business demand of storage data determines whether the data need persistence.It is understood that needing the data of persistence can be with
It is subsequent data to be analyzed useful, needs the data of persistence, the data for not needing persistence can be only to current
Operate useful data.
In the specific implementation of the embodiment of the present invention, common business demand has offline statistics and real-time statistics, right
It is not concerned with the timeliness of data in offline statistics, but compares and pays close attention to later analysis, needing will be under such data persistence
Come, such as the product how many common daily user has used system to provide, face takes how many when to user on the product
Between, these are all the data for needing Long-term analysis, are got off then requiring persistence.Some business are only to need to pay close attention to current shape
State, for example product is checked either with or without delay machine, such a demand only focuses on current state, then there is no need to these data
Persistence is got off.
In an alternative embodiment, for not needing the data of persistence, above system can will data be put in storage store to
In Distributed Message Queue kafka, also it can store to the data cache module of other storage ephemeral datas, it is straight for other services
Connect use.
S102, to need the data to be put in storage of persistence that storage configuration is arranged.
It should be noted that the data to be put in storage for needing persistence, storage configuration can be arranged in above system for it,
The configuration can be the pre-set configuration of operation that developer as needed carries out database, for example, can match in advance
Set config.By carrying out storage configuration in advance, so that according to demand to operation more flexibleization of database.
S103 stores data to be put in storage to corresponding database according to storage configuration.
Data to be put in storage are stored to corresponding database specifically, above system can be configured according to above-mentioned storage.It can
Choosing, before being stored in database, above system can also receive the user demand of user's input, and can be based on the demand pair
Data to be put in storage carry out data filtering, filter out the unwanted data of user.
In an alternative embodiment, task can be set when detecting the request of each data loading in above system
The data volume of data to be put in storage is monitored, for example, monitoring is picked up, is put down, is bought, behavioral datas amount and the distinct device such as touching
Data volume can be referred to output abnormality prompt information, above-mentioned abnormal prompt information when the variation abnormality of above-mentioned data volume
At the time of showing abnormal occur and the location of abnormal data.Optionally, above system can be monitored in CPU or Memory
The function of (memory) cracking can react in CPU or Memory or data volume exception and solve this respect
The problem of.
In embodiments of the present invention, determine whether storage data need persistence by the business demand of data to be put in storage,
To need the data to be put in storage of persistence that storage configuration is arranged, data to be put in storage are stored to corresponding data according to storage configuration
Library.It supports the concurrent request of mass data and the distribution statistics of data, corresponding processing is done by preset configuration, is improved
Flexibility to database manipulation.
It should be noted that step shown in the flowchart of the accompanying drawings can be in such as a group of computer-executable instructions
It is executed in computer system, although also, logical order is shown in flow charts, and it in some cases, can be with not
The sequence being same as herein executes shown or described step.
Below in conjunction with attached drawing 2, describe in detail to data distribution provided in an embodiment of the present invention storage processing system.
It should be noted that attached data distribution shown in Fig. 2 is put in storage processing system, for executing the side of embodiment illustrated in fig. 1 of the present invention
Method, for ease of description, only parts related to embodiments of the present invention are shown, disclosed by specific technical details, please refers to
Present invention embodiment shown in FIG. 1.
Fig. 2 is referred to, for the embodiment of the invention provides the structural schematic diagrams that a kind of data distribution is put in storage processing system.Such as
Shown in Fig. 2, the data distribution storage processing system 10 of the embodiment of the present invention may include: storage data detection module 101, storage
Configuration settings module 102, the first data memory module 103, business demand determining module 104, the second data memory module 105,
Data monitoring module 106, abnormal prompt module 107 and data filtering module 108.
Be put in storage data detection module 101, for according to the business demands of data to be put in storage determine data to be put in storage whether need
Want persistence.
It should be noted that the available data loading of above system 10 is requested, business demand determining module 104 can root
Determine that the corresponding business demand of data to be put in storage, so-called business demand can indicate that Data Data to be put in storage is corresponding according to the request
Business process.
In the specific implementation, above system 10 can provide all data to be put in storage unified entrance, it is put in storage data
Detection module 101 can determine whether the data need persistence according to the business demand of data to be put in storage.It is understood that
It needs the data of persistence can be and subsequent data is analyzed useful, need the data of persistence, do not need persistence
Data can be only data useful to current operation.
In the specific implementation of the embodiment of the present invention, common business demand has offline statistics and real-time statistics, right
It is not concerned with the timeliness of data in offline statistics, but compares and pays close attention to later analysis, needing will be under such data persistence
Come, such as the product how many common daily user has used system to provide, face takes how many when to user on the product
Between, these are all the data for needing Long-term analysis, are got off then requiring persistence.Some business are only to need to pay close attention to current shape
State, for example product is checked either with or without delay machine, such a demand only focuses on current state, then there is no need to these data
Persistence is got off.
In an alternative embodiment, for not needing the data of persistence, the second data memory module 105 can will be wait be put in storage
Data are stored into Distributed Message Queue kafka, also be can store to the data cache module of other storage ephemeral datas, are supplied
Other services directly use.
It is put in storage configuration settings module 102, for configuring to need the data to be put in storage of persistence that storage is arranged.
It should be noted that the data to be put in storage for needing persistence, storage configuration settings module 102 can set for it
It is placed in library configuration, which can be the pre-set configuration of operation that developer as needed carries out database, for example,
Config can be pre-configured with.By carrying out storage configuration in advance, so that according to demand to operation more flexibleization of database.
First data memory module 103, for being stored data to be put in storage to corresponding database according to storage configuration.
In the specific implementation, the first data memory module 103 can according to above-mentioned storage configure will data be put in storage store to
Corresponding database.Optionally, before being stored in database, above system 10 can also receive the user demand of user's input,
Data filtering module 108 can treat storage data based on the demand and carry out data filtering, filter out the unwanted data of user.
In an alternative embodiment, above system 10 is when detecting the request of each data loading, data monitoring mould
Task can be set to monitor the data volume of data to be put in storage in block 106, for example, monitoring is picked up, puts down, and buys, the behaviors such as touching
The data volume of data volume and distinct device, when the variation abnormality of above-mentioned data volume, abnormal prompt module 107 can export different
At the time of normal prompt information, above-mentioned abnormal prompt information can indicate abnormal occur and the location of abnormal data.Optionally,
Above system can be monitored can when the function of CPU perhaps Memory (memory) is in CPU or Memory or data volume exception
It reacts with cracking and solves the problems, such as this respect.
In embodiments of the present invention, determine whether storage data need persistence by the business demand of data to be put in storage,
To need the data to be put in storage of persistence that storage configuration is arranged, data to be put in storage are stored to corresponding data according to storage configuration
Library.It supports the concurrent request of mass data and the distribution statistics of data, corresponding processing is done by preset configuration, is improved
Flexibility to database manipulation.
Those of ordinary skill in the art will appreciate that realizing all or part of the process in above-described embodiment method, being can be with
Relevant hardware is instructed to complete by computer program, the program can be stored in computer-readable storage medium
In, the program is when being executed, it may include such as the process of the embodiment of above-mentioned each method.Wherein, the storage medium can be magnetic
Dish, CD, read-only memory (Read-Only Memory, ROM) or random access memory (Random Access
Memory, RAM) etc..
The above disclosure is only the preferred embodiments of the present invention, cannot limit the right model of the present invention with this certainly
It encloses, therefore equivalent changes made in accordance with the claims of the present invention, is still within the scope of the present invention.
Claims (10)
1. a kind of data distribution enters library processing method characterized by comprising
Determine whether the data to be put in storage need persistence according to the business demand of data to be put in storage；
To need the data to be put in storage of persistence that storage configuration is arranged；
It is configured according to the storage and stores the data to be put in storage to corresponding database.
2. the method according to claim 1, wherein the method also includes:
The corresponding business demand of data to be put in storage is determined according to acquired data loading request.
3. the method according to claim 1, wherein the method also includes:
The data to be put in storage for not needing persistence are stored into Distributed Message Queue kafka.
4. the method according to claim 1, wherein the method also includes:
Monitor the data volume of the data to be put in storage；
When the variation abnormality of the data volume, output abnormality prompt information, what the abnormal prompt information instruction occurred extremely
The location of moment and abnormal data.
5. the method according to claim 1, wherein the method also includes:
Data filtering is carried out to the data to be put in storage based on user demand.
6. a kind of data distribution is put in storage processing system characterized by comprising
It is put in storage data detection module, determines whether the data to be put in storage need to hold for the business demand according to data to be put in storage
Longization；
It is put in storage configuration settings module, for configuring to need the data to be put in storage of persistence that storage is arranged；
First data memory module stores the data to be put in storage to corresponding database for being configured according to the storage.
7. system according to claim 6, which is characterized in that the system also includes:
Business demand determining module, for determining the corresponding business need of data to be put in storage according to acquired data loading request
It asks.
8. system according to claim 6, which is characterized in that the system also includes:
Second data memory module, the data to be put in storage for that will not need persistence are stored to Distributed Message Queue kafka
In.
9. system according to claim 6, which is characterized in that the system also includes:
Data monitoring module, for monitoring the data volume of the data to be put in storage；
Abnormal prompt module, for when the variation abnormality of the data volume, output abnormality prompt information, the abnormal prompt to be believed
Breath instruction is abnormal at the time of occur and the location of abnormal data.
10. system according to claim 6, which is characterized in that the system also includes:
Data filtering module, for carrying out data filtering to the data to be put in storage based on user demand.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910470399.2A CN110263079A (en) | 2019-05-31 | 2019-05-31 | Data distribution enters library processing method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910470399.2A CN110263079A (en) | 2019-05-31 | 2019-05-31 | Data distribution enters library processing method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110263079A true CN110263079A (en) | 2019-09-20 |
Family
ID=67916227
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910470399.2A Pending CN110263079A (en) | 2019-05-31 | 2019-05-31 | Data distribution enters library processing method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110263079A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103024014A (en) * | 2012-12-03 | 2013-04-03 | 北京京东世纪贸易有限公司 | Method and system for distributing and processing mass data passing through message queues |
| CN104834860A (en) * | 2015-05-09 | 2015-08-12 | 福建六壬网安股份有限公司 | Dynamic warehousing method for security events |
| CN108734414A (en) * | 2018-06-05 | 2018-11-02 | 山东省海洋资源与环境研究院 | A kind of marine environmental monitoring data normalization storage and the system of analysis and appraisal |
| US20180357264A1 (en) * | 2015-05-29 | 2018-12-13 | Nuodb, Inc. | Table partitioning within distributed database systems |
| US20190034460A1 (en) * | 2017-07-27 | 2019-01-31 | Sap Se | Co-locating Microservice Persistence Containers within Tenant-Specific Database |
| CN109522315A (en) * | 2018-10-26 | 2019-03-26 | 苏宁易购集团股份有限公司 | A kind of data base processing method and system |
-
2019
- 2019-05-31 CN CN201910470399.2A patent/CN110263079A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103024014A (en) * | 2012-12-03 | 2013-04-03 | 北京京东世纪贸易有限公司 | Method and system for distributing and processing mass data passing through message queues |
| CN104834860A (en) * | 2015-05-09 | 2015-08-12 | 福建六壬网安股份有限公司 | Dynamic warehousing method for security events |
| US20180357264A1 (en) * | 2015-05-29 | 2018-12-13 | Nuodb, Inc. | Table partitioning within distributed database systems |
| US20190034460A1 (en) * | 2017-07-27 | 2019-01-31 | Sap Se | Co-locating Microservice Persistence Containers within Tenant-Specific Database |
| CN108734414A (en) * | 2018-06-05 | 2018-11-02 | 山东省海洋资源与环境研究院 | A kind of marine environmental monitoring data normalization storage and the system of analysis and appraisal |
| CN109522315A (en) * | 2018-10-26 | 2019-03-26 | 苏宁易购集团股份有限公司 | A kind of data base processing method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109634728B (en) | Job scheduling method and device, terminal equipment and readable storage medium | |
| CN108427581A (en) | System micro services method and terminal device | |
| CN110389748A (en) | Business data processing method and terminal device | |
| CN111190892B (en) | Method and device for processing abnormal data in data backfilling | |
| CN109962855A (en) | A kind of current-limiting method of WEB server, current-limiting apparatus and terminal device | |
| CN109669976A (en) | Data service method and equipment based on ETL | |
| CN107544849A (en) | A kind of WEB server concurrency management method and system | |
| CN106484459B (en) | Flow control method and device applied to JavaScript | |
| CN106357791A (en) | Method, device and system for processing services | |
| CN109359019A (en) | Application program capacity monitoring method, device, electronic equipment and storage medium | |
| CN109597837A (en) | Storage method, querying method and the relevant device of time series data | |
| CN111552521A (en) | Application data reporting method, device, server and storage medium | |
| JP2023548985A (en) | Quantification of robotic process automation related resource usage | |
| JPH07244644A (en) | Agent management system | |
| CN109241037A (en) | A kind of method, apparatus that realizing database processing, equipment and storage medium | |
| CN110263079A (en) | Data distribution enters library processing method and system | |
| CN116719623A (en) | Job scheduling method, job result processing method and device | |
| CN107818501B (en) | Actuarial method and device | |
| CN116089040A (en) | Service flow scheduling method and device, electronic equipment and storage medium | |
| US20090083747A1 (en) | Method for managing application programs by utilizing redundancy and load balance | |
| CN108062224A (en) | Data read-write method, device and computing device based on file handle | |
| CN107656849A (en) | A kind of software system performance positioning problems method and device | |
| CN113448867A (en) | Software pressure testing method and device | |
| CN111352802A (en) | Monitoring processing method, device, equipment and storage medium of network bus | |
| CN110597216A (en) | Transaction arrangement method and device for manufacturing execution system and readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190920 |
|
| RJ01 | Rejection of invention patent application after publication |