CN107977417A - A kind of ultralight amount framework method accessed towards big data content high frequency - Google Patents
A kind of ultralight amount framework method accessed towards big data content high frequency Download PDFInfo
- Publication number
- CN107977417A CN107977417A CN201711176320.2A CN201711176320A CN107977417A CN 107977417 A CN107977417 A CN 107977417A CN 201711176320 A CN201711176320 A CN 201711176320A CN 107977417 A CN107977417 A CN 107977417A
- Authority
- CN
- China
- Prior art keywords
- data
- database
- memory
- logical operation
- access
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 17
- 238000013461 design Methods 0.000 claims abstract description 10
- 238000005457 optimization Methods 0.000 claims abstract description 6
- 230000008859 change Effects 0.000 claims description 15
- 238000004364 calculation method Methods 0.000 claims description 8
- 235000013399 edible fruits Nutrition 0.000 claims description 6
- 238000006243 chemical reaction Methods 0.000 claims description 4
- 230000000694 effects Effects 0.000 claims description 4
- 230000006399 behavior Effects 0.000 claims description 3
- 238000012937 correction Methods 0.000 claims description 3
- 230000003993 interaction Effects 0.000 claims description 3
- 238000012986 modification Methods 0.000 claims description 3
- 230000004048 modification Effects 0.000 claims description 3
- 238000012544 monitoring process Methods 0.000 claims description 3
- 230000036961 partial effect Effects 0.000 claims description 3
- 230000002269 spontaneous effect Effects 0.000 claims description 3
- 230000004044 response Effects 0.000 abstract description 8
- 238000011161 development Methods 0.000 abstract description 7
- 230000008878 coupling Effects 0.000 abstract description 3
- 238000010168 coupling process Methods 0.000 abstract description 3
- 238000005859 coupling reaction Methods 0.000 abstract description 3
- 238000010586 diagram Methods 0.000 description 12
- 238000012360 testing method Methods 0.000 description 5
- 238000010276 construction Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000006855 networking Effects 0.000 description 3
- 230000008569 process Effects 0.000 description 2
- 241001282153 Scopelogadus mizolepis Species 0.000 description 1
- BUGBHKTXTAQXES-UHFFFAOYSA-N Selenium Chemical compound [Se] BUGBHKTXTAQXES-UHFFFAOYSA-N 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 238000003780 insertion Methods 0.000 description 1
- 230000037431 insertion Effects 0.000 description 1
- 238000011056 performance test Methods 0.000 description 1
- 229910052711 selenium Inorganic materials 0.000 description 1
- 239000011669 selenium Substances 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2471—Distributed queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Probability & Statistics with Applications (AREA)
- Fuzzy Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of ultralight amount framework method accessed towards big data content high frequency, comprise the following steps:A. the direct result of the access of user is obtained from platform internal memory;B. substantial amounts of data are gathered from database and carries out logical operation;C. the data loading of step b is ultimately derived from database;D. by write back data database, while calculated according to the design linkage of step c to step b to step a and store the region of memory of access strategy extremely.The above-mentioned ultralight amount framework method application accessed towards big data content high frequency or platform(It is not limited to WEB)Realize high concurrent;Response speed is substantially improved without other strategies;Developer's technical ability threshold is reduced, memory can will be reflexed to the operation of database by simply configuring;Two aspect development amounts of raising development efficiency, database and memory are reduced to the former task amount;The optimization of cooperation, develops relevant module coupling and significantly reduces.
Description
Technical field
The present invention relates to cloud security product/service technology field, and in particular to a kind of to be accessed towards big data content high frequency
Ultralight amount framework method.
Background technology
Developed in recent years, various mobile terminal APP and platform show one's talent, and the data volume presentation of user's scale of construction and generation is quick-fried
Hairdo increases.Each large platform all seemed in technologies such as big data industry of having an effect, Hadoop, MapReduce under special scenes
In heavyweight, configuration and implementation complexity limit industry development speed, and the construction cycle is tediously long in practice process, and cost is numerous
Weight.The design method only needs simply to configure, you can be calculated towards big scale of construction data content, in the WEB application of access
Obtain high concurrent, the improved efficiency of High Availabitity.
The content of the invention
It is a kind of high towards big data content the technical problem to be solved in the present invention is having overcome the deficiencies of the prior art and provide
The ultralight amount framework method that frequency accesses, application or platform(It is not limited to WEB)Realize high concurrent;Significantly carried without other strategies
Rise response speed;Developer's technical ability threshold is reduced, memory can will be reflexed to the operation of database by simply configuring;
Two aspect development amounts of raising development efficiency, database and memory are reduced to the former task amount;The optimization of cooperation, is opened
Send out module coupling relevant and significantly reduce.
To reach above-mentioned purpose, the technical solution adopted by the present invention is:It is a kind of to surpass towards what big data content high frequency accessed
Light weight framework method, comprises the following steps:
A. the direct result of the access of user is obtained from platform internal memory, an only step, without being interacted with database;With
All access in family can be obtained based on access strategy in memory as a result, access strategy uses the Hash pattern of result;
B. substantial amounts of data are gathered from database and carry out logical operation, by the cooperation of other distributed systems of original data also or
The direct operation result of the machine is fed directly to user;Logical operation module can be tied by being obtained in external distributed computing system
Fruit, when necessary stores result of calculation to the region of access strategy;
C. the accuracy of the performance direct relation step a of step b, and the opportunity that a performance part of step b is loaded by data
Determine;The data source for realizing logical operation of part scene is not necessarily loaded from the data of database, but final source
In database;
The interaction opportunity of memory and database is divided into initial phase and data change stage:Initial phase will from database
Data are once or step formula is loaded onto logical operation module, by module unified calculation conversion to the retrievable direct result of user
It is stored in access strategy module;
There are two kinds of scenes in the data change stage:(1), data-base content modification, this scene is by user or platform side's active variable
More;(2), access strategy failure, that is, occur user access be hit results, in this case by writeback policies by knot
Fruit feeds back to database, and to sacrifice partial properties requirement, to reduce the complexity of internal memory operation, guarantee is loaded onto logical operation mould
The data source of block is fixed;
Under the scene that data actively and passively change, the change of database data is finally embodied in, monitor database change becomes
Fail data in reasonable plan, the behavior changed by data interception and loading and more data replacement logical operation module;It will dissipate
Fall the data in withered and fallen time space in a manner of towards tangent plane programming or other skills realize final monitoring, can finally divide
From database and logical operation module;
D. in writeback policies, a certain threshold value is descend below with hit rate, then by write back data database, while according to step
The design linkage of c to step b to step a calculates and stores the region of memory of access strategy extremely;Tieed up in real time under connected effect
Hold the uniformity of data and keep high level hit rate.
The Hash pattern of the step a results includes the subpatterns such as error correction, speech recognition and information security.
Linkage design is carried out to the step d, step b, step c and step a, realizes the closed loop of the spontaneous adjustment of system, and
Next node is given by what all consumption-types worked, realizes that optimization consumption-type works by inside modules in node module, i.e.,
Outwards it is Distributed Calculation;Inwardly calculated for division memory.
Due to the utilization of above-mentioned technical proposal, the present invention has following advantages compared with prior art:
Using or platform(It is not limited to WEB)Realize high concurrent;Response speed is substantially improved without other strategies;Reduce exploitation
Personnel's technical ability threshold, memory can will be reflexed to by simply configuring to the operation of database;Improve development efficiency, database
The former task amount is reduced to two aspect development amounts of memory;The optimization of cooperation, develops relevant module coupling
Significantly reduce.
Brief description of the drawings
Technical solution of the present invention is described further below in conjunction with the accompanying drawings:
Fig. 1 is framework method schematic diagram of the present invention;
Fig. 2 is framework method schematic diagram of the present invention;
Fig. 3 is framework method schematic diagram of the present invention;
Fig. 4 is the framework networking schematic diagram of the present invention;
Fig. 5 is affairs percent of pass schematic diagram of the present invention;
Fig. 6 is WEB CPU of the present invention, memory usage schematic diagram;
Fig. 7 is database CPU of the present invention, memory usage schematic diagram;
Fig. 8 is database IO utilization rate schematic diagrames after present invention pressurization;
WEB CPU, memory usage schematic diagram after Fig. 9 cancels for pressurization;
Database CPU, memory usage schematic diagram after Figure 10 cancels for pressurization;
Figure 11 is affairs percent of pass schematic diagram.
Embodiment
With reference to specific embodiment, the present invention is described in further detail.
A kind of ultralight amount framework method accessed towards big data content high frequency as shown in Figure 1 to Figure 3, including following step
Suddenly:
A. the direct result of the access of user is obtained from platform internal memory, an only step, without being interacted with database;With
All access in family can be obtained based on access strategy in memory as a result, access strategy uses the Hash pattern of result;
B. substantial amounts of data are gathered from database and carry out logical operation, by the cooperation of other distributed systems of original data also or
The direct operation result of the machine is fed directly to user;Logical operation module can be tied by being obtained in external distributed computing system
Fruit, when necessary stores result of calculation to the region of access strategy;
C. the accuracy of the performance direct relation step a of step b, and the opportunity that a performance part of step b is loaded by data
Determine;The data source for realizing logical operation of part scene is not necessarily loaded from the data of database, but final source
In database;
The interaction opportunity of memory and database is divided into initial phase and data change stage:Initial phase will from database
Data are once or step formula is loaded onto logical operation module, by module unified calculation conversion to the retrievable direct result of user
It is stored in access strategy module;
There are two kinds of scenes in the data change stage:(1), data-base content modification, this scene is by user or platform side's active variable
More;(2), access strategy failure, that is, occur user access be hit results, in this case by writeback policies by knot
Fruit feeds back to database, and to sacrifice partial properties requirement, to reduce the complexity of internal memory operation, guarantee is loaded onto logical operation mould
The data source of block is fixed;
Under the scene that data actively and passively change, the change of database data is finally embodied in, monitor database change becomes
Fail data in reasonable plan, the behavior changed by data interception and loading and more data replacement logical operation module;It will dissipate
Fall the data in withered and fallen time space in a manner of towards tangent plane programming or other skills realize final monitoring, can finally divide
From database and logical operation module;
D. in writeback policies, a certain threshold value is descend below with hit rate, then by write back data database, while according to step
The design linkage of c to step b to step a calculates and stores the region of memory of access strategy extremely;Tieed up in real time under connected effect
Hold the uniformity of data and keep high level hit rate.
The Hash pattern of the step a results includes the subpatterns such as error correction, speech recognition and information security.
Linkage design is carried out to the step d, step b, step c and step a, realizes the closed loop of the spontaneous adjustment of system, and
Next node is given by what all consumption-types worked, realizes that optimization consumption-type works by inside modules in node module, i.e.,
Outwards it is Distributed Calculation;Inwardly calculated for division memory.
In an example of present design, short chain(Hereafter access the short connection of hang-up short message)Representing user needs to visit
The direct result resource asked, real data source are the basic data of tenantry, and logical operation module is needed according to short chain address
Simulated with reference to the basic data of tenantry and the basic information of transferor, we obtain after tentatively this programme is realized
Data:
Pressure point analysis:
Analyzed according to the business characteristic of the example, when with user volume increase, a large number of users concurrently clicks on hang-up short message short chain can
Pressure can be caused to system.When large concurrent clicks on hang-up short message short chain, system should return to correct advertisement URL, also
Will the insertion click details into database, and correctly generation tenantry consumes note in the case where generating mass data in time
Record, transferor commission is detailed and commission counts, and correctly deducts tenantry's expense.Then this performance test emphasis is the big use of simulation
Concurrently click on hang-up short message short chain, the bearing capacity of observing system in family.
Pressure estimation:
Configured according to client hardware, 1 client is maximum per second to simulate 1000 user concurrents, it is assumed that it is short concurrently to click on on-hook
Letter:Hang-up short message generates:Phone at the same time:Transferor scale=1:5:50:500(It is only for reference).Then 1000 user concurrents are sold
Square gauge mould is 500,000, is looked like for the transferor of 500,000 scales, and when peak value has 5 general-purpose families to make a phone call at the same time, has 5000 users at the same time
Hang up the telephone, there are 1000 users to click on hang-up short message short chain at the same time.
Test point refers to following table:
Testing tool includes
Loadruner:Concurrent instrument;
selenium :Add 100 tenantries, 100 advertisements, 1000 transferors, 1000 hang-up short message short chains of generation.
Test scene design refers to following table:
Hardware configuration and networking refer to content
Server configuration see the table below:
Test machine configuration see the table below:
Networking diagram is as shown in Figure 4.
Details as Follows for test:
Scene 1
Affairs percent of pass is as shown in Figure 5;
Concurrent is hit 110879 times, and by 110879 times, affairs percent of pass is 100%;
The average time that transaction response time server returns to URL advertising addresses is 0.031 second;
Pressure process WEB, MYSQL server CPU, memory usage difference are as shown in Figure 6 and Figure 7;
Database IO utilization rates are as shown in Figure 8 after pressurization.
After pressurization is cancelled, CPU, the memory usage of WEB, MYSQL server are distinguished as shown in Figure 9 and Figure 10.
Scene 2
Affairs percent of pass:
Concurrent is hit 101000 times, and by 100813 times, failure 187 times, affairs percent of pass is 99.814%, as shown in figure 11;
The transaction response time:
The average time that server returns to URL advertising addresses is 1.729 seconds.
Scene 3:
Affairs percent of pass 99.78%, average response time 1.248 seconds.
Scene 4:
Affairs percent of pass 100%, average response time 0.042 second.
Scene 5:
Affairs percent of pass 100%, average response time 0.877 second.
It the above is only the concrete application example of the present invention, protection scope of the present invention be not limited in any way.All use is equal
Conversion or equivalence replacement and the technical solution that is formed, all fall within rights protection scope of the present invention.
Claims (3)
1. a kind of ultralight amount framework method accessed towards big data content high frequency, it is characterised in that comprise the following steps:
The direct result of access of user is obtained from platform internal memory, an only step, without being interacted with database;User
All access can be obtained based on access strategy in memory as a result, access strategy uses the Hash pattern of result;
Gather substantial amounts of data from database and carry out logical operation, by the cooperation of other distributed systems of original data also or this
The direct operation result of machine is fed directly to user;Logical operation module can be tied by being obtained in external distributed computing system
Fruit, when necessary stores result of calculation to the region of access strategy;
The accuracy of the performance direct relation step a of step b, and a performance part of step b is determined by the opportunity that data load
It is fixed;The data source for realizing logical operation of part scene is not necessarily loaded from the data of database, but is ultimately derived from
Database;
The interaction opportunity of memory and database is divided into initial phase and data change stage:Initial phase will from database
Data are once or step formula is loaded onto logical operation module, by module unified calculation conversion to the retrievable direct result of user
It is stored in access strategy module;
There are two kinds of scenes in the data change stage:(1), data-base content modification, this scene is by user or platform side's active variable
More;(2), access strategy failure, that is, occur user access be hit results, in this case by writeback policies by knot
Fruit feeds back to database, and to sacrifice partial properties requirement, to reduce the complexity of internal memory operation, guarantee is loaded onto logical operation mould
The data source of block is fixed;
Under the scene that data actively and passively change, the change of database data is finally embodied in, monitor database change becomes
Fail data in reasonable plan, the behavior changed by data interception and loading and more data replacement logical operation module;It will dissipate
Fall the data in withered and fallen time space in a manner of towards tangent plane programming or other skills realize final monitoring, can finally divide
From database and logical operation module;
In writeback policies, a certain threshold value is descend below with hit rate, then by write back data database, while according to step c
Design linkage to step b to step a calculates and stores the region of memory of access strategy extremely;Tieed up in real time under connected effect
Hold the uniformity of data and keep high level hit rate.
2. the ultralight amount framework method according to claim 1 accessed towards big data content high frequency, it is characterised in that:Institute
Stating the Hash pattern of step a results includes the subpatterns such as error correction, speech recognition and information security.
3. the ultralight amount framework method according to claim 1 accessed towards big data content high frequency, it is characterised in that:It is right
The step d, step b, step c and step a carry out linkage design, realize the closed loop of the spontaneous adjustment of system, and by all consumption
Next node is given in type work, realizes that optimization consumption-type works by inside modules in node module, i.e., is outwards distribution
Formula calculates;Inwardly calculated for division memory.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201711176320.2A CN107977417A (en) | 2017-11-22 | 2017-11-22 | A kind of ultralight amount framework method accessed towards big data content high frequency |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201711176320.2A CN107977417A (en) | 2017-11-22 | 2017-11-22 | A kind of ultralight amount framework method accessed towards big data content high frequency |
Publications (1)
Publication Number | Publication Date |
---|---|
CN107977417A true CN107977417A (en) | 2018-05-01 |
Family
ID=62010985
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201711176320.2A Pending CN107977417A (en) | 2017-11-22 | 2017-11-22 | A kind of ultralight amount framework method accessed towards big data content high frequency |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN107977417A (en) |
Citations (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20080162443A1 (en) * | 2006-12-27 | 2008-07-03 | Fujitsu Limited | Method, apparatus, and computer program product for controlling query |
CN101493826A (en) * | 2008-12-23 | 2009-07-29 | 中兴通讯股份有限公司 | Database system based on WEB application and data management method thereof |
CN101592938A (en) * | 2009-06-30 | 2009-12-02 | 刘文祥 | Numerical control network and various system thereof |
CN102508881A (en) * | 2011-10-18 | 2012-06-20 | 国网电力科学研究院 | Method for clustering multiple nodes of memory database of power information system |
CN102646121A (en) * | 2012-02-23 | 2012-08-22 | 武汉大学 | Two-stage storage method combined with RDBMS (relational database management system) and Hadoop cloud storage |
CN102654863A (en) * | 2011-03-02 | 2012-09-05 | 华北计算机系统工程研究所 | Real-time database history data organizational management method |
CN103686716A (en) * | 2013-12-19 | 2014-03-26 | 复旦大学 | Android access control system for enhancing confidentiality and integrality |
US8862551B2 (en) * | 2005-12-29 | 2014-10-14 | Nextlabs, Inc. | Detecting behavioral patterns and anomalies using activity data |
CN106230627A (en) * | 2016-07-28 | 2016-12-14 | 浪潮软件股份有限公司 | WEB access peak relieving method based on customizable strategy |
CN106649856A (en) * | 2016-12-30 | 2017-05-10 | 金蝶软件(中国)有限公司 | Database access device, system and method |
-
2017
- 2017-11-22 CN CN201711176320.2A patent/CN107977417A/en active Pending
Patent Citations (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8862551B2 (en) * | 2005-12-29 | 2014-10-14 | Nextlabs, Inc. | Detecting behavioral patterns and anomalies using activity data |
US20080162443A1 (en) * | 2006-12-27 | 2008-07-03 | Fujitsu Limited | Method, apparatus, and computer program product for controlling query |
CN101493826A (en) * | 2008-12-23 | 2009-07-29 | 中兴通讯股份有限公司 | Database system based on WEB application and data management method thereof |
CN101592938A (en) * | 2009-06-30 | 2009-12-02 | 刘文祥 | Numerical control network and various system thereof |
CN102654863A (en) * | 2011-03-02 | 2012-09-05 | 华北计算机系统工程研究所 | Real-time database history data organizational management method |
CN102508881A (en) * | 2011-10-18 | 2012-06-20 | 国网电力科学研究院 | Method for clustering multiple nodes of memory database of power information system |
CN102646121A (en) * | 2012-02-23 | 2012-08-22 | 武汉大学 | Two-stage storage method combined with RDBMS (relational database management system) and Hadoop cloud storage |
CN103686716A (en) * | 2013-12-19 | 2014-03-26 | 复旦大学 | Android access control system for enhancing confidentiality and integrality |
CN106230627A (en) * | 2016-07-28 | 2016-12-14 | 浪潮软件股份有限公司 | WEB access peak relieving method based on customizable strategy |
CN106649856A (en) * | 2016-12-30 | 2017-05-10 | 金蝶软件(中国)有限公司 | Database access device, system and method |
Non-Patent Citations (1)
Title |
---|
张延松等: "内存数据库可控的page-color优化技术研究" * |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US20210326729A1 (en) | Recommendation Model Training Method and Related Apparatus | |
CN106470133B (en) | System pressure testing method and device | |
Di et al. | Characterizing and modeling cloud applications/jobs on a Google data center | |
JP6716727B2 (en) | Streaming data distributed processing method and apparatus | |
CN109309596B (en) | Pressure testing method and device and server | |
CN105791178B (en) | Message assemble method and device | |
CN102999608A (en) | System and method for tree table demonstration of large data | |
Li et al. | Optimizing energy of http requests in android applications | |
CN113568577B (en) | Distributed grouping storage method based on alliance block chain | |
CN108667840A (en) | Injection loophole detection method and device | |
CN109191287A (en) | A kind of sharding method, device and the electronic equipment of block chain intelligence contract | |
CN103677983B (en) | The dispatching method and device of application | |
CN110474820A (en) | Traffic playback method, device, electronic equipment | |
CN107888717A (en) | A kind of domain name determines method, apparatus and electronic equipment | |
US20190215262A1 (en) | System and method for dynamically testing networked target systems | |
CN109978547A (en) | Risk behavior control method and system, equipment and storage medium | |
Varghese et al. | DocLite: A docker-based lightweight cloud benchmarking tool | |
CN106909436A (en) | Produce the method and system of the dependency relation of virtual machine message queue application program | |
CN106897313B (en) | Mass user service preference evaluation method and device | |
US20120209584A1 (en) | Advanced Metering Infrastructure Simulation | |
Cao | Load balancing design of web cluster based on Nginx under novel virtualization platform | |
Chahal et al. | Migrating a recommendation system to cloud using ml workflow | |
Mittal et al. | Cloud testing-the future of contemporary software testing | |
CN109947736A (en) | The method and system calculated in real time | |
CN107977417A (en) | A kind of ultralight amount framework method accessed towards big data content high frequency |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20180501 |
|
WD01 | Invention patent application deemed withdrawn after publication |