CN107977417A - A kind of ultralight amount framework method accessed towards big data content high frequency - Google Patents

A kind of ultralight amount framework method accessed towards big data content high frequency Download PDF

Info

Publication number
CN107977417A
CN107977417A CN201711176320.2A CN201711176320A CN107977417A CN 107977417 A CN107977417 A CN 107977417A CN 201711176320 A CN201711176320 A CN 201711176320A CN 107977417 A CN107977417 A CN 107977417A
Authority
CN
China
Prior art keywords
data
database
memory
logical operation
access
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201711176320.2A
Other languages
Chinese (zh)
Inventor
叶俊
王秋月
许玲玲
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Jiangsu Wing Enterprise Cloud Communications Technology Co Ltd
Original Assignee
Jiangsu Wing Enterprise Cloud Communications Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Jiangsu Wing Enterprise Cloud Communications Technology Co Ltd filed Critical Jiangsu Wing Enterprise Cloud Communications Technology Co Ltd
Priority to CN201711176320.2A priority Critical patent/CN107977417A/en
Publication of CN107977417A publication Critical patent/CN107977417A/en
Pending legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24Querying
    • G06F16/245Query processing
    • G06F16/2458Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
    • G06F16/2471Distributed queries
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23Updating

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Mathematical Physics (AREA)
  • Computational Linguistics (AREA)
  • Software Systems (AREA)
  • Probability & Statistics with Applications (AREA)
  • Fuzzy Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention discloses a kind of ultralight amount framework method accessed towards big data content high frequency, comprise the following steps:A. the direct result of the access of user is obtained from platform internal memory;B. substantial amounts of data are gathered from database and carries out logical operation;C. the data loading of step b is ultimately derived from database;D. by write back data database, while calculated according to the design linkage of step c to step b to step a and store the region of memory of access strategy extremely.The above-mentioned ultralight amount framework method application accessed towards big data content high frequency or platform(It is not limited to WEB)Realize high concurrent;Response speed is substantially improved without other strategies;Developer's technical ability threshold is reduced, memory can will be reflexed to the operation of database by simply configuring;Two aspect development amounts of raising development efficiency, database and memory are reduced to the former task amount;The optimization of cooperation, develops relevant module coupling and significantly reduces.

Description

A kind of ultralight amount framework method accessed towards big data content high frequency
Technical field
The present invention relates to cloud security product/service technology field, and in particular to a kind of to be accessed towards big data content high frequency Ultralight amount framework method.
Background technology
Developed in recent years, various mobile terminal APP and platform show one's talent, and the data volume presentation of user's scale of construction and generation is quick-fried Hairdo increases.Each large platform all seemed in technologies such as big data industry of having an effect, Hadoop, MapReduce under special scenes In heavyweight, configuration and implementation complexity limit industry development speed, and the construction cycle is tediously long in practice process, and cost is numerous Weight.The design method only needs simply to configure, you can be calculated towards big scale of construction data content, in the WEB application of access Obtain high concurrent, the improved efficiency of High Availabitity.
The content of the invention
It is a kind of high towards big data content the technical problem to be solved in the present invention is having overcome the deficiencies of the prior art and provide The ultralight amount framework method that frequency accesses, application or platform(It is not limited to WEB)Realize high concurrent;Significantly carried without other strategies Rise response speed;Developer's technical ability threshold is reduced, memory can will be reflexed to the operation of database by simply configuring; Two aspect development amounts of raising development efficiency, database and memory are reduced to the former task amount;The optimization of cooperation, is opened Send out module coupling relevant and significantly reduce.
To reach above-mentioned purpose, the technical solution adopted by the present invention is:It is a kind of to surpass towards what big data content high frequency accessed Light weight framework method, comprises the following steps:
A. the direct result of the access of user is obtained from platform internal memory, an only step, without being interacted with database;With All access in family can be obtained based on access strategy in memory as a result, access strategy uses the Hash pattern of result;
B. substantial amounts of data are gathered from database and carry out logical operation, by the cooperation of other distributed systems of original data also or The direct operation result of the machine is fed directly to user;Logical operation module can be tied by being obtained in external distributed computing system Fruit, when necessary stores result of calculation to the region of access strategy;
C. the accuracy of the performance direct relation step a of step b, and the opportunity that a performance part of step b is loaded by data Determine;The data source for realizing logical operation of part scene is not necessarily loaded from the data of database, but final source In database;
The interaction opportunity of memory and database is divided into initial phase and data change stage:Initial phase will from database Data are once or step formula is loaded onto logical operation module, by module unified calculation conversion to the retrievable direct result of user It is stored in access strategy module;
There are two kinds of scenes in the data change stage:(1), data-base content modification, this scene is by user or platform side's active variable More;(2), access strategy failure, that is, occur user access be hit results, in this case by writeback policies by knot Fruit feeds back to database, and to sacrifice partial properties requirement, to reduce the complexity of internal memory operation, guarantee is loaded onto logical operation mould The data source of block is fixed;
Under the scene that data actively and passively change, the change of database data is finally embodied in, monitor database change becomes Fail data in reasonable plan, the behavior changed by data interception and loading and more data replacement logical operation module;It will dissipate Fall the data in withered and fallen time space in a manner of towards tangent plane programming or other skills realize final monitoring, can finally divide From database and logical operation module;
D. in writeback policies, a certain threshold value is descend below with hit rate, then by write back data database, while according to step The design linkage of c to step b to step a calculates and stores the region of memory of access strategy extremely;Tieed up in real time under connected effect Hold the uniformity of data and keep high level hit rate.
The Hash pattern of the step a results includes the subpatterns such as error correction, speech recognition and information security.
Linkage design is carried out to the step d, step b, step c and step a, realizes the closed loop of the spontaneous adjustment of system, and Next node is given by what all consumption-types worked, realizes that optimization consumption-type works by inside modules in node module, i.e., Outwards it is Distributed Calculation;Inwardly calculated for division memory.
Due to the utilization of above-mentioned technical proposal, the present invention has following advantages compared with prior art:
Using or platform(It is not limited to WEB)Realize high concurrent;Response speed is substantially improved without other strategies;Reduce exploitation Personnel's technical ability threshold, memory can will be reflexed to by simply configuring to the operation of database;Improve development efficiency, database The former task amount is reduced to two aspect development amounts of memory;The optimization of cooperation, develops relevant module coupling Significantly reduce.
Brief description of the drawings
Technical solution of the present invention is described further below in conjunction with the accompanying drawings:
Fig. 1 is framework method schematic diagram of the present invention;
Fig. 2 is framework method schematic diagram of the present invention;
Fig. 3 is framework method schematic diagram of the present invention;
Fig. 4 is the framework networking schematic diagram of the present invention;
Fig. 5 is affairs percent of pass schematic diagram of the present invention;
Fig. 6 is WEB CPU of the present invention, memory usage schematic diagram;
Fig. 7 is database CPU of the present invention, memory usage schematic diagram;
Fig. 8 is database IO utilization rate schematic diagrames after present invention pressurization;
WEB CPU, memory usage schematic diagram after Fig. 9 cancels for pressurization;
Database CPU, memory usage schematic diagram after Figure 10 cancels for pressurization;
Figure 11 is affairs percent of pass schematic diagram.
Embodiment
With reference to specific embodiment, the present invention is described in further detail.
A kind of ultralight amount framework method accessed towards big data content high frequency as shown in Figure 1 to Figure 3, including following step Suddenly:
A. the direct result of the access of user is obtained from platform internal memory, an only step, without being interacted with database;With All access in family can be obtained based on access strategy in memory as a result, access strategy uses the Hash pattern of result;
B. substantial amounts of data are gathered from database and carry out logical operation, by the cooperation of other distributed systems of original data also or The direct operation result of the machine is fed directly to user;Logical operation module can be tied by being obtained in external distributed computing system Fruit, when necessary stores result of calculation to the region of access strategy;
C. the accuracy of the performance direct relation step a of step b, and the opportunity that a performance part of step b is loaded by data Determine;The data source for realizing logical operation of part scene is not necessarily loaded from the data of database, but final source In database;
The interaction opportunity of memory and database is divided into initial phase and data change stage:Initial phase will from database Data are once or step formula is loaded onto logical operation module, by module unified calculation conversion to the retrievable direct result of user It is stored in access strategy module;
There are two kinds of scenes in the data change stage:(1), data-base content modification, this scene is by user or platform side's active variable More;(2), access strategy failure, that is, occur user access be hit results, in this case by writeback policies by knot Fruit feeds back to database, and to sacrifice partial properties requirement, to reduce the complexity of internal memory operation, guarantee is loaded onto logical operation mould The data source of block is fixed;
Under the scene that data actively and passively change, the change of database data is finally embodied in, monitor database change becomes Fail data in reasonable plan, the behavior changed by data interception and loading and more data replacement logical operation module;It will dissipate Fall the data in withered and fallen time space in a manner of towards tangent plane programming or other skills realize final monitoring, can finally divide From database and logical operation module;
D. in writeback policies, a certain threshold value is descend below with hit rate, then by write back data database, while according to step The design linkage of c to step b to step a calculates and stores the region of memory of access strategy extremely;Tieed up in real time under connected effect Hold the uniformity of data and keep high level hit rate.
The Hash pattern of the step a results includes the subpatterns such as error correction, speech recognition and information security.
Linkage design is carried out to the step d, step b, step c and step a, realizes the closed loop of the spontaneous adjustment of system, and Next node is given by what all consumption-types worked, realizes that optimization consumption-type works by inside modules in node module, i.e., Outwards it is Distributed Calculation;Inwardly calculated for division memory.
In an example of present design, short chain(Hereafter access the short connection of hang-up short message)Representing user needs to visit The direct result resource asked, real data source are the basic data of tenantry, and logical operation module is needed according to short chain address Simulated with reference to the basic data of tenantry and the basic information of transferor, we obtain after tentatively this programme is realized Data:
Pressure point analysis:
Analyzed according to the business characteristic of the example, when with user volume increase, a large number of users concurrently clicks on hang-up short message short chain can Pressure can be caused to system.When large concurrent clicks on hang-up short message short chain, system should return to correct advertisement URL, also Will the insertion click details into database, and correctly generation tenantry consumes note in the case where generating mass data in time Record, transferor commission is detailed and commission counts, and correctly deducts tenantry's expense.Then this performance test emphasis is the big use of simulation Concurrently click on hang-up short message short chain, the bearing capacity of observing system in family.
Pressure estimation:
Configured according to client hardware, 1 client is maximum per second to simulate 1000 user concurrents, it is assumed that it is short concurrently to click on on-hook Letter:Hang-up short message generates:Phone at the same time:Transferor scale=1:5:50:500(It is only for reference).Then 1000 user concurrents are sold Square gauge mould is 500,000, is looked like for the transferor of 500,000 scales, and when peak value has 5 general-purpose families to make a phone call at the same time, has 5000 users at the same time Hang up the telephone, there are 1000 users to click on hang-up short message short chain at the same time.
Test point refers to following table:
Testing tool includes
Loadruner:Concurrent instrument;
selenium :Add 100 tenantries, 100 advertisements, 1000 transferors, 1000 hang-up short message short chains of generation.
Test scene design refers to following table:
Hardware configuration and networking refer to content
Server configuration see the table below:
Test machine configuration see the table below:
Networking diagram is as shown in Figure 4.
Details as Follows for test:
Scene 1
Affairs percent of pass is as shown in Figure 5;
Concurrent is hit 110879 times, and by 110879 times, affairs percent of pass is 100%;
The average time that transaction response time server returns to URL advertising addresses is 0.031 second;
Pressure process WEB, MYSQL server CPU, memory usage difference are as shown in Figure 6 and Figure 7;
Database IO utilization rates are as shown in Figure 8 after pressurization.
After pressurization is cancelled, CPU, the memory usage of WEB, MYSQL server are distinguished as shown in Figure 9 and Figure 10.
Scene 2
Affairs percent of pass:
Concurrent is hit 101000 times, and by 100813 times, failure 187 times, affairs percent of pass is 99.814%, as shown in figure 11;
The transaction response time:
The average time that server returns to URL advertising addresses is 1.729 seconds.
Scene 3:
Affairs percent of pass 99.78%, average response time 1.248 seconds.
Scene 4:
Affairs percent of pass 100%, average response time 0.042 second.
Scene 5:
Affairs percent of pass 100%, average response time 0.877 second.
It the above is only the concrete application example of the present invention, protection scope of the present invention be not limited in any way.All use is equal Conversion or equivalence replacement and the technical solution that is formed, all fall within rights protection scope of the present invention.

Claims (3)

1. a kind of ultralight amount framework method accessed towards big data content high frequency, it is characterised in that comprise the following steps:
The direct result of access of user is obtained from platform internal memory, an only step, without being interacted with database;User All access can be obtained based on access strategy in memory as a result, access strategy uses the Hash pattern of result;
Gather substantial amounts of data from database and carry out logical operation, by the cooperation of other distributed systems of original data also or this The direct operation result of machine is fed directly to user;Logical operation module can be tied by being obtained in external distributed computing system Fruit, when necessary stores result of calculation to the region of access strategy;
The accuracy of the performance direct relation step a of step b, and a performance part of step b is determined by the opportunity that data load It is fixed;The data source for realizing logical operation of part scene is not necessarily loaded from the data of database, but is ultimately derived from Database;
The interaction opportunity of memory and database is divided into initial phase and data change stage:Initial phase will from database Data are once or step formula is loaded onto logical operation module, by module unified calculation conversion to the retrievable direct result of user It is stored in access strategy module;
There are two kinds of scenes in the data change stage:(1), data-base content modification, this scene is by user or platform side's active variable More;(2), access strategy failure, that is, occur user access be hit results, in this case by writeback policies by knot Fruit feeds back to database, and to sacrifice partial properties requirement, to reduce the complexity of internal memory operation, guarantee is loaded onto logical operation mould The data source of block is fixed;
Under the scene that data actively and passively change, the change of database data is finally embodied in, monitor database change becomes Fail data in reasonable plan, the behavior changed by data interception and loading and more data replacement logical operation module;It will dissipate Fall the data in withered and fallen time space in a manner of towards tangent plane programming or other skills realize final monitoring, can finally divide From database and logical operation module;
In writeback policies, a certain threshold value is descend below with hit rate, then by write back data database, while according to step c Design linkage to step b to step a calculates and stores the region of memory of access strategy extremely;Tieed up in real time under connected effect Hold the uniformity of data and keep high level hit rate.
2. the ultralight amount framework method according to claim 1 accessed towards big data content high frequency, it is characterised in that:Institute Stating the Hash pattern of step a results includes the subpatterns such as error correction, speech recognition and information security.
3. the ultralight amount framework method according to claim 1 accessed towards big data content high frequency, it is characterised in that:It is right The step d, step b, step c and step a carry out linkage design, realize the closed loop of the spontaneous adjustment of system, and by all consumption Next node is given in type work, realizes that optimization consumption-type works by inside modules in node module, i.e., is outwards distribution Formula calculates;Inwardly calculated for division memory.
CN201711176320.2A 2017-11-22 2017-11-22 A kind of ultralight amount framework method accessed towards big data content high frequency Pending CN107977417A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201711176320.2A CN107977417A (en) 2017-11-22 2017-11-22 A kind of ultralight amount framework method accessed towards big data content high frequency

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201711176320.2A CN107977417A (en) 2017-11-22 2017-11-22 A kind of ultralight amount framework method accessed towards big data content high frequency

Publications (1)

Publication Number Publication Date
CN107977417A true CN107977417A (en) 2018-05-01

Family

ID=62010985

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201711176320.2A Pending CN107977417A (en) 2017-11-22 2017-11-22 A kind of ultralight amount framework method accessed towards big data content high frequency

Country Status (1)

Country Link
CN (1) CN107977417A (en)

Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080162443A1 (en) * 2006-12-27 2008-07-03 Fujitsu Limited Method, apparatus, and computer program product for controlling query
CN101493826A (en) * 2008-12-23 2009-07-29 中兴通讯股份有限公司 Database system based on WEB application and data management method thereof
CN101592938A (en) * 2009-06-30 2009-12-02 刘文祥 Numerical control network and various system thereof
CN102508881A (en) * 2011-10-18 2012-06-20 国网电力科学研究院 Method for clustering multiple nodes of memory database of power information system
CN102646121A (en) * 2012-02-23 2012-08-22 武汉大学 Two-stage storage method combined with RDBMS (relational database management system) and Hadoop cloud storage
CN102654863A (en) * 2011-03-02 2012-09-05 华北计算机系统工程研究所 Real-time database history data organizational management method
CN103686716A (en) * 2013-12-19 2014-03-26 复旦大学 Android access control system for enhancing confidentiality and integrality
US8862551B2 (en) * 2005-12-29 2014-10-14 Nextlabs, Inc. Detecting behavioral patterns and anomalies using activity data
CN106230627A (en) * 2016-07-28 2016-12-14 浪潮软件股份有限公司 WEB access peak relieving method based on customizable strategy
CN106649856A (en) * 2016-12-30 2017-05-10 金蝶软件(中国)有限公司 Database access device, system and method

Patent Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8862551B2 (en) * 2005-12-29 2014-10-14 Nextlabs, Inc. Detecting behavioral patterns and anomalies using activity data
US20080162443A1 (en) * 2006-12-27 2008-07-03 Fujitsu Limited Method, apparatus, and computer program product for controlling query
CN101493826A (en) * 2008-12-23 2009-07-29 中兴通讯股份有限公司 Database system based on WEB application and data management method thereof
CN101592938A (en) * 2009-06-30 2009-12-02 刘文祥 Numerical control network and various system thereof
CN102654863A (en) * 2011-03-02 2012-09-05 华北计算机系统工程研究所 Real-time database history data organizational management method
CN102508881A (en) * 2011-10-18 2012-06-20 国网电力科学研究院 Method for clustering multiple nodes of memory database of power information system
CN102646121A (en) * 2012-02-23 2012-08-22 武汉大学 Two-stage storage method combined with RDBMS (relational database management system) and Hadoop cloud storage
CN103686716A (en) * 2013-12-19 2014-03-26 复旦大学 Android access control system for enhancing confidentiality and integrality
CN106230627A (en) * 2016-07-28 2016-12-14 浪潮软件股份有限公司 WEB access peak relieving method based on customizable strategy
CN106649856A (en) * 2016-12-30 2017-05-10 金蝶软件(中国)有限公司 Database access device, system and method

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
张延松等: "内存数据库可控的page-color优化技术研究" *

Similar Documents

Publication Publication Date Title
US20210326729A1 (en) Recommendation Model Training Method and Related Apparatus
CN106470133B (en) System pressure testing method and device
Di et al. Characterizing and modeling cloud applications/jobs on a Google data center
JP6716727B2 (en) Streaming data distributed processing method and apparatus
CN109309596B (en) Pressure testing method and device and server
CN105791178B (en) Message assemble method and device
CN102999608A (en) System and method for tree table demonstration of large data
Li et al. Optimizing energy of http requests in android applications
CN113568577B (en) Distributed grouping storage method based on alliance block chain
CN108667840A (en) Injection loophole detection method and device
CN109191287A (en) A kind of sharding method, device and the electronic equipment of block chain intelligence contract
CN103677983B (en) The dispatching method and device of application
CN110474820A (en) Traffic playback method, device, electronic equipment
CN107888717A (en) A kind of domain name determines method, apparatus and electronic equipment
US20190215262A1 (en) System and method for dynamically testing networked target systems
CN109978547A (en) Risk behavior control method and system, equipment and storage medium
Varghese et al. DocLite: A docker-based lightweight cloud benchmarking tool
CN106909436A (en) Produce the method and system of the dependency relation of virtual machine message queue application program
CN106897313B (en) Mass user service preference evaluation method and device
US20120209584A1 (en) Advanced Metering Infrastructure Simulation
Cao Load balancing design of web cluster based on Nginx under novel virtualization platform
Chahal et al. Migrating a recommendation system to cloud using ml workflow
Mittal et al. Cloud testing-the future of contemporary software testing
CN109947736A (en) The method and system calculated in real time
CN107977417A (en) A kind of ultralight amount framework method accessed towards big data content high frequency

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
WD01 Invention patent application deemed withdrawn after publication

Application publication date: 20180501

WD01 Invention patent application deemed withdrawn after publication