CN103761248B - The method and system of data query are carried out using memory database - Google Patents
The method and system of data query are carried out using memory database Download PDFInfo
- Publication number
- CN103761248B CN103761248B CN201310717225.4A CN201310717225A CN103761248B CN 103761248 B CN103761248 B CN 103761248B CN 201310717225 A CN201310717225 A CN 201310717225A CN 103761248 B CN103761248 B CN 103761248B
- Authority
- CN
- China
- Prior art keywords
- entity
- loading
- memory database
- data
- memory
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2453—Query optimisation
- G06F16/24534—Query rewriting; Transformation
- G06F16/24539—Query rewriting; Transformation using cached or materialised query results
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Present invention relates particularly to the method and system that data query is carried out using memory database, wherein, method comprises the following steps：1）Monitoring service is established, entity, and the loading SQL of entity, loading strategy are set in monitoring service, loading SQL is the SQL performed when entity loads in the business library where the entity, and loading strategy is the data alternative condition of entity；2）For each entity, with reference to corresponding loading strategy, using corresponding data in the entity are selected in the business library where corresponding loading SQL from the entity, it is loaded into memory database, the interior database is disposed using different machine；3）Querying command is received, data corresponding with querying command are found out in memory database and are returned.The present invention accelerates the speed of data query.
Description
【Technical field】
The invention belongs to enterprise information management system field (ERP), is related in information management system field on checking, examining
Meter and associated monitoring business normal data administrative skill, and in particular to using memory database carry out data query method and
System.
【Background technology】
During data monitoring, it is sometimes necessary to the data of multiple business libraries to be inquired about, business library is database,
There is at least one entity in business library, contains multiple business datums in entity；Current querying method is in some conventional sides
In case generally in the following way：
1st, in the database（Such as Oracle）, by way of the tables of data that DBLINK inquires about needs is by chained list
Main business storehouse is created to, because the establishment of chained list can take database connection, database connection resource can be caused excessive, limited
Connection resource in the case of produce excessive connection query performance can be caused low, serious situation can cause database corruption,
Creating chained list at the same time needs to lift authority of the active user in Oracle, and there are some potential safety problems；
2nd, by the data query of multiple business libraries into memory, then multiple data sets are calculated in memory, so
When data volume is excessive（Such as million, ten million bar）Substantial amounts of memory headroom can be taken, influences the performance of production system, serious meeting
Cause production system to be delayed the situation of machine, and need to handle single service logic for each inquiry, autgmentability is poor.
【The content of the invention】
The invention solves first technical problem be to provide it is a kind of using memory database carry out data query side
Method, it accelerates the speed of data query.
The invention solves second technical problem be to provide and a kind of carry out data query using memory database and be
System, it accelerates the speed of data query.
Above-mentioned first technical problem solves by the following technical programs：
The method that data query is carried out using memory database, it is characterised in that comprise the following steps：
1）Monitoring service is established, entity, and the loading SQL of entity, loading strategy, loading are set in monitoring service
SQL is the SQL performed when entity loads in the business library where the entity, and loading strategy is the data alternative condition of entity；
2）For each entity, with reference to corresponding loading strategy, the industry where corresponding loading SQL from the entity is used
Business selects corresponding data in the entity in storehouse, is loaded into memory database, and the interior database is disposed using different machine；
3）Querying command is received, data corresponding with querying command are found out in memory database and are returned.
In step 2）In, an entity state table is correspondingly generated in memory database, entity state token, which is loaded with, to be added
The each entity attributes being downloaded in memory database, attribute include entity name, loading SQL, loading strategy.
In step 2）Each entity specially in monitoring service carries out following operation：Judged first according to entity name
The corresponding data of the entity, either with or without record, internal storage data are loaded into if not recording by the entity in entity state table
In storehouse, mark to carry the entity attributes in entity state；If record then judge the entity this time load strategy whether with
The loading strategy of correspondent entity is consistent in entity state table；If consistent, this time loading of the entity need not carry out；If
It is inconsistent, then the data of the entity in memory database are deleted, and it is interior the entity this time to be needed the data loaded be loaded into
In deposit data storehouse, while update the loading number of the entity in entity state table, nearest load time, loading SQL, loading plan
Slightly.
In step 1）In monitoring service in set entity loading sequence；In step 2）In, by the loading sequence pair
Each entity is regularly loaded.
In step 1）With step 2）Between further include step 1 '):All entities on entity table are traveled through, according to the nearest of entity
Load time, loading number, physical size calculate the loading frequency of each entity, then according to loading frequency from high in the end
Entity is loaded into memory database successively.
The attribute further includes the entity row record number of loading entity, entity row record size；In step 2）In, each
Before secondary loading entity, judge whether the sum of the pre-load amount of entity and the currently used amount of memory database are more than memory valve
Value, if greater than the entity deletion that will then meet deletion condition from memory database, while will be with this in entity state table
The corresponding attribute of entity is deleted a bit；Entity row record number × entity row record size of entity pre-load amount=current loading entity
× calculation error ratio, calculation error is than=actual the ratio for consuming memory and calculated value.
Further include timing and delete data step：At regular intervals, when the currently used amount for determining memory database is big
The data that the entity of deletion condition will be met from memory database in memory threshold values are deleted, while will be with entity state table
These corresponding attributes of entity for meeting deletion condition are deleted.
Above-mentioned second technical problem solves by the following technical programs：
The system that data query is carried out using memory database, it is characterised in that including：
1）Setup module, for establishing monitoring service, sets entity in monitoring service, and the loading SQL of entity plus
Strategy is carried, loading SQL is the SQL performed when entity loads in the business library where the entity, and loading strategy is the data of entity
Alternative condition；
2）Load-on module, for each entity, with reference to corresponding loading strategy, using corresponding loading SQL from this
Corresponding data in the entity are selected in business library where entity, are loaded into memory database, the interior database uses
Different machine deployment；
3）Enquiry module, for receiving querying command, data corresponding with querying command is found out in memory database and are returned
Return.
Correspondingly generate an entity state table in load-on module, in memory database, entity state token be loaded with by
The each entity attributes being loaded into memory database, attribute include entity name, loading SQL, loading strategy.
In load-on module following operation is carried out for each entity in monitoring service：Judged first according to entity name
The corresponding data of the entity, either with or without record, internal storage data are loaded into if not recording by the entity in entity state table
In storehouse, mark to carry the entity attributes in entity state；If record then judge the entity this time load strategy whether with
The loading strategy of correspondent entity is consistent in entity state table；If consistent, this time loading of the entity need not carry out；If
It is inconsistent, then the data of the entity in memory database are deleted, and it is interior the entity this time to be needed the data loaded be loaded into
In deposit data storehouse, while update the loading number of the entity in entity state table, nearest load time, loading SQL, loading plan
Slightly.
The loading sequence of entity is set in the monitoring service in setup module；In load step, by the loading time
The each entity of ordered pair is regularly loaded.
Further include intelligent sequencing module:All entities on entity table are traveled through, according to the nearest load time of entity, loading time
Number, physical size calculate the loading frequency of each entity, are then from high in the end successively loaded entity according to loading frequency
Into memory database.
The attribute further includes the entity row record number of loading entity, entity row record size；In load-on module, every
Before once loading entity, judge whether the sum of the pre-load amount of entity and the currently used amount of memory database are more than memory valve
Value, if greater than the entity deletion that will then meet deletion condition from memory database, while will be with this in entity state table
The corresponding attribute of entity is deleted a bit；Entity row record number × entity row record size of entity pre-load amount=current loading entity
× calculation error ratio, calculation error is than=actual the ratio for consuming memory and calculated value.
Further include timing and delete data step：At regular intervals, when the currently used amount for determining memory database is big
The data that the entity of deletion condition will be met from memory database in memory threshold values are deleted, while will be with entity state table
These corresponding attributes of entity for meeting deletion condition are deleted.
As seen from the above technical solution, the present invention uses memory database as middle database, by each industry before inquiry
The data in business storehouse are drawn into memory database by ETL, and inquiry is then performed in memory database, can thus be accomplished more
The inter-library inquiry of business library, solves the problems, such as that inter-library inquiry is slower with query performance, as shown in figure 4, experimental data proves, this hair
Bright method several orders of magnitude higher than the efficiency of traditional rule service inquiry.The data volume for reducing ETL by loading strategy and extracting,
Lift query performance.Trans-sectoral business library inquiry, and the inter-library project plan comparison of tradition are more efficient by the way of memory database, can
Autgmentability is more preferable, and safer, because occupying data by the way of DBLINK chained lists in conventional solution
The connection resource in storehouse, can cause database connection excessive and cannot respond to new request, and in addition user will also assign renewal
Authority, can there are security risk, and use memory database scheme can farthest mitigate application server and business
The pressure of database, even if the memory database machine of delaying nor affects on the normal operation of application service, it is only necessary to restart internal storage data
Storehouse services, and whole process is operated by UI completely.Search efficiency, because involved business datum is all in memory
In database, the magnetic disc i/o time of traditional database is eliminated, search efficiency several orders of magnitude higher than traditional database,
As long as ensureing the high hit rate of data loading, the query performance of system relatively greatly improves.
【Brief description of the drawings】
Fig. 1 is the flow chart of one the method for the present invention of embodiment；
Fig. 2 is the different machine deployment diagram of memory database；
Fig. 3 is the specific example figure of entity state table；
Fig. 4 is using with not using memory database to carry out the comparative result figure of data query；
Fig. 5 is the rule schema of entity loading；
Fig. 6 is the structure diagram of two present system of embodiment.
【Embodiment】
Embodiment one
As shown in Figure 1, carrying out the method for data query using memory database, comprise the following steps：
1）Setting steps：Monitoring service is established, entity, and the loading SQL of entity, loading plan are set in monitoring service
Slightly, loading SQL is the SQL performed when entity loads in the business library where the entity, and loading strategy selects for the data of entity
Condition, loading strategy are different and different according to current context environment；
2）Load step：For each entity, with reference to corresponding loading strategy, using corresponding loading SQL from the reality
Corresponding data in the entity are selected in business library where body, are loaded into memory database, the interior database is using different
Machine is disposed；
3）Query steps：Querying command is received, data corresponding with querying command are found out in memory database and are returned.On
State the physics deployment explanation of memory database：Build dedicated memory database server（In general, 64 behaviour of machine configuration requirement
Make system, it is more than 32G memories, the network bandwidth of 4 cores and above mainstream CPU, gigabit/10,000,000,000,64 JDK）, it is all be related to it is trans-sectoral
The business datum in business storehouse is all first loaded onto in memory database（Support the databases such as Oracle, SqlServer, DB2）, subsequently look into
Ask and performed in memory database.In practical application, memory database server is supported embedded（It is embedded into application service）, same to machine
（With application service in same server）, the deployment of different machine（From application service in two different servers）Three models；See
Fig. 2, the present invention are disposed using different machine, and deployment supports Windows, linux system at the same time.
Wherein, in step 2）In, an entity state table is correspondingly generated in memory database, entity state token is loaded with
For each entity attributes, as shown in figure 3, attribute includes entity name, last load time, loading number, entity row note
Recording number, entity row record size, loading strategy and loading SQL, entity name includes entity English name, entity Chinese.
Entity English name：The English description of entity, entity can be established in memory database by table name of entity name
Table.
Entity Chinese：The Chinese description of entity.
The last load time：The time of entity last time loading.
Load number：The number that entity loads altogether.
Entity row records number：The data being loaded in the entity share how many row records.
Entity row record size：The size of the row record of entity, computational methods consume big for the field of entity all properties
It is the sum of small.
Loading strategy：Record the loading strategy of the entity；
Load SQL：The SQL performed when entity loads in the business library where its entity.
In order to preferably carry out data query, it is improved in the present embodiment from following aspects.
First, loading entity efficiency is improved
In actual operation, can there is a situation where that some entities need repeated loading, system is lifting loading efficiency, is saved
Load time, every time load entity when can carry out it is following operate, therefore, in above-mentioned steps 2）In every time loading entity when can carry out
Operate below：First according to entity name come judge the entity in entity state table either with or without record, if not recording
The corresponding data of the entity are loaded into memory database, mark to carry the entity attributes in entity state；If record
Then judge whether this time loading for the entity be tactful consistent with the loading strategy of correspondent entity in entity state table；If consistent,
Then this time loading of the entity need not carry out；If it is inconsistent, the data of the entity in memory database are deleted, and will
The entity this time needs the data loaded to be loaded into memory database, while updates the loading time of the entity in entity state table
Several, nearest load time, loading SQL, loading strategy.
Entity this step avoids identical loading strategy reduces the data volume of loading next with then no longer loading,
Improve the efficiency of inquiry.
2nd, entity load mode
There are both schemes：
1st, manually planning：In step 1）In monitoring service in set entity loading sequence；In step 2）In, by described
Loading sequence regularly loads each entity.This is manually arranged as needed, easy to neatly tackle needs.
2nd, intelligence arranges：In step 1）With step 2）Between further include step 1 '):Travel through all entities on entity table, root
Factually the nearest load time of body, loading number, physical size calculate the loading frequency of each entity, then according to loading
Frequency from high in the end successively loads entity into memory database, thus by the of a relatively high entity of access frequency all load into
Memory, this loading is by scheduler program in non-traffic active stage（As at night）Loading, so just eliminated loading at second day
Process, so as to shorten query time, lifting search efficiency.
3rd, the garbage reclamation action based on memory database
Since memory is limited resources, there is relevant security mechanism when entity loads, to control the remaining amount of memory.
Specifically there are two schemes：
3.1 entities are deleted when loading：In step 2）In, before entity is loaded each time, judge the pre-load amount of entity
Whether it is more than memory threshold values with the sum of the currently used amount of memory database（Memory database maximum uses memory value, Ke Yigen
Freely set according to actual conditions）, if greater than the entity deletion that will then meet deletion condition from memory database, while in reality
Attribute corresponding with these entities is deleted in body state table；The entity row record number of entity pre-load amount=current loading entity
× entity row record size × calculation error ratio, calculation error is than=actual the ratio for consuming memory and calculated value.
3.2 timings are deleted：At regular intervals（Such as 10 minutes, can freely it be set according to actual conditions）, work as judgement
Currently used amount to memory database is more than memory threshold values, will meet deletion bar from memory database using " genetic algorithm "
The entity of part is deleted, while deletes attribute corresponding with these entities in entity state table.
Above-mentioned deletion condition is typically to be set according to entity usage time, access times, frequency of use, it is therefore an objective to will
The entity being of little use is deleted.
4th, UI is visualized
The configuration of memory database, operation, attended operation all visualize, there is provided and special UI interfaces are operated,
User experience is improved, while provides the operating console of memory database, similar to PL/SQL, is logged in and authorized by user
Afterwards, DML, DDL, DCL action statement, while the load condition of query entity can be performed directly in memory database, it is convenient
Implementation/developer is safeguarded and issue track.
Embodiment two
As shown in fig. 6, the system of data query is carried out using memory database, including：
1）Setup module, for establishing monitoring service, sets entity in monitoring service, and the loading SQL of entity plus
Strategy is carried, loading SQL is the SQL performed when entity loads in the business library where the entity, and loading strategy is the data of entity
Alternative condition, loading strategy are different and different according to current context environment；
2）Load-on module, for each entity, with reference to corresponding loading strategy, using corresponding loading SQL from this
Corresponding data in the entity are selected in business library where entity, are loaded into memory database, the interior database uses
Different machine deployment；
3）Enquiry module, for receiving querying command, data corresponding with querying command is found out in memory database and are returned
Return.
The physics deployment explanation of above-mentioned memory database：Build dedicated memory database server（In general, machine configures
It is required that it is more than 64 bit manipulation systems, 32G memories, the network bandwidth of 4 cores and above mainstream CPU, gigabit/10,000,000,000,64 JDK）, own
It is related to all first being loaded onto in memory database across the business datum of business library（Support the data such as Oracle, SqlServer, DB2
Storehouse）, subsequent query is in memory database execution.In practical application, memory database server is supported embedded（It is embedded into using clothes
Business）, same to machine（With application service in same server）, the deployment of different machine（From application service in two different servers）Three kinds
Pattern；See Fig. 2, the present invention is disposed using different machine, and deployment supports Windows, linux system at the same time.
Wherein, an entity state table is correspondingly generated in load-on module, in memory database, entity state token carries
Have and be directed to each entity attributes, as shown in figure 3, attribute includes entity name, last load time, loading number, entity row
Recording number, entity row record size, loading strategy and loading SQL, entity name includes entity English name, entity Chinese name
Claim.
Entity English name：The English description of entity, entity can be established in memory database by table name of entity name
Table.
Entity Chinese：The Chinese description of entity.
The last load time：The time of entity last time loading.
Load number：The number that entity loads altogether.
Entity row records number：The data being loaded in the entity share how many row records.
Entity row record size：The size of the row record of entity, computational methods consume big for the field of entity all properties
It is the sum of small.
Loading strategy：Record the loading strategy of the entity；
Load SQL：The SQL performed when entity loads in the business library where its entity.
In order to preferably carry out data query, it is improved in the present embodiment from following aspects.
First, loading entity efficiency is improved
In actual operation, can there is a situation where that some entities need repeated loading, system is lifting loading efficiency, is saved
Load time, can be carried out when loading entity every time it is following operate, therefore, in above-mentioned load-on module every time loading entity when can be into
The following operation of row：First according to entity name come judge the entity in entity state table either with or without record, if do not recorded
Then the corresponding data of the entity are loaded into memory database, mark to carry the entity attributes in entity state；If note
Record then judges whether this time loading for the entity be tactful consistent with the loading strategy of correspondent entity in entity state table；If one
Cause, then this time loading of the entity need not carry out；If it is inconsistent, the data of the entity in memory database are deleted,
And this time need the data loaded to be loaded into memory database the entity, while update the entity in entity state table plus
Carry number, nearest load time, loading SQL, loading strategy.
Entity this step avoids identical loading strategy reduces the data volume of loading next with then no longer loading,
Improve the efficiency of inquiry.
2nd, entity load mode
There are both schemes：
1st, manually planning：The loading sequence of entity is set in the monitoring service in setup module；In load-on module, press
The loading sequence regularly loads each entity.This is manually arranged as needed, easy to neatly tackle
Need.
2nd, intelligence arranges：Further include intelligent sequencing module:All entities on entity table are traveled through, according to the nearest loading of entity
Time, loading number, physical size calculate the loading frequency of each entity, then according to loading frequency from high in the end successively
Entity is loaded into memory database, is thus all loaded the of a relatively high entity of access frequency into memory, this loading passes through
Scheduler program is in non-traffic active stage（As at night）Loading, so in second day process for just eliminating loading, is looked into so as to shorten
Ask time, lifting search efficiency.
3rd, the garbage reclamation action based on memory database
Since memory is limited resources, there is relevant security mechanism when entity loads, to control the remaining amount of memory.
Specifically there are two schemes：
3.1 entities are deleted when loading：In load-on module, before entity is loaded each time, the preloading of entity is judged
Whether the sum of amount and the currently used amount of memory database are more than memory threshold values（Memory database maximum uses memory value, can be with
Freely set according to actual conditions）, if greater than the entity deletion that will then meet deletion condition from memory database, while
Attribute corresponding with these entities is deleted in entity state table；The entity row record of entity pre-load amount=current loading entity
Number × entity row record size × calculation error ratio, calculation error is than=actual the ratio for consuming memory and calculated value.
3.2 timings are deleted：At regular intervals（Such as 10 minutes, can freely it be set according to actual conditions）, work as judgement
Currently used amount to memory database is more than memory threshold values, will meet deletion bar from memory database using " genetic algorithm "
The entity of part is deleted, while deletes attribute corresponding with these entities in entity state table.
Above-mentioned deletion condition is typically to be set according to entity usage time, access times, frequency of use, it is therefore an objective to will
The entity being of little use is deleted.
4th, UI is visualized
The configuration of memory database, operation, attended operation all visualize, there is provided and special UI interfaces are operated,
User experience is improved, while provides the operating console of memory database, similar to PL/SQL, is logged in and authorized by user
Afterwards, DML, DDL, DCL action statement, while the load condition of query entity can be performed directly in memory database, it is convenient
Implementation/developer is safeguarded and issue track.
The present invention is not limited to above-described embodiment, simple replacement based on above-described embodiment, not making creative work,
Should belong to the invention discloses scope.
Claims (10)
1. the method for data query is carried out using memory database, it is characterised in that comprise the following steps：
1) monitoring service is established, entity, and the loading SQL of entity, loading strategy are set in monitoring service, and loading SQL is
The SQL performed when entity loads in the business library where the entity, loading strategy are the data alternative condition of entity；
2) each entity is directed to, with reference to corresponding loading strategy, uses the business library where corresponding loading SQL from the entity
Corresponding data in middle selection entity, are loaded into memory database, the memory database is disposed using different machine；
3) querying command is received, data corresponding with querying command are found out in memory database and are returned；
An entity state table is correspondingly generated in step 2), in memory database, entity state token, which is loaded with, to be loaded into
Each entity attributes in memory database, attribute include entity name, loading SQL, loading strategy；
Each entity in step 2) is specially monitoring service carries out following operation：The reality is judged according to entity name first
The corresponding data of the entity, either with or without record, memory database are loaded into if not recording by body in entity state table
In, mark to carry the entity attributes in entity state；If record then judge the entity this time load strategy whether with reality
The loading strategy of correspondent entity is consistent in body state table；If consistent, this time loading of the entity need not carry out；If no
Unanimously, then the data of the entity in memory database are deleted, and this time needs the data loaded to be loaded into memory the entity
In database, while update loading number, nearest load time, loading SQL, the loading strategy of the entity in entity state table.
2. the method according to claim 1 that data query is carried out using memory database, it is characterised in that in step 1)
In monitoring service in set entity loading sequence；In step 2), each entity is carried out by the loading sequence secondary
Load to sequence.
3. the method according to claim 1 that data query is carried out using memory database, it is characterised in that in step 1)
Step 1 is further included between step 2) '):All entities on entity table are traveled through, according to the nearest load time of entity, loading time
Number, physical size calculate the loading frequency of each entity, are then from high in the end successively loaded entity according to loading frequency
Into memory database.
4. the method according to claim 1 that data query is carried out using memory database, it is characterised in that the attribute
Further include entity row record number, the entity row record size of loading entity；In step 2), before entity is loaded each time,
Judge whether the sum of the pre-load amount of entity and the currently used amount of memory database are more than memory threshold values, if greater than then from interior
The entity for meeting deletion condition is deleted in deposit data storehouse, while deletes attribute corresponding with these entities in entity state table
Remove；Entity row record number × entity row record size × calculation error ratio of entity pre-load amount=current loading entity, meter
Error is calculated than=actual the ratio for consuming memory and calculated value.
5. the method according to claim 1 that data query is carried out using memory database, it is characterised in that it is fixed to further include
When delete data step：At regular intervals, when the currently used amount for determining memory database is more than memory threshold values from memory
The data that the entity of deletion condition will be met in database are deleted, while will meet deletion condition with these in entity state table
The corresponding attribute of entity delete.
6. the system of data query is carried out using memory database, it is characterised in that including：
1) setup module, for establishing monitoring service, sets entity, and the loading SQL of entity, loading plan in monitoring service
Slightly, loading SQL is the SQL performed when entity loads in the business library where the entity, and loading strategy selects for the data of entity
Condition；
2) load-on module, for each entity, with reference to corresponding loading strategy, using corresponding loading SQL from the entity
Corresponding data in the entity are selected in the business library at place, are loaded into memory database, the memory database is using different
Machine is disposed；
3) enquiry module, for receiving querying command, data corresponding with querying command is found out in memory database and are returned；
An entity state table is correspondingly generated in load-on module, in memory database, entity state token, which is loaded with, to be loaded
Each entity attributes into memory database, attribute include entity name, loading SQL, loading strategy；
In load-on module following operation is carried out for each entity in monitoring service：The reality is judged according to entity name first
The corresponding data of the entity, either with or without record, memory database are loaded into if not recording by body in entity state table
In, mark to carry the entity attributes in entity state；If record then judge the entity this time load strategy whether with reality
The loading strategy of correspondent entity is consistent in body state table；If consistent, this time loading of the entity need not carry out；If no
Unanimously, then the data of the entity in memory database are deleted, and this time needs the data loaded to be loaded into memory the entity
In database, while update loading number, nearest load time, loading SQL, the loading strategy of the entity in entity state table.
7. the system according to claim 6 that data query is carried out using memory database, it is characterised in that mould is being set
The loading sequence of entity is set in monitoring service in the block；In load step, each entity is carried out by the loading sequence
Regularly load.
8. the system according to claim 6 that data query is carried out using memory database, it is characterised in that further include intelligence
Can sorting module:All entities on entity table are traveled through, are calculated according to the nearest load time of entity, loading number, physical size
Go out the loading frequency of each entity, then from high in the end successively loaded entity into memory database according to loading frequency.
9. the system according to claim 6 that data query is carried out using memory database, it is characterised in that the attribute
Further include entity row record number, the entity row record size of loading entity；In load-on module, each time load entity it
Before, judge whether the sum of the pre-load amount of entity and the currently used amount of memory database are more than memory threshold values, if greater than then
The entity for meeting deletion condition is deleted from memory database, while will category corresponding with these entities in entity state table
Property delete；Entity row record number × entity row record size × calculation error ratio of entity pre-load amount=current loading entity
Rate, calculation error is than=actual the ratio for consuming memory and calculated value.
10. the system according to claim 6 that data query is carried out using memory database, it is characterised in that further include
Data step is deleted in timing：At regular intervals, when the currently used amount for determining memory database is more than memory threshold values from interior
The data that the entity of deletion condition will be met in deposit data storehouse are deleted, while will meet deletion bar with these in entity state table
The corresponding attribute of entity of part is deleted.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310717225.4A CN103761248B (en) | 2013-12-23 | 2013-12-23 | The method and system of data query are carried out using memory database |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310717225.4A CN103761248B (en) | 2013-12-23 | 2013-12-23 | The method and system of data query are carried out using memory database |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103761248A CN103761248A (en) | 2014-04-30 |
| CN103761248B true CN103761248B (en) | 2018-04-17 |
Family
ID=50528487
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310717225.4A Active CN103761248B (en) | 2013-12-23 | 2013-12-23 | The method and system of data query are carried out using memory database |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103761248B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106202080A (en) * | 2015-04-30 | 2016-12-07 | 中国移动通信集团公司 | A kind of webpage rendering intent, server and terminal unit |
| CN110888939A (en) * | 2018-09-06 | 2020-03-17 | 北京京东尚科信息技术有限公司 | Data management method and device |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101221585A (en) * | 2008-02-03 | 2008-07-16 | 华为技术有限公司 | Data storage method and device |
| CN101320392A (en) * | 2008-07-17 | 2008-12-10 | 中兴通讯股份有限公司 | High-capacity data access method and device of internal memory database |
| CN101329685A (en) * | 2008-07-30 | 2008-12-24 | 烽火通信科技股份有限公司 | Implementing method of memory database on household gateway |
| CN102054034A (en) * | 2010-12-27 | 2011-05-11 | 华中科技大学 | Implementation method for business basic data persistence of enterprise information system |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8046557B2 (en) * | 2005-12-05 | 2011-10-25 | Intelitrac Inc. | Apparatus and method for on-demand in-memory database management platform |
| CN101414917B (en) * | 2007-10-19 | 2011-03-16 | 华为技术有限公司 | Method for saving internal memory space, data management network element and network system |
| CN102279885A (en) * | 2011-08-16 | 2011-12-14 | 中兴通讯股份有限公司 | Method and device for operating data by memory database |
-
2013
- 2013-12-23 CN CN201310717225.4A patent/CN103761248B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101221585A (en) * | 2008-02-03 | 2008-07-16 | 华为技术有限公司 | Data storage method and device |
| CN101320392A (en) * | 2008-07-17 | 2008-12-10 | 中兴通讯股份有限公司 | High-capacity data access method and device of internal memory database |
| CN101329685A (en) * | 2008-07-30 | 2008-12-24 | 烽火通信科技股份有限公司 | Implementing method of memory database on household gateway |
| CN102054034A (en) * | 2010-12-27 | 2011-05-11 | 华中科技大学 | Implementation method for business basic data persistence of enterprise information system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103761248A (en) | 2014-04-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20220277025A1 (en) | System and Method for Analysis and Management of Data Distribution in a Distributed Database Environment | |
| JP6025149B2 (en) | System and method for managing data | |
| Santos et al. | Real-time data warehouse loading methodology | |
| CN109271450A (en) | Database synchronization method, device, server and storage medium | |
| CN103688259B (en) | For the method by compressing and file storage carries out automaticdata placement | |
| US9135071B2 (en) | Selecting processing techniques for a data flow task | |
| CN106777225A (en) | The moving method and system of a kind of data | |
| CN109144994A (en) | Index updating method, system and relevant apparatus | |
| CN104462185B (en) | A kind of digital library's cloud storage system based on mixed structure | |
| US11294958B2 (en) | Managing a distributed knowledge graph | |
| CN103095805A (en) | Cloud storage system of data intelligent and decentralized management | |
| CN102651007A (en) | Method and device for managing database indexes | |
| CN105074724A (en) | Efficient query processing using histograms in a columnar database | |
| CN110209728A (en) | A kind of Distributed Heterogeneous Database synchronous method, electronic equipment and storage medium | |
| CN103455526A (en) | ETL (extract-transform-load) data processing method, device and system | |
| CN104615710A (en) | Electronic map frame data updating method | |
| CN102982186B (en) | Based on maintaining method and the system of the range partition table of oracle database system | |
| CN110019469A (en) | Distributed data base data processing method, device, storage medium and electronic device | |
| CN106991190A (en) | A kind of database automatically creates subdata base system | |
| CN106648897A (en) | SOLR cluster extension method and system supporting resource balancing | |
| CN108446315A (en) | Big data moving method, device, equipment and storage medium | |
| CN107590257A (en) | A kind of data base management method and device | |
| CN106780157B (en) | Ceph-based power grid multi-temporal model storage and management system and method | |
| CN108153804A (en) | A kind of metadata daily record update method of symmetric distributed file system | |
| CN109669975A (en) | A kind of industry big data processing system and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |