CN106960016A - The method and system of data are inserted to database - Google Patents
The method and system of data are inserted to database Download PDFInfo
- Publication number
- CN106960016A CN106960016A CN201710124487.8A CN201710124487A CN106960016A CN 106960016 A CN106960016 A CN 106960016A CN 201710124487 A CN201710124487 A CN 201710124487A CN 106960016 A CN106960016 A CN 106960016A
- Authority
- CN
- China
- Prior art keywords
- data
- header information
- buffer queue
- database
- data header
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 38
- 238000012795 verification Methods 0.000 claims description 18
- 238000012937 correction Methods 0.000 claims description 7
- 238000003780 insertion Methods 0.000 abstract description 31
- 230000037431 insertion Effects 0.000 abstract description 31
- 238000012550 audit Methods 0.000 description 17
- 230000006870 function Effects 0.000 description 12
- 238000005457 optimization Methods 0.000 description 7
- 230000008859 change Effects 0.000 description 6
- 238000013500 data storage Methods 0.000 description 5
- 230000000694 effects Effects 0.000 description 5
- 230000008569 process Effects 0.000 description 4
- 230000006872 improvement Effects 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 241001269238 Data Species 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 238000000151 deposition Methods 0.000 description 2
- 238000007796 conventional method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2308—Concurrency control
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The disclosure is directed to a kind of method and system that data are inserted to database, it is related to database technical field, wherein the method for inserting data to database includes：Obtain the data header information and performance data information for being inserted into data；The data header information that gets is judged with the presence or absence of in the data head table in database, and wherein data head table is stored with the data header information of data and the corresponding relation of data sequence number；When the data header information got is present in data head table, the data sequence number corresponding with data header information is obtained in data head table；Data sequence number is inserted into data message table together with the corresponding performance data information of the data header information with getting, be stored with the data sequence number of every data and the corresponding relation of performance data information in the data message table.Thus, the efficiency in insertion of data into data storehouse is improved, and the scene inserted for various databases is respectively provided with good versatility.
Description
Technical field
This disclosure relates to database technical field, more particularly to a kind of method and system that data are inserted to database.
Background technology
With the development of industry 4.0, industrial control network has a networking, IPization, intelligentized trend, and this also brings simultaneously
Requirement to industrial control network security, so increasing network audit product is applied to industrial control network.
Because industrial control network has the requirement of hard real-time, so inserting the requirement of efficiency very to database in network audit product
It is high.
In presently relevant technology, Configuration Values of the main reliable database of technology of raising database insertion data efficiency itself
Optimization and SQL (Structured Query Language, SQL) statement level optimize both schemes, and this two
The scheme of kind has respective usage scenario respectively, there is also very big using restricted.
Specifically, the implementation on the first scheme, various business or PostgreSQL database software all provide a series of excellent
Change config option, corresponding optimization can be done for different application scenarios, and specifically distribute rationally according to database software
Difference and there may be difference.Because the configuration of various database software has differences, cause different database software it
Between transplantability it is very poor；The optimization means of time are changed in addition, this scheme is typically resource, can be consumed in more systems
Deposit, it is possible to causing the overall operational efficiency reduction of system.
On the implementation of second scheme, because nearly all common business and PostgreSQL database software are all supported at present
Using SQL statement come operating database, such as inquire about, insert, delete and edit etc..Inserted for certain specific big data quantity
Enter scene, table structure can be optimized according to specific business characteristic, optimize specific SQL statement and realize, so as to reach
Improve the purpose that data insert efficiency.Although, table structure and SQL statement for be between different database software can be with
Transplanting；But, this table structure and SQL statement for specific transactions scene optimizes, for other business scenarios just
No longer applicable, this scheme is also poor for the versatility of different business scenarios.
The content of the invention
To overcome problem present in correlation technique, the disclosure provides a kind of method to database insertion data and is
System.
Treated according to the first aspect of the embodiment of the present disclosure there is provided a kind of method that data are inserted to database, including acquisition
Insert the data header information and performance data information of data；Judge that the data header information got whether there is in the number
Closed according in the data head table in storehouse, wherein the data head table is stored with, the data header information of data is corresponding with data sequence number
System；When the data header information got is present in the data head table, in the data head table obtain with it is described
The corresponding data sequence number of data header information；The data sequence number is corresponding together with the data header information with getting
The performance data information is inserted into the data message table, the data sequence number for every data that is stored with the data message table with
The corresponding relation of performance data information.
According to the second aspect of the embodiment of the present disclosure there is provided a kind of system that data are inserted to database, including first obtain
Unit is taken, the data header information and performance data information of data are inserted into for obtaining；Judging unit, for judging what is got
The data header information whether there is in the data head table in the database, every number wherein the data head table is stored with
According to data header information and data sequence number corresponding relation；Second acquisition unit, for when the data header information got
When being present in the data head table, the data sequence number corresponding with the data header information is obtained in the data head table；
Unit is inserted, for the data sequence number to be believed together with the corresponding performance data of the data header information with getting
Breath is inserted into the data message table, data sequence number and the performance data information of every data that is stored with the data message table
Corresponding relation.
The technical scheme provided by this disclosed embodiment can include the following benefits：
Based on embodiment of the present invention technical scheme so that data head table and data message table in database store number jointly
According to that is, data head table is used for the data header information of data storage and the function code data letter of data message table data storage
Breath, and complete by data sequence number the association of the data header information and function code data message of same data；Thus on the one hand, originally
The technical scheme for the database insertion data that inventive embodiments are provided is the optimization for database structure, and for database
Application scenarios be not particularly limited, application it is upper have preferable versatility；On the other hand, the implementation pair of the embodiment of the present invention
The consumption of system resource and occupancy are less, and the efficiency of database can be lifted on the whole；Another further aspect, each toward database
During middle insertion data, all only need to inquire about data head table, need not inquire about in database and be deposited compared to traditional database
The function code data information portion of storage, improves the efficiency in insertion of data into data storehouse.
It should be appreciated that the general description of the above and detailed description hereinafter are only exemplary and explanatory, not
The disclosure can be limited.
Brief description of the drawings
Accompanying drawing herein is merged in specification and constitutes the part of this specification, shows the implementation for meeting the present invention
Example, and for explaining principle of the invention together with specification.
Fig. 1 is the structural representation of the database using the embodiment of the present invention；
Fig. 2 is a kind of stream of method that data are inserted into database shown in Fig. 1 according to an exemplary embodiment
Cheng Tu；
Fig. 3 is a kind of method that data are inserted into database shown in Fig. 1 according to another exemplary embodiment
Flow chart；
Fig. 4 is that a kind of of step 202 in a kind of method that data are inserted to database shown in Fig. 3 specifically performs flow
Figure；
Fig. 5 is a kind of structured flowchart of system that data are inserted to database according to an exemplary embodiment；
Fig. 6 is a kind of structured flowchart of system that data are inserted to database according to an exemplary embodiment；
Fig. 7 is a kind of structured flowchart of system that data are inserted to database according to another exemplary embodiment；
Fig. 8 is a kind of structured flowchart of system that data are inserted to database according to another exemplary embodiment.
Reference
The judging unit of 301 first acquisition unit 302
303 second acquisition units 304 insert unit
The buffer queue update module of 3021 buffer queue correction verification module 3022
30211 HASH matching components
Embodiment
Here exemplary embodiment will be illustrated in detail, its example is illustrated in the accompanying drawings.Following description is related to
During accompanying drawing, unless otherwise indicated, the same numbers in different accompanying drawings represent same or analogous key element.Following exemplary embodiment
Described in embodiment do not represent and the consistent all embodiments of the present invention.On the contrary, they be only with it is such as appended
The example of the consistent apparatus and method of some aspects be described in detail in claims, the present invention.
Embodiment one
The structural representation of the database using the embodiment of the present invention is illustrated that referring to Fig. 1, as shown in figure 1, in data
Include data head table and data message table in storehouse, the data storage by way of data head table is combined with data message table.Its
In, record has data sequence number 1,2~data sequence number of data sequence number n data sequence number row and and data sequence number in data head table
The corresponding data header information row of row；Then there are data sequence number row and function code data message row in data message table.Data
Data link is set up by data sequence number row between information table and data head table, for example：The corresponding data header information of data sequence number 1
With the composition data 1 of function code information 1 corresponding to data sequence number 1.
In one embodiment, be stored with n datas in the database, each data in the n datas all by
It is divided into two parts, the two parts are stored into data head table and data message table, and passes through the data sequence in two tables
Number row are associated.It is possible thereby to intuitively find out very much, using the data stored in the database of the embodiment of the present invention according to
Information content difference is stored respectively, and is associated based on data sequence number.
The embodiment of the present invention can be applied in any product, device and the server of database is configured with, accordingly to change
The database insertion efficiency of kind each product, device and server.The embodiment of the present invention be particularly suitable for use in audit Platform Server or
In device, efficiency is inserted thus, it is possible to the data for being effectively improved audit platform.
It is illustrated that one kind according to an exemplary embodiment inserts data into database shown in Fig. 1 referring to Fig. 2
Method flow chart, as shown in Fig. 2 the present embodiment implement method be suitable to data are inserted into the database shown in Fig. 1, i.e.,
The corresponding relation of the data header information for having data storage and data sequence number is comprised at least in the database for being inserted into data
The data message table of data head table and the data sequence number of data storage and the corresponding relation of performance data information.More specifically, should
The method for inserting data to database includes：
Step 101：Obtain the data header information and performance data information for being inserted into data；
The data header information can include the source in source IP, purpose IP and the protocol name, and same data flow of data
IP, purpose IP and protocol name all same.
Step 102：Judge the data header information got with the presence or absence of in the data head table in database；
If the judged result of step 102 is yes, i.e., if the data header information got is present in the data head table
Redirect execution step 1031；And if the judged result of step 102 is no, if the data header information got is not present in
Step 1032 is then jumped in the data head table；
Step 1031：The data sequence number corresponding with data header information is obtained in data head table；
Step 1032：The data header information got is inserted into data head table, and determine in data head table with obtain with
The corresponding data sequence number of the data header information that gets；
Step 104：Data sequence number is inserted into together with the corresponding performance data information of the data header information with getting
Data message table.
By obtaining the data sequence number of the data header information corresponding to data in data head table, and pass through the data sequence number
The data update the data information table, and the performance data information stored in the data message table can be exactly the specific letter of data
Cease, for example can be various functions code, mapped between the data head table and data message table by data sequence number；Past every time
When inserting data in database, all only need to inquire about data head table, compared to traditional database, it is not necessary to inquire about database
In function code part, and simply improvement to database structure, and improve the efficiency of data insertion.In addition, for number
According to the inquiry of data stored in storehouse can be based only upon data head table inquiry just can with the positioning of complete paired data sequence number,
It is then based on data sequence number and completes calling to function code data message specified in data message table, data can be lifted
The efficiency data query in storehouse.
Embodiment two
Embodiment two can be regarded as the further optimization of embodiment one, because in embodiment one, being inserted to database
The embodiment of data, although the data head matrix section for only needing to inquire about in database just can with the insertion of complete paired data
The efficiency in insertion of data into data storehouse is improved to a certain extent, but still with the place being modified, in this open phase of embodiment two
For the more excellent embodiment of embodiment one.
The scheme by buffer queue, that is, the step 102 are introduced in step 102 in method shown in embodiment one
Execution can include：
By the buffer queues of the multiple data header informations being cached with the data head table to the data header information that gets
Matching verification is performed, to determine that the data header information got whether there is in the data head table.
When the data header information got is present in buffer queue, it is determined that the data header information got is present in this
In data head table；When the data header information got is not present in buffer queue, it is determined that the data header information got is not
It is present in the data head table.
The present embodiment can be applied in any product, device and the server of database is configured with, and improve each with corresponding
The database insertion efficiency of product, device and server.The embodiment of the present invention is particularly suitable for use in audit Platform Server or device
In, insert efficiency thus, it is possible to the data for being effectively improved audit platform.
Thus, audit Platform Server or device are direct to being received by the caching sequence comprising multiple data header informations
The data head for obtaining data performs verification matching so that can just be completed in the caching stage of data to the data being inserted into data
Positioned in storehouse, accelerate the efficiency in insertion of data into data storehouse.
If the data head being inserted into can be queried in buffer queue, then what the data being inserted into were searched
Efficiency is highest；But the length of buffer queue is all finite length.Make it that the data being inserted into completely can be slow
To deposit find in queue and belong to a kind of ideal state, if in order to ensure that the data being inserted into are believed by the data head in the buffer queue
Breath hit probability is improved, then, correspondingly, the efficiency in insertion of data into data storehouse should also be as being higher.In view of this, it is of the invention
Embodiment is disclosed by a kind of data temperature algorithm to update the scheme of the buffer queue, thus ensures the buffer queue
High temperature, improves and is inserted into the probability that the buffered queue of data is hit.More specifically, being illustrated that with reference to Fig. 3 of the invention another
The flow chart of the method that data are inserted to database of one example, the method for inserting data to database is specifically included：
Step 201：Obtain the data header information and performance data information for being inserted into data；
Step 202：By the buffer queues of the multiple data header informations being cached with data head table to the data that get
Header performs matching verification, to determine that the data header information got whether there is in data head table；
The process of matching verification is exactly to judge to obtain with the presence or absence of matching in multiple data header informations in the buffer queue
The data header information got, when result of determination is matching and shows that the data header information got is present in data head table
When, then step 206 is performed, and when result of determination is mismatch and shows that the data header information got is not present in data head
When in table, then step 203 is performed.
Step 203：Judge buffer queue whether saturation；
When the judged result in step 203 for buffer queue is saturation, then step 2041 is performed, and when the judgement
As a result it is unsaturated for buffer queue, then perform step 2042；
Step 2041：It is determined that the minimum data head of data hot value ranking in all data header informations under buffer queue
Information, and pass through the minimum data header information of the data header information replacement data hot value ranking got；
Step 2042：The data header information got is inserted into buffer queue；
Step 205：The corresponding data sequence number of the data header information of fetched data is determined in the data head table；
Step 206：Data sequence number is inserted into together with the corresponding performance data information of the data header information with getting
Data message table.
, for example can be according to each data in buffer queue on the embodiment of the determination of minimum dsc data in step 2041
The timestamp that the number of times and last time that header is matched are matched determines the number of each data header information in the buffer queue
According to hot value；According to the sequence of the data hot value of each data header information in the buffer queue, to determine in the buffer queue
The minimum data header information of data hot value ranking in all data header informations.As an example, can be for last time quilt
Each set weight determines data temperature to the number of times that the timestamp and data header information matched somebody with somebody are matched,
It is understood that as illustrated in embodiment one, the data header information can the source IP comprising data,
Purpose IP and protocol name, accordingly, the minimum data header information of the data hot value ranking may refer to the minimum source of temperature
IP, purpose IP and protocol name, thus replace the data header information newly inserted information least conventional in buffer queue, thus real
The efficient utilization rate of buffer queue resource is showed so that the buffer queue after renewal has higher relative to data are inserted into
With hit rate, the data insertion efficiency of database is further increased.
The embodiment of the present invention can not limited as being illustrated in embodiment one for the application scenarios of database.
But need herein it is emphasized that when the embodiment of the present invention is practiced in environment of industrial network, can produce more
The significant technique effect illustrated above；Trace it to its cause and be：Found by the analysis to actual industry control network message, absolutely
Data flow in most of industry control networks is all TCP (the Transmission Control Protocol, transmission control of long connection
Agreement processed) flow, the data head in data is substantially very fixed to maintain change in a smaller scope, and thus the present invention is real
The buffer queue based on data temperature algorithm that example provided is applied to solve data sequence number and gauge outfit Field Inquiry in data head table
Efficiency.
Embodiment three
Embodiment three can be regarded as preferred embodiment of the embodiment two under certain application scenarios, because of embodiment two
In a part of content elaborate that can update buffer queue by temperature algorithm be inserted into the raising hit rates of data, to improve
The data of database insert the scheme of efficiency.But, buffer queue is generally based on fifo queue (FIFO, First
Input First Output) mode realize that but being updated based on data temperature for buffer queue is coordinated in embodiment two
FIFO mode, which updates, determines that buffer queue just also has the place having much room for improvement, because FIFO mode most likely results in number
It is updated according to the earlier data header information of hot value ranking.In view of this, the embodiment of the present invention is disclosed another
Embodiment three, in the way of replacing the first in first out (FIFO) of buffer queue generally.
More specifically, providing the implementation of the further optimization for the step 202 being directed in embodiment two in the present embodiment
Mode, the implementation of the step 202 can be determined based on HASH.It is illustrated that referring to Fig. 4 shown in Fig. 3 and inserts number to database
According to method in step 202 specific execution flow chart, can specifically include following sub-step：
Step 2021：It is determined that corresponding to the caching HASH value groups of buffer queue, caching HASH values group is included in buffer queue
Each HASH values corresponding to each data header information；
Step 2022：It is determined that the first HASH values corresponding to the data header information got；
Step 2023：Matching verification is performed to the first HASH values by caching HASH values group, to determine the data got
Header whether there is in buffer queue.
The present embodiment can be applied in any product, device and the server of database is configured with, and improve each with corresponding
The database insertion efficiency of product, device and server.The embodiment of the present invention is particularly suitable for use in audit Platform Server or device
In, insert efficiency thus, it is possible to the data for being effectively improved audit platform.
Thus, the embodiment of the present invention employs HASH algorithms to realize buffer queue so that between data head and HASH values
It is the relation mapped one by one；Thus, on the one hand the lookup based on HASH values can avoid whole delay for the implementation of the embodiment of the present invention
The query process of necessary data head table, can greatly be improved when depositing the inquiry of queue, and eliminating insertion data every time
Data insert efficiency；On the other hand, for the type of database software, configuration parameter and sql like language all it is any it is special will
Ask, so the application of the embodiment of the present invention is with very high versatility and transplantability.
In the foregoing embodiments, the determination of each HASH values corresponding to each data header information in buffer queue, can
To be realized by following mode：
Source IP, protocol type, purpose IP e.g., including in data header information perform HASH algorithms.
α=((SIP ＜ ＜ 32) ^ (Proto ＜ ＜ 56)) | (DIP^ (Proto ＜ ＜ 24))
Wherein, α refers to the HASH values of 64, and SIP refers to the source IP of data, and Proto refers to the protocol type of data, DIP
Refer to the purpose IP of data.
The embodiment of the present invention can not be limited the application scenarios of database as being illustrated in embodiment one.But
Being need to be herein it is emphasized that when the embodiment of the present invention is practiced in environment of industrial network, can produce and more show
The technique effect illustrated above write；As embodiment two is somebody's turn to do, the data flow in industry control network is generally long company
Connect, and the division of subnet typically will not be very complicated in industry control network so that it can ensure that HASH values are sent out with reference to the HASH algorithms
The probability very little of raw collision.
It is understood that specifically being described in various embodiments above merely exemplary so that the public becomes apparent from geography
Solve the spirit of the present invention, limiting the scope of the invention that it goes without doing.For example, being realized with HASH algorithms in embodiment three
Buffer queue is not essential, when for the fewer network scenarios of data flow, the method that directly can be searched using traversal.Cause
For when data flow is fewer, traversal number of times also will not be very big, can ignore because of the inquiry of buffer queue to data base querying
The influence of efficiency.
Example IV
Fig. 5 is a kind of structured flowchart of system that data are inserted to database according to an exemplary embodiment.Such as
Shown in Fig. 5, the system includes first acquisition unit 301, judging unit 302, second acquisition unit 303 and insertion unit 304.
Wherein, judging unit 302, the data header information for judging to get whether there is in the data head table in database, its
Middle data head table is stored with the data header information of data and the corresponding relation of data sequence number；Second acquisition unit 303, is obtained for working as
When the data header information got is present in data head table, the data sequence corresponding with data header information is obtained in data head table
Number；Unit 304 is inserted, for data sequence number to be inserted together with the corresponding performance data information of the data header information with getting
To data message table, be stored with the data sequence number of every data and the corresponding relation of performance data information in the data message table.
The present embodiment can be applied in any product, device and the server of database is configured with, and improve each with corresponding
The database insertion efficiency of product, device and server.The embodiment of the present invention is particularly suitable for use in audit Platform Server or device
In, insert efficiency thus, it is possible to the data for being effectively improved audit platform.
In the implementation process of the present embodiment system, by obtaining the data header information corresponding to data in data head table
Data sequence number, and by the data sequence number, the data update the data information table, the function number stored in the data message table
It is believed that breath can be exactly the specifying information of data, for example can be various functions code, the data head table and data message table it
Between mapped by data sequence number；When every time toward insertion data in database, all only need to inquire about data head table, compared to biography
The database of system, it is not necessary to the function code part inquired about in database, and the simply improvement to database structure, and improve
The efficiency of data insertion.In addition, the inquiry for data stored in database can be based only upon the inquiry of data head table
Just it can be then based on data sequence number with the positioning of complete paired data sequence number and complete to function code specified in data message table
Calling for data message, can lift the efficiency data query of database.
Embodiment five
Embodiment five can regard the further optimization to example IV as, because being inserted in example IV to database
Enter data technical scheme although only need to inquire about database in data head table, data sequence number part just can be with complete paired data
Call, the efficiency in insertion of data into data storehouse can be improved to a certain extent, but still with the place being modified, implement herein
The open more excellent embodiment relative to example IV of example five.
The structured flowchart of the system that data are inserted to database of one embodiment of the invention, such as Fig. 6 are illustrated that referring to Fig. 6
Shown, the system for inserting data to database includes first acquisition unit 301, judging unit 302, second acquisition unit 303
And insertion unit 304, wherein include buffer queue correction verification module 3021 in judging unit 302,.On second acquisition unit
303 and insert unit 304 particularly relevant technical characteristic and technique effect be referred to embodiment illustrated in fig. 5, same section
Not in this to go forth.It should be pointed out that the buffer queue correction verification module 3021, is configured as by being cached with the data head
The buffer queue of multiple data header informations in table performs matching verification to the data header information got, to determine what is got
Data header information whether there is in the data head table.More specifically, the buffer queue correction verification module 3021, is configured as obtaining
The data header information got, which is present in when in buffer queue, then determines that the data header information got is present in the data head table,
Then determine that the data header information got is not present in the number when the data header information got is not present in buffer queue
According in head table.
The present embodiment can be applied in any product, device and the server of database is configured with, and improve each with corresponding
The database insertion efficiency of product, device and server.The embodiment of the present invention is particularly suitable for use in audit Platform Server or device
In, insert efficiency thus, it is possible to the data for being effectively improved audit platform.
Thus, the data head for directly obtaining data to receiving by the caching sequence comprising multiple data header informations is performed
Verification matching so that just can complete to position fetched data in database in the caching stage of data, accelerate data
Insert the efficiency of database.
Embodiment six
Embodiment six can be seen as the further public affairs on the basis of embodiment five to technical solution of the present invention
Open, if the data head being inserted into can be found in buffer queue, then search efficiency is highest；But buffer queue
Length is all finite length.Make it that the data being inserted into can be found in buffer queue completely and belong to a kind of preferable shape
Condition, but it is higher if ensure that the data being inserted into hit probability by the data header information in the buffer queue, accordingly
The efficiency in ground insertion of data into data storehouse should also be as being higher.In view of this, the embodiment of the present invention provides a kind of data heat herein
Degree algorithm updates the buffer queue to ensure the high temperature of the buffer queue.
The structural representation of the system that data are inserted to database of further embodiment of this invention is illustrated that referring to Fig. 7,
The embodiment is additionally arranged included in judging unit 302 compared with the embodiment shown in Fig. 6 and is connected to buffer queue correction verification module
3021 buffer queue update module 3022.The buffer queue update module 3022 is configured as it is determined that the data head got
After information is not present in the data head table, buffer queue whether saturation is judged；When buffer queue saturation, really
It is scheduled on the minimum data header information of data hot value ranking in all data header informations under buffer queue, and by getting
The minimum data header information of data header information replacement data hot value ranking, and when buffer queue does not have saturation, to caching
The data header information got is inserted in queue.
Mould is determined it is highly preferred that also including in the embodiment and being connected to the minimum dsc data of the buffer queue update module
Block (not shown), be configured to according to each data header information is matched in buffer queue number of times and last time be matched when
Between stamp determine the buffer queue in each data header information data hot value；According to the number of each data header information in the buffer queue
According to the sequence of hot value, to determine the minimum data head of data hot value ranking in all data header informations of the buffer queue
Information.As an example, can be the number of times being matched for the timestamp and data header information being matched for the last time each institute
The weight of setting determines data temperature.
It is understood that the data header information can include source IP, purpose IP and the protocol name of data, accordingly should
The minimum data header information of data hot value ranking may refer to the minimum source IP of temperature, purpose IP and protocol name, thus will be new
The data header information of insertion replaces information least conventional in buffer queue, thus achieves the effective of buffer queue resource and most
That changes greatly passes through so that the buffer queue after renewal is further carried relative to data are inserted into higher match hit rate
The high data insertion efficiency of database.
The embodiment of the present invention can be applied in any product, device and the server of database is configured with, accordingly to change
The database insertion efficiency of kind each product, device and server.The embodiment of the present invention be particularly suitable for use in audit Platform Server or
In device, efficiency is inserted thus, it is possible to the data for being effectively improved audit platform.
The embodiment of the present invention can not limited as being illustrated in example IV for the application scenarios of database.
But need herein it is emphasized that when the embodiment of the present invention is practiced in environment of industrial network, can produce more
The significant technique effect illustrated above；Trace it to its cause and be：Found by the analysis to actual industry control network message, absolutely
Data flow in most of industry control networks, which is all that data head in the TCP flow of long connection, data is substantially very fixed, maintains one
Change in individual smaller scope, the buffer queue based on data temperature algorithm that thus embodiment of the present invention is provided is solved
The efficiency of data sequence number and gauge outfit Field Inquiry in data head table.
Embodiment seven
Embodiment seven can be seen as the further disclosure for embodiment five or six pairs of technical solution of the present invention, real
Preferred embodiment of the embodiment five or six under certain application scenarios can be regarded as by applying example seven, because in embodiment five or six
A part be depicted and buffer queue can be updated by temperature algorithm and be inserted into the raising hit rates of data, to improve data
The data insertion efficiency in storehouse.But, what the mode that buffer queue is generally based on first in first out (FIFO) was realized, but embodiment
The mode for updating cooperation FIFO based on data temperature of buffer queue, which updates, in five or six determines that just also presence needs buffer queue
Improved place, because FIFO mode most likely results in data hot value ranking, earlier data header information is updated
Fall.In view of this, the embodiment of the present invention is disclosed the preferred embodiment seven, to replace buffer queue generally
First in first out (FIFO) mode.
The block diagram that data system is inserted to database of yet another embodiment of the invention is illustrated that referring to Fig. 8, by Fig. 8 institutes
The system for inserting data to database shown is additionally arranged compared with the system to database insertion data shown in Fig. 6 in the Fig. 8
HASH matching components 30211 in buffer queue module 3021, are configured to determine that the caching corresponding to buffer queue
HASH value groups, caching HASH values group includes each HASH values in buffer queue corresponding to each data header information；It is determined that get
The first HASH values corresponding to data header information；Matching verification is performed to the first HASH values by caching HASH values group, to determine
The data header information got whether there is in buffer queue.
The embodiment of the present invention can be applied in any product, device and the server of database is configured with, accordingly to change
The database insertion efficiency of kind each product, device and server.The embodiment of the present invention be particularly suitable for use in audit Platform Server or
In device, efficiency is inserted thus, it is possible to the data for being effectively improved audit platform.
Thus, the embodiment of the present invention employs HASH algorithms to realize buffer queue so that between data head and HASH values
It is the relation mapped one by one；Thus, on the one hand the lookup based on HASH values can avoid whole delay for the implementation of the embodiment of the present invention
The query process of necessary data head table, can greatly be improved when depositing the inquiry of queue, and eliminating insertion data every time
Data insert efficiency；On the other hand, for the type of database software, configuration parameter and sql like language all it is any it is special will
Ask, so the application of the embodiment of the present invention is with very high versatility and transplantability.
More specifically, the embodiment of the present invention continues the determination of open HASH values herein, it can specifically include according to data head
Source IP, protocol type, purpose IP in information perform HASH algorithms.It is specific as follows on the HASH algorithms：α=((SIP ＜ ＜
32) ^ (Proto ＜ ＜ 56)) | (DIP^ (Proto ＜ ＜ 24))
Wherein, α refers to the HASH values of 64, and SIP refers to the source IP of data, and Proto refers to the protocol type of data, DIP
Refer to the purpose IP of data.
The embodiment of the present invention can not be limited the application scenarios of database as being illustrated in embodiment one.But
Being need to be herein it is emphasized that when the embodiment of the present invention is practiced in environment of industrial network, can produce and more show
The technique effect illustrated above write；As embodiment two is somebody's turn to do, the data flow in industry control network is generally long company
Connect, and the division of subnet typically will not be very complicated in industry control network so that it can ensure that HASH values are sent out with reference to the HASH algorithms
The probability very little of raw collision.
It is understood that specifically being described in various embodiments above merely exemplary so that the public becomes apparent from geography
Solve the spirit of the present invention, limiting the scope of the invention that it goes without doing.For example, being realized with HASH algorithms in embodiment three
Buffer queue is not essential, when for the fewer network scenarios of data flow, the method that directly can be searched using traversal.Cause
For when data flow is fewer, traversal number of times also will not be very big, can ignore because of the inquiry of buffer queue to data base querying
The influence of efficiency.
On one in the concrete mode of the execution operation of the modules in the system of above-described embodiment, unit and component
Part is described in detail in the embodiment about this method, no longer will be gone to live in the household of one's in-laws on getting married in detail herein for the repeating part
State.
Those skilled in the art will readily occur to its of the present invention after considering specification and putting into practice invention disclosed herein
Its embodiment.The application be intended to the present invention any modification, purposes or adaptations, these modifications, purposes or
Person's adaptations follow the general principle of the present invention and including the undocumented common knowledge in the art of the disclosure
Or conventional techniques.Description and embodiments are considered only as exemplary, and true scope and spirit of the invention are by following
Claim is pointed out.
It should be appreciated that the invention is not limited in the precision architecture for being described above and being shown in the drawings, and
And various modifications and changes can be being carried out without departing from the scope.The scope of the present invention is only limited by appended claim.
Claims (10)
1. a kind of method that data are inserted to database, it is characterised in that including：
Obtain the data header information and performance data information for being inserted into data；
The data header information got is judged with the presence or absence of in the data head table in the database, wherein the data
Head table is stored with the data header information of every data and the corresponding relation of data sequence number；
When the data header information got is present in the data head table, in the data head table obtain with it is described
The corresponding data sequence number of data header information；
The data sequence number is inserted into together with the corresponding performance data information of the data header information with getting
Be stored with the data message table, the data message table every data data sequence number it is corresponding with performance data information close
System.
2. according to the method described in claim 1, it is characterised in that whether the data header information for judging to get is deposited
In being the data head table in the database, wherein the data head table, which is stored with, the data header information and number of every data
Include according to the corresponding relation of sequence number：
The data head got is believed by the buffer queue for the multiple data header informations being cached with the data head table
Breath performs matching verification, to determine that the data header information got whether there is in the data head table.
3. method according to claim 2, it is characterised in that it is determined that the data header information got is not present in
After in the data head table, also include：Update the buffer queue.
4. method according to claim 3, it is characterised in that the renewal buffer queue includes：
Judge the buffer queue whether saturation；
When the buffer queue saturation, it is determined that data hot value in all data header informations under the buffer queue
The minimum data header information of ranking；
The minimum data header information of the data hot value ranking is replaced by the data header information got, and
When the buffer queue does not have saturation, the data header information got is inserted into the buffer queue.
5. method according to claim 4, it is characterised in that all data heads of the determination under the buffer queue
The minimum data header information of hot value ranking includes in information：
According to being determined the timestamp that each data header information is matched in buffer queue number of times and last time are matched
The data hot value of each data header information in buffer queue；
According to the sequence of the data hot value of each data header information in the buffer queue, to determine the institute in the buffer queue
There is the minimum data header information of data hot value ranking in data header information.
6. method according to claim 2, it is characterised in that described multiple in the data message table by being cached with
The buffer queue of data header information judges that the data header information got whether there is to be included in the buffer queue：
It is determined that corresponding to the caching HASH value groups of the buffer queue, the caching HASH values group includes each in the buffer queue
Each HASH values corresponding to data header information；
It is determined that the first HASH values corresponding to the data header information got；
Matching verification is performed to the first HASH values by the caching HASH values group, to determine the data head got
Information whether there is in the buffer queue.
7. according to the method described in claim 1, it is characterised in that the data header information for judging to get whether
It is present in the data head table in the database, the data header information of data and data sequence wherein the data head table is stored with
Number corresponding relation after also include：
When the data header information got is not present in the data head table, by the data header information got
Insert the data head table；
The corresponding data sequence number of the data header information to obtain with get is determined in the data head table.
8. a kind of device that data are inserted to database, it is characterised in that including：
First acquisition unit, the data header information and performance data information of data are inserted into for obtaining；
Judging unit, the data header information for judging to get whether there is the data head table in the database
In, the data header information of every data and the corresponding relation of data sequence number wherein the data head table is stored with；
Second acquisition unit, for when the data header information got is present in the data head table, in the number
According to the data sequence number corresponding with the data header information of acquisition in head table；
Insert unit, for by the data sequence number together with the corresponding function number of the data header information with getting
It is believed that breath is inserted into the data message table, data sequence number and the performance data of every data that is stored with the data message table are believed
The corresponding relation of breath.
9. device according to claim 8, it is characterised in that the judging unit includes：
Buffer queue correction verification module, the buffer queue pair for multiple data header informations by being cached with the data head table
The data header information got performs matching verification, to determine that the data header information got whether there is in described
In data head table.
10. device according to claim 9, it is characterised in that described device also includes：
Buffer queue update module, is connected to the buffer queue correction verification module, for it is determined that the data head got
After information is not present in the data head table, the buffer queue is updated.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710124487.8A CN106960016A (en) | 2017-03-03 | 2017-03-03 | The method and system of data are inserted to database |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710124487.8A CN106960016A (en) | 2017-03-03 | 2017-03-03 | The method and system of data are inserted to database |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106960016A true CN106960016A (en) | 2017-07-18 |
Family
ID=59470072
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710124487.8A Pending CN106960016A (en) | 2017-03-03 | 2017-03-03 | The method and system of data are inserted to database |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106960016A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113505141A (en) * | 2021-07-05 | 2021-10-15 | 浪潮云信息技术股份公司 | Automated method and system for realizing database SQL optimized execution based on artificial intelligence |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102084360A (en) * | 2008-04-06 | 2011-06-01 | 弗森-艾奥公司 | Apparatus, system, and method for validating that a correct data segment is read from a data storage device |
| CN103336849A (en) * | 2013-07-24 | 2013-10-02 | 昆明理工大学 | Method and device for increasing retrieval speed in database retrieval system |
| CN103647794A (en) * | 2013-12-31 | 2014-03-19 | 北京启明星辰信息安全技术有限公司 | Data correlation method based on J2EE platform and correlation plug-in |
| CN104516894A (en) * | 2013-09-27 | 2015-04-15 | 国际商业机器公司 | Method and device for managing time series database |
| CN104932956A (en) * | 2015-06-19 | 2015-09-23 | 华南理工大学 | Big-data-oriented cloud disaster tolerant backup method |
| CN105138675A (en) * | 2015-09-08 | 2015-12-09 | 上海上讯信息技术股份有限公司 | Database auditing method and device |
-
2017
- 2017-03-03 CN CN201710124487.8A patent/CN106960016A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102084360A (en) * | 2008-04-06 | 2011-06-01 | 弗森-艾奥公司 | Apparatus, system, and method for validating that a correct data segment is read from a data storage device |
| CN103336849A (en) * | 2013-07-24 | 2013-10-02 | 昆明理工大学 | Method and device for increasing retrieval speed in database retrieval system |
| CN104516894A (en) * | 2013-09-27 | 2015-04-15 | 国际商业机器公司 | Method and device for managing time series database |
| CN103647794A (en) * | 2013-12-31 | 2014-03-19 | 北京启明星辰信息安全技术有限公司 | Data correlation method based on J2EE platform and correlation plug-in |
| CN104932956A (en) * | 2015-06-19 | 2015-09-23 | 华南理工大学 | Big-data-oriented cloud disaster tolerant backup method |
| CN105138675A (en) * | 2015-09-08 | 2015-12-09 | 上海上讯信息技术股份有限公司 | Database auditing method and device |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113505141A (en) * | 2021-07-05 | 2021-10-15 | 浪潮云信息技术股份公司 | Automated method and system for realizing database SQL optimized execution based on artificial intelligence |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP5623585B2 (en) | Method, apparatus and system for use in flow statistics | |
| CN104168341B (en) | The localization method and CDN dispatching methods and device of IP address | |
| CN103927331B (en) | Data querying method, data querying device and data querying system | |
| CN104301242B (en) | Communication stream control system, communication method of flow control and communication stream processing routine | |
| US20080056291A1 (en) | Methods and system for dynamic reallocation of data processing resources for efficient processing of sensor data in a distributed network | |
| WO2010098034A1 (en) | Distributed database management system and distributed database management method | |
| CN108173982A (en) | The NAT processing method and processing devices of straddle message | |
| CN108197237A (en) | Visualization data, which collect, shows system | |
| CN106790556A (en) | A kind of NAT conversation managing methods based on distributed system | |
| CN106815258A (en) | A kind of date storage method and coordinator node | |
| CN104239417A (en) | Dynamic adjustment method and dynamic adjustment device after data fragmentation in distributed database | |
| CN109145189A (en) | A kind of data deposit and search system and its method based on block chain | |
| CN112835977B (en) | Database management method and system based on block chain | |
| US6760813B1 (en) | Devices, softwares and methods for preemptively updating expiring entries in a cache memory to anticipate future inquiries | |
| CN107040612A (en) | A kind of information recommendation method and device | |
| CN108989399A (en) | A kind of cluster common data querying method, system, device and readable storage medium storing program for executing | |
| CN111723126B (en) | Block chain-based cold and hot time sequence data classification processing method and storage system | |
| US20170359433A1 (en) | Scalable management of composite data collected with varied identifiers | |
| CN107947977A (en) | A kind of collocation method of interchanger, device, electronic equipment and storage medium | |
| CN112417517A (en) | Data standard management method based on block chain | |
| CN109189902A (en) | The method and device of automatic-answering back device consulting | |
| CN106960016A (en) | The method and system of data are inserted to database | |
| US20120254224A1 (en) | Collaborative Search Network | |
| CN105610881B (en) | A kind of distributed caching range query method, apparatus and system | |
| CN101685460A (en) | Database device, method and system for processing shared data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |