CN103559217A - Heterogeneous database oriented massive multicast data storage implementation method - Google Patents
Heterogeneous database oriented massive multicast data storage implementation method Download PDFInfo
- Publication number
- CN103559217A CN103559217A CN201310488096.6A CN201310488096A CN103559217A CN 103559217 A CN103559217 A CN 103559217A CN 201310488096 A CN201310488096 A CN 201310488096A CN 103559217 A CN103559217 A CN 103559217A
- Authority
- CN
- China
- Prior art keywords
- data file
- model
- warehouse
- cache data
- message
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Security & Cryptography (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention provides a heterogeneous database oriented massive multicast data storage implementation method. The heterogeneous database oriented massive multicast data storage implementation method comprises receiving multicast massages according to a certain limit cycle and quantity restrictions; storing the required messages into a first level cache data file and triggering the analysis on the first level cache data file through signals; storing the massages into corresponding models of second level cache data files and establishing second level cache data file storage catalogues and related data files and control files corresponding to the models; performing synchronous triggering of storage operation on models of storage threads according to the signals, performing analysis on the second level cache data files to be stored and inputting the analyzed data into corresponding databases through calling of an SQL * Loader and storage control file data. According to the heterogeneous database oriented massive multicast data storage implementation method, the real-time stable receiving processing of massive multicast data can be achieved, the synchronous efficient storage of multi-objective databases can be achieved, the processing capacity on the massive multicast data is improved in practical application, the integrity of the data is ensured, and the efficiency of the application system is improved.
Description
Technical field
The invention belongs to computer data acquiring field, be specifically related to a kind of warehouse-in of the magnanimity multicast packet towards heterogeneous database implementation method.
Background technology
When large scale system polytypic batch concurrent testing, face the situations such as task is heavy, the many and a large amount of repeated tests of model, the magnanimity test data of test generation need to receive in real time and be saved in the database of each model, and that the processing of this mass data has is real-time, data volume is large, data layout is complicated and the large feature of warehouse-in pressure.
Conventionally the warehouse-in of multicast packet is pressed the difference of data buffer storage mode, has based on shared buffer with based on two kinds of warehouse-in modes of data file.Warehouse-in based on shared buffer, general " producer-consumer " model or its modification of adopting, " producer " deposits the multicast packet receiving in buffer memory, and " consumer " accesses shared drive, after taking-up Data Analysis, deposits database in.Warehouse-in based on data file, adopts the mode of multithreading asynchronous operation to complete message reception, resolves and warehouse-in, and receiving thread is monitored the network port, and receiving group message, writes data cached file by some cycles by message, generates raw data file; Resolve thread and resolve raw data file, generate warehouse-in document; Warehouse-in thread is used SQL*Loader technology will put document input database in storage.
In large scale system polytypic batch concurrent testing, database has a plurality of and isomery, and above two kinds of warehouse-in modes respectively have deficiency in this case:
The warehouse-in of the first based on shared buffer, the defect of existence is: cache size is difficult for definite, easily overflows, and particularly, when big data quantity, system is extremely unstable, cannot guarantee integrality and the security of data; The read-write of shared buffer is synchronous operation, and efficiency is low; Data loading efficiency efficiency when adopting SQL*Loader is high, at database, is a plurality of and isomery in the situation that, cannot guarantee to put in storage speed.
The warehouse-in mode of the second based on data file, although can guarantee receiving efficiency, stability and the integrality of data, adopt the logging data of SQL*Loader technology, at warehouse-in efficiency, also promote to some extent, but at parsing and the warehouse-in in the face of mass data, particularly during the warehouse-in of a plurality of heterogeneous databases, cannot guarantee high efficiency.
Summary of the invention
During for large scale system polytypic batch concurrent testing, the magnanimity multicast packet of generation needs the situation of a plurality of heterogeneous databases of typing, and the present invention proposes a kind of warehouse-in of the magnanimity multicast packet towards heterogeneous database implementation method.By two-level cache data file, multicast packet stored and classified, the mode of putting in storage by multithreading guarantees that parsing and the warehouse-in of the data of each model have exclusive warehouse-in thread to complete, when having improved data-handling efficiency, solved the problem of a plurality of heterogeneous database warehouse-ins.
A kind of warehouse-in of the magnanimity multicast packet towards heterogeneous database implementation method provided by the invention, comprises the following steps:
Step 1: start multicast message receiving thread, start receiving group message;
Step 2: first the multicast message receiving is carried out to integrality judgement, if imperfect, continue to receive; If complete, the type of message is tested, if not needed type, abandon this message, continue to receive new multicast message; If required message deposits message in current level cache data file to be written in; When the multicast message quantity that reaches receiving cycle or reception when the time of receiving group message reaches specified quantity, record has completed the one-level data file number of reception, close and the current level cache data file to be written of rename, generate new level cache data file to be written simultaneously;
Step 3: judge whether to need the level cache data file of parsing, if having, read the multicast message in level cache data file to be resolved, start to resolve in advance; Otherwise, continue to wait for;
Pre-method of resolving is: resolve the information source field in multicast message header, obtain the affiliated model of message, judge that information corresponding to this model is whether in model queue, if not in model queue, obtain the information of this model and add in model queue, and creating L2 cache data file storage directory corresponding to this model and associated data files and control documents, multicast message is write to L2 cache data file, invokes thread pond, starts this model warehouse-in thread simultaneously; If in model queue, directly multicast message is write in corresponding L2 cache data file;
After a level cache data file is parsed in advance, record has completed the level cache data file number of parsing, close and the current all L2 cache data files to be written of rename, more in new model queue, L2 cache data file number is preserved in corresponding the completing of each model, generate new L2 cache data file to be written simultaneously, when wait is resolved in advance, store data next time;
Step 4: each model warehouse-in thread judges that completing of corresponding model preserve L2 cache data file number and whether be greater than and put L2 cache data file number in storage, if not, the data that there is no new needs warehouse-in be described, continues wait; If, according to putting L2 cache data file number in storage, obtain next one L2 cache data file to be put in storage, resolve this L2 cache data file, the warehouse-in that data after resolving is saved to corresponding model is prepared in data file, after being parsed, calling data loading tool SQL*Loader and warehouse-in control documents are prepared the data typing correspondence database in data file by warehouse-in, simultaneously the corresponding L2 cache of warehouse-in data file number in new model queue more.
In described step 2, current level cache data file to be written is unique and have a Permanent File Name; Relevant to the level cache data file number that completes reception in completing the name of level cache data file of write state.
In described step 3, the information of described model comprise model number, corresponding database user name and password, save data table name, complete and preserve L2 cache data file number and put L2 cache data file number in storage.The associated data files that described model is corresponding and control documents comprise L2 cache data file, warehouse-in preparation data file, warehouse-in control documents and misdata log file.
In described step 3, according to the level cache data file number that completes parsing, obtain next level cache data file to be resolved.When starting model warehouse-in thread, using the information of this model as parameter, pass to warehouse-in thread.
Described warehouse-in control documents, the attribute-name of preparing in data file path, data table name and tables of data according to the warehouse-in of warehouse-in control documents masterplate and corresponding model creates.
The advantage of the warehouse-in of the magnanimity multicast packet towards heterogeneous database implementation method of the present invention is:
(1) data receiver real-time: multicast receives, multicast message is resolved in advance and in-stockroom operation asynchronous working, guarantees to accept in real time and preserve multicast message;
(2) data integrity: message integrality judgement during receiving group message guarantees the integrality of wall scroll message, is directly deposited in the integrality that level cache data file has guaranteed data receiver after reception message;
(3) stability: preserve multicast message with the form of cache file, rather than be kept in internal memory, shielded the complicacy of memory management and the internal memory that may occur such as overflows at the mistake, strengthened the stability of system;
(4) shadow data storehouse isomerism: according to the different structure of tables of data in heterogeneous database, the warehouse-in that generates counter structure is prepared data file, call warehouse-in control documents masterplate and automatically generate corresponding warehouse-in control documents, realized the unified warehouse-in for heterogeneous database;
(5) data loading high efficiency: adopt multithreading, make each model have exclusive warehouse-in thread to carry out Data Analysis and warehouse-in, adopt SQL*Loader technology simultaneously, guaranteed the high efficiency of system warehouse-in.
Accompanying drawing explanation
Fig. 1 is the module diagram of realizing the warehouse-in of the magnanimity multicast packet towards heterogeneous database implementation method of the present invention;
Fig. 2 is the process flow diagram of the warehouse-in of the magnanimity multicast packet towards heterogeneous database implementation method of the present invention;
Fig. 3 is the file system schematic diagram of the warehouse-in of the magnanimity multicast packet towards heterogeneous database implementation method of the present invention;
Fig. 4 puts control documents template style figure in storage in the warehouse-in of the magnanimity multicast packet towards heterogeneous database implementation method of the present invention.
Embodiment
Hereinafter with reference to accompanying drawing, also describe technical scheme of the present invention in detail in conjunction with the embodiments.
The warehouse-in of the magnanimity multicast packet towards the heterogeneous database implementation method that the present invention proposes, by according to some cycles or restricted number receiving multicast data, by data apportion buffer memory to level cache data file, and complete the pre-parsing to level cache data file with semaphore triggering mode, according to directory index divide model by data buffer storage to L2 cache data file, use Thread Pool Technology to customize exclusive warehouse-in thread, with semaphore triggering mode, complete the parsing of L2 cache data file and warehouse-in, realized magnanimity multicast packet real-time stabilization and received and preserve.By pre-parsing, level cache data file is split into a plurality of L2 cache data files by model, greatly alleviate the burden of packet parsing.Use multithreading to make between model, warehouse-in thread and database corresponding one by one, avoided warehouse-in thread frequent problem of switching between database, guaranteed stability and the high efficiency of system.
As shown in Figure 1, a kind of warehouse-in of the magnanimity multicast packet towards heterogeneous database implementation method provided by the invention, its applied software systems, comprise multicast receiver module 1, the pre-parsing module 2 of multicast message and enter library module 3.
Multicast receiver module 1 major function comprises multicast message reception and filtration and multicast message preservation.
Multicast message receives and filters: monitor multicast receiving port, the type of the group packet voice receiving is differentiated, filter out unwanted message.
Multicast message is preserved: required message is saved in to level cache data file.Multicast receiver module 1 presses receiving cycle or receiving group number is divided level cache data file, and the current level cache data file in write state has Permanent File Name, relevant to the level cache data file number that completes reception in completing the name of level cache data file of write state.
Multicast message is pre-resolves and distribution: read the multicast message in level cache data file, according to the difference of multicast message source model, message is saved in respectively in L2 cache data file corresponding to each model.
Model queue management: the information of all multicast messages source model is preserved in model queue, comprise the username and password of model number, correspondence database, the table name of save data, tables of data attribute, complete and preserve secondary data number of files and put secondary data number of files in storage, the major function of model queue management is inquiry, preservation and the deletion of type information, for entering library module 3, provides necessary data supporting.
The major function that enters library module 3 comprises that warehouse-in thread management, multicast message are resolved, warehouse-in control documents is constructed and SQL*Loader warehouse-in.
Warehouse-in thread management: safeguard warehouse-in thread pool, control warehouse-in thread creation, startup, hang-up, recovery and recovery.
Multicast message is resolved: read L2 cache data file, resolve multicast message.
Warehouse-in control documents structure: call warehouse-in control documents masterplate, create the corresponding warehouse-in of model control documents, according to the information in model queue, structure warehouse-in control documents.
SQL*Loader warehouse-in: call SQL*Loader and warehouse-in control documents, by data typing correspondence database.
In conjunction with Fig. 1 and Fig. 2, the warehouse-in of the magnanimity multicast packet towards heterogeneous database implementation method of the present invention is described.
Step 1: start multicast message receiving thread, start receiving group message.Enter library and read configuration information, initialization multicast receiver module 1, the pre-parsing module 2 of multicast message and enter library module 3, multicast receiver module 1 starts multicast message receiving thread, starts receiving group message.Data multicast communication mechanism in the embodiment of the present invention is UDP cast communication.Described configuration information comprises multicast address and port, database address, port and SID (Security Identifiers, secure identifier).
Step 2: multicast receiver module 1 is monitored the network port, receiving group message.First carry out integrality judgement, if multicast message is imperfect, continue to receive, until receive complete; If multicast message is complete, the type of the multicast message receiving is tested, if not needed type, dropping packets, continues to receive new message; If required message is saved in message information in the level cache data file of current appointment; The multicast message quantity that reaches receiving cycle or reception when the time of receiving group message reaches specified quantity, close current level cache data file, and by this level cache data file of established rule rename, and record has completed the one-level data file number of reception, generate new level cache data file simultaneously, wait for writing of the new multi-cast message data receiving.
As shown in Figure 3, the file directory in the present invention is divided into two-stage, and level cache data file belongs to the file under first class catalogue.Level cache data file is divided into two classes by state:
A) first kind is the cache file just writing at receiving group message: when having multicast message to write cache file for the first time, by system, automatically generated, when multicast message receiving cycle or quantity reach setting value, close; This type of data buffering file has Permanent File Name, and unique;
B) Equations of The Second Kind is to have completed the cache file that data write: when this type of level cache data file is closed in first kind level cache data file, after completing the one-level data file number of reception and set Naming conventions rename, produce, generate new first kind level cache data file simultaneously.
The multicast message of described reception comprises header and imformosome.Header mainly comprises the fields such as length, model number, time, information type, information source and information object.
Length field represents the length of message except length field, for without symbol binary number, accounts for 2 bytes, and low byte is front, and high byte is rear.
Model number field, for model under identification information, accounts for 8 bytes.
Time field, for representing the generation time of this information, is accurate to 0.1ms, without symbol binary number, accounts for 6 bytes, and each byte is defined as follows:
1-2 byte is used for representing the day of year based on January 1st, 2010, unsigned int, and low byte is front;
3-6 byte is for representing the 0.1ms number of accumulative total on the same day, and without symbol short, low byte is front.
Information type field, for identifying type and the function of multicast message, accounts for 4 bytes, with ascii string, represents.
Information source field, for identifying the original producer of multicast message, accounts for 3 bytes.
Information destination field, for identifying the sending destination of multicast message, accounts for 3 bytes.
Imformosome is an elongated data structure, is the particular content of multicast message, can be formed also and can be comprised of binary number by ascii character, and the information type of form in header determined.
Step 3: judge whether level cache data file to be resolved, if having, resolve in advance level cache data file, otherwise, continue to wait for.
Whether the level cache data file number that completes reception in pre-parsing module 2 determining steps two of multicast message is greater than the level cache data file number that has completed parsing, if the two is equal, illustrates and there is no new need data to be processed, continues to wait for; If the former is greater than the latter, according to the level cache data file number that completes parsing, obtain next level cache data file to be resolved, read multi-cast message data, start to resolve in advance.During pre-parsing, only resolve the information source field in header, obtain the model that produces this message, Query Database obtains database information and the tables of data information that this model is relevant.
2 pairs of Equations of The Second Kind level cache data files of the pre-parsing module of multicast message are resolved in advance, by information source model, message are distributed and are saved in L2 cache data file corresponding to each model.
The information source field that the pre-parsing module 2 of multicast message is resolved in multicast message header, obtain the affiliated model of message, judge that the information of this model is whether in model queue, if not in model queue, obtain the information of this model and add in model queue, creating L2 cache data file storage directory corresponding to this model and associated data files and control documents, multicast message is write to L2 cache data file, invokes thread pond, starts this model warehouse-in thread simultaneously; If in model queue, directly multicast message is write in L2 cache data file corresponding to this model.
The information of described model comprise model number, corresponding database user name and password, save data table name, complete and preserve L2 cache data file number and put L2 cache data file number in storage.
The associated data files that described model is corresponding and control documents comprise L2 cache data file, warehouse-in preparation data file, warehouse-in control documents and misdata log file.
When starting model warehouse-in thread, using this type information as parameter, pass to warehouse-in thread.
After a level cache data file is parsed in advance, record has completed the level cache data file number of parsing, close all current L2 cache data files to be written, and by established rule rename L2 cache data file, more in new model queue, L2 cache data file number is preserved in corresponding completing, generate new L2 cache data file simultaneously, when wait is resolved in advance, store data next time.
As shown in Figure 3, L2 cache data file belongs to the data file in file directory, and second-level directory is model catalogue, the corresponding second-level directory of each model in model queue.The L2 cache data file that second-level directory comprises corresponding model, warehouse-in are prepared data file, warehouse-in control documents and misdata log file.
Take certain model as example, and L2 cache data file is divided into two classes by state:
A) first kind is writing the cache file of multicast message: the corresponding level cache data file of resolving in advance of this type of L2 cache data file, when creating this model catalogue, by system, automatically generated, after the pre-parsing of level cache data file, close, this class data file has Permanent File Name, and unique;
B) Equations of The Second Kind is to have completed the cache file that data write: this type of L2 cache data file is when first kind L2 cache data file is closed, after completing the secondary data number of files of reception and set Naming conventions rename, produce, generate new first kind L2 cache data file simultaneously.
Step 4: enter 3 pairs of Equations of The Second Kind L2 cache data files of library module and resolve, and the data after resolving are saved in the warehouse-in preparation data file of corresponding model, calling data loading tool SQL*Loader method and warehouse-in control documents, prepare database corresponding to data typing model in data file by warehouse-in.
Each model warehouse-in thread judges that completing of corresponding model preserve L2 cache data file number and whether be greater than and put L2 cache data file number in storage, if not, the data that there is no new needs warehouse-in be described, continues wait; If, according to putting L2 cache data file number in storage, obtain next one L2 cache data file to be put in storage, resolve this L2 cache data file, the warehouse-in that data after resolving is saved to corresponding model is prepared in data file, after being parsed, call SQL*Loader and warehouse-in control documents is prepared the data typing correspondence database in data file by warehouse-in, simultaneously the L2 cache of the warehouse-in data file number of corresponding model in new model queue more.Adopt multithreading management warehouse-in thread, make each model have exclusive warehouse-in thread to carry out Data Analysis and warehouse-in.
Warehouse-in is prepared data file, misdata log file and warehouse-in control documents and is belonged to the file under the catalogue of the second level in file system together, as shown in Figure 3.The data list structure that warehouse-in is prepared in the form of data file and the database of corresponding model is corresponding one by one, warehouse-in control documents is by calling warehouse-in control documents masterplate, according to the information of the database of corresponding model and concrete data list structure, create, thereby realize the warehouse-in of heterogeneous database.
Warehouse-in is prepared data file when creating the L2 cache data file storage directory of this model, by system, automatically generated, according to data tableau format in correspondence database, the data that parse are write to warehouse-in and prepare data file, file is csv form, after this enters data loading in database data document and completes, empty this document; This class data file has Permanent File Name, and unique under the L2 cache data file storage directory of model.
Misdata log file, when creating the L2 cache data file storage directory of this model, is generated automatically by system, when calling SQL*Loader by data input database, if there is misdata, by this data write error mistake document data record.
Warehouse-in control documents is when creating the L2 cache data file storage directory of this model, and the database relevant information according to corresponding model in warehouse-in control documents template and model queue, is generated automatically by system.
Warehouse-in control documents masterplate belongs to the file under first order catalogue in file system, warehouse-in control documents masterplate as shown in Figure 4, wherein " file_path ", " table_name " and " table_element " represent respectively the attribute-name in warehouse-in preparation data file path, data table name and the tables of data for the treatment of input database, warehouse-in control documents masterplate is called while generating concrete warehouse-in control documents, " $ file_path $ ", " $ table_name $ " and " $ table_element $ " are by the actual value that is replaced by of correspondence.
Claims (9)
1. towards a magnanimity multicast packet warehouse-in implementation method for heterogeneous database, it is characterized in that, comprise the following steps:
Step 1: start multicast message receiving thread, start receiving group message;
Step 2: first the multicast message receiving is carried out to integrality judgement, if imperfect, continue to receive; If complete, the type of message is tested, if not needed type, abandon this message, continue to receive new multicast message; If required message deposits message in current level cache data file to be written in; When the multicast message quantity that reaches receiving cycle or reception when the time of receiving group message reaches specified quantity, record has completed the one-level data file number of reception, close and the current level cache data file to be written of rename, generate new level cache data file to be written simultaneously;
Step 3: judge whether to need the level cache data file of parsing, if having, read the multicast message in level cache data file to be resolved, start to resolve in advance; Otherwise, continue to wait for;
Pre-method of resolving is: resolve the information source field in multicast message header, obtain the affiliated model of message, judge that information corresponding to this model is whether in model queue, if not in model queue, obtain the information of this model and add in model queue, and creating L2 cache data file storage directory corresponding to this model and associated data files and control documents, multicast message is write in L2 cache data file, invokes thread pond, starts this model warehouse-in thread simultaneously; If in model queue, directly multicast message is write in the L2 cache data file of corresponding model; Associated data files and control documents that described model is corresponding comprise: L2 cache data file, warehouse-in are prepared data file, warehouse-in control documents and misdata log file;
After a level cache data file is parsed in advance, record has completed the level cache data file number of parsing, close and the current all L2 cache data files to be written of rename, more in new model queue, L2 cache data file number is preserved in corresponding the completing of each model, generates new L2 cache data file to be written simultaneously;
Step 4: each model warehouse-in thread judges that completing of corresponding model preserve L2 cache data file number and whether be greater than and put L2 cache data file number in storage, if not, the data that there is no new needs warehouse-in be described, continues wait; If, according to putting L2 cache data file number in storage, obtain next one L2 cache data file to be put in storage, resolve this L2 cache data file, the warehouse-in that data after resolving is saved to corresponding model is prepared in data file, after being parsed, calling data loading tool SQL*Loader and warehouse-in control documents are prepared the data typing correspondence database in data file by warehouse-in, simultaneously the L2 cache of the warehouse-in data file number of corresponding model in new model queue more.
2. a kind of magnanimity multicast packet towards heterogeneous database according to claim 1 is put implementation method in storage, it is characterized in that, in described step 2, current level cache data file to be written is unique and have a Permanent File Name, relevant to the level cache data file number that completes reception in completing the name of level cache data file of write state.
3. a kind of warehouse-in of the magnanimity multicast packet towards heterogeneous database implementation method according to claim 1, is characterized in that the multicast message described in step 2 comprises header and imformosome; Header comprises length, model number, time, information type, information source and information destination field;
Length field represents the length of message except length field, for without symbol binary number, accounts for 2 bytes, and low byte is front, and high byte is rear;
Model number field is used for identifying the affiliated model of message, accounts for 8 bytes;
Time field, for representing the generation time of message, is accurate to 0.1ms, for without symbol binary number, accounts for 6 bytes, and each byte is defined as follows:
1-2 byte, for representing the day of year based on January 1st, 2010, is unsigned int, and low byte is front;
3-6 byte is for representing the 0.1ms number of same day accumulative total, and for without symbol short, low byte is front;
Information type field, for identifying type and the function of message, accounts for 4 bytes, with ascii string, represents;
Information source field, for identifying the original producer of message, accounts for 3 bytes;
Information destination field, for identifying the sending destination of message, accounts for 3 bytes;
Imformosome is an elongated data structure, is the particular content of message, ascii character or binary number, consists of, and the information type of form in header determined.
4. a kind of magnanimity multicast packet towards heterogeneous database according to claim 1 is put implementation method in storage, it is characterized in that, the information of the model described in step 3 comprise model number, corresponding database user name and password, save data table name, complete and preserve L2 cache data file number and put L2 cache data file number in storage.
5. a kind of warehouse-in of the magnanimity multicast packet towards heterogeneous database implementation method according to claim 1, is characterized in that described step 3 is obtained next level cache data file to be resolved according to the level cache data file number that completes parsing.
6. a kind of warehouse-in of the magnanimity multicast packet towards heterogeneous database implementation method according to claim 1, is characterized in that, when the startup model described in step 3 is put thread in storage, passes to warehouse-in thread using the information of this model as parameter.
7. a kind of magnanimity multicast packet towards heterogeneous database according to claim 1 is put implementation method in storage, it is characterized in that, warehouse-in described in step 3 and step 4 is prepared data file, the form of this document is corresponding one by one with the data list structure in the database of corresponding model, and it is unique that warehouse-in corresponding to each model prepared data file, and have Permanent File Name.
8. a kind of magnanimity multicast packet towards heterogeneous database according to claim 1 is put implementation method in storage, it is characterized in that, misdata log file described in step 3 and step 4, when calling SQL*Loader misdata is appearred in data input database, writes misdata.
9. a kind of magnanimity multicast packet towards heterogeneous database according to claim 1 is put implementation method in storage, it is characterized in that, warehouse-in control documents described in step 3 and step 4, the attribute-name of preparing in data file path, data table name and tables of data according to the warehouse-in of warehouse-in control documents masterplate and corresponding model creates.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310488096.6A CN103559217B (en) | 2013-10-17 | 2013-10-17 | A kind of massive multicast data towards isomeric data storehouse warehouse-in implementation method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310488096.6A CN103559217B (en) | 2013-10-17 | 2013-10-17 | A kind of massive multicast data towards isomeric data storehouse warehouse-in implementation method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103559217A true CN103559217A (en) | 2014-02-05 |
| CN103559217B CN103559217B (en) | 2016-06-01 |
Family
ID=50013464
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310488096.6A Expired - Fee Related CN103559217B (en) | 2013-10-17 | 2013-10-17 | A kind of massive multicast data towards isomeric data storehouse warehouse-in implementation method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103559217B (en) |
Cited By (23)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104077368A (en) * | 2014-06-18 | 2014-10-01 | 国电南瑞科技股份有限公司 | History data two-level caching multi-stage submitting method for dispatching monitoring system |
| CN104657462A (en) * | 2015-02-10 | 2015-05-27 | 北京宇航系统工程研究所 | Massive measurement data near-real-time storage method |
| CN104750814A (en) * | 2015-03-30 | 2015-07-01 | 大连理工大学 | Multisensor-based multivariate and heterogeneous data steam automatic storage method |
| CN104834635A (en) * | 2014-02-07 | 2015-08-12 | 中国移动通信集团广东有限公司 | Data processing method and device |
| CN105468705A (en) * | 2015-11-18 | 2016-04-06 | 广东南方通信建设有限公司 | Mobile communication background data file importing method |
| CN105808534A (en) * | 2014-12-27 | 2016-07-27 | 株洲南车时代电气股份有限公司 | A method of extracting original data in a heterogeneous data fusion file |
| CN105808778A (en) * | 2016-03-30 | 2016-07-27 | 中国银行股份有限公司 | Method and device for extracting, transforming and loading mass data |
| CN106230916A (en) * | 2016-07-26 | 2016-12-14 | 深圳市乐唯科技开发有限公司 | Data receiver that a kind of anti-user interface is stuck and management method |
| CN107204939A (en) * | 2017-05-27 | 2017-09-26 | 南京南瑞继保电气有限公司 | A kind of message processing method based on two-level cache |
| CN107798138A (en) * | 2017-11-23 | 2018-03-13 | 广东电网有限责任公司电力科学研究院 | A kind of real time data stowage and device |
| CN108595511A (en) * | 2018-03-23 | 2018-09-28 | 中国人民解放军91977部队 | A kind of diversification meteorological model data classification storage processing method and system |
| CN109033289A (en) * | 2018-07-13 | 2018-12-18 | 天津瑞能电气有限公司 | A kind of banking procedure of the high frequency real time data for micro-capacitance sensor |
| CN109657413A (en) * | 2019-01-23 | 2019-04-19 | 中国人民解放军国防科技大学 | Simulation component model warehousing method, model library, computer equipment and storage medium |
| CN111221829A (en) * | 2019-11-15 | 2020-06-02 | 浙江鸿程计算机系统有限公司 | Processing method for dealing with warehousing bottleneck problem of relational database |
| CN111245891A (en) * | 2019-12-27 | 2020-06-05 | 特瓦特能源科技有限公司 | Real-time data display method and device |
| CN112328542A (en) * | 2020-11-25 | 2021-02-05 | 天津凯发电气股份有限公司 | Method for importing data in heterogeneous data file into database |
| CN112764908A (en) * | 2021-01-26 | 2021-05-07 | 北京鼎普科技股份有限公司 | Network data acquisition processing method and device and electronic equipment |
| CN112785284A (en) * | 2020-12-31 | 2021-05-11 | 银清科技有限公司 | Message storage method and device based on structured document |
| CN112822090A (en) * | 2019-11-15 | 2021-05-18 | 北京新媒传信科技有限公司 | Communication method, device and system of instant messaging tool suitable for mobile terminal |
| CN113032295A (en) * | 2021-02-25 | 2021-06-25 | 西安电子科技大学 | Data packet second-level caching method, system and application |
| CN113127551A (en) * | 2021-04-25 | 2021-07-16 | 在秀网络科技(深圳)有限公司 | Data calling method and system for data bearing platform replacing front-end equipment |
| CN114422624A (en) * | 2022-01-25 | 2022-04-29 | 电信科学技术第五研究所有限公司 | Data receiving method |
| CN116662037A (en) * | 2023-07-24 | 2023-08-29 | 杭州鉴智机器人科技有限公司 | Processing method and device for shared memory, electronic equipment and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101277272A (en) * | 2008-05-16 | 2008-10-01 | 北京航空航天大学 | Method for implementing magnanimity broadcast data warehouse-in |
| US20110016528A1 (en) * | 2008-08-15 | 2011-01-20 | Venus Info Tech Inc. | Method and Device for Intrusion Detection |
| CN102750377A (en) * | 2012-06-21 | 2012-10-24 | 深圳市谷米科技有限公司 | Massive data storage and retrieval method |
| CN102891873A (en) * | 2011-07-21 | 2013-01-23 | 腾讯科技（深圳）有限公司 | Method for storing log data and log data storage system |
| CN102929951A (en) * | 2012-10-08 | 2013-02-13 | 深圳市博瑞得科技有限公司 | Mass data storage method and device with data binding |
| CN103237059A (en) * | 2013-04-02 | 2013-08-07 | 西安翔迅科技有限责任公司 | Method for interacting traffic information data and commands |
-
2013
- 2013-10-17 CN CN201310488096.6A patent/CN103559217B/en not_active Expired - Fee Related
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101277272A (en) * | 2008-05-16 | 2008-10-01 | 北京航空航天大学 | Method for implementing magnanimity broadcast data warehouse-in |
| US20110016528A1 (en) * | 2008-08-15 | 2011-01-20 | Venus Info Tech Inc. | Method and Device for Intrusion Detection |
| CN102891873A (en) * | 2011-07-21 | 2013-01-23 | 腾讯科技（深圳）有限公司 | Method for storing log data and log data storage system |
| CN102750377A (en) * | 2012-06-21 | 2012-10-24 | 深圳市谷米科技有限公司 | Massive data storage and retrieval method |
| CN102929951A (en) * | 2012-10-08 | 2013-02-13 | 深圳市博瑞得科技有限公司 | Mass data storage method and device with data binding |
| CN103237059A (en) * | 2013-04-02 | 2013-08-07 | 西安翔迅科技有限责任公司 | Method for interacting traffic information data and commands |
Non-Patent Citations (2)
| Title |
|---|
| 陈蔚文，马世龙: "航天器测试信息系统数据资源管理平台的设计与实现", 《2007中国控制与决策学术年会论文集》, 31 December 2007 (2007-12-31) * |
| 马世龙 等: "面向航天器自动化测试的通用测试语言及系统关键技术研究与应用", 《中国科技成果》, no. 13, 31 December 2012 (2012-12-31), pages 62 - 63 * |
Cited By (32)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104834635A (en) * | 2014-02-07 | 2015-08-12 | 中国移动通信集团广东有限公司 | Data processing method and device |
| CN104077368A (en) * | 2014-06-18 | 2014-10-01 | 国电南瑞科技股份有限公司 | History data two-level caching multi-stage submitting method for dispatching monitoring system |
| CN105808534A (en) * | 2014-12-27 | 2016-07-27 | 株洲南车时代电气股份有限公司 | A method of extracting original data in a heterogeneous data fusion file |
| CN105808534B (en) * | 2014-12-27 | 2019-06-11 | 株洲中车时代电气股份有限公司 | A kind of method that isomeric data merges extraction initial data in file |
| CN104657462B (en) * | 2015-02-10 | 2017-12-22 | 北京宇航系统工程研究所 | A kind of magnanimity measurement data quasi real time storage method |
| CN104657462A (en) * | 2015-02-10 | 2015-05-27 | 北京宇航系统工程研究所 | Massive measurement data near-real-time storage method |
| CN104750814A (en) * | 2015-03-30 | 2015-07-01 | 大连理工大学 | Multisensor-based multivariate and heterogeneous data steam automatic storage method |
| CN105468705A (en) * | 2015-11-18 | 2016-04-06 | 广东南方通信建设有限公司 | Mobile communication background data file importing method |
| CN105808778A (en) * | 2016-03-30 | 2016-07-27 | 中国银行股份有限公司 | Method and device for extracting, transforming and loading mass data |
| CN105808778B (en) * | 2016-03-30 | 2019-04-26 | 中国银行股份有限公司 | A kind of mass data extracts, conversion, loading method and device |
| CN106230916A (en) * | 2016-07-26 | 2016-12-14 | 深圳市乐唯科技开发有限公司 | Data receiver that a kind of anti-user interface is stuck and management method |
| CN107204939A (en) * | 2017-05-27 | 2017-09-26 | 南京南瑞继保电气有限公司 | A kind of message processing method based on two-level cache |
| CN107798138A (en) * | 2017-11-23 | 2018-03-13 | 广东电网有限责任公司电力科学研究院 | A kind of real time data stowage and device |
| CN108595511B (en) * | 2018-03-23 | 2022-04-01 | 中国人民解放军91977部队 | Diversified meteorological hydrological data classification storage processing method and system |
| CN108595511A (en) * | 2018-03-23 | 2018-09-28 | 中国人民解放军91977部队 | A kind of diversification meteorological model data classification storage processing method and system |
| CN109033289A (en) * | 2018-07-13 | 2018-12-18 | 天津瑞能电气有限公司 | A kind of banking procedure of the high frequency real time data for micro-capacitance sensor |
| CN109657413A (en) * | 2019-01-23 | 2019-04-19 | 中国人民解放军国防科技大学 | Simulation component model warehousing method, model library, computer equipment and storage medium |
| CN109657413B (en) * | 2019-01-23 | 2023-05-23 | 中国人民解放军国防科技大学 | Simulation component model warehousing method, model library, computer equipment and storage medium |
| CN111221829A (en) * | 2019-11-15 | 2020-06-02 | 浙江鸿程计算机系统有限公司 | Processing method for dealing with warehousing bottleneck problem of relational database |
| CN112822090B (en) * | 2019-11-15 | 2023-03-31 | 北京新媒传信科技有限公司 | Communication method, device and system of instant messaging tool suitable for mobile terminal |
| CN112822090A (en) * | 2019-11-15 | 2021-05-18 | 北京新媒传信科技有限公司 | Communication method, device and system of instant messaging tool suitable for mobile terminal |
| CN111245891A (en) * | 2019-12-27 | 2020-06-05 | 特瓦特能源科技有限公司 | Real-time data display method and device |
| CN112328542A (en) * | 2020-11-25 | 2021-02-05 | 天津凯发电气股份有限公司 | Method for importing data in heterogeneous data file into database |
| CN112785284A (en) * | 2020-12-31 | 2021-05-11 | 银清科技有限公司 | Message storage method and device based on structured document |
| CN112764908A (en) * | 2021-01-26 | 2021-05-07 | 北京鼎普科技股份有限公司 | Network data acquisition processing method and device and electronic equipment |
| CN112764908B (en) * | 2021-01-26 | 2024-01-26 | 北京鼎普科技股份有限公司 | Network data acquisition processing method and device and electronic equipment |
| CN113032295A (en) * | 2021-02-25 | 2021-06-25 | 西安电子科技大学 | Data packet second-level caching method, system and application |
| CN113127551A (en) * | 2021-04-25 | 2021-07-16 | 在秀网络科技(深圳)有限公司 | Data calling method and system for data bearing platform replacing front-end equipment |
| CN114422624A (en) * | 2022-01-25 | 2022-04-29 | 电信科学技术第五研究所有限公司 | Data receiving method |
| CN114422624B (en) * | 2022-01-25 | 2024-05-17 | 电信科学技术第五研究所有限公司 | Data receiving method |
| CN116662037A (en) * | 2023-07-24 | 2023-08-29 | 杭州鉴智机器人科技有限公司 | Processing method and device for shared memory, electronic equipment and storage medium |
| CN116662037B (en) * | 2023-07-24 | 2023-10-20 | 杭州鉴智机器人科技有限公司 | Processing method and device for shared memory, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103559217B (en) | 2016-06-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103559217A (en) | Heterogeneous database oriented massive multicast data storage implementation method | |
| CN105138592B (en) | A kind of daily record data storage and search method based on distributed structure/architecture | |
| WO2019134226A1 (en) | Log collection method, device, terminal apparatus, and storage medium | |
| US11625412B2 (en) | Storing data items and identifying stored data items | |
| TW201800967A (en) | Method and device for processing distributed streaming data | |
| WO2021073510A1 (en) | Statistical method and device for database | |
| CN107133231B (en) | Data acquisition method and device | |
| CN111221791A (en) | Method for importing multi-source heterogeneous data into data lake | |
| CN108228322B (en) | Distributed link tracking and analyzing method, server and global scheduler | |
| WO2014110940A1 (en) | A method, apparatus and system for storing, reading the directory index | |
| CN110414259A (en) | A kind of method and apparatus for constructing data element, realizing data sharing | |
| CN114328759A (en) | Data construction and management method and terminal of data warehouse | |
| CN113177090A (en) | Data processing method and device | |
| CN109344137A (en) | A kind of log storing method and system | |
| CN109902070B (en) | WiFi log data-oriented analysis storage search method | |
| CN103853504B (en) | Data access method of caching system | |
| CN115858322A (en) | Log data processing method and device and computer equipment | |
| CN109063201B (en) | Impala online interactive query method based on mixed storage scheme | |
| CN107590199A (en) | A kind of multi-thread data storehouse design method towards internal memory | |
| CN107832464B (en) | Data bleaching method and device | |
| CN111654410B (en) | Gateway request monitoring method, device, equipment and medium | |
| CN117194298B (en) | Control method, device, equipment and storage medium | |
| CN117171272A (en) | Data synchronization method and device | |
| CN117435670A (en) | Heterogeneous data source synchronization method, heterogeneous data source synchronization system, electronic equipment and storage medium | |
| CN117648086A (en) | Script file conversion method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20160601 Termination date: 20201017 |