CN110502540A - Data processing method, device, computer equipment and storage medium - Google Patents
Data processing method, device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN110502540A CN110502540A CN201910666341.5A CN201910666341A CN110502540A CN 110502540 A CN110502540 A CN 110502540A CN 201910666341 A CN201910666341 A CN 201910666341A CN 110502540 A CN110502540 A CN 110502540A
- Authority
- CN
- China
- Prior art keywords
- data
- target data
- unit
- plug
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
This application involves a kind of data processing method, device, computer equipment and storage medium, computer equipment have in detecting data source data it is to be synchronized when, it is determining with the matched reading plug-in unit of data source according to preset configuration relation；It include data source and the corresponding relationship and purpose library for reading plug-in unit and the corresponding relationship for writing plug-in unit in configuration relation；Then it calls and reads plug-in unit, by reading the target data in plug-in unit reading data source；And determine the corresponding purpose library of target data；Plug-in unit is write with purpose storehouse matching finally, calling, by writing plug-in unit target data is synchronized to purpose library.The real-time of data processing without batch processing, can be improved, and reduce the consumption to network by real time data synchronization to be synchronized into purpose library using the above method.
Description
Technical field
This application involves database technical field, more particularly to a kind of data processing method, device, computer equipment and
Storage medium.
Background technique
With the rise and rapid development of network, a large amount of information and data are produced, are gone to arrange number with the method for science
According to as problem in the urgent need to address.The data of different data sources are integrated, inside data of enterprise isolated island can be got through,
Stream compression is realized, to realize with different view to the Accurate Analysis and accurate judgement of enterprise operation various aspects information.
In traditional technology, computer equipment mainly passes through data warehouse technology (Extract-Transform-Load, abbreviation
ETL technology) data are extracted from each operation system, and cleaning conversion is carried out to data, undesirable data are filtered out,
Such as incomplete data, the data of mistake or duplicate data, it then loads data into data warehouse, realizing will enterprise
Dispersion, messy, the skimble-scamble Data Integration of standard in industry is to together.
But as the expansion of data scale and data variation speed are promoted, computer equipment using the above method without
The real-time change of data, is not able to satisfy the demand of real time business scene in method perception service system.
Summary of the invention
Based on this, it is necessary in view of the above technical problems, provide a kind of data processing method, device, computer equipment and
Storage medium.
A kind of data processing method, the above method include:
Have in detecting data source data it is to be synchronized when, according to preset configuration relation, determination is matched with data source
Read plug-in unit；It include data source and the corresponding relationship and purpose library for reading plug-in unit and the corresponding relationship for writing plug-in unit in configuration relation；
It calls and reads plug-in unit, by reading the target data in plug-in unit reading data source；
Determine the corresponding purpose library of target data；
Calling writes plug-in unit with purpose storehouse matching, and target data is synchronized to purpose library by writing plug-in unit.
It in one of the embodiments, further include the number of target data to be extracted in data source in above-mentioned configuration relation
According to the corresponding relationship of type and data type and purpose library；
By reading the target data in plug-in unit reading data source, comprising:
Data type based on target data, by reading the target data in plug-in unit reading data source；
Determine the corresponding purpose library of target data, comprising:
Data type based on target data determines purpose corresponding with target data library.
The above method in one of the embodiments, further include:
Target data is cached by first order memory cache；
According to the blocked state in purpose library, the target data cached in first order memory cache write-in second level memory is delayed
It deposits or file cache；
Target data is synchronized to purpose library by writing plug-in unit, comprising:
By writing plug-in unit, the target data cached in second level memory cache or file cache is synchronized to purpose library.
It is above-mentioned in one of the embodiments, that second level memory is written into the target data cached in first order memory cache
Caching or file cache, comprising:
The target data cached in first order memory cache is cached to message publishing system；
According to the blocked state in purpose library, target data is obtained from message publishing system and second level memory cache is written
Or file cache.
The above-mentioned blocked state according to purpose library in one of the embodiments, obtains target from message publishing system
Simultaneously second level memory cache or file cache is written in data, comprising:
The overflow value in purpose library is obtained, overflow value is used to characterize the blocked state in purpose library；
When overflow value is less than or equal to preset spilling threshold value, the target data cached in message publishing system is written
Second level memory cache；Overflow threshold value by second level memory cache can buffer data size determine；
When overflow value is greater than and overflows threshold value, file cache is written into the target data cached in message publishing system.
The overflow value in above-mentioned acquisition purpose library in one of the embodiments, comprising:
When data reach second level memory cache, whether detection second level memory cache overflows；
If so, overflow value adds 1；
If it is not, then overflow value is reset.
The above-mentioned target data by second level memory cache or file cache is synchronized in one of the embodiments,
Purpose library, comprising:
It is when spatial cache in second level memory cache or file cache is less than pre-set space threshold value, target data is same
It walks to purpose library；
Alternatively,
According to preset caching period, the target that will be cached in second level memory cache in caching period or file cache
Data are synchronized to purpose library.
In one of the embodiments, above by the target data read in plug-in unit reading data source, comprising:
The data that data source is parsed by reading plug-in unit store log；Record has the behaviour to target data in data storage log
It notes down；
The data manipulation mode of target data is determined according to operation note；Data manipulation mode includes Data Definition Language
DDL and data manipulation language DML；
If the data manipulation mode of target data is DDL, the corresponding DDL operation of target data is read by reading plug-in unit；
If the data manipulation mode of target data is DML, target data is read by reading plug-in unit；
Target data is synchronized to purpose library by writing plug-in unit, comprising:
If the data manipulation mode of target data is DDL, the DDL of target data will be grasped in data source by writing plug-in unit
Make, be converted into operating with the target DDL of purpose storehouse matching, and data are carried out to the data in purpose library using target DDL operation
Change；
If the data manipulation mode of target data is DML, purpose library is written into target data by writing plug-in unit.
A kind of data processing equipment, above-mentioned apparatus include:
Matching module, for have in detecting data source data it is to be synchronized when, according to preset configuration relation, determine with
The matched reading plug-in unit of data source；Include data source in configuration relation and reads the corresponding relationship of plug-in unit and purpose library and write plug-in unit
Corresponding relationship；
Calling module reads plug-in unit for calling, by reading the target data in plug-in unit reading data source；
Determining module, for determining the corresponding purpose library of target data；
Synchronization module writes plug-in unit with purpose storehouse matching for calling, by writing plug-in unit target data is synchronized to purpose
Library.
A kind of computer equipment, including memory and processor, memory are stored with computer program, and above-mentioned processor is held
The step of realizing above-mentioned data processing method when row computer program.
A kind of computer readable storage medium, is stored thereon with computer program, and the computer program is held by processor
The step of above-mentioned data processing method is realized when row.
Above-mentioned data processing method, device, computer equipment and storage medium, computer equipment is in detecting data source
Have data it is to be synchronized when, it is determining with the matched reading plug-in unit of data source according to preset configuration relation；It include data in configuration relation
The corresponding relationship and purpose library of source and reading plug-in unit and the corresponding relationship for writing plug-in unit；Then it calls and reads plug-in unit, read by reading plug-in unit
Take the target data in data source；And determine the corresponding purpose library of target data；Finally, calling slotting with writing for purpose storehouse matching
Target data is synchronized to purpose library by writing plug-in unit by part.Due to including to be inserted with the matched reading of data source in computer equipment
Part so that have in detecting data source data it is to be synchronized when, matched reading plug-in unit can be called to read target data, then again
Calling writes plug-in unit and target data is synchronized in purpose library, so as to by real time data synchronization to be synchronized into purpose library, and
Batch processing is not needed, the real-time of data processing is improved；Further, since computer equipment in real time writes target device
Enter in purpose library, data volume reduces when being written relative to batch, reduces the consumption to network.
Detailed description of the invention
Fig. 1 is the applied environment figure of data processing method in one embodiment；
Fig. 2 is the flow diagram of data processing method in one embodiment；
Fig. 3 is the flow diagram of data processing method in another embodiment；
Fig. 4 is the flow diagram of data processing method in another embodiment；
Fig. 5 is the flow diagram of data processing method in another embodiment；
Fig. 6 is the flow diagram of data processing method in another embodiment；
Fig. 7 is the structural block diagram of data processing equipment in one embodiment；
Fig. 8 is the structural block diagram of data processing equipment in another embodiment；
Fig. 9 is the structural block diagram of data processing equipment in another embodiment；
Figure 10 is the structural block diagram of data processing equipment in another embodiment；
Figure 11 is the internal structure chart of computer equipment in one embodiment.
Specific embodiment
It is with reference to the accompanying drawings and embodiments, right in order to which the objects, technical solutions and advantages of the application are more clearly understood
The application is further elaborated.It should be appreciated that specific embodiment described herein is only used to explain the application, not
For limiting the application.
Data processing method provided by the present application can be applied in application environment as shown in Figure 1.Wherein, computer
Equipment 100 can be connect with multiple data sources and multiple purpose libraries.Wherein, computer equipment can with independent server or
It is the server cluster of multiple server compositions to realize.
In one embodiment, as shown in Fig. 2, providing a kind of data processing method, it is applied in Fig. 1 in this way
It is illustrated for computer equipment 100, comprising:
S101, have in detecting data source data it is to be synchronized when, it is determining with data source according to preset configuration relation
The reading plug-in unit matched；It include that the corresponding relationship of data source and reading plug-in unit and purpose library are closed with the corresponding of plug-in unit is write in configuration relation
System.
Wherein, above-mentioned data source can be various businesses system, such as selling operation system, financial affair work system and people
Thing operation system etc., can also be obtain data application program, for data source type it is not limited here.By never
With the extraction data in data source, then data are integrated, data basis can be provided for enterprise diagnosis and judgement.
Above-mentioned data can be the structural data in data source, be also possible to unstructured data, such as video, sound
Frequently, table etc., for above-mentioned data type it is not limited here.
Specifically, computer equipment can extract instruction according to the data received, have data to wait for determine in data source
It is synchronous；Synchrodata can also be determined the need in real time according to the log generated in data source, exist for above-mentioned detection mode
This is without limitation.
Due to different to the read-write modes of data in different data sources and different purpose libraries, stored in computer equipment
With different data sources matched multiple reading plug-in units, and from the multiple of different purpose storehouse matchings write plug-in unit, above-mentioned reading plug-in unit and
Writing plug-in unit can be stored in the form of data packet in program data processing routine catalogue.It is above-mentioned to match for above-mentioned configuration relation
It sets in relationship and can will read plug-in unit in table form, write plug-in unit and data source and target source is associated, it can also be by adding
The mode of identifier is added to be associated, it is not limited here.Wherein each data source can only correspond to a reading plug-in unit, each mesh
Library can only corresponding one write plug-in unit；In addition, the same reading plug-in unit can be associated with from different data sources, it is same to write plug-in unit
It can also be associated with from different purpose libraries.
S102, it calls and reads plug-in unit, by reading the target data in plug-in unit reading data source.
In computer equipment after the corresponding reading plug-in unit of data source has been determined, above-mentioned reading can be called in program directory
Plug-in unit, and by reading the target data in plug-in unit reading data source.Wherein, above-mentioned target data refers to be synchronized in data source
Data, above-mentioned target data can be identical type, be also possible to different types.Computer equipment can be read together simultaneously
Target data in one data source can also read the target data in different data sources, it is not limited here simultaneously.
Specifically, computer equipment can all read data to be synchronized in data source when reading target data,
Data can be carried out with screening and obtain target data, it is not limited here.
In addition, computer equipment by read plug-in unit read data source data when, in addition to read data source in target
Data, can also read information relevant to data source, such as the generation times of data, the type of service of data source and continue
Data volume size of the target data taken etc., it is not limited here.
S103, the corresponding purpose library of target data is determined.
After computer equipment is by reading the target data that plug-in unit is read in data source, target data may further determine that
Which database is target data be synchronized in by corresponding purpose library with determination.
The purpose library connecting with computer equipment can be relevant database, can be used for processing structure data；
It can be non-relational database, can be used for handling unstructured data.For example, above-mentioned purpose library can be column storage
Database Vertica is also possible to Hadoop database, can also be KUDU database, it is not limited here.
Computer equipment can determine the corresponding purpose library of target data according to the type of target data；For example, working as data
When target data in source corresponds to two kinds of data types, then corresponding purpose library can be two different databases；In addition,
Computer equipment can also according to the type of data source, to determine purpose library corresponding to the target data extracted in data source,
For example, the target data extracted in financial affair work system can be written to corresponding when data source is financial affair work system
In purpose library；For above-mentioned purpose library method of determination it is not limited here.
S104, calling and purpose storehouse matching write plug-in unit, and target data is synchronized to purpose library by writing plug-in unit.
Further, it after computer equipment has determined the corresponding purpose library of target data, can call and purpose library
That matches writes plug-in unit, and target data is synchronized in purpose library.
Specifically, computer equipment can first cache data, when progress data are synchronous then again by caching
Data are synchronized in purpose library by writing plug-in unit.
Computer equipment can be target data writing direct purpose library in synchronous target data, be also possible to pair
Data in purpose library are modified, so that the data in purpose library are synchronous with target data, herein for the above-mentioned method of synchronization
Without limitation.
Above-mentioned data processing method, computer equipment have in detecting data source data it is to be synchronized when, according to preset
Configuration relation, the determining and matched reading plug-in unit of data source；It include the corresponding relationship of data source and reading plug-in unit in configuration relation, and
Purpose library and the corresponding relationship for writing plug-in unit；Then it calls and reads plug-in unit, by reading the target data in plug-in unit reading data source；And really
The corresponding purpose library of the data that set the goal；Plug-in unit is write with purpose storehouse matching finally, calling, it is by writing plug-in unit that target data is synchronous
To purpose library.Due to include in computer equipment with the matched reading plug-in unit of data source so that there is number in detecting data source
According to it is to be synchronized when, matched reading plug-in unit can be called to read target data, then recalled and write plug-in unit and be synchronized to target data
In purpose library, so as to which real time data synchronization to be synchronized into purpose library, without batch processing, is improved at data
The real-time of reason；Further, since target device is written in purpose library in real time for computer equipment, when being written relative to batch
Data volume reduces, and reduces the consumption to network.
It in one embodiment, further include the data class of target data to be extracted in data source in above-mentioned configuration relation
The corresponding relationship of type and data type and purpose library；Computer equipment is by reading the number of targets in plug-in unit reading data source
According to when, can pass through and read plug-in unit and read target data in data source based on the data type of target data.For example, data source A
In have data 1, data 2 and data 3 comprising data type, and data type to be extracted in data source is contained in configuration relation
For data 2, computer equipment, can root sentence configuration relation reading number when reading the target data in data source by reading plug-in unit
According to the data 2 in the A of source.
Correspondingly, computer equipment, can be based on the data of target data when determining the corresponding purpose library of target data
Type determines purpose corresponding with target data library.That is, containing different types of number of targets in above-mentioned configuration relation
It is data 2 according to corresponding purpose library, such as when reading the data type that plug-in unit is read, according to the corresponding mesh of data 2 in configuration relation
Library be database B, then can determine the corresponding purpose library of target data be database B.
Above-mentioned data processing method, computer equipment can according to the demand of user, by configuration relation in data source
Data carry out screening obtain target data, data processing method can be made to meet customer need for greater flexibility；Further, may be used
To reduce the data volume reading plug-in unit and reading, so that reading data duration is reduced, so that data processing real-time is higher.
Fig. 3 is the flow diagram of data processing method in another embodiment, and the present embodiment is related to computer equipment pair
The process that data are cached, on the basis of the above embodiments, as shown in figure 3, the above method further include:
S201, target data is cached by first order memory cache.
S202, according to the blocked state in purpose library, the second level is written into the target data cached in first order memory cache
Memory cache or file cache.
Wherein, target data can be stored in above-mentioned first order memory cache avoid data repetition create and
Transmission, can be improved data processing speed.In addition, in the first order memory cache data cleansing can be carried out to target data
Processing, filters undesirable data, such as incomplete data, the data of mistake and duplicate data etc.；It can be with
Conversion operation, such as data granularity conversion etc. are carried out to target data.
Further, computer equipment can be according to the blocked state in the corresponding purpose library of target data, to determine first
In the target data write-in second level memory cache cached in grade memory cache, or in write-in file cache.Wherein, the second level
For memory cache compared with file cache, the readwrite performance under high concurrent state is excellent, and can support cluster with component, automatically
Manage load balancing；In file cache, data are safer, and the target data stored after system is powered down will not lose, and
In the case where file cache off-capacity, it is easy extension and increases capacity.Computer equipment joint uses second level memory cache
Target data is handled with file cache, can effectively alleviate purpose library blocked state, so that data write-in is more steady
It is fixed.Between first order memory cache and second level memory cache or file cache, data switch can be set, computer is set
The standby conducting direction that data switch can be determined by the blocked state in purpose library.
Correspondingly, after second level memory cache or file cache is written in target data by computer equipment, Ke Yitong
It crosses and writes plug-in unit, the target data cached in second level memory cache or file cache is synchronized to purpose library.
Specifically, computer equipment can be by the target data real-time synchronization in second level memory cache or file cache
To purpose library；Optionally, pre-set space threshold value can also be less than in the spatial cache in second level memory cache or file cache
When, target data is synchronized to purpose library.For example, when being filled with target data in the memory of the second level and then having been built with purpose library
Vertical connection, by the target data batch synchronization in the memory cache of the second level to purpose library, avoids second level memory cache to each
The write-in of data caused by connection is frequently established slowly with purpose library when data is written, and is written to improve to purpose library
The writing speed of target data.
In addition, computer equipment can also according to preset caching period, by second level memory cache in caching period or
The target data cached in person's file cache is synchronized to purpose library.When the number that the first memory cache is written into the second memory cache
When according to measuring smaller, time length comparison required for the second memory cache is write completely is long, the real-time of the data write-in of influence；Therefore,
Computer equipment sets the soul village period of the second memory cache, such as the target data that the second memory cache will be written in 5 minutes
In batch write-in purpose library.
Above-mentioned data processing method, computer equipment will be delayed according to the blocked state in purpose library in first order memory cache
The target data write-in second level memory cache or file cache deposited, can be written with data stabilization, reduce the impact to network.
Fig. 4 is the flow diagram of data processing method in another embodiment, and the present embodiment is related to computer equipment will
The concrete mode of second level memory cache or file cache is written in target data, on the basis of the above embodiments, such as Fig. 4 institute
Show, above-mentioned S202 includes:
S301, the target data cached in first order memory cache is cached to message publishing system.
Wherein, above-mentioned message publishing system can be kafka system, provide data-interface in kafka system, so that outside
Portion's system can consume the target data cached in fafka system before purpose library is written in target data.
S302, according to the blocked state in purpose library, from obtaining target data in message publishing system and be written in the second level
Deposit caching or file cache.
Further, computer equipment can obtain target data from message publishing system and second level memory is written and delay
It deposits or file cache.
Above-mentioned data processing method, by caching the target data cached in the first memory cache to news release system
System, allows external system consumption target data at any time, improves the Facility Consumption of target data.
Fig. 5 be another embodiment in data processing method flow diagram, the present embodiment be related to computer equipment from
Target data is obtained in message publishing system and the process of second level memory cache or file cache is written, in above-described embodiment
On the basis of, above-mentioned S302 includes:
S401, the overflow value for obtaining purpose library, overflow value are used to characterize the blocked state in purpose library.
Specifically, the overflow value in the available purpose library of computer equipment, above-mentioned overflow value are used to characterize the resistance in purpose library
Plug-like state.When blocking when data are written to purpose library in second level memory cache, correspondingly second level memory cache can go out
Existing data are overflowed, therefore computer equipment can be when continuous overflow occurs in second level memory cache, it is believed that purpose library hinders
Plug.
Computer equipment, can be to the case where second level memory cache overflows when data being written every time when obtaining overflow value
Added up, when data reach second level memory cache, whether detection second level memory cache overflows；If so, overflowing
Value plus 1 out；If it is not, then overflow value is reset.For example, when data being written to second level memory cache continuous 10 times, in the second level
Caching is deposited to overflow, overflow value adds up to be 10 at this time, when data being written to second level memory cache the 11st time, the second level
Memory cache does not overflow, at this point, computer equipment thinks that second level memory cache data-handling capacity can satisfy demand,
Overflow value is reset.
S402, when overflow value is less than or equal to preset spilling threshold value, the number of targets that will be cached in message publishing system
According to write-in second level memory cache；Overflow threshold value by second level memory cache can buffer data size determine.
S403, overflow value be greater than overflow threshold value when, by the target data cached in message publishing system write-in file delay
It deposits.
Overflow value can be compared, then by computer equipment after obtaining overflow value with preset spilling threshold value
It is determined according to comparison result and writes data into second level memory cache or file cache.Wherein, above-mentioned spilling threshold value is by
Secondary memory caching can buffer data size determine, for example, above-mentioned spilling threshold value can be can buffer data size 20%, if the
Secondary memory caching can cache 100 datas, then above-mentioned spilling threshold value can be 20, that is to say, that when overflow value be less than etc.
When 20, computer equipment thinks that second level memory cache can satisfy data processing needs, not will cause data write-in resistance
Plug can make in data stabilization write-in purpose library.When being overflowed to second level memory cache write-in data for continuous 20 times, meter
Calculating machine equipment can consider that purpose library is blocked, so that target data is written in file cache, alleviate the write-in in purpose library
Pressure.
Target data is written in the second level to determine by obtaining overflow value for above-mentioned data processing method, computer equipment
Caching or file cache are deposited, allows computer equipment to adjust writing mode in time according to the blocked state in purpose library, makes
Obtaining more steadily to be written purpose library for target data.
Fig. 6 is the flow diagram of data processing method in another embodiment, and the present embodiment is related to computer equipment reading
A kind of mode of target data is taken, on the basis of the above embodiments, as shown in fig. 6, above-mentioned S102 includes:
S501, the data that data source is parsed by reading plug-in unit store log；Recording in data storage log has to number of targets
According to operation note.
Wherein, after the data change in data source, log can be stored with automatically generated data, to record to target data
Change process and change result.For example, above-mentioned data storage log can be binlog file, above-mentioned binlog file be can be
The file of one binary format, the SQL statement information that data source is updated for recording user, such as change Data source table and
The SQL statement of change content can all be recorded in binlog, but not will record to the inquiry of the contents such as library table.Above-mentioned binlog
File can be row mode, can recorde by the total data of modification data, even if only having updated a field, binlog file
In also will record the data of all fields；It can also be statement pattern, record is to execute sentence, does not need to record every a line
Variation, reduces binlog file journalization amount；Furthermore it is also possible to be mixed mode, it is not limited here.
S502, the data manipulation mode that target data is determined according to operation note；Data manipulation mode includes data definition
Language DDL and data manipulation language DML.
Computer equipment can store log according to data to determine mesh to be synchronized after parsing data storage log
The data manipulation mode of data is marked, above-mentioned data manipulation mode may include Data Definition Language DDL and data manipulation language
DML, when being DDL for data manipulation mode, which can be list structured data, can be operated using DDL, such as create
Table structure etc. is built and operates to change target data；When being DML for data manipulation mode, which can be adopted
DML operation is waited to change with " selection ", " insertion ", " update " and " deletion ".
If the data manipulation mode of S503, target data is DDL, the corresponding DDL behaviour of target data is read by reading plug-in unit
Make.
Computer equipment is determining that the data manipulation mode of target data is DDL according to data storage log, can pass through
It reads plug-in unit and reads the corresponding DDL operation of target data, without reading target data.
Further, read DDL operation after, computer equipment can by write plug-in unit by data source to number of targets
According to DDL operation, be converted into the target DDL of purpose storehouse matching operate, and using target DDL operate to the data in purpose library
Carry out data change.For example, computer equipment determines in data source according to data storage log has carried out increase a line to table
Operation, can by the DDL for increasing a line operate read out, then by DDL operation convert as corresponding operation in purpose library
Mode, and modify in purpose library to corresponding table.By above-mentioned DDL from adaptation method, computer equipment is to table
When structured data synchronizes, does not need staff and safeguard table structure by hand in purpose library, but can be by from turn
DDL after changing operates the synchronous target data in purpose library.
If the data manipulation mode of S504, target data is DML, target data is read by reading plug-in unit.
Computer equipment is determining that the data manipulation mode of target data is DML according to data storage log, can pass through
It reads plug-in unit and reads target data, then target data is written in purpose library by writing plug-in unit.
Above-mentioned data processing method, computer equipment store the data manipulation mode that log judges target data by data
It is DDL or DML, and in the case where data manipulation mode is DDL, only reads corresponding DDL and operate and converted, lead to
DDL after crossing conversion, which is operated, carries out the synchronization that target data is realized in operation to data in purpose library.By the above method in number
It in the case where being DDL according to operation mode, does not need to be read and writen target data, further improves what data synchronized
Speed promotes the synchronous real-time of data.
It should be understood that although each step in the flow chart of Fig. 2-6 is successively shown according to the instruction of arrow,
These steps are not that the inevitable sequence according to arrow instruction successively executes.Unless expressly stating otherwise herein, these steps
Execution there is no stringent sequences to limit, these steps can execute in other order.Moreover, at least one in Fig. 2-6
Part steps may include that perhaps these sub-steps of multiple stages or stage are not necessarily in synchronization to multiple sub-steps
Completion is executed, but can be executed at different times, the execution sequence in these sub-steps or stage is also not necessarily successively
It carries out, but can be at least part of the sub-step or stage of other steps or other steps in turn or alternately
It executes.
In one embodiment, as shown in fig. 7, providing a kind of data processing equipment, comprising: matching module 10 calls
Module 20, determining module 30 and synchronization module 40, in which:
Matching module 10, for have in detecting data source data it is to be synchronized when, according to preset configuration relation, determine
With the matched reading plug-in unit of data source；In configuration relation include data source and read plug-in unit corresponding relationship and purpose library with write it is slotting
The corresponding relationship of part；
Calling module 20 reads plug-in unit for calling, by reading the target data in plug-in unit reading data source；
Determining module 30, for determining the corresponding purpose library of target data；
Synchronization module 40 writes plug-in unit with purpose storehouse matching for calling, by writing plug-in unit target data is synchronized to mesh
Library.
Above method embodiment, realization principle and skill may be implemented in data processing equipment provided by the embodiments of the present application
Art effect is similar, and details are not described herein.
It in one embodiment, on the basis of the above embodiments, further include to be extracted in data source in configuration relation
Target data data type and data type and purpose library corresponding relationship；Above-mentioned calling module 20 is specifically used for: base
In the data type of target data, by reading the target data in plug-in unit reading data source；Above-mentioned synchronization module 40 is specifically used for:
Data type based on target data determines purpose corresponding with target data library.
In one embodiment, on the basis of the above embodiments, as shown in figure 8, above-mentioned apparatus further includes cache module
50, it is used for: target data is cached by first order memory cache；It, will be in the first order according to the blocked state in purpose library
Deposit the target data write-in second level memory cache or file cache cached in caching；Above-mentioned synchronization module 40 is specifically used for:
By writing plug-in unit, the target data cached in second level memory cache or file cache is synchronized to purpose library.
In one embodiment, on the basis of the above embodiments, as shown in figure 9, above-mentioned cache module 50 includes:
Cache unit 501, for caching the target data cached in first order memory cache to message publishing system；
Writing unit 502 obtains target data and is write from message publishing system for the blocked state according to purpose library
Enter second level memory cache or file cache.
In one embodiment, on the basis of the above embodiments, above-mentioned writing unit 502 is specifically used for: obtaining purpose
The overflow value in library, overflow value are used to characterize the blocked state in purpose library；When overflow value is less than or equal to preset spilling threshold value,
Second level memory cache is written into the target data cached in message publishing system；Overflow threshold value by second level memory cache can
Buffer data size determines；For when overflow value is greater than and overflows threshold value, file to be written in the target cached in message publishing system
Caching.
In one embodiment, on the basis of the above embodiments, above-mentioned writing unit 502 is specifically used for reaching in data
When the memory cache of the second level, whether detection second level memory cache overflows；If so, overflow value adds 1；If it is not, then overflowing
Value is reset.
In one embodiment, on the basis of the above embodiments, above-mentioned cache module 50 is specifically used for: in the second level
When depositing the spatial cache in caching or file cache less than pre-set space threshold value, target data is synchronized to purpose library；Alternatively, pressing
According to preset caching period, the target data cached in second level memory cache in caching period or file cache is synchronized to
Purpose library.
In one embodiment, on the basis of the above embodiments, as shown in Figure 10, above-mentioned calling module 20 includes:
Resolution unit 201, the data for parsing data source by reading plug-in unit store log；It is recorded in data storage log
There is the operation note to target data；
Determination unit 202, for determining the data manipulation mode of target data according to operation note；Data manipulation mode packet
Include Data Definition Language DDL and data manipulation language DML；
First reading unit 203, for the data manipulation mode in target data be DDL when, pass through read plug-in unit read mesh
Mark the corresponding DDL operation of data；
Second reading unit 204, for the data manipulation mode in target data be DML when, pass through read plug-in unit read mesh
Mark data.
Above-mentioned synchronization module 40 is specifically used for: when the data manipulation mode of target data is DDL, will be counted by writing plug-in unit
The DDL of target data is operated according in source, is converted into operating with the target DDL of purpose storehouse matching, and using target DDL operation pair
Data in purpose library carry out data change；When the data manipulation mode of target data is DML, by writing plug-in unit for number of targets
According to write-in purpose library.
Above method embodiment, realization principle and skill may be implemented in data processing equipment provided by the embodiments of the present application
Art effect is similar, and details are not described herein.
Specific about data processing equipment limits the restriction that may refer to above for data processing method, herein not
It repeats again.Modules in above-mentioned data processing equipment can be realized fully or partially through software, hardware and combinations thereof.On
Stating each module can be embedded in the form of hardware or independently of in the processor in computer equipment, can also store in a software form
In memory in computer equipment, the corresponding operation of the above modules is executed in order to which processor calls.
In one embodiment, a kind of computer equipment is provided, which can be server, internal junction
Composition can be as shown in figure 11.The computer equipment include by system bus connect processor, memory, network interface and
Database.Wherein, the processor of the computer equipment is for providing calculating and control ability.The memory packet of the computer equipment
Include non-volatile memory medium, built-in storage.The non-volatile memory medium is stored with operating system, computer program and data
Library.The built-in storage provides environment for the operation of operating system and computer program in non-volatile memory medium.The calculating
The database of machine equipment handles data for storing data.The network interface of the computer equipment is used to pass through with external terminal
Network connection communication.To realize a kind of data processing method when the computer program is executed by processor.
It will be understood by those skilled in the art that structure shown in Figure 11, only part relevant to application scheme
The block diagram of structure, does not constitute the restriction for the computer equipment being applied thereon to application scheme, and specific computer is set
Standby may include perhaps combining certain components or with different component layouts than more or fewer components as shown in the figure.
In one embodiment, a kind of computer equipment, including memory and processor are provided, is stored in memory
Computer program, the processor perform the steps of when executing computer program
Have in detecting data source data it is to be synchronized when, according to preset configuration relation, determination is matched with data source
Read plug-in unit；It include data source and the corresponding relationship and purpose library for reading plug-in unit and the corresponding relationship for writing plug-in unit in configuration relation；
It calls and reads plug-in unit, by reading the target data in plug-in unit reading data source；
Determine the corresponding purpose library of target data；
Calling writes plug-in unit with purpose storehouse matching, and target data is synchronized to purpose library by writing plug-in unit.
It in one embodiment, further include the data type of target data to be extracted in data source in configuration relation,
And the corresponding relationship of data type and purpose library；Processor also performs the steps of when executing computer program based on target
The data type of data, by reading the target data in plug-in unit reading data source；Data type based on target data, determine with
The corresponding purpose library of target data.
In one embodiment, it is also performed the steps of when processor executes computer program by target data by the
Level-one memory cache is cached；According to the blocked state in purpose library, the target data cached in first order memory cache is write
Enter second level memory cache or file cache；By writing plug-in unit, by what is cached in second level memory cache or file cache
Target data is synchronized to purpose library.
In one embodiment, it also performs the steps of when processor executes computer program by first order memory cache
The target data of middle caching is cached to message publishing system；According to the blocked state in purpose library, obtained from message publishing system
Simultaneously second level memory cache or file cache is written in target data.
In one embodiment, the spilling for obtaining purpose library is also performed the steps of when processor executes computer program
Value, overflow value are used to characterize the blocked state in purpose library；When overflow value is less than or equal to preset spilling threshold value, message is sent out
The target data write-in second level memory cache cached in distribution system；Overflow threshold value by second level memory cache can be data cached
Amount determines；When overflow value is greater than and overflows threshold value, file cache is written into the target data cached in message publishing system.
In one embodiment, it is also performed the steps of when processor executes computer program when data reach the second level
When memory cache, whether detection second level memory cache overflows；If so, overflow value adds 1；If it is not, then overflow value is reset.
In one embodiment, it also performs the steps of when processor executes computer program in second level memory cache
Or the spatial cache in file cache be less than pre-set space threshold value when, target data is synchronized to purpose library；Alternatively, according to default
Caching period, the target data cached in second level memory cache in caching period or file cache is synchronized to purpose
Library.
In one embodiment, it is also performed the steps of when processor executes computer program and parses number by reading plug-in unit
Log is stored according to the data in source；Record has the operation note to target data in data storage log；It is determined according to operation note
The data manipulation mode of target data；Data manipulation mode includes Data Definition Language DDL and data manipulation language DML；If mesh
The data manipulation mode for marking data is DDL, reads the corresponding DDL operation of target data by reading plug-in unit；If the number of target data
It is DML according to operation mode, reads target data by reading plug-in unit；If the data manipulation mode of target data is DDL, by writing
Plug-in unit will operate the DDL of target data in data source, be converted into operating with the target DDL of purpose storehouse matching, and use target
DDL operation carries out data change to the data in purpose library；If the data manipulation mode of target data is DML, slotting by writing
Purpose library is written in target data by part.
Computer equipment provided in this embodiment, implementing principle and technical effect are similar with above method embodiment, In
This is repeated no more.
In one embodiment, a kind of computer readable storage medium is provided, computer program is stored thereon with, is calculated
Machine program performs the steps of when being executed by processor
Have in detecting data source data it is to be synchronized when, according to preset configuration relation, determination is matched with data source
Read plug-in unit；It include data source and the corresponding relationship and purpose library for reading plug-in unit and the corresponding relationship for writing plug-in unit in configuration relation；
It calls and reads plug-in unit, by reading the target data in plug-in unit reading data source；
Determine the corresponding purpose library of target data；
Calling writes plug-in unit with purpose storehouse matching, and target data is synchronized to purpose library by writing plug-in unit.
It in one embodiment, further include the data type of target data to be extracted in data source in configuration relation,
And the corresponding relationship of data type and purpose library；It also performs the steps of when computer program is executed by processor based on mesh
The data type for marking data, by reading the target data in plug-in unit reading data source；Data type based on target data determines
Purpose corresponding with target data library.
In one embodiment, it is also performed the steps of when computer program is executed by processor and passes through target data
First order memory cache is cached；According to the blocked state in purpose library, the target data that will be cached in first order memory cache
Second level memory cache or file cache is written；By writing plug-in unit, will be cached in second level memory cache or file cache
Target data be synchronized to purpose library.
In one embodiment, it is also performed the steps of when computer program is executed by processor and delays first order memory
The target data for depositing middle caching is cached to message publishing system；According to the blocked state in purpose library, obtained from message publishing system
It takes target data and second level memory cache or file cache is written.
In one embodiment, it is also performed the steps of when computer program is executed by processor and obtains overflowing for purpose library
It is worth out, overflow value is used to characterize the blocked state in purpose library；When overflow value is less than or equal to preset spilling threshold value, by message
The target data write-in second level memory cache cached in delivery system；It overflows threshold value and number is cached by second level memory cache
It is determined according to amount；When overflow value is greater than and overflows threshold value, file cache is written into the target data cached in message publishing system.
In one embodiment, it is also performed the steps of when computer program is executed by processor when data reach second
When grade memory cache, whether detection second level memory cache overflows；If so, overflow value adds 1；If it is not, then overflow value is clear
Zero.
In one embodiment, it is also performed the steps of when computer program is executed by processor slow in second level memory
Deposit or file cache in spatial cache be less than pre-set space threshold value when, target data is synchronized to purpose library；Alternatively, according to pre-
If caching period, data cached be synchronized to purpose for what is cached in second level memory cache in caching period or file cache
Library.
In one embodiment, it is also performed the steps of when computer program is executed by processor by reading plug-in unit parsing
The data of data source store log；Record has the operation note to target data in data storage log；It is true according to operation note
The data manipulation mode for the data that set the goal；Data manipulation mode includes Data Definition Language DDL and data manipulation language DML；If
The data manipulation mode of target data is DDL, reads the corresponding DDL operation of target data by reading plug-in unit；If target data
Data manipulation mode is DML, reads target data by reading plug-in unit；If the data manipulation mode of target data is DDL, pass through
The DDL of target data will be operated in data source by writing plug-in unit, be converted into operating with the target DDL of purpose storehouse matching, and use mesh
It marks DDL operation and data change is carried out to the data in purpose library；If the data manipulation mode of target data is DML, by writing
Purpose library is written in target data by plug-in unit.
Computer readable storage medium provided in this embodiment, implementing principle and technical effect and above method embodiment
Similar, details are not described herein.
Those of ordinary skill in the art will appreciate that realizing all or part of the process in above-described embodiment method, being can be with
Relevant hardware is instructed to complete by computer program, the computer program can be stored in a non-volatile computer
In read/write memory medium, the computer program is when being executed, it may include such as the process of the embodiment of above-mentioned each method.Wherein,
To any reference of memory, storage, database or other media used in each embodiment provided herein,
Including non-volatile and/or volatile memory.Nonvolatile memory may include read-only memory (ROM), programming ROM
(PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM) or flash memory.Volatile memory may include
Random access memory (RAM) or external cache.By way of illustration and not limitation, RAM is available in many forms,
Such as static state RAM (SRAM), dynamic ram (DRAM), synchronous dram (SDRAM), double data rate sdram (DDRSDRAM), enhancing
Type SDRAM (ESDRAM), synchronization link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM
(RDRAM), direct memory bus dynamic ram (DRDRAM) and memory bus dynamic ram (RDRAM) etc..
Each technical characteristic of above embodiments can be combined arbitrarily, for simplicity of description, not to above-described embodiment
In each technical characteristic it is all possible combination be all described, as long as however, the combination of these technical characteristics be not present lance
Shield all should be considered as described in this specification.
The several embodiments of the application above described embodiment only expresses, the description thereof is more specific and detailed, but simultaneously
It cannot therefore be construed as limiting the scope of the patent.It should be pointed out that coming for those of ordinary skill in the art
It says, without departing from the concept of this application, various modifications and improvements can be made, these belong to the protection of the application
Range.Therefore, the scope of protection shall be subject to the appended claims for the application patent.
Claims (10)
1. a kind of data processing method, which is characterized in that the described method includes:
Have in detecting data source data it is to be synchronized when, according to preset configuration relation, it is determining with data source is matched reads to insert
Part；It include data source and the corresponding relationship and purpose library for reading plug-in unit and the corresponding relationship for writing plug-in unit in the configuration relation；
The reading plug-in unit is called, the target data in the data source is read by the reading plug-in unit；
Determine the corresponding purpose library of the target data；
It calls and writes plug-in unit with the purpose storehouse matching, write plug-in unit by described the target data is synchronized to the purpose
Library.
2. the method according to claim 1, wherein further include in the configuration relation in the data source to
The data type and data type of the target data of extraction and the corresponding relationship in purpose library；
It is described that target data in the data source is read by the reading plug-in unit, comprising:
Based on the data type of the target data, the target data in the data source is read by the reading plug-in unit；
The corresponding purpose library of the determination target data, comprising:
Data type based on target data determines purpose corresponding with target data library.
3. according to the method described in claim 2, it is characterized in that, the method also includes:
The target data is cached by first order memory cache；
According to the blocked state in the purpose library, the target data cached in the first order memory cache is written in the second level
Deposit caching or file cache；
It is described to write plug-in unit by described the target data is synchronized to the purpose library, comprising:
Plug-in unit is write by described, the target data cached in the second level memory cache or the file cache is synchronized to
The purpose library.
4. according to the method described in claim 3, it is characterized in that, the target that will be cached in the first order memory cache
Second level memory cache or file cache is written in data, comprising:
The target data cached in the first order memory cache is cached to message publishing system；
According to the blocked state in the purpose library, target data is obtained from the message publishing system and the second level is written
Memory cache or the file cache.
5. according to the method described in claim 4, it is characterized in that, the blocked state according to the purpose library, from described
Target data is obtained in message publishing system and the second level memory cache or the file cache is written, comprising:
The overflow value in purpose library is obtained, the overflow value is used to characterize the blocked state in the purpose library；
When the overflow value is less than or equal to preset spilling threshold value, the target data that will be cached in the message publishing system
The second level memory cache is written；It is described overflow threshold value by the second level memory cache can buffer data size determine；
When the overflow value is greater than the spilling threshold value, the file is written into the target cached in the message publishing system
Caching.
6. according to the method described in claim 5, it is characterized in that, the overflow value for obtaining purpose library, comprising:
When data reach the second level memory cache, detect whether the second level memory cache overflows；
If so, the overflow value adds 1；
If it is not, then the overflow value is reset.
7. according to the method described in claim 3, it is characterized in that, by the second level memory cache or the file cache
In target data be synchronized to the purpose library, comprising:
When spatial cache in the second level memory cache or the file cache is less than pre-set space threshold value, by the mesh
Mark data are synchronized to the purpose library；
Alternatively,
According to preset caching period, will delay in second level memory cache described in the caching period or the file cache
The target data deposited is synchronized to the purpose library.
8. the method according to claim 1, wherein described read in the data source by the reading plug-in unit
Target data, comprising:
Log is stored by the data that the reading plug-in unit parses data source；Recording in the data storage log has to target data
Operation note；
The data manipulation mode of the target data is determined according to the operation note；The data manipulation mode includes that data are fixed
Adopted language DDL and data manipulation language DML；
If the data manipulation mode of the target data is DDL, the corresponding DDL of the target data is read by the reading plug-in unit
Operation；
If the data manipulation mode of the target data is DML, the target data is read by the reading plug-in unit；
It is described to write plug-in unit by described the target data is synchronized to the purpose library, comprising:
If the data manipulation mode of the target data is DDL, by it is described write plug-in unit by the data source to number of targets
According to DDL operation, be converted into the target DDL of the purpose storehouse matching operate, and using the target DDL operate to the mesh
Library in data carry out data change；
It, will be described in target data write-in by the plug-in unit of writing if the data manipulation mode of the target data is DML
Purpose library.
9. a kind of data processing equipment, which is characterized in that described device includes:
Matching module, for have in detecting data source data it is to be synchronized when, according to preset configuration relation, determining and data
The matched reading plug-in unit in source；Include data source in the configuration relation and reads the corresponding relationship of plug-in unit and purpose library and write plug-in unit
Corresponding relationship；
Calling module reads the target data in the data source by the reading plug-in unit for calling the reading plug-in unit；
Determining module, for determining the corresponding purpose library of the target data；
Synchronization module writes plug-in unit with the purpose storehouse matching for calling, and to write plug-in unit same by the target data by described
It walks to the purpose library.
10. a kind of computer equipment, including memory and processor, the memory are stored with computer program, feature exists
In the step of processor realizes any one of claims 1 to 8 the method when executing the computer program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910666341.5A CN110502540A (en) | 2019-07-23 | 2019-07-23 | Data processing method, device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910666341.5A CN110502540A (en) | 2019-07-23 | 2019-07-23 | Data processing method, device, computer equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110502540A true CN110502540A (en) | 2019-11-26 |
Family
ID=68586733
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910666341.5A Pending CN110502540A (en) | 2019-07-23 | 2019-07-23 | Data processing method, device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110502540A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111078697A (en) * | 2019-12-11 | 2020-04-28 | 秒针信息技术有限公司 | Data storage method and device, storage medium and electronic equipment |
| CN111159264A (en) * | 2019-11-28 | 2020-05-15 | 京东数字科技控股有限公司 | Data processing method and device |
| CN113411271A (en) * | 2021-06-16 | 2021-09-17 | 阿波罗智联(北京)科技有限公司 | Data access method, device, equipment and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070038642A1 (en) * | 2004-09-15 | 2007-02-15 | Scott Durgin | Method for providing extensible software components within a distributed synchronization system |
| CN103399856A (en) * | 2013-07-01 | 2013-11-20 | 北京科东电力控制系统有限责任公司 | Explosive type data caching and processing system for SCADA system and method thereof |
| CN106557486A (en) * | 2015-09-25 | 2017-04-05 | 阿里巴巴集团控股有限公司 | A kind of storage method and device of data |
| US20180081956A1 (en) * | 2013-11-04 | 2018-03-22 | Guangdong Electronics Industry Institute Ltd. | Method for automatically synchronizing multi-source heterogeneous data resources |
| CN109189785A (en) * | 2018-08-10 | 2019-01-11 | 平安科技（深圳）有限公司 | Date storage method, device, computer equipment and storage medium |
| CN109992595A (en) * | 2019-04-11 | 2019-07-09 | 北京启迪区块链科技发展有限公司 | Different database conversion method, apparatus, equipment and storage medium |
-
2019
- 2019-07-23 CN CN201910666341.5A patent/CN110502540A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070038642A1 (en) * | 2004-09-15 | 2007-02-15 | Scott Durgin | Method for providing extensible software components within a distributed synchronization system |
| CN103399856A (en) * | 2013-07-01 | 2013-11-20 | 北京科东电力控制系统有限责任公司 | Explosive type data caching and processing system for SCADA system and method thereof |
| US20180081956A1 (en) * | 2013-11-04 | 2018-03-22 | Guangdong Electronics Industry Institute Ltd. | Method for automatically synchronizing multi-source heterogeneous data resources |
| CN106557486A (en) * | 2015-09-25 | 2017-04-05 | 阿里巴巴集团控股有限公司 | A kind of storage method and device of data |
| CN109189785A (en) * | 2018-08-10 | 2019-01-11 | 平安科技（深圳）有限公司 | Date storage method, device, computer equipment and storage medium |
| CN109992595A (en) * | 2019-04-11 | 2019-07-09 | 北京启迪区块链科技发展有限公司 | Different database conversion method, apparatus, equipment and storage medium |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111159264A (en) * | 2019-11-28 | 2020-05-15 | 京东数字科技控股有限公司 | Data processing method and device |
| CN111078697A (en) * | 2019-12-11 | 2020-04-28 | 秒针信息技术有限公司 | Data storage method and device, storage medium and electronic equipment |
| CN111078697B (en) * | 2019-12-11 | 2023-11-03 | 秒针信息技术有限公司 | Data storage method and device, storage medium and electronic equipment |
| CN113411271A (en) * | 2021-06-16 | 2021-09-17 | 阿波罗智联(北京)科技有限公司 | Data access method, device, equipment and storage medium |
| US11816031B2 (en) | 2021-06-16 | 2023-11-14 | Apollo Intelligent Connectivity (Beijing) Technology Co., Ltd. | Method and apparatus for ingesting data, device and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| KR102392944B1 (en) | Data backup methods, storage media and computing devices | |
| CN107766575B (en) | Read-write separation database access method and device | |
| EP3238106B1 (en) | Compaction policy | |
| CN110362632B (en) | Data synchronization method, device, equipment and computer readable storage medium | |
| CN102542071B (en) | Distributed data processing system and method | |
| CN110502540A (en) | Data processing method, device, computer equipment and storage medium | |
| CN110751275B (en) | Graph training system, data access method and device, electronic device and storage medium | |
| CN107491523B (en) | Method and device for storing data object | |
| CN105989076A (en) | Data statistical method and device | |
| CN107169083A (en) | Public security bayonet socket magnanimity vehicle data storage and retrieval method and device, electronic equipment | |
| CN107943841A (en) | Stream data processing method, system and computer-readable recording medium | |
| US20180329921A1 (en) | Method and apparatus for storing time series data | |
| CN109271420A (en) | Information-pushing method, device, computer equipment and storage medium | |
| US10255234B2 (en) | Method for storing data elements in a database | |
| US8898677B2 (en) | Data arrangement calculating system, data arrangement calculating method, master unit and data arranging method | |
| CN109542907A (en) | Database caches construction method, device, computer equipment and storage medium | |
| CN105956123A (en) | Local updating software-based data processing method and apparatus | |
| CN101635734A (en) | Method and device for storing and managing downloaded data on non-volatile storage medium | |
| CN109815425A (en) | Caching data processing method, device, computer equipment and storage medium | |
| CA3151383A1 (en) | Method of and device for enquiring, by pagination, sub-tables divided from database, and computer equipment | |
| CN110083579A (en) | Incremental data synchronous method, apparatus, computer equipment and computer storage medium | |
| CN109783788A (en) | Tables of data complementing method, device, computer equipment and storage medium | |
| CN104765792B (en) | A kind of method, apparatus and system of dimension data storage | |
| CN110069457A (en) | One kind is based on scratch file delet method and system in distributed file storage system | |
| CN111966632B (en) | Streaming data storage method, reading method, equipment and readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20191126 |
|
| RJ01 | Rejection of invention patent application after publication |