CN108897876A - A kind of data cut-in method and device - Google Patents
A kind of data cut-in method and device Download PDFInfo
- Publication number
- CN108897876A CN108897876A CN201810718790.5A CN201810718790A CN108897876A CN 108897876 A CN108897876 A CN 108897876A CN 201810718790 A CN201810718790 A CN 201810718790A CN 108897876 A CN108897876 A CN 108897876A
- Authority
- CN
- China
- Prior art keywords
- task execution
- thread
- data
- source
- data source
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application provides a kind of data cut-in method and device, and this method includes that task schedule thread obtains execution cycle time；Task schedule thread carries out data scanning to memory database according to execution cycle time；If pending mission bit stream is arrived in scanning, is calculated according to the available resources of current server, the maximum real time resources occupancy for executing resource and current server of each task execution thread, obtain the actual quantity of required task execution thread；If actual quantity is greater than 1, the data in source data source are concurrently read；The data of reading are summarized and by the data transmissions summarized to target data source by convergence thread.Corresponding access way can be arranged according to the type of source data source and target data source by user in the application, be applicable in different business scenarios, coding is write and modified without user, reduce the workload of user.And the quantity of available thread is optimized, the efficiency of data access is improved.
Description
Technical field
This application involves technical field of data processing more particularly to a kind of data cut-in methods and device.
Background technique
In recent years, the application of computer system has been deep into all trades and professions, and in internet industry, enterprise exists simultaneously more
Kind computer application is to inside and outside offer service, and each application has respective data storage method, in order to guarantee different storages
The data consistency of system part, it is often necessary to it is synchronous that data are carried out between different storage systems.
Currently, existing data access synchronous method is staff using inside some particular source of timer poll
Whether (for example, table and table between) in database or each data source (for example, between types of databases) have data update,
To realize that its data is synchronous.But the execution program that data synchronize usually carries out hard coded by user to realize, i.e., data are synchronous
Variant variables in the process are replaced with fixed value；If user need to change data synchronize in execution parameter, such as timer
Duration etc. then needs to modify its corresponding variant variables replaced by fixed value, increases so as to cause the workload of user.And
Since different types of data source has different data access, the method for above-mentioned hard coded can not support Various types of data source
Access synchronous working, user need to modify the corresponding coding of data access, lead to user according to the type of actual data source
The increase of workload.
Summary of the invention
The application provides a kind of data cut-in method and device, to solve in existing data access synchronous method, data
Synchronous execution program usually carries out hard coded by user and realizes, if user need to change data synchronize in execution parameter,
It then needs to modify its corresponding variant variables replaced by fixed value, increases so as to cause the workload of user.And due to not
Same type data source has different data access, and the method for above-mentioned hard coded can not support the access in Various types of data source same
Work is walked, user need to modify the corresponding coding of data access, lead to amount of user effort according to the type of actual data source
Increase the problem of.
In a first aspect, the application provides a kind of data cut-in method, including：
Task schedule thread obtains execution cycle time；
The task schedule thread carries out data scanning to memory database according to the execution cycle time, described in judgement
Whether pending mission bit stream is had in memory database, and the pending mission bit stream includes type of the user according to source data source
The second access way information, the source data that the first access way information for being arranged, user are arranged according to the type of target data source
The basic information in source and the basic information of target data source；
If the task schedule thread scans to pending mission bit stream, according to the available resources of current server,
The maximum execution resource of each task execution thread and the real time resources occupancy of current server are calculated, and are handled
The actual quantity of task execution thread needed for pending mission bit stream；
Judge whether the actual quantity of the task execution thread is greater than 1, if the actual number of the task execution thread
Amount is greater than 1, then multiple task execution threads are according to the basic information of the first access way information and source data source, and
Hair reads the data in the source data source；
Convergence thread summarizes the data that the task execution thread is read；
Basic information of the convergence thread according to the second access way information and target data source, the number that will summarize
According to being sent to the target data source.
Second aspect, the application provide a kind of data access device, including：
Module is obtained, obtains execution cycle time for task schedule thread；
Scan module carries out data to memory database according to the execution cycle time for the task schedule thread
Scanning, judges whether there is pending mission bit stream in the memory database, the pending mission bit stream include user according to
The second access side that first access way information of the type setting in source data source, user are arranged according to the type of target data source
The basic information of formula information, the basic information in source data source and target data source；
First judgment module, if for the task schedule thread scans to pending mission bit stream, according to current
The available resources of server, the maximum real time resources occupancy for executing resource and current server of each task execution thread
It is calculated, obtains the actual quantity of task execution thread needed for handling pending mission bit stream；
Second judgment module, for judging whether the actual quantity of the task execution thread is greater than 1, if the task
The actual quantity of execution thread is greater than 1, then multiple task execution threads are according to the first access way information and source number
According to the basic information in source, the data in the source data source are concurrently read；
Summarizing module summarizes the data that the task execution thread is read for converging thread；
Delivery module is believed for the convergence thread according to the basis of the second access way information and target data source
Breath, by the data transmission summarized to the target data source.
From the above technical scheme, the application provides a kind of data cut-in method and device, can be by user according to source number
Corresponding access way is set according to the type of source and target data source, is applicable in different business scenarios, have higher transplantability,
Platform scalability and high reusability, and coding is write and modifies without user, reduce the workload of user.And processing is connect
The quantity for entering the available thread of task optimizes, and improves the efficiency of data access.
Detailed description of the invention
In order to illustrate more clearly of the technical solution of the application, letter will be made to attached drawing needed in the embodiment below
Singly introduce, it should be apparent that, for those of ordinary skills, without any creative labor,
It is also possible to obtain other drawings based on these drawings.
Fig. 1 is a kind of flow chart for data cut-in method that one embodiment of the application provides；
Fig. 2 is a kind of flow chart for data cut-in method that another embodiment of the application provides；
Fig. 3 is a kind of structural schematic diagram of data access device provided by the present application；
Fig. 4 is the structural schematic diagram of first judgment module；
Fig. 5 is the structural schematic diagram of the second judgment module；
Fig. 6 is the structural schematic diagram of delivery module.
Specific embodiment
Include the following steps referring to Fig. 1 in a first aspect, embodiments herein provides a kind of data cut-in method：
Step S101：Task schedule thread obtains execution cycle time.
Execution cycle time can be by user according to real data source renewal time self-setting, for example, user can set
Execution cycle time is 20 days 12 April in 2018:00, so that task schedule thread starts to scan in the time.
Step S102：The task schedule thread carries out data to memory database according to the execution cycle time and sweeps
It retouches, judges whether there is pending mission bit stream in the memory database, the pending mission bit stream includes user according to source
The second access way that first access way information of the type setting of data source, user are arranged according to the type of target data source
The basic information of information, the basic information in source data source and target data source.If the task schedule thread scans are to wait hold
Row mission bit stream, thens follow the steps S103.
Pending mission bit stream is stored in memory database, can be reduced the sweep time of task schedule thread, is improved
Search rate.Source data source is the data source for having carried out data change；Target data source is need to carry out data with source data source
Synchronous data source.Source data source and target data source can be include relevant database, non-relational database (NOSQL number
According to library), Excel file, one of message and restful interface or a variety of data sources.For example, in enterprise computer system
In, the SQL Server database of Microsoft is used in market department, Hbase database is used in research and development department, due to market department
The data of door acquisition can provide reference for research and development department, so the SQL Server database that can use market department is as source
Data source, and the Hbase database that research and development department uses is as target data source.
Step S103：According to the available resources of current server, each task execution thread it is maximum execute resource and
The real time resources occupancy of current server is calculated, and the reality of task execution thread needed for handling pending mission bit stream is obtained
Border quantity.
Step S104：Judge whether the actual quantity of the task execution thread is greater than 1, if the task execution thread
Actual quantity be greater than 1, then follow the steps S105-S107；If the actual quantity of the task execution thread is equal to 1, hold
Row step S106-S107.
Step S105：Multiple task execution threads are according to the basis of the first access way information and source data source
Information concurrently reads the data in the source data source.
Step S106：Convergence thread summarizes the data that the task execution thread is read.
Step S107：The convergence thread according to the basic information of the second access way information and target data source,
By the data transmission summarized to the target data source.
From the above technical scheme, the embodiment of the present application provides a kind of data cut-in method, can be by user according to source number
Corresponding access way is set according to the type of source and target data source, is applicable in different business scenarios, have higher transplantability,
Platform scalability and high reusability, and coding is write and modifies without user, reduce the workload of user.And processing is connect
The quantity for entering the available thread of task optimizes, and improves the efficiency of data access.
Referring to fig. 2, another embodiment of the application provides a kind of data cut-in method, includes the following steps：
Step S201：Task schedule thread obtains execution cycle time.
Step S202：The task schedule thread carries out data to memory database according to the execution cycle time and sweeps
It retouches, judges whether there is pending mission bit stream in the memory database, the pending mission bit stream includes user according to source
The second access way that first access way information of the type setting of data source, user are arranged according to the type of target data source
The basic information of information, the basic information in source data source and target data source.If the task schedule thread scans are to wait hold
Row mission bit stream, thens follow the steps S203.
Task schedule thread starts to carry out data scanning to memory database in execution cycle time.Memory database
It is to place the data in the database directly operated in memory, relative to disk, the reading and writing data speed of memory will be higher by several
Pending mission bit stream is stored in memory database by the order of magnitude, can reduce the sweep time of task schedule thread, and raising is looked into
Look for rate.
Source data source is the data source for having carried out data change；Target data source is need to be same with source data source progress data
The data source of step, that is, the data source for needing to be consistent with the data in source data source.Source data source and target data source
Can be includes relevant database, non-relational database (NOSQL database), Excel file, message and restful interface
One of or a variety of data sources.Relevant database is commonly divided into oracle database, musql database and sql number
According to library etc.；Non-relational database is commonly divided into mongodb database and hbase database etc.；Message can be divided into
ActiveMQ message etc..Since the type of database that source data source and target data source is included is different, access data are connect
It is also not identical to enter mode.The type of database that user can be included according to source data source and target data source is arranged its and corresponding connects
Enter mode.The basic information in source data source may include the IP in source data source, request port, library literary name section and field data types
Etc. information；The basic information of target data source may include the IP of target data source, request port, library literary name section and field data
The information such as type.
Step S203：Resource is executed according to the available resources of current server, the maximum of each task execution thread, according to
Following formula calculates, and obtains the standard number of task execution thread needed for handling pending mission bit stream；
N=E1/E2,
Wherein, the standard number of task execution thread needed for N is indicated, E1 indicate the available resources of current server, E2 table
Show that the maximum of each task execution thread executes resource.
The maximum resource that executes of each task execution thread refers to that the upper limit value of resource can be performed in a task execution thread,
For example, a task execution thread can at most grab 1000 datas, each data are 10kb, then this task execution thread
It is 1000*10kb=1M that maximum, which executes resource,.
Resource is executed divided by the maximum of each task execution thread using the available resources of current server, so that it may be obtained everywhere
The standard number of task execution thread needed for managing pending mission bit stream, for example, the available resources of current server are 100M, often
A maximum resource that executes for executing mission thread is 1M, the then criterion numeral of task execution thread needed for handling pending mission bit stream
Amount is 100.
Step S204：If the Current resource occupancy of current server is greater than preset threshold, and the task execution line
The standard number of journey is greater than 1, then reduces the standard number of the task execution thread, until meeting preset condition, is handled
The actual quantity of task execution thread needed for pending mission bit stream, the preset condition are the real-time money of the current server
Source occupancy is less than or equal to preset threshold, and the volume residual of task execution thread needed for the pending mission bit stream of processing is greater than
Or it is equal to 1, alternatively, the real time resources occupancy of the current server is greater than preset threshold, and handle pending mission bit stream
The volume residual of required task execution thread is 1.
The preset threshold of the Current resource occupancy of current server such as preset threshold can be arranged by user's sets itself
It is 80 percent.If the Current resource occupancy of current server is greater than preset threshold, and has a plurality of task execution thread
Data access task is executed, then needs to reduce the item number of task execution thread, is accounted for avoid the Current resource in current server
When larger with rate, and the item number of task execution thread is more, influences the arithmetic speed of server and the feelings of loss of data occur
Condition occurs, and guarantees that current server is constantly in good operating status.
The item number for reducing task execution thread, will meet preset condition, i.e., the real time resources of the described current server occupy
Rate is less than or equal to preset threshold, and the volume residual of task execution thread needed for the pending mission bit stream of processing is greater than or equal to
1, alternatively, the real time resources occupancy of the current server is greater than preset threshold, and handles and appoint needed for pending mission bit stream
The volume residual for execution thread of being engaged in is 1；That is, if task execution thread standard number reduce to a certain extent, and
At at least one, the real time resources occupancy of current server is less than preset threshold, then the criterion numeral of task execution thread
Amount stops reducing；If the standard number of task execution thread, which is reduced to, only remains one, and the current real-time money of current server
Source occupancy is also greater than preset threshold, then is also required to the standard number for stopping reducing task execution thread, to guarantee at least one
Item executes the achievable current data of mission thread and accesses task, prevents data access task from stopping.
Step S205：Judge that the actual quantity of the task execution thread is greater than 1, if the reality of the task execution thread
Border quantity is greater than 1, thens follow the steps S206-S209；If the actual quantity of the task execution thread is equal to 1, step is executed
Rapid S207-S209.
Step S206：Presently described task execution thread is according to the first information access way, the basis in source data source
The corresponding vernier information of initial data to be read in information and source data source, is read out the data in the source data source.
Vernier information has recorded the location information of corresponding data, the case where the actual quantity of task execution thread is greater than 1
Under, a plurality of task execution thread is successively read out the data in source data source according to default execution sequence, and initial data is
The data that first need of current task execution thread are read；The corresponding vernier information of initial data records current task execution thread
The corresponding location information of data that first need is read.
Step S207：The corresponding vernier information update of termination data after presently described task execution thread is read is to interior
Deposit data library, so that the termination data after next task execution thread is read according to presently described task execution thread are corresponding
Vernier information, the data in the source data source are continued to read.
Terminating data is the last item data that current task execution thread is read, and terminates the corresponding vernier information note of data
Record the corresponding location information of data of the last one reading of current task execution thread.Next task execution thread is according to as predecessor
The corresponding vernier information of termination data that execution thread of being engaged in is read, continuation sequence read subsequent data, it can be achieved that multiple tasks
Execution thread concurrently reads the uniqueness of data, it is ensured that the accuracy for reading the data in data source guarantees data access just
True rate.
Step S208：Convergence thread summarizes the data that the task execution thread is read.
Step S209：The convergence thread according to the basic information of the second access way information and target data source,
By the data transmission summarized to the target data source.
From the above technical scheme, the embodiment of the present application provides a kind of data cut-in method, can be by user according to source number
Corresponding access way is set according to the type of source and target data source, is applicable in different business scenarios, have higher transplantability,
Platform scalability and high reusability, and coding is write and modifies without user, reduce the workload of user.And processing is connect
The quantity for entering the available thread of task optimizes, and improves the efficiency of data access.
Further, the pending mission bit stream is also deposited to database configuration record list, and database configuration record list exists
It is stored in the storage unit of non-memory database, such as disk.
In the case where pending mission bit stream is also deposited to database configuration record list, above-described embodiment step S208 packet
It includes：
Step S2081：The convergence thread transfers the pending mission bit stream in the database configuration record list；
Step S2082：The convergence thread is according to the of the pending mission bit stream in the database configuration record list
The basic information of two access way information and target data source, by the data transmission summarized to the target data source.
It is stored in due to database configuration record list in the storage unit of non-memory database, it can be by pending mission bit stream
It is backed up, can also facilitate lookup of the user to pending mission bit stream.
Second aspect, referring to Fig. 3, another embodiment of the application provides a kind of data access device, described device packet
It includes：
Module 301 is obtained, obtains execution cycle time for task schedule thread；
Scan module 302 carries out memory database according to the execution cycle time for the task schedule thread
Data scanning judges whether there is pending mission bit stream in the memory database, and the pending mission bit stream includes user
It is connect according to the first access way information of the type in source data source setting, user according to second that the type of target data source is arranged
Enter the basic information of mode information, the basic information in source data source and target data source；
First judgment module 303, if basis is worked as the task schedule thread scans to pending mission bit stream
The available resources of preceding server, the maximum real time resources occupancy for executing resource and current server of each task execution thread
Rate is calculated, and the actual quantity of task execution thread needed for handling pending mission bit stream is obtained；
Second judgment module 304, for judging whether the actual quantity of the task execution thread is greater than 1, if described
The actual quantity of task execution thread is greater than 1, then multiple task execution threads according to the first access way information and
The basic information in source data source concurrently reads the data in the source data source；
Summarizing module 305 summarizes the data that the task execution thread is read for converging thread；
Delivery module 306, for the convergence thread according to the base of the second access way information and target data source
Plinth information, by the data transmission summarized to the target data source.
Further, referring to fig. 4, the first judgment module 303 includes：
Computing unit, 401, for being executed according to the available resources of current server, the maximum of each task execution thread
Resource calculates according to following formula, obtains the standard number of task execution thread needed for handling pending mission bit stream；
N=E1/E2,
Wherein, the standard number of task execution thread needed for N is indicated, E1 indicate the available resources of current server, E2 table
Show that the maximum of each task execution thread executes resource；
Judging unit 402, if the Current resource occupancy for current server is greater than preset threshold, and the task
The standard number of execution thread is greater than 1, then reduces the standard number of the task execution thread, until meeting preset condition, obtains
The actual quantity of the task execution thread to needed for handling pending mission bit stream, the preset condition are the current server
Real time resources occupancy is less than or equal to preset threshold, and the remainder of task execution thread needed for the pending mission bit stream of processing
Amount is greater than or equal to 1, alternatively, the real time resources occupancy of the current server is greater than preset threshold, and handles pending
The volume residual of task execution thread needed for information of being engaged in is 1.
Further, referring to Fig. 5, second judgment module 304 includes：
Reading unit 501, for presently described task execution thread according to the first information access way, source data source
Basic information and source data source in the corresponding vernier information of initial data to be read, the data in the source data source are carried out
It reads；
Updating unit 502, for the corresponding vernier information of termination data after reading presently described task execution thread
It is updated to memory database, so that next task execution thread is according to the termination after the reading of presently described task execution thread
The data in the source data source are continued to read by the corresponding vernier information of data.
Further, the pending mission bit stream is also deposited to database configuration record list.
Further, referring to Fig. 6, the delivery module 306 includes：
Unit 601 is transferred, transfers the letter of the pending task in the database configuration record list for the convergence thread
Breath；
Transmission unit 602 is believed for the convergence thread according to the pending task in the database configuration record list
The basic information of second access way information and target data source of breath, by the data transmission summarized to the target data source.
From the above technical scheme, the embodiment of the present application provides a kind of data access device, can be by user according to source number
Corresponding access way is set according to the type of source and target data source, is applicable in different business scenarios, have higher transplantability,
Platform scalability and high reusability, and coding is write and modifies without user, reduce the workload of user.And processing is connect
The quantity for entering the available thread of task optimizes, and improves the efficiency of data access.
It is required that those skilled in the art can be understood that the technology in the embodiment of the present application can add by software
The mode of general hardware platform realize.Based on this understanding, the technical solution in the embodiment of the present application substantially or
Or the part that contributes to existing technology can be embodied in the form of software products, which can deposit
Storage is in storage medium, such as ROM/RAM, magnetic disk, CD, including some instructions computer equipment to as (can be with
It is personal computer, server or the network equipment etc.) execute certain part institutes of each embodiment of the application or embodiment
The method stated.
Various embodiments are described in a progressive manner for this specification, same and similar part between each embodiment
Can cross-reference, each embodiment focuses on the differences from other embodiments, especially for device reality
For applying example, since it is substantially similar to the method embodiment, so being described relatively simple, related place is referring to embodiment of the method
Part explanation.
Claims (10)
1. a kind of data cut-in method, which is characterized in that the method includes：
Task schedule thread obtains execution cycle time；
The task schedule thread carries out data scanning to memory database according to the execution cycle time, judges the memory
Whether pending mission bit stream is had in database, and the pending mission bit stream includes that user is arranged according to the type in source data source
The first access way information, user be arranged according to the type of target data source the second access way information, source data source
The basic information of basic information and target data source；
If the task schedule thread scans are to pending mission bit stream, according to the available resources of current server, each
The maximum execution resource of task execution thread and the real time resources occupancy of current server are calculated, and obtain handling wait hold
The actual quantity of task execution thread needed for row mission bit stream；
Judge whether the actual quantity of the task execution thread is greater than 1, if the actual quantity of the task execution thread is big
In 1, then multiple task execution threads are concurrently read according to the basic information of the first access way information and source data source
Take the data in the source data source；
Convergence thread summarizes the data that the task execution thread is read；
The convergence thread passes the data summarized according to the basic information of the second access way information and target data source
It send to the target data source.
2. the method as described in claim 1, which is characterized in that the available resources according to current server, each task
The maximum execution resource of execution thread and the real time resources occupancy of current server are calculated, and obtain handling pending
The quantity of task execution thread needed for business information includes：
Resource is executed according to the available resources of current server, the maximum of each task execution thread, is calculated according to following formula,
Obtain the standard number of task execution thread needed for handling pending mission bit stream；
N=E1/E2,
Wherein, the standard number of task execution thread needed for N is indicated, E1 indicate the available resources of current server, and E2 indicates every
The maximum of a task execution thread executes resource；
If the Current resource occupancy of current server is greater than preset threshold, and the standard number of the task execution thread is big
In 1, then the standard number of the task execution thread is reduced, until meeting preset condition, obtains handling pending mission bit stream
The actual quantity of required task execution thread, the preset condition be the current server real time resources occupancy be less than or
Volume residual equal to preset threshold, and task execution thread needed for the pending mission bit stream of processing is greater than or equal to 1, alternatively,
The real time resources occupancy of the current server is greater than preset threshold, and task execution line needed for the pending mission bit stream of processing
The volume residual of journey is 1.
3. the method as described in claim 1, which is characterized in that the multiple task execution thread connects according to described first
Enter the basic information of mode information and source data source, the data concurrently read in the source data source include：
Presently described task execution thread is according to the first information access way, the basic information in source data source and source data source
The corresponding vernier information of interior initial data to be read, is read out the data in the source data source；
By the corresponding vernier information update of termination data after the reading of presently described task execution thread to memory database, so that
Next task execution thread is right according to the corresponding vernier information of termination data after the reading of presently described task execution thread
The data in the source data source continue to read.
4. the method as described in claim 1, which is characterized in that the pending mission bit stream is also deposited to database to configure and be recorded
Table.
5. method as claimed in claim 4, which is characterized in that the convergence thread according to the second access way information and
The data transmission summarized to the target data source includes by the basic information of target data source：
The convergence thread transfers the pending mission bit stream in the database configuration record list；
The thread that converges is according to the second access way information of the pending mission bit stream in the database configuration record list
With the basic information of target data source, by the data transmission summarized to the target data source.
6. a kind of data access device, which is characterized in that described device includes：
Module is obtained, obtains execution cycle time for task schedule thread；
Scan module carries out data to memory database according to the execution cycle time for the task schedule thread and sweeps
It retouches, judges whether there is pending mission bit stream in the memory database, the pending mission bit stream includes user according to source
The second access way that first access way information of the type setting of data source, user are arranged according to the type of target data source
The basic information of information, the basic information in source data source and target data source；
First judgment module, if for the task schedule thread scans to pending mission bit stream, according to current service
The maximum real time resources occupancy progress for executing resource and current server of the available resources of device, each task execution thread
It calculates, obtains the actual quantity of task execution thread needed for handling pending mission bit stream；
Second judgment module, for judging whether the actual quantity of the task execution thread is greater than 1, if the task execution
The actual quantity of thread is greater than 1, then multiple task execution threads are according to the first access way information and source data source
Basic information, concurrently read the data in the source data source；
Summarizing module summarizes the data that the task execution thread is read for converging thread；
Delivery module, for the convergence thread according to the basic information of the second access way information and target data source,
By the data transmission summarized to the target data source.
7. device as claimed in claim 6, which is characterized in that the first judgment module includes：
Computing unit, for executing resource according to the available resources of current server, the maximum of each task execution thread, according to
Following formula calculates, and obtains the standard number of task execution thread needed for handling pending mission bit stream；
N=E1/E2,
Wherein, the standard number of task execution thread needed for N is indicated, E1 indicate the available resources of current server, and E2 indicates every
The maximum of a task execution thread executes resource；
Judging unit, if the Current resource occupancy for current server is greater than preset threshold, and the task execution line
The standard number of journey is greater than 1, then reduces the standard number of the task execution thread, until meeting preset condition, is handled
The actual quantity of task execution thread needed for pending mission bit stream, the preset condition are the real-time money of the current server
Source occupancy is less than or equal to preset threshold, and the volume residual of task execution thread needed for the pending mission bit stream of processing is greater than
Or it is equal to 1, alternatively, the real time resources occupancy of the current server is greater than preset threshold, and handle pending mission bit stream
The volume residual of required task execution thread is 1.
8. device as claimed in claim 6, which is characterized in that second judgment module includes：
Reading unit, for presently described task execution thread according to the first information access way, the basis in source data source
The corresponding vernier information of initial data to be read in information and source data source, is read out the data in the source data source；
Updating unit, for the corresponding vernier information update of termination data after reading presently described task execution thread to interior
Deposit data library, so that the termination data after next task execution thread is read according to presently described task execution thread are corresponding
Vernier information, the data in the source data source are continued to read.
9. device as claimed in claim 6, which is characterized in that the pending mission bit stream is also deposited to database to configure and be recorded
Table.
10. device as claimed in claim 9, which is characterized in that the delivery module includes：
Unit is transferred, transfers the pending mission bit stream in the database configuration record list for the convergence thread；
Transmission unit, for the convergence thread according to second of the pending mission bit stream in the database configuration record list
The basic information of access way information and target data source, by the data transmission summarized to the target data source.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810718790.5A CN108897876A (en) | 2018-06-29 | 2018-06-29 | A kind of data cut-in method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810718790.5A CN108897876A (en) | 2018-06-29 | 2018-06-29 | A kind of data cut-in method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108897876A true CN108897876A (en) | 2018-11-27 |
Family
ID=64347634
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810718790.5A Pending CN108897876A (en) | 2018-06-29 | 2018-06-29 | A kind of data cut-in method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108897876A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109857774A (en) * | 2018-12-26 | 2019-06-07 | 广州海达安控智能科技有限公司 | Based on Multi-sensor Fusion deformation measurement data statistical method and device |
| CN109918187A (en) * | 2019-03-12 | 2019-06-21 | 北京同城必应科技有限公司 | Method for scheduling task, device, equipment and storage medium |
| CN110069493A (en) * | 2019-02-28 | 2019-07-30 | 平安科技（深圳）有限公司 | Data processing method, device, computer equipment and storage medium |
| CN110287018A (en) * | 2019-07-04 | 2019-09-27 | 中国工商银行股份有限公司 | Batch tasks method of combination and device |
| CN110334018A (en) * | 2019-06-18 | 2019-10-15 | 梁俊杰 | A kind of big data introduction method and relevant device |
| CN110795423A (en) * | 2019-09-23 | 2020-02-14 | 紫光云(南京)数字技术有限公司 | Data extraction method for rapid cleaning and conversion |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102915377A (en) * | 2012-11-14 | 2013-02-06 | 深圳市宏电技术股份有限公司 | Method and system for converting or synchronizing databases |
| CN103699638A (en) * | 2013-12-23 | 2014-04-02 | 国云科技股份有限公司 | Method for realizing cross-database type synchronous data based on configuration parameters |
| US8832173B2 (en) * | 2009-01-20 | 2014-09-09 | Sap Ag | System and method of multithreaded processing across multiple servers |
| CN105389312A (en) * | 2014-09-04 | 2016-03-09 | 上海福网信息科技有限公司 | Big data migration method and tool |
| CN105592314A (en) * | 2015-12-17 | 2016-05-18 | 清华大学 | Parallel decoding method and device |
| CN106933673A (en) * | 2015-12-30 | 2017-07-07 | 阿里巴巴集团控股有限公司 | Adjust the method and device of component logic number of threads |
-
2018
- 2018-06-29 CN CN201810718790.5A patent/CN108897876A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8832173B2 (en) * | 2009-01-20 | 2014-09-09 | Sap Ag | System and method of multithreaded processing across multiple servers |
| CN102915377A (en) * | 2012-11-14 | 2013-02-06 | 深圳市宏电技术股份有限公司 | Method and system for converting or synchronizing databases |
| CN103699638A (en) * | 2013-12-23 | 2014-04-02 | 国云科技股份有限公司 | Method for realizing cross-database type synchronous data based on configuration parameters |
| CN105389312A (en) * | 2014-09-04 | 2016-03-09 | 上海福网信息科技有限公司 | Big data migration method and tool |
| CN105592314A (en) * | 2015-12-17 | 2016-05-18 | 清华大学 | Parallel decoding method and device |
| CN106933673A (en) * | 2015-12-30 | 2017-07-07 | 阿里巴巴集团控股有限公司 | Adjust the method and device of component logic number of threads |
Non-Patent Citations (2)
| Title |
|---|
| 万川梅: "《MySQL数据库应用教程》", 31 July 2017 * |
| 老任物联网杂谈: "《JVM最大线程数计算方法》", 9 May 2011 * |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109857774A (en) * | 2018-12-26 | 2019-06-07 | 广州海达安控智能科技有限公司 | Based on Multi-sensor Fusion deformation measurement data statistical method and device |
| CN109857774B (en) * | 2018-12-26 | 2024-04-23 | 广州市中海达测绘仪器有限公司 | Deformation monitoring data statistics method and device based on multi-sensor fusion |
| CN110069493A (en) * | 2019-02-28 | 2019-07-30 | 平安科技（深圳）有限公司 | Data processing method, device, computer equipment and storage medium |
| CN110069493B (en) * | 2019-02-28 | 2024-05-07 | 平安科技（深圳）有限公司 | Data processing method, device, computer equipment and storage medium |
| CN109918187A (en) * | 2019-03-12 | 2019-06-21 | 北京同城必应科技有限公司 | Method for scheduling task, device, equipment and storage medium |
| CN110334018A (en) * | 2019-06-18 | 2019-10-15 | 梁俊杰 | A kind of big data introduction method and relevant device |
| CN110287018A (en) * | 2019-07-04 | 2019-09-27 | 中国工商银行股份有限公司 | Batch tasks method of combination and device |
| CN110287018B (en) * | 2019-07-04 | 2021-08-13 | 中国工商银行股份有限公司 | Batch task arranging method and device |
| CN110795423A (en) * | 2019-09-23 | 2020-02-14 | 紫光云(南京)数字技术有限公司 | Data extraction method for rapid cleaning and conversion |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108897876A (en) | A kind of data cut-in method and device | |
| US7788237B2 (en) | Method and system for tracking changes in a document | |
| US7680848B2 (en) | Reliable and scalable multi-tenant asynchronous processing | |
| KR100509794B1 (en) | Method of scheduling jobs using database management system for real-time processing | |
| CN106250226B (en) | Method for scheduling task and system based on consistency hash algorithm | |
| CN110806933B (en) | Batch task processing method, device, equipment and storage medium | |
| US9438665B1 (en) | Scheduling and tracking control plane operations for distributed storage systems | |
| CN107704597A (en) | Relevant database to Hive ETL script creation methods | |
| KR100538371B1 (en) | Method and System for Incorporating legacy applications into a distributed data processing environment | |
| CN109257399B (en) | Cloud platform application program management method, management platform and storage medium | |
| US10158709B1 (en) | Identifying data store requests for asynchronous processing | |
| CN105159604A (en) | Disk data read-write method and system | |
| CN105635311A (en) | Method for synchronizing resource pool information in cloud management platform | |
| CN104199912B (en) | A kind of method and device of task processing | |
| US10333800B2 (en) | Allocating physical nodes for processes in an execution plan | |
| CN108881485A (en) | The method for ensureing the high concurrent system response time under big data packet | |
| CN113037529B (en) | Reserved bandwidth allocation method, device, equipment and storage medium | |
| CN103780686A (en) | Method and system for customizing application approval procedure in cloud organization | |
| US20150324486A1 (en) | Grouping records in buckets distributed across nodes of a distributed database system to perform comparison of the grouped records | |
| CN106888264B (en) | A kind of method for interchanging data and device | |
| CN104298761A (en) | Implementation method for master data matching between heterogeneous software systems | |
| CN107409086B (en) | Mass data management in communication applications through multiple mailboxes | |
| CN105978744A (en) | Resource allocation method, device and system | |
| CN109842671A (en) | A kind of method and system of browser and interapplication communications of declaring dutiable goods automatically | |
| US10979303B1 (en) | Segmentation of maintenance on distributed systems |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: 230000 zone B, 19th floor, building A1, 3333 Xiyou Road, hi tech Zone, Hefei City, Anhui Province Applicant after: Dingfu Intelligent Technology Co.,Ltd. Address before: Room 630, 6th floor, Block A, Wanliu Xingui Building, 28 Wanquanzhuang Road, Haidian District, Beijing Applicant before: DINFO (BEIJING) SCIENCE DEVELOPMENT Co.,Ltd. |
|
| CB02 | Change of applicant information | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20181127 |
|
| RJ01 | Rejection of invention patent application after publication |