CN103488733B - A kind of method and device for importing data - Google Patents
A kind of method and device for importing data Download PDFInfo
- Publication number
- CN103488733B CN103488733B CN201310425562.6A CN201310425562A CN103488733B CN 103488733 B CN103488733 B CN 103488733B CN 201310425562 A CN201310425562 A CN 201310425562A CN 103488733 B CN103488733 B CN 103488733B
- Authority
- CN
- China
- Prior art keywords
- database server
- data
- server
- speed
- load
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
Abstract
The invention discloses a kind of method and device for importing data, to solve to import in the prior art the method for data may result in servers go down, import data it is less efficient the problem of.Intermediate server obtains the load state information of each database server, and determine therefrom that the speed that data are imported to each database server, when receiving the task of client addition, according to the server identification included in the task, with the speed that data are imported to the corresponding database server of the server identification determined, data are imported to the corresponding database server of the server identification.Pass through the above method, intermediate server can be according to the load of database server, data are imported to database server with corresponding speed, therefore the risk of high capacity database server paralysis is reduced, and, intermediate server can carry out data importing in real time without importing data in the specified period, and this also increases the efficiency for importing data.
Description
Technical field
The present invention relates to communication technical field, more particularly to a kind of method and device for importing data.
Background technology
At present, user can be imported data in MySQL server by client and be stored, and be used with saving user
The memory space of terminal.
Because MySQL server stores the data of many users, therefore, in order to ensure what is stored in MySQL server
The security of data, is generally kept apart client with MySQL server using intermediate server, user can not pass through client
MySQL server directly is imported data to, and to be imported data to by intermediate server in MySQL server, such as Fig. 1 institutes
Show.
Fig. 1 is imports the system structure diagram of data in the prior art, and the system shown in Fig. 1 includes client, work
For Deferred Job server clusters, data storage area and the MySQL server cluster of intermediate server cluster.Wherein,
Deferred Job server clusters as each Deferred Job servers of intermediate server by constituting, MySQL server
Cluster is made up of each MySQL server.
Based on the system shown in Fig. 1, when user will be imported data in MySQL server, number is first upload the data to
According in memory block（Because Deferred Job servers have not been used to the memory space of data storage, therefore by data storage
The data that area storage user uploads）, and task is added to Deferred Job servers by client, the task of addition is included
The server identification of the Data Identification for the data to be imported and the MySQL server to be imported into.Deferred Job are serviced
The Data Identification that device is included in the task according to addition, obtains corresponding data, and appointing according to addition from data storage area
The data of acquisition, imported into corresponding MySQL server by the server identification included in business in the specified period.
In the prior art, it is contemplated that relatively low in 2 points of loads to 6 points of each MySQL servers of every morning, it is therefore existing
There are Deferred Job servers in technology that the data of acquisition are imported into corresponding MySQL at 2 points to 6 points of every morning to take
It is engaged in device.
But, in practical application scene, do not ensure that 2 points to 6 points of the load in the morning of each MySQL server
It is very low, that is, it cannot be guaranteed that each MySQL server is suitable for carrying out data importing for 2 points to 6 points in the morning, if some
MySQL server in the morning 2 points to 6 points load it is still very high, then this period to these MySQL servers import
Data will certainly sharply increase the load of MySQL server, may result in MySQL server paralysis.In addition, 2 points in the morning
Data importing, which is carried out, to 6 points of this periods does not also ensure that all data can import MySQL in this period
Server, if still having data not yet to import MySQL server at the end of 6:00 AM, can only also wait until next 2 points of morning
It could be imported to 6 points, this reduces the efficiency for importing data.
The content of the invention
The embodiment of the present invention provides a kind of method and device for importing data, to solve to import data in the prior art
Method may result in MySQL server paralysis, and import data it is less efficient the problem of.
A kind of method for importing data provided in an embodiment of the present invention, including：
Intermediate server obtains the load state information of each database server；And
For each database server, according to the load state information of the database server got, it is determined that to
The database server import data speed, wherein, the load of the database server is bigger, it is determined that to the database take
The speed for device importing data of being engaged in is smaller；And
When receiving the task of client addition, according to the Data Identification included in the task, data storage is obtained
The corresponding data of Data Identification described in area；And
According to the server identification included in the task, to determine database corresponding to the server identification
Server imports the speed of data, and the data got are imported to the corresponding database server of the server identification.
A kind of method for importing data provided in an embodiment of the present invention, including：
Database server provides the load state information of the database server to intermediate server, makes the centre
Server is according to the load state information of the database server, it is determined that importing the speed of data to the database server
Rate；
The data that the speed for receiving the intermediate server to determine is imported.
A kind of device for importing data provided in an embodiment of the present invention, including：
Load acquisition module, the load state information for obtaining each database server；
Determining module, for for each database server, being taken according to the database that load acquisition module is got
Be engaged in the load state information of device, it is determined that import the speed of data to the database server, wherein, the database server it is negative
Carry it is bigger, it is determined that to the database server import data speed it is smaller；
Data acquisition module, for when receiving the task of client addition, according to the data included in the task
Mark, obtains the corresponding data of Data Identification described in data storage area；
Import modul, for according to the server identification included in the task, with the determining module determine to
The corresponding database server of the server identification imports the speed of data, is taken to the corresponding database of the server identification
Business device imports the data that the data acquisition module is got.
A kind of device for importing data provided in an embodiment of the present invention, including：
Module is provided, the load state information for providing the database server to intermediate server, make it is described in
Between server according to the load state information of the database server, it is determined that importing the speed of data to the database server
Rate；
Receiving module, the data imported for receiving the intermediate server with the speed determined.
The embodiment of the present invention provides a kind of method and device for importing data, and intermediate server obtains each database server
Load state information, and determine therefrom that the speed that data are imported to each database server, wherein, database server it is negative
Carry it is bigger, it is determined that to the database server import data speed it is smaller, when receive client addition task when, root
According to the server identification included in the task, data are imported to determine to the corresponding database server of the server identification
Speed, import data to the corresponding database server of the server identification.By the above method, intermediate server can basis
The load of database server, imports data, therefore reduce high capacity database with corresponding speed to database server
The risk of servers go down, moreover, intermediate server can carry out data in real time and lead without importing data in the specified period
Enter, this also increases the efficiency for importing data.
Brief description of the drawings
Fig. 1 is the system structure diagram for importing data in the prior art；
Fig. 2 is the process provided in an embodiment of the present invention for importing data；
Fig. 3 is provided in an embodiment of the present invention using intermediate server as Deferred Job servers, database server
For the importing data storage system architectures schematic diagram exemplified by MySQL server；
Fig. 4 is the apparatus structure schematic diagram of the first importing data provided in an embodiment of the present invention；
Fig. 5 is the apparatus structure schematic diagram of second provided in an embodiment of the present invention importing data.
Embodiment
In order to avoid in the prior art, intermediate server is easily caused when importing data to the database server of high capacity
The problem of database server of high capacity is paralysed, and improve in the efficiency of data importing, the embodiment of the present invention, intermediate server
The load state information of each database server is obtained, and determines therefrom that the speed that data are imported to each database server,
When importing data to some database server, data are carried out with the speed for importing data to the database server of determination and led
Enter.
With reference to Figure of description, the embodiment of the present invention is described in detail.
Fig. 2 is the process of importing data provided in an embodiment of the present invention, specifically includes following steps：
S201：Intermediate server obtains the load state information of each database server.
In embodiments of the present invention, intermediate server can periodically obtain the load condition letter of each database server
Breath.Specifically, intermediate server can obtain each according to the first setting cycle by the way that the acquisition process of operation is middle from the central database
The load state information of database server, wherein, the load condition letter of each database server preserved in central database
Breath is that each database server is reported according to the second setting cycle to central database.That is, for a database
For server, the database server can be according to the second setting cycle, by the offer process of operation on central database
Report the load state information of the database server, make intermediate server is middle from the central database to obtain the database server
Load state information.
Further, in order to ensure in the availability of acquisition process run in intermediate server, the embodiment of the present invention, in
Between server the acquisition process of operation can be also monitored by finger daemon, when monitoring the acquisition process and being not present,
The acquisition process is restarted by the finger daemon.It is similar, can in order to ensure the offer process run in database server
With property, database server can be also monitored by finger daemon to the offer process of operation, when monitoring the offer process
In the absence of when, the offer process is restarted by the finger daemon.
S202：For each database server, according to the load state information of the database server got, really
Orient the speed that the database server imports data.
Wherein, the load of the database server is bigger, it is determined that to the database server import data speed get over
It is small.
In embodiments of the present invention, the load state information for the database server that intermediate server is got includes data
The central processing unit of storehouse server（Central Processing Unit, CPU）Quantity, cpu load value and memory input
Output（Input Output, IO）Load value.Then intermediate server is for a database server, according to the number got
Determined according to the load state information of storehouse server can be to the method for speed that the database server imports data：It is determined that obtaining
The cpu load value for the database server got and the ratio of CPU quantity, as cpu load average, according to for the data
The functional relation of the speed of the default cpu load average of storehouse server, memory I O load values with importing data, and determine
The cpu load average of the database server and the memory I O load values obtained, it is determined that importing number to the database server
According to speed.
Wherein, cpu load average, memory I O load values are preset with importing the speed of data for a database server
The method of the functional relation of rate can be specifically：Several cpu load values and memory I O are preset for the database server
Load value, determines importing number of the database server in the case of each default cpu load value and memory I O load values
According to expected rate, according to each default cpu load value and memory I O load values, and each default CPU determined is born
Load value and the corresponding importing data expected rate of memory I O load values, fitting cpu load value and memory I O load values are with leading
Enter the functional relation of the speed of data.
For example, being C1, C2 for its predeterminable cpu load average of a database server, preset its memory I O and bear
Load value is I1, I2, then determines the database server in the importing number that cpu load value is when C1, memory I O load values are I1
According to expected rate V1, determine the database server in the importing data that cpu load value is when C1, memory I O load values are I2
Expected rate V2, determines the database server in the importing data phase that cpu load value is when C2, memory I O load values are I1
Speed V3 is hoped, it is that importing data when C2, memory I O load values are I2 are expected in cpu load value to determine the database server
Speed V4, and binary function f (C, I) is fitted, meet the binary function of fitting：f(C1,I1)=V1、f(C1,I2)=V2、f
(C2,I1)=V3、f(C2,I2)=V4.So, the binary function f (C, I) of fitting is default aiming at the database server
The functional relation of the speed of cpu load average, memory I O load values with importing data.
Follow-up, intermediate server then can be according to functional relation f (C, I), and database server got
Cpu load average and memory I O load values, it is determined that to the database server import data speed.
S203：When receiving the task of client addition, according to the Data Identification included in the task, obtain data and deposit
The corresponding data of the Data Identification of this in storage area.
S204：According to the server identification included in the task, to determine data corresponding to the server identification
Storehouse server imports the speed of data, and the data got are imported to the corresponding database server of the server identification.
In embodiments of the present invention, above-mentioned intermediate server is determined to each database server by step S201~S202
Importing the method for the speed of data can be performed by the first setting cycle, i.e. no matter whether intermediate server receives client
The task of the importing data of addition, intermediate server all according to the first setting cycle, obtains the load shape of each database server
State information, the load state information that each database server is obtained every time determines that the speed that data are imported to each database server
Rate.
And when intermediate server receives the task of client addition, then can be according to the numbers of the data included in the task
According to mark, the corresponding data of the Data Identification in data storage area are obtained, and according to the database server included in the task
Server identification, to determine the speed that data are imported to the corresponding database server of the server identification, to the clothes
Business device identifies corresponding database server and imports the data got.
By the above method, intermediate server can be according to the load of database server, with the speed of response to database
Server imports data, when the load of a database server is larger, and intermediate server is imported to the database server
The speed of data is relatively low, thus reduce high capacity database server paralysis risk, moreover, intermediate server without
In the specified period（Such as every 2 points to 6 points of morning）Data are imported, data importing can be carried out in real time, can also be effectively improved
Import the efficiency of data.
Further, intermediate server is controlling to import number to database server by step S204 as shown in Figure 2
According to speed when, the step of above-mentioned control imports speed can be performed by tc flow control tools.Specifically, tc flows can be passed through
Control instrument performs following orders：
tc qdisc add dev$DEV root handle1:htb default11
tc class add dev$DEV parent1:classid1:1htb rate1000mbit ceil1000mbit
tc class add dev$DEV parent1:classid1:2htb rate${limit}Mbps
iptables-t mangle-A POSTROUTING-p tcp-d$ip-j MARK--set-mark5
tc filter add dev$DEV parent1:0prio1protocol ip handle5fw flowid1:2
The function of above-mentioned first line command is to create the queue for needing to control to import speed；
The function of second line command is that a classification, the speed of the classification are created in the queue that the first line command is created
The upper limit is 1000M/s；
The function of the third line order is one subclassification of establishment, the subclassification collection in the classification that the second line command is created
Into the attribute of parent category, i.e. the rate-limit of the subclassification is also 1000M/s；
The function of fourth line order refers to that constant current will controlled rule to the speed of the flow of which IP address；
The function of 5th boat order is the subclassification for making the taking effect rules of fourth line be created in the third line order.
Wherein, the DEV in mentioned order represents the port that flow goes out from middle server flows, and limit represents flowing out stream
The size of amount, unit is M/s, and ip is exactly to flow out the IP address that flow is pointed to, namely the targeting database server to be imported
IP address.
It can be seen that, by tc flow control tools control to database server import the speed of data when, only need by
Being used to be added to the port that the database server imports data in intermediate server needs to control to import in the queue of speed,
And the IP address of the database server is specified, tc flows instrument can be to the speed of the data for importeding into the database server
It is controlled, is specifically according to for the default cpu load average of the database server, memory I O load values and importing number
According to speed functional relation speed control.
In addition, intermediate server is got after the load state information of each database server every time, it can preserve and get
Each database server load state information, and contrast the load state information of this each database server got
The load state information of each database server got with the last time.For a database server, if middle clothes
The business device last time gets the load state information of the database server, and this does not get the negative of the database server
Status information is carried, then illustrates that the database server is offline, intermediate server can be determined no longer to the database server
Import the speed of data.Opposite, for a database server, if intermediate server is last not to get the data
The load state information of storehouse server, and this gets the load state information of the database server, then illustrates the data
Storehouse server is the database server newly reached the standard grade, and intermediate server can control to take to the database by the method shown in Fig. 2
The speed for device importing data of being engaged in.
Intermediate server described in the embodiment of the present invention includes but is not limited to Deferred Job servers, database service
Device includes but is not limited to MySQL server.Fig. 3 is the clothes provided in an embodiment of the present invention by Deferred Job of intermediate server
Device, database server be engaged in for the importing data storage system architectures schematic diagram exemplified by MySQL server, is wrapped in the system shown in Fig. 3
Include client, Deferred Job server clusters, data storage area, central database and MySQL server cluster.Its
In, Deferred Job server clusters are made up of each Deferred Job servers, and MySQL server cluster is by each MySQL
Server is constituted.
In the system as shown in fig. 3, each MySQL server periodically reports MySQL server certainly to central database
The load state information of body, Deferred Job servers periodically obtain the negative of each MySQL server from the central database
Status information is carried, and the speed to each MySQL server importing data is determined according to the load state information of acquisition, when user will
When importing data in MySQL server, first it upload the data in data storage area, and by client to Deferred
Job servers add task, the Data Identification that Deferred Job servers are included in the task according to addition, from data storage
Obtain corresponding data in area, and the server identification included in the task according to addition, to determine to the server mark
Know the speed that corresponding database server imports data, import what is obtained to the corresponding database server of the server identification
Data.
It is above the method for importing data provided in an embodiment of the present invention, based on same invention thinking, the present invention is implemented
Example also provides a kind of device for importing data, as shown in Figure 4 and Figure 5.
Fig. 4 is the apparatus structure schematic diagram of the first importing data provided in an embodiment of the present invention, is specifically included：
Load acquisition module 401, the load state information for obtaining each database server；
Determining module 402, for for each database server, the number got according to load acquisition module 401
According to the load state information of storehouse server, it is determined that the speed of data is imported to the database server, wherein, the database service
The load of device is bigger, it is determined that to the database server import data speed it is smaller；
Data acquisition module 403, for when receiving the task of client addition, according to the number included in the task
According to mark, the corresponding data of Data Identification described in data storage area are obtained；
Import modul 404, for according to the server identification included in the task, being determined with the determining module 402
What is gone out imports the speed of data to the corresponding database server of the server identification, to the corresponding number of the server identification
The data that the data acquisition module 403 is got are imported according to storehouse server.
The load acquisition module 401 is specifically for according to the first setting cycle, by the acquisition process of operation from center
The load state information of each database server is obtained in database；
The load acquisition module 401 is additionally operable to, and the acquisition process is monitored by finger daemon, when monitoring
When the acquisition process is not present, the acquisition process is restarted by the finger daemon.
The load state information for the database server that the load acquisition module 401 is got includes：Database service
Central processor CPU quantity, cpu load value and the memory input and output I/O load value of device；
The determining module 402 is specifically for it is determined that the cpu load value of the database server got and CPU numbers
The ratio of amount, is used as cpu load average；According to for the default cpu load average of the database server, memory I O loads
The functional relation of speed of the value with importing data, and the cpu load average of the database server determined and depositing for obtaining
Reservoir I/O load value, it is determined that importing the speed of data to the database server.
The device of specific above-mentioned importing data as shown in Figure 4 can be located in intermediate server.
Fig. 5 is the apparatus structure schematic diagram of second provided in an embodiment of the present invention importing data, is specifically included：
Module 501 is provided, the load state information for providing the database server to intermediate server makes described
Intermediate server is according to the load state information of the database server, it is determined that importing data to the database server
Speed；
Receiving module 502, the data imported for receiving the intermediate server with the speed determined.
The offer module 501 is specifically for according to the second setting cycle, by the offer process of operation to centre data
Storehouse reports the load state information of the database server；
The offer module 501 is additionally operable to, and the offer process is monitored by finger daemon；It is described when monitoring
When offer process is not present, the offer process is restarted by the finger daemon.
The device of specific above-mentioned importing data as shown in Figure 5 can be located in database server.
The embodiment of the present invention provides a kind of method and device for importing data, and intermediate server obtains each database server
Load state information, and determine therefrom that the speed that data are imported to each database server, wherein, database server it is negative
Carry it is bigger, it is determined that to the database server import data speed it is smaller, when receive client addition task when, root
According to the server identification included in the task, data are imported to determine to the corresponding database server of the server identification
Speed, import data to the corresponding database server of the server identification.By the above method, intermediate server can basis
The load of database server, imports data, therefore reduce high capacity database with corresponding speed to database server
The risk of servers go down, moreover, intermediate server can carry out data in real time and lead without importing data in the specified period
Enter, this also increases the efficiency for importing data.
It should be understood by those skilled in the art that, embodiments herein can be provided as method, system or computer program
Product.Therefore, the application can be using the reality in terms of complete hardware embodiment, complete software embodiment or combination software and hardware
Apply the form of example.Moreover, the application can be used in one or more computers for wherein including computer usable program code
Usable storage medium（Including but not limited to magnetic disk storage, CD-ROM, optical memory etc.）The computer program production of upper implementation
The form of product.
In a typical configuration, computing device includes one or more processors (CPU), input/output interface, net
Network interface and internal memory.
Internal memory potentially includes the volatile memory in computer-readable medium, random access memory (RAM) and/or
The forms such as Nonvolatile memory, such as read-only storage (ROM) or flash memory (flash RAM).Internal memory is computer-readable medium
Example.
Computer-readable medium includes permanent and non-permanent, removable and non-removable media can be by any method
Or technology come realize information store.Information can be computer-readable instruction, data structure, the module of program or other data.
The example of the storage medium of computer includes, but are not limited to phase transition internal memory (PRAM), static RAM (SRAM), moved
State random access memory (DRAM), other kinds of random access memory (RAM), read-only storage (ROM), electric erasable
Programmable read only memory (EEPROM), fast flash memory bank or other memory techniques, read-only optical disc read-only storage (CD-ROM),
Digital versatile disc (DVD) or other optical storages, magnetic cassette tape, the storage of tape magnetic rigid disk or other magnetic storage apparatus
Or any other non-transmission medium, the information that can be accessed by a computing device available for storage.Define, calculate according to herein
Machine computer-readable recording medium does not include the data-signal and carrier wave of non-temporary computer readable media (transitory media), such as modulation.
The application is with reference to method, the equipment according to the embodiment of the present application（System）And the flow of computer program product
Figure and/or block diagram are described.It should be understood that can be by every first-class in computer program instructions implementation process figure and/or block diagram
Journey and/or the flow in square frame and flow chart and/or block diagram and/or the combination of square frame.These computer programs can be provided
The processor of all-purpose computer, special-purpose computer, Embedded Processor or other programmable data processing devices is instructed to produce
A raw machine so that produced by the instruction of computer or the computing device of other programmable data processing devices for real
The device for the function of being specified in present one flow of flow chart or one square frame of multiple flows and/or block diagram or multiple square frames.
These computer program instructions, which may be alternatively stored in, can guide computer or other programmable data processing devices with spy
Determine in the computer-readable memory that mode works so that the instruction being stored in the computer-readable memory, which is produced, to be included referring to
Make the manufacture of device, the command device realize in one flow of flow chart or multiple flows and/or one square frame of block diagram or
The function of being specified in multiple square frames.
These computer program instructions can be also loaded into computer or other programmable data processing devices so that in meter
Series of operation steps is performed on calculation machine or other programmable devices to produce computer implemented processing, thus in computer or
The instruction performed on other programmable devices is provided for realizing in one flow of flow chart or multiple flows and/or block diagram one
The step of function of being specified in individual square frame or multiple square frames.
Although having been described for the preferred embodiment of the application, those skilled in the art once know basic creation
Property concept, then can make other change and modification to these embodiments.So, appended claims are intended to be construed to include excellent
Select embodiment and fall into having altered and changing for the application scope.
Obviously, those skilled in the art can carry out various changes and modification without departing from this Shen to the embodiment of the present application
Please embodiment spirit and scope.So, if these modifications and variations of the embodiment of the present application belong to the application claim
And its within the scope of equivalent technologies, then the application is also intended to comprising including these changes and modification.
Claims (8)
1. a kind of method for importing data, it is characterised in that including：
Intermediate server obtains the load state information of each database server；And
For each database server, according to the load state information of the database server got, it is determined that to the number
The speed of data is imported according to storehouse server, wherein, the load of the database server is bigger, it is determined that to the database server
The speed for importing data is smaller；And
When receiving the task of client addition, according to the Data Identification included in the task, obtain in data storage area
The corresponding data of the Data Identification；And
According to the server identification included in the task, to determine database service corresponding to the server identification
Device imports the speed of data, and the data got are imported to the corresponding database server of the server identification；
Wherein, the central processor CPU quantity of the load state information of the database server including database server,
Cpu load value and memory input and output I/O load value；And
According to the load state information of the database server got, it is determined that importing the speed of data to the database server
Rate, is specifically included：
It is determined that the ratio of the cpu load value of the database server got and CPU quantity, is used as cpu load average；
According to the speed for the default cpu load average of the database server, memory I O load values with importing data
Functional relation, and determine the database server cpu load average and obtain memory I O load values, it is determined that to this
Database server imports the speed of data；
It is determined that imported to the database server also includes after the speed of data：Will be to the database by tc flows instrument
The speed that server imports data is updated to the speed determined.
2. the method as described in claim 1, it is characterised in that intermediate server obtains the load condition of each database server
Information, is specifically included：
The intermediate server obtains each data according to the first setting cycle by the way that the acquisition process of operation is middle from the central database
The load state information of storehouse server, wherein, the load condition letter of each database server preserved in the central database
Breath be each database server according to the second setting cycle, reported by the offer process of operation to the central database.
3. method as claimed in claim 2, it is characterised in that methods described also includes：
The intermediate server is monitored by finger daemon to the acquisition process, is not deposited when monitoring the acquisition process
When, the acquisition process is restarted by the finger daemon；And/or,
The database server is monitored by finger daemon to the offer process, when monitoring the offer process not
In the presence of, the offer process is restarted by the finger daemon.
4. a kind of method for importing data, it is characterised in that including：
Database server provides the load state information of the database server to intermediate server, makes the intermediary service
Device is according to the load state information of the database server, it is determined that importing the speed of data to the database server；
The data that the speed for receiving the intermediate server to determine is imported；
Wherein, the central processor CPU quantity of the load state information of the database server including database server,
Cpu load value and memory input and output I/O load value；And
According to the load state information of the database server got, it is determined that importing the speed of data to the database server
Rate, is specifically included：
It is determined that the ratio of the cpu load value of the database server got and CPU quantity, is used as cpu load average；
According to the speed for the default cpu load average of the database server, memory I O load values with importing data
Functional relation, and determine the database server cpu load average and obtain memory I O load values, it is determined that to this
Database server imports the speed of data；
Determined in the intermediate server to after the speed of database server importing data, in addition to：Taken in the middle of described
The speed that data are imported to the database server is updated to the speed determined by business device by tc flows instrument.
5. a kind of device for importing data, it is characterised in that including：
Load acquisition module, the load state information for obtaining each database server；
Determining module, for for each database server, the database server got according to load acquisition module
Load state information, it is determined that to the database server import data speed, wherein, the load of the database server is got over
Greatly, it is determined that to the database server import data speed it is smaller；
Data acquisition module, for when receiving the task of client addition, according to the Data Identification included in the task,
Obtain the corresponding data of Data Identification described in data storage area；
Import modul, for according to the server identification included in the task, with the determining module determine to described
The corresponding database server of server identification imports the speed of data, to the corresponding database server of the server identification
Import the data that the data acquisition module is got；
Wherein, the load state information for the database server that the load acquisition module is got includes：Database server
Central processor CPU quantity, cpu load value and memory input and output I/O load value；
The determining module is specifically for it is determined that the cpu load value and the ratio of CPU quantity of the database server got
Value, is used as cpu load average；According to for the default cpu load average of the database server, memory I O load values with leading
Enter the functional relation of the speed of data, and the database server determined cpu load average and the memory I O that obtains
Load value, it is determined that importing the speed of data to the database server；
The import modul is additionally operable to：Determined in the determining module to after the speed of database server importing data,
Described in the speed that data are imported to the database server is updated to determine by the import modul by tc flows instrument
Speed.
6. device as claimed in claim 5, it is characterised in that the load acquisition module is specifically for according to the first setting
In the cycle, pass through the acquisition process of the operation middle load state information for obtaining each database server from the central database；
The load acquisition module is additionally operable to, and the acquisition process is monitored by finger daemon, when monitoring described obtain
When taking the process to be not present, the acquisition process is restarted by the finger daemon.
7. a kind of device for importing data, it is characterised in that including：
Module is provided, the load state information for providing the database server to intermediate server makes to take in the middle of described
Device be engaged according to the load state information of the database server, it is determined that importing the speed of data to the database server；
Receiving module, the data imported for receiving the intermediate server with the speed determined；
Wherein, the central processor CPU quantity of the load state information of the database server including database server,
Cpu load value and memory input and output I/O load value；And
According to the load state information of the database server got, it is determined that importing the speed of data to the database server
Rate, is specifically included：
It is determined that the ratio of the cpu load value of the database server got and CPU quantity, is used as cpu load average；
According to the speed for the default cpu load average of the database server, memory I O load values with importing data
Functional relation, and determine the database server cpu load average and obtain memory I O load values, it is determined that to this
Database server imports the speed of data；
Determined in the intermediate server to after the speed of database server importing data, in addition to：Taken in the middle of described
The speed that data are imported to the database server is updated to the speed determined by business device by tc flows instrument.
8. device as claimed in claim 7, it is characterised in that the offer module specifically for, according to the second setting cycle,
The load state information of the database server is reported to central database by the offer process of operation；
The offer module is additionally operable to, and the offer process is monitored by finger daemon；When monitor it is described provide into
When journey is not present, the offer process is restarted by the finger daemon.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310425562.6A CN103488733B (en) | 2013-09-17 | 2013-09-17 | A kind of method and device for importing data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310425562.6A CN103488733B (en) | 2013-09-17 | 2013-09-17 | A kind of method and device for importing data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103488733A CN103488733A (en) | 2014-01-01 |
| CN103488733B true CN103488733B (en) | 2017-10-27 |
Family
ID=49828959
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310425562.6A Active CN103488733B (en) | 2013-09-17 | 2013-09-17 | A kind of method and device for importing data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103488733B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105335494B (en) * | 2015-10-20 | 2019-03-26 | 广州品唯软件有限公司 | User data introduction method and system |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN100388183C (en) * | 2003-07-05 | 2008-05-14 | 华为技术有限公司 | Server load equalization method for implementing weighted minimum linked allocation |
| GB2483111A (en) * | 2010-08-27 | 2012-02-29 | Zeus Technology Ltd | Monitoring connections to servers and memory management |
| CN102957549A (en) * | 2011-08-19 | 2013-03-06 | 深圳市快播科技有限公司 | System and method for managing cross-platform remote servers |
| CN103118076B (en) * | 2013-01-11 | 2015-05-13 | 烽火通信科技股份有限公司 | Upgraded server cluster system and load balancing method thereof |
-
2013
- 2013-09-17 CN CN201310425562.6A patent/CN103488733B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN103488733A (en) | 2014-01-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9384227B1 (en) | Database system providing skew metrics across a key space | |
| US20120291133A1 (en) | Security compliant data storage management | |
| EP3002924B1 (en) | Stream-based object storage solution for real-time applications | |
| US11388232B2 (en) | Replication of content to one or more servers | |
| US10318346B1 (en) | Prioritized scheduling of data store access requests | |
| US9633105B2 (en) | Multidimensional data representation | |
| US10334013B2 (en) | Avoiding web request failures before they occur by component analysis | |
| CN103631873B (en) | A kind of data compression method and storage system | |
| CN110096489A (en) | A kind of data query method, system, device and electronic equipment | |
| CN104113605A (en) | Enterprise cloud application development monitoring processing method | |
| CN104363282A (en) | Cloud computing resource scheduling method and device | |
| CN107291720A (en) | A kind of method, system and computer cluster for realizing batch data processing | |
| CN104021088A (en) | Log storage method and device | |
| CN110457182A (en) | A kind of load balancing cluster example operating index monitoring system | |
| CN106657182A (en) | Cloud file processing method and cloud file processing device | |
| CN103488733B (en) | A kind of method and device for importing data | |
| CN102566942A (en) | File striping writing method, device and system | |
| CN110266555A (en) | Method for analyzing web site service request | |
| CN111770150B (en) | Access flow control method and device and electronic equipment | |
| CN103560974B (en) | Method and device for maintaining tokens | |
| CN108664322A (en) | Data processing method and system | |
| US20230040444A1 (en) | Systems and methods for modulating data objects to effect state changes | |
| CN109254928A (en) | A kind of method of log processing, system, equipment and computer readable storage medium | |
| US20190166081A1 (en) | Dynamic communication session management | |
| CN108563781A (en) | Internet of Things big data processing method based on Hadoop and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20230418 Address after: Room 501-502, 5/F, Sina Headquarters Scientific Research Building, Block N-1 and N-2, Zhongguancun Software Park, Dongbei Wangxi Road, Haidian District, Beijing, 100193 Patentee after: Sina Technology (China) Co.,Ltd. Address before: 100080, International Building, No. 58 West Fourth Ring Road, Haidian District, Beijing, 20 floor Patentee before: Sina.com Technology (China) Co.,Ltd. |
|
| TR01 | Transfer of patent right |