CN106897325B - Data loading method and device - Google Patents
Data loading method and device Download PDFInfo
- Publication number
- CN106897325B CN106897325B CN201510967644.2A CN201510967644A CN106897325B CN 106897325 B CN106897325 B CN 106897325B CN 201510967644 A CN201510967644 A CN 201510967644A CN 106897325 B CN106897325 B CN 106897325B
- Authority
- CN
- China
- Prior art keywords
- target data
- loading
- loaded
- delay time
- random delay
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
Abstract
The application provides a data loading method and a data loading device, wherein the method comprises the following steps: generating random delay time when target data is determined to be loaded; and when the random delay time is up, executing the processing of loading the target data from a database. The method and the device reduce the pressure of large-batch data loading requests on the database, and ensure the stability of the database.
Description
Technical Field
The present application relates to network technologies, and in particular, to a data loading method and apparatus.
Background
In service deployment, for a certain service, a larger number of servers may need to be deployed to process the service. When these servers are started, a large number of servers generally acquire initialization data required for starting from the database, and load the initialization data into the server memory to complete the initialization configuration of the service application. For the database, receiving the data acquisition requests of a large number of servers at the same time will put a great pressure on the database, and there is a possibility of overwhelming the database.
Disclosure of Invention
In view of this, the present application provides a data loading method and apparatus, so as to reduce the pressure on a database caused by a large amount of data loading requests, and ensure the stability of the database.
Specifically, the method is realized through the following technical scheme:
in a first aspect, a data loading method is provided, including:
generating random delay time when target data is determined to be loaded;
and when the random delay time is up, executing the processing of loading the target data from a database.
In a second aspect, a data loading apparatus is provided, including:
the time generation module is used for generating random delay time when the target data is determined to be loaded;
and the loading execution module is used for executing the processing of loading the target data from the database when the random delay time is reached.
According to the data loading method and device, a random delay time is generated when data are loaded, the data are loaded when the delay time is up, and the random delay time corresponding to each data to be loaded is different in probability, so that the loading time of each data is dispersed, the pressure of a large batch of data loading requests on a database is reduced, and the stability of the database is ensured.
Drawings
Fig. 1 is an application scenario of a data loading method according to an exemplary embodiment of the present application;
FIG. 2 is a flow chart illustrating a method of loading data according to an exemplary embodiment of the present application;
FIG. 3 is a flow chart illustrating another method of data loading according to an exemplary embodiment of the present application;
FIG. 4 is a flow chart illustrating yet another method of data loading in an exemplary embodiment of the present application;
FIG. 5 is a schematic structural diagram of a data loading apparatus according to an exemplary embodiment of the present application;
fig. 6 is a schematic structural diagram of another data loading apparatus according to an exemplary embodiment of the present application.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.
Fig. 1 illustrates an application scenario of the data loading method of the present application, as shown in fig. 1, for example, for a certain service, a large number of servers are configured, including: server 11, server 12, server 13, etc., may be tens or hundreds in actual implementations. When the servers are started, the initialized configuration data need to be loaded, and then the service processing requests and the services can be normally received. The initialized configuration data may be stored in the database 14, and each server needs to acquire the configuration data from the database 14 and load the configuration data.
When a large number of servers are started at the same time, the servers may load data to the database 14 at the same time, which may put a large strain on the servers and may result in insufficient database connection resources. The data loading method is used for solving the problem, avoiding the situation that the database is under huge pressure, and ensuring the stability of the database. The execution flow of the method can refer to the example of fig. 2, and the flow can be executed by one of the servers:
in step 201, when it is determined that the target data needs to be loaded, a random delay time is generated.
In this step, it is determined that the target data needs to be loaded, for example, the server starts to be started, and it may be checked whether the initialization data is loaded at the next time, and if the initialization data is not loaded, it is necessary to first go to the database to obtain the loaded data to complete the initialization configuration of the server. In the example of initialization by starting up the server, the configuration data used for initialization may be referred to as target data, and in other application scenarios, the data to be loaded may be referred to as target data.
And generating the delay time, wherein the loading of the target data can be delayed and executed instead of immediately executing the loading of the target data when the target data is determined to be loaded, and the delay time is the generated random delay time in the step. The random delay time may be randomly generated, and the random delay time generated by different servers may be different.
For example, assuming that a large number of servers including server 11 to server 13 are all started at the same time in the scenario of fig. 1, according to the method of this step, each server will generate a random delay time at the time of starting, for example, for server 11, the generated random delay time may be 5 seconds, and for server 12, the generated random delay time may be 6 seconds. In addition, even for the same server, the generated random delay time can be different when the server is started at two different times.
In step 202, when the random delay time is reached, the target data is loaded from the database.
For example, taking the random delay time generated by the server 11 as 5 seconds as an example, when determining that the target data needs to be loaded, the server 11 may request the database to obtain the target data load after 5 seconds. For the scenario illustrated in step 201, when a large number of servers are started at the same time, target data needs to be loaded, but since the delay time is randomly generated, the random delay time of each server may be different, and the data loading time of each server may be dispersed, for example, after the servers are started at the same time, server 11 starts loading data after 5 seconds, and server 12 starts loading data after 6 seconds. For the database, the scattered reception of the data loading requests of the servers greatly reduces the pressure compared with the reception at the same time point in the traditional mode.
Furthermore, although the delay time is randomly generated, and the servers started at the same time do not load data at the same time point, it is still possible that the random delay time of some servers is the same. For example, in 100 servers started simultaneously, there may be 15 random delay times generated by the servers, but for the database, compared with the original data loading request received by 100 servers simultaneously, the data loading request received by 15 servers simultaneously still can be received, and the pressure has been reduced significantly.
According to the data loading method, the random delay time is generated, and data loading is performed after the random delay time, so that large batches of data loading requests can be dispersed in time, the pressure on the database is reduced, and the stability of the database is ensured.
In another example, a loading status flag may be set to indicate the status of whether the target data has been loaded more clearly and conveniently, and the random delay time may be generated when it is determined that the target data has not been loaded according to the loading status flag. Referring to the flow shown in FIG. 3:
in step 301, the system start may be a server start, and at this time, initialization configuration is to be performed, so before initialization, it may be determined whether the target data has been loaded according to the preset loading state identifier, in this embodiment, it is determined whether the initialization data is loaded.
Referring to step 302, the loading status flag set in this example may be a flag, and when the flag is true, it may be set to indicate that data has been loaded; when flag is false, it indicates that the data is not loaded yet. In this step, it may be determined whether flag is equal to true.
If yes, indicating that the server has been initialized, go to step 305 and end the process; if not, indicating that initialization data is to be loaded, execution continues with step 303.
In step 303, the server does not load immediately, but generates a random delay time, even if it determines to load data. For example, a time period may be predetermined within which a random delay time is generated. Illustratively, the time period of 0-30 seconds can be randomly generated. In a specific implementation, multiple servers applied to the same service may use the same time period, for example, in the scenario of fig. 1, each server is applied to the same service, and when generating the random delay time, each server may generate time within the time period of 0 to 30 seconds.
The determination of the preset time period may be determined according to a service to which the target data is applied. For example, for a service, when the server starts to start, a service processing request of the service arrives at the server after 30 seconds, and the server needs to complete the initialization configuration as soon as possible before the service processing request arrives, so that, for example, the time period for generating the random delay time may be set to 10 seconds. In addition, when the time period is set, the number of servers can be considered, so that even if the random delay time of partial servers is the same, the number of servers loaded at the same time point can be ensured not to be too large.
In the example shown in fig. 3, it can also be seen that the loading state identifier may be used to indicate whether the target data has been loaded, and whether to generate the random delay time may be determined according to the identifier, so that the loading state identifier may be used as a trigger condition for loading the target data, which is convenient for a manager to control when to load the target data by using the loading state identifier.
For example, assuming that the subsequent target data is updated after the successful target data is loaded, and the server needs to reload the target data at this time, it is possible to modify the loading status flag to indicate that the target data is not loaded, e.g., modify flag true to flag false. Meanwhile, the value of the flag is periodically checked, for example, the state of the flag may be queried every 3 seconds, and when the flag is found to be false, it may be determined that the target data needs to be reloaded at this time. The server may generate a random delay time and load the updated target data from the database when the random delay time is reached. In this example, the loading status identifier is queried periodically, and it can be determined whether data needs to be loaded according to the identifier, and when it is determined that data needs to be loaded, the data is loaded by delaying the time at random.
FIG. 4 illustrates the flow of another business process method, in this example, assuming that the loading was not successful when the data was loaded from the database at step 304. For example, when the server successfully loads the initialization data, the flag value may be modified to true, and when the flag value is periodically checked, if the flag is still false, it may be determined that the loading is not successful, and at this time, the flag value is not modified, and a random delay time is continuously generated, see the flow of fig. 4. The delay time generated when the loading fails can be called another random delay time, and then when the another random delay time is reached, the target data is loaded from the database again, and the above steps are repeatedly executed until the data loading is successful.
According to the data loading method, the loading state identifier is set, so that random delay time can be generated when the identifier indicates that the target data is not loaded, and then the data is loaded, a large amount of data can be loaded and dispersed in a time period, the target data can be successfully loaded, the pressure of a database is reduced, and the stability of the database is ensured.
The application also provides a data loading device, which can be applied to a server and can execute the data loading method of the embodiment of the application so as to enable the server to load data at random time delay. As shown in fig. 5, the apparatus may include: a time generation module 51 and a load execution module 52.
A time generation module 51, configured to generate a random delay time when it is determined that target data needs to be loaded;
and the load execution module 52 is configured to execute the process of loading the target data from the database when the random delay time arrives.
Further, the time generation module 51 is configured to determine that the target data needs to be loaded when it is determined that the target data is not loaded according to a preset loading state identifier, where the loading state identifier is used to indicate whether the target data is loaded.
Further, the time generating module 51 is configured to generate a random delay time within a preset time period; and the preset time period is determined according to the service applied by the target data.
Further, the time generating module 51 is configured to generate another random delay time again when the load executing module fails to load the target data from the database; and the load execution module 52 is configured to load the target data from the database again when the another random delay time arrives.
Fig. 6 illustrates another structure of the data loading apparatus, and as shown in fig. 6, the time generation module 51 may include: an identity management submodule 511 and a time management submodule 512.
The identifier management sub-module 511 is configured to, after the target data is loaded, modify the loading status identifier to indicate that the target data is not loaded if the target data is updated;
the time management submodule 512 is configured to determine that target data needs to be loaded according to the modified loading state identifier, and generate random delay time;
and a load execution module 52, configured to load the updated target data from the database when the random delay time arrives.
The data loading device of the embodiment can disperse a large amount of data loading requests in time by generating the random delay time and loading data after the random delay time, thereby reducing the pressure on the database and ensuring the stability of the database.
The above description is only exemplary of the present application and should not be taken as limiting the present application, as any modification, equivalent replacement, or improvement made within the spirit and principle of the present application should be included in the scope of protection of the present application.
Claims (8)
1. A data loading method is applied to initialization during server startup, configuration data used for initialization is called target data, and the method comprises the following steps:
when the target data is determined to be loaded, generating a random delay time within a preset time period; the preset time period is determined according to the time for sending the service processing request to the server after the server starts to start the service applied by the target data;
and when the random delay time is up, executing the processing of loading the target data from a database.
2. The method of claim 1, wherein determining that target data needs to be loaded comprises:
and determining that the target data is not loaded according to a preset loading state identifier, wherein the loading state identifier is used for indicating whether the target data is loaded or not.
3. The method of claim 2, further comprising:
after the target data is loaded, if the target data is updated, the loading state identifier is modified to represent that the target data is not loaded;
and according to the modified loading state identifier, determining that target data needs to be loaded, generating random delay time, and loading updated target data from the database when the random delay time is reached.
4. The method of claim 1, further comprising:
and if the target data loading from the database fails, generating another random delay time again, and loading the target data from the database again when the another random delay time is reached.
5. A data loading device, which is used for initialization during server startup, and the configuration data used for initialization is called target data, and the device comprises:
the time generation module is used for generating a random delay time within a preset time period when the target data is determined to be loaded; the preset time period is determined according to the time for sending the service processing request to the server after the server starts to start the service applied by the target data;
and the loading execution module is used for executing the processing of loading the target data from the database when the random delay time is reached.
6. The apparatus of claim 5,
the time generation module is used for determining that the target data needs to be loaded when the target data is determined not to be loaded according to a preset loading state identifier, and the loading state identifier is used for indicating whether the target data is loaded or not.
7. The apparatus of claim 6, wherein the time generation module comprises: the identifier management submodule and the time management submodule;
the identifier management submodule is used for modifying the loading state identifier into a state indicating that the target data is not loaded if the target data is updated after the target data is loaded;
the time management submodule is used for determining that target data needs to be loaded according to the modified loading state identifier and generating random delay time;
and the loading execution module is used for loading the updated target data from the database when the random delay time is reached.
8. The apparatus of claim 5,
the time generation module is used for generating another random delay time again when the loading execution module fails to load the target data from the database;
and the loading execution module is used for loading the target data from the database again when the other random delay time is reached.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510967644.2A CN106897325B (en) | 2015-12-21 | 2015-12-21 | Data loading method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510967644.2A CN106897325B (en) | 2015-12-21 | 2015-12-21 | Data loading method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106897325A CN106897325A (en) | 2017-06-27 |
| CN106897325B true CN106897325B (en) | 2020-11-17 |
Family
ID=59190219
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510967644.2A Active CN106897325B (en) | 2015-12-21 | 2015-12-21 | Data loading method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106897325B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111783002A (en) * | 2020-06-30 | 2020-10-16 | 北京达佳互联信息技术有限公司 | Webpage data loading method and device, electronic equipment and storage medium |
| CN113190384B (en) * | 2021-05-21 | 2022-07-22 | 重庆紫光华山智安科技有限公司 | Data recovery control method, device, equipment and medium based on erasure codes |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080177892A1 (en) * | 2007-01-19 | 2008-07-24 | International Business Machines Corporation | Method for service oriented data extraction transformation and load |
| CN101447997B (en) * | 2008-12-31 | 2012-10-10 | 中国建设银行股份有限公司 | Data processing method, server and data processing system |
| CN102880678A (en) * | 2012-09-11 | 2013-01-16 | 哈尔滨工程大学 | Embedded real-time memory database |
| CN104184765B (en) * | 2013-05-23 | 2018-04-17 | 阿里巴巴集团控股有限公司 | A kind of request control method and client terminal device and server-side device |
-
2015
- 2015-12-21 CN CN201510967644.2A patent/CN106897325B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN106897325A (en) | 2017-06-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109743390B (en) | Task scheduling method and device, computer equipment and storage medium | |
| CN107748695B (en) | Timed task processing method and device, storage medium and computer equipment | |
| US10225145B2 (en) | Method and device for updating client | |
| CN109194538B (en) | Testing method, device, server and storage medium based on distributed coordination | |
| CN108256118B (en) | Data processing method, device, system, computing equipment and storage medium | |
| CN106712981B (en) | Node change notification method and device | |
| CN106570097B (en) | Sequence generation method and device | |
| CN106201574B (en) | Application interface starting method and device | |
| CN107634964B (en) | WAF (Wireless Access Filter) testing method and device | |
| RU2749339C1 (en) | Equipment control method, cluster system, electronic device and readable data media | |
| CN105159771A (en) | Method and apparatus for processing inter-application chain type start | |
| CN109918187B (en) | Task scheduling method, device, equipment and storage medium | |
| US20130204926A1 (en) | Information processing system, information processing device, client terminal, and computer readable medium | |
| CN106897325B (en) | Data loading method and device | |
| CN110659131A (en) | Task processing method, electronic device, computer device, and storage medium | |
| CN111698281B (en) | Resource downloading method and device, electronic equipment and storage medium | |
| CN113472679A (en) | Interface current limiting method and system | |
| WO2018068655A1 (en) | License management method and system | |
| CN111046377B (en) | Method and device for loading dynamic link library, electronic equipment and storage medium | |
| CN107425994B (en) | Method, terminal and server for realizing remote parameter management | |
| CN114185763A (en) | Dynamic allocation method, device, storage medium and electronic equipment | |
| CN111104330A (en) | Distributed test method, server and distributed test system | |
| CN111147585A (en) | Equipment upgrading method, device, storage medium and system | |
| CN110266438B (en) | Data transmission method, device and system and computer readable storage medium | |
| CN111309467A (en) | Task distribution method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20200921 Address after: Cayman Enterprise Centre, 27 Hospital Road, George Town, Grand Cayman Islands Applicant after: Innovative advanced technology Co.,Ltd. Address before: Cayman Enterprise Centre, 27 Hospital Road, George Town, Grand Cayman Islands Applicant before: Advanced innovation technology Co.,Ltd. Effective date of registration: 20200921 Address after: Cayman Enterprise Centre, 27 Hospital Road, George Town, Grand Cayman Islands Applicant after: Advanced innovation technology Co.,Ltd. Address before: A four-storey 847 mailbox in Grand Cayman Capital Building, British Cayman Islands Applicant before: Alibaba Group Holding Ltd. |
|
| TA01 | Transfer of patent application right | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |