CN117290329A - Method for improving access concurrency performance of MPP data warehouse based on high-availability connection pool - Google Patents

Method for improving access concurrency performance of MPP data warehouse based on high-availability connection pool Download PDF

Info

Publication number
CN117290329A
CN117290329A CN202310151902.4A CN202310151902A CN117290329A CN 117290329 A CN117290329 A CN 117290329A CN 202310151902 A CN202310151902 A CN 202310151902A CN 117290329 A CN117290329 A CN 117290329A
Authority
CN
China
Prior art keywords
preset
control module
central control
connection
data warehouse
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202310151902.4A
Other languages
Chinese (zh)
Inventor
刘坤
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shenzhen Jinxinquan Technology Co ltd
Original Assignee
Shenzhen Jinxinquan Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shenzhen Jinxinquan Technology Co ltd filed Critical Shenzhen Jinxinquan Technology Co ltd
Priority to CN202310151902.4A priority Critical patent/CN117290329A/en
Publication of CN117290329A publication Critical patent/CN117290329A/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/25—Integrating or interfacing systems involving database management systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/28—Databases characterised by their database models, e.g. relational or object models
    • G06F16/283—Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP

Landscapes

  • Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

The invention relates to the technical field of high-availability connection pools, in particular to a method for improving access concurrency performance of an MPP data warehouse based on a high-availability connection pool, which comprises the following steps: the central control module judges whether the operation connection accords with a preset standard according to the average response time length; judging whether to adjust the maximum allowable value of the flow transmission in the operation connection or call a plurality of new connections according to the average response time length so as to construct a plurality of data channels in the user terminal and the secondary data warehouse, and correcting the ratio of the comprehensive flow transmission values in each operation connection; and the central control module judges whether to newly add connection in the connection pool according to the total number of the connections to be used, and carries out secondary correction on the ratio of the comprehensive flow transmission values when the number of the connections in the connection pool exceeds the preset number of the connections, and processes the condition that multiple users access a single database at the same time so as to improve the access concurrency performance of the MPP data warehouse.

Description

Method for improving access concurrency performance of MPP data warehouse based on high-availability connection pool
Technical Field
The invention relates to the technical field of high-availability connection pools, in particular to a method for improving access concurrency performance of an MPP data warehouse based on a high-availability connection pool.
Background
The customer group of the mobile service is very huge, when users of the related system of the mobile user are accessed, the problem of simultaneously accessing a large number of clients is faced, and when the user quantity is increased to a certain degree, the pressure of concurrent user access cannot be born; the invention processes the electronic digital data and plans the construction of the data channel of the secondary data warehouse.
Chinese patent publication No.: CN111597173a discloses a data warehouse system, which comprises a data mobile phone unit, a data storage unit and a data access unit, and is capable of providing data storage and access service for each service system in the internet enterprise, uploading service data related to the service system to the data warehouse system, performing unified cleaning and storage by the data warehouse system, and performing unified authentication processing when each service system accesses the data, thus the following problems are found: the condition that multiple users access a single database at the same time is not processed, and the phenomenon of network breakdown is easily caused by the fact that multiple users access the single database at the same time.
Disclosure of Invention
Therefore, the invention provides a method for improving the access concurrency performance of an MPP data warehouse based on a high-availability connection pool, which is used for solving the problem of network collapse caused by simultaneous access of a single database by multiple users in the prior art.
To achieve the above object, the present invention provides a method for improving access concurrency performance of an MPP data warehouse based on a high-availability connection pool, comprising,
when a single terminal in the user side sends an access request, the central control module invokes a single connection from the connection pool according to the information in the access request to construct a data channel of a corresponding secondary data warehouse in the user side and the MPP data warehouse; the central control module marks the connection as operation connection when the connection completes the construction of the data channels of the user side and the secondary data warehouse;
the central control module sequentially records an access time node when the terminal sends an access request, an output time node when the terminal outputs data from the secondary data warehouse and a receiving time node when the user receives the data when the construction of the data channels of the user side and the secondary data warehouse is completed, and the central control module obtains the response time of the operation connection for the access request according to the access time node and the output time node and obtains the transmission time of the operation connection for the access request according to the output time node and the receiving time node;
The central control module counts the response time and the transmission time of each access request of a single connection in a preset time to obtain the average response time and the average transmission time of the operation connection between the user terminal and the secondary data warehouse in the preset time, and judges whether the operation connection accords with a preset standard according to the average response time;
the central control module judges whether to adjust the maximum allowable value of the flow transmission in the operation connection or call a plurality of new connections to construct a plurality of data channels in the user terminal and the secondary data warehouse according to the average response time when judging that the single connection does not meet the preset standard, and judges whether to correct the ratio of the comprehensive flow transmission values in each operation connection according to the comprehensive transmission time of each operation connection after finishing the judgment;
and the central control module judges whether to add connection in the connection pool according to the total number of the connections to be used when judging that new connection is required to be called, and judges whether to carry out secondary correction on the comprehensive flow transmission value ratio of each operation connection when the number of the connections in the connection pool exceeds a preset connection number standard.
Further, the central control module calculates the average response time length Ta0 of the operation connection connected with the single secondary data warehouse in the preset time length under a first preset condition, and setsWherein n is the number of access requests that the operation connection delivers within a preset time period, ta i is an i-th response time period corresponding to an i-th access request that the operation connection delivers to the secondary data warehouse within the preset time period, i=1, 2,3 is set, and n, e i is an i-th time period correction coefficient for the i-th response time period;
the first preset condition is that the central control module judges that the operation duration of the single operation connection reaches a preset duration.
Further, the central control module determines a determination mode for the operation connection according to the average response time length Ta0 under the first preset condition, wherein,
the first judging mode is that the central control module judges that the operation connection of the operation connection meets a preset standard; the first judging mode meets the condition that the average response time length Ta0 of the operation connection in the preset time length is smaller than or equal to a first preset average response time length Ta1 preset in the central control module;
the second judging mode is that the central control module judges that the operation connection connected with the secondary data warehouse does not accord with a preset standard, and the central control module adjusts the maximum allowable value of the flow transmission for the operation connection to a corresponding value according to the difference value between the average response time length Ta0 and the first preset average response time length Ta1; the second judgment mode meets the condition that the average response time length Ta0 is longer than the first preset response time length Ta1 and Ta0 is smaller than a second preset response time length Ta2 preset by the central control module;
The third judging mode is that the central control module judges that the operation connection of the secondary data warehouse connection does not accord with a preset standard, and the central control module calls a corresponding number of new connections according to the difference value between the average response time length Ta0 and the second preset average response time length Ta2 so as to construct a plurality of data channels in the user side and the secondary data warehouse; the third determination mode satisfies that the average response time Ta0 is greater than the second preset average response time Ta2.
Further, the central control module sequentially determines the assignment mode of the correction coefficient of each time length according to the instantaneous data transmission quantity of the operation connection under the second preset condition under the time node that the operation connection transmits the access request to the terminal in the preset time length to the secondary data warehouse, wherein, for the i-th time length correction coefficient ei,
the first assignment mode is that the central control module assigns ei to 1.15; the first assignment mode satisfies that an ith instantaneous data transmission quantity Qi of the operation connection is smaller than or equal to a first preset instantaneous data transmission quantity Qa preset in the central control module when the terminal transmits an ith access request to the secondary data warehouse;
The second assignment mode is that the central control module assigns ei to 1; the second assignment mode satisfies that an ith instantaneous data transmission quantity Qi of the operation connection is larger than a first preset instantaneous data transmission quantity Qa preset in the central control module and Qi is smaller than or equal to a second preset instantaneous data transmission quantity Qb preset in the central control module when the terminal transmits an ith access request to the secondary data warehouse;
a third assignment mode is that the central control module assigns ei to 0.95; the third assignment mode satisfies that the ith instantaneous data transmission quantity Qi of the operation connection is larger than the second preset instantaneous data transmission quantity Qb when the terminal transmits the ith access request to the secondary data warehouse;
and the second preset condition is that the central control module calculates the average response time of the operation connection in the preset time.
Further, the central control module calculates a difference DeltaTax between the average response time length Ta0 and the first preset average response time length Ta1 under a third preset condition, sets DeltaTax=Ta1-Ta 0, records DeltaTax as a first-stage response time length difference of the operation connection, determines an adjustment mode of a maximum allowable value of flow transmission for the operation connection according to DeltaTax,
The first adjusting mode is that the central control module adjusts the maximum allowable value C of the flow transmission to a corresponding value by using a preset first preset adjusting coefficient beta 1, and the preset value is 1 < beta 1 < 1.2; the first adjusting mode meets the condition that the first-level difference DeltaTax is smaller than or equal to a first preset first-level response time length difference DeltaTax 1 set by the central control module;
the second adjusting mode is that the central control module adjusts the maximum allowable value C of the flow transmission to a corresponding value by using a second preset adjusting coefficient beta 2 preset, and beta 1 < beta 2 < 1.3 is set; the second adjustment mode meets the condition that the first-level difference value delta Tax is larger than the first preset first-level response time length difference value delta Tax1, and the delta Tax is smaller than or equal to a second preset first-level response time length difference value delta Tax2 set by the central control module;
the third adjusting mode is that the central control module adjusts the maximum allowable value C of the flow transmission to a corresponding value by using a preset third preset adjusting coefficient beta 3, and beta 2 < beta 3 < 1.5 is set; the third adjusting mode meets the condition that the first-level difference DeltaTax is larger than the second preset first-level response time length difference DeltaTax 2;
the central control module marks the maximum allowable value of the flow transmission for the operation connection after adjustment as C ', and sets C' =C multiplied by beta j, wherein j=1, 2 and 3;
The third preset condition is that the central control module adjusts the maximum allowable value of the flow transmission for the operation connection to a corresponding value according to the difference value between the average response time length Ta0 and the first preset average response time length Ta 1.
Further, the central control module detects the average transmission time length of the operation connection between the user terminal and the secondary data warehouse in the preset time length under a fourth preset condition, and for the kth operation connection respectively connected with the user terminal and the secondary data warehouse, the central control module marks the average transmission time of the operation connection in the preset time length as Tbk0, wherein k=1, 2,3, m, m is the total number of operation connections connected with the user terminal and the secondary data warehouse, and the setting is performedTb i is an ith response time length corresponding to an ith access request which is transmitted to the secondary data warehouse in a preset time length by the operation connection;
and the fourth preset condition is that the central control module adjusts the maximum allowable value of the flow transmission for the operation connection to C'.
Further, the central control module sequentially counts the average transmission time length of the data transmitted by each operation connection in the preset time length under the fourth preset condition to obtain the comprehensive transmission time length Tb of each operation connection in the preset time length, and determines the comprehensive flow transmission value for each operation connection according to Tb A ratio correction method in which
The first correction mode is that the central control module sets the ratio of the comprehensive flow transmission value for each operation connection as a preset transmission value ratio B0, wherein B0 is the ratio of the maximum data transmission value allowed in a single operation connection to the maximum data transmission value which can be born by the operation connection; the first correction mode meets the condition that the comprehensive transmission duration Tb is smaller than or equal to a first preset comprehensive transmission duration Tb1 set in the central control module;
the second correction mode is that the central control module uses a first correction coefficient e1 to set a comprehensive flow transmission value ratio B1 for each operation connection, wherein e1 is more than 1.2 and less than 1.4, and B1=B0×e1; the second correction mode satisfies that the integrated transmission duration Tb is greater than the first preset integrated transmission duration Tb1 and less than or equal to a second preset integrated transmission duration Tb2 set in the central control module;
the third correction mode is that the central control module uses a second correction coefficient to set the comprehensive flow transmission value ratio B2, e1 < e2 < 1.5 for each operation connection, and B2=B0×e2; the third modification manner satisfies that the integrated transmission duration Tb is longer than the second preset integrated transmission duration Tb2.
Further, the central control module calculates the difference delta Tay between the average response time length Ta0 and the second preset average response time length Ta2 under a fifth preset condition, sets delta tay=ta0-Ta 2, marks delta Tay as the difference of the secondary response time lengths of the operation connection, determines a calling mode for the connection quantity connected with the secondary data warehouse according to delta Tay,
the first calling mode is that the central control module uses a preset first regulating coefficient gamma 1 to change the number of the connections connected with the secondary data warehouse to a first number M1,1.3 < gamma 1 < 1.5, and the setting is carried outWherein M is the same as the second levelAn initial number of connections to which the data warehouse is connected; the first calling mode meets the condition that the secondary difference value delta Tay is smaller than or equal to a preset secondary response time length difference value delta Tay0 set by the central control module;
the second calling mode is that the central control module uses a preset second regulating coefficient gamma 2 to change the number of the connections connected with the secondary data warehouse to a first number M2, gamma 1 is less than gamma 2 and less than 1.6, and the setting is carried outThe second calling mode meets the condition that the secondary difference value delta Tay is larger than the preset secondary response time length difference value delta Tay0;
And the fifth preset condition is that the central control module judges that the operation connection connected with the secondary data warehouse does not meet a preset standard, and the central control module calls a corresponding number of new connections according to the difference value between the average response time length Ta0 and the second preset average response time length Ta2 so as to construct a plurality of data channels in the user side and the secondary data warehouse.
Further, the central control module determines a secondary correction mode of the integrated flow transmission value ratio for each operation connection according to the number difference value between the newly added total connection number and the preset connection number standard under the sixth preset condition, wherein,
the first secondary correction mode is that the central control module uses the corrected comprehensive flow transmission value ratio Bz, wherein z=1, 2; the first secondary correction mode meets the quantity difference standard preset by the central control module;
the second secondary correction mode is to secondarily correct the corrected integrated flow transmission value ratio to Bz ', and Bz' =1.05×Bz is set; the second correction mode meets the condition that the quantity difference value is larger than a quantity difference value standard preset by the central control module;
And the sixth preset condition is that the central control module judges that the newly added connection number is larger than a preset connection number standard in the connection pool.
Further, the central control module acquires corresponding keywords from the access request sent by the terminal so as to connect the terminal with the corresponding secondary data warehouse by using connection to construct a data channel of the corresponding secondary data warehouse in the user terminal and the MPP data warehouse.
Compared with the prior art, the method has the beneficial effects that the central control module judges whether the operation connection accords with the preset standard according to the average response time length; judging whether to adjust the maximum allowable value of the flow transmission in the operation connection or call a plurality of new connections according to the average response time length so as to construct a plurality of data channels in the user terminal and the secondary data warehouse, and correcting the ratio of the comprehensive flow transmission values in each operation connection; and the central control module judges whether to newly add connection in the connection pool according to the total number of the connections to be used, and carries out secondary correction on the ratio of the comprehensive flow transmission values when the number of the connections in the connection pool exceeds the preset number of the connections, and processes the condition that multiple users access a single database at the same time so as to improve the access concurrency performance of the MPP data warehouse.
Further, when the central control module judges that the operation duration of the single operation connection reaches the preset duration, according to the number of access requests transmitted by the operation connection in the preset duration and the ith response duration corresponding to the ith access request transmitted by the operation connection to the secondary data warehouse in the preset duration, the average response duration Ta0 of the operation connection connected with the single secondary data warehouse in the preset duration is calculated, and the access concurrency performance of the MPP data warehouse is further improved while the average response duration of the operation connection connected with the single secondary data warehouse in the preset duration is ensured to be accurately and timely acquired.
Further, when the central control module judges that the operation time length of a single operation connection reaches the preset time length, whether the operation connection meets the preset standard is judged according to the average response time length Ta0, when the central control module judges that the operation connection does not meet the preset standard, the maximum allowable value of the flow transmission of the operation connection is selected in a corresponding mode to be adjusted or a corresponding number of new connections are called, so that a plurality of data channels are built in the user side and the secondary data warehouse, the working state of the operation connection is accurately judged, meanwhile, the solution mode when the operation connection does not meet the preset standard is further determined according to the average response time length, and the access concurrency performance of the MPP data warehouse is further improved.
Further, when the central control module calculates the average response time of the operation connection in the preset time, a corresponding correction coefficient ei is selected according to the instantaneous data transmission quantity of the operation connection under the time node of the operation connection in the preset time for transmitting the access request to the secondary data warehouse to each terminal, and the access concurrency performance of the MPP data warehouse is further improved while the accurate acquisition of the average response time of the operation connection connected with the single secondary data warehouse in the preset time is ensured.
Further, when the average response time length Ta0 is greater than the first preset response time length Ta1 and Ta0 is smaller than the second preset response time length Ta2 preset by the central control module, the central control module adjusts the maximum allowable value of the flow transmission for the operation connection to a corresponding value according to the difference value between the average response time length Ta0 and the first preset average response time length Ta1, and when the response is ensured to be made according to the access request sent by the terminal in time, the access concurrency performance of the MPP data warehouse is further improved.
Further, the central control module adjusts the maximum allowable value of the flow transmission aiming at the operation connection to C', and calculates the average transmission duration aiming at the user side and the secondary data warehouse in the preset duration according to the ith instantaneous data transmission quantity Qi, the first preset instantaneous data transmission quantity Qa and the ith response duration corresponding to the ith access request which is transmitted to the secondary data warehouse in the preset duration by the operation connection, thereby ensuring the accurate acquisition of the average transmission duration and further improving the access concurrency performance of the MPP data warehouse.
Further, the central control module calculates the comprehensive transmission duration Tb of each operation connection in the preset duration according to the average transmission time of the operation connection in the preset duration, corrects the comprehensive flow transmission value ratio in each operation connection to a corresponding value according to Tb, and selects the corresponding comprehensive flow transmission value ratio to further improve the access concurrency performance of the MPP data warehouse when the average transmission duration is found to be too high.
Further, the central control module calls a corresponding number of new connections according to the difference value between the average response time length Ta0 and the second preset average response time length Ta2 in a third judging mode so as to construct a plurality of data channels in the user side and the secondary data warehouse, and the access concurrency performance of the MPP data warehouse is further improved while the effective reduction of the response time length is ensured.
Further, when the newly added connection number is determined to be larger than the preset connection number standard in the connection pool, the central control module carries out secondary correction on the ratio of the comprehensive flow transmission values of all the operation connections according to the number difference value between the newly added total connection number and the preset connection number standard, and the access concurrency performance of the MPP data warehouse is further improved while the situation that the influence on normal data transmission between the user side and the secondary data warehouse due to excessive connection numbers in the connection pool is avoided.
Further, the central control module acquires the corresponding keywords according to the access request sent by the terminal so as to connect the terminal with the corresponding secondary data warehouse by using the connection to construct a data channel of the corresponding secondary data warehouse in the user terminal and the MPP data warehouse, and the access concurrency performance of the MPP data warehouse is further improved while the user is ensured to connect the corresponding secondary data warehouse.
Drawings
FIG. 1 is a flow chart of steps of a method for improving concurrency performance of MPP data warehouse access based on a high-availability connection pool in accordance with an embodiment of the present invention;
FIG. 2 is a flowchart of determining a determination mode for the operation connection according to an average response time length according to an embodiment of the present invention;
FIG. 3 is a flowchart of an adjustment method for determining a maximum allowable value of flow transmission for the operation connection according to a difference between an average response time length and a first preset average response time length in an embodiment of the present invention;
fig. 4 is a flowchart of determining a calling manner for the number of connections connected to the secondary data repository according to a difference between an average response time length and a second preset average response time length according to an embodiment of the present invention.
Detailed Description
In order that the objects and advantages of the invention will become more apparent, the invention will be further described with reference to the following examples; it should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
Preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are merely for explaining the technical principles of the present invention, and are not intended to limit the scope of the present invention.
It should be noted that, in the description of the present invention, terms such as "upper," "lower," "left," "right," "inner," "outer," and the like indicate directions or positional relationships based on the directions or positional relationships shown in the drawings, which are merely for convenience of description, and do not indicate or imply that the apparatus or elements must have a specific orientation, be constructed and operated in a specific orientation, and thus should not be construed as limiting the present invention.
Furthermore, it should be noted that, in the description of the present invention, unless explicitly specified and limited otherwise, the terms "mounted," "connected," and "connected" are to be construed broadly, and may be either fixedly connected, detachably connected, or integrally connected, for example; can be mechanically or electrically connected; can be directly connected or indirectly connected through an intermediate medium, and can be communication between two elements. The specific meaning of the above terms in the present invention can be understood by those skilled in the art according to the specific circumstances.
Referring to fig. 1, fig. 2, fig. 3, and fig. 4, the method for improving the concurrency performance of MPP data warehouse access based on a high-availability connection pool according to the embodiments of the present invention includes the steps of determining a determination mode for the operation connection according to an average response time length, determining an adjustment mode for a maximum allowable value of flow transmission for the operation connection according to a difference value between the average response time length and a first preset average response time length, and determining a call mode for the number of connections connected to the secondary data warehouse according to a difference value between the average response time length and a second preset average response time length; the embodiment of the invention provides a step flow chart of a method for improving the access concurrency performance of an MPP data warehouse based on a high-availability connection pool, which comprises the following steps:
when a single terminal in the user side sends an access request, the central control module invokes a single connection from the connection pool according to the information in the access request to construct a data channel of a corresponding secondary data warehouse in the user side and the MPP data warehouse; the central control module marks the connection as operation connection when the connection completes the construction of the data channels of the user side and the secondary data warehouse;
The central control module sequentially records an access time node when the terminal sends an access request, an output time node when the terminal outputs data from the secondary data warehouse and a receiving time node when the user receives the data when the construction of the data channels of the user side and the secondary data warehouse is completed, and the central control module obtains the response time of the operation connection for the access request according to the access time node and the output time node and obtains the transmission time of the operation connection for the access request according to the output time node and the receiving time node;
the central control module counts the response time and the transmission time of each access request of a single connection in a preset time to obtain the average response time and the average transmission time of the operation connection between the user terminal and the secondary data warehouse in the preset time, and judges whether the operation connection accords with a preset standard according to the average response time;
the central control module judges whether to adjust the maximum allowable value of the flow transmission in the operation connection or call a plurality of new connections to construct a plurality of data channels in the user terminal and the secondary data warehouse according to the average response time when judging that the single connection does not meet the preset standard, and judges whether to correct the ratio of the comprehensive flow transmission values in each operation connection according to the comprehensive transmission time of each operation connection after finishing the judgment;
And the central control module judges whether to add connection in the connection pool according to the total number of the connections to be used when judging that new connection is required to be called, and judges whether to carry out secondary correction on the comprehensive flow transmission value ratio of each operation connection when the number of the connections in the connection pool exceeds a preset connection number standard.
Specifically, the central control module calculates the average response time length Ta0 of the operation connection connected with the single secondary data warehouse in the preset time length under a first preset condition, and setsWherein n is the number of access requests transmitted by the operation connection within a preset time period, tai is an ith response time period corresponding to an ith access request transmitted by the operation connection to the secondary data warehouse within the preset time period, i=1, 2,3 is set, and n, ei is an ith time period correction coefficient for the ith response time period;
the first preset condition is that the central control module judges that the operation duration of the single operation connection reaches a preset duration.
Specifically, the central control module determines a determination mode for the operation connection according to the average response time length Ta0 under the first preset condition, wherein,
The first judging mode is that the central control module judges that the operation connection of the operation connection meets a preset standard; the first judging mode meets the condition that the average response time length Ta0 of the operation connection in the preset time length is smaller than or equal to a first preset average response time length Ta1 preset in the central control module;
the second judging mode is that the central control module judges that the operation connection connected with the secondary data warehouse does not accord with a preset standard, and the central control module adjusts the maximum allowable value of the flow transmission for the operation connection to a corresponding value according to the difference value between the average response time length Ta0 and the first preset average response time length Ta1; the second judgment mode meets the condition that the average response time length Ta0 is longer than the first preset response time length Ta1 and Ta0 is smaller than a second preset response time length Ta2 preset by the central control module;
the third judging mode is that the central control module judges that the operation connection of the secondary data warehouse connection does not accord with a preset standard, and the central control module calls a corresponding number of new connections according to the difference value between the average response time length Ta0 and the second preset average response time length Ta2 so as to construct a plurality of data channels in the user side and the secondary data warehouse; the third determination mode satisfies that the average response time Ta0 is greater than the second preset average response time Ta2;
Wherein ta1=120, ta2=2048.
Specifically, the central control module sequentially determines the assignment mode of the time length correction coefficients according to the instantaneous data transmission quantity of the operation connection under the second preset condition under the time node that the operation connection transmits the access request to the terminals in the preset time length to the secondary data warehouse, wherein, for the ith time length correction coefficient ei,
the first assignment mode is that the central control module assigns ei to 1.15; the first assignment mode satisfies that an ith instantaneous data transmission quantity Qi of the operation connection is smaller than or equal to a first preset instantaneous data transmission quantity Qa preset in the central control module when the terminal transmits an ith access request to the secondary data warehouse;
the second assignment mode is that the central control module assigns ei to 1; the second assignment mode satisfies that an ith instantaneous data transmission quantity Qi of the operation connection is larger than a first preset instantaneous data transmission quantity Qa preset in the central control module and Qi is smaller than or equal to a second preset instantaneous data transmission quantity Qb preset in the central control module when the terminal transmits an ith access request to the secondary data warehouse;
a third assignment mode is that the central control module assigns ei to 0.95; the third assignment mode satisfies that the ith instantaneous data transmission quantity Qi of the operation connection is larger than the second preset instantaneous data transmission quantity Qb when the terminal transmits the ith access request to the secondary data warehouse;
The second preset condition is that the central control module calculates the average response time of the operation connection in the preset time;
wherein qa=33 and qb=155.
Specifically, the central control module calculates a difference Δtax between the average response time length Ta0 and the first preset average response time length Ta1 under a third preset condition, sets Δtax=ta1-Ta 0, marks Δtax as a first-stage response time length difference of the operation connection, determines an adjustment mode of a maximum allowable value of flow transmission for the operation connection according to the Δtax, wherein,
the first adjusting mode is that the central control module adjusts the maximum allowable value C of the flow transmission to a corresponding value by using a preset first preset adjusting coefficient beta 1, and the preset value is 1 < beta 1 < 1.2; the first adjusting mode meets the condition that the first-level difference DeltaTax is smaller than or equal to a first preset first-level response time length difference DeltaTax 1 set by the central control module;
the second adjusting mode is that the central control module adjusts the maximum allowable value C of the flow transmission to a corresponding value by using a second preset adjusting coefficient beta 2 preset, and beta 1 < beta 2 < 1.3 is set; the second adjustment mode meets the condition that the first-level difference value delta Tax is larger than the first preset first-level response time length difference value delta Tax1, and the delta Tax is smaller than or equal to a second preset first-level response time length difference value delta Tax2 set by the central control module;
The third adjusting mode is that the central control module adjusts the maximum allowable value C of the flow transmission to a corresponding value by using a preset third preset adjusting coefficient beta 3, and beta 2 < beta 3 < 1.5 is set; the third adjusting mode meets the condition that the first-level difference DeltaTax is larger than the second preset first-level response time length difference DeltaTax 2;
the central control module marks the maximum allowable value of the flow transmission for the operation connection after adjustment as C ', and sets C' =C multiplied by beta j, wherein j=1, 2 and 3;
the third preset condition is that the central control module adjusts a maximum allowable value of flow transmission for the operation connection to a corresponding value according to a difference value between the average response time length Ta0 and the first preset average response time length Ta 1;
where β1=1.15, β2=1.26, β3=1.44, tax 1=276, tax 2=552.
Specifically, the central control module detects, under a fourth preset condition, an average transmission time length of the operation connection between the user terminal and the secondary data warehouse in the preset time length, and for a kth operation connection respectively connected with the user terminal and the secondary data warehouse, the central control module marks the average transmission time of the operation connection in the preset time length as Tbk0, where k=1, 2,3, where m, m is a total number of operation connections connected with the user terminal and the secondary data warehouse, and sets Tb i is an ith response time length corresponding to an ith access request which is transmitted to the secondary data warehouse in a preset time length by the operation connection;
and the fourth preset condition is that the central control module adjusts the maximum allowable value of the flow transmission for the operation connection to C'.
Specifically, the central control module sequentially counts the average transmission duration of the data transmitted by each operation connection in the preset duration under the fourth preset condition to obtain the comprehensive transmission duration Tb of each operation connection in the preset duration, and determines a correction mode for the ratio of the comprehensive flow transmission value in each operation connection according to Tb, wherein the correction mode is set
The first correction mode is that the central control module sets the ratio of the comprehensive flow transmission value for each operation connection as a preset transmission value ratio B0, wherein B0 is the ratio of the maximum data transmission value allowed in a single operation connection to the maximum data transmission value which can be born by the operation connection; the first correction mode meets the condition that the comprehensive transmission duration Tb is smaller than or equal to a first preset comprehensive transmission duration Tb1 set in the central control module;
the second correction mode is that the central control module uses a first correction coefficient e1 to set a comprehensive flow transmission value ratio B1 for each operation connection, wherein e1 is more than 1.2 and less than 1.4, and B1=B0×e1; the second correction mode satisfies that the integrated transmission duration Tb is greater than the first preset integrated transmission duration Tb1 and less than or equal to a second preset integrated transmission duration Tb2 set in the central control module;
The third correction mode is that the central control module uses a second correction coefficient to set the comprehensive flow transmission value ratio B2, e1 < e2 < 1.5 for each operation connection, and B2=B0×e2; the third correction mode satisfies that the integrated transmission duration Tb is longer than the second preset integrated transmission duration Tb2;
wherein b0=0.61, e1=1.26, e2=1.45, tb1=0.38, tb2=1.2.
Specifically, the central control module calculates the difference DeltaTay between the average response time length Ta0 and the second preset average response time length Ta2 under a fifth preset condition, sets DeltaTay=Ta0-Ta 2, marks DeltaTay as the difference of the secondary response time length of the operation connection, determines the calling mode of the connection quantity connected with the secondary data warehouse according to DeltaTay,
the first calling mode is that the central control module uses a preset first regulating coefficient gamma 1 to change the number of the connections connected with the secondary data warehouse to a first number M1,1.3 < gamma 1 < 1.5, and the setting is carried outWherein M is the initial number of connections to the secondary data warehouse; the first calling mode meets the condition that the secondary difference value delta Tay is smaller than or equal to a preset secondary response time length difference value delta Tay0 set by the central control module;
The second calling mode is that the central control module uses a preset second regulating coefficient gamma 2 to change the number of the connections connected with the secondary data warehouse to a first number M2, gamma 1 is less than gamma 2 and less than 1.6, and the setting is carried outThe second calling mode satisfies that the secondary difference DeltaTay is larger than the preset secondary response time difference△Tay0；
The fifth preset condition is that the central control module judges that the operation connection connected with the secondary data warehouse does not meet a preset standard, and the central control module calls a corresponding number of new connections according to the difference value between the average response time length Ta0 and the second preset average response time length Ta2 so as to construct a plurality of data channels in the user side and the secondary data warehouse;
wherein Δtay0=3084, γ1=1.22, γ2=1.52.
Specifically, the central control module determines a secondary correction mode of the integrated flow transmission value ratio for each operation connection according to the number difference value between the newly added total connection number and the preset connection number standard under the sixth preset condition, wherein,
the first secondary correction mode is that the central control module uses the corrected comprehensive flow transmission value ratio Bz, wherein z=1, 2; the first secondary correction mode meets the quantity difference standard preset by the central control module;
The second secondary correction mode is to secondarily correct the corrected integrated flow transmission value ratio to Bz ', and Bz' =1.05×Bz is set; the second correction mode meets the condition that the quantity difference value is larger than a quantity difference value standard preset by the central control module;
the sixth preset condition is that the central control module judges that the newly added connection number is larger than a preset connection number standard in the connection pool;
wherein the number difference criterion is equal to 10.
Specifically, the central control module acquires corresponding keywords from the access request sent by the terminal so as to connect the terminal with the corresponding secondary data warehouse by using connection to construct a data channel of the corresponding secondary data warehouse in the user terminal and the MPP data warehouse.
Thus far, the technical solution of the present invention has been described in connection with the preferred embodiments shown in the drawings, but it is easily understood by those skilled in the art that the scope of protection of the present invention is not limited to these specific embodiments. Equivalent modifications and substitutions for related technical features may be made by those skilled in the art without departing from the principles of the present invention, and such modifications and substitutions will be within the scope of the present invention.
The foregoing description is only of the preferred embodiments of the invention and is not intended to limit the invention; various modifications and variations of the present invention will be apparent to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims (10)

1. A method for improving concurrency performance of MPP data warehouse accesses based on a high-availability connection pool, comprising:
when a single terminal in the user side sends an access request, the central control module invokes a single connection from the connection pool according to the information in the access request to construct a data channel of a corresponding secondary data warehouse in the user side and the MPP data warehouse; the central control module marks the connection as operation connection when the connection completes the construction of the data channels of the user side and the secondary data warehouse;
the central control module sequentially records an access time node when the terminal sends an access request, an output time node when the terminal outputs data from the secondary data warehouse and a receiving time node when the user receives the data when the construction of the data channels of the user side and the secondary data warehouse is completed, and the central control module obtains the response time of the operation connection for the access request according to the access time node and the output time node and obtains the transmission time of the operation connection for the access request according to the output time node and the receiving time node;
The central control module counts the response time and the transmission time of each access request of a single connection in a preset time to obtain the average response time and the average transmission time of the operation connection between the user terminal and the secondary data warehouse in the preset time, and judges whether the operation connection accords with a preset standard according to the average response time;
the central control module judges whether to adjust the maximum allowable value of the flow transmission in the operation connection or call a plurality of new connections to construct a plurality of data channels in the user terminal and the secondary data warehouse according to the average response time when judging that the single connection does not meet the preset standard, and judges whether to correct the ratio of the comprehensive flow transmission values in each operation connection according to the comprehensive transmission time of each operation connection after finishing the judgment;
and the central control module judges whether to add connection in the connection pool according to the total number of the connections to be used when judging that new connection is required to be called, and judges whether to carry out secondary correction on the comprehensive flow transmission value ratio of each operation connection when the number of the connections in the connection pool exceeds a preset connection number standard.
2. The method for improving concurrency performance of MPP data warehouse access based on high-availability connection pool of claim 1, wherein the central control module calculates the average response duration Ta0 of the operational connections connected to a single secondary data warehouse for the preset duration under a first preset condition, and sets upWherein n is the number of access requests transmitted by the operation connection within a preset time period, tai is an ith response time period corresponding to an ith access request transmitted by the operation connection to the secondary data warehouse within the preset time period, i=1, 2,3 is set, and n, ei is an ith time period correction coefficient for the ith response time period;
the first preset condition is that the central control module judges that the operation duration of the single operation connection reaches a preset duration.
3. The method for improving concurrency performance of MPP data warehouse access based on high-availability connectivity pool as set forth in claim 2, wherein the central control module determines a decision mode for the operational connectivity based on the average response time period Ta0 under the first preset condition, wherein,
the first judging mode is that the central control module judges that the operation connection of the operation connection meets a preset standard; the first judging mode meets the condition that the average response time length Ta0 of the operation connection in the preset time length is smaller than or equal to a first preset average response time length Ta1 preset in the central control module;
The second judging mode is that the central control module judges that the operation connection connected with the secondary data warehouse does not accord with a preset standard, and the central control module adjusts the maximum allowable value of the flow transmission for the operation connection to a corresponding value according to the difference value between the average response time length Ta0 and the first preset average response time length Ta 1; the second judgment mode meets the condition that the average response time length Ta0 is longer than the first preset response time length Ta1 and Ta0 is smaller than a second preset response time length Ta2 preset by the central control module;
the third judging mode is that the central control module judges that the operation connection of the secondary data warehouse connection does not accord with a preset standard, and the central control module calls a corresponding number of new connections according to the difference value between the average response time length Ta0 and the second preset average response time length Ta2 so as to construct a plurality of data channels in the user side and the secondary data warehouse; the third determination mode satisfies that the average response time Ta0 is greater than the second preset average response time Ta2.
4. The method for improving concurrency performance of MPP data warehouse access based on high-availability connection pool as set forth in claim 3, wherein said central control module sequentially determines the manner of assigning each time length correction factor based on the instantaneous data transmission amount of the operation connection at the time node at which the operation connection transmits the access request to said secondary data warehouse for a preset time length for the operation connection under the second preset condition, wherein for said i-th time length correction factor ei,
The first assignment mode is that the central control module assigns ei to 1.15; the first assignment mode satisfies that an ith instantaneous data transmission quantity Qi of the operation connection is smaller than or equal to a first preset instantaneous data transmission quantity Qa preset in the central control module when the terminal transmits an ith access request to the secondary data warehouse;
the second assignment mode is that the central control module assigns ei to 1; the second assignment mode satisfies that an ith instantaneous data transmission quantity Qi of the operation connection is larger than a first preset instantaneous data transmission quantity Qa preset in the central control module and Qi is smaller than or equal to a second preset instantaneous data transmission quantity Qb preset in the central control module when the terminal transmits an ith access request to the secondary data warehouse;
a third assignment mode is that the central control module assigns ei to 0.95; the third assignment mode satisfies that the ith instantaneous data transmission quantity Qi of the operation connection is larger than the second preset instantaneous data transmission quantity Qb when the terminal transmits the ith access request to the secondary data warehouse;
and the second preset condition is that the central control module calculates the average response time of the operation connection in the preset time.
5. The method for improving concurrency performance of an MPP data warehouse based on a high-availability connection pool as set forth in claim 4, wherein the central control module calculates a difference Δtax between the average response time period Ta0 and the first preset average response time period Ta1 under a third preset condition, sets Δtax=ta1-Ta 0, the central control module marks Δtax as a first-level response time period difference of the operation connection and determines an adjustment manner of a maximum allowable value of traffic transmission for the operation connection according to Δtax,
the first adjusting mode is that the central control module adjusts the maximum allowable value C of the flow transmission to a corresponding value by using a preset first preset adjusting coefficient beta 1, and the preset value is 1 < beta 1 < 1.2; the first adjusting mode meets the condition that the first-level difference DeltaTax is smaller than or equal to a first preset first-level response time length difference DeltaTax 1 set by the central control module;
the second adjusting mode is that the central control module adjusts the maximum allowable value C of the flow transmission to a corresponding value by using a second preset adjusting coefficient beta 2 preset, and beta 1 < beta 2 < 1.3 is set; the second adjustment mode meets the condition that the first-level difference value delta Tax is larger than the first preset first-level response time length difference value delta Tax1, and the delta Tax is smaller than or equal to a second preset first-level response time length difference value delta Tax2 set by the central control module;
The third adjusting mode is that the central control module adjusts the maximum allowable value C of the flow transmission to a corresponding value by using a preset third preset adjusting coefficient beta 3, and beta 2 < beta 3 < 1.5 is set; the third adjusting mode meets the condition that the first-level difference DeltaTax is larger than the second preset first-level response time length difference DeltaTax 2;
the central control module marks the maximum allowable value of the flow transmission for the operation connection after adjustment as C ', and sets C' =C multiplied by beta j, wherein j=1, 2 and 3;
the third preset condition is that the central control module adjusts the maximum allowable value of the flow transmission for the operation connection to a corresponding value according to the difference value between the average response time length Ta0 and the first preset average response time length Ta 1.
6. The method for improving concurrency performance for MPP data warehouse access based on high-availability connection pool of claim 5, wherein the central control module detects the average transmission time of the operation connection between the user terminal and the secondary data warehouse for the average transmission time of the operation connection between the user terminal and the secondary data warehouse in the preset time period under a fourth preset condition, and the central control module marks the average transmission time of the operation connection within the preset time period as Tbk0 for the kth operation connection respectively connected between the user terminal and the secondary data warehouse, wherein k = 1,2,3, where, m, m is the total number of operation connections connected between the user terminal and the secondary data warehouse, and is set as Wherein Tbi is an ith response time length corresponding to an ith access request which is transmitted to the secondary data warehouse in a preset time length by the operation connection;
and the fourth preset condition is that the central control module adjusts the maximum allowable value of the flow transmission for the operation connection to C'.
7. According to claimThe method for improving access concurrency performance of an MPP data warehouse based on a high-availability connection pool as set forth in claim 6, wherein the central control module sequentially counts average transmission time length of data transmitted by each operation connection in a preset time length under the fourth preset condition to obtain comprehensive transmission time length Tb of each operation connection in the preset time length, and determines a correction mode for a ratio of comprehensive flow transmission values in each operation connection according to Tb, wherein
The first correction mode is that the central control module sets the ratio of the comprehensive flow transmission value for each operation connection as a preset transmission value ratio B0, wherein B0 is the ratio of the maximum data transmission value allowed in a single operation connection to the maximum data transmission value which can be born by the operation connection; the first correction mode meets the condition that the comprehensive transmission duration Tb is smaller than or equal to a first preset comprehensive transmission duration Tb1 set in the central control module;
The second correction mode is that the central control module uses a first correction coefficient e1 to set a comprehensive flow transmission value ratio B1 for each operation connection, wherein e1 is more than 1.2 and less than 1.4, and B1=B0×e1; the second correction mode satisfies that the integrated transmission duration Tb is greater than the first preset integrated transmission duration Tb1 and less than or equal to a second preset integrated transmission duration Tb2 set in the central control module;
the third correction mode is that the central control module uses a second correction coefficient to set the comprehensive flow transmission value ratio B2, e1 < e2 < 1.5 for each operation connection, and B2=B0×e2; the third modification manner satisfies that the integrated transmission duration Tb is longer than the second preset integrated transmission duration Tb2.
8. The method for improving concurrency performance of MPP data warehouse access based on high-availability connection pool of claim 7, wherein said central control module calculates a difference Δtay between average response time Ta0 and said second preset average response time Ta2 under a fifth preset condition, sets Δtay = Ta0-Ta2, and wherein central control module marks Δtay as a difference in secondary response time of said operational connection and determines a calling manner for said number of connections to said secondary data warehouse based on Δtay,
The first calling mode is that the central control module uses a preset first regulating coefficient gamma 1 to change the number of the connections connected with the secondary data warehouse to a first number M1,1.3 < gamma 1 < 1.5, and the setting is carried outWherein M is the initial number of connections to the secondary data warehouse; the first calling mode meets the condition that the secondary difference value delta Tay is smaller than or equal to a preset secondary response time length difference value delta Tay0 set by the central control module;
the second calling mode is that the central control module uses a preset second regulating coefficient gamma 2 to change the number of the connections connected with the secondary data warehouse to a first number M2, gamma 1 is less than gamma 2 and less than 1.6, and the setting is carried outThe second calling mode meets the condition that the secondary difference value delta Tay is larger than the preset secondary response time length difference value delta Tay0;
and the fifth preset condition is that the central control module judges that the operation connection connected with the secondary data warehouse does not meet a preset standard, and the central control module calls a corresponding number of new connections according to the difference value between the average response time length Ta0 and the second preset average response time length Ta2 so as to construct a plurality of data channels in the user side and the secondary data warehouse.
9. The method for improving concurrency performance of an MPP data warehouse based on a pool of highly available connections as set forth in claim 8, wherein said central control module determines a quadratic modification of the ratio of integrated traffic transfer values for each of said operational connections based on the number difference of the newly added total number of connections to said preset number of connections criteria under said sixth preset condition,
The first secondary correction mode is that the central control module uses the corrected comprehensive flow transmission value ratio Bz, wherein z=1, 2; the first secondary correction mode meets the quantity difference standard preset by the central control module;
the second secondary correction mode is to secondarily correct the corrected integrated flow transmission value ratio to Bz ', and Bz' =1.05×Bz is set; the second correction mode meets the condition that the quantity difference value is larger than a quantity difference value standard preset by the central control module;
and the sixth preset condition is that the central control module judges that the newly added connection number is larger than a preset connection number standard in the connection pool.
10. The method for improving access concurrency performance of an MPP data warehouse based on a high-availability connection pool as set forth in claim 1, wherein the central control module obtains a corresponding keyword from an access request sent by the terminal to connect the terminal with the corresponding secondary data warehouse using a connection to construct a data channel between the user terminal and the corresponding secondary data warehouse in the MPP data warehouse.
CN202310151902.4A 2023-02-15 2023-02-15 Method for improving access concurrency performance of MPP data warehouse based on high-availability connection pool Pending CN117290329A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310151902.4A CN117290329A (en) 2023-02-15 2023-02-15 Method for improving access concurrency performance of MPP data warehouse based on high-availability connection pool

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310151902.4A CN117290329A (en) 2023-02-15 2023-02-15 Method for improving access concurrency performance of MPP data warehouse based on high-availability connection pool

Publications (1)

Publication Number Publication Date
CN117290329A true CN117290329A (en) 2023-12-26

Family

ID=89239650

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310151902.4A Pending CN117290329A (en) 2023-02-15 2023-02-15 Method for improving access concurrency performance of MPP data warehouse based on high-availability connection pool

Country Status (1)

Country Link
CN (1) CN117290329A (en)

Similar Documents

Publication Publication Date Title
CN1290372C (en) Class based bandwidth scheduling for CDMA air interfaces
JPH11284570A (en) Power control method
US20030112762A1 (en) Method of radio resource management for integrated voice and data CDMA networks
CN114785565B (en) Data security exchange system based on network boundary
CN117290329A (en) Method for improving access concurrency performance of MPP data warehouse based on high-availability connection pool
US7492784B2 (en) Call acceptance control method, mobile communication system and base station device
WO2021098403A1 (en) Resource allocation method, server, and storage medium
CN109450585B (en) Low-delay GMP mapping method and system based on state machine self-adaptive control
US6775233B1 (en) Rate processor sharing method and apparatus for scheduling data transmissions in a CDMA wireless communication system
EP2717616B1 (en) Random access method, mobile terminal, and mobile communication system
CN110858956B (en) Processing method, device, equipment and medium for IP short message gateway load unbalance
US20040209636A1 (en) Smooth method for adjusting downlink transmitted power
CN115617860A (en) Intelligent query method and system based on big data
US7228106B2 (en) Method and apparatus for adjusting usage of DSP of handheld terminal
US20170244641A1 (en) Transmission control method and apparatus for network services and controller
CN115915369A (en) PUCCH power control method, terminal, device and storage medium
CN111416869A (en) Method for adjusting node weight in distributed cluster
CN113301170A (en) Data processing method based on communication node
CN109639459B (en) Method for adjusting rated bandwidth, terminal and storage medium
US20110249565A1 (en) Systems and Methods for Adapting Network Capacity Assigned to a Partition
US7181235B2 (en) Hybrid method for adjusting downlink transmitted power
CN116074262B (en) Resource optimization allocation method based on big data platform
CN117061584B (en) Audio transmission method based on dual-channel link
CN114726729B (en) Admission control method and system of wireless access network facing network slicing
CN111970544B (en) Downlink flow scheduling policy adjusting method, server and readable storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination