CN117539947A - Data source dynamic adaptation method and system based on lightweight platform - Google Patents

Data source dynamic adaptation method and system based on lightweight platform Download PDF

Info

Publication number
CN117539947A
CN117539947A CN202410009688.3A CN202410009688A CN117539947A CN 117539947 A CN117539947 A CN 117539947A CN 202410009688 A CN202410009688 A CN 202410009688A CN 117539947 A CN117539947 A CN 117539947A
Authority
CN
China
Prior art keywords
database
data
access
interceptor
data source
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202410009688.3A
Other languages
Chinese (zh)
Other versions
CN117539947B (en
Inventor
吴浩然
朱晶晶
黄永洋
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Guancheng Information Technology Suzhou Co ltd
Original Assignee
Guancheng Information Technology Suzhou Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Guancheng Information Technology Suzhou Co ltd filed Critical Guancheng Information Technology Suzhou Co ltd
Priority to CN202410009688.3A priority Critical patent/CN117539947B/en
Publication of CN117539947A publication Critical patent/CN117539947A/en
Application granted granted Critical
Publication of CN117539947B publication Critical patent/CN117539947B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/25Integrating or interfacing systems involving database management systems
    • G06F16/252Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/30Authentication, i.e. establishing the identity or authorisation of security principals
    • G06F21/31User authentication
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/60Protecting data
    • G06F21/62Protecting access to data via a platform, e.g. using keys or access control rules
    • G06F21/6218Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44Arrangements for executing specific programs
    • G06F9/445Program loading or initiating
    • G06F9/44505Configuring for program initiating, e.g. using registry, configuration files
    • G06F9/4451User profiles; Roaming
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02DCLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Software Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Databases & Information Systems (AREA)
  • Computer Security & Cryptography (AREA)
  • Computer Hardware Design (AREA)
  • Data Mining & Analysis (AREA)
  • Health & Medical Sciences (AREA)
  • Bioethics (AREA)
  • General Health & Medical Sciences (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention provides a data source dynamic adaptation method based on a lightweight platform, which relates to the technical field of data source adaptation and comprises the following steps: acquiring user requirements, dynamically loading configuration classes, generating corresponding data adapters based on the user requirements through a data source analyzer, acquiring original addresses and database passwords corresponding to a database according to the data adapters, adapting manufacturer data materials in the database, creating a database dividing mark, generating an optimal adaptation scheme, introducing a responsibility chain mechanism when the front end accesses the rear end, setting interceptor responsibility for each interceptor in the rear end, judging whether interception is needed or not through a context object, and calling according to an access request of the database to acquire database access permission; and generating a database management rule through a dynamic management algorithm, acquiring a data source in the database, inquiring and editing the data source in the database, and returning database data corresponding to the database mark if the corresponding database mark is inquired.

Description

Data source dynamic adaptation method and system based on lightweight platform
Technical Field
The invention relates to the technical field of data source adaptation, in particular to a data source dynamic adaptation method and system based on a lightweight platform.
Background
Under the condition of diversified business systems, the requirements and the systems for data processing and data application are more and more, the problems of inconsistent database grammar, dynamic loading driving and dynamic expansion driving need to be solved for the dynamic multiple data sources in the butt joint, and the development process is complex and easy to make mistakes.
In the prior art, CN109299150a discloses a solution for a configurable multi-data source adaptation rule engine, comprising the steps of: configuring multiple data sources, configuring the number of active rules according to the configured multiple data sources, triggering the running of the rules by the user behavior generation event MQ after the rules are in online effect, and judging the execution behavior according to the rule result.
CN108846118A discloses a method for realizing intelligent adaptation to multiple data sources, which relates to the technical field of data source processing, and comprises the following steps: step 1, configuring key information of a data source; step 2, defining a method name of a calling method according to key information of a data source to be controlled; and step 3, matching the method name of the calling method with key information of the data source, and determining the data source used by the calling method according to the matching result.
In summary, although the prior art can realize the adaptation to multiple data sources, the prior art cannot automatically adapt according to the access requirement, and needs to manually adjust options and codes, so that a scheme for automatically adapting the data sources is needed to solve the defects in the prior art.
Disclosure of Invention
The embodiment of the invention provides a data source dynamic adaptation method and a system based on a lightweight platform, which are used for carrying out real-time adaptation on different data sources based on user access requirements, accessing a database and extracting data.
In a first aspect of the embodiments of the present invention, a method for dynamically adapting a data source based on a lightweight platform is provided, including:
acquiring user requirements, dynamically loading configuration classes, generating corresponding data adapters based on the user requirements through a data source analyzer, acquiring original addresses and database passwords corresponding to a database according to the data adapters, adapting manufacturer data materials in the database through a data source adaptation algorithm, creating a database dividing mark and generating an optimal adaptation scheme;
according to the optimal adaptation scheme, introducing a responsibility chain mechanism when the front end accesses the rear end, setting corresponding interceptor responsibility for each interceptor in the rear end, judging whether interception is needed or not through a pre-introduced context object, and calling according to an access request of the database to acquire a database access permission;
and generating a database management rule through a dynamic management algorithm according to the database access permission, acquiring a data source in the database through the database management rule, inquiring and editing the data source in the database, and returning the database data corresponding to the database dividing mark if the database dividing mark is inquired.
In an alternative embodiment of the present invention,
the obtaining the user requirement, dynamically loading the configuration class, and generating the corresponding data adapter based on the user requirement through the data source analyzer comprises the following steps:
acquiring a database to be accessed according to user requirements, generating a configuration class, creating a configuration manager, monitoring an event according to the configuration manager, and reloading the configuration file and updating the configuration class if the configuration file changes;
according to the updated configuration class, a data source analyzer is selected, the data source analyzer obtains dynamic parameters of each configuration item in the configuration class, determines parameters corresponding to each interface of the data adapter according to the dynamic parameters, generates the data adapter, and registers the data adapter into a system through a configuration file.
In an alternative embodiment of the present invention,
the step of obtaining the original address and the database password corresponding to the database according to the data adapter, and adapting manufacturer data in the database through a data source adaptation algorithm, wherein the step of creating a database dividing mark and generating an adaptation scheme comprises the following steps:
according to the data adapter, acquiring a database address and a database password corresponding to a database matched with the user demand, generating an objective function based on a data source adaptation algorithm according to the database address and the database password, and taking the database address and the database password as an optimization target of the objective function;
Randomly generating an initial adaptation scheme as an initial solution of the objective function, setting initial temperature corresponding to the initial adaptation scheme, reducing the initial temperature, carrying out neighborhood search according to the initial adaptation scheme for each different temperature, replacing the initial adaptation scheme if a new adaptation scheme is better than the initial adaptation scheme, repeating the search until the search is completed, marking the adaptation scheme when the search is completed as an optimal adaptation scheme, marking a database corresponding to the optimal adaptation scheme as a required database, and generating a corresponding database dividing mark.
In an alternative embodiment of the present invention,
according to the optimal adaptation scheme, introducing a responsibility chain mechanism when the front end accesses the back end, setting corresponding interceptor responsibility for each interceptor in the back end, judging whether interception is needed or not through a pre-introduced context object, and calling according to the database access request, wherein the acquiring the database access permission comprises the following steps:
according to the optimal adaptation scheme, the front end performs data access to the rear end, a service control layer in the rear end intercepts the data access through interceptors, corresponding interceptor responsibilities are set for each interceptor in the rear end according to the responsibility chain mechanism, and the interceptors are connected according to the execution sequence according to the interceptor responsibilities to form an organic responsibility chain;
According to the organic responsibility chain, when a database access request is obtained, processing logic of a first interceptor is executed from the head of the organic responsibility chain, and whether to terminate the request or transmit the request to the next interceptor is judged according to the corresponding interceptor responsibility until all interceptors are executed or the database access request is intercepted;
and if the database access request is intercepted, generating corresponding return data according to the interceptors generating the interception, and if all interceptors are executed and the interception is not generated, generating a database access permission according to the database access request.
In an alternative embodiment of the present invention,
the organic responsibility chain further comprises an interceptor dynamic management scheme:
generating an interceptor interface in the organic responsibility chain according to the interceptor dynamic management scheme, and setting an interceptor manager on the interceptor interface, wherein the interceptor manager judges an entry point, an exit point and an interceptor execution sequence of the user demand according to the user demand;
according to the entry point and the exit point, deleting the interceptors positioned before the entry point and the interceptors positioned after the exit point in the organic responsibility chain, acquiring an interceptor list according to the execution sequence of the interceptors, and adding or deleting the interceptors at specific positions.
In an alternative embodiment of the present invention,
generating a database management rule through a dynamic management algorithm according to the database access permission, acquiring a data source in the database through the database management rule, inquiring and editing the data source in the database, and returning database data corresponding to the database separation mark if the database separation mark is inquired, wherein the database data corresponding to the database separation mark comprises the following steps:
acquiring a configuration class corresponding to the database according to the database access permission, and setting a corresponding initial configuration rule for each configuration item in the configuration class by combining the access frequency, the access time window and the access data volume limit of the database in the system through a dynamic management algorithm introduced in advance;
according to the initial configuration rule, historical access data are obtained, the performance of a database is predicted according to a performance optimization model in the dynamic management algorithm and the historical access data, a performance prediction result is obtained, the initial configuration rule is optimized according to the performance prediction result, corresponding effective conditions are added, an optimal configuration rule is obtained, and the operation is repeated until the preset iteration times are reached;
summarizing the optimal configuration rule corresponding to each configuration item, generating the database management rule, calling a data source in the database by the rear end according to the database management rule, inquiring whether the data source contains a preset database dividing mark or not according to the data source, returning database data corresponding to the database dividing mark if the database dividing mark is contained, and returning an empty inquiry result if the database dividing mark is not contained.
In an alternative embodiment of the present invention,
the setting of the corresponding initial configuration rule for each configuration item in the configuration class by combining the access authority, the access time window and the access data volume limit of the database in the system through the pre-introduced dynamic management algorithm comprises the following steps:
acquiring the access frequency, access time window and access data volume of the database, and determining the life cycle and modification authority of the initial configuration rule;
collecting system access quantity and database load in real time according to the dynamic management algorithm, and performing aggregation operation and window operation on the system access quantity and the database load collected in real time through a stream processing engine to obtain access peaks and access valleys;
and generating a primary configuration rule according to the access peak value and the access valley value by combining an access time window and a database load.
In a second aspect of the embodiments of the present invention, a data source dynamic adaptation system based on a lightweight platform is provided, including:
the first unit is used for acquiring user demands, dynamically loading configuration classes, generating corresponding data adapters based on the user demands through a data source analyzer, acquiring original addresses and database passwords corresponding to a database according to the data adapters, adapting manufacturer data materials in the database through a data source adaptation algorithm, creating a database dividing mark and generating an optimal adaptation scheme;
The second unit is used for introducing a responsibility chain mechanism when the front end accesses the rear end according to the optimal adaptation scheme, setting corresponding interceptor responsibility for each interceptor in the rear end, judging whether interception is needed or not through a context object which is introduced in advance, and calling according to an access request of the database to acquire a database access permission;
and the third unit is used for generating a database management rule through a dynamic management algorithm according to the database access permission, acquiring a data source in the database through the database management rule, inquiring and editing the data source in the database, and returning the database data corresponding to the database dividing mark if the database dividing mark is inquired.
In a third aspect of an embodiment of the present invention,
there is provided an electronic device including:
a processor;
a memory for storing processor-executable instructions;
wherein the processor is configured to invoke the instructions stored in the memory to perform the method described previously.
In a fourth aspect of an embodiment of the present invention,
there is provided a computer readable storage medium having stored thereon computer program instructions which, when executed by a processor, implement the method as described above.
According to the invention, the system can dynamically load the configuration class by acquiring the user requirement, so that the system can dynamically adjust the configuration according to the specific requirement of the user when running, the flexibility and the adaptability are improved, and the original address and the password of the database are acquired through the data adapter. The system can be connected to a correct database according to user requirements, so that effective adaptation and transmission of data are realized, whether interception is needed or not is judged through a pre-introduced context object, the system can realize flexible request processing flow, each interceptor is ensured to work at a correct time, a database management rule is generated through a dynamic management algorithm, a data source in the database is acquired according to the rule, the system can inquire and edit the database according to the rule, and the rationality and flexibility of database management are ensured.
Drawings
FIG. 1 is a schematic flow chart of a data source dynamic adaptation method based on a lightweight platform according to an embodiment of the invention;
Fig. 2 is a schematic structural diagram of a data source dynamic adaptation system based on a lightweight platform according to an embodiment of the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The technical scheme of the invention is described in detail below by specific examples. The following embodiments may be combined with each other, and some embodiments may not be repeated for the same or similar concepts or processes.
Fig. 1 is a schematic flow chart of a data source dynamic adaptation method based on a lightweight platform according to an embodiment of the present invention, as shown in fig. 1, the method includes:
s1, obtaining user demands, dynamically loading configuration classes, generating corresponding data adapters based on the user demands through a data source analyzer, obtaining original addresses and database passwords corresponding to a database according to the data adapters, adapting manufacturer data materials in the database through a data source adaptation algorithm, creating a database dividing mark and generating an optimal adaptation scheme;
The configuration class is a class used for storing and managing system configuration information and comprises database connection information, log level, cache configuration and the like, the data source analyzer is a component used for analyzing data source information in configuration files or other configuration forms, the data source adapter is a component used for interacting with different types of data sources, the data source adaptation algorithm is an algorithm or rule used for adapting to differences among different data sources, the data type mapping and SQL dialect are related to the differences, and the database splitting mark is an mark used in a distributed database or a database splitting and table architecture and used for guiding how data is divided into different databases or tables.
In an alternative embodiment of the present invention,
the obtaining the user requirement, dynamically loading the configuration class, and generating the corresponding data adapter based on the user requirement through the data source analyzer comprises the following steps:
acquiring a database to be accessed according to user requirements, generating a configuration class, creating a configuration manager, monitoring an event according to the configuration manager, and reloading the configuration file and updating the configuration class if the configuration file changes;
according to the updated configuration class, a data source analyzer is selected, the data source analyzer obtains dynamic parameters of each configuration item in the configuration class, determines parameters corresponding to each interface of the data adapter according to the dynamic parameters, generates the data adapter, and registers the data adapter into a system through a configuration file.
Defining a database configuration class, comprising database connection information, such as a host, a port, a user name, a password and the like, receiving configuration file path parameters, loading initial configuration information in an initialization stage, providing a method for loading the configuration information and a configuration change monitoring method, starting a monitoring mechanism, monitoring the change of a configuration file, calling a processing method of a configuration manager to process a configuration change event when the configuration file is changed, reloading configuration, generating a new database configuration class, updating configuration in the configuration manager if the new configuration is different from the old configuration, and triggering a configuration update event;
creating a data source analyzer, adding a dynamic parameter analyzing method, analyzing according to the dynamic parameters in the database configuration class, creating a data adapter and receiving the dynamic parameters, establishing database connection during initialization, setting a method corresponding to the database connection and query operation, analyzing configuration by using the data source analyzer in configuration change processing, generating the dynamic parameters, generating a new data adapter by using the dynamic parameters, and registering the new data adapter in the system.
In this embodiment, by combining the configuration class and the configuration manager, the system can flexibly acquire and update the configuration information of the database according to the user requirements, so that the system can easily adapt to different database types and connection information without modifying codes, the event monitoring mechanism of the configuration manager enables the system to sense the change of the configuration file in real time, once the configuration file changes, the system can quickly reload the configuration, ensure the timely update of the configuration information, and dynamically select and switch the adapter during running by registering the data adapter with the configuration file without redeploying an application, so that the system is more flexible, can adapt to database changes and new data source accesses, and in combination, the flexibility and maintainability of the system can be obviously improved, the system is suitable for coping with the continuously changing database connection requirements, and the complexity of system maintenance and upgrade is reduced.
In an alternative embodiment of the present invention,
the step of obtaining the original address and the database password corresponding to the database according to the data adapter, and adapting manufacturer data in the database through a data source adaptation algorithm, wherein the step of creating a database dividing mark and generating an adaptation scheme comprises the following steps:
according to the data adapter, acquiring a database address and a database password corresponding to a database matched with the user demand, generating an objective function based on a data source adaptation algorithm according to the database address and the database password, and taking the database address and the database password as an optimization target of the objective function;
randomly generating an initial adaptation scheme as an initial solution of the objective function, setting initial temperature corresponding to the initial adaptation scheme, reducing the initial temperature, carrying out neighborhood search according to the initial adaptation scheme for each different temperature, replacing the initial adaptation scheme if a new adaptation scheme is better than the initial adaptation scheme, repeating the search until the search is completed, marking the adaptation scheme when the search is completed as an optimal adaptation scheme, marking a database corresponding to the optimal adaptation scheme as a required database, and generating a corresponding database dividing mark.
The neighborhood search is an optimization algorithm used for searching an optimal solution in a search space, the algorithm tries to generate a new solution by slightly changing in the neighborhood of the current solution, and decides whether to accept the new solution according to a certain rule, and the requirement database is divided into a plurality of parts according to a certain rule or algorithm and each part is called a database according to the system requirement and an optimization target.
The data adapter is utilized to acquire a database address and a database password matched with the data adapter from user demands, an objective function is designed based on a data source adaptation algorithm, information such as database connection performance, stability and load balancing is considered, an integrated evaluation index is defined by way of example, factors such as connection speed, response time and load balancing are comprehensively considered to form the objective function, the acquired database address and database password are used as optimization targets of the objective function, input of the objective function is the database address and password, and output of the objective function is used as an evaluation index.
An initial adaptation scheme of a group of database addresses and passwords is randomly generated, initial temperatures are set, the initial temperatures are generally higher, solutions with larger differences are accepted at the initial stage of searching, the initial temperatures are gradually reduced by defining a cooling function, neighborhood searching is conducted for each different temperature, a new adaptation scheme is generated in the neighborhood of the current adaptation scheme, the new solution can be obtained by slightly changing the current solution, the difference of objective function values of the new and old adaptation schemes is calculated, namely whether the new solution is superior to the current solution or not is evaluated, the new solution is accepted according to probability, the probability is reduced along with the reduction of the temperature, the reduction range of the acceptance probability along with the reduction of the temperature can be adjusted according to different requirements, the searching process is repeated until searching is completed, in the searching process, the optimal adaptation scheme in each iteration is recorded, the optimal adaptation scheme refers to the solution with the minimum objective function value at the current temperature, after the final searching is completed, the database corresponding to the optimal adaptation scheme when the searching is completed is marked as a required database, a corresponding database mark is generated, and the required database is marked.
In this embodiment, the objective function generated based on the data source adaptation algorithm comprehensively considers a plurality of factors such as connection performance and security, so that the system can optimize according to various requirements, the overall system performance is improved, the database corresponding to the optimal adaptation scheme is identified as a required sub-database, the system is described to pay attention to the optimization of the database address and the password, the requirements of the sub-database are considered, the overall structure of the database is optimized, the query efficiency and the load balance are improved, the initial temperature is set by randomly generating the initial adaptation scheme, the searching is performed in a mode of gradually reducing the temperature, the global searching is performed in a solution space to find a better adaptation scheme, and in sum, the embodiment realizes the dynamic adjustment of the database connection information and the sub-database requirements, improves the adaptability, the flexibility and the performance of the system, and can better meet the database requirements in different application scenarios.
S2, introducing a responsibility chain mechanism when the front end accesses the rear end according to the optimal adaptation scheme, setting corresponding interceptor responsibility for each interceptor in the rear end, judging whether interception is required or not through a pre-introduced context object, and calling according to an access request of the database to acquire a database access permission;
The chain of responsibilities is a behavioral design model that aims to pass requests from one processor to the next until the request is processed or the end of the chain is reached, each processor being responsible for determining whether the request can be processed and whether the request is passed to the next processor, the interceptor, a design model applied in software development that allows for intervention and execution of additional operations during processing of the request, typically used to perform certain operations before or after the request reaches the target object, such as authentication, logging, performance monitoring, etc., the context object is an object that holds and passes information related to the current execution state, typically including information about the environment, data, configuration, etc., of the current execution, and may be passed and shared between different components of the system.
In an alternative embodiment of the present invention,
according to the optimal adaptation scheme, introducing a responsibility chain mechanism when the front end accesses the back end, setting corresponding interceptor responsibility for each interceptor in the back end, judging whether interception is needed or not through a pre-introduced context object, and calling according to the database access request, wherein the acquiring the database access permission comprises the following steps:
According to the optimal adaptation scheme, the front end performs data access to the rear end, a service control layer in the rear end intercepts the data access through interceptors, corresponding interceptor responsibilities are set for each interceptor in the rear end according to the responsibility chain mechanism, and the interceptors are connected according to the execution sequence according to the interceptor responsibilities to form an organic responsibility chain;
according to the organic responsibility chain, when a database access request is obtained, processing logic of a first interceptor is executed from the head of the organic responsibility chain, and whether to terminate the request or transmit the request to the next interceptor is judged according to the corresponding interceptor responsibility until all interceptors are executed or the database access request is intercepted;
and if the database access request is intercepted, generating corresponding return data according to the interceptors generating the interception, and if all interceptors are executed and the interception is not generated, generating a database access permission according to the database access request.
The organic responsibility chain is a variant of the responsibility chain mode, emphasizes the cooperation and organic connection between all nodes (interceptors) in the responsibility chain, and aims to enable all nodes to be more flexible and dynamic, and the connection mode of the nodes can be dynamically adjusted according to the requirements of the system so as to adapt to different business scenes.
Designing a class for each interceptor, realizing an interceptor interface, defining an interception logic, creating an interceptor responsibility chain object for maintaining and executing the interceptor, defining an interceptor list in the responsibility chain object, setting corresponding interceptor responsibilities for each interceptor including authority verification, log recording, security check and the like, and connecting the interceptors according to the execution sequence by adding interceptor examples in the responsibility chain object according to the execution sequence to form an organic responsibility chain;
when the system receives a database access request, a service control layer acquires interceptor responsibility chain objects, processing logic of a first interceptor is executed from the head of the responsibility chain, each interceptor executes the interceptor one by one according to the order of the responsibility chain, processes specific logic, judges whether to terminate the request or transmit the request to the next interceptor according to the responsibility of each interceptor, and if a certain interceptor judges that the request needs to be terminated, the responsibility chain does not transmit the request any more;
if a certain interceptor judges that the database access request needs to be intercepted, corresponding return data is generated according to logic of the interceptor, the return data comprises error information and interception prompts, if all interceptors do not intercept after the execution is finished, the database access request is proved to accord with permission conditions, and the system generates database access permission according to the database access request.
In this embodiment, a responsibility chain mechanism and an organic responsibility chain design are adopted, so that the interceptors can cooperatively work to form a flexible and organic responsibility chain, the system can dynamically adjust the connection mode of the interceptors according to specific service requirements so as to adapt to different service scenarios, through the interceptor responsibility chain, the system realizes fine-grained control of database access requests, each interceptor can complete specific tasks such as authority verification, log record and the like, thereby better meeting the functional requirements of the system, and through dynamic adaptation, responsibility chain and interception processing, the system can improve the security and ensure that only legal requests can access the database. Meanwhile, the stability of the system is enhanced, abnormal conditions can be flexibly processed through the interceptor, and system breakdown or data leakage caused by improper database access is prevented.
In an alternative embodiment of the present invention,
the organic responsibility chain further comprises an interceptor dynamic management scheme:
Generating an interceptor interface in the organic responsibility chain according to the interceptor dynamic management scheme, and setting an interceptor manager on the interceptor interface, wherein the interceptor manager judges an entry point, an exit point and an interceptor execution sequence of the user demand according to the user demand;
according to the entry point and the exit point, deleting the interceptors positioned before the entry point and the interceptors positioned after the exit point in the organic responsibility chain, acquiring an interceptor list according to the execution sequence of the interceptors, and adding or deleting the interceptors at specific positions.
Defining interceptor interfaces, including execution methods and necessary attributes, to ensure that each interceptor meets interface specifications, creating an interceptor manager, wherein the interceptor manager is used for dynamically managing the interceptors according to user requirements, and analyzing an entry point, an exit point and an interceptor execution sequence according to the user requirements;
according to the interceptor manager and the user requirements, an interceptor list is generated, and is ordered according to the execution sequence, according to the entry point and the exit point, the interceptors positioned before the entry point and after the exit point in the organic responsibility chain are deleted, according to the interceptor execution sequence, the generated interceptor list is inserted into a specific position of the organic responsibility chain, if necessary, the interceptors in the original organic responsibility chain are deleted, the interceptors are instantiated and added into the organic responsibility chain, each interceptor is ensured to be normally executed, and when a database access request exists, the organic responsibility chain is executed according to the new interceptor configuration sequence.
In this embodiment, by means of a dynamic management scheme of the interceptors, the system can flexibly configure the interceptors in the organic responsibility chain according to the user requirements, so as to realize dynamic adaptation of the database access request, ensure that the system can flexibly configure the interceptors according to different requirements during operation, improve the dynamic adaptability of the system, and enable the system to adapt to different user requirements by means of requirement analysis and the interceptor manager to customize the entry points, the exit points and the execution sequence of the interceptors, thereby improving the user experience and the customized support.
S3, generating a database management rule through a dynamic management algorithm according to the database access permission, acquiring a data source in the database through the database management rule, inquiring and editing the data source in the database, and returning database data corresponding to the database separation mark if the database separation mark is inquired
The dynamic management algorithm is an algorithm for adjusting system behavior in real time or in a running mode, and is generally used for adaptively adjusting system configuration, resource allocation or behavior strategies according to environment, user requirements or other dynamically changing factors in the running mode, and the database management rule is a set of definition and convention used for guiding and standardizing design, maintenance, access and protection of a database, and generally comprises requirements and suggestions on aspects of database structure, data integrity, safety, performance optimization and the like.
In an alternative embodiment of the present invention,
generating a database management rule through a dynamic management algorithm according to the database access permission, acquiring a data source in the database through the database management rule, inquiring and editing the data source in the database, and returning database data corresponding to the database separation mark if the database separation mark is inquired, wherein the database data corresponding to the database separation mark comprises the following steps:
acquiring a configuration class corresponding to the database according to the database access permission, and setting a corresponding initial configuration rule for each configuration item in the configuration class by combining the access frequency, the access time window and the access data volume limit of the database in the system through a dynamic management algorithm introduced in advance;
According to the initial configuration rule, historical access data are obtained, the performance of a database is predicted according to a performance optimization model in the dynamic management algorithm and the historical access data, a performance prediction result is obtained, the initial configuration rule is optimized according to the performance prediction result, corresponding effective conditions are added, an optimal configuration rule is obtained, and the operation is repeated until the preset iteration times are reached;
summarizing the optimal configuration rule corresponding to each configuration item, generating the database management rule, calling a data source in the database by the rear end according to the database management rule, inquiring whether the data source contains a preset database dividing mark or not according to the data source, returning database data corresponding to the database dividing mark if the database dividing mark is contained, and returning an empty inquiry result if the database dividing mark is not contained.
Obtaining database access permission from a system, wherein the database access permission comprises key information such as a database address, a user name, a password and the like, obtaining configuration types related to the database, wherein the configuration types comprise configuration items related to database connection and access, such as connection pool size and overtime setting, introducing a dynamic management algorithm in advance, predicting and analyzing according to parameters such as access frequency, access time window and access data volume limitation and the like of the database in the system and the overall load condition of the system, setting initial configuration rules according to the prediction result of the dynamic management algorithm for each configuration item, including dynamic adjustment of the connection pool size, optimization of overtime setting and applying the initial configuration rules to the database configuration types;
Acquiring historical access data from database access records or logs, wherein the historical access data comprises key performance indexes such as access frequency, access time window, response time, error rate and the like, acquiring a current database configuration rule according to a previously set initial configuration rule, establishing a prediction model of database performance by combining the historical access data based on a performance optimization model in a dynamic management algorithm, wherein the performance optimization model is used for predicting performance under different configuration rules, predicting the performance under the current database configuration rule by using the established performance prediction model to obtain a performance prediction result, optimizing the initial configuration rule by adjusting database configuration parameters such as the size of a connection pool, optimizing query sentences, increasing indexes and the like according to the performance prediction result, adding effective conditions for the optimal configuration rule, ensuring that the rule is applied only under specific conditions, repeating the execution of the prediction and optimization, and continuously optimizing the configuration rule through an iteration process until the preset iteration times are reached.
Summarizing the optimal configuration rules subjected to iterative optimization for each configuration item, integrating the optimal configuration rules corresponding to each configuration item into database management rules, ensuring that all necessary configuration information and optimization strategies are contained, calling corresponding databases according to the database management rules in back-end service, inquiring whether a data source contains a preset database dividing mark for each database, and if the data source contains the preset database dividing mark, continuing to execute the next step; otherwise, returning an empty query result, which indicates that no related data exists in the database, and determining a specific database to be accessed according to a preset database dividing mark. And calling a corresponding data source, executing query operation, acquiring database data corresponding to the database dividing mark, and returning the data obtained by query to the back-end service for further processing or transmitting to the front-end.
In this embodiment, by means of a dynamic management algorithm introduced in advance, the system can dynamically adjust an initial configuration rule of each configuration item in the configuration class according to access frequency, access time window, access data volume limitation and the like of the database, so that the system can flexibly adapt to different database access requirements, the configuration dynamic property and adaptability are improved, the system performs performance prediction and optimizes the configuration rule through historical access data and a performance optimization model, the system can continuously improve the database performance through iterative optimization, and optimal configuration can be realized under different running situations, so that the response speed and resource utilization rate of the system are improved.
In an alternative embodiment of the present invention,
the setting of the corresponding initial configuration rule for each configuration item in the configuration class by combining the access authority, the access time window and the access data volume limit of the database in the system through the pre-introduced dynamic management algorithm comprises the following steps:
acquiring the access frequency, access time window and access data volume of the database, and determining the life cycle and modification authority of the initial configuration rule;
Collecting system access quantity and database load in real time according to the dynamic management algorithm, and performing aggregation operation and window operation on the system access quantity and the database load collected in real time through a stream processing engine to obtain access peaks and access valleys;
and generating a primary configuration rule according to the access peak value and the access valley value by combining an access time window and a database load.
Acquiring access frequency, access time window and access data volume from a database management system, determining the life cycle of an initial configuration rule, namely defining the effective period of the configuration rule, setting authority to modify the initial configuration rule, ensuring the stability and compliance of configuration, utilizing a dynamic management algorithm, collecting the access volume of the system and the load condition of a database in real time by monitoring system logs and database performance indexes, utilizing a stream processing engine to aggregate the access volume and the database load collected in real time by real-time statistics of the access volume and window average calculation of the database load, and combining the result of the window operation to obtain the peak value and the valley value of the access volume of the system;
according to the information such as the access peak value, the access valley value, the access time window, the database load and the like, a dynamic management algorithm is utilized to generate an initial configuration rule by setting configuration items such as the size of a connection pool, the query timeout time, the concurrent connection number and the like, the generated initial configuration rule is applied to a database configuration class, the initial configuration rule is ensured to be effective in a system, the change of the access quantity and the database load is monitored in real time when the system operates, and the configuration rule in the database configuration class is dynamically adjusted according to the real-time analysis result of the dynamic management algorithm.
In this embodiment, by acquiring the access frequency, the access time window and the access data amount of the database, the system can flexibly set the life cycle of the initial configuration rule, ensure that the rule is effective in a certain period, and meanwhile, clearly define the modification authority, ensure that only authorized personnel can modify the configuration rule, and improve the stability and safety of the system.
Fig. 2 is a schematic structural diagram of a data source dynamic adapting system based on a lightweight platform according to an embodiment of the present invention, as shown in fig. 2, the system includes:
the first unit is used for acquiring user demands, dynamically loading configuration classes, generating corresponding data adapters based on the user demands through a data source analyzer, acquiring original addresses and database passwords corresponding to a database according to the data adapters, adapting manufacturer data materials in the database through a data source adaptation algorithm, creating a database dividing mark and generating an optimal adaptation scheme;
the second unit is used for introducing a responsibility chain mechanism when the front end accesses the rear end according to the optimal adaptation scheme, setting corresponding interceptor responsibility for each interceptor in the rear end, judging whether interception is needed or not through a context object which is introduced in advance, and calling according to an access request of the database to acquire a database access permission;
and the third unit is used for generating a database management rule through a dynamic management algorithm according to the database access permission, acquiring a data source in the database through the database management rule, inquiring and editing the data source in the database, and returning the database data corresponding to the database dividing mark if the database dividing mark is inquired.
The present invention may be a method, apparatus, system, and/or computer program product. The computer program product may include a computer readable storage medium having computer readable program instructions embodied thereon for performing various aspects of the present invention.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and not for limiting the same; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some or all of the technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit of the invention.

Claims (10)

1. A method for dynamic adaptation of a data source based on a lightweight platform, comprising:
acquiring user requirements, dynamically loading configuration classes, generating corresponding data adapters based on the user requirements through a data source analyzer, acquiring original addresses and database passwords corresponding to a database according to the data adapters, adapting manufacturer data materials in the database through a data source adaptation algorithm, creating a database dividing mark and generating an optimal adaptation scheme;
According to the optimal adaptation scheme, introducing a responsibility chain mechanism when the front end accesses the rear end, setting corresponding interceptor responsibility for each interceptor in the rear end, judging whether interception is needed or not through a pre-introduced context object, and calling according to an access request of the database to acquire a database access permission;
and generating a database management rule through a dynamic management algorithm according to the database access permission, acquiring a data source in the database through the database management rule, inquiring and editing the data source in the database, and returning the database data corresponding to the database dividing mark if the database dividing mark is inquired.
2. The method of claim 1, wherein the obtaining user requirements, dynamically loading configuration classes, generating, by a data source parser, corresponding data adapters based on the user requirements comprises:
acquiring a database to be accessed according to user requirements, generating a configuration class, creating a configuration manager, monitoring an event according to the configuration manager, and reloading the configuration file and updating the configuration class if the configuration file changes;
According to the updated configuration class, a data source analyzer is selected, the data source analyzer obtains dynamic parameters of each configuration item in the configuration class, determines parameters corresponding to each interface of the data adapter according to the dynamic parameters, generates the data adapter, and registers the data adapter into a system through a configuration file.
3. The method of claim 1, wherein the obtaining, according to the data adapter, the original address and the database password corresponding to the database, adapting vendor data in the database through a data source adaptation algorithm, creating a database splitting flag, and generating an adaptation scheme includes:
according to the data adapter, acquiring a database address and a database password corresponding to a database matched with the user demand, generating an objective function based on a data source adaptation algorithm according to the database address and the database password, and taking the database address and the database password as an optimization target of the objective function;
randomly generating an initial adaptation scheme as an initial solution of the objective function, setting initial temperature corresponding to the initial adaptation scheme, reducing the initial temperature, carrying out neighborhood search according to the initial adaptation scheme for each different temperature, replacing the initial adaptation scheme if a new adaptation scheme is better than the initial adaptation scheme, repeating the search until the search is completed, marking the adaptation scheme when the search is completed as an optimal adaptation scheme, marking a database corresponding to the optimal adaptation scheme as a required database, and generating a corresponding database dividing mark.
4. The method according to claim 1, wherein the introducing a responsibility chain mechanism when the front end accesses the back end according to the optimal adaptation scheme, setting a corresponding interceptor responsibility for each interceptor in the back end, determining whether interception is required by a context object introduced in advance, and calling according to the database access request, and obtaining the database access permission includes:
according to the optimal adaptation scheme, the front end performs data access to the rear end, a service control layer in the rear end intercepts the data access through interceptors, corresponding interceptor responsibilities are set for each interceptor in the rear end according to the responsibility chain mechanism, and the interceptors are connected according to the execution sequence according to the interceptor responsibilities to form an organic responsibility chain;
according to the organic responsibility chain, when a database access request is obtained, processing logic of a first interceptor is executed from the head of the organic responsibility chain, and whether to terminate the request or transmit the request to the next interceptor is judged according to the corresponding interceptor responsibility until all interceptors are executed or the database access request is intercepted;
And if the database access request is intercepted, generating corresponding return data according to the interceptors generating the interception, and if all interceptors are executed and the interception is not generated, generating a database access permission according to the database access request.
5. The method of claim 4, wherein the organic responsibility chain further comprises an interceptor dynamic management scheme:
generating an interceptor interface in the organic responsibility chain according to the interceptor dynamic management scheme, and setting an interceptor manager on the interceptor interface, wherein the interceptor manager judges an entry point, an exit point and an interceptor execution sequence of the user demand according to the user demand;
according to the entry point and the exit point, deleting the interceptors positioned before the entry point and the interceptors positioned after the exit point in the organic responsibility chain, acquiring an interceptor list according to the execution sequence of the interceptors, and adding or deleting the interceptors at specific positions.
6. The method according to claim 1, wherein generating a database management rule according to the database access permission by a dynamic management algorithm, acquiring a data source in the database by the database management rule, querying and editing the data source in the database, and returning database data corresponding to the database splitting mark if the database splitting mark is queried, comprises:
Acquiring a configuration class corresponding to the database according to the database access permission, and setting a corresponding initial configuration rule for each configuration item in the configuration class by combining the access frequency, the access time window and the access data volume limit of the database in the system through a dynamic management algorithm introduced in advance;
according to the initial configuration rule, historical access data are obtained, the performance of a database is predicted according to a performance optimization model in the dynamic management algorithm and the historical access data, a performance prediction result is obtained, the initial configuration rule is optimized according to the performance prediction result, corresponding effective conditions are added, an optimal configuration rule is obtained, and the operation is repeated until the preset iteration times are reached;
summarizing the optimal configuration rule corresponding to each configuration item, generating the database management rule, calling a data source in the database by the rear end according to the database management rule, inquiring whether the data source contains a preset database dividing mark or not according to the data source, returning database data corresponding to the database dividing mark if the database dividing mark is contained, and returning an empty inquiry result if the database dividing mark is not contained.
7. The method of claim 6, wherein the setting of the corresponding initial configuration rule for each configuration item in the configuration class by the pre-introduced dynamic management algorithm in combination with access rights, access time window and access data volume limit of the database in the system comprises:
acquiring the access frequency, access time window and access data volume of the database, and determining the life cycle and modification authority of the initial configuration rule;
collecting system access quantity and database load in real time according to the dynamic management algorithm, and performing aggregation operation and window operation on the system access quantity and the database load collected in real time through a stream processing engine to obtain access peaks and access valleys;
and generating an initial configuration rule according to the access peak value and the access valley value by combining an access time window and a database load.
8. A lightweight platform-based data source dynamic adaptation system, comprising:
the first unit is used for acquiring user demands, dynamically loading configuration classes, generating corresponding data adapters based on the user demands through a data source analyzer, acquiring original addresses and database passwords corresponding to a database according to the data adapters, adapting manufacturer data materials in the database through a data source adaptation algorithm, creating a database dividing mark and generating an optimal adaptation scheme;
The second unit is used for introducing a responsibility chain mechanism when the front end accesses the rear end according to the optimal adaptation scheme, setting corresponding interceptor responsibility for each interceptor in the rear end, judging whether interception is needed or not through a context object which is introduced in advance, and calling according to an access request of the database to acquire a database access permission;
and the third unit is used for generating a database management rule through a dynamic management algorithm according to the database access permission, acquiring a data source in the database through the database management rule, inquiring and editing the data source in the database, and returning the database data corresponding to the database dividing mark if the database dividing mark is inquired.
9. An electronic device, comprising:
a processor;
a memory for storing processor-executable instructions;
wherein the processor is configured to invoke the instructions stored in the memory to perform the method of any of claims 1 to 7.
10. A computer readable storage medium having stored thereon computer program instructions, which when executed by a processor, implement the method of any of claims 1 to 7.
CN202410009688.3A 2024-01-04 2024-01-04 Data source dynamic adaptation method and system based on lightweight platform Active CN117539947B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202410009688.3A CN117539947B (en) 2024-01-04 2024-01-04 Data source dynamic adaptation method and system based on lightweight platform

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202410009688.3A CN117539947B (en) 2024-01-04 2024-01-04 Data source dynamic adaptation method and system based on lightweight platform

Publications (2)

Publication Number Publication Date
CN117539947A true CN117539947A (en) 2024-02-09
CN117539947B CN117539947B (en) 2024-03-12

Family

ID=89786474

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202410009688.3A Active CN117539947B (en) 2024-01-04 2024-01-04 Data source dynamic adaptation method and system based on lightweight platform

Country Status (1)

Country Link
CN (1) CN117539947B (en)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9948655B1 (en) * 2016-04-15 2018-04-17 AtScale, Inc. Data access authorization for dynamically generated database structures
CN108667930A (en) * 2018-05-09 2018-10-16 聚龙股份有限公司 A kind of configuration of dynamic data source and automation switching method
CN113810466A (en) * 2021-08-12 2021-12-17 北京科东电力控制系统有限责任公司 Middleware for multi-source heterogeneous data, system and method applying middleware
CN114595201A (en) * 2022-03-23 2022-06-07 四川省明厚天信息技术股份有限公司 Method, equipment and storage medium for inquiring acquisition record of interface access log
CN117313050A (en) * 2023-09-08 2023-12-29 山东浪潮科学研究院有限公司 Authority configuration, storage, distribution, authorization and authentication method suitable for low-code platform

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9948655B1 (en) * 2016-04-15 2018-04-17 AtScale, Inc. Data access authorization for dynamically generated database structures
CN108667930A (en) * 2018-05-09 2018-10-16 聚龙股份有限公司 A kind of configuration of dynamic data source and automation switching method
CN113810466A (en) * 2021-08-12 2021-12-17 北京科东电力控制系统有限责任公司 Middleware for multi-source heterogeneous data, system and method applying middleware
CN114595201A (en) * 2022-03-23 2022-06-07 四川省明厚天信息技术股份有限公司 Method, equipment and storage medium for inquiring acquisition record of interface access log
CN117313050A (en) * 2023-09-08 2023-12-29 山东浪潮科学研究院有限公司 Authority configuration, storage, distribution, authorization and authentication method suitable for low-code platform

Also Published As

Publication number Publication date
CN117539947B (en) 2024-03-12

Similar Documents

Publication Publication Date Title
US8181173B2 (en) Determining priority for installing a patch into multiple patch recipients of a network
US7035882B2 (en) Data storage system
US8775412B2 (en) Method and system for a self-healing query access plan
US7130838B2 (en) Query optimization via a partitioned environment
US7031958B2 (en) Patterned based query optimization
KR101120814B1 (en) Systems and methods that optimize row level database security
US7502775B2 (en) Providing cost model data for tuning of query cache memory in databases
US20040167975A1 (en) Method, system, and program for managing devices in a network
US20080183684A1 (en) Caching an Access Plan for a Query
US8566307B2 (en) Database query governor with tailored thresholds
US11720825B2 (en) Framework for multi-tenant data science experiments at-scale
CN113792077B (en) Data processing method, program product, readable medium and electronic device
US20220121633A1 (en) Learning-based workload resource optimization for database management systems
US11308066B1 (en) Optimized database partitioning
CN106776706A (en) Method for managing user right and device based on caching
US11645281B1 (en) Caching query plans in database systems
CN117539947B (en) Data source dynamic adaptation method and system based on lightweight platform
US20230333971A1 (en) Workload generation for optimal stress testing of big data management systems
US11151272B2 (en) Monitoring usage of an application to identify characteristics and trigger security control
US11392356B1 (en) Online machine learning based compilation
SE1051167A1 (en) A system and method for performing partial evaluation in order to construct a simplified policy
CN114020446A (en) Cross-multi-engine routing processing method, device, equipment and storage medium
CN113722349A (en) SQL (structured query language) auditing method and device based on distributed database and computer equipment
KR20180104498A (en) Method and apparatus for processing query
US7664752B2 (en) Authorization over a distributed and partitioned management system

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant