CN114860770A - Optimization method, system, computer device and medium based on data service - Google Patents
Optimization method, system, computer device and medium based on data service Download PDFInfo
- Publication number
- CN114860770A CN114860770A CN202210396312.3A CN202210396312A CN114860770A CN 114860770 A CN114860770 A CN 114860770A CN 202210396312 A CN202210396312 A CN 202210396312A CN 114860770 A CN114860770 A CN 114860770A
- Authority
- CN
- China
- Prior art keywords
- data
- service
- data service
- parameters
- cache
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 69
- 238000005457 optimization Methods 0.000 title claims abstract description 60
- 230000004044 response Effects 0.000 claims abstract description 26
- 238000006243 chemical reaction Methods 0.000 claims abstract description 13
- 238000004590 computer program Methods 0.000 claims description 18
- 230000008569 process Effects 0.000 claims description 15
- 238000012545 processing Methods 0.000 claims description 12
- 238000012544 monitoring process Methods 0.000 claims description 10
- 230000007246 mechanism Effects 0.000 claims description 7
- 230000006978 adaptation Effects 0.000 claims description 4
- 230000006870 function Effects 0.000 description 10
- 238000004458 analytical method Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 4
- 230000009466 transformation Effects 0.000 description 4
- 230000015556 catabolic process Effects 0.000 description 3
- 238000013523 data management Methods 0.000 description 3
- 238000006731 degradation reaction Methods 0.000 description 3
- 230000003044 adaptive effect Effects 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000002955 isolation Methods 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 230000006399 behavior Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000010485 coping Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000004806 packaging method and process Methods 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 230000006641 stabilisation Effects 0.000 description 1
- 238000011105 stabilization Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2453—Query optimisation
- G06F16/24534—Query rewriting; Transformation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2453—Query optimisation
- G06F16/24534—Query rewriting; Transformation
- G06F16/24539—Query rewriting; Transformation using cached or materialised query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/08—Protocols for interworking; Protocol conversion
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer Security & Cryptography (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses an optimization method, a system, computer equipment and a medium based on data service, wherein the optimization method based on the data service comprises the following steps: acquiring a data service request sent by an HTTP (hyper text transport protocol), wherein the data service request comprises a data query parameter, an HTTP request parameter and a local parameter; assembling and converting the data query parameters, the HTTP request parameters, the local parameters and the back-end service parameters through an adapter to call resource group services; if the data query parameters conform to the application conversion rules, converting the logic plan corresponding to the data query parameters into a physical execution strategy; and performing data retrieval on the resource group adapter cache based on the physical execution strategy, acquiring a retrieval result, converting the retrieval result into an HTTP response result through the adapter, and returning the HTTP response result. The method effectively improves the response timeliness of the data service, reduces the repeated data reading of the resource group, and improves the overall applicability of the data service system.
Description
Technical Field
The present invention relates to the field of data management technologies, and in particular, to an optimization method, system, computer device, and medium based on data services.
Background
Data-as-a-service (DaaS) is a Data management strategy that aims to improve the agility of business innovation by using Data as business assets. DaaS is intermediate between PaaS (Platform-as-a-Service) and SaaS (Software-as-a-Service). Like SaaS, DaaS provides a way for usage agencies to manage the large amounts of data generated on a daily basis and to abstract valuable information from these data on a service-wide basis in order to drive various decisions. DaaS generally provides data from various sources on demand through an API, aims to simplify access to data, provides data sets or data streams that can be used in multiple formats, and thus enables the use of data virtualization.
The API gateway is required to be used as a unified gateway when being constructed, the current limiting and authentication functions of the micro-service gateway need to be taken over, integrated platform middleware and the like of traditional application services provided by a using mechanism can be connected, and the unified gateway integrating multiple functions inevitably has system performance problems, concurrency problems and the like which influence the usability of data services.
Disclosure of Invention
The embodiment of the invention provides an optimization method, a system, computer equipment and a medium based on data service, which aim to solve the problem that the system performance, concurrency and other problems of a data service system influence the availability of the data service in the using process.
A method of data service based optimization, comprising:
acquiring a data service request sent by an HTTP (hyper text transport protocol), wherein the data service request comprises data query parameters, HTTP request parameters and local parameters;
assembling and converting the data query parameters, the HTTP request parameters, the local parameters and the back-end service parameters through an adapter to form back-end service parameters adaptive to a back-end service protocol for calling resource group service;
if the data query parameters conform to the application conversion rules, converting the logic plan corresponding to the data query parameters into a physical execution strategy;
and performing data retrieval on the resource group adapter cache based on the physical execution strategy, acquiring a retrieval result, converting the retrieval result into an HTTP response result through the adapter, and returning the HTTP response result.
Preferably, the retrieving result is obtained, including:
if the retrieval result is empty, carrying out hierarchical query on the data query parameters according to the attributes through a linker to obtain a query hierarchical result;
and acquiring the stored data closest to the data query parameters as a retrieval result based on the query layering result.
Preferably, the data service request further comprises a service type; the method for forming the back-end service parameters adaptive to the back-end service protocol to call the resource group service further comprises the following steps:
if the service type belongs to the cache strategy request, searching API cache data corresponding to the data service request through the API gateway cache, and obtaining a searching result;
if the search result is empty, continuing to search the API cache data on the resource group adapter cache;
and if the API cache data is not searched in the resource group adapter cache, continuously executing the step of carrying out hierarchical query on the data query parameters according to the attributes through the linker to obtain a query hierarchical result.
Preferably, before searching the API cache data corresponding to the data service request through the API gateway cache, the method further includes:
the API gateway stores API cache data in response to requests from the data repository during its lifetime.
Preferably, the data service request further comprises an item ID;
continuing to look up API cache data on the resource group adapter cache, comprising:
acquiring a corresponding project resource group based on the project ID;
and acquiring the API cache data through the adaptation of the project resource group.
Preferably, the resource group service includes at least two project resource groups stored per project resource; the method further comprises the following steps:
resource invocation of at least two sets of project resources is controlled by a workload orchestration mechanism.
Preferably, the data service request is obtained through an API gateway; the method further comprises the following steps:
monitoring the data service request to obtain a monitoring result;
if the data service request belongs to a current limiting object, performing current limiting processing on the data service request;
and if the execution process of the data service request meets the failure value of the sliding window period, automatically fusing the data service request.
A data service based optimization system, comprising:
the system comprises an acquisition service request module, a data service request module and a data service processing module, wherein the acquisition service request module is used for acquiring a data service request sent by an HTTP (hyper text transport protocol), and the data service request comprises a data query parameter, an HTTP request parameter and a local parameter;
the calling resource group service module is used for splicing and converting the data query parameters, the HTTP request parameters, the local parameters and the back-end service parameters through the adapter to form back-end service parameters matched with the back-end service protocol for calling resource group services;
the conversion logic plan module is used for converting the logic plan corresponding to the data query parameter into a physical execution strategy if the data query parameter accords with the application conversion rule;
and the retrieval result acquisition module is used for carrying out data retrieval on the resource group adapter cache based on the physical execution strategy, acquiring a retrieval result, converting the retrieval result into an HTTP response result through the adapter and returning the HTTP response result.
A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, the processor implementing the data service based optimization method when executing the computer program.
A computer-readable medium, in which a computer program is stored which, when being executed by a processor, carries out the above-mentioned data service based optimization method.
According to the optimization method, the optimization system, the optimization computer equipment and the optimization medium based on the data service, the HTTP protocol is converted through the adapter to call the resource group service, the corresponding retrieval result is obtained through the resource group adapter cache, the response timeliness of the data service can be effectively improved, multiple data reading of the resource group is reduced, the retrieval pressure of the resource group is reduced, the data retrieval speed is improved, and the overall applicability of the data service system is improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the description of the embodiments of the present invention will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to these drawings without inventive labor.
FIG. 1 is a schematic diagram of an application environment of an optimization method based on data services according to an embodiment of the present invention;
FIG. 2 is a flow chart of a method for data service based optimization in an embodiment of the present invention;
FIG. 3 is a first flowchart of a method for data service based optimization according to another embodiment of the present invention;
FIG. 4 is a second flowchart of a method for data service based optimization in another embodiment of the present invention;
FIG. 5 is a third flowchart of a data service based optimization method in another embodiment of the present invention;
FIG. 6 is a fourth flowchart of a data service based optimization method in another embodiment of the present invention;
FIG. 7 is a schematic diagram of a data services based optimization system in an embodiment of the present invention;
FIG. 8 is a block diagram of an overall architecture of a data services based optimization system in accordance with another embodiment of the present invention;
FIG. 9 is a schematic diagram of a computer device in an embodiment of the invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The data service-based optimization method provided by the embodiment of the invention can be applied to the application environment shown in fig. 1, and the data service-based optimization method is applied to a data service-based optimization system, and the data service-based optimization system comprises a client and a resource group platform, wherein the client communicates with the resource group platform through a network. The client is also called a client, and refers to a program corresponding to the resource group platform and providing local services for the client. Further, the client is a computer program, an APP program of the intelligent device or a third-party applet embedded with other APPs. The client can be installed on computer equipment such as but not limited to various personal computers, notebook computers, smart phones, tablet computers and portable wearable equipment. The resource group platform can be implemented as a stand-alone resource group platform or as a cluster of resource group platforms consisting of multiple resource group platforms.
In an embodiment, as shown in fig. 2, an optimization method based on data services is provided, which is described by taking the example that the method is applied to the resource group platform in fig. 1, and specifically includes the following steps:
s10, acquiring a data service request sent by an HTTP (hyper text transport protocol), wherein the data service request comprises a data query parameter, an HTTP request parameter and a local parameter.
The data service request is a request for providing data service to implement a subsequent data virtualization scenario, where data virtualization (data virtualization) is a general term used to describe all data management methods, and is a process of data integration to obtain more data information, and this process usually introduces other technologies, such as a database, an application program, a file system, a web technology, a big data technology, and the like.
The data query parameters comprise the type of query data, parameter application scenes and other parameters related to the data service.
The HTTP request parameter is a parameter that encapsulates the data service request to conform to the HTTP protocol.
The local parameter (Context local parameter) refers to a parameter set locally when the server provides a corresponding data service, for example, a server memory, a peak value per second, and the like.
And S20, assembling and converting the data query parameters, the HTTP request parameters, the local parameters and the back-end service parameters through the adapter to form back-end service parameters matched with a back-end service protocol for calling resource group service.
The adapter is used for filling and adapting data and supporting access to a data source. Each data provider also provides a data adapter.
The resource group service is a data service that the resource group platform can provide in the present application.
And S30, if the data query parameters accord with the application conversion rules, converting the logic plan corresponding to the data query parameters into a physical execution strategy.
The application conversion rule is a preset rule capable of converting the data query parameters.
The implementation of the logical plan (logistic plan) class mainly comprises three phases:
1. converting each node of the syntax tree into a corresponding logic plan node through an AstBuilder code syntax analyzer to form an unresolved logic operator tree which does not contain data information and column information; 2. acting a series of rules on the unresolved logical operator tree by an Analyzer Analyzer to generate an analyzed logical operator tree; 3. and applying a series of optimization rules to the logic operator tree through an Optimizer so as to improve an inefficient structure and generate an optimized logic operator tree on the premise of ensuring correct results.
The physical execution is to convert the logical execution plan into a physical execution plan SparkPlan, and then may directly run on a spare-Core (kernel of spare framework) to generate RDD (resource Distributed databases), so as to solve the problem that the computing framework is not efficient in processing different application scenarios,
specifically, the technical implementation that the adapter can parse SQL statements, optimize queries, and map syntax to corresponding data sources supports that multiple data sources (mysql, oracle, postgresql, hive, etc.) need to perform protocol conversion and service call for data sources (Dubbo, resutful api) of non-databases. The work that the adapter needs to do includes: the method comprises the steps of obtaining HTTP request parameters and Context local parameters, assembling back-end service parameters, completing protocol conversion from an HTTP protocol to back-end service, calling the back-end service to obtain a response result and converting the response result into an HTTP response result.
The optimization query translates the logical plan into a physical execution strategy. This process uses a set of transformation rules to complete the evaluation until an optimal point is reached, each rule having a schema that can match a sub-tree of the query plan and determine whether a transformation should be applied. The result is to replace the match with a logically equivalent sub-query. The system contains several optimized application transformation rules, such as: such as Predicate Pushdown Rules (Predicate Pushdown Rules), Limit Pushdown Rules (Limit Pushdown Rules), column clipping Rules (ColumnPruning Rules), and disassociation Rules (disassociate Rules).
And S40, performing data retrieval on the resource group adapter cache based on the physical execution strategy, acquiring a retrieval result, converting the retrieval result into an HTTP response result through the adapter, and returning the HTTP response result.
The resource group adapter cache is a local cache provided by the resource group platform through a cache strategy, and is used for relieving the storage and searching pressure of the resource group database.
In particular, the caching policy provides user, token, application level caching, which can be used to differentiate between hot and cold data caching to effectively reduce the pressure on the real API provider (resource group database), thereby reducing the pressure on the application container, application cache, and database of the API service provider step by step. If the cache is hit, a real API provider does not need to be requested, and the access time is effectively reduced.
The adapter layer cache is suitable for data which is rarely modified, non-important data, constant data and data which is difficult to modify by a third party.
According to the optimization method based on the data service, the adapter is used for converting the HTTP protocol to call the resource group service, the corresponding retrieval result is obtained through the resource group adapter cache, the response timeliness of the data service can be effectively improved, multiple data reading on the resource group is reduced, the retrieval pressure of the resource group is relieved, the data retrieval speed is improved, and the overall applicability of the data service system is improved.
In an embodiment, as shown in fig. 3, in step S40, the obtaining of the search result specifically includes the following steps:
and S41, if the retrieval result is empty, performing hierarchical query on the data query parameters according to the attributes through the linker to obtain a query hierarchical result.
And S42, acquiring the storage data closest to the data query parameters as a retrieval result based on the query layering result.
The hierarchical storage is a data storage mode realized by the hierarchical query, helps a user to perform query by quick cold start under the condition of not performing pre-calculation, can remarkably improve the performance of ultra-multidimensional flexible analysis and detailed query, is beneficial to comprehensively covering various analysis scenes, and brings more possibility of analysis and exploration based on mass data for the user.
The intelligent hierarchical storage can support flexible ad hoc inquiry scenes of multi-dimensional random combination, such as label analysis or user behavior analysis and the like, and brings greater assistance to fine operation and auxiliary business decision of mechanisms using data services; meanwhile, data can be managed by connecting an external model through hierarchical storage, mainstream commercial intelligent BI tools and excels can be connected seamlessly, and data analysts can be enabled to analyze the data.
In particular, the linker reports location and other data attributes, such as partitions, ordering and grouping, and indexing. The linker may return multiple query tiers for a single table, each query tier having different attributes, the optimized query may be the query to select the most efficient query tier; this functionality can be used to manage and optimize development and analysis of queries for easy optimization of new queries by adding physical hierarchies.
In a specific embodiment, as described in fig. 4, after step S20, the data service request further includes the service type. After a backend service parameter adapted to the backend service protocol is formed to invoke resource group services, the method further specifically includes the following steps:
s201, if the service type belongs to the cache strategy request, searching API cache data corresponding to the data service request through the API gateway cache, and obtaining a searching result.
S202, if the search result is empty, continuing to search the API cache data on the resource group adapter cache.
S203, if the API cache data is not searched in the resource group adapter cache, the step of carrying out hierarchical query on the data query parameters according to the attributes through the linker to obtain a query hierarchical result is continuously executed.
The cache policy request is a preset request which can adopt the API gateway to cache and obtain corresponding data, for example, some idempotent, frequently queried, data timeliness insensitive requests and other openable request cache policies.
Specifically, the API gateway cache may effectively reduce the pressure of the real API provider, thereby reducing the pressure of the application container, the application cache, and the database of the API service provider step by step. The cache of the API gateway can effectively reduce the access time of the background API, the cache is directly accessed from the API gateway, if the cache is hit, a real API provider does not need to be requested, and the access time is effectively reduced.
The API gateway cache is enabled to cache the end node's response. By means of caching, the number of calls initiated to the end node can be reduced, while reducing the delay of requests issued to the API.
In an embodiment, before step S,201, that is, before searching the API cache data corresponding to the data service request through the API gateway cache, the method further includes the following steps:
s2010. the API gateway stores API cache data in response to a request from the data repository during its lifetime.
Specifically, the API gateway may cache responses from the compute nodes in the resource group during a specified Time-To-Live (TTL) period (in seconds). Then, in responding to the request, the API gateway looks up the end node response from the API gateway cache without making the request to the end node. Preferably, the default TTL value of the API cache may be set to 600 seconds, and TTL = 0 indicates that the interface cache is disabled.
In a particular embodiment, the data service request further includes an item ID. As shown in fig. 5, in step S201, that is, continuously searching for API cache data on the resource group adapter cache, the method specifically includes the following steps:
and S2011, acquiring a corresponding project resource group based on the project ID.
S2012, the API cache data is obtained through the adaptation of the project resource group.
The management of the project resource group adopts a project mode to manage resource nodes (special nodes for realizing resource packaging and resource isolation), corresponding nodes and process numbers are initially distributed after service creation, resource elastic expansion is realized by matching with service arrangement according to load conditions, and multi-level resource elastic expansion is supported. Furthermore, besides the project resource group, a public resource pool can be set as a redundancy backup for coping with the load at peak time.
In one embodiment, the resource group service includes at least two project resource groups stored per project resource. The optimization method based on the data service further comprises the following steps:
and S50, controlling resource calling of at least two project resource groups through a workload arranging mechanism.
In particular, the workload orchestration mechanism is responsible for sharing workloads among resources, for service discovery and high availability, etc., for managing clusters, providing automated orchestration, coordination, and management of resource data access portals, complex computer systems, middleware (middlewares), and traffic of the clusters.
The at least two project resource groups facilitate project isolation for resource usage, especially to accommodate multi-user/tenant scenarios.
In one embodiment, the data service request is obtained through an API gateway. As shown in fig. 6, the optimization method based on data service further includes the following steps:
and S61, monitoring the data service request to obtain a monitoring result.
And S62, if the data service request belongs to a current limiting object, performing current limiting processing on the data service request.
And S63, if the execution process of the data service request meets the failure value of the sliding window period, automatically fusing the data service request.
The current limiting object includes preset objects that limit network traffic, request frequency, traffic transmission speed, and the like, for example, an interface severe credential, an APP, an IP address, and the like.
The sliding window is a circled window that moves back in sequence as the ACK or process reads data. For example, in the window of the transmitting end, if an ACK message is received in a certain communication, which indicates that the previous messages have been received, the whole available window is moved to the right in sequence.
Specifically, the present embodiment can provide access protection for a resource group platform through multiple dimensions, such as interface authentication credential (token) throttling, App throttling, IP throttling, resource group throttling, and the like. Each API sets the processing overtime time, and carries out quick failure processing on overtime requests, so that resource occupation is avoided.
Further, the stabilization of services is an important cornerstone for the sustainable development of data service platforms (resource group platforms). In addition, in the distributed system, each service itself has many uncontrollable factors, such as slow processing of the thread pool, overtime of the request, insufficient resources, refusal of the request, and even unavailability of the direct service, downtime of the host, downtime of the database, downtime of the cache, and the like. For some non-core services, if a large number of exceptions occur, the services can be degraded through fusing or service degradation and lossy services are provided, so that the flexible availability of the services is ensured, and the avalanche effect is avoided.
Service fusing generally refers to a protection measure adopted in a software system to prevent the whole system from being in failure due to the overload phenomenon of a service caused by some reasons.
The service degradation is that under the condition that the pressure of the server is increased sharply, certain service interfaces or pages are closed by using limited resources according to the current service condition, so that the server resources are released to ensure the normal operation of core tasks.
In this embodiment, the resource group platform supports a fusing degradation function, monitors statistical information of the request in real time, automatically fuses after reaching a failure threshold configured in a specified sliding window period, and returns a default value.
According to the optimization method based on the data service, the resource group service is called by converting the HTTP protocol through the adapter, and the corresponding retrieval result is obtained by caching the resource group adapter, so that the response timeliness of the data service can be effectively improved, the multiple data reading of the resource group can be reduced, the retrieval pressure of the resource group can be reduced, the data retrieval speed can be improved, and the overall applicability of the data service system can be improved.
It should be understood that, the sequence numbers of the steps in the foregoing embodiments do not imply an execution sequence, and the execution sequence of each process should be determined by functions and internal logic of the process, and should not limit the implementation process of the embodiments of the present invention in any way.
In an embodiment, a data service-based optimization system is provided, and the data service-based optimization system corresponds to the data service-based optimization method in the foregoing embodiments one to one. As shown in fig. 7 and 8, the data service-based optimization system includes a get service request module 10, a call resource group service module 20, a transformation logic plan module 30, and a get search result module 40. The detailed description of each functional module is as follows:
an acquiring service request module 10, configured to acquire a data service request sent by an HTTP protocol, where the data service request includes a data query parameter, an HTTP request parameter, and a local parameter;
the calling resource group service module 20 is used for splicing and converting the data query parameters, the HTTP request parameters, the local parameters and the back-end service parameters through the adapter to form back-end service parameters matched with a back-end service protocol for calling resource group services;
a conversion logic plan module 30, configured to convert the logic plan corresponding to the data query parameter into a physical execution policy if the data query parameter meets the application conversion rule;
and the retrieval result obtaining module 40 is used for performing data retrieval on the resource group adapter cache based on the physical execution strategy, obtaining a retrieval result, converting the retrieval result into an HTTP response result through the adapter, and returning the HTTP response result.
Preferably, the module 40 for obtaining retrieval results includes a sub-module for obtaining hierarchical results and a sub-module for obtaining retrieval results. The functional modules are explained in detail as follows:
and the hierarchical result obtaining sub-module is used for carrying out hierarchical query on the data query parameters according to the attributes through the linker to obtain a query hierarchical result if the retrieval result is empty.
And the retrieval result obtaining sub-module is used for obtaining the storage data closest to the data query parameters as the retrieval result based on the query layering result.
Preferably, the optimization system based on data service further comprises a module for obtaining search results, a module for continuing to search the API cache and a module for continuing to execute the query hierarchy. The functional modules are explained in detail as follows:
and the search result obtaining module is used for searching API cache data corresponding to the data service request through the API gateway cache if the service type belongs to the cache strategy request, and obtaining a search result.
And the continuous search API cache module is used for continuously searching API cache data on the resource group adapter cache if the search result is empty.
And the continuous execution query layering module is used for continuously executing the step of carrying out layering query on the data query parameters according to the attributes through the linker to obtain a query layering result if the API cache data is not searched on the resource group adapter cache.
Preferably, the data service-based optimization system further comprises a data storage and caching module. The functional modules are explained in detail as follows:
and the storage cache data module is used for responding to the request of the data resource library by the gateway in the life period to store the API cache data.
Preferably, the continue-find API cache module includes a get project resource group submodule and a get cache data submodule. The functional modules are explained in detail as follows:
and the acquisition project resource group submodule is used for acquiring the corresponding project resource group based on the project ID.
And the cache data obtaining submodule is used for obtaining the API cache data through the adaptation of the project resource group.
Preferably, the data service-based optimization system further comprises a control resource calling module. The functional modules are explained in detail as follows:
and the control resource calling module is used for controlling the resource calling of at least two project resource groups through a workload arranging mechanism.
Preferably, the optimization system based on data service further comprises a monitoring result obtaining module, a current limiting processing service request module and a fusing service request module. The functional modules are explained in detail as follows:
and the monitoring result obtaining module is used for monitoring the data service request and obtaining a monitoring result.
And the flow limiting processing service request module is used for performing flow limiting processing on the data service request if the data service request belongs to a flow limiting object.
And the fusing service request module is used for automatically fusing the data service request if the executing process of the data service request meets the failure value of the sliding window period.
For specific limitations of the data service based optimization system, reference may be made to the above limitations of the data service based optimization method, which are not described herein again. The various modules in the data service based optimization system described above may be implemented in whole or in part by software, hardware, and combinations thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as shown in fig. 9. The computer device includes a processor, a memory, a network interface, and a database connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile medium, an internal memory. The non-volatile medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile media. The database of the computer device is used for data related to an optimization method based on data services. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a method of data service based optimization.
In one embodiment, a computer device is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the processor implements the data service-based optimization method of the above embodiments, such as S10 to S40 shown in fig. 2. Alternatively, the processor, when executing the computer program, implements the functions of the modules/units of the optimization system based on data service in the above-described embodiment, such as the functions of the modules 10 to 40 shown in fig. 8. To avoid repetition, further description is omitted here.
In one embodiment, a computer readable medium is provided, on which a computer program is stored, and the computer program is executed by a processor to implement the data service-based optimization method of the above embodiments, such as S10 to S40 shown in fig. 2. Alternatively, the computer program, when executed by the processor, implements the functions of each module/unit in the data service based optimization system in the above-described system embodiment, for example, the functions of modules 10 to 40 shown in fig. 8. To avoid repetition, further description is omitted here.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a non-volatile computer readable medium, and can include the processes of the embodiments of the methods described above when the computer program is executed. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-mentioned division of the functional units and modules is illustrated, and in practical applications, the above-mentioned function distribution may be performed by different functional units and modules according to needs, that is, the internal structure of the system is divided into different functional units or modules to perform all or part of the above-mentioned functions.
The above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not substantially depart from the spirit and scope of the embodiments of the present invention, and are intended to be included within the scope of the present invention.
Claims (10)
1. A method for optimizing based on data service, comprising:
acquiring a data service request sent by an HTTP (hyper text transport protocol), wherein the data service request comprises a data query parameter, an HTTP request parameter and a local parameter;
assembling and converting the data query parameters, the HTTP request parameters, the local parameters and the back-end service parameters through an adapter to form back-end service parameters matched with a back-end service protocol for calling resource group service;
if the data query parameters conform to the application conversion rules, converting the logic plan corresponding to the data query parameters into a physical execution strategy;
and performing data retrieval on the resource group adapter cache based on the physical execution strategy, acquiring a retrieval result, converting the retrieval result into an HTTP response result through the adapter, and returning the HTTP response result.
2. The data service-based optimization method according to claim 1, wherein the obtaining of the search result comprises:
if the retrieval result is empty, performing hierarchical query on the data query parameters according to attributes through a linker to obtain a query hierarchical result;
and acquiring the storage data closest to the data query parameters as the retrieval result based on the query layering result.
3. The data service-based optimization method of claim 2, wherein the data service request further comprises a service type;
the forming of the backend service parameters adapted to the backend service protocol is used for invoking resource group services, and the method further comprises the following steps:
if the service type belongs to the cache strategy request, searching API cache data corresponding to the data service request through an API gateway cache, and obtaining a search result;
if the search result is empty, continuing to search the API cache data on the resource group adapter cache;
if the API cache data is not searched on the resource group adapter cache, the step of carrying out hierarchical query on the data query parameters according to attributes through the linker to obtain query hierarchical results is continuously executed.
4. The data service-based optimization method of claim 3, wherein before the searching for API cache data corresponding to the data service request through an API gateway cache, further comprising:
and the API gateway responds to the request of the data resource library to store the API cache data in the life period of the API gateway.
5. The data service-based optimization method of claim 3, wherein the data service request further comprises an item ID;
the continuing to search the API cache data on the resource group adapter cache comprises:
acquiring a corresponding project resource group based on the project ID;
and acquiring the API cache data through the adaptation of the project resource group.
6. The data-service-based optimization method of claim 1, wherein the resource group service includes at least two project resource groups stored per project resource; the method further comprises the following steps:
controlling resource invocation of at least two of the sets of project resources through a workload orchestration mechanism.
7. The data service-based optimization method of claim 1, wherein the data service request is obtained through an API gateway; the method further comprises the following steps:
monitoring the data service request to obtain a monitoring result;
if the data service request belongs to a current limiting object, performing current limiting processing on the data service request;
and if the execution process of the data service request meets the failure value of the sliding window period, automatically fusing the data service request.
8. A data service based optimization system, comprising:
the system comprises an acquisition service request module, a data service request module and a data service processing module, wherein the acquisition service request module is used for acquiring a data service request sent by an HTTP (hyper text transport protocol), and the data service request comprises a data query parameter, an HTTP request parameter and a local parameter;
the calling resource group service module is used for splicing and converting the data query parameters, the HTTP request parameters, the local parameters and the back-end service parameters through the adapter to form back-end service parameters matched with a back-end service protocol for calling resource group services;
the conversion logic plan module is used for converting the logic plan corresponding to the data query parameter into a physical execution strategy if the data query parameter accords with an application conversion rule;
and the retrieval result acquisition module is used for carrying out data retrieval on the resource group adapter cache based on the physical execution strategy, acquiring a retrieval result, converting the retrieval result into an HTTP response result through the adapter and returning the HTTP response result.
9. A computer arrangement comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the data service based optimization method according to any one of claims 1 to 7 when executing the computer program.
10.A computer-readable medium, in which a computer program is stored which, when being executed by a processor, carries out a method for data service based optimization according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210396312.3A CN114860770B (en) | 2022-04-15 | 2022-04-15 | Optimization method, system, computer device and medium based on data service |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210396312.3A CN114860770B (en) | 2022-04-15 | 2022-04-15 | Optimization method, system, computer device and medium based on data service |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114860770A true CN114860770A (en) | 2022-08-05 |
| CN114860770B CN114860770B (en) | 2022-12-16 |
Family
ID=82631959
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210396312.3A Active CN114860770B (en) | 2022-04-15 | 2022-04-15 | Optimization method, system, computer device and medium based on data service |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114860770B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116680299A (en) * | 2023-06-05 | 2023-09-01 | 粤港澳大湾区(广东)国创中心 | Database query method, system, storage medium and device |
| CN117221374A (en) * | 2023-09-11 | 2023-12-12 | 广州Tcl互联网小额贷款有限公司 | API (application program interface) calling method and system based on API gateway |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101958912A (en) * | 2010-10-28 | 2011-01-26 | 华为技术有限公司 | Classification level query method, system and uniform resource locator server |
| CN104834675A (en) * | 2015-04-02 | 2015-08-12 | 浪潮集团有限公司 | Query performance optimization method based on user behavior analysis |
| CN104885077A (en) * | 2012-09-28 | 2015-09-02 | 甲骨文国际公司 | Managing continuous queries with archived relations |
| CN109508432A (en) * | 2018-11-28 | 2019-03-22 | 段红 | A kind of high-performance FAAS system based on block chain |
| CN109600381A (en) * | 2018-12-18 | 2019-04-09 | 浩云科技股份有限公司 | A kind of method for interface adaptation, apparatus and system |
| CN111382174A (en) * | 2018-12-28 | 2020-07-07 | 百度在线网络技术（北京）有限公司 | Multi-party data combined query method, device, server and storage medium |
| CN111552540A (en) * | 2020-04-23 | 2020-08-18 | 无锡华云数据技术服务有限公司 | Resource synchronization method based on VMware cloud platform and super-fusion cloud platform |
| US10769000B1 (en) * | 2019-08-27 | 2020-09-08 | Cloudflare, Inc. | Method and apparatus for transparently enabling compatibility between multiple versions of an application programming interface (API) |
| CN112000448A (en) * | 2020-07-17 | 2020-11-27 | 北京计算机技术及应用研究所 | Micro-service architecture-based application management method |
| CN112131449A (en) * | 2020-09-21 | 2020-12-25 | 西北大学 | Implementation method of cultural resource cascade query interface based on elastic search |
| CN112395303A (en) * | 2019-08-15 | 2021-02-23 | 阿里巴巴集团控股有限公司 | Query execution method and device, electronic equipment and computer readable medium |
-
2022
- 2022-04-15 CN CN202210396312.3A patent/CN114860770B/en active Active
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101958912A (en) * | 2010-10-28 | 2011-01-26 | 华为技术有限公司 | Classification level query method, system and uniform resource locator server |
| CN104885077A (en) * | 2012-09-28 | 2015-09-02 | 甲骨文国际公司 | Managing continuous queries with archived relations |
| CN104834675A (en) * | 2015-04-02 | 2015-08-12 | 浪潮集团有限公司 | Query performance optimization method based on user behavior analysis |
| CN109508432A (en) * | 2018-11-28 | 2019-03-22 | 段红 | A kind of high-performance FAAS system based on block chain |
| CN109600381A (en) * | 2018-12-18 | 2019-04-09 | 浩云科技股份有限公司 | A kind of method for interface adaptation, apparatus and system |
| CN111382174A (en) * | 2018-12-28 | 2020-07-07 | 百度在线网络技术（北京）有限公司 | Multi-party data combined query method, device, server and storage medium |
| CN112395303A (en) * | 2019-08-15 | 2021-02-23 | 阿里巴巴集团控股有限公司 | Query execution method and device, electronic equipment and computer readable medium |
| US10769000B1 (en) * | 2019-08-27 | 2020-09-08 | Cloudflare, Inc. | Method and apparatus for transparently enabling compatibility between multiple versions of an application programming interface (API) |
| CN111552540A (en) * | 2020-04-23 | 2020-08-18 | 无锡华云数据技术服务有限公司 | Resource synchronization method based on VMware cloud platform and super-fusion cloud platform |
| CN112000448A (en) * | 2020-07-17 | 2020-11-27 | 北京计算机技术及应用研究所 | Micro-service architecture-based application management method |
| CN112131449A (en) * | 2020-09-21 | 2020-12-25 | 西北大学 | Implementation method of cultural resource cascade query interface based on elastic search |
Non-Patent Citations (3)
| Title |
|---|
| FREDERICO VALENTE ET AL.: "A RESTful Image Gateway for Multiple Medical Image Repositories", 《IEEE TRANSACTIONS ON INFORMATION TECHNOLOGY IN BIOMEDICINE》 * |
| 潘卿帅: "智能服务中心服务化关键技术的研究与实现", 《中国优秀硕士学位论文全文数据库 信息科技辑》 * |
| 韩万江 等: "基于微服务架构的分布式灾情管理系统设计", 《中国地震》 * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116680299A (en) * | 2023-06-05 | 2023-09-01 | 粤港澳大湾区(广东)国创中心 | Database query method, system, storage medium and device |
| CN116680299B (en) * | 2023-06-05 | 2024-04-09 | 粤港澳大湾区(广东)国创中心 | Database query method, system, storage medium and device |
| CN117221374A (en) * | 2023-09-11 | 2023-12-12 | 广州Tcl互联网小额贷款有限公司 | API (application program interface) calling method and system based on API gateway |
| CN117221374B (en) * | 2023-09-11 | 2024-05-24 | 广州Tcl互联网小额贷款有限公司 | API (application program interface) calling method and system based on API gateway |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114860770B (en) | 2022-12-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114860770B (en) | Optimization method, system, computer device and medium based on data service | |
| CN107291947B (en) | Semi-structured data query method and distributed NewSQL database system | |
| US10831562B2 (en) | Method and system for operating a data center by reducing an amount of data to be processed | |
| US9298775B2 (en) | Changing the compression level of query plans | |
| US11055352B1 (en) | Engine independent query plan optimization | |
| US20170329835A1 (en) | Context-aware workload dispatching | |
| CN109446252B (en) | Unified access method and system for power grid regulation and control | |
| US11520740B2 (en) | Efficiently deleting data from objects in a multi-tenant database system | |
| US9930113B2 (en) | Data retrieval via a telecommunication network | |
| US10860606B2 (en) | Efficiently deleting data from objects in a multi tenant database system | |
| CN115729951B (en) | Data query method, system, device and computer readable storage medium | |
| US20240012819A1 (en) | PROCESSING EXTERNAL FUNCTIONS USING USER-DEFINED FUNCTIONS (UDFs) | |
| US11144359B1 (en) | Managing sandbox reuse in an on-demand code execution system | |
| US20220164344A1 (en) | Data processing method, data processing device and non-volatile computer-readable storage media | |
| CN108874837A (en) | Database divides library method, apparatus, middleware and storage medium and electronic equipment | |
| US11256695B1 (en) | Hybrid query execution engine using transaction and analytical engines | |
| US20140237087A1 (en) | Service pool for multi-tenant applications | |
| CN107220265B (en) | Database statement compiling and executing method and device | |
| US11816073B1 (en) | Asynchronously forwarding database commands | |
| US10200301B1 (en) | Logical control groups for distributed system resources | |
| CN112685451A (en) | Data query processing method and device, computer equipment and storage medium | |
| EP2990960A1 (en) | Data retrieval via a telecommunication network | |
| CN116910082A (en) | SQL sentence processing method, device, server and medium | |
| CN118567792A (en) | Running method and device of data query task, processor and electronic equipment | |
| CN111931033A (en) | Retrieval method, retrieval device and server |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CB03 | Change of inventor or designer information |
Inventor after: Li Peng Inventor after: Gao Hailing Inventor after: Yu Yang Inventor after: Gao Jingjun Inventor before: Li Peng Inventor before: Yu Yang Inventor before: Gao Jingjun |
|
| CB03 | Change of inventor or designer information |