CN117827953A - Data fusion method, system, equipment and medium based on distributed service - Google Patents
Data fusion method, system, equipment and medium based on distributed service Download PDFInfo
- Publication number
- CN117827953A CN117827953A CN202311865964.8A CN202311865964A CN117827953A CN 117827953 A CN117827953 A CN 117827953A CN 202311865964 A CN202311865964 A CN 202311865964A CN 117827953 A CN117827953 A CN 117827953A
- Authority
- CN
- China
- Prior art keywords
- data
- target
- service
- request
- data fusion
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000007500 overflow downdraw method Methods 0.000 title claims abstract description 36
- 230000004927 fusion Effects 0.000 claims abstract description 132
- 238000004458 analytical method Methods 0.000 claims abstract description 18
- 235000006719 Cassia obtusifolia Nutrition 0.000 claims description 8
- 235000014552 Cassia tora Nutrition 0.000 claims description 8
- 244000201986 Cassia tora Species 0.000 claims description 8
- 238000000034 method Methods 0.000 abstract description 29
- 230000008569 process Effects 0.000 abstract description 12
- 238000012545 processing Methods 0.000 abstract description 11
- 238000004321 preservation Methods 0.000 abstract 1
- 230000005540 biological transmission Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 4
- 230000004069 differentiation Effects 0.000 description 3
- 238000003491 array Methods 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 239000004973 liquid crystal related substance Substances 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 239000003795 chemical substances by application Substances 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000007499 fusion processing Methods 0.000 description 1
- 238000012946 outsourcing Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to a data fusion method, a system, equipment and a medium based on distributed service, wherein the method is characterized in that a preset configuration file and an analysis tool package corresponding to the preset configuration file are acquired, a remote call service request of a target service is acquired based on the analysis tool package and initialized, a target routing rule, a target call mode and a target data fusion strategy corresponding to the target service are acquired, request data of the remote call service request are acquired and fused, and the request data are returned to a requesting party of the remote call service request. The method avoids the processing of data which does not need to be fused or stored in a distributed mode, and omits unnecessary processes of comparison, fusion and the like, thereby improving the preservation efficiency and accuracy of the data and improving the freshness, accuracy and comprehensiveness of the fused data when the big data is applied.
Description
Technical Field
The present invention relates to the field of distributed technologies, and in particular, to a data fusion method, system, device, and medium based on distributed services.
Background
In the process of building a large data platform, a specific tool is generally used for data aggregation, and then data cleaning and data fusion are carried out. However, because of uncertainty of each service system, various data sources are generally required to be adapted first and then data is gathered in a data warehouse, and problems such as untimely data synchronization, interruption of data synchronization, repeated data, missing data and the like are caused in the gathering process of the data, so that subsequent data fusion is affected, and finally, the effect is not ideal when the data are output to each big data application.
For example, data of a certain industry is generally generated by a market-level business system first, then pushed or extracted into a provincial data warehouse, and the provincial data warehouse is pushed or extracted into a provincial data warehouse after gathering data of all local markets of the provincial. The data convergence links existing nationwide are numerous, and some data can only exist in a business system of a certain city or in a provincial data warehouse after a period of time, and cannot be converged into a data warehouse of a higher level in time, so that the data convergence links are not comprehensive enough. When large data application in the city or province is used, the local data and the national data need to be queried simultaneously in order to pursue the comprehensiveness, and the problem of preferential use of the same data exists. The same piece of data is the most fresh and accurate in local, and the local data should preferably cover the data returned by the national interface; and the local area has no data of foreign city or province, so that the whole province or national interface data is needed. Therefore, the freshness, accuracy and comprehensiveness of the data consulted by the user can be guaranteed to the greatest extent.
In summary, the data fusion effect caused by various reasons in the traditional data convergence process is not ideal, and a brand-new data fusion method is needed to improve the freshness, accuracy and comprehensiveness of the fused data in the application of big data.
Disclosure of Invention
In view of the foregoing, it is necessary to provide a data fusion method, system, device and medium based on distributed services, so as to solve the problem of how to improve the freshness, accuracy and comprehensiveness of data when the distributed big data is applied in the prior art.
In order to achieve the technical purpose, the invention adopts the following technical scheme:
in a first aspect, the present invention provides a data fusion method based on distributed services, including:
acquiring a preset configuration file and an analysis tool pack corresponding to the preset configuration file, wherein the preset configuration file is used for configuring various distributed service protocol data, various routing rule data, various calling mode data and various data fusion strategies;
analyzing the preset configuration file based on the analysis tool pack and initializing a system for running the data fusion method based on the distributed service;
acquiring a remote call service request of a target service, and acquiring a target routing rule, a target call mode and a target data fusion strategy corresponding to the target service based on the configured distributed service protocol data and the remote call service request;
and acquiring and fusing the request data of the remote call service request based on the target routing rule, the target call mode and the target data fusion strategy, and returning the request data to a requester of the remote call service request.
Further, the obtaining and fusing the request data of the remote call service request based on the target routing rule, the target call mode and the target data fusion policy, and returning the request data to the requester of the remote call service request includes:
obtaining a strategy of calling local service and remote service by the target service according to the target routing rule and the target calling mode;
judging the corresponding relation between the return result of the target remote call service request and the target call mode based on the strategies of the target service call local service and the remote service;
and acquiring and fusing request data of the remote call service request through the target data fusion strategy based on the corresponding relation between the return result of the target remote call service request and the target call mode, and returning the request data to a requester of the remote call service request.
Further, the plurality of distributed service protocol data includes DUBBO protocol, RMI protocol, and restul protocol; the plurality of routing rule data comprises default routing rules and custom conditional routing rules; the plurality of call mode data comprises a local call mode only and a remote call mode only, a local call priority remote call model and a remote call priority local call mode; the data fusion strategies comprise a general merging strategy, a linked list merging strategy, an combination merging strategy, an overlay strategy, a JavaBean type data merging strategy and a custom merging strategy.
Further, the preset configuration file comprises a predefined DUBBO service registry, a global default routing rule, a global default call mode, a global default data fusion policy and service interface information.
Further, the preset configuration file further comprises an optional routing rule, an optional calling mode and an optional data fusion strategy.
In a second aspect, the present invention also provides a data fusion system based on distributed services, including:
the configuration module is used for acquiring a preset configuration file and an analysis tool package corresponding to the preset configuration file, wherein the preset configuration file is used for configuring various distributed service protocol data, various routing rule data, various calling mode data and various data fusion strategies;
the initialization module is used for analyzing the preset configuration file based on the analysis tool package and initializing a system for running the data fusion method based on the distributed service;
the request interception module is used for acquiring a remote call service request of a target service and acquiring a target routing rule, a target call mode and a target data fusion strategy corresponding to the target service based on the configured distributed service protocol data and the remote call service request;
and the data return module is used for acquiring and fusing the request data of the remote call service request based on the target routing rule, the target call mode and the target data fusion strategy, and returning the request data to a requester of the remote call service request.
In a third aspect, the invention also provides an electronic device comprising a memory and a processor, wherein,
a memory for storing a program;
and a processor coupled to the memory for executing the program stored in the memory to implement the steps in the distributed service based data fusion method in any of the implementations described above.
In a fourth aspect, the present invention further provides a computer readable storage medium storing a computer readable program or instructions, where the program or instructions, when executed by a processor, implement the steps in a distributed service-based data fusion method in any of the above implementations.
The invention provides a data fusion method, a system, equipment and a medium based on distributed services, wherein the method firstly acquires a preset configuration file and an analysis tool pack corresponding to the preset configuration file, wherein the preset configuration file is used for configuring various distributed service protocol data, various routing rule data, various calling mode data and various data fusion strategies, and then analyzes the preset configuration file based on the analysis tool pack and initializes a system for running the data fusion method based on the distributed services. And finally, acquiring and fusing the request data of the remote call service request based on the target routing rule, the target call mode and the target data fusion strategy, and returning the request data to a requesting party of the remote call service request. Compared with the prior art, the invention realizes the differentiation processing of different data by pre-configuring a plurality of distributed service protocol data, a plurality of routing rule data, a plurality of calling mode data and a plurality of data fusion strategies and selecting a proper target routing rule, a target calling mode and a target data fusion strategy when in application. For data such as data which do not need to be fused and are not stored in a distributed mode and the like are directly released, the data in different calling modes are directly processed based on the modes needed by the data, unnecessary comparison, fusion and other processes are omitted, the resources of a computer are only applied to the data which need to be fused, the storage efficiency and accuracy of the data are greatly improved, and the freshness, accuracy and comprehensiveness of the data in big data application after fusion are improved.
Drawings
FIG. 1 is a flow chart of a method for data fusion based on distributed services according to an embodiment of the present invention;
FIG. 2 is a system architecture diagram of an embodiment of a distributed service-based data fusion system provided by the present invention;
fig. 3 is a schematic structural diagram of an embodiment of an electronic device according to the present invention.
Detailed Description
Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form a part hereof, and together with the description serve to explain the principles of the invention, and are not intended to limit the scope of the invention.
It is to be understood that technical terms, acronyms, and the like appearing hereinafter are prior art and those skilled in the art are able to understand their meanings based on context and are not described here too much for reasons of brevity.
In the description of the present application, the meaning of "a plurality" is two or more, unless explicitly defined otherwise.
Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the invention. The appearances of such phrases in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those of skill in the art will explicitly and implicitly appreciate that the embodiments described herein may be combined with other embodiments.
The invention provides a data fusion method, a device, equipment and a storage medium based on distributed services, which are respectively described below.
Referring to fig. 1, in one embodiment of the present invention, a data fusion method based on distributed services is disclosed, including:
s101, acquiring a preset configuration file and an analysis tool pack corresponding to the preset configuration file, wherein the preset configuration file is used for configuring various distributed service protocol data, various routing rule data, various calling mode data and various data fusion strategies;
s102, analyzing the preset configuration file based on the analysis tool package and initializing a system for running the data fusion method based on the distributed service;
s103, acquiring a remote call service request of a target service, and acquiring a target routing rule, a target call mode and a target data fusion strategy corresponding to the target service based on the configured distributed service protocol data and the remote call service request;
s104, acquiring and fusing the request data of the remote call service request based on the target routing rule, the target call mode and the target data fusion strategy, and returning the request data to a requester of the remote call service request.
Compared with the prior art, the invention realizes the differentiation processing of different data by pre-configuring a plurality of distributed service protocol data, a plurality of routing rule data, a plurality of calling mode data and a plurality of data fusion strategies and selecting a proper target routing rule, a target calling mode and a target data fusion strategy when in application. For data such as data which do not need to be fused and are not stored in a distributed mode and the like are directly released, the data in different calling modes are directly processed based on the modes needed by the data, unnecessary comparison, fusion and other processes are omitted, the resources of a computer are only applied to the data which need to be fused, the storage efficiency and accuracy of the data are greatly improved, and the freshness, accuracy and comprehensiveness of the data in big data application after fusion are improved.
Further, in a preferred embodiment, the plurality of distributed service protocol data includes DUBBO protocol, RMI protocol, and restul protocol; the plurality of routing rule data comprises default routing rules and custom conditional routing rules; the plurality of call mode data comprises a local call mode only and a remote call mode only, a local call priority remote call model and a remote call priority local call mode; the data fusion strategies comprise a general merging strategy, a linked list merging strategy, an combination merging strategy, an overlay strategy, a JavaBean type data merging strategy and a custom merging strategy.
The DUBBO protocol is a Java-based high-performance, simple and easy-to-use distributed service framework. The Remote Procedure Call (RPC) communication mode is adopted, data transmission is carried out through a TCP protocol of a transmission layer, multiple serialization modes and load balancing strategies are supported, and remote call among different services can be realized.
The RMI protocol (Remote Method Invocation, remote method call) is a communication protocol in the Java platform that implements distributed objects. The Java virtual machine system allows remote call among objects on different Java virtual machines, enables Java objects distributed on different machines to be called like local objects, and achieves transparent access of distributed objects.
The RESTFUL protocol (Representational State Transfer, presentation layer state transition) is a style of software architecture based on the HTTP protocol for building a distributed system. It provides a set of constraints and principles to facilitate the design and implementation of a scalable, high-performance, reliable and useful network system. The restul protocol uses standard HTTP methods (e.g., GET, POST, PUT, DELETE) to access and manipulate resources, locates resources through URIs (uniform resource identifiers), and uses JSON or XML formats for data transmission and representation.
In the data fusion strategy:
the general merging strategy is a basic data fusion strategy, which merges data from different sources, and may cause data redundancy or conflict. This strategy typically merges data according to certain rules, but may not handle complex merge situations for a particular data type.
Linked list merge strategies are commonly used to handle data fusion of ordered lists. The method combines a plurality of ordered linked lists according to respective ordering rules to generate an integral ordered linked list.
An array merge strategy merges multiple arrays into one larger array. This strategy is applicable to situations where multiple sets of data of the same type need to be consolidated into one whole.
The overwrite policy is used to overwrite a previous data item with a subsequent data item when the same data item is present at the time of data fusion. Such policies are often used in the case of updating or modifying data.
The JavaBean type data merging strategy is used for carrying out data fusion on JavaBean objects. The policy compares the attributes of the objects and merges or overlays the attributes according to certain rules.
The custom merge strategy is a user-defined data fusion strategy. In such a policy, a user may customize the rules and logic of data merging according to specific needs to meet specific fusion needs.
Further, in a preferred embodiment, the preset configuration file includes a predefined DUBBO service registry, a global default routing rule, a global default call mode, a global default data fusion policy, and service interface information.
Wherein:
the DUBBO service registry is an important component in the DUBBO framework for registering and managing services between different applications. Through the service registry, applications providing services can register their service information to the service registry, and applications requiring use of these services can find the corresponding services through the service registry and invoke them.
The global default service registry is a default configuration-based service registry in the DUBBO framework that is specified in the global configuration of the framework. If the application does not explicitly specify a service registry, then a global default service registry will be used.
The global default routing rule is a default configuration-based routing rule in the DUBBO framework that specifies on which servers the different services should perform operations. If the application does not explicitly specify a routing rule, then a global default routing rule will be used.
The global default call mode is a call mode based on default configuration in the DUBBO framework, and specifies the call mode (e.g., synchronous call or asynchronous call) that needs to be used by different services. If the application does not explicitly specify a call mode, then a global default call mode will be used.
The global default data fusion policy is a data fusion policy based on default configuration in the DUBBO framework, and specifies a merging mode to be used in data fusion. If the application does not explicitly specify a data fusion policy, then a global default data fusion policy will be used.
The service interface information is a set of information in the DUBBO framework for describing the service interface, including a service interface name, a method name, a parameter type, a return value type, and the like. Through the service interface information, the DUBBO framework can identify and manage different services and generate corresponding agents and call codes according to the interface information.
Further, in a preferred embodiment, the preset configuration file further includes an optional routing rule, an optional calling mode, and an optional data fusion policy. The optional routing rules, the optional calling modes and the optional data fusion policy are flexible configuration modes provided in the DUBBO framework, and the appropriate rules, modes and policies can be selected according to requirements. The routing rules are used for selecting service providers, the calling modes comprise synchronous calling, asynchronous calling and unidirectional calling, and the data fusion strategy is used for merging data items. Selecting appropriate rules, patterns, and policies helps to improve the efficiency and accuracy of service invocation and data fusion.
It will be appreciated that the selectable routing rules, the selectable call patterns, and the selectable data fusion policies are the types of the aforementioned plurality of routing rule data, the plurality of call pattern data, and the plurality of data fusion policies, respectively.
Further, in a preferred embodiment, the obtaining a remote call service request of the target service, and obtaining, based on the configured distributed service protocol data and the remote call service request, a target routing rule, a target call mode and a target data fusion policy corresponding to the target service specifically includes:
intercepting the remote call service request for obtaining the target service, judging whether the target service is a preset distributed service data fusion cache object or not according to the remote call service request based on the configured distributed service protocol data;
if not, directly releasing the remote call service request;
if yes, obtaining a target routing rule, a target calling mode and a target data fusion strategy corresponding to the target service based on the configured distributed service protocol data and the remote calling service request.
The process can determine whether the data fusion caching is needed or not by intercepting the remote call service request of the required target service and judging according to the preset distributed service protocol data. If the required target service is not the preset distributed service data fusion cache object, the remote call service request can be directly released without additional processing, so that the data transmission rate is improved. Thus, the request of the non-preset cache object can be guaranteed to be rapidly transmitted, and unnecessary delay is avoided. If the required target service is determined to be the preset distributed service data fusion cache object, the target routing rule, the target calling mode and the target data fusion policy need to be acquired based on the configured distributed service protocol data and the remote calling service request. These target configuration information will be used for subsequent service invocation and data fusion procedures.
Through the above embodiment, under the condition that the preset condition is met, the efficiency of data transmission can be optimized through processing the data fusion cache object, and the request of the non-preset cache object can be rapidly processed. Therefore, the performance and response speed of the whole system can be improved, and the accuracy and the high efficiency of data transmission and processing are ensured.
Further, in a preferred embodiment, the obtaining and fusing the request data of the remote invocation service request based on the target routing rule, the target invocation mode and the target data fusion policy, and returning the request data to the requester of the remote invocation service request specifically includes:
obtaining a strategy of calling local service and remote service by the target service according to the target routing rule and the target calling mode;
judging the corresponding relation between the return result of the target remote call service request and the target call mode based on the strategies of the target service call local service and the remote service;
and acquiring and fusing request data of the remote call service request through the target data fusion strategy based on the corresponding relation between the return result of the target remote call service request and the target call mode, and returning the request data to a requester of the remote call service request.
Specifically, in one preferred embodiment, the obtaining and fusing, by the target data fusion policy, the request data of the remote call service request based on the correspondence between the returned result of the target remote call service request and the target call mode in the foregoing process, and returning the request data to the requester of the remote call service request specifically includes:
judging whether one return result corresponds to multiple calling modes in the target calling modes or not based on the corresponding relation between the return result of the target remote calling service request and the target calling modes;
if not, directly acquiring request data of the remote call service request based on the target call mode, and returning the request data to a requester of the remote call service request;
if yes, acquiring and fusing the request data of the remote call service request through the target data fusion strategy, and returning the request data to a requester of the remote call service request.
And the process selects a proper calling mode to return data, and for the data which does not need to be fused, the omitted fusion process determines the strategy of calling the local service and the remote service by the target service according to the target routing rule and the target calling mode. And judging the corresponding relation between the returned result of the target remote call service request and the target call mode according to the strategy.
If the returned result and the target calling mode have only one corresponding relation, the request data of the remote calling service request is directly obtained according to the target calling mode and returned to the requester. And if the returned result and the target call mode are judged to have various corresponding relations, acquiring and fusing the request data of the remote call service request according to the target data fusion strategy, and returning the request data to the requester. Therefore, the performance and response speed of the whole system can be further improved, and meanwhile, the accuracy and the high efficiency of data are ensured.
Through the optimized embodiment, based on the target routing rule, the target calling mode and the target data fusion strategy, the request data of the remote calling service request are obtained and fused, and the result is returned to the requester. Therefore, the fusion and acquisition processes of the data can be ensured to be more flexible, efficient and accurate, and the actual requirements are met.
The present invention also provides a more detailed embodiment for more clearly illustrating the above-described process:
step S1, presetting a distributed service protocol, a routing rule, a calling mode and a data fusion strategy supported by a system:
the distributed service protocol supported by the preset system of the embodiment may include DUBBO protocol, RMI protocol, restul protocol, etc., and the distributed service supported by the preset system described herein is only used for explaining the embodiment, and is not used for limiting the embodiment, and can be expanded in a self-defined manner;
the preset routing rule in this embodiment includes a default route and a custom condition route, where the default route directly takes the configuration of the calling mode, and the condition rule can be custom-applied to different calling modes according to parameter conditions.
The preset calling modes of the embodiment comprise a local calling mode and a remote calling mode, wherein the local calling is preferential to the remote calling, and the remote calling is preferential to the local calling;
the preset data fusion policy in this embodiment includes a general merge policy, a linked list merge policy, an array merge policy, an overlay policy, a java bean type data merge policy, and a custom merge policy. Data fusion logic implementations, such as arrays, linked lists, linear tables, javabeans, etc., that additionally contain some common data structure objects.
Step S2, writing a preset configuration file and a corresponding analysis tool kit which need to be subjected to distributed service data fusion:
the preset distributed service protocol in this embodiment is, for example, a DUBBO protocol, and the preset configuration file needs to include a predefined DUBBO service registry (generally at least one local and one remote), so that the subsequent system connects to the corresponding registry to perform distributed service remote call. The system also comprises a global default service registry, a global default routing rule, a global default calling mode and a global default data fusion strategy. In addition, the configuration file written in the invention also needs to contain specific service interface full path names (and service interface information), method names (optional: all methods in the default interface are not selected), optional routing rules (optional: routing rules not selected as global default), optional call modes (optional: call modes not selected as global default), and optional data fusion strategies (optional: data fusion strategies not selected as global default).
And S3, analyzing the data fusion configuration file when a system of the data fusion method based on the distributed service is initialized, and caching a service interface and a corresponding method name, a routing rule, a calling mode and a data fusion strategy. When the configuration file is updated, supporting real-time dynamic refreshing of cache data:
and S4, intercepting a remote call service request, judging whether the service is in a distributed service data fusion cache object, directly releasing if the service is not in the distributed service data fusion cache object, and acquiring a corresponding service routing rule and a corresponding call mode (namely a target routing rule and a target call mode) if the service is in the distributed service data fusion cache object.
Step S5, judging how to call the local service and the remote service according to the routing rule and the call mode, namely obtaining the strategy of calling the local service and the remote service by the target service:
example 1: when a certain service interface is used for searching the basic information of the motor vehicles nationwide according to the number plate number in a fuzzy way, the routing rule configured by the service is a custom condition rule, and the calling mode is local priority remote. The custom condition is that only local services are invoked when the request parameters start with "jaw a", otherwise remote services are invoked.
Example 2: when a certain service interface is used for inquiring basic information of national personnel according to names and the date of birth, the routing rule configured by the service is a default rule, and the calling mode is local priority remote. The local service and the remote service are finally invoked, and the two invoking results are stored in a linear table to wait for data fusion.
Step S6, judging the corresponding relation between the return result of the target remote call service request and the target call mode, if only one mode is directly returned in the mode call, if a plurality of modes exist, carrying out data fusion after acquiring the data fusion strategy:
example 1: the calling mode of a certain service configuration is remote or local only, and at the moment, the mode calling has only one mode, so that a result can be directly returned without data fusion.
Example 2: and if the calling mode of the certain service configuration is local calling and preferential remote calling and the data fusion strategy is a JavaBean type data merging strategy, the local service and the remote service are required to be called simultaneously, and the returned writable and readable attribute values of the local result object are all rewritten into the corresponding attribute of the remote result object. For example, the personnel A exists in a return result of local service call, the marital attribute of the personnel A is married, and the household address attribute is the household address immediately after the last mobile update; the marital attribute of the personnel A in the remote service calling return result is not married, the household address attribute is the household address before household change, and the combined personnel A information is more fresh and more accurate than the remote service calling result. The personnel B is an outsourcing personnel, the local service call return result does not contain the personnel, the remote service call return result contains the personnel and can be directly returned, and the information of the personnel B after data fusion is more comprehensive than the result of the local service call only.
And S7, returning after data fusion.
In order to better implement the data fusion method based on the distributed service in the embodiment of the present invention, referring to fig. 2 correspondingly, fig. 2 is a schematic structural diagram of an embodiment of a data fusion system based on the distributed service provided by the present invention, where the data fusion system 200 based on the distributed service provided by the embodiment of the present invention includes:
the configuration module 210 is configured to obtain a preset configuration file and an analysis tool package corresponding to the preset configuration file, where the preset configuration file is configured to configure multiple distributed service protocol data, multiple routing rule data, multiple call mode data, and multiple data fusion policies;
an initialization module 220, configured to parse the preset configuration file based on the parsing tool package and initialize a system for running the data fusion method based on the distributed service;
the request interception module 230 is configured to obtain a remote call service request of a target service, and obtain a target routing rule, a target call mode and a target data fusion policy corresponding to the target service based on the configured distributed service protocol data and the remote call service request;
the data return module 240 is configured to obtain and fuse the request data of the remote call service request based on the target routing rule, the target call mode and the target data fusion policy, and return the request data to the requester of the remote call service request.
What needs to be explained here is: the corresponding system 200 provided in the foregoing embodiments may implement the technical solutions described in the foregoing method embodiments, and the specific implementation principles of the foregoing modules or units may be referred to the corresponding content in the foregoing method embodiments, which is not repeated herein.
Referring to fig. 3, fig. 3 is a schematic structural diagram of an electronic device according to an embodiment of the invention. Based on the above data fusion method based on the distributed service, the present invention further provides a data fusion device 300 based on the distributed service, that is, the electronic device, where the data fusion device 300 based on the distributed service may be a computing device such as a mobile terminal, a desktop computer, a notebook computer, a palm computer, and a server. The distributed service based data fusion device 300 includes a processor 310, a memory 320, and a display 330. Fig. 3 shows only some of the components of the distributed service based data fusion device, but it should be understood that not all of the illustrated components need be implemented, and more or fewer components may alternatively be implemented.
The memory 320 may be an internal storage unit of the distributed service based data fusion device 300 in some embodiments, such as a hard disk or a memory of the distributed service based data fusion device 300. The memory 320 may also be an external storage device of the distributed service-based data fusion device 300 in other embodiments, for example, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash Card (Flash Card) or the like, which are provided on the distributed service-based data fusion device 300. Further, the memory 320 may also include both internal and external storage units of the distributed service based data fusion device 300. The memory 320 is used for storing application software installed in the distributed service-based data fusion apparatus 300 and various types of data, such as program codes installed in the distributed service-based data fusion apparatus 300. The memory 320 may also be used to temporarily store data that has been output or is to be output. In an embodiment, the memory 320 stores a data fusion program 340 based on a distributed service, where the data fusion program 340 based on the distributed service may be executed by the processor 310, so as to implement the data fusion method based on the distributed service according to embodiments of the present application.
The processor 310 may in some embodiments be a central processing unit (Central Processing Unit, CPU), microprocessor or other data processing chip for executing program code or processing data stored in the memory 320, such as performing a distributed service based data fusion method or the like.
The display 330 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch, or the like in some embodiments. The display 330 is used to display information at the distributed service-based data fusion device 300 and to display a visual user interface. The components 310-330 of the distributed service based data fusion device 300 communicate with each other over a system bus.
In one embodiment, the steps in the distributed service-based data fusion method as described above are implemented when the processor 310 executes the distributed service-based data fusion program 340 in the memory 320.
The present embodiment also provides a computer-readable storage medium having stored thereon a distributed service-based data fusion program which, when executed by a processor, implements the steps of the above embodiments.
The invention provides a data fusion method, a system, equipment and a medium based on distributed services, wherein the method firstly acquires a preset configuration file and an analysis tool pack corresponding to the preset configuration file, wherein the preset configuration file is used for configuring various distributed service protocol data, various routing rule data, various calling mode data and various data fusion strategies, and then analyzes the preset configuration file based on the analysis tool pack and initializes a system for running the data fusion method based on the distributed services. And finally, acquiring and fusing the request data of the remote call service request based on the target routing rule, the target call mode and the target data fusion strategy, and returning the request data to a requesting party of the remote call service request. Compared with the prior art, the invention realizes the differentiation processing of different data by pre-configuring a plurality of distributed service protocol data, a plurality of routing rule data, a plurality of calling mode data and a plurality of data fusion strategies and selecting a proper target routing rule, a target calling mode and a target data fusion strategy when in application. For data such as data which do not need to be fused and are not stored in a distributed mode and the like are directly released, the data in different calling modes are directly processed based on the modes needed by the data, unnecessary comparison, fusion and other processes are omitted, the resources of a computer are only applied to the data which need to be fused, the storage efficiency and accuracy of the data are greatly improved, and the freshness, accuracy and comprehensiveness of the data in big data application after fusion are improved.
The present invention is not limited to the above-mentioned embodiments, and any changes or substitutions that can be easily understood by those skilled in the art within the technical scope of the present invention are intended to be included in the scope of the present invention.
Claims (10)
1. A distributed service-based data fusion method, comprising:
acquiring a preset configuration file and an analysis tool pack corresponding to the preset configuration file, wherein the preset configuration file is used for configuring various distributed service protocol data, various routing rule data, various calling mode data and various data fusion strategies;
analyzing the preset configuration file based on the analysis tool pack and initializing a system for running the data fusion method based on the distributed service;
acquiring a remote call service request of a target service, and acquiring a target routing rule, a target call mode and a target data fusion strategy corresponding to the target service based on the configured distributed service protocol data and the remote call service request;
and acquiring and fusing the request data of the remote call service request based on the target routing rule, the target call mode and the target data fusion strategy, and returning the request data to a requester of the remote call service request.
2. The distributed service-based data fusion method according to claim 1, wherein the obtaining the remote call service request of the target service, and obtaining the target routing rule, the target call mode and the target data fusion policy corresponding to the target service based on the configured distributed service protocol data and the remote call service request, includes:
intercepting the remote call service request for obtaining the target service, judging whether the target service is a preset distributed service data fusion cache object or not according to the remote call service request based on the configured distributed service protocol data;
if not, directly releasing the remote call service request;
if yes, obtaining a target routing rule, a target calling mode and a target data fusion strategy corresponding to the target service based on the configured distributed service protocol data and the remote calling service request.
3. The distributed service-based data fusion method according to claim 1, wherein the obtaining and fusing the request data of the remote invocation service request based on the target routing rule, the target invocation mode, and the target data fusion policy, and returning the request data to the requestor of the remote invocation service request, includes:
obtaining a strategy of calling local service and remote service by the target service according to the target routing rule and the target calling mode;
judging the corresponding relation between the return result of the target remote call service request and the target call mode based on the strategies of the target service call local service and the remote service;
and acquiring and fusing request data of the remote call service request through the target data fusion strategy based on the corresponding relation between the return result of the target remote call service request and the target call mode, and returning the request data to a requester of the remote call service request.
4. The distributed service-based data fusion method according to claim 3, wherein the obtaining and fusing, by the target data fusion policy, the request data of the remote invocation service request based on the correspondence between the returned result of the target remote invocation service request and the target invocation mode, and returning the request data to the requestor of the remote invocation service request includes:
judging whether one return result corresponds to multiple calling modes in the target calling modes or not based on the corresponding relation between the return result of the target remote calling service request and the target calling modes;
if not, directly acquiring request data of the remote call service request based on the target call mode, and returning the request data to a requester of the remote call service request;
if yes, acquiring and fusing the request data of the remote call service request through the target data fusion strategy, and returning the request data to a requester of the remote call service request.
5. The distributed service-based data fusion method of claim 1, wherein the plurality of distributed service protocol data includes DUBBO protocol, RMI protocol, and restul protocol; the plurality of routing rule data comprises default routing rules and custom conditional routing rules; the plurality of call mode data comprises a local call mode only and a remote call mode only, a local call priority remote call model and a remote call priority local call mode; the data fusion strategies comprise a general merging strategy, a linked list merging strategy, an combination merging strategy, an overlay strategy, a JavaBean type data merging strategy and a custom merging strategy.
6. The distributed service-based data fusion method of claim 1, wherein the preset configuration file comprises a predefined DUBBO service registry, a global default routing rule, a global default call mode, a global default data fusion policy, and service interface information.
7. The distributed service-based data fusion method of claim 6, wherein the preset configuration file further comprises an optional routing rule, an optional calling mode, and an optional data fusion policy.
8. A distributed service-based data fusion system, comprising:
the configuration module is used for acquiring a preset configuration file and an analysis tool package corresponding to the preset configuration file, wherein the preset configuration file is used for configuring various distributed service protocol data, various routing rule data, various calling mode data and various data fusion strategies;
the initialization module is used for analyzing the preset configuration file based on the analysis tool package and initializing a system for running the data fusion method based on the distributed service;
the request interception module is used for acquiring a remote call service request of a target service and acquiring a target routing rule, a target call mode and a target data fusion strategy corresponding to the target service based on the configured distributed service protocol data and the remote call service request;
and the data return module is used for acquiring and fusing the request data of the remote call service request based on the target routing rule, the target call mode and the target data fusion strategy, and returning the request data to a requester of the remote call service request.
9. An electronic device comprising a memory and a processor, wherein,
the memory is used for storing programs;
the processor, coupled to the memory, for executing the program stored in the memory to implement the steps in the distributed service based data fusion method of any of the above claims 1 to 7.
10. A computer readable storage medium storing a computer readable program or instructions which, when executed by a processor, is capable of carrying out the steps of the distributed service based data fusion method of any one of the preceding claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311865964.8A CN117827953A (en) | 2023-12-28 | 2023-12-28 | Data fusion method, system, equipment and medium based on distributed service |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311865964.8A CN117827953A (en) | 2023-12-28 | 2023-12-28 | Data fusion method, system, equipment and medium based on distributed service |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117827953A true CN117827953A (en) | 2024-04-05 |
Family
ID=90514961
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311865964.8A Pending CN117827953A (en) | 2023-12-28 | 2023-12-28 | Data fusion method, system, equipment and medium based on distributed service |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117827953A (en) |
-
2023
- 2023-12-28 CN CN202311865964.8A patent/CN117827953A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8332520B2 (en) | Web server for managing session and method thereof | |
| US6442565B1 (en) | System and method for transmitting data content in a computer network | |
| US7269664B2 (en) | Network portal system and methods | |
| US20200110756A1 (en) | Data delivery architecture for transforming client response data | |
| US8117280B2 (en) | Task computing | |
| US7673029B2 (en) | Grid automation bus to integrate management frameworks for dynamic grid management | |
| US10091086B2 (en) | System and method for providing an application programming interface manager for use with a service bus runtime | |
| CN105765578B (en) | Parallel access of data in a distributed file system | |
| US20040117452A1 (en) | XML-based network management system and method for configuration management of heterogeneous network devices | |
| US20030055878A1 (en) | Programmatic management of software resources in a content framework environment | |
| CN110413288A (en) | Using dispositions method, device, server and storage medium | |
| JPH04230567A (en) | Dispersed type constitution profile for computing system | |
| US10102230B1 (en) | Rate-limiting secondary index creation for an online table | |
| US20140237485A1 (en) | System and method for aggregate data from multiple sources to provide a single cim object | |
| US8978041B2 (en) | Service virtualization container | |
| US20030115243A1 (en) | Distributed process execution system and method | |
| US20030115575A1 (en) | Method and system for sharing resources in hierarchical backplanes | |
| US20120143949A1 (en) | Achieving interoperability for web services using a map engine | |
| US20230027121A1 (en) | System and Method for Aggregating Data in a Remote Address Space | |
| US20100332604A1 (en) | Message selector-chaining | |
| CN117827953A (en) | Data fusion method, system, equipment and medium based on distributed service | |
| CN116032614A (en) | Container network micro-isolation method, device, equipment and medium | |
| US10114864B1 (en) | List element query support and processing | |
| US20110138288A1 (en) | Method, system, and computer program product for tagging of portlets in a portal infrastructure | |
| US20090228900A1 (en) | Systems and methods for application programming using persistent objects |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |